Compare commits

...

1273 Commits

Author SHA1 Message Date
Alexander Neumann
40791fff64 Add version for 0.13.0 2022-03-26 20:09:59 +01:00
Alexander Neumann
a53a4a23fd Update manpages and auto-completion 2022-03-26 20:09:59 +01:00
Alexander Neumann
b567c08e80 Generate CHANGELOG.md for 0.13.0 2022-03-26 20:09:40 +01:00
Alexander Neumann
0ca89b6fec Prepare changelog for 0.13.0 2022-03-26 20:09:39 +01:00
Alexander Neumann
d7e46c187a Merge pull request #3678 from restic/changelogs
Review, reword and polish unreleased changelog entries
2022-03-26 19:44:30 +01:00
Alexander Neumann
6aefe3e111 Merge pull request #3677 from restic/3490-polish
check: Adjust help and documentation for check --read-data-subset
2022-03-26 19:34:51 +01:00
Leo R. Lundgren
03137a34db Review, reword and polish unreleased changelog entries 2022-03-26 13:01:53 +01:00
Leo R. Lundgren
c7d637ec39 check: Adjust help and documentation for check --read-data-subset 2022-03-26 00:11:04 +01:00
Alexander Neumann
6087c4ad75 Merge pull request #3656 from lgommans/forget-security
forget: Update docs for readability and append-only considerations
2022-03-24 21:36:19 +01:00
Leo R. Lundgren
cdf478c8f4 doc: More updates to forget documentation and security considerations 2022-03-23 23:12:19 +01:00
Luc Gommans
80969a6347 Update docs according to comments from MichaelEischer in PR #3656 2022-03-23 23:12:19 +01:00
Leo R. Lundgren
676d5d498c doc: Update forget security considerations and thread model 2022-03-23 23:12:19 +01:00
Luc Gommans
9c1d49e312 Document "forget" security considerations and add references
Removing data based on a policy when the attacker had the opportunity to
add data to your repository comes with some considerations. This is
added to the 060_forget.rst documentation.

That document is also updated to reflect that restic now considers
the current system time while running "forget".

References to the security considerations section are added:
- In `restic forget --help`
- In the threat model (design.rst)
- In the (030) setup section where an append-only setup is referenced

A reference is also to be added to the `rest-server` readme's
append-only paragraph (see my fork).

This commit also resolves a typo (amount->number for countable noun),
changes a password length recommendation into the metric that
actually matters when creating passwords (entropy) since I was editing
these doc files anyway, and updates the outdated copyright year in
`conf.py`.

Some wording in 060_forget (line 21..22) was changed to clarify what
"forget" and "prune" do, to try and avoid the apparent misconception
that "forget" does not remove any data.
2022-03-23 23:12:19 +01:00
Alexander Neumann
ca1e2316cf Merge pull request #3665 from MichaelEischer/sane-list-locks
list: Never lock the repository when listing lock files
2022-03-21 11:14:44 +01:00
Alexander Neumann
0b8b524f12 Merge pull request #3512 from MichaelEischer/cleaner-lock-refresh
Prevent lock refresh from leaving behind lots of stale locks
2022-03-21 11:10:37 +01:00
Alexander Neumann
a350625554 Merge pull request #3524 from MichaelEischer/atomic-sftp
sftp: Implement atomic uploads
2022-03-21 11:08:22 +01:00
Alexander Neumann
32e61f2620 Update changelog/unreleased/issue-1106
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-03-21 11:04:04 +01:00
Alexander Neumann
8388f66c4c Merge pull request #3668 from greatroar/symlink-size
Report symlink sizes from FUSE mount
2022-03-21 11:02:32 +01:00
Alexander Neumann
0937008648 Merge pull request #3654 from MichaelEischer/limit-huge-tree-streams
Limit number of large tree blobs loaded in parallel by StreamTrees
2022-03-21 11:01:04 +01:00
Alexander Neumann
3a285f91bc Merge pull request #2311 from vincentbernat/feature/negative-pattern
filter: ability to use negative patterns
2022-03-20 14:02:30 +01:00
Alexander Neumann
29a5778626 Improve wording 2022-03-20 13:46:16 +01:00
Michael Eischer
53656f019a filter: address review comments 2022-03-20 13:33:08 +01:00
Michael Eischer
cd190bee14 filter: short circuit if no negative patterns 2022-03-20 13:33:08 +01:00
Vincent Bernat
2ee07ded2b filter: ability to use negative patterns
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix #233
2022-03-20 13:33:08 +01:00
Michael Eischer
12606b575f filter: Cleanup variable naming 2022-03-20 13:33:08 +01:00
Michael Eischer
5f145f0c7e filter: introduce pattern struct 2022-03-20 13:33:08 +01:00
Vincent Bernat
13c40d4199 filter: additional tests for filter.List() 2022-03-20 13:33:08 +01:00
Alexander Neumann
13aae82635 Merge pull request #3673 from restic/update-go
Update go
2022-03-20 12:02:32 +01:00
Alexander Neumann
b85d035956 Fix calens install step 2022-03-20 11:36:45 +01:00
Alexander Neumann
47aa4613f7 Force Go to use Module Mode 2022-03-20 11:30:01 +01:00
Alexander Neumann
a9a5acb8ce Update golangci-lint 2022-03-20 11:26:56 +01:00
Alexander Neumann
6dee59b789 Install gox before checking out code
Otherwise newer Go versions complain that the hash for the installed
version of gox is not in the go.mod, which we don't want anyways because
the tests should use the latest version of gox.
2022-03-20 11:26:56 +01:00
Alexander Neumann
2e19d19216 Use latest Go version for cross-compile and lint 2022-03-20 11:26:56 +01:00
Alexander Neumann
18a1de0de1 Use "go get" or "go install" selectively
Go 1.18 dropped support for installing binaries via "go get", Go <= 1.16
does not support it. So we need to use the right verb depending on the
Go version.
2022-03-20 11:26:56 +01:00
Alexander Neumann
9b57fcc6b0 Fix build.go, minimum Go version is 1.14 2022-03-20 10:54:33 +01:00
Alexander Neumann
17878036d8 Update tests to Go 1.18 2022-03-20 10:54:24 +01:00
Jason Lenz
2b1932a258 Report symlink sizes from FUSE mount for snapshot dir
Fixes #3667.
2022-03-17 22:21:47 -05:00
greatroar
fdc738fb70 Report symlink sizes from FUSE mount
Fixes #3667.
2022-03-13 16:48:35 +01:00
MichaelEischer
daea461f15 Merge pull request #3663 from jimt/find-msgs
Remove period from find messages
2022-03-07 22:23:49 +01:00
Jim Tittsler
a3d99217a4 Remove period from find messages
Simplifies cut-and-paste of IDs (and makes stylistically
consistent with other messages) #3659
2022-03-07 11:16:04 +09:00
MichaelEischer
e0ab689ccd Merge pull request #3664 from DanielG/fix-doc-warning
doc: Fix block quote warning
2022-03-06 21:56:26 +01:00
Michael Eischer
7af69fd7b9 list: Never lock the repository when listing lock files
There's no point in locking the repository just to list the currently
existing lock files. This won't work for an exclusively locked
repository and is also confusing to users.
2022-03-06 21:44:51 +01:00
Daniel Gröber
49b67c8aaa doc: Fix block quote warning 2022-03-06 18:15:55 +01:00
rawtaz
44d543ede3 Merge pull request #3653 from MichaelEischer/fix-ls-option-help
ls: Fix description for --host, --tag and --path options
2022-02-19 23:06:24 +01:00
Michael Eischer
5ef4ee7760 ls: Fix description for --host, --tag and --path options 2022-02-19 22:36:02 +01:00
Michael Eischer
254c8743fc Limit number of large tree blobs loaded in parallel by StreamTrees
Load tree blobs with more than 50MB only from a single goroutine. Very
large tree blobs with for example 400 MB size can otherwise require
roughly 1GB * streamTreeParallelism memory.
2022-02-19 12:26:09 +01:00
MichaelEischer
ad4f4dbc7a Merge pull request #3645 from greatroar/stdin-parent
Don't set a parent for --stdin backups
2022-02-19 11:36:51 +01:00
greatroar
63f6a9b085 Don't set a parent for --stdin backups
Loading any parent tree for these only wastes time and memory.
Fixes #3641, where it was shown that the most recent tree will get
picked.

--parent is now implicitly ignored when --stdin is given.
2022-02-19 10:41:33 +01:00
MichaelEischer
4a2d5a146d Merge pull request #3507 from ahmgithubahm/document-AWS_PROFILE-support
Document AWS_PROFILE support
2022-02-18 23:40:25 +01:00
Michael Eischer
1efc26899d Update docs for AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE 2022-02-18 23:31:10 +01:00
MichaelEischer
8df246d0f3 Merge pull request #3628 from gum3ng/issue_3127
[#issue 3127] Add xattr support for Solaris
2022-02-17 22:07:39 +01:00
gum3ng
dd30083c2b [#issue 3127] Add xattr support for Solaris 2022-02-13 14:24:37 +05:30
MichaelEischer
fb4c5af5c4 Merge pull request #3642 from gco/master
Fix test failures on Solaris
2022-02-12 22:07:14 +01:00
MichaelEischer
18ec49ddfa Merge pull request #3644 from duritong/centos-epel-repo
add a note about installation via epel for RHEL / CentOS Stream 8 & 9
2022-02-12 21:41:46 +01:00
Michael Eischer
5ec312ca06 sftp: Implement atomic uploads
Create a temporary file with a sufficiently random name to essentially
avoid any chance of conflicts. Once the upload has finished remove the
temporary suffix. Interrupted upload thus will be ignored by restic.
2022-02-12 20:30:49 +01:00
Michael Eischer
aebd24e414 Add changelog 2022-02-12 20:21:58 +01:00
Michael Eischer
d72181c8c1 Ensure that the lock cleanup handler is run after the global one
cleanup handlers run in the order in which they are added. As Go calls
init() functions in lexical order, the cleanup handler from global.go
was registered before that from lock.go, which is the correct order.

Make this order explicit to ensure that this won't break accidentally.
2022-02-12 20:21:58 +01:00
Michael Eischer
c6fd13425b remember the refreshed lock file even if removal failed
This ensures that restic won't create lots of new lock files without
deleting them later on.

In some cases a Delete operation on a backend can return a "File does
not exist" error even though the Delete operation succeeded. This can
for example be caused by request retries. This caused restic to forget
about the new lock file and continue trying to remove the old (already
deleted) lock file.
2022-02-12 20:21:58 +01:00
MichaelEischer
cc90f2ba6b Merge pull request #2816 from greatroar/noatime
Set O_NOATIME flag on Linux
2022-02-07 21:38:31 +01:00
MichaelEischer
d8f58fb7bf Merge pull request #3592 from vgt/jsondiff
Add json output for diff command
2022-02-07 21:33:30 +01:00
duritong
a4786dda5a Update doc/020_installation.rst
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-02-06 21:31:18 +01:00
Michael Eischer
aaa7f94139 Add changelog for O_NOATIME 2022-02-06 15:00:37 +01:00
Michael Eischer
6b17a7110c backup: Set O_NOATIME in the right place
The archiver uses FS.OpenFile, where FS is an instance of the FS
interface. This is different from fs.OpenFile, which uses the OpenFile
method provided by the fs package.
2022-02-06 15:00:37 +01:00
greatroar
7080fed7ae Set O_NOATIME flag on Linux
Citing Kerrisk, The Linux Programming Interface:

    The O_NOATIME flag is intended for use by indexing and backup
    programs. Its use can significantly reduce the amount of disk
    activity, because repeated disk seeks back and forth across the
    disk are not required to read the contents of a file and to update
    the last access time in the file’s i-node[.]

restic used to do this, but the functionality was removed along with the
fadvise call in #670.
2022-02-06 15:00:34 +01:00
Michael Eischer
74f29ad09b diff: add basic test for json output format 2022-02-06 11:44:15 +01:00
Michael Eischer
5f34ad523f diff: fix test failure and add remark on quiet to changelog 2022-02-05 23:02:07 +01:00
MichaelEischer
58236ead12 Merge pull request #3619 from aneeshusa/avoid-time-travel-paradoxes-when-finding-parents
Avoid choosing parent snapshot newer than time of current snapshot
2022-02-05 22:51:24 +01:00
Michael Eischer
8ae4d86a84 rename snapshot timestamp filter variable 2022-02-05 22:42:38 +01:00
mh
3f0184ba2a add a note about installation via epel 2022-02-05 22:18:33 +01:00
rawtaz
90473ea9ff Merge pull request #3638 from leogott/patch-1
Documentation: Update msys2 wiki url after move
2022-01-27 17:07:45 +01:00
Leona 'leo' Gottfried
4e84e8ab3f Update msys2 wiki url after move
https://github.com/msys2/msys2/wiki/Porting was permanently moved and redirects to https://www.msys2.org/
I substituted the new location of the /wiki/Porting page in the docs
2022-01-27 11:09:50 +01:00
Greg
2e9180638e Fix test failures on Solaris
Add exceptions for symlinks, sticky bits, and device nodes in the same places where the BSDSs and/or Darwin have them.
2022-01-25 18:05:56 -08:00
Aneesh Agrawal
058dfc20da Avoid choosing parent snapshot newer than time of current snapshot
Currently, `restic backup` (if a `--parent` is not provided)
will choose the most recent matching snapshot as the parent snapshot.
This makes sense in the usual case,
where we tag the snapshot-being-created with the current time.

However, this doesn't make sense if the user has passed `--time`
and is currently creating a snapshot older than the latest snapshot.
Instead, choose the most recent snapshot
which is not newer than the snapshot-being-created's timestamp,
to avoid any time travel.

Impetus for this change:
I'm using restic for the first time!
I have a number of existing BTRFS snapshots
I am backing up via restic to serve as my initial set of backups.
I initially `restic backup`'d the most recent snapshot to test,
then started backing up each of the other snapshots.
I noticed in `restic cat snapshot <id>` output
that all the remaining snapshots have the most recent as the parent.
2022-01-23 23:55:00 -05:00
Aneesh Agrawal
502fc3281c Add CONTRIBUTING.md docs to not edit man pages
Document this code review feedback I got for other contributors.
2022-01-23 23:54:24 -05:00
David Vogt
77c850148a Add json output for diff command 2022-01-23 23:22:26 +01:00
MichaelEischer
df89aa0087 Merge pull request #3623 from invine/issue-3464
Skip lock file creation on forget with --no-lock and --dry-run
2022-01-23 18:12:34 +01:00
Pavel Frolov
792523b28b [issue 3464] skip lock creation in case of dry-run 2022-01-23 06:44:41 +03:00
MichaelEischer
f0a8182493 Merge pull request #3626 from fkusche/issue-3620-doc-unreferenced-packs
Update documentation regarding unreferenced packs
2022-01-22 15:15:25 +01:00
Florian Kusche
6183d0be53 Update output of restic check 2022-01-16 12:34:20 +01:00
rawtaz
7f6fc78f95 Merge pull request #3544 from restic/fix-b2-delete-retry
b2: Successful delete if file does not exist
2022-01-13 23:28:30 +01:00
rawtaz
abfbacf3d3 Merge pull request #3591 from MichaelEischer/prune-fix-max-repack
prune: Handle --max-repack-size=0 as expected
2022-01-13 03:53:20 +01:00
Florian Kusche
b0c1d0f9cd Update documentation regarding unreferenced packs
Also removes an unnecessary space at the end of the last line.
2022-01-09 11:45:03 +01:00
rawtaz
8b6fe845d4 Merge pull request #3618 from mattxtaz/master
Add missing colon in prune stats output and realign the fields
2022-01-06 23:07:56 +01:00
mattxtaz
6ff32ee4d3 Add missing colon in prune stats output and change padding to 14 chars to align the fields 2022-01-06 21:15:15 +00:00
rawtaz
2ff3b7d69c Merge pull request #3615 from gum3ng/issue_3558
doc: Add a FAQ section for invalid Windows filenames
2022-01-02 18:06:02 +01:00
gum3ng
9589de16db [issue 3558]: Add a FAQ section for invalid Windows filenames 2022-01-02 22:24:00 +05:30
MichaelEischer
2c3e5d943d Merge pull request #3593 from DarkKirb/parallelize-restic-copy
Parallelize blob upload/download for restic copy
2021-12-29 22:31:54 +01:00
Charlotte 🦝 Delenk
e2bb384a60 Parallelize blob upload/download for restic copy
Currently restic copy will copy each blob from every snapshot serially,
which has performance implications on high-latency backends such as b2.

This commit introduces 8x parallelism for blob downloads/uploads which
can improve restic copy operations up to 8x for repositories with many
small blobs on b2.

This commit also addresses the TODO comment in the copyTree function.

Related work:

A more thorough improvement of the restic copy performance can be found
in PR #3513
2021-12-29 18:59:09 +01:00
MichaelEischer
e5985e0d63 Merge pull request #3602 from cqjjjzr/fix-rclone-sigint
fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever (#3601)
2021-12-29 16:56:37 +01:00
MichaelEischer
8832837a8a Merge pull request #3607 from jtagcat/quieter_cleanup
logging: quiet 'removing n old cache dirs'
2021-12-29 16:06:35 +01:00
jtagcat
f92130d878 logging: quiet 'removing n old cache dirs'
Closes #3595

Choosing to include `stdoutIsTerminal()` as:
 - all other instances with `!opts.JSON` do so
 - this likely will not affect anything, especially when autorun
 - this seems to not be a meaningful enough summary
     to include in auto-backup reports

JSON is still likely not guaranteed to work and this is a suboptimal
  solution to this. Ideally, #1804 should refactor all print statements,
  and define+document(+handle) when stdoutIsTerminal() should be used.
  Else, it may end up more inconsistent and bulky
  (duplicate lines, longer files).
2021-12-29 01:08:29 +02:00
Charlie Jiang
a5b0e0bef4 fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2021-12-28 13:14:46 +08:00
rawtaz
e6e51b84ac Merge pull request #3605 from gum3ng/issue_3463
Improve clarity of error message in restic ls command
2021-12-27 21:45:23 +01:00
MichaelEischer
c5c3dfe10f Merge pull request #3590 from metalsp0rk/documentation-enhance
Document Safe Passwords and Clarify B2 App Key information
2021-12-27 20:21:43 +01:00
Kyle Brennan
19ec4d8f17 Document safe passwords. Fix #2238 2021-12-27 10:43:18 -08:00
Kyle Brennan
47ecd950b8 Enhance details about user application keys. Fix #2672 2021-12-27 10:43:15 -08:00
MichaelEischer
051cc7ce71 Merge pull request #3589 from metalsp0rk/copy-no-lock
Make Copy respect no lock
2021-12-27 19:11:02 +01:00
Michael Eischer
64e733f3d6 tweak copy --no-lock changelog 2021-12-27 18:22:25 +01:00
Gautam Menghani
017614c41a [#issue 3463] Improve clarity of error message in restic ls command 2021-12-27 22:42:27 +05:30
Michael Eischer
0cfdb82ea4 prune: Handle --max-repack-size=0 as expected
Previously the flag was ignored and `--max-repack-size=1` had to be
used.
2021-12-27 15:48:56 +01:00
Kyle Brennan
d5ed5da85c update changelog for copy --no-lock 2021-12-03 12:16:40 -08:00
Kyle Brennan
8eb83029a8 Make copy honor --no-lock 2021-12-03 09:50:28 -08:00
MichaelEischer
882d58abce Merge pull request #3163 from palbr/patch-1
Add PGP fingerprint to 020_installation.rst
2021-11-19 23:58:50 +01:00
Peter Albrecht
8de4401bb5 Changed URL for key-file
The keyfile provided by restic's own webserver (https://restic.net) should be
more stable than relying on public keyservers. So I changed the URL to the
GPG keyfile, as recommended by MichaelEischer.
2021-11-19 15:47:59 +01:00
MichaelEischer
f7a9b90eb9 Merge pull request #3573 from magandrez/feat/ls-report-symbolic-notation
Provide mode in symbolic notation for `restic ls --json`
2021-11-18 21:29:04 +01:00
MichaelEischer
aa214f99b4 Merge pull request #3565 from mathstuf/doc-missing-metadata
doc: mention metadata that is not backed up currently
2021-11-18 21:28:55 +01:00
MichaelEischer
4a25bbaed3 Merge pull request #3578 from adsultana/patch-1
docs: Fix link to "help wanted" issues
2021-11-18 21:20:30 +01:00
Michael Eischer
583edc39b8 doc: reorder backup metadata exceptions 2021-11-18 21:17:38 +01:00
Manuel González
212b2f651f Add file mode in symbolic notation to ls --json
This aligns `restic ls --json` with `restic find --json`, utilizing the same
naming.
2021-11-16 19:45:50 +02:00
Andrew Sultana
15ab96ecd6 docs: Fix link to "help wanted" issues 2021-11-16 00:03:34 +00:00
Ben Boeckel
d71afb3d32 doc: mention metadata that is not backed up currently
See: #3497
See: #1622
See: #2075
2021-11-13 18:40:26 -05:00
MichaelEischer
4bf05d91a1 Merge pull request #3571 from garrmcnu/blazer-unknown-authority
Update Backblaze B2 blazer module
2021-11-13 22:46:49 +01:00
MichaelEischer
de3afc1005 Merge pull request #3574 from gurjeet/rename_aws_s3_to_amazon_s3
Use S3's proper product name, Amazon S3
2021-11-13 22:30:32 +01:00
Michael Eischer
2ea998f70e Add PR link to changelog 2021-11-13 22:25:54 +01:00
Garry McNulty
e8fa3855e7 Update blazer
If a request fails with "x509: certificate signed by unknown authority",
the B2 backend now returns the error without retrying the request.

Closes #3556
Closes #2355
2021-11-13 22:25:54 +01:00
Gurjeet Singh
34a6a24544 Use S3's proper product name, Amazon S3
Per Amazon's product page [1], S3 is officially called "Amazon S3". The
restic project uses the phrase "AWS S3" in some places. This patch
corrects the product name.

[1]:https://aws.amazon.com/s3/
2021-11-13 22:21:06 +01:00
MichaelEischer
1d8a0b06cb Merge pull request #3575 from MichaelEischer/adjust-http2-canary-test
rest: Adjust http2-missing-eof test to golang >= 1.17.3
2021-11-13 22:13:24 +01:00
Michael Eischer
50053a85d3 rest: Adjust http2 missing eof test to golang >= 1.17.3, >= 1.16.10
The missing eof with http2 when a response included a content-length
header but no data, has been fixed in golang 1.17.3/1.16.10. Therefore
just drop the canary test and schedule it for removal once go 1.18 is
required as minimum version by restic.
2021-11-13 21:57:30 +01:00
MichaelEischer
f1cfb97237 Merge pull request #3514 from phcreery/rclone_timeout
rclone: extend timeout from 60s to 240s
2021-11-07 18:10:34 +01:00
MichaelEischer
cb81ee9396 Merge pull request #3474 from kitone/fix-issue-3382
Honor RESTIC_CACHE_DIR environment variable
2021-11-07 17:57:54 +01:00
Michael Eischer
b0e64deb27 rclone: Fix timeout calculation 2021-11-07 17:49:33 +01:00
phcreery
43d173b042 rclone: add timeout option and documentation 2021-11-07 17:49:21 +01:00
rawtaz
1b152a2c4d Merge pull request #3568 from MichaelEischer/fix-local-fd-leak
local: Fix fd leak when encountering files directly inside data/
2021-11-07 01:36:48 +01:00
MichaelEischer
15cc3c0e23 Merge pull request #3566 from mathstuf/check-progress-output-stomping
check: wait for progress bar output
2021-11-06 23:34:49 +01:00
kitone
5904f80cfa restic cache should display the name of the cache without shortening it in the case of the restic check 2021-11-06 20:18:51 +01:00
Ben Boeckel
4d579c4387 check: wait for progress bar output
Further code will also output to the terminal and the bar's cursor
positioning causes its output to overlap with the remaining output in a
racy way.

Fixes: #3344
2021-11-06 15:05:09 -04:00
Michael Eischer
15d7313387 local: Fix fd leak when encountering files directly inside data/ 2021-11-06 19:44:57 +01:00
MichaelEischer
6c84ea1412 Merge pull request #3548 from gum3ng/issue_3490
Support for specifying amount of data in read-data-subset
2021-11-05 23:28:11 +01:00
MichaelEischer
78c7dd53ef Merge pull request #3526 from greatroar/dump-refactor
Refactor internal/dump + concurrent load/write
2021-11-05 22:38:39 +01:00
MichaelEischer
a34bfa8269 Merge pull request #3562 from restic/rawtaz-mount-usage
mount: Improve usage information when mounted
2021-11-05 22:10:32 +01:00
kitone
0425a30420 add changelog issue-3382 2021-11-04 15:13:48 +01:00
kitone
1b23675f21 cache --cleanup should handle directories created by restic check.
Because there is no guarantee that a cleanup of these directories will occur
after the "restic check", we extend the behavior to detect and manage these
specific cache directories and allow their cleanup too.
2021-11-04 15:10:38 +01:00
Gautam Menghani
836fbb9133 [#issue 3490] Support for specifying file size in read-data-subset 2021-11-02 15:25:46 +05:30
greatroar
c71729dfc4 Refactor internal/dump + concurrent load/write
Package internal/dump has been reworked so its API consists of a single
type Dumper that handles tar and zip formats. Tree loading and node
writing happen concurrently.
2021-11-01 23:01:55 +01:00
Leo R. Lundgren
711ceb0109 mount: Improve usage information when mounted 2021-11-01 20:59:20 +01:00
MichaelEischer
829c0a67af Merge pull request #3520 from nxt-engineering/Docker
Docker Multistage build
2021-10-24 17:37:54 +02:00
Alexander Neumann
fb5d9345a7 Merge pull request #3510 from MichaelEischer/fix-archiver-early-on-abort
archiver: Fix TestArchiverAbortEarlyOnError test
2021-10-16 15:37:41 +02:00
kitone
95eb859b54 Honor RESTIC_CACHE_DIR environment variable
Fix #3382: restic check doesn't obey the RESTIC_CACHE_DIR environment variable
2021-10-10 16:00:02 +02:00
Michael Eischer
257740b0cc b2: Successful delete if file does not exist
When deleting a file, B2 sometimes returns a "500 Service Unavailable"
error but nevertheless correctly deletes the file. Due to retries in
the B2 library blazer, we sometimes also see a "400 File not present"
error. The retries of restic for the delete request then fail with
"404 File with such name does not exist.".

As we have to rely on request retries in a distributed system to handle
temporary errors, also consider a delete request to be successful if the
file is reported as not existing. This should be safe as B2 claims to
provide a strongly consistent bucket listing and thus a missing file
shouldn't mysteriously show up again later on.
2021-10-09 23:51:12 +02:00
MichaelEischer
46d08d9404 Merge pull request #3535 from jtagcat/writingverbose
Change: selfupdate: 'writing restic to' as verbose
2021-10-09 22:41:51 +02:00
jtagcat
a7853057ab Change: selfupdate: 'writing restic to' as verbose
Running restic self-update --quiet no longer
prints "writing restic to /usr/local/bin/restic".

The only output printed with -q is failures or
"successfully updated restic to version 0.12.1"

https://github.com/restic/restic/pull/3535

fix test fail: changelog title can't end with `.`

shorten changelog title
2021-10-09 23:24:18 +03:00
MichaelEischer
eb282532dc Merge pull request #3534 from jtagcat/patch-1
docs/example: useradd: restic now system account
2021-10-09 20:42:43 +02:00
MichaelEischer
f2a3b3b4a1 Merge pull request #3537 from greatroar/dump-lru
Check cap instead of len in bloblru
2021-10-09 20:00:27 +02:00
MichaelEischer
58e8b34633 Merge pull request #3532 from garrmcnu/s3-credentials-config
s3: Add warning if key ID or secret is empty
2021-10-09 19:32:32 +02:00
MichaelEischer
a02cea6e83 Merge pull request #3539 from jnoxon/fix-ec2-metadata-v2
get iam roles working with ec2 metadata v2
2021-10-07 23:00:27 +02:00
Garry McNulty
708d7a2574 s3: Add warning if key ID or secret is empty
Also add debug message if no credential types are available.

Closes #2388
2021-10-06 23:13:40 +01:00
Jeff Noxon
6f4b5ab8d1 get iam roles working with ec2 metadata v2 2021-10-03 20:06:55 -05:00
greatroar
634a9c162d Check cap instead of len in bloblru
restic dump uses bloblru.Cache to keep buffers alive, but also reuses
evicted buffers. That means large buffers may be used to store small
blobs, causing the cache to think it's using less memory than it
actually does.
2021-10-03 09:34:17 +02:00
jtagcat
632ca2ef52 docs/example: useradd: restic now system account
and: use /sbin/nologin
and: use long flags in the useradd command
and: download v0.12.1 instead of v0.9.6
2021-10-03 00:51:06 +03:00
MichaelEischer
24088f8307 Merge pull request #3528 from greatroar/tree-insert
internal/restic: Don't allocate in Tree.Insert
2021-09-27 22:22:03 +02:00
greatroar
c892c0bab9 internal/restic: Don't allocate in Tree.Insert
name         old time/op    new time/op    delta
BuildTree-8    34.6µs ± 4%     7.0µs ± 3%  -79.68%  (p=0.000 n=18+19)

name         old alloc/op   new alloc/op   delta
BuildTree-8    34.0kB ± 0%     0.9kB ± 0%  -97.37%  (p=0.000 n=20+20)

name         old allocs/op  new allocs/op  delta
BuildTree-8       108 ± 0%         1 ± 0%  -99.07%  (p=0.000 n=20+20)
2021-09-26 18:08:48 +02:00
MichaelEischer
78dac2fd48 Merge pull request #3523 from greatroar/swift-v2
Upgrade ncw/swift to v2
2021-09-24 21:41:36 +02:00
Michael Eischer
5ea8bba1a1 swift: restore context err check for list() 2021-09-24 21:19:46 +02:00
MichaelEischer
a5e103a212 Merge pull request #3522 from greatroar/dump-lru
Use LRU cache in restic dump
2021-09-24 20:33:58 +02:00
greatroar
e7ec0453b1 Upgrade ncw/swift to v2 2021-09-24 19:08:37 +02:00
Uli Martens
1ebcb1d097 Add changelog entry for PR #3508 2021-09-24 15:45:09 +02:00
greatroar
fe04d024c7 Use LRU cache in restic dump 2021-09-24 15:45:08 +02:00
Uli Martens
718966a81a Move Blobcache into dedicated internal package 2021-09-24 15:45:00 +02:00
greatroar
4f33eca634 Remove unused Writer arg to internal/dump.writeDump 2021-09-24 15:40:42 +02:00
MichaelEischer
cc110c42e6 Merge pull request #2657 from mansam/add-skip-tls-verification-flag
Add --insecure-tls flag to disable SSL cert verification
2021-09-22 21:40:01 +02:00
Sam Lucidi
897d8e662c Add --insecure-tls flag to disable SSL cert verification
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
cimnine
4a95af5290 GitHub action for Docker build 2021-09-21 08:23:33 +02:00
cimnine
f28c8bc1c2 Multistage Docker build 2021-09-21 08:23:33 +02:00
MichaelEischer
1827b16ade Merge pull request #3519 from greatroar/maphash
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
greatroar
8b758c78a3 Require Go 1.14 to build 2021-09-19 16:18:19 +02:00
greatroar
8d2996eaaa Replace siphash by hash/maphash
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.

Benchmark results on linux/amd64, -benchtime=3s:

name                                             old time/op    new time/op    delta
IndexHasUnknown-8                                  50.6ns ±10%    41.0ns ±19%  -18.92%  (p=0.000 n=9+10)
IndexHasKnown-8                                    52.6ns ±12%    41.5ns ±12%  -21.13%  (p=0.000 n=9+10)
IndexMapHash-8                                     3.64µs ± 1%    2.00µs ± 0%  -45.09%  (p=0.000 n=10+9)
IndexAlloc-8                                        700ms ± 1%     601ms ± 6%  -14.18%  (p=0.000 n=8+10)
IndexAllocParallel-8                                205ms ± 5%     192ms ± 8%   -6.18%  (p=0.043 n=10+10)
MasterIndexAlloc-8                                  319ms ± 1%     279ms ± 5%  -12.58%  (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8                      156ns ± 8%     147ns ± 6%   -5.46%  (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8                    150ns ± 7%     142ns ± 8%   -5.69%  (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8              74.4ns ± 6%    72.0ns ± 9%     ~     (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8            67.4ns ± 9%    65.5ns ± 7%     ~     (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8        461ns ± 2%     445ns ± 2%   -3.49%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8      408ns ±11%     378ns ± 5%   -7.22%  (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8        479ns ± 1%     437ns ± 4%   -8.82%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8      406ns ± 8%     343ns ±15%  -15.44%  (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8       480ns ± 1%     455ns ± 5%   -5.15%  (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8     391ns ±18%     382ns ± 8%     ~     (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8                        71.0ns ± 8%    57.2ns ±11%  -19.36%  (p=0.000 n=9+10)
PackerManager-8                                     254ms ± 1%     254ms ± 1%     ~     (p=0.285 n=15+15)

name                                             old speed      new speed      delta
IndexMapHash-8                                   1.12GB/s ± 1%  2.05GB/s ± 0%  +82.13%  (p=0.000 n=10+9)
PackerManager-8                                   208MB/s ± 1%   207MB/s ± 1%     ~     (p=0.281 n=15+15)

name                                             old alloc/op   new alloc/op   delta
IndexMapHash-8                                      0.00B          0.00B          ~     (all equal)
IndexAlloc-8                                        400MB ± 0%     400MB ± 0%     ~     (p=1.000 n=9+10)
IndexAllocParallel-8                                401MB ± 0%     401MB ± 0%   +0.00%  (p=0.000 n=10+10)
MasterIndexAlloc-8                                  258MB ± 0%     262MB ± 0%   +1.42%  (p=0.000 n=9+10)
PackerManager-8                                    73.1kB ± 0%    73.1kB ± 0%     ~     (p=0.382 n=13+13)

name                                             old allocs/op  new allocs/op  delta
IndexMapHash-8                                       0.00           0.00          ~     (all equal)
IndexAlloc-8                                         907k ± 0%      907k ± 0%   -0.00%  (p=0.000 n=10+10)
IndexAllocParallel-8                                 907k ± 0%      907k ± 0%   +0.00%  (p=0.009 n=10+10)
MasterIndexAlloc-8                                   327k ± 0%      317k ± 0%   -3.06%  (p=0.000 n=10+10)
PackerManager-8                                       744 ± 0%       744 ± 0%     ~     (all equal)
2021-09-19 16:05:18 +02:00
MichaelEischer
58efe21eca Merge pull request #3264 from amozoss/upstream-master
Refactor backup progress
2021-09-19 14:54:42 +02:00
MichaelEischer
71fcf48533 Merge pull request #2594 from greatroar/concurrent-restore-verify
Concurrent restore --verify
2021-09-19 14:52:31 +02:00
Michael Eischer
921e328b56 restore: Fix linting error 2021-09-19 14:41:07 +02:00
Michael Eischer
e62d4f622f Reword changelog 2021-09-19 14:41:07 +02:00
Michael Eischer
2cdc0719af restorer: Sanitize verify errors 2021-09-19 14:01:26 +02:00
greatroar
bdcdfaf6b4 restore --verify: buffer reuse consistency and comment 2021-09-19 13:11:27 +02:00
greatroar
2b94742ca5 Replace no-op closures in restorer by nil check 2021-09-19 13:11:16 +02:00
greatroar
d357744104 Handle canceled context in restore --verify properly 2021-09-19 13:11:05 +02:00
greatroar
d4225ec803 Simplify buffer growing in Restorer.verifyFile
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
greatroar
de8521ae56 Report number of successes from Restorer.VerifyFiles
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
greatroar
bb066cf7d3 Concurrent Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
greatroar
556424d61b Reuse buffer in Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
greatroar
92ae951ffa Report timing from restic restore --verify 2021-09-19 12:53:09 +02:00
greatroar
973fa921cb Test and document Restorer.VerifyFiles 2021-09-19 12:52:11 +02:00
Michael Eischer
e0d615c264 archiver: Fix TestArchiverAbortEarlyOnError test
This can be caused when the test has uploaded four blobs, then queues
two blobs for upload which are delayed. Then a seventh file can be
opened which lead to a test failure.
2021-09-12 22:17:17 +02:00
MichaelEischer
ef5672a902 Merge pull request #3509 from ajaspers/patch-1
Update instructions for reproducing build
2021-09-12 21:40:37 +02:00
ajaspers
c0eddc9969 Update instructions for reproducing build
Dependencies are fetched at build time and stored in the GOPATH. These paths end up being in the final binary.

Bump restic version to latest and go version to the 1.16.6, which was used to build restic 0.12.1.
2021-09-12 09:02:57 -07:00
Michael Eischer
fbb0e6499a ui: consolidate backup ui in ui/backup package 2021-09-12 16:20:15 +02:00
Michael Eischer
503d4c3e2f Add changelog 2021-09-12 16:00:49 +02:00
Michael Eischer
cccb0d4064 ui: assert that ProgressPrinter interface is implemented 2021-09-12 15:26:40 +02:00
Michael Eischer
a144c986f2 backup: Reenable JSON status updates with redirected output
After the refactoring status updates were no longer printed in quiet
mode or when the output is not an interactive terminal. However, the
JSON output is often piped to e.g. another program. Thus, don't set the
update frequency to 0 in that case. The status updates are still
disabled for backup --quiet.

This also reduces the status update frequency to 60fps compared to a
potentially much higher value before the refactoring.
2021-09-12 15:26:40 +02:00
Michael Eischer
d62bfed65d ui: move SetDryRun to ProgressReporter 2021-09-12 15:25:58 +02:00
Michael Eischer
77b129ec74 ui: remove unused Summary field 2021-09-12 15:25:58 +02:00
Michael Eischer
3024239e40 ui/json: restore summary output 2021-09-12 15:25:58 +02:00
Michael Eischer
5ccf583b8a ui: restore a few comments 2021-09-12 15:25:58 +02:00
Michael Eischer
80cbaf6d38 ui: Simplify ReportTotal methods 2021-09-12 15:25:58 +02:00
Dan Willoughby
448419990c Refactor backup progress
Move the shared logic into the progress

Allows logic to be shared with forth coming restore status
2021-09-12 15:25:58 +02:00
Andy
7baa9a570d document AWS_PROFILE support
Since restic (or whatever library it is using) seems to respect/use AWS_PROFILE, it's worth documenting this.
2021-09-10 14:06:46 +01:00
Alexander Neumann
bf9c8771a4 Merge pull request #3482 from MichaelEischer/changelog-3429
Add changelog for #3246 and #3429
2021-09-08 09:15:40 +02:00
Michael Eischer
5e84f38f31 Add changelog for #3246 2021-09-07 21:18:11 +02:00
Alexander Neumann
8fe122d675 Merge pull request #3488 from MichaelEischer/rebuild-broken-index
Fix `rebuild-index` for damaged index
2021-09-07 17:00:23 +02:00
Alexander Neumann
74c47f1f12 Merge pull request #3502 from restic/rawtaz-issue-519
doc: Add note about scheduling to backup section
2021-09-07 16:58:17 +02:00
MichaelEischer
fa5ca8af81 Merge pull request #2856 from aawsome/remove-readahead
Simplify cache logic
2021-09-04 20:25:49 +02:00
MichaelEischer
b45d88e124 Merge pull request #3496 from juergenhoetzel/imporove-snapshot-filter-warning-message
Improve snapshot filter warning message
2021-09-03 23:42:27 +02:00
MichaelEischer
bc4cbd775b Merge pull request #2880 from aawsome/enhance-recover
Improve recover command
2021-09-03 23:40:43 +02:00
MichaelEischer
a29777f467 Merge pull request #3501 from greatroar/printprogress
Streamline progress printing in cmd/restic
2021-09-03 23:34:36 +02:00
Alexander Weiss
bce87922c0 Improve recover
- only save directories not referenced by any snapshot
- dont't write empty snapshot
- use progress bar
2021-09-03 21:36:57 +02:00
Alexander Weiss
81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
greatroar
7f0aa49f45 cmd/restic: Streamline progress printing
* PrintProgress no longer does unnecessary Sprintf calls, and performs
  fewer allocations in general
* newProgressMax's callback checks whether the terminal supports
  line updates once instead of once per call
* the callback looks up the terminal width once per call instead of
  twice (on Windows)
* the status shortening now uses the Unicode-aware version from
  internal/ui/termstatus (future-proofing)
2021-09-03 11:48:22 +02:00
greatroar
5aaa3e93c1 internal/ui/termstatus: Optimize and publish Truncate
name               old time/op  new time/op  delta
TruncateASCII-8     347ns ± 1%    69ns ± 1%  -80.02%  (p=0.000 n=9+10)
TruncateUnicode-8   447ns ± 3%   348ns ± 1%  -22.04%  (p=0.000 n=10+10)
2021-09-03 11:48:22 +02:00
MichaelEischer
ec2e3b260e Merge pull request #3499 from greatroar/wrappedconn-pointer
rclone: Return one fewer value from run
2021-08-31 21:48:36 +02:00
rawtaz
26914abe62 doc: Add note about scheduling to backup section
Explains that restic doesn't have built-in scheduling
and mentions a few keywords one can search for.
2021-08-29 22:03:22 +02:00
greatroar
950b818274 rclone: Return one fewer value from run 2021-08-26 18:12:08 +02:00
Juergen Hoetzel
defe19fdf6 Quote snapshot prefix in error string 2021-08-25 16:11:28 +02:00
Juergen Hoetzel
409e4936af Improve snapshot filter warning message
Include the root-cause why the snapshot prefix is ignored.
2021-08-25 15:46:21 +02:00
MichaelEischer
10b39d7591 Merge pull request #3487 from andreaso/rest-proto-readme-link
Fix REST protocol link in README file
2021-08-22 18:47:32 +02:00
Michael Eischer
194ed19557 Add changelog 2021-08-22 18:29:58 +02:00
Michael Eischer
877fc9f352 rebuild-index: test that invalid indexes are skipped and removed 2021-08-22 18:24:19 +02:00
Michael Eischer
64258a2c2a rebuild-index: Ignore broken index files
Previously, an e.g. truncated index could prevent rebuild-index from
working.
2021-08-22 18:23:47 +02:00
Andreas Olsson
c520672982 Fix REST protocol link in README file
1. All other document links goes to the _Read the Docs_ site
2. The GitHub reStructuredText renderer doesn't appear to do includes,
   making for a rather empty read.
2021-08-22 13:16:51 +02:00
Alexander Neumann
9374c3ce81 Merge pull request #3485 from lbausch/fix_typo
Fix typo in changelog
2021-08-21 13:16:20 +02:00
Lorenz Bausch
4d56b34096 Fix typo in changelog 2021-08-21 12:16:04 +02:00
Alexander Neumann
66382b2861 Update Go to 1.17 2021-08-17 21:38:34 +02:00
Michael Eischer
1fab5892b5 Add changelog for #3429 2021-08-16 17:42:16 +02:00
MichaelEischer
c898f7a6bf Merge pull request #3479 from BUFU1610/patch-1
fixed --keep-within-X options in example
2021-08-15 20:06:26 +02:00
BUFU
7659790923 fixed --keep-within-X options in example
The order of the words was wrong in the example, i.e. fixed to --keep-within-hourly from --keep-hourly-within.
2021-08-15 19:56:22 +02:00
MichaelEischer
ecf34783ef Merge pull request #3480 from MichaelEischer/fix-rest-tests
Fix rest backend tests
2021-08-15 18:59:02 +02:00
Michael Eischer
68370feeee backends: Remove TestSaveFilenames test
Filenames are expected to match the sha256 sum of the file content. This
rule is now enforced by the rest server thus making this test useless.
2021-08-15 18:24:16 +02:00
Michael Eischer
574c83e47f rest: Fix test to use paths which are the sha256 sum of the data 2021-08-15 18:19:43 +02:00
Michael Eischer
e6a5801155 rest: Fix test backend url
The rest config normally uses prepareURL to sanitize URLs and ensures
that the URL ends with a slash. However, the test used an URL without a
trailing slash, which after the rest server changes causes test
failures.
2021-08-15 18:16:17 +02:00
Alexander Neumann
d90efd7704 Fix test 2021-08-09 10:30:10 +02:00
Alexander Neumann
9fe5a87785 Merge pull request #3429 from MichaelEischer/safe-key-change
key: Check that new key works before deleting the old one
2021-08-09 10:07:15 +02:00
Alexander Neumann
7f1608dc77 Merge pull request #3246 from restic/content-hash-for-upload
Calculate content hashes during upload
2021-08-08 17:24:08 +02:00
Michael Eischer
f4c5dec05d backend: test that a wrong hash fails an upload 2021-08-04 22:17:46 +02:00
Michael Eischer
7c1903e1ee panic if hash returns an error
Add a sanity check that the interface contract is honoured.
2021-08-04 22:17:46 +02:00
Michael Eischer
51b7e3119b mem: calculate md5 content hash for uploads
The mem backend is primarily used for testing. This ensures that the
upload hash calculation gets appropriate test coverage.
2021-08-04 22:17:46 +02:00
Michael Eischer
a009b39e4c gs/swift: calculate md5 content hash for upload 2021-08-04 22:17:46 +02:00
Michael Eischer
1d3e99f475 azure: check upload using md5 content hash
For files below 256MB this uses the md5 hash calculated while assembling
the pack file. For larger files the hash for each 100MB part is
calculated on the fly. That hash is also reused as temporary filename.
As restic only uploads encrypted data which includes among others a
random initialization vector, the file hash shouldn't be susceptible to
md5 collision attacks (even though the algorithm is broken).
2021-08-04 22:17:46 +02:00
Michael Eischer
9aa2eff384 Add plumbing to calculate backend specific file hash for upload
This enables the backends to request the calculation of a
backend-specific hash. For the currently supported backends this will
always be MD5. The hash calculation happens as early as possible, for
pack files this is during assembly of the pack file. That way the hash
would even capture corruptions of the temporary pack file on disk.
2021-08-04 22:17:46 +02:00
Michael Eischer
ee2f14eaf0 s3: enable content hash calculation for uploads 2021-08-04 22:12:12 +02:00
MichaelEischer
553ea36ca6 Merge pull request #2838 from greatroar/cache-conflicts
Make cache concurrency-safe
2021-08-04 22:11:50 +02:00
greatroar
6586e90acf Modernize internal/cache error handling 2021-08-04 22:02:42 +02:00
greatroar
ea04f40eb3 Save cached files to a temporary location first 2021-08-04 22:02:42 +02:00
greatroar
f9b6f8fd45 Replace duplicate type checking in cache with a function 2021-08-04 22:02:42 +02:00
MichaelEischer
1b1a2115fa Merge pull request #3436 from greatroar/local-save-tmp
Save files under temporary name in local backend
2021-08-04 22:01:53 +02:00
Michael Eischer
65908647e3 Add changelog 2021-08-04 21:51:53 +02:00
greatroar
81e2499d19 Sync directory to get durable writes in local backend 2021-08-04 21:51:53 +02:00
greatroar
195a5cf996 Save files under temporary name in local backend
Fixes #3435.
2021-08-04 21:51:53 +02:00
MichaelEischer
bc97a3d1f9 Merge pull request #3300 from aawsome/backup-dryrun
backup: add --dry-run/-n flag
2021-08-04 21:50:38 +02:00
Michael Eischer
702cff636f Add use case to changelog 2021-08-04 21:19:29 +02:00
Alexander Weiss
780e11b7e2 Adapt changelog 2021-08-04 21:19:29 +02:00
erin
4126435663 resolve rawtaz's review comments
make majority of suggestions from review by @rawtaz verbatim, with one clarification on my part in changelog
2021-08-04 21:19:29 +02:00
Alexander Weiss
d107a2cfdf Separate dry run tests 2021-08-04 21:19:29 +02:00
Alexander Weiss
38a8a48a25 Simplify dry run backend 2021-08-04 21:19:29 +02:00
Ryan Hitchman
77bf148460 backup: add --dry-run/-n flag to show what would happen.
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.

Fixes #1542

Example usage:

$ restic -vv --dry-run . | grep add
new       /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified  /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified  /cmd/restic/global.go, saved in 0.000s (0 B added)
new       /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new       /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified  /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified  /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified  /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified  /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified  /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
2021-08-04 21:19:29 +02:00
MichaelEischer
533ac4fd95 Merge pull request #3467 from greatroar/wrappedconn-pointer
Use rclone.wrappedConn by pointer
2021-08-04 21:18:53 +02:00
Alexander Neumann
7049f1cbfc Set development version for 0.12.1 2021-08-03 11:45:39 +02:00
Alexander Neumann
dc7a8aab24 Add version for 0.12.1 2021-08-03 11:45:36 +02:00
Alexander Neumann
94983a1f36 Update manpages and auto-completion 2021-08-03 11:45:36 +02:00
Alexander Neumann
a92faca10e Generate CHANGELOG.md for 0.12.1 2021-08-03 11:45:19 +02:00
Alexander Neumann
b19cd8c50f Prepare changelog for 0.12.1 2021-08-03 11:45:19 +02:00
Alexander Neumann
b862732318 Merge pull request #3468 from restic/rework-changelog
Reword changelog entries
2021-08-03 10:06:56 +02:00
Leo R. Lundgren
cb844e7136 Polish changelog entries 2021-08-03 00:01:09 +02:00
Alexander Neumann
b7fe1fe6b4 Reword changelog entries 2021-08-03 00:01:09 +02:00
MichaelEischer
c98bbdcdbe Merge pull request #3457 from systemmonkey42/feature/untagged
Feature to match untagged snapshots only when listing or forgetting
2021-08-02 23:19:39 +02:00
David le Blanc
326fefcd80 Allow --tag and --keep-tag to match untagged snapshots 2021-08-02 23:06:20 +02:00
greatroar
fa3eed1998 Use rclone.wrappedConn by pointer
This shaves a kilobyte off the Linux binary by not generating a
non-pointer interface implementation.
2021-08-01 09:11:50 +02:00
MichaelEischer
5571c3f7fd Merge pull request #3453 from MichaelEischer/http2-zero-length-workaround
rest: Workaround Http2 zero-length reply bug
2021-07-31 20:30:06 +02:00
Alexander Neumann
d8ea10db8c rest: Rework handling HTTP2 zero-length replies bug
Add comment that the check is based on the stdlib HTTP2 client. Refactor
the checks into a function. Return an error if the value in the
Content-Length header cannot be parsed.
2021-07-31 17:12:24 +02:00
Alexander Neumann
77551597b2 Merge pull request #3416 from torfason/keep-hourly-within
Keep hourly within
2021-07-30 10:36:41 +02:00
Alexander Neumann
92f293cd0b Merge pull request #3427 from MichaelEischer/find-packs-from-index
find: List missing pack files based on the index
2021-07-30 10:33:02 +02:00
Magnus Thor Torfason
2081bd12fb forget: Ensure future snapshots do not affect --keep-within-*
Ensure that only snapshots made in the past are taken into account when running restic forget with the within switches (--keep-within, --keep-within- hourly, and friends)
2021-07-24 16:14:43 +00:00
Magnus Thor Torfason
74ebc650ab forget: Add --keep-within-hourly (and friends)
Allow keeping hourly/daily/weekly/monthly/yearly snapshots for a given time period.

This adds the following flags/parameters to restic forget:
  --keep-within-hourly duration
  --keep-within-daily duration
  --keep-within-weekly duration
  --keep-within-monthly duration
  --keep-within-yearly duration

Includes following changes:
  - Add tests for --keep-within-hourly (and friends)
  - Add documentation for --keep-within-hourly (and friends)
  - Add changelog for --keep-within-hourly (and friends)
2021-07-24 16:14:43 +00:00
Alexander Neumann
c707d71b72 Merge pull request #3401 from MichaelEischer/goroutine-shutdown-cleanups
Goroutine shutdown cleanups
2021-07-11 16:32:28 +02:00
Alexander Neumann
691866ce43 Merge pull request #3452 from MichaelEischer/add-s390x-releases
Add release binaries for linux/s390x
2021-07-11 16:30:25 +02:00
Alexander Neumann
efd918c59e Merge pull request #3454 from MichaelEischer/update-dependencies
Update dependencies
2021-07-11 16:29:31 +02:00
Michael Eischer
7d28006e2e Add changelog 2021-07-10 22:39:01 +02:00
Michael Eischer
0880afe67b Use our generate command instead of cobra's complete command 2021-07-10 19:44:18 +02:00
Michael Eischer
100baf74c0 Update cobra
The most noteworthy change seems to be

https://github.com/spf13/cobra/pull/1192
Have Cobra create 'completion' command automatically
2021-07-10 19:43:14 +02:00
Michael Eischer
c733ae6b16 Readd and update indirect dependencies
The azure-sdk-for-go is the only remaining module without a go.mod file.
Thus we only need indirect dependencies for it. Remove all other
indirect dependencies.
2021-07-10 19:21:43 +02:00
Michael Eischer
989b398fee Misc library updates
The most noteworthy change is the xattr update which includes
https://github.com/pkg/xattr/pull/54 that adds xattr support for solaris
and illumos
2021-07-10 19:04:47 +02:00
Michael Eischer
bbc8146934 Update minio/sha256-simd
Apparently the standard Go sha256 implementation is now faster than the
assembly implementation. The library now only adds support for SHA
extensions available in some processors.

See https://github.com/minio/sha256-simd/pull/57 for more details.
2021-07-10 18:34:16 +02:00
Michael Eischer
aa22ebac69 Update backend dependencies
Possibly interesting changes:

https://github.com/Azure/azure-sdk-for-go/pull/14521
Retry on HTTP client errors

https://github.com/minio/minio-go/pull/1452
fix: make sure getObject returns error on truncated responses

f5854403a9
http2: close Transport connection on write errors
2021-07-10 18:27:13 +02:00
Michael Eischer
097ed659b2 rest: test that zero-length replies over HTTP2 work correctly
The first test function ensures that the workaround works as expected.
And the second test function is intended to fail as soon as the issue
has been fixed in golang to allow us to eventually remove the
workaround.
2021-07-10 17:22:42 +02:00
Michael Eischer
185a55026b rest: workaround for HTTP2 zero-length replies bug
The golang http client does not return an error when a HTTP2 reply
includes a non-zero content length but does not return any data at all.
This scenario can occur e.g. when using rclone when a file stored in a
backend seems to be accessible but then fails to download.
2021-07-10 16:59:01 +02:00
Michael Eischer
495831d53c add release binaries for linux/s390x 2021-07-10 00:10:24 +02:00
Michael Eischer
3442dc87fb find: Address review comments 2021-07-06 21:04:34 +02:00
Michael Eischer
a81f34ae47 Add changelog 2021-07-06 21:04:34 +02:00
Michael Eischer
95b44490a0 find: search blob ids for pack in index if pack is missing
If a pack file is missing try to determine the contained pack ids based
on the repository index. This helps with assessing the damage to a
repository before running `rebuild-index`.
2021-07-06 21:04:34 +02:00
Michael Eischer
3caab3c7ac find: Print not found pack files 2021-07-06 21:04:34 +02:00
Michael Eischer
40745b4f82 find: stop file listing after resolving all pack files 2021-07-06 21:04:34 +02:00
Michael Eischer
6c01078f3d find: support resolving multiple pack ids to blobs
Just passing the list of blobs to packsToBlobs would also work in most
cases, however, it could cause unexpected results when multiple pack
files have the same prefix. Forget found prefixes to prevent this.
2021-07-06 21:04:34 +02:00
Leo R. Lundgren
790294dc26 contributing: Change freenode to libera 2021-07-04 11:05:36 +02:00
Alexander Neumann
30d968b0e4 Merge pull request #3449 from MichaelEischer/fix-restore-retries
restore: Correctly handle partial pack download errors
2021-07-04 10:55:09 +02:00
Michael Eischer
43b82d69b4 Add changelog 2021-06-29 21:27:00 +02:00
Michael Eischer
bd316d3893 restore: Test partial pack downloads in filerestorer 2021-06-29 21:11:30 +02:00
Michael Eischer
e8bbb05328 restore: Correctly handle partial pack download errors
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
rawtaz
58be5172ff Merge pull request #3437 from MichaelEischer/fix-prune-output
prune: Add missing newlines in error descriptions
2021-06-28 00:56:51 +02:00
MichaelEischer
cb6fd281a0 Merge pull request #3442 from restic/rawtaz-doc-verbose
doc: Correct position of --verbose in backup docs
2021-06-27 18:19:37 +02:00
rawtaz
eb61de7b3a Merge pull request #3287 from tjrana/clean-up-pr-template
Clean up PR template
2021-06-25 21:06:50 +02:00
rawtaz
98a88b483d doc: Correct position of --verbose in backup docs 2021-06-25 19:26:46 +02:00
MichaelEischer
6a4c1ed50d Merge pull request #3438 from MichaelEischer/fast-fuse-changelog
Add changelog for #3426
2021-06-20 15:05:19 +02:00
Michael Eischer
409306db2b Add changelog for #3426 2021-06-20 14:46:37 +02:00
Michael Eischer
aad8864835 prune: Add missing newlines in error descriptions 2021-06-20 14:25:40 +02:00
MichaelEischer
c1eb7ac1a1 Merge pull request #3420 from greatroar/local-errs
Modernize error handling in local backend
2021-06-20 14:20:40 +02:00
greatroar
e5f0f67ba0 Modernize error handling in local backend
* Stop prepending the operation name: it's already part of os.PathError,
  leading to repetitive errors like "Chmod: chmod /foo/bar: operation not
  permitted".

* Use errors.Is to check for specific errors.
2021-06-18 11:13:27 +02:00
Alexander Neumann
45eb30388f Merge pull request #3426 from MichaelEischer/fast-fuse
mount: enable fuse readahead
2021-06-14 10:27:41 +02:00
Michael Eischer
454b6d608e key: Add test that failed key saves don't break the repository 2021-06-13 13:46:48 +02:00
TJ Rana
a61a0255a8 Modify pull request template 2021-06-12 19:34:46 -04:00
Michael Eischer
6add186867 key: Check that a new key file actually works 2021-06-12 23:09:19 +02:00
MichaelEischer
a476752962 Merge pull request #3421 from greatroar/s3-fileinfo
Return s3.fileInfos by pointer
2021-06-12 18:55:18 +02:00
MichaelEischer
e8d20ea32c Merge pull request #3409 from greatroar/lchown-mknod
Make restic.{lchown,mknod} regular functions
2021-06-12 18:22:38 +02:00
Michael Eischer
fe43f53528 mount: enable fuse readahead
Apparently readahead was disabled by default. Enable readahead with the
Linux default size of 128kB. Larger values seem to have no effect.
This can speed up reading from the fuse mount by at least factor 5.

Speedup for a 1G random file stored in a local repository:
(Only one result shown, but times were quite stable, restarted restic
after each command)
$ dd if=/dev/urandom bs=1M count=1024 of=rand
$ shasum -a 256 tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  tmp/rand

before: $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  fuse/snapshots/latest/tmp/rand

real    0m18.294s
user    0m4.522s
sys     0m3.305s

before: $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real    0m14.924s
user    0m0.000s
sys     0m4.625s

after:  $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  fuse/snapshots/latest/tmp/rand

real    0m6.106s
user    0m3.115s
sys     0m0.182s

after:  $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real    0m3.096s
user    0m0.017s
sys     0m0.241s
2021-06-12 17:07:30 +02:00
greatroar
0d4f16b6ba Return s3.fileInfos by pointer
Since the fileInfos are returned in a []interface, they're already
allocated on the heap. Making them pointers explicitly means the
compiler doesn't need to generate fileInfo and *fileInfo versions of the
methods on this type. The binary becomes about 7KiB smaller on
Linux/amd64.
2021-06-07 19:48:43 +02:00
greatroar
0666c4d244 Make restic.{lchown,mknod} regular functions
This way, they can be inlined and dead code can be removed on Windows.
Also fixed some comments.
2021-05-27 22:51:40 +02:00
rawtaz
fdbd65485e Merge pull request #3402 from MichaelEischer/misc-fixes
Various small code cleanups
2021-05-24 11:30:31 +02:00
rawtaz
2daf033156 Merge pull request #3403 from MichaelEischer/fast-cat
cat: only load index if really necessary
2021-05-24 11:02:10 +02:00
MichaelEischer
5dad45f005 Merge pull request #3312 from soraxas/master
Bump cobra and add completions for fish
2021-05-23 13:58:27 +02:00
MichaelEischer
7eb6372123 Merge pull request #3257 from greatroar/ls-json-empty
ls: print "size":0 for empty files in JSON
2021-05-23 13:49:56 +02:00
Michael Eischer
61b368ddea cat: only load index if really necessary 2021-05-23 13:11:55 +02:00
Michael Eischer
fd8bce8184 backup: cleanly shutdown goroutines on error 2021-05-23 13:02:44 +02:00
greatroar
d7322a5f36 restic ls --json: print "size":0 for empty files 2021-05-21 21:06:00 +02:00
Tin Lai
9cc1ecdd45 bump cobra and add completions for fish
Signed-off-by: Tin Lai <oscar@tinyiu.com>
2021-05-21 13:47:52 +10:00
Alexander Neumann
af3de702c7 Merge pull request #3332 from restic/debug-1999
Merge `debug examine` to salvage damaged pack files
2021-05-18 09:38:40 +02:00
Alexander Neumann
226cd8d4d1 Merge pull request #3310 from MichaelEischer/copy-unstable-tree
`copy` raw tree blobs
2021-05-18 09:36:51 +02:00
Alexander Neumann
4cabad8c34 Merge pull request #3325 from MichaelEischer/fix-mintty-output
Fix windows terminal output for mintty
2021-05-18 09:29:24 +02:00
Michael Eischer
cf92c58460 Properly wrap errors in readerat helper 2021-05-17 21:08:23 +02:00
Michael Eischer
5767c65c62 delete: properly close fileChan if context is canceled 2021-05-17 21:05:54 +02:00
Michael Eischer
75c990504d azure/gs: Fix default value in connections help text 2021-05-17 20:56:51 +02:00
Michael Eischer
5a87a0ba0a find: use Str() to format short ids 2021-05-17 20:56:49 +02:00
Alexander Neumann
38ccddc84f Merge pull request #3399 from oli-h/oli-h-patch-060_forget.rst
Sudden find: fix within first "forget" example
2021-05-17 11:39:36 +02:00
Alexander Neumann
d7b5061aa5 Merge pull request #3394 from MichaelEischer/apple-m1-build
Build darwin/arm64 binaries for Apple M1
2021-05-17 09:56:44 +02:00
rawtaz
27141ae87f Merge pull request #2999 from restic/issue-2109
backup: Improve wording for --one-file-system description
2021-05-15 00:23:36 +02:00
Leo R. Lundgren
90d75651e6 backup: Improve wording for --one-file-system description 2021-05-15 00:06:27 +02:00
rawtaz
2a915069a8 Merge pull request #3393 from MichaelEischer/fix-filter-crash
filter: Fix crash for '**' pattern
2021-05-14 23:58:44 +02:00
Michael Eischer
55bea6e7a6 filter: Fix crash for '**' pattern 2021-05-14 23:50:31 +02:00
Michael Eischer
af6f6fba15 Build darwin/arm64 binaries for Apple M1 2021-05-14 23:04:45 +02:00
MichaelEischer
6d8ceefd67 Merge pull request #3373 from greatroar/simplify-limiter
Simplify internal/limiter
2021-05-14 20:27:31 +02:00
MichaelEischer
7349c6d338 Merge pull request #3167 from renard/limit-snapshots-list
Add option to limit snapshots list
2021-05-13 20:26:33 +02:00
Sébastien Gross
2a92b68e65 cmd/snapshots: Add option to limit snapshots list
This patch adds a `--latest` option to limit snapshots list to the n
last snapshots. It is very similar to the `--last` one but does not
limit to one entry. It also deprecates the `--last` flag usage in
favor of `--latest 1`

Output example:

    $ restic snapshots --latest 2
    repository 0d3eb989 opened successfully, password is correct
    ID        Time                 Host        Tags        Paths
    ------------------------------------------------------------
    5a33bdcc  2020-12-14 12:30:00  local                   /home
    73887d8e  2020-12-15 12:30:00  local                   /home
    ------------------------------------------------------------
    2 snapshots

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2021-05-13 20:18:23 +02:00
MichaelEischer
64b00d28b1 Merge pull request #3345 from greatroar/sftp-enospc
Check for ENOSPC and remove broken files in SFTP
2021-05-13 20:09:38 +02:00
oli-h
23f9cb838d Sudden find: fix within first "forget" example
I think this was a typo or copy/paste problem when created the doc.
Feel free to merge. Cheers
2021-05-10 08:55:42 +02:00
rawtaz
01261770bb Merge pull request #3386 from adrian5/doc-change
doc: Add direct link to GitHub issue
2021-05-09 00:12:21 +02:00
adrian5
a0f1c74000 doc: Add direct link to GitHub issue 2021-05-08 21:13:11 +00:00
Alexander Neumann
be6fc02c04 Merge pull request #3376 from restic/rawtaz-doc-exclude
doc: Polish exclude file documentation
2021-04-27 19:25:00 +02:00
Alexander Neumann
3ce5544796 Merge pull request #3321 from restic/doc-files-from
doc: Improve docs for --files-from et al
2021-04-27 19:15:31 +02:00
rawtaz
556caa326f doc: Polish exclude file documentation 2021-04-25 20:36:11 +02:00
greatroar
ae170e2b38 Simplify internal/limiter 2021-04-24 11:54:43 +02:00
rawtaz
f7316cea07 Merge pull request #3371 from cyounkins/max_unused_default
doc: default for --max-unused
2021-04-23 03:18:05 +02:00
Craig Younkins
32a84ab3e4 doc: default for --max-unused 2021-04-22 21:12:37 -04:00
rawtaz
6c7eabf08c Merge pull request #3363 from davegallant/design-doc-grammar-fix
doc: Fix grammar in design.rst
2021-04-15 11:01:38 +02:00
Dave Gallant
4aaf10c356 doc: Fix grammar in design.rst 2021-04-14 17:25:53 -04:00
Michael Eischer
f65e1211d9 Add changelog entry 2021-04-11 20:02:09 +02:00
Michael Eischer
7cb8ea69ba Add test to mintty pipe detection 2021-04-11 20:02:09 +02:00
Michael Eischer
80564a9bc9 Properly detect mintty output redirection
mintty on windows always uses pipes to connect stdout between processes
and for the terminal output. The previous implementation always assumed
that stdout connected to a pipe means that stdout is displayed on a
mintty terminal. However, this detection breaks when using pipes to
connect processes and for powershell which uses pipes when redirecting
to a file.

Now the pipe filename is queried and matched against the pattern used by
msys / cygwin when connected to the terminal. In all other cases assume
that a pipe is just a regular pipe.
2021-04-11 20:02:09 +02:00
Michael Eischer
5e6af77b7a Unify interactive terminal detection code
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.

This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.

stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
2021-04-11 20:02:09 +02:00
MichaelEischer
cc254dfefe Merge pull request #3362 from greatroar/darwin-preallocate
Use FcntlFstore to preallocate on Mac
2021-04-10 22:47:56 +02:00
greatroar
23531be272 Use FcntlFstore to preallocate on Mac 2021-04-10 16:54:07 +02:00
Alexander Neumann
b922fc851b Merge pull request #3356 from restic/rawtaz-doc-dollar-sign
doc: Clarify dollar sign expansion in exclude files
2021-04-09 12:30:54 +02:00
rawtaz
ccfd5f1d4a Merge pull request #3298 from jniggemann/patch-1
doc: Add note about bash completion path
2021-04-09 00:17:17 +02:00
Jan Niggemann
9a1f685179 doc: Add note about bash completion path 2021-04-08 23:53:38 +02:00
rawtaz
b5e40b370c doc: Clarify dollar sign expansion in exclude files 2021-04-08 23:46:53 +02:00
rawtaz
74c0607c92 Merge pull request #3319 from MichaelEischer/skip-prealloc-test
restorer: Skip preallocate test if not supported by the filesystem
2021-04-07 18:59:06 +02:00
Leo R. Lundgren
5861bb031c doc: Improve docs for --files-from et al 2021-04-07 18:31:46 +02:00
rawtaz
c2569ff923 Merge pull request #3354 from greatroar/filechange-docs
docs: on Windows, the filesize must match for "unchanged"
2021-04-07 15:29:46 +02:00
greatroar
ecbe7f3a99 Backup docs: on Windows, the filesize must match for "unchanged" 2021-04-04 17:01:48 +02:00
Michael Eischer
88731d8c28 Add changelog 2021-03-27 17:08:35 +01:00
greatroar
dc88ca79b6 Handle lack of space and remove broken files in SFTP backend 2021-03-27 15:02:14 +01:00
MichaelEischer
efb10b3c40 Merge pull request #3347 from jazcap53/fix_ambiguous_warning
Change ambiguous Warning message
2021-03-27 12:54:07 +01:00
MichaelEischer
d456437ad1 Merge pull request #3343 from hluup/3334
Display 'created new cache in ' message only if output is terminal
2021-03-27 12:43:37 +01:00
MichaelEischer
4c61825249 Merge pull request #3335 from cyounkins/patch-1
prune --max-unused does not limit metadata repacking
2021-03-27 12:40:35 +01:00
Andrew Jarcho
9f44129c2f Change ambiguous Warning message
Fixes #3346

On branch fix_ambiguous_warning
Changes to be committed:
    modified:   cmd_backup.go
2021-03-24 10:44:47 -04:00
Hendrik Luup
5592c17e4a Display 'created new cache in ' message only if output is terminal 2021-03-22 11:31:08 +02:00
Craig Younkins
aa0a7b78a8 doc: prune --max-unused unlimited will still repack metadata
Adding minor clarification to documentation of `prune --max-unused unlimited` to indicate metadata will still be repacked. The referenced PR indicates `max-repack-size` CAN limit metadata repacking.

Ref: https://forum.restic.net/t/max-unused-unlimited-still-repacks/3661
Ref: https://github.com/restic/restic/pull/3228
2021-03-21 13:19:09 -04:00
Alexander Neumann
88a23521dd Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check
s3: Fix sanity check
2021-03-11 13:13:46 +01:00
MichaelEischer
e678acafcf Merge pull request #3331 from greatroar/upgrade-sftp
Upgrade pkg/sftp to 1.13
2021-03-10 22:27:32 +01:00
Michael Eischer
54d58edacc debug: fix usage for examine command 2021-03-10 22:22:33 +01:00
Michael Eischer
5975ed61f3 debug: fix linter warning 2021-03-10 21:22:53 +01:00
Michael Eischer
dc62ec5933 debug: use Printf/Warnf for output 2021-03-10 21:20:21 +01:00
Michael Eischer
547d9b384d debug: cleanup repair code 2021-03-10 21:15:38 +01:00
greatroar
187a77fb27 Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist 2021-03-10 21:05:24 +01:00
Michael Eischer
fa7b9d5dfe debug: Cleanup pack size checks 2021-03-10 20:57:14 +01:00
Michael Eischer
6774fc6454 debug: check arguments and cleanup error handling 2021-03-10 20:43:22 +01:00
Michael Eischer
096f15db5c debug: extract examinePack function 2021-03-10 20:21:05 +01:00
Michael Eischer
84491ff40b debug: store repaired and correct blobs 2021-03-10 20:03:44 +01:00
Alexander Neumann
b3c3121622 debug: Save raw decrypt (disregarding signature) 2021-03-10 20:03:44 +01:00
Alexander Neumann
ce4b6d0874 examine: add byte repair mode 2021-03-10 20:03:44 +01:00
Alexander Neumann
52061e817c debug: fix percentage 2021-03-10 20:03:44 +01:00
Alexander Neumann
133ac42a0b debug: check file content hash 2021-03-10 20:03:44 +01:00
Alexander Neumann
90f975fa1c debug: make output less verbose 2021-03-10 20:03:44 +01:00
Alexander Neumann
086993bae1 debug: check packs not in index, implement repair 2021-03-10 20:03:44 +01:00
Alexander Neumann
d6f78163d4 Add 'debug examine' command to debug #1999 2021-03-10 20:03:44 +01:00
MichaelEischer
c9b4fadd91 Merge pull request #3294 from Achilleshiel/fix-copy-repofile
Add RepositoryFile2 Option for secondary repository
2021-03-08 22:50:31 +01:00
Michael Eischer
da458a55db Cleanup comments in secondary repo test 2021-03-08 22:41:13 +01:00
Wouter Horlings
9ccdba9df6 Add repositoryFile2 option
The `init` and `copy` commands can now use `--repository-file2` flag and
the `$RESTIC_REPOSITORY_FILE2` environment variable.

This also fixes the conflict with the `--repository-file` and `--repo2`
flag.

Tests are added for the initSecondaryGlobalOpts function.

This adds a NOK function to the test helper functions. This NOK tests if
err is not nil, and otherwise fail the test.

With the NOK function a couple of sad paths are tested in the
initSecondaryGlobalOpts function.

In total the tests checks wether the following are passed correct:
   - Password
   - PasswordFile
   - Repo
   - RepositoryFile

The following situation must return an error to pass the test:
   - no Repo or RepositoryFile defined
   - Repo and RepositoryFile defined both
2021-03-08 22:41:13 +01:00
MichaelEischer
a0f9d73d44 Merge pull request #3305 from juergenhoetzel/terminal-race-condition
backup: In case of error also Wait() for terminal goroutine
2021-03-08 22:20:18 +01:00
Michael Eischer
427781928f copy: test that trees with unstable json encoding are properly copied 2021-03-08 22:16:48 +01:00
Michael Eischer
2fc7abac35 copy: copy raw bytes of tree blobs
This avoids problems when for some reason the JSON encoding changes.
This also ensures forward compatibility with future restic versions
which might e.g. add new fields to the tree metadata.
2021-03-08 22:16:48 +01:00
Michael Eischer
f9c581f219 Tweak changelog 2021-03-08 22:11:34 +01:00
Juergen Hoetzel
18fccb5995 backup: In case of error also Wait() for terminal goroutine
This prevents a race condition where the final summary message can get lost.
2021-03-08 22:11:34 +01:00
Michael Eischer
2a9f0f19b6 s3: Fix sanity check
The sanity check shouldn't replace the error message if there is already
one.
2021-03-08 20:23:57 +01:00
rawtaz
d686fa25de Merge pull request #3323 from restic/doc-setcap
doc: Clarify setcap applying only to current binary
2021-03-08 19:32:11 +01:00
rawtaz
f000f41c91 doc: Clarify setcap applying only to current binary 2021-03-08 19:21:06 +01:00
MichaelEischer
64fe733fa0 Merge pull request #3320 from MichaelEischer/fix-ci
Update to Go 1.16 and pin go version used for golangcil-lint
2021-03-08 19:18:10 +01:00
Michael Eischer
781378a65e Bump go version to 1.16 2021-03-04 23:24:26 +01:00
Michael Eischer
64a9272c9a go mod tidy for go 1.16 2021-03-04 23:18:53 +01:00
Michael Eischer
ee89e33f12 CI: Pin go version used for golint to 1.15.x
This prevents unexpected lint failures when a new go version is
released.
2021-03-04 23:12:48 +01:00
Michael Eischer
f0e9068ef2 go mod tidy for go 1.16 2021-03-04 22:56:08 +01:00
Michael Eischer
f70aca6f6f restorer: Skip preallocate test if not supported by the filesystem 2021-03-04 20:33:46 +01:00
MichaelEischer
814a399e4c Merge pull request #3283 from dennypage/master
Treat an empty password as a fatal error for repository init.
2021-02-28 00:49:21 +01:00
Michael Eischer
13730e3844 Add link to pr to changelog 2021-02-28 00:40:52 +01:00
Denny Page
0f41e99ea7 Treat an empty password as a fatal error for repository init. 2021-02-28 00:40:00 +01:00
MichaelEischer
6712c6de73 Merge pull request #3309 from MichaelEischer/fix-random-check-crash
check: Fix crash of --read-data-subset=x% on empty repository
2021-02-27 16:24:56 +01:00
Michael Eischer
9e852af5be check: Fix crash of --read-data-subset=x% on empty repository
Rounding up to at least one pack file should only be done if at least a
pack file exists.
2021-02-27 16:05:36 +01:00
MichaelEischer
4baebdc6f5 Merge pull request #3308 from MichaelEischer/fix-not-a-directory
local: Ignore files in intermediate folders
2021-02-27 15:34:36 +01:00
Michael Eischer
a293fd9aef local: Ignore files in intermediate folders
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
2021-02-27 13:47:55 +01:00
TJ Rana
91e8f0e486 Clean up writing style 2021-02-16 22:19:11 -05:00
TJ Rana
d7d562b287 Clarify checklist items 2021-02-16 22:11:22 -05:00
TJ Rana
76e1ba6fd0 Clean up pull request template 2021-02-16 22:03:33 -05:00
rawtaz
8eb6a5805b Merge pull request #3286 from MichaelEischer/fix-quiet-backup
backup: Correctly handle --quiet flag
2021-02-15 22:35:04 +01:00
Michael Eischer
65bd2a9a49 backup: Correctly handle --quiet flag
The quiet flag changes the backup output to assume a non-interactive
terminal. However, the output progress interval was not set to 0 by
default.
2021-02-15 22:14:58 +01:00
Alexander Neumann
12f0ccc237 helpers: Also push versioned image to Docker 2021-02-15 20:08:40 +01:00
Alexander Neumann
bb53fcfc0d Set development version for 0.12.0 2021-02-14 11:44:24 +01:00
Alexander Neumann
27f241334e Add version for 0.12.0 2021-02-14 11:44:20 +01:00
Alexander Neumann
4e99a3d650 Update manpages and auto-completion 2021-02-14 11:44:20 +01:00
Alexander Neumann
1cb1cd6f44 Generate CHANGELOG.md for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann
1a34260cf0 Prepare changelog for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann
13d52c88fb Merge pull request #3282 from restic/changelog-unreleased
changelog: Correct and slightly polish some unreleased changelog files
2021-02-14 10:49:33 +01:00
Leo R. Lundgren
4b5ca1e914 changelog: Correct and slightly polish some unreleased changelog files 2021-02-14 00:57:54 +01:00
MichaelEischer
917f5b910a Merge pull request #3270 from restic/fix-ls-json
ls: Check for non-nil error before calling panic()
2021-02-09 22:05:04 +01:00
Alexander Neumann
c0f2c1d871 ls: Check for non-nil error before calling panic() 2021-02-07 21:12:54 +01:00
Alexander Neumann
9985368d46 Merge pull request #3255 from MichaelEischer/restorer-check-error
restorer: Check dropped error
2021-02-03 16:10:46 +01:00
Alexander Neumann
2dd592a06c Merge pull request #3254 from lorenz/allow-http2
Allow HTTP/2
2021-02-02 21:08:20 +01:00
Lorenz Brun
362338dd60 Add changelog entry 2021-02-01 20:20:34 +01:00
MichaelEischer
6ac032be64 Merge pull request #3256 from aawsome/rebuild-index-bar-done
rebuild-index: Add missing bar.Done()
2021-01-31 18:48:14 +01:00
MichaelEischer
0ce05d5725 Merge pull request #3236 from Wouter0100/patch-1
doc: Note only path-style URL support for S3
2021-01-31 18:44:58 +01:00
Michael Eischer
0aed8d47d7 docs: Fix typo in S3 URL support 2021-01-31 18:33:15 +01:00
Alexander Weiss
39a26066f7 rebuild-index: add missing bar.Done() 2021-01-31 18:28:02 +01:00
Michael Eischer
47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
MichaelEischer
b3dc127af5 Merge pull request #3207 from aawsome/filerestorer-fix-errorhandling
Fix error handling in filerestorer and fix #3166
2021-01-31 17:47:46 +01:00
Wouter van Os
8442c43209 doc: Update wording for S3 path-only requirement 2021-01-31 15:10:03 +01:00
Alexander Weiss
6e942693ba Fix #3166 2021-01-31 14:22:57 +01:00
Alexander Weiss
5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss
573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
MichaelEischer
b8550a21f2 Merge pull request #3253 from restic/improve-prune-error
prune: Improve error message for missing files
2021-01-31 12:03:20 +01:00
Alexander Neumann
027a51529d prune: Improve error message for missing files
This commit changes the error message so that a list of file names is
printed. Before, just the raw map was printed, which is not a great user
interface.
2021-01-31 11:31:27 +01:00
Lorenz Brun
5427119205 Allow HTTP/2 2021-01-31 02:44:30 +01:00
Alexander Neumann
f647614e24 Merge pull request #3248 from MichaelEischer/backend-cleanups
Backend code and test cleanups
2021-01-30 21:37:33 +01:00
Michael Eischer
e0867c9682 backend: try to cleanup test leftovers 2021-01-30 21:23:20 +01:00
Michael Eischer
f740b2fb23 mem: check upload length before storing upload 2021-01-30 21:23:20 +01:00
Alexander Neumann
0e5f2fff71 Merge pull request #3243 from restic/fix-scanner-overlap
backup: Fix total size for overlapping targets
2021-01-30 21:17:21 +01:00
MichaelEischer
99228be623 Merge pull request #3250 from restic/add-golangci-lint-config
Add golangci lint config, add many error checks
2021-01-30 21:09:38 +01:00
Alexander Neumann
04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann
f867e65bcd Fix issues reported by staticcheck 2021-01-30 20:43:53 +01:00
Alexander Neumann
a00e27adf6 Add entry to changelog 2021-01-30 20:22:25 +01:00
Alexander Neumann
0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
Alexander Neumann
aef3658a5f Address review comments 2021-01-30 20:02:37 +01:00
Alexander Neumann
200f09522d Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann
cbd88c457a backup: Improve error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann
1a0eb05bfa errcheck: Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann
3c753c071c errcheck: More error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann
16313bfcc9 errcheck: Add error check for MergeFinalIndexes() 2021-01-30 20:02:37 +01:00
Alexander Neumann
75f53955ee errcheck: Add error checks
Most added checks are straight forward.
2021-01-30 20:02:37 +01:00
Alexander Neumann
1632a84e7b Add a section explaining golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann
b3d5bf7c99 Update golangci-lint version 2021-01-30 20:02:37 +01:00
Alexander Neumann
57627a307f Add config for golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann
6ab7d49a03 Merge pull request #3251 from MichaelEischer/rest-dropped-error
rest: handle dropped error in save operation
2021-01-30 19:44:50 +01:00
Michael Eischer
a53778cd83 rest: handle dropped error in save operation 2021-01-30 19:25:04 +01:00
Alexander Neumann
dd94efb307 Merge pull request #3249 from MichaelEischer/fix-dropped-gs-error
gs: Don't drop error when finishing upload
2021-01-30 16:24:57 +01:00
Michael Eischer
8a486eafed gs: Don't drop error when finishing upload
The error returned when finishing the upload of an object was dropped.
This could cause silent upload failures and thus data loss in certain
cases. When a MD5 hash for the uploaded blob is specified, a wrong
hash/damaged upload would return its error via the Close() whose error
was dropped.
2021-01-30 13:31:32 +01:00
MichaelEischer
4d576c2f79 Merge pull request #3217 from M4a1x/read-data-subset-fix
Fix missing rand seed for restic check --read-data-subset=x%
2021-01-29 23:34:13 +01:00
Max Stabel
f9e1fa26ff Fix missing rand seed for restic check --read-data-subset=x% 2021-01-29 23:18:35 +01:00
MichaelEischer
fb3cf3f885 Merge pull request #3245 from aawsome/prune-fix-statistics-cacheable
prune: Fix statistics for --repack-cacheable-only
2021-01-29 23:14:49 +01:00
Alexander Weiss
e08e65dc30 prune: Simplify logic selecting packs to repack 2021-01-29 22:27:22 +01:00
Alexander Weiss
daeb4cdf8f prune: Fix statistics for --repack-cacheable-only 2021-01-29 22:27:22 +01:00
Alexander Neumann
cdd704920d azure: Pass data length to Azure libray
The azureAdapter was used directly without a pointer, but the Len()
method was only defined with a pointer receiver (which means Len() is
not present on a azureAdapter{}, only on a pointer to it).
2021-01-29 21:08:41 +01:00
Alexander Neumann
bbdf18c4a2 Merge pull request #3176 from MichaelEischer/backend-content-length
Pass upload size to backends and sanity check it
2021-01-29 20:33:44 +01:00
Michael Eischer
1f583b3d8e backend: test that incomplete uploads fail 2021-01-29 13:51:53 +01:00
Michael Eischer
c73316a111 backends: add sanity check for the uploaded file size
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.

Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
2021-01-29 13:51:51 +01:00
Michael Eischer
4526d5d197 swift: explicitly pass upload size to library
This allows properly setting the content-length which could help the
server-side to detect incomplete uploads.
2021-01-29 13:50:46 +01:00
Michael Eischer
dca9b6f5db azure: explicitly pass upload size
Previously the fallback from the azure library was to read the whole
blob into memory and use that to determine the upload size.
2021-01-29 13:50:46 +01:00
Alexander Neumann
a16ce65295 Merge pull request #3244 from MichaelEischer/better-damage-reports
Print more details about possible repository damages
2021-01-29 11:45:45 +01:00
Alexander Neumann
5c41120c70 Add entry to changelog 2021-01-29 11:31:36 +01:00
Alexander Neumann
5c617859ab backup/scanner: Fix total size for overlapping targets
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.

This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
2021-01-29 11:31:36 +01:00
Alexander Neumann
81211750ba archiver/tree: Introduce functions Leaf() and NodeNames() 2021-01-29 11:11:28 +01:00
rawtaz
de7e3a0648 Merge pull request #2823 from greatroar/trust-mtime
Add --ignore-ctime flag to backup and document change detection
2021-01-29 00:02:19 +01:00
greatroar
6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
Michael Eischer
58b5679f14 prune: reword missing blobs error
The previous wording could be understood such that the prune run did
damage the repository.
2021-01-28 21:48:24 +01:00
Michael Eischer
7b8886c052 prune: report missing but unneeded pack files
This indicates a damaged repository so add some output to help with
debugging.
2021-01-28 21:46:01 +01:00
Michael Eischer
ff95999246 rebuild-index: report added/removed/reindexed files
This should help with investigating missing pack files.
2021-01-28 21:46:01 +01:00
Michael Eischer
b71c52797a find: correctly expand multiple blob ids
For example `restic find --show-pack-id --blob f78dc991 5b9e4366 ddd8c7d4`
would previously only expand one blob if all of them belong to the same
file.
2021-01-28 21:21:54 +01:00
MichaelEischer
82140967d3 Merge pull request #3228 from aawsome/prune-all-trees
prune: Remove all unused trees
2021-01-28 21:04:27 +01:00
MichaelEischer
43cb26010a Merge pull request #3242 from greatroar/fprintln
internal/ui/termstatus: Use Fprintln to get a newline
2021-01-28 20:34:06 +01:00
MichaelEischer
35033d9b79 Merge pull request #3177 from MichaelEischer/fix-2759
prune: don't print stacktrace on console
2021-01-28 20:28:45 +01:00
Alexander Neumann
84822d44d4 Merge pull request #2536 from MatthewVance/threat-model
Update threat model
2021-01-28 14:26:03 +01:00
Matthew Vance
58c7f4694d Update threat model 2021-01-28 14:20:42 +01:00
Alexander Neumann
4d40c70214 Merge pull request #3211 from MichaelEischer/sftp-speedup
Speed-up caching/pack download via SFTP
2021-01-28 14:16:55 +01:00
Alexander Neumann
44169d0dc4 Merge pull request #3205 from MichaelEischer/fix-quiet-verbose
Properly check that --quiet and --verbose are not combined
2021-01-28 13:53:31 +01:00
Alexander Neumann
6aa7e9f9c6 Merge pull request #3174 from MichaelEischer/parallelize-lock-loading
Parallelize lock file loading
2021-01-28 13:52:12 +01:00
Alexander Neumann
bdfedf1f5b Merge pull request #3173 from MichaelEischer/unify-index-loading
Unify index loading
2021-01-28 13:50:42 +01:00
greatroar
b9cfe6f68a internal/ui/termstatus: Use Fprintln to get a newline 2021-01-28 13:30:10 +01:00
Alexander Neumann
72eec8c0c4 Merge pull request #3106 from MichaelEischer/parallel-tree-walk
Parallelize tree walk in prune and copy and add progress bar to check
2021-01-28 12:06:42 +01:00
Michael Eischer
68608a89ad restic: add comment about StreamTrees shutdown 2021-01-28 11:10:50 +01:00
Michael Eischer
1e306be000 Add changelog entry 2021-01-28 11:10:50 +01:00
Michael Eischer
ddb7697d29 restic: Test progress reporting of StreamTrees 2021-01-28 11:10:50 +01:00
Michael Eischer
313ad0e32f progress/counter: Fix test for final report call 2021-01-28 11:10:50 +01:00
Michael Eischer
e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer
505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer
eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer
258ce0c1e5 parallel: report progress for StreamTrees
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
2021-01-28 11:08:43 +01:00
Michael Eischer
3d6a3e2555 copy: Remove treeCloner struct 2021-01-28 11:08:43 +01:00
Michael Eischer
0caad1e890 copy: parallelize tree walk 2021-01-28 11:08:43 +01:00
Michael Eischer
f2a1b125cb restic: Actually parallelize FindUsedBlobs 2021-01-28 11:08:43 +01:00
Michael Eischer
6e03f80ca2 check: Split the parallelized tree loader into a reusable component
The actual code change is minimal
2021-01-28 11:08:43 +01:00
Michael Eischer
1d7bb01a6b check: Cleanup tree loading and switch to use errgroup
The helper methods are now wired up in the Structure method.
2021-01-28 11:08:43 +01:00
Alexander Neumann
a4689eb3b9 Merge pull request #3199 from MichaelEischer/non-interactive-counter
Don't print progress on non-interactive terminals
2021-01-28 10:53:38 +01:00
Alexander Neumann
c5a66e9181 ui: Simlify channel receive 2021-01-28 10:42:02 +01:00
Wouter van Os
b5972f184c doc: Note only path-style URL support for S3
This adds a node to mention that currently only path-style URL's are supported for S3, as per code at:
- aa0faa8c7d/internal/backend/s3/config.go (L42-L45)
- aa0faa8c7d/internal/backend/s3/config.go (L48-L62)
2021-01-23 16:54:08 +01:00
Alexander Weiss
d7dc19a496 prune: Always repack packs containing tree blobs 2021-01-15 16:42:04 +01:00
Michael Eischer
f3442ce8a5 Test that WriteTo of a backend's Load remains accessible 2021-01-03 22:23:53 +01:00
Michael Eischer
678e75e1c2 sftp: enforce use of optimized upload method
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
2021-01-03 22:23:53 +01:00
Michael Eischer
6b5b29dbee limiter: add unit tests 2021-01-03 22:23:53 +01:00
Michael Eischer
f35f2c48cd limiter: support WriteTo in LimitBackend for read rate limiting 2021-01-03 22:23:53 +01:00
Michael Eischer
bcb852a8d0 hashing: support WriteTo in the reader 2021-01-03 22:23:53 +01:00
MichaelEischer
aa0faa8c7d Merge pull request #3208 from restic/add-mips
Add mips* architectures to CI and release
2021-01-03 16:06:10 +01:00
MichaelEischer
f7ec263a22 Merge pull request #3109 from aawsome/optimize-filerestorer
restore: Don't save pack content in memory
2021-01-03 14:53:41 +01:00
Alexander Neumann
7d665fa1f4 Add entry to changelog 2021-01-03 14:41:11 +01:00
Michael Eischer
69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Neumann
36db248e30 Split cross compilation targets into two jobs 2021-01-03 10:50:54 +01:00
Alexander Neumann
eb72b10f55 Add mips* architectures to CI and release 2021-01-03 10:41:54 +01:00
Alexander Neumann
622f4c7daa Run cloud backends also on pushes 2021-01-03 10:38:10 +01:00
Alexander Neumann
f8c50394d6 Revert "Run cloud tests if secrets are available"
This reverts commit aa648bdcac.
2021-01-03 10:28:01 +01:00
Alexander Neumann
aa648bdcac Run cloud tests if secrets are available 2021-01-03 10:26:55 +01:00
Alexander Neumann
e8abc79ce9 Add hints for keeping the list of architectures in sync 2021-01-01 10:09:04 +01:00
Alexander Weiss
34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss
7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss
07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss
3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Michael Eischer
97388b3504 Properly check that --quiet and --verbose are not combined
If --verbose is specified once, then globalOptions.Verbose == 1.
Previously --quiet --verbose would silently ignore the --verbose flag.
2020-12-30 21:24:18 +01:00
Alexander Neumann
8b84c96d9d Merge pull request #3204 from MichaelEischer/archiver-tomb-race
archiver: fix race condition during worker startup
2020-12-30 20:04:20 +01:00
Michael Eischer
debc4a3a99 archiver: fix race condition during worker startup
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.

To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
2020-12-30 17:31:22 +01:00
MichaelEischer
e1efc193e1 Merge pull request #3139 from aawsome/prune-healing
prune: Add healing of repository in some situations
2020-12-29 22:17:27 +01:00
Alexander Weiss
f0113139ea prune: Correct error message 2020-12-29 20:20:05 +01:00
Alexander Weiss
f6df94a50e prune: Add self-healing
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
2020-12-29 20:20:05 +01:00
MichaelEischer
31e56f1ad5 Merge pull request #3197 from SkYNewZ/fix/3183
Fix tag handling for multiple tag lists
2020-12-29 18:44:38 +01:00
MichaelEischer
7fda2f2ad8 Merge pull request #3134 from greatroar/unlock-warn
Warn when unlock fails instead of returning an error
2020-12-29 18:30:08 +01:00
greatroar
dec5008369 Warn when unlock fails instead of returning an error
Only one caller was checking the error.
2020-12-29 17:48:20 +01:00
Quentin Lemaire
873505ed3b Update related changelog 2020-12-29 17:12:46 +01:00
Quentin Lemaire
25ecf9eafb fix(cmd_tag): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Quentin Lemaire
e88f3fb80c fix(cmd_backup): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Alexander Neumann
b2efa0af39 Merge pull request #3164 from MichaelEischer/improve-context-cancel
Improve context cancel handling in archiver and backends
2020-12-29 17:03:42 +01:00
Michael Eischer
25f4acdaa8 Add changelog entry 2020-12-29 16:32:18 +01:00
Michael Eischer
cff4955a48 ui: remove dead struct member 2020-12-29 16:32:18 +01:00
Michael Eischer
05a987b07c Support values less than 1 for RESTIC_PROGRESS_FPS
For example set the variable to 0.016666 to print the progress once per
minute.
2020-12-29 16:32:18 +01:00
Michael Eischer
92da5168e1 ui: force backup progress update on signal 2020-12-29 16:32:18 +01:00
Michael Eischer
34afc93ddc ui/progress: extract signal handling into own package 2020-12-29 16:32:18 +01:00
Michael Eischer
023eea6463 ui: don't shorten non-interactive progress output 2020-12-29 16:32:18 +01:00
Michael Eischer
684600cf42 ui: update status for the backup command on non-interactive terminals
Allow the backup command to print status on non-interactive terminals.
The output is disabled by setting a MinUpdatePause == 0.
2020-12-29 16:03:43 +01:00
Michael Eischer
85fe5feadb Unify progress report frequency calculation 2020-12-29 16:03:43 +01:00
Michael Eischer
969141b5e9 Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals
This makes it possible to use the environment variable to also get
regular progress updates on non-interactive terminals.
2020-12-29 16:03:43 +01:00
Michael Eischer
13ce981794 ui: cleanup backup status shutdown 2020-12-29 16:03:43 +01:00
Michael Eischer
c2ef049f1b ui/progress: don't print progress on non-interactive terminals
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
2020-12-29 16:03:43 +01:00
MichaelEischer
a488d4c847 Merge pull request #2833 from greatroar/aix
AIX port
2020-12-29 12:32:50 +01:00
Alexander Neumann
4133b1ea65 Synchronize OS and architectures for testing 2020-12-29 11:11:50 +01:00
Alexander Neumann
46d2ca5095 Remove old test script 2020-12-29 11:02:48 +01:00
Quentin Lemaire
334d8ce724 feat(tags): Create Flatten() method 2020-12-29 10:59:46 +01:00
Alexander Neumann
c661518df9 Merge pull request #2793 from greatroar/rclone-doc
Clarify rclone-over-SSH docs
2020-12-29 10:46:34 +01:00
Michael Eischer
0d81f16343 Add AIX as cross-compile target to CI 2020-12-29 01:35:01 +01:00
greatroar
3b09ae9074 AIX port 2020-12-29 01:35:01 +01:00
greatroar
18531e3d6f Portability fixes to internal/restic
syscall.Mknod is not available on AIX.
2020-12-29 01:35:01 +01:00
Michael Eischer
ca07317815 add changelog 2020-12-28 21:06:47 +01:00
Michael Eischer
d0ca8fb0b8 backend: test that a canceled context prevents RetryBackend operations 2020-12-28 21:06:47 +01:00
Michael Eischer
08b7f2b58d archiver: test that context canceled error is not dropped 2020-12-28 21:06:47 +01:00
Michael Eischer
e483b63c40 retrybackend: Fail operations when context is already canceled
Depending on the used backend, operations started with a canceled
context may fail or not. For example the local backend still works in
large parts when called with a canceled context. Backends transfering
data via http don't work. It is also not possible to retry failed
operations in that state as the RetryBackend will abort with a 'context
canceled' error.

Ensure uniform behavior of all backends by checking for a canceled
context by checking for a canceled context as a first step in the
RetryBackend. This ensures uniform behavior across all backends, as
backends are always wrapped in a RetryBackend.
2020-12-28 21:06:47 +01:00
Michael Eischer
fc60b560ba archiver: Let saveTree report a canceled context as an error
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
2020-12-28 21:06:47 +01:00
Michael Eischer
736e964317 archiver: Don't loose error if background context is canceled
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
2020-12-28 21:06:47 +01:00
MichaelEischer
9c41e4a343 Merge pull request #3192 from DRON-666/fix-options
Fix formatting of `restic options` command
2020-12-23 22:53:40 +01:00
DRON-666
332b1896d1 Some options fixes
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
2020-12-23 23:26:04 +03:00
MichaelEischer
cb6b0f6255 Merge pull request #3181 from tWido/feature-no-retry-permission
Don't retry when "Permission denied" occurs in local backend
2020-12-23 20:24:29 +01:00
MichaelEischer
1e73aac610 Merge pull request #3179 from aawsome/check-filesize
check: Remove filesize counter
2020-12-23 20:22:02 +01:00
Alexander Weiss
2a1add7538 check: remove file size counter 2020-12-23 02:34:31 +01:00
tWido
7dab113035 Don't retry when "Permission denied" occurs in local backend 2020-12-22 23:41:12 +01:00
MichaelEischer
8efb874f48 Merge pull request #3148 from aawsome/simplify-index-rebuild
rebuild-index: code simplification
2020-12-22 23:27:53 +01:00
Michael Eischer
de99207046 repository: tweak comment for packs method 2020-12-22 23:01:58 +01:00
Alexander Weiss
d8d2cc6dd9 Simplify rebuild-index code 2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer
b9f5d3fe13 repository: Add test for ForAllIndexes 2020-12-22 22:36:18 +01:00
Michael Eischer
a12c5f1d37 repository: move otherwise unused LoadIndex to tests 2020-12-22 22:36:18 +01:00
Michael Eischer
24474a36f4 repository: deduplicate index loading implementation 2020-12-22 22:36:18 +01:00
Michael Eischer
ccc84af73d debug/list: parallelize index loading 2020-12-22 22:36:18 +01:00
Michael Eischer
96904f8972 check: extract parallel index loading 2020-12-22 22:36:18 +01:00
MichaelEischer
69f9d269eb Merge pull request #3189 from restic/fix-counter-segfault
ui/progress: Use mutex instead of atomic
2020-12-22 22:14:08 +01:00
rawtaz
ec59c73489 Merge pull request #3188 from restic/forget-prune-dry-run
forget: Enable --dry-run together with --prune
2020-12-22 21:09:15 +01:00
Alexander Neumann
6c514adb8a ui/progress: Use mutex instead of atomic
The counter value needs to be aligned to 64 bit in memory for the
atomic functions to work on some platform (such as 32 bit ARM).

The atomic package says in its documentation:

> These functions require great care to be used correctly. Except for
> special, low-level applications, synchronization is better done with
> channels or the facilities of the sync package.

This commit replaces the atomic functions with a simple sync.Mutex, so
we don't have to care about alignment.
2020-12-22 21:03:27 +01:00
Alexander Neumann
edf89e1c74 forget: Enable --dry-run together with --prune 2020-12-22 20:58:02 +01:00
Alexander Neumann
f7c7c2f730 Merge pull request #3175 from MichaelEischer/fix-3099-changelog
Fix changelog for #3099
2020-12-20 11:29:57 +01:00
Michael Eischer
cfea79d0c5 prune: don't print stacktrace on console 2020-12-19 14:28:48 +01:00
Michael Eischer
5cd40f8b58 Add changelog 2020-12-19 11:38:38 +01:00
MichaelEischer
d32949ee54 Merge pull request #3081 from DRON-666/dump-zip
Add zip support to the `dump` command
2020-12-19 11:33:33 +01:00
DRON-666
83b10dbb12 Deduplicate dumper closing logic 2020-12-19 02:42:46 +03:00
DRON-666
e136dd8696 Deduplicate test code 2020-12-19 02:06:54 +03:00
DRON-666
33adb58817 Minor fixes and linter suggestions 2020-12-19 02:04:17 +03:00
DRON-666
da9053b184 Some gramma fixes in documentation 2020-12-19 01:16:15 +03:00
DRON-666
ef1aeb8724 dump: Update docs and changelog 2020-12-19 01:09:47 +03:00
DRON-666
2ca76afc2b dump: Add new option --archive 2020-12-19 01:09:47 +03:00
DRON-666
89ab6d557e dump: Add zip dumper 2020-12-19 01:09:47 +03:00
DRON-666
0256f95994 dump: Split tar and walk logic 2020-12-19 01:09:47 +03:00
Michael Eischer
bfadc82a20 Fix changelog for #3099 2020-12-18 20:58:24 +01:00
Michael Eischer
34b6130a0e restic: parallelize lock file loading 2020-12-18 20:46:16 +01:00
MichaelEischer
22260d130d Merge pull request #3170 from greatroar/dont-retry
Don't retry permanent errors in backends
2020-12-17 23:29:56 +01:00
greatroar
9341a83b05 Changelog entry for #2453 and #3170 2020-12-17 23:15:48 +01:00
greatroar
66d904c905 Make invalid handles permanent errors 2020-12-17 12:47:53 +01:00
greatroar
746dbda413 Mark "ssh exited" errors in SFTP as permanent 2020-12-17 12:43:09 +01:00
greatroar
f7784bddb3 Don't retry when "no space left on device" in local backend
Also adds relevant documentation to the restic.Backend interface.
2020-12-17 12:43:09 +01:00
Alexander Neumann
1cdd38d9e0 Merge pull request #3165 from restic/2747-doc-forget
doc: Clarify calendar boundaries for --keep-* options
2020-12-16 08:38:03 +01:00
Leo R. Lundgren
b3c0d2f45b doc: Clarify calendar boundaries for --keep-* options 2020-12-15 12:43:06 +01:00
Peter Albrecht
cd25e36811 Add PGP fingerprint to 020_installation.rst
I like the idea of verifying the integrity of applications, I download from the internet. So I was very happy to see that restic does provide SHA256-checksums which are signed with the maintainers PGP key.

The only thing I miss: I could not find a direct way to download the used PGP key and verify the keys fingerprint.

Doing some searches, I found:
* https://github.com/restic/rest-server/issues/121
* https://restic.net/blog/2015-09-16/verifying-code-archive-integrity/

To help other restic users, I think you should add information about your PGP key/fingerprint to this installation doc, too. To save you some precious time, I created a draft, how this doc might be expanded, in this pull-request. You are free to accept it or change the text to your liking.

I copied the key/fingerprint text from: ``restic/restic/master/doc/090_participating.rst``

Thank you for your work in restic!
2020-12-13 17:00:00 +01:00
Alexander Neumann
e96677cafb Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables
swift: Add support for id based keystone v3 auth parameters
2020-12-12 16:27:38 +01:00
Michael Eischer
1d69341e88 swift: Add support for id based keystone v3 auth parameters
This adds support for the following environment variables, which were
previously missing:

OS_USER_ID            User ID for keystone v3 authentication
OS_USER_DOMAIN_ID     User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID  Project domain ID for keystone v3 authentication
OS_TRUST_ID           Trust ID for keystone v3 authentication
2020-12-11 19:22:34 +01:00
Alexander Neumann
36c5d39c2c Fix issues reported by semgrep 2020-12-11 09:41:59 +01:00
Alexander Neumann
7facc8ccc1 Merge pull request #2505 from aawsome/fix-repo-configfile
Fix repo configfile
2020-12-07 07:52:37 +01:00
Alexander Neumann
ba31c6fdaa Merge pull request #3150 from MichaelEischer/fix-windows-redir-output
termstatus: Fix canUpdateStatus detection for redirected output on windows
2020-12-06 21:17:27 +01:00
Alexander Neumann
b58799d83a Merge pull request #3152 from MichaelEischer/fix-backup-background-hang
backup: Fix shutdown hang when running in the background on linux
2020-12-06 21:16:00 +01:00
Alexander Neumann
0d5b764f90 Merge pull request #3130 from aawsome/snapshots-parallel
Make loading snapshots parallel
2020-12-06 21:08:18 +01:00
Alexander Weiss
d6b3859e48 Add changelog 2020-12-06 19:29:18 +01:00
Michael Eischer
b48f579530 termstatus: Fix canUpdateStatus detection for redirected output
The canUpdateStatus check was simplified in #2608, but it accidentally flipped
the condition. The correct check is as follows: If the output is a pipe then
restic probably runs in mintty/cygwin. In that case it's possible to
update the output status. In all other cases it isn't.

This commit inverts to condition again to offer the previous and correct
behavior.
2020-12-06 19:02:42 +01:00
Michael Eischer
401ef92c5f backup: Fix shutdown hang when running in the background
On shutdown the backup commands waits for the terminal output goroutine
to stop. However while running in the background the goroutine ignored
the canceled context.
2020-12-06 18:53:41 +01:00
Alexander Weiss
e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss
26f85779be Parallelize ForAllSnapshots 2020-12-06 05:09:58 +01:00
Alexander Weiss
5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
MichaelEischer
3264eae9f6 Merge pull request #3149 from aawsome/fix-rebuild-index
Bugfix for rebuild-index
2020-12-05 22:18:22 +01:00
Alexander Weiss
83c8a9b058 Bugfix: packSizeFromList should save size from List() 2020-12-05 20:58:36 +01:00
Alexander Neumann
43cf301450 Merge pull request #3141 from MichaelEischer/fix-fuse-dir-owner
fuse: Properly set uid/gid for directories
2020-12-01 16:06:24 +01:00
Michael Eischer
d05c88a5d6 fuse: Properly set uid/gid for directories
In #2584 this was changed to use the uid/gid of the root node. This
would be okay for the top-level directory of a snapshot, however, this
change also applied to normal directories within a snapshot. This
change reverts the problematic part and adds a test that directory
attributes are represented correctly.
2020-11-30 23:41:49 +01:00
Alexander Neumann
058b102db0 Merge pull request #3138 from MichaelEischer/fix-debug-build 2020-11-30 12:27:51 +01:00
rawtaz
fcebc7d250 Merge pull request #3135 from restic/manual
doc: Fix misc missing/incorrect text in manual
2020-11-29 19:20:51 +01:00
MichaelEischer
f2959127b6 Merge pull request #3065 from greatroar/local-subdirs
Don't recurse in local backend's List if not required
2020-11-29 19:03:59 +01:00
Leo R. Lundgren
61460dee52 doc: Fix misc missing/incorrect text in manual 2020-11-29 18:59:24 +01:00
Michael Eischer
54a6d98945 Enable debug builds for CI 2020-11-29 18:47:00 +01:00
Michael Eischer
f72f6c9c80 Fix debug build 2020-11-29 18:44:36 +01:00
MichaelEischer
52b98f7f95 Merge pull request #3017 from greatroar/files-from0
Add backup options --files-from-verbatim and --files-from-raw
2020-11-29 18:15:21 +01:00
Alexander Neumann
04d856e601 Merge pull request #3136 from restic/rawtaz-copy-doc
doc: Emphasize double transfer and duplication in copy command
2020-11-29 13:57:10 +01:00
Alexander Neumann
a7b49c4889 Merge pull request #3119 from restic/keep-mountpoints
Keep mountpoints as empty directories for --one-file-system
2020-11-29 11:22:12 +01:00
rawtaz
a568211b98 Merge pull request #3128 from vrenaville/tzdata
[FIX] Timezone in docker image
2020-11-28 22:43:18 +01:00
Leo R. Lundgren
f70b10d0ee doc: Emphasize double transfer and duplication in copy command 2020-11-28 19:49:34 +01:00
greatroar
55bf76ba0c backup: Add --files-from-{verbatim,raw} options 2020-11-28 18:22:31 +01:00
Alexander Neumann
162117c42c Add changelog 2020-11-28 17:00:31 +01:00
Alexander Neumann
82ae942965 backup: Keep mountpoints for --one-file-system
When a file system is mounted at a directory, lstat() returns attributes
of the root node of the mounted file system, including the device ID of
the other file system. The previous code used when --one-file-system is
specified excluded the directory itself because of that.

This commit changes the code so that mountpoints are kept as empty
directories, its attributes set to the root note of the mounted file
system. The behavior mimics `tar`, which does the same.
2020-11-28 17:00:31 +01:00
Alexander Neumann
f576d3d826 Add tests 2020-11-28 17:00:31 +01:00
Alexander Neumann
037f0a4c91 Refactor device ID checking 2020-11-28 17:00:31 +01:00
Alexander Neumann
2f9346a5af Merge pull request #3125 from metalsp0rk/cat-respect-no-lock
Make restic cat respect --no-lock
2020-11-28 12:45:59 +01:00
Alexander Neumann
a3105799c9 Print warning when unlocking the repo fails 2020-11-28 11:31:25 +01:00
Kyle Brennan
666768cd17 cat: Respect --no-lock flag 2020-11-28 11:30:02 +01:00
vrenaville
adc7a6555f [FIX] Timezone in docker image 2020-11-27 08:05:19 +01:00
Alexander Neumann
9a97095a4c Merge pull request #3120 from aawsome/blob-implementation
Blob implementation
2020-11-22 20:59:30 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss
e3013271a6 Harmonize naming 2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Neumann
c844580e0f Merge pull request #3101 from aawsome/packsizes
Compute packsizes in MasterIndex
2020-11-22 15:49:19 +01:00
Alexander Weiss
67c938f232 Use PackSize in prune 2020-11-21 22:13:54 +01:00
Alexander Weiss
a851c53cbe Use PackSize in checker 2020-11-21 22:13:54 +01:00
Alexander Weiss
4960b841e6 Use PackSize in rebuild-index 2020-11-21 22:13:54 +01:00
Alexander Weiss
ce5d630681 Add MasterIndex.PackSize() 2020-11-21 22:13:54 +01:00
Alexander Weiss
c3ddde9e7d Return hdrSize in ListPack 2020-11-21 22:13:54 +01:00
rawtaz
cac481634c Merge pull request #3115 from johanbove/patch-1
Docs: Update 04_backup.rst
2020-11-20 11:23:03 +01:00
Johan Bové
c23b1a4cba Update 04_backup.rst
Fixed typo - _files_ are included from _folders_, not other _files_.
2020-11-20 07:52:23 +01:00
Alexander Neumann
110a32a08b Merge pull request #3113 from aawsome/fix-prune-stats-duplicates
Fix statistics in prune for duplicates
2020-11-19 20:34:11 +01:00
greatroar
8e213e82fc backend/local: replace fs.Walk with custom walker
This code is more strict in what it expects to find in the backend:
depending on the layout, either a directory full of files or a directory
full of such directories.
2020-11-19 16:46:42 +01:00
Alexander Neumann
8a150ee91f Merge pull request #3112 from MichaelEischer/reduce-progress-log-spam
Limit progress bar updates to once per second on non-terminal outputs
2020-11-19 07:54:36 +01:00
Alexander Weiss
cb5ec7ea6b Fix statistics in prune for duplicates
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.

If not all duplicates are marked for repacking, we lack the
information which

The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
2020-11-18 22:30:22 +01:00
Michael Eischer
625410f003 Limit progress bar updates to once per second on non-terminal outputs
The code accidentally checked whether stdin is a terminal instead of
stdout, the former is not relevant here as the output is printed on
stdout.
2020-11-18 22:12:07 +01:00
Alexander Neumann
75eff92b56 Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init
do not require gs bucket permissions to init repository
2020-11-18 16:53:45 +01:00
eleith
a24e986b2b do not require gs bucket permissions to init repository
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.

these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.

this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.

this way, restic can still initialize a repository if the service
account does have object permissions

fixes: https://github.com/restic/restic/issues/3100
2020-11-18 06:14:11 -08:00
rawtaz
6822ce8479 Merge pull request #3105 from restic/fix-init-repo-file
Allow using --repository-file in init
2020-11-17 21:54:44 +01:00
Alexander Neumann
d857fb6e59 Allow using --repository-file in init 2020-11-17 20:43:46 +01:00
rawtaz
342520b648 Merge pull request #3103 from greatroar/rephrase-pr3102
Rephrase #3095/#3102 changelog entry
2020-11-17 12:52:12 +01:00
greatroar
028f2b8c0e Rephrase #3095/#3102 changelog entry 2020-11-17 12:37:06 +01:00
rawtaz
1b6e8c888f Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param
Remove `--drive-use-trash=false` from rclone param
2020-11-17 11:21:20 +01:00
Alexander Neumann
5f3b802ee7 Merge pull request #3099 from MichaelEischer/check-less-memory 2020-11-16 10:11:25 +01:00
Michael Eischer
022dc35be9 Add changelog 2020-11-15 19:02:51 +01:00
Michael Eischer
1f43cac12d check: Only track data blobs when unused blobs should be reported
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
Michael Eischer
6da66c15d8 check: Simplify referenced blob tracking
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
2020-11-15 18:42:55 +01:00
Michael Eischer
3500f9490c check: Simplify blob status tracking
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
Michael Eischer
b8c7543a55 check: Merge 'size could not be found' and 'not found in index' errors
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.

The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
2020-11-15 18:41:50 +01:00
MichaelEischer
45ba456291 Merge pull request #3038 from fgma/check-data-subset-percentage
Check data subset: check random percentage subset
2020-11-15 18:24:13 +01:00
Michael Eischer
caac38ed27 check: Tweak subset percentage over 100% error message 2020-11-15 18:13:50 +01:00
Michael Eischer
15c537f9db Rename changelog entry to contain issue id 2020-11-15 18:13:50 +01:00
fgma
8f9cea8cc0 Check data subset: check random percentage subset 2020-11-15 18:13:50 +01:00
Alexander Neumann
3c0c0c132b Merge pull request #3006 from aawsome/new-rebuild-index
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
Alexander Neumann
9968220652 Merge pull request #2850 from greatroar/packer-malloc
Decrease allocation rate in internal/pack
2020-11-15 17:19:49 +01:00
kitone
0649828555 Add changelog 2020-11-15 17:09:30 +01:00
kitone
3c03b35212 Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error 2020-11-15 17:09:30 +01:00
greatroar
ab2b7d7f9a Decrease allocation rate in internal/pack
internal/repository benchmark results:

name             old time/op    new time/op    delta
PackerManager-8     179ms ± 1%     181ms ± 1%   +0.78%  (p=0.009 n=10+10)

name             old speed      new speed      delta
PackerManager-8   294MB/s ± 1%   292MB/s ± 1%   -0.77%  (p=0.009 n=10+10)

name             old alloc/op   new alloc/op   delta
PackerManager-8    91.3kB ± 0%    72.2kB ± 0%  -20.92%  (p=0.000 n=9+7)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.38k ± 0%     0.76k ± 0%  -45.20%  (p=0.000 n=10+7)
2020-11-15 16:51:47 +01:00
greatroar
9a8a2cae4c Move pack testing logic to test file 2020-11-15 16:45:49 +01:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss
3d1d5295cc Add changelog 2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss
187c8fb259 Parallelize MasterIndex.Save() 2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
Alexander Weiss
5898cb341f Use CreateIndexFromPacks() in test 2020-11-15 07:05:05 +01:00
Alexander Weiss
43732bb885 Add CreateIndexFromPacks() 2020-11-15 07:04:51 +01:00
MichaelEischer
6feaf6bd1f Merge pull request #3030 from greatroar/concrete-statt
Replace restic.statT interface by concrete types
2020-11-14 23:32:17 +01:00
greatroar
c45f8ee075 Replace restic.statT interface by concrete types
name                old time/op    new time/op    delta
NodeFillUser-8        1.81µs ± 9%    1.50µs ± 5%  -17.07%  (p=0.000 n=19+20)
NodeFromFileInfo-8    1.76µs ± 4%    1.49µs ± 6%  -15.63%  (p=0.000 n=20+19)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)
NodeFromFileInfo-8      496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
NodeFromFileInfo-8      3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
MichaelEischer
3601a9b6cd Merge pull request #2690 from SkYNewZ/master
Fix #2688: Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
Michael Eischer
fdec8051ab tags: Tweak description of --add/set/remove options 2020-11-14 22:55:30 +01:00
MichaelEischer
333c5a19d4 Merge pull request #3082 from aawsome/check-sizes
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
Quentin Lemaire
a8ad6b9a4b fix(tags): Change tags list type according to restic.TagList 2020-11-14 16:24:58 +00:00
Quentin Lemaire
b0882b3f3c fix(snapshots): Update help message to better match the 'forget' command one 2020-11-14 15:48:57 +00:00
Quentin Lemaire
e74110a833 docs: Write new entry to changelog/unreleased 2020-11-14 15:48:56 +00:00
Quentin Lemaire
ae441d3134 fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Neumann
913a34f568 Merge pull request #3094 from restic/fix-ci-tests
Fix ci tests
2020-11-14 11:24:28 +01:00
Alexander Neumann
468612b108 Disable fuse test for macOS 2020-11-14 11:02:49 +01:00
Alexander Weiss
7eabcabf68 Adjust changelog 2020-11-14 00:42:49 +01:00
Alexander Weiss
17bb77b1f9 check: Also check blob length and offset 2020-11-14 00:42:49 +01:00
Alexander Weiss
80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
tofran
94a154c7ca Remove --drive-use-trash=false from rclone param
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
Alexander Neumann
219d9a62f2 Download assets from repo 2020-11-13 21:56:14 +01:00
rawtaz
e8713bc209 Merge pull request #3093 from restic/fix-diff-metadata
diff: Correctly count top-level blobs
2020-11-13 21:50:29 +01:00
Alexander Neumann
04d1983800 Merge pull request #3090 from fgma/vss-fix-386
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
fgma
88208c3db2 vss: improved changelog entry 2020-11-13 21:13:17 +01:00
Alexander Neumann
59ea5a4208 diff: Correctly count top-level blobs 2020-11-13 21:11:21 +01:00
Alexander Neumann
145830005b Use old versions, otherwise tar doesn't work 2020-11-12 21:35:01 +01:00
Alexander Neumann
8ad9f88993 helpers: Improve error message 2020-11-12 20:38:31 +01:00
fgma
859d89b032 vss: add changelog file for issue 3090 2020-11-12 19:38:22 +01:00
fgma
f9223cd827 vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386 2020-11-12 19:31:00 +01:00
Alexander Neumann
0be906a92f CI: Use netcologne mirror 2020-11-11 20:59:23 +01:00
rawtaz
dfb9326b1b Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
Alexander Neumann
e4e0ce09ad CI: Update links for Windows binaries 2020-11-11 20:51:01 +01:00
Alexander Neumann
0334114865 CI: Hardcode mirror URL 2020-11-11 20:42:47 +01:00
Alexander Neumann
b1bbdcb637 Improve changelog 2020-11-11 20:30:30 +01:00
Alexander Neumann
4a0b7328ec s3: Remove dots for config description 2020-11-11 20:20:35 +01:00
Alexander Neumann
3e0456d88b Highlight that s3.list-objects-v1 is a temporary 2020-11-11 20:11:35 +01:00
Alexander Neumann
dd94174379 Merge pull request #3086 from greatroar/diff-errors
Improve error reporting from restic diff
2020-11-11 19:48:03 +01:00
greatroar
63e32c44c0 Improve error reporting from restic diff
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
Nick Douma
f013662e3f Remove separate section on Ceph, and move s3.list-objects-v1 note to S3 section 2020-11-11 15:11:14 +01:00
Nick Douma
4320ff2bbf Add changelog entry for s3.list-objects-v1 2020-11-11 12:33:22 +01:00
Nick Douma
354b7e89cc Document the extended s3.list-objects-v1 flag in a new Ceph section 2020-11-11 12:32:46 +01:00
Nick Douma
829959390a Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1 2020-11-11 11:54:38 +01:00
Nick Douma
ccd55d529d Add s3.list-objects-v1 extended option and default to false 2020-11-11 11:54:36 +01:00
Nick Douma
4ddcc17135 Add support for boolean extended options 2020-11-11 11:54:27 +01:00
MichaelEischer
407843c5f9 Merge pull request #3034 from hoyho/gh_master
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
MichaelEischer
46d31ab86d Merge pull request #3058 from greatroar/counter
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
Alexander Neumann
c986823d3f Merge pull request #3048 from aawsome/check-pack-index
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
Alexander Weiss
239931578c check: check index for packs that are read 2020-11-09 17:28:14 +01:00
hoyho
9df52327cc bugfix: omit ENOTDATA for extended attributes
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
rawtaz
605db3b389 Merge pull request #3077 from MichaelEischer/tame-golangci-lint
Switch to the default set of golangci-lint linters
2020-11-08 22:34:27 +01:00
Michael Eischer
8de129e12f Switch to the default set of golangci-lint linters 2020-11-08 22:17:50 +01:00
Alexander Neumann
2072f0a481 Merge pull request #3076 from restic/check-go-mod-tidy
Check that go.mod/go.sum are up to date
2020-11-08 21:37:22 +01:00
Alexander Neumann
5731e391f8 Merge pull request #3075 from restic/warn-prune-no-cache
UI: Add several hints
2020-11-08 20:37:10 +01:00
Alexander Neumann
6a0a1d1f1c Check that go.mod/go.sum are up to date 2020-11-08 20:32:25 +01:00
Alexander Neumann
a8d21b5dcf prune: Warn if no cache is present 2020-11-08 20:25:35 +01:00
Alexander Neumann
823d0afd6e backup: Always print parent snapshot info 2020-11-08 20:25:35 +01:00
Alexander Neumann
a5989707ac Move test for golang-ci 2020-11-08 19:58:16 +01:00
Alexander Neumann
3a0cfafeb5 Fix quotes 2020-11-08 19:52:37 +01:00
Alexander Neumann
c923bd957d Fix test for pull request 2020-11-08 19:51:11 +01:00
Alexander Neumann
1a3f885d3d Only run golangci-lint for PRs 2020-11-08 17:34:07 +01:00
Alexander Neumann
3bf43d7951 Merge pull request #2475 from restic/use-github-actions
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
Alexander Neumann
561da92396 Replace badges in README 2020-11-08 17:26:59 +01:00
Alexander Neumann
5cf42884c8 Run tests on GitHub Actions 2020-11-08 16:04:57 +01:00
MichaelEischer
9e4e0077fb Merge pull request #3069 from greatroar/xattr-eintr
Update pkg/xattr to handle EINTR on Linux
2020-11-08 12:52:23 +01:00
MichaelEischer
1758da855f Merge pull request #3068 from aawsome/context-mem-backend
Return context error in mem backend
2020-11-08 12:24:12 +01:00
greatroar
15ea90feed Update pkg/xattr to handle EINTR on Linux
Updates #2659. This is a case where the stdlib will not handle EINTR for
us, even with Go 1.16. That xattr calls are directly affected can be
seen in the report for issue #2968.
2020-11-08 09:34:24 +01:00
Alexander Weiss
826cfa0533 fix context in archiver tests 2020-11-08 08:24:24 +01:00
Alexander Weiss
fef408a8bd Return context error in mem backend 2020-11-08 00:05:53 +01:00
greatroar
a2d4209322 Don't recurse in local backend's List if not required
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.

Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9).
2020-11-07 08:54:13 +01:00
Alexander Neumann
275f713211 Remove Travis and AppVeyor 2020-11-06 21:49:52 +01:00
MichaelEischer
4707bdb204 Merge pull request #2842 from aawsome/rebuild-index-inmem
Rebuild index in prune by using in-memory index
2020-11-06 20:51:20 +01:00
Alexander Neumann
47277c4b4c Add comments, clarify computation 2020-11-06 20:23:30 +01:00
Alexander Weiss
d2e53730d6 Add test that repo.List is only called once 2020-11-06 20:23:30 +01:00
Alexander Weiss
fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Weiss
38cc4393f6 Add Masterindex.Save(); Add Index.Packs() 2020-11-06 20:23:30 +01:00
Alexander Neumann
7f6f31c34b Remove Hound 2020-11-06 11:43:58 +01:00
Alexander Neumann
164b4cb2f6 Add changelog for #3014 2020-11-06 10:05:42 +01:00
Alexander Neumann
4a9b05aff1 Merge pull request #3063 from aawsome/fix-3062
Fix #3062
2020-11-05 19:36:40 +01:00
Alexander Weiss
aaf1c44362 Fix #3062 2020-11-05 17:05:42 +01:00
Alexander Neumann
a5592e83f7 Merge pull request #3014 from ivandeex/pr-stream-reset
Fix sporadic stream reset between rclone and restic
2020-11-05 16:11:50 +01:00
Ivan Andreev
ab2790d9de Fix http2 stream reset between restic and rest backends #3014 2020-11-05 15:57:40 +03:00
Alexander Neumann
8a2a326189 Merge pull request #2535 from ncw/fix-2528
s3: add bucket-lookup parameter to select path or dns style bucket lookup
2020-11-05 12:39:34 +01:00
Nick Craig-Wood
86b5d8ffaa s3: add bucket-lookup parameter to select path or dns style bucket lookup
This is to enable restic working with Alibaba cloud

Fixes #2528
2020-11-05 12:20:10 +01:00
Alexander Neumann
636b2f2e94 Merge pull request #2941 from MichaelEischer/parallel-repack
prune: Parallelize repack step
2020-11-05 11:00:41 +01:00
Alexander Neumann
ae5302c7a8 Add comment that keepBlobs is modified 2020-11-05 10:33:38 +01:00
Alexander Neumann
866a52ad4e Remove unneeded seek
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
Alexander Neumann
a4507610a0 Fix typo 2020-11-05 10:31:49 +01:00
Alexander Neumann
7def2d8ea7 Use a non-constant seed 2020-11-05 10:31:49 +01:00
Alexander Neumann
ee0112ab3b Clarify message about expected error 2020-11-05 10:31:49 +01:00
Michael Eischer
b373f164fe prune: Parallelize repack command 2020-11-05 10:31:49 +01:00
Alexander Neumann
8a0dbe7c1a helpers: Remove old changelog files for release 2020-11-05 10:26:00 +01:00
Alexander Neumann
4e3ad8b3b1 Remove changelog files from 0.11.0 release 2020-11-05 10:22:39 +01:00
Alexander Neumann
5144141321 Merge pull request #2718 from aawsome/new-cleanup-command
Reimplementation of prune
2020-11-05 10:12:19 +01:00
Alexander Neumann
d35d279455 Set development version for 0.11.0 2020-11-05 09:41:40 +01:00
Alexander Neumann
b7e1ece1e0 Add version for 0.11.0 2020-11-05 09:41:35 +01:00
Alexander Neumann
c5300a2c56 Update manpages and auto-completion 2020-11-05 09:41:35 +01:00
Alexander Neumann
1bd92896d7 Generate CHANGELOG.md for 0.11.0 2020-11-05 09:41:22 +01:00
Alexander Neumann
9dba01021e Prepare changelog for 0.11.0 2020-11-05 09:41:22 +01:00
Alexander Neumann
23f6b8c3fd Add .vscode to gitignore 2020-11-05 09:40:56 +01:00
Alexander Neumann
79a50e3b1f helpers: Create changelog subdir for release 2020-11-05 09:39:06 +01:00
Alexander Neumann
b8a5ca2d10 Add gitignore to changelog/unreleased 2020-11-05 09:01:32 +01:00
fgma
916b2d303b vss: fix potential crash (not reachable in restic) (#3045)
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
Alexander Neumann
a06f5c28c0 Merge pull request #3055 from greatroar/cleanup
Don't separately allocate sync.{Mutex,Once} if not necessary
2020-11-04 11:11:21 +01:00
greatroar
c2f3eee5af Don't separately allocate sync.{Mutex,Once} if not necessary
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
rawtaz
62345abe4a Merge pull request #3053 from maikell/patch-1
Update README.md
2020-11-03 19:06:59 +01:00
Maikel van Leeuwen
e024fc6d4d Update README.md
Typo in link. 
Links to  https://forum.restic.net%3E/ should be https://forum.restic.net
2020-11-03 18:51:11 +01:00
Alexander Neumann
1ca60bccfb Refactor condition for MaxRepackBytes
Don't depend on the string (opts.MaxRepackSize) for the condition,
instead check if there's a (positive) limit configured.
2020-11-03 16:42:21 +01:00
Alexander Neumann
7f86eb4ec0 Move helper function 2020-11-03 16:42:21 +01:00
Alexander Neumann
c1a3de4a6e Refactor max-unused calculation, add unlimited option
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.

Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.

Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.

Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.

I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:

      x ≤ p * (u+x)
    ⇔ x ≤ p*u + p*x
    ⇔ x - p*x ≤ p*u
    ⇔ x * (1-p) ≤ p*u
    ⇔ x ≤ p/(1-p) * u
2020-11-03 16:42:21 +01:00
Alexander Neumann
f8c4dd7b1a Split packe rewrite logic into two case branches
The comma is too sublte, let's split this into two separate branches.
2020-11-03 16:42:21 +01:00
Alexander Neumann
a5b80452fe Add comment that usedBlobs is modified 2020-11-03 16:42:21 +01:00
Alexander Neumann
aff1e220f5 Split struct members, add comments 2020-11-03 16:42:21 +01:00
Alexander Neumann
095155d9ce Remove RepackSmall 2020-11-03 16:42:21 +01:00
Alexander Neumann
1dd9fdce74 Reword changelog slightly 2020-11-03 16:42:21 +01:00
Alexander Weiss
b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss
7f9a0a5907 Reimplementation of prune 2020-11-03 16:42:21 +01:00
Alexander Weiss
3b591ed987 Add Verboseff 2020-11-03 16:42:21 +01:00
Alexander Weiss
ce7d613749 Add Blob.Handle() 2020-11-03 16:42:21 +01:00
Alexander Weiss
581d90cf91 Make some pack parameters public 2020-11-03 16:42:21 +01:00
Alexander Neumann
0db9024aad Merge pull request #3052 from restic/error-on-invalid-size
Return an error for invalid sizes
2020-11-03 12:13:49 +01:00
Alexander Neumann
21ba15577e Return an error for invalid sizes 2020-11-03 11:53:49 +01:00
Alexander Neumann
2fb1957ca4 Rewrite README as Markdown 2020-11-03 09:20:09 +01:00
Alexander Neumann
9a88fb253b Merge pull request #3051 from greatroar/sanitize-env
Sanitize environment before starting backend processes (rclone, ssh)
2020-11-02 21:18:57 +01:00
Alexander Neumann
f14436953a Merge pull request #3050 from greatroar/widechars
Fix string truncation in ui/termstatus
2020-11-02 16:58:51 +01:00
greatroar
11fbaaae9a Sanitize environment before starting backend processes (rclone, ssh)
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.

Fixes #2192.
2020-11-02 16:41:23 +01:00
greatroar
3ed84ff0c6 Fix string truncation in ui/termstatus
Fixes #3046.
2020-11-02 12:50:49 +01:00
Alexander Neumann
8e965ed4eb Merge pull request #3039 from MichaelEischer/tar-proper-dirs
Properly dump directories to tar
2020-11-02 12:30:26 +01:00
Alexander Neumann
5f0fa2129e Improve readability
It's time to use a switch statement.
2020-11-02 11:24:26 +01:00
Alexander Neumann
04dfa19c7e Improve changelog 2020-11-02 11:23:09 +01:00
Alexander Neumann
6509c207f4 Merge pull request #2997 from MichaelEischer/faster-excludes
Speedup exclude/include checking
2020-11-02 11:07:21 +01:00
Alexander Neumann
445b845267 Merge pull request #2978 from MichaelEischer/warn-tree-error
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
Alexander Neumann
3ff37215df Merge pull request #2935 from MichaelEischer/upgrade-minio
Upgrade minio SDK to version 7
2020-11-02 09:09:10 +01:00
MichaelEischer
5d379b5359 Merge pull request #3042 from rubiojr/remove-unused-function
Remove unused function
2020-11-01 21:55:22 +01:00
Sergio Rubio
e708628cfd Remove unused function
Not currently used, and it'd need to be added to the MasterIndex interface first.
2020-10-28 13:24:49 +01:00
MichaelEischer
bb4b3481a6 Merge pull request #3031 from greatroar/debug-no-seek
Allow debug logging to pipes and terminals
2020-10-25 17:59:06 +01:00
MichaelEischer
ad3a52e6f0 Merge pull request #3026 from greatroar/refactor-ui
internal/ui refactoring
2020-10-25 17:51:39 +01:00
rawtaz
e8b4d8d8bc Merge pull request #2998 from greatroar/env-docs
Systematize documentation of environment variables
2020-10-24 22:49:31 +02:00
Michael Eischer
1aa61e6def Add changelog 2020-10-24 22:49:29 +02:00
Michael Eischer
8d7d6ad2d5 dump: include username in tar 2020-10-24 22:42:01 +02:00
Michael Eischer
fe09e6f865 dump: test proper permissions and directory name 2020-10-24 22:42:01 +02:00
Michael Eischer
1e3c9a2c11 dump: Fix file permission to tar mapping
The file permissions included a go specific directory bit which
accidentially forced the usage of the GNU header format. This leads
to problems with 7zip on Windows or when extended attributes are
used.
2020-10-24 22:42:01 +02:00
Michael Eischer
e21dcb0eea dump: Additional ACL tests 2020-10-24 22:41:01 +02:00
MichaelEischer
31b8d7a639 Merge pull request #2274 from fgma/master
Support for Volume Shadow Copy Service (VSS) on windows
2020-10-24 15:43:34 +02:00
fgma
5695f9ebd2 vss: Implement VSS support for Windows
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.

Snapshotting volumes also covers mountpoints but skips UNC paths.
2020-10-24 11:35:57 +02:00
greatroar
8091151638 doc: Update restic mount availability and requirements
NetBSD doesn't support restic mount either, so it's easier to list the
positive cases. Also noted that FUSE for macOS is required on the Mac.
2020-10-22 09:57:30 +02:00
greatroar
ae179ee63e Systematize documentation of environment variables
Cache locations were documented inconsistently in three places.

The backup docs mentioned PATH being used to find fusermount, which is
never run by restic backup. It now mentions ssh and rclone, which are
used by backends.

The notion of a "system-wide" environment variable makes no sense.
TMPDIR is now mentioned because it allows for optimization and may
have security implications.
2020-10-22 09:57:30 +02:00
MichaelEischer
0590e3e12d Merge pull request #3035 from greatroar/no-unmount-on-failure
restic mount: don't umount unless we actually tried to mount
2020-10-21 22:10:42 +02:00
greatroar
3807d13bdc restic mount: don't umount unless we actually tried to mount
Previously, failure to open a repository would result in an "unable to
umount" message.
2020-10-21 12:15:05 +02:00
greatroar
63be3704d9 Allow debug logging to pipes and terminals 2020-10-19 22:30:30 +02:00
greatroar
35419de232 Simplify ui.StdioWrapper.Write
Instead of looping to find line breaks, make it look for the last one.
2020-10-17 20:30:46 +02:00
greatroar
863a590a81 Refactor termstatus.Term.{Print,Error} methods 2020-10-17 20:30:46 +02:00
greatroar
7c0b6a82db Remove unused public method ui.linesWriter.Flush 2020-10-17 20:30:24 +02:00
MichaelEischer
96a912b65a Merge pull request #3025 from aawsome/fix-index-ids
Fix setting of ID in DecodeIndex
2020-10-17 09:54:45 +02:00
Alexander Weiss
b44ecde8b0 Fix setting of ID in DecodeIndex 2020-10-17 09:12:58 +02:00
MichaelEischer
39fe1e96fe Merge pull request #3018 from greatroar/background-check
Fix IsProcessBackground on Linux with stdin redirection
2020-10-16 22:50:19 +02:00
MichaelEischer
4ba237bb93 Merge pull request #3019 from greatroar/refactor-decodeindex
Refactor index decoding
2020-10-15 23:22:33 +02:00
MichaelEischer
ce87fbd7dc Merge pull request #3022 from greatroar/refactor-runworkers
Defer channel closing outside repository.RunWorkers
2020-10-14 22:44:42 +02:00
greatroar
b27375f5ce defer close(ch) outside repository.RunWorkers 2020-10-14 15:50:16 +02:00
greatroar
720e0ee0c7 if cond { return true }; return false => return cond 2020-10-13 20:56:43 +02:00
greatroar
27db3ec262 Refactor index decoding
Decoding old-format indices no longer requires loading and decrypting
twice.
2020-10-13 20:47:50 +02:00
greatroar
f80b07b2c8 Fix IsProcessBackground on Linux with stdin redirection
The previous implementation assumed that stdin was a terminal.
It now checks the terminal's fd.
2020-10-13 13:12:02 +02:00
MichaelEischer
6003dada14 Merge pull request #3011 from restic/doc-alpine
doc: Add restic install instructions for Alpine Linux
2020-10-12 23:29:40 +02:00
MichaelEischer
41a45ae908 Merge pull request #3016 from greatroar/uid-gid-lookup
Simplify/optimize cached UID/GID lookups
2020-10-12 23:24:55 +02:00
MichaelEischer
8299c5559c Merge pull request #3015 from greatroar/changelog-template
Improved changelog template
2020-10-12 23:12:29 +02:00
Alexander Neumann
56883817d8 Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
greatroar
f0cd16e5ea Cache uint32-typed ids in lookup{Username,Group}
NodeFillUser-8        1.92µs ± 5%    1.75µs ± 3%   -8.89%  (p=0.000 n=10+10)
NodeFromFileInfo-8    1.89µs ± 7%    1.76µs ± 4%   -6.69%  (p=0.001 n=10+10)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)
NodeFromFileInfo-8      504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
NodeFromFileInfo-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
2020-10-12 15:08:15 +02:00
greatroar
a03fe4562e Remove unused error return in lookup{Username,Group} 2020-10-12 14:48:23 +02:00
greatroar
efc075df87 Improved changelog template 2020-10-12 12:15:15 +02:00
Leo R. Lundgren
f500b0d90e doc: Add restic install instructions for Alpine Linux 2020-10-11 18:31:07 +02:00
Alexander Neumann
c193cea119 Pass Context to NewRestorer() 2020-10-10 15:24:26 +02:00
Alexander Neumann
f0d49ca600 Merge pull request #2933 from MichaelEischer/less-context-todo
Replace most usages of context.TODO()
2020-10-10 15:03:44 +02:00
MichaelEischer
164d8af3dd Merge pull request #2906 from kitone/fix-inconsistent-timestamps-permissions
Restore inconsistent timestamps permissions
2020-10-10 14:56:43 +02:00
kitone
052564007a Add changelog 2020-10-10 14:37:51 +02:00
kitone
6099f81692 Fix #1212 restore code produces inconsistent timestamps/permissions.
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
2020-10-10 13:46:44 +02:00
kitone
9d7f616190 Improve restorer debug log information 2020-10-10 13:46:43 +02:00
kitone
295ddb9e57 Add test case for inconsistent timestamps and permissions restoration
Reproduce from https://github.com/restic/restic/issues/1212
2020-10-10 13:46:43 +02:00
Michael Eischer
e638b46a13 Embed context into ReaderAt
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
Michael Eischer
d6cfe857b7 pass proper context into MasterIndex.RebuildIndex 2020-10-09 22:39:07 +02:00
Michael Eischer
2964d2ad15 Skip unlocking for nil locks
Now that lockRepo receives a context, it is possible that it is canceled
before a lock was created. Thus `unlockRepo` must be able to handle this
case.
2020-10-09 22:39:06 +02:00
Michael Eischer
0c9efa9c2a Pass context to lockRepo 2020-10-09 22:39:06 +02:00
Michael Eischer
37a5e2d681 rest: use global context on repository creation 2020-10-09 22:39:06 +02:00
Michael Eischer
4b0fcaed45 unlock: use proper context for locks cleanup
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
2020-10-09 22:37:56 +02:00
Michael Eischer
645a6efaf2 restorer: remove redundant type specification 2020-10-09 22:37:56 +02:00
Michael Eischer
dc2e664209 integration_fuse_test: use global context
No need to use the TODO context.
2020-10-09 22:37:56 +02:00
Michael Eischer
a449450021 init: pass proper context to master key generation
This is no change in behavior as a canceled context did later on cause
the config file creation to fail. Therefore this change just lets the
repository initialization fail a bit earlier.
2020-10-09 22:37:56 +02:00
Michael Eischer
603bb0e309 restore: Use proper context while loading snapshot 2020-10-09 22:37:56 +02:00
Michael Eischer
27456f6545 debug: use proper context
This allows the debug commands to be properly interrupted.
2020-10-09 22:37:56 +02:00
Michael Eischer
c458e114d4 pass context to Find / FindSnapshot
This allows proper interruption of restic while it searches for
snapshots or key files.
2020-10-09 22:37:56 +02:00
Michael Eischer
45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer
307a6ba3a3 Upgrade minio sdk to v7
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
2020-10-09 22:37:24 +02:00
Michael Eischer
50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
Alexander Neumann
5fd3dbccb7 Merge pull request #2963 from MichaelEischer/fix-status-deadlock
backup: Fix possible deadlock of scanner goroutine
2020-10-09 21:35:50 +02:00
Alexander Neumann
35655a481b Merge pull request #2931 from MichaelEischer/reduce-debug-overhead
Reduce debug log overhead
2020-10-09 21:35:11 +02:00
Alexander Neumann
30cb553c8d Merge pull request #2932 from MichaelEischer/proper-rclone-create
Call rclone.Create to create a new repository for the rclone backend
2020-10-09 21:29:15 +02:00
Alexander Neumann
2f3eeff2e7 Merge pull request #2964 from MichaelEischer/fix-repro-instructions
Fix instructions to reproduce the release binaries
2020-10-09 21:28:24 +02:00
MichaelEischer
356ac404cd Merge pull request #2996 from greatroar/setconsolecursorposition
Use windows.SetConsoleCursorPosition in ui/termstatus
2020-10-09 18:19:02 +02:00
rawtaz
ab769abeaf Merge pull request #3002 from MichaelEischer/update-contributing
Update links to help wanted labels in CONTRIBUTING.md and link to forum
2020-10-09 17:24:25 +02:00
Michael Eischer
4036991c91 Update links to help wanted labels in CONTRIBUTING.md and link to forum 2020-10-09 16:15:43 +02:00
Michael Eischer
88c8e903d2 filter: Fix glob matching on absolute path marker on windows
A pattern part containing "/" is used to mark a path or a pattern as
absolute. However, on Windows the path separator is "\" such that glob
patterns like "?" could match the marker. The code now explicitly skips
the marker when the pattern does not represent an absolute path.
2020-10-09 16:11:05 +02:00
greatroar
740758a5fa Optimize filter pattern matching
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.

name                          old time/op    new time/op    delta
FilterLines-8                   44.7ms ± 5%    44.9ms ± 5%     ~     (p=0.631 n=10+10)
FilterPatterns/Relative-8       13.6ms ± 4%    13.4ms ± 5%     ~     (p=0.165 n=10+10)
FilterPatterns/Absolute-8       10.9ms ± 5%    10.7ms ± 4%     ~     (p=0.052 n=10+10)
FilterPatterns/Wildcard-8       53.7ms ± 5%    50.4ms ± 5%   -6.00%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8     128ms ± 2%      95ms ± 1%  -25.54%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=1.000 n=9+8)
FilterPatterns/Absolute-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.903 n=9+8)
FilterPatterns/Wildcard-8       19.7MB ± 0%    19.7MB ± 0%   -0.00%  (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-8        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-8     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-09 15:51:14 +02:00
MichaelEischer
862ee4b2c9 Merge pull request #2970 from labkode/fopenskip
Skip fopen for file change check to avoid network penalty
2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador
958dc6aafc Skip fopen for file change check to avoid network penalty 2020-10-09 10:53:28 +02:00
MichaelEischer
88cc444779 Merge pull request #2934 from MichaelEischer/upgrade-backoff
Upgrade github.com/cenkalti/backoff module
2020-10-08 20:33:30 +02:00
MichaelEischer
4a424af1d5 Merge pull request #2989 from MichaelEischer/remove-local-chmod
local: mark repository files as read-only and handle chmod errors
2020-10-08 19:04:13 +02:00
Michael Eischer
bc65da2baf Add changelog 2020-10-08 14:45:26 +02:00
Michael Eischer
b79f18209f Upgrade github.com/cenkalti/backoff module
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
2020-10-07 22:04:59 +02:00
Michael Eischer
8388e67c4b filter: cleanup path separator conversion 2020-10-07 21:14:07 +02:00
Michael Eischer
0acc3c5923 filter: special case patterns without globbing characters
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.

name                          old time/op    new time/op    delta
FilterLines-4                   34.8ms ±17%    41.2ms ±23%  +18.36%  (p=0.000 n=10+10)
FilterPatterns/Relative-4       21.7ms ± 6%    12.1ms ±23%  -44.46%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       10.0ms ± 5%     9.1ms ±11%   -9.80%  (p=0.006 n=10+9)
FilterPatterns/Wildcard-4       47.0ms ± 7%    42.2ms ± 5%  -10.19%  (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4     190ms ± 1%     131ms ±20%  -31.47%  (p=0.000 n=8+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.870 n=9+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.145 n=10+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    19.7MB ± 0%  +37.91%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.421 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:55:43 +02:00
Michael Eischer
54a124de3b filter: explicitly test separate ListWithChild function 2020-10-07 20:47:52 +02:00
Michael Eischer
bcc3bddcf4 filter: only check whether a child path could match when necessary
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.3ms ± 9%    21.7ms ± 6%   -6.68%  (p=0.004 n=10+10)
FilterPatterns/Absolute-4       13.9ms ± 7%    10.0ms ± 5%  -27.61%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4       51.4ms ± 7%    47.0ms ± 7%   -8.51%  (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4     551ms ± 9%     190ms ± 1%  -65.41%  (p=0.000 n=10+8)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.665 n=10+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.480 n=9+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    14.3MB ± 0%     ~     (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:52 +02:00
Michael Eischer
17c53efb0d filter: Optimize double wildcard expansion
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       22.7ms ± 5%    23.3ms ± 9%     ~     (p=0.353 n=10+10)
FilterPatterns/Absolute-4       14.2ms ±13%    13.9ms ± 7%     ~     (p=0.853 n=10+10)
FilterPatterns/Wildcard-4        266ms ±16%      51ms ± 7%  -80.67%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4     554ms ± 6%     551ms ± 9%     ~     (p=0.436 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.349 n=10+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.073 n=10+9)
FilterPatterns/Wildcard-4        141MB ± 0%      14MB ± 0%  -89.89%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     0.09M ± 0%  -94.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:48 +02:00
Michael Eischer
7959796269 filter: Special case for absolute paths
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.6ms ±20%    22.7ms ± 5%     ~     (p=0.684 n=10+10)
FilterPatterns/Absolute-4       32.3ms ± 8%    14.2ms ±13%  -56.01%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        334ms ±17%     266ms ±16%  -20.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     709ms ± 7%     554ms ± 6%  -21.89%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%   +0.00%  (p=0.046 n=9+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.464 n=10+10)
FilterPatterns/Wildcard-4        141MB ± 0%     141MB ± 0%     ~     (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     1.63M ± 0%     ~     (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:29 +02:00
Michael Eischer
375c2a56de filter: Parse filter patterns only once
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       30.3ms ±10%    23.6ms ±20%  -22.12%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       49.0ms ± 3%    32.3ms ± 8%  -33.94%  (p=0.000 n=8+10)
FilterPatterns/Wildcard-4        345ms ± 9%     334ms ±17%     ~     (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4     3.93s ± 2%     0.71s ± 7%  -81.98%  (p=0.000 n=9+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       4.63MB ± 0%    3.57MB ± 0%  -22.98%  (p=0.000 n=9+9)
FilterPatterns/Absolute-4       8.54MB ± 0%    3.57MB ± 0%  -58.20%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        146MB ± 0%     141MB ± 0%   -2.93%  (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4     907MB ± 0%       4MB ± 0%  -99.61%  (p=0.000 n=9+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        66.6k ± 0%     22.2k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4        88.7k ± 0%     22.2k ± 0%  -75.00%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.70M ± 0%     1.63M ± 0%   -3.92%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     4.46M ± 0%     0.02M ± 0%  -99.50%  (p=0.000 n=10+10)
2020-10-07 20:47:27 +02:00
Michael Eischer
b8eacd1364 filter: Reduce redundant path and pattern splitting
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       62.1ms ±15%    30.3ms ±10%  -51.22%  (p=0.000 n=9+10)
FilterPatterns/Absolute-4        111ms ±10%      49ms ± 3%  -56.08%  (p=0.000 n=10+8)
FilterPatterns/Wildcard-4        393ms ±15%     345ms ± 9%  -12.30%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     10.0s ± 3%      3.9s ± 2%  -60.53%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       16.4MB ± 0%     4.6MB ± 0%  -71.76%  (p=0.000 n=10+9)
FilterPatterns/Absolute-4       31.4MB ± 0%     8.5MB ± 0%  -72.77%  (p=0.000 n=9+10)
FilterPatterns/Wildcard-4        168MB ± 0%     146MB ± 0%  -13.19%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.23GB ± 0%    0.91GB ± 0%  -71.96%  (p=0.000 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4         178k ± 0%       67k ± 0%  -62.50%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4         266k ± 0%       89k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.87M ± 0%     1.70M ± 0%   -9.47%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     17.7M ± 0%      4.5M ± 0%  -74.87%  (p=0.000 n=9+10)
2020-10-07 18:13:19 +02:00
Michael Eischer
e73c281142 filter: Benchmark absolute paths, wildcards and long filter lists 2020-10-07 17:54:36 +02:00
Michael Eischer
fdd3b14db3 filter: test some corner cases 2020-10-07 17:09:44 +02:00
Michael Eischer
f4282aa6fd local: mark repository files as read-only
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258.
2020-10-07 12:29:37 +02:00
Michael Eischer
40ee17167e local: Ignore permission errors on chmod call in Save/Remove operation
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
2020-10-07 12:29:37 +02:00
greatroar
f3e933f0c1 Use windows.SetConsoleCursorPosition in ui/termstatus 2020-10-07 11:18:54 +02:00
MichaelEischer
0ae02f3030 Merge pull request #2844 from aawsome/prune-integration-tests
Add prune integration tests for many edge cases
2020-10-06 23:16:41 +02:00
rawtaz
e401859afb Merge pull request #2992 from lbausch/repo_wording
Replace repo/repos with repository/repositories in `--no-lock` help text
2020-10-06 23:03:55 +02:00
Lorenz Bausch
af4100e07d Replace repo/repos with repository/repositories in --no-lock help text 2020-10-06 22:49:48 +02:00
Alexander Neumann
ea5bbe0857 Remove dysfunctional link to saythanks 2020-10-06 20:43:44 +02:00
Alexander Weiss
6822a58413 Add prune integration tests for many edge cases 2020-10-06 20:20:05 +02:00
MichaelEischer
09d39e260d Merge pull request #2980 from greatroar/error-wrapping
Remove repetitive error wrapping from internal/cache
2020-10-06 15:19:00 +02:00
Michael Eischer
1579d2a8ec Remove some unused assignments 2020-10-06 14:55:13 +02:00
Michael Eischer
eba5dd831f Fix typos reported by misspell 2020-10-06 14:55:13 +02:00
Michael Eischer
9ffb698c8d Modernize context import in internal/fuse
Migrated using `go fix`.
2020-10-06 14:55:13 +02:00
Michael Eischer
efbb850d92 Remove a few redundant type specifiers
This is the result of running `gofmt -s -w **/*.go`
2020-10-06 14:55:13 +02:00
MichaelEischer
cfd57c480a Merge pull request #2991 from restic/nonsense
cache/check: Clarify error message when given arguments
2020-10-06 00:40:12 +02:00
Leo R. Lundgren
e105a3f391 cache/check: Clarify error message when given arguments 2020-10-06 00:08:59 +02:00
rawtaz
5d8cfff3f2 Merge pull request #2988 from greatroar/snapshots-json
Regression test for #2979
2020-10-05 21:02:52 +02:00
greatroar
673dda77c0 Less repetitive error messages in internal/cache
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
2020-10-05 20:28:54 +02:00
MichaelEischer
1ab4c710e1 Merge pull request #2987 from greatroar/no-password-for-help
Don't read password for generate, help or self-update
2020-10-05 20:16:19 +02:00
greatroar
feedf0ebce Don't read password for generate, help or self-update
Fixes #2951.
2020-10-05 19:54:07 +02:00
MichaelEischer
1a490acd67 Merge pull request #2982 from greatroar/archiver-error-handling
Check error in archiver before calling Select
2020-10-05 19:19:47 +02:00
Michael Eischer
187518a8a3 docs: Properly reproduce zip archives for windows binaries 2020-10-05 18:24:46 +02:00
Michael Eischer
a232c833dc docs: Update binary reproduction instructions and refer to older instructions 2020-10-05 18:24:46 +02:00
MichaelEischer
c84643c6a9 Merge pull request #2981 from greatroar/mkdircachedir
Simplify cache directory creation
2020-10-05 18:15:23 +02:00
MichaelEischer
8390a8aaf3 Merge pull request #2986 from greatroar/upgrade-xxhash
Upgrade cespare/xxhash to 2.1.1
2020-10-05 17:49:08 +02:00
greatroar
1e1a1f3078 Upgrade cespare/xxhash to 2.1.1 2020-10-05 17:25:39 +02:00
MichaelEischer
abe9fa261f Merge pull request #2983 from greatroar/upgrade-siphash
Upgrade siphash to 1.2.2
2020-10-05 17:18:07 +02:00
MichaelEischer
6001b45bf7 Merge pull request #2984 from restic/snapshots-json
snapshots: Make --json output [] instead of null when no snapshots
2020-10-05 17:14:23 +02:00
greatroar
6bee0aafc2 Regression test for #2979 2020-10-05 15:08:09 +02:00
greatroar
8252ea8e3d Upgrade siphash to 1.2.2
New version is potentially faster on 32-bit ARM.
2020-10-05 14:56:13 +02:00
Leo R. Lundgren
137d20a06a snapshots: Make --json output [] instead of null when no snapshots 2020-10-05 13:14:10 +02:00
greatroar
c4e2203e45 Check error in archiver before calling Select
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
greatroar
7d0fa1a686 Simplify cache directory creation 2020-10-05 10:46:45 +02:00
rawtaz
a23d90d270 Merge pull request #2975 from restic/rawtaz-doc-resume
Add FAQ on whether restic can resume backups
2020-10-03 23:12:34 +02:00
MichaelEischer
2630411530 Merge pull request #2974 from gottwald/swap-gcs-lib
Swap deprecated GCS lib with replacement
2020-10-03 21:58:11 +02:00
rawtaz
b9b82d878d Add FAQ on whether restic can resume backups
This new FAQ entry explains that restic can resume interrupted backups.
2020-10-03 21:39:45 +02:00
Ingo Gottwald
8b8e230771 Swap deprecated GCS lib with replacement 2020-10-03 18:55:56 +02:00
rawtaz
7c3c6fa431 Merge pull request #2977 from MichaelEischer/downgrade-cobra
Downgrade cobra to v0.0.5
2020-10-03 18:23:13 +02:00
Michael Eischer
29908906b7 Downgrade cobra to v0.0.5 2020-10-03 18:13:07 +02:00
rawtaz
bbeb439f41 Merge pull request #2937 from andreaso/self-update-output-path-fix
Don't require `self-update --output` placeholder file
2020-10-03 13:35:20 +02:00
Andreas Olsson
ce14df303b Don't require self-update --output placeholder file
This removes the requirement on `restic self-update --output` to point
to a path of an existing file, to overwrite. In case the specified
path does exist we still want to verify that it's a regular file,
rather than a directory or a device, which gets overwritten.

We also want to verify that a path to a new file exists within an
existing directory. The alternative being running into that issue
after the actual download, etc has completed.

While at it I also replace `errors.Errorf` with the more appropriately
verbose `errors.Fatalf`.

Resolves #2491
2020-10-03 07:02:23 +02:00
MichaelEischer
3c6671b18b Merge pull request #2973 from restic/rawtaz-repo2
Correct data type for --repo2 option
2020-10-02 22:22:41 +02:00
MichaelEischer
86ff1f2bf6 Merge pull request #2972 from restic/rawtaz-verbose
Clarify max verbose level in option description
2020-10-02 22:21:45 +02:00
rawtaz
0cce6dc31c Correct data type for --repo2 option 2020-10-02 21:10:52 +02:00
rawtaz
6253ff0187 Clarify max verbose level in option description
Clarifies that the max verbosity level is 3, in the description of the --verbose option.
2020-10-02 20:25:34 +02:00
rawtaz
e9943e864f Merge pull request #2955 from jtagcat/clarify--compact
help: --compact: clarify description
2020-10-02 15:59:08 +02:00
jtagcat
a687261804 --help: --compact: clarify description 2020-10-02 16:55:56 +03:00
MichaelEischer
4df8861e09 Merge pull request #2967 from gottwald/fix-deprctd-gs-constructor
Replace deprecated method in gs backend
2020-10-01 11:01:49 +02:00
Ingo Gottwald
00cedd22aa Replace deprecated method in gs backend 2020-10-01 10:02:42 +02:00
MichaelEischer
f361ed66de Merge pull request #2910 from mtdcr/repository-file
Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
2020-10-01 01:04:23 +02:00
Michael Eischer
7b50a65492 Update backup help output in documentation 2020-10-01 00:50:27 +02:00
Michael Eischer
c18b119a9b Document new option --repository-file 2020-10-01 00:50:27 +02:00
Michael Eischer
61035d68bc Add test for --repository-file 2020-10-01 00:50:27 +02:00
Andreas Oberritter
97f7855de3 Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
As an alternative to -r, this allows to read the repository URL
from a file in order to prevent certain types of information leaks,
especially for URLs containing credentials.

Fixes #1458, fixes #2900.
2020-10-01 00:50:26 +02:00
Michael Eischer
d44df9d00d backup: Always remove the status lines once a backup ends 2020-09-30 23:13:10 +02:00
Michael Eischer
8d0ba55ecd backup: Fix possible deadlock of scanner goroutine
When the backup is interrupted for some reason while the scanner is
still active this could lead to a deadlock. Interruptions are triggered
by canceling the context object used by both the backup progress UI and
the scanner. It is possible that a context is canceled between the
respective check in the scanner and it calling the `ReportTotal` method
of the UI. The latter method sends a message to the UI goroutine.
However, a canceled context will also stop that goroutine, which can
cause the channel send operation to block indefinitely.

This is resolved by adding a `closed` channel which is closed once the
UI goroutine is stopped and serves as an escape hatch for reported UI
updates.

This change covers not just the ReportTotal method but all potentially
affected methods of the progress UI implementation.
2020-09-30 23:13:10 +02:00
rawtaz
34ea960559 Merge pull request #2966 from MichaelEischer/recover-help-typo
recover: Fix typo in the command help
2020-09-30 18:16:26 +02:00
rawtaz
5ea01b00df Merge pull request #2965 from MichaelEischer/rework-password-prompt
Clarify verbose password prompt
2020-09-30 18:15:05 +02:00
MichaelEischer
48d0ab5276 Merge pull request #2959 from restic/keep-text
backup: Correct keep policy text
2020-09-30 17:57:00 +02:00
Michael Eischer
2a79c1a44d recover: Fix typo in the command help
Reported-by: Steve Divskinsy <stevesbrain@users.noreply.github.com>
2020-09-30 17:44:34 +02:00
MichaelEischer
fd02407863 Merge pull request #2849 from classmarkets/gcs-access-token
gs: support authentication with access token
2020-09-30 17:42:56 +02:00
Michael Eischer
aea9f7d286 clarify verbose password prompt 2020-09-30 17:25:54 +02:00
Leo R. Lundgren
028c9a5343 backup: Correct keep policy text
Makes the following corrections to the "Applying Policy:" output:

- keep the last 1 snapshots snapshots => keep 1 latest snapshots
- keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots
2020-09-28 14:26:53 +02:00
MichaelEischer
94136132e3 Merge pull request #2957 from plumbeo/patch-3
Don’t print excessively detailed debug messages on object deletion when —verbose is used
2020-09-27 22:08:07 +02:00
plumbeo
009bd907f2 Don’t print too many messages when deleting files
Print detailed debug messages on file deletions only when --verbose=2 is used
2020-09-27 14:24:04 +02:00
MichaelEischer
14b312f00d Merge pull request #2658 from creativeprojects/issue-2241
Don't echo authentication passwords (rest backend)
2020-09-22 22:17:53 +02:00
Fred
206cadfab4 Hide password from repository URLs 2020-09-22 22:00:51 +02:00
MichaelEischer
4875f7b659 Merge pull request #2614 from greatroar/simplify-fs
Simplify internal/fs
2020-09-21 22:05:10 +02:00
MichaelEischer
6f2093e491 Merge pull request #2940 from andreaso/appveyor-tar-https
Use https:// to download appveyor tar
2020-09-21 21:50:26 +02:00
MichaelEischer
16f31b2f73 Merge pull request #2939 from J0WI/patch-1
Update Go version to 1.15 in Docker build script
2020-09-21 21:46:33 +02:00
greatroar
0d65b78168 Simplify os.ModeType|os.ModeCharDevice => os.ModeType
Since Go 1.12, ModeCharDevice is included in ModeType:
golang/go@a2a3dd00c9
2020-09-21 14:21:32 +02:00
greatroar
95ebba85ff Remove stray Printf from internal/fs 2020-09-21 14:21:32 +02:00
greatroar
59b343a9bf Remove OS-specific versions of fs.MkdirAll
Go has supported Windows paths correctly since 1.11, see
https://github.com/golang/go/issues/10900 and the commit referenced
there.
2020-09-21 14:21:32 +02:00
greatroar
1557c58eef Fix and simplify fs.Reader
fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0
correctly. fakeFileInfo.sys is always nil, so omit the field.
2020-09-21 14:21:32 +02:00
greatroar
c504aa505c Remove unused fs.Rename 2020-09-21 14:21:32 +02:00
greatroar
1b20f6beec Remove io.Writer from fs.File
It was only used in a single test, which now uses plain *os.File instead.
2020-09-21 14:21:32 +02:00
rawtaz
10e3340863 Merge pull request #2945 from YoshieraHuang/max-file-size
Fix nil check in rejectBySize
2020-09-21 14:18:45 +02:00
yoshiera
3cf29a777d Fix nil check in rejectBySize 2020-09-21 19:20:24 +08:00
Andreas Olsson
fd1f7b7268 Use https:// to download appveyor tar
When supported one might as well use https://.
2020-09-20 18:49:46 +02:00
J0WI
090a73f7c5 Update Go version to 1.15 in Docker build script 2020-09-20 15:42:12 +00:00
rawtaz
1cfb01a8a6 Merge pull request #2936 from andreaso/contrib-update-version
Update versions in contribution documentation
2020-09-20 13:27:13 +02:00
Andreas Olsson
ce62d3d689 Update versions in contribution documentation
Reflects 3c44598 and 429f97b.
2020-09-20 10:32:27 +02:00
Michael Eischer
8c36317b71 rclone: use configured number of connections during create 2020-09-19 19:11:43 +02:00
Michael Eischer
60a5c35de9 rclone: close connection to rclone if open fails 2020-09-19 19:11:43 +02:00
Michael Eischer
9333f707fa init: use Create method for rclone backend
This properly issues the initial repository creation command

Fixes #1896
2020-09-19 19:11:43 +02:00
Alexander Neumann
429f97b887 Set development version for 0.10.0 2020-09-19 17:38:47 +02:00
Alexander Neumann
40832b2927 Add version for 0.10.0 2020-09-19 17:38:26 +02:00
Alexander Neumann
c8a94eced7 Update manpages and auto-completion 2020-09-19 17:38:26 +02:00
Alexander Neumann
ee6e981b4e Generate CHANGELOG.md for 0.10.0 2020-09-19 17:38:10 +02:00
Alexander Neumann
96fd982f6a Prepare changelog for 0.10.0 2020-09-19 17:37:59 +02:00
Alexander Neumann
6ff0082c02 Merge pull request #2928 from MichaelEischer/init-copy-chunker
init: Add `--copy-chunker-parameters` option
2020-09-19 17:32:13 +02:00
Alexander Neumann
95c1d7d959 Merge pull request #2927 from restic/update-deps
Update dependencies
2020-09-19 17:29:35 +02:00
Michael Eischer
07f4e7d10b Only log HTTP requests when a debug log is configured
The logged HTTP requests are only visible when a debug log is
configured.
2020-09-19 17:07:53 +02:00
Michael Eischer
f003410402 init: Add --copy-chunker-params option
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
2020-09-19 16:53:05 +02:00
Michael Eischer
9ad8250a78 Reduce overhead of debug calls if no log is enabled
In case no debug log is configured, then calls to debug.Log only incur
the costs to check a single boolean flag making the call really cheap.
2020-09-19 16:45:01 +02:00
Michael Eischer
655430550b Extract parameters for second repository from copy command 2020-09-19 16:07:55 +02:00
MichaelEischer
1823b8195c Merge pull request #2930 from rawtaz/password-file-text
Improve wording for --password-file and related options
2020-09-19 16:06:15 +02:00
Leo R. Lundgren
311ad2d2d0 Improve wording for --password-file and related options 2020-09-19 15:47:32 +02:00
Alexander Neumann
a10b44a265 Add note about fuse 2020-09-19 14:35:55 +02:00
Alexander Neumann
baf3a9aa3b Merge pull request #2929 from rawtaz/password-command-text
Improve wording for --password-command and --password-command2 options
2020-09-19 14:30:39 +02:00
Leo R. Lundgren
ffe6dce7e7 Improve wording for --password-command and --password-command2 options 2020-09-19 14:23:25 +02:00
Alexander Neumann
8ce0ce387f Add changelog 2020-09-19 14:22:31 +02:00
Alexander Neumann
3c44598bf6 Drop Go 1.12, require at least 1.13 2020-09-19 14:20:04 +02:00
Alexander Neumann
3bb55fd6bf Merge pull request #2914 from YoshieraHuang/max-file-size
Support excluding files by size
2020-09-19 14:12:15 +02:00
Alexander Neumann
36efefa7bd Update dependencies, pin fuse library
This updates all dependencies which causes the following changes:
 * The fuse library is set to the last version supporting macOS
 * The minimal version of Go required to build restic is now 1.13
2020-09-19 14:06:37 +02:00
yoshiera
ac4b8c98ac Support excluding files by size 2020-09-19 19:52:09 +08:00
MichaelEischer
4dcd6abf37 Merge pull request #2616 from greatroar/no-cache-interface
Remove practically unused cache interface
2020-09-18 19:50:47 +02:00
greatroar
cb3f531050 Make Cache.Path private
It's only used inside the package.
2020-09-18 11:17:29 +02:00
greatroar
23fcbb275a Remove restic.Cache interface
It was used in one code path, which then asserted its concrete type as
*cache.Cache.

Privatised some of the interface methods.
2020-09-18 10:48:13 +02:00
greatroar
6e3215a80d Remove unused error type from internal/cache 2020-09-18 10:38:14 +02:00
greatroar
9abef3bf1a Move internal/fs.TestChdir to internal/test.Chdir 2020-09-17 10:43:33 +02:00
Alexander Neumann
b10dce541e Merge pull request #2598 from MichaelEischer/diff-speedup
Diff speedup
2020-09-13 16:42:05 +02:00
Michael Eischer
4f221c4022 Add changelog entry 2020-09-13 16:28:46 +02:00
Michael Eischer
f5c448aa65 diff: Optimize diff calculation for shared subtrees
When the diff calculation compares two trees with identical id then no
differences between them can ever show up. Optimize for that case by
simply traversing the tree only once to collect all referenced blobs for
a proper calculation of added and removed blobs.

Just skipping the common subtrees is not possible as this would skew the
results if the added or removed blobs are shared with one of the
subtrees.
2020-09-13 16:28:46 +02:00
Michael Eischer
c0fc85d303 diff: Add integration test 2020-09-13 16:28:46 +02:00
Alexander Neumann
0c48e515f0 Merge pull request #2630 from MichaelEischer/fix-staticcheck
Fix lots of small issues reported by staticcheck
2020-09-13 16:19:22 +02:00
Alexander Neumann
97950ab81a options: Fix test for Go >= 1.15 2020-09-12 17:36:44 +02:00
Alexander Neumann
59fca85844 Remove darwin/386 from integration tests 2020-09-12 17:09:33 +02:00
Alexander Neumann
e207257714 Fix Appveyor 2020-09-12 16:59:03 +02:00
Alexander Neumann
82e1cbed4f Update Go versions 2020-09-12 16:52:21 +02:00
Alexander Neumann
8903b6c88a helpers: Rework list of architectures
This also removes darwin/386 which is not supported on Go 1.15 any more.
2020-09-12 16:39:41 +02:00
Alexander Neumann
93583c01b1 helpers: Fetch modules for release binaries 2020-09-12 16:25:36 +02:00
MichaelEischer
88664ba222 Merge pull request #2893 from MichaelEischer/restore-preallocate
restorer: Preallocate files
2020-09-08 22:43:05 +02:00
Michael Eischer
121233e1b3 Add preallocate tests 2020-09-07 21:41:47 +02:00
Michael Eischer
8cc9514879 restorer: pre-allocate files before loading chunks 2020-09-07 21:41:47 +02:00
Michael Eischer
2e7d475029 Process packs in order of first appearance 2020-09-06 21:25:35 +02:00
rawtaz
d3a286928a Merge pull request #2905 from MichaelEischer/fix-ls-help
ls: Explicitly enforce that the user specifies a snapshot ID
2020-09-05 10:44:01 +02:00
Michael Eischer
c46edcd9d6 error strings should not end with punctuation 2020-09-05 10:07:17 +02:00
Michael Eischer
1ede018ea6 error variable names should start with 'Err' 2020-09-05 10:07:17 +02:00
Michael Eischer
b77e933d80 Convert underscore variable names to camelCase 2020-09-05 10:07:16 +02:00
Michael Eischer
d0329cf3eb Adjust comments to match name of exported methods 2020-09-05 10:07:16 +02:00
Michael Eischer
dc31529fc3 Unindent else block after if block ending with a return statement 2020-09-05 10:07:16 +02:00
Michael Eischer
4784540f04 repository: Simplify worker group code 2020-09-05 10:07:16 +02:00
Michael Eischer
84ea2389ae archiver_test: Only skip symlinks on windows 2020-09-05 10:07:16 +02:00
Michael Eischer
b4a7ce86cf uint cannot be less than zero 2020-09-05 10:07:16 +02:00
Michael Eischer
7ee0964880 crypto: Remove unused error 2020-09-05 10:07:16 +02:00
Michael Eischer
f6f11400c2 restic: Remove unused variable value 2020-09-05 10:07:16 +02:00
Michael Eischer
e2dc5034d3 restic/file: Use FileType for all File constants
The data type of a variable is only carried on when using iota.
2020-09-05 10:07:16 +02:00
Michael Eischer
9a1b3cb5d9 restorer: Remove unused treeID struct member from test 2020-09-05 10:07:16 +02:00
Michael Eischer
b22655367c integration_test: Replace fprintf without format string 2020-09-05 10:07:16 +02:00
Michael Eischer
068a3ce23f Remove redundant return 2020-09-05 10:07:16 +02:00
Michael Eischer
ee05501ce7 archiver: Remove unused chmod function 2020-09-05 10:07:16 +02:00
Michael Eischer
014600bee6 archiver: Remove unused variable from test 2020-09-05 10:07:16 +02:00
Michael Eischer
d9a80e07b9 repository: Simplify index age calculation 2020-09-05 10:07:16 +02:00
Michael Eischer
d19f05c960 Use buf.String() instead of string(buf.Bytes()) 2020-09-05 10:07:16 +02:00
Michael Eischer
460e2ffbf6 Collapse a few boolean operations 2020-09-05 10:07:14 +02:00
Michael Eischer
49b6aac3fa Use !bytes.Equal instead of bytes.Compare !=0 to check for inequality 2020-09-05 10:06:23 +02:00
Michael Eischer
2f8335554c Remove a few unused variables 2020-09-05 10:06:23 +02:00
Michael Eischer
37113282ca Merge a few variable declaration and initializations 2020-09-05 10:05:34 +02:00
Michael Eischer
337725c354 Use sort.Strings 2020-09-05 10:05:34 +02:00
MichaelEischer
2ddb7ffb7e Merge pull request #2255 from Kidswiss/tar
Fix dumping issues with / and the first sub level
2020-09-01 21:52:17 +02:00
Michael Eischer
81dcfea11a dump: Mention tar output in command help text 2020-08-31 22:43:10 +02:00
MichaelEischer
55071ee367 Merge pull request #2859 from buschjost/stats-filter-by-tag-and-path
Add filter by tag and path to stats command
2020-08-31 22:11:01 +02:00
Michael Eischer
dcf9ded977 dump: Use slashes as path separators in tar on windows
This conversion is necessary as paths returned by filepath.Rel use the
operating-system native path separator
2020-08-30 18:26:50 +02:00
Michael Eischer
bcf44a9c3f dump: Properly handle file paths without / prefix
filepath.Rel failed if the requested path did not start with a `/` e.g.
`restic` instead of `/restic`.
2020-08-30 18:25:42 +02:00
Oliver Buschjost
a7b4c19abf Fix display of warning about ignored filters 2020-08-30 17:14:56 +02:00
Oliver Buschjost
d3fcfeba3a Add filter by tag and path to stats command 2020-08-30 17:14:48 +02:00
Michael Eischer
e69449bf2c dump: Add test for splitPath 2020-08-30 16:00:48 +02:00
Simon Beck
da4193c3ef Add changelog 2020-08-30 16:00:48 +02:00
Simon Beck
fe6445e0f4 dump: Added unit tests for tar output 2020-08-30 16:00:48 +02:00
MichaelEischer
ea81a0e282 Merge pull request #2606 from middelink/fix-323
Add copy functionality.
2020-08-30 10:18:24 +02:00
Simon Beck
80a11960dd dump: Always dump relative paths into tarballs
Tarballs should only contain relative paths.
2020-08-29 21:14:34 +02:00
Simon Beck
d6f739ec22 dump: Extract tar code to internal/dump 2020-08-29 21:14:34 +02:00
Simon Beck
b98598e55f dump: Fix dumping issues with / and the first sub level
There was an issue that prevented the dump command from working
correctly when either:

* `/` contained multiple nodes (e.g. `restic backup /`)
* dumping a file in the first sublevel was attempted (e.g. `/foo`)
2020-08-29 21:14:29 +02:00
Michael Eischer
d5f86effa1 ls: Explicitly enforce that the user specifies a snapshot ID
The help messages suggested that the `ls` command work without
explicitly passing a snapshot ID. However, this was never the case:
without a snapshot ID the command just failed with the error
`Ignoring "", it is not a snapshot id`.

Fixes #2299
2020-08-29 14:28:53 +02:00
MichaelEischer
c34c731698 Merge pull request #2702 from MichaelEischer/fix-backup-dir-count
backup: Fix reporting of directory count in summary
2020-08-29 11:29:21 +02:00
Michael Eischer
412623b848 copy: Reuse buffer for downloaded blobs 2020-08-29 10:48:44 +02:00
Michael Eischer
bbe8b73f03 Update help text of backup command in docs 2020-08-29 10:48:44 +02:00
Michael Eischer
91e8d998cd Add documentation for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer
9a4796594a integration tests: Fix checking of wrong snapshot 2020-08-29 10:48:44 +02:00
Michael Eischer
15374d22e9 integration tests: Add basic tests for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer
88ad58d6cd integration tests: Redirect directory diff into intermediate buffer 2020-08-29 10:48:44 +02:00
Michael Eischer
591a8c4cdf integration tests: Deduplicate backup test-data setup code 2020-08-29 10:48:44 +02:00
Michael Eischer
ec9a53b7e8 copy: Mark and skip previously copied snapshots
Use the `Original` field of the copied snapshot to store a persistent
snapshot ID. This can either be the ID of the source snapshot if
`Original` was not yet set or the previous value stored in the
`Original` field. In order to still copy snapshots modified using the
tags command the source snapshot is compared to all snapshots in the
destination repository which have the same persistent ID. Snapshots are
only considered equal if all fields except `Original` and `Parent`
match. That way modified snapshots are still copied while avoiding
duplicate copies at the same time.
2020-08-29 10:48:44 +02:00
MichaelEischer
34a3adfd8d Merge pull request #2898 from aawsome/add-iexclude-file
backup: Add --iexclude-file option
2020-08-28 21:59:16 +02:00
Alexander Weiss
9867c4bbb4 Add flag --iexclude-file to backup 2020-08-28 21:04:41 +02:00
MichaelEischer
efb4a981cf Merge pull request #2904 from jasonccox/consistent-usage-strings
Update usage strings to put flags before args
2020-08-28 20:39:14 +02:00
Jason Cox
2447f3f110 Update usage strings to put flags before args
The standard UNIX-style ordering of command-line arguments places
optional flags before other positional arguments. All of restic's
commands support this ordering, but some of the usage strings showed the
flags after the positional arguments (which restic also parses just
fine). This change updates the doc strings to reflect the standard
ordering.

Because the `restic help` command comes directly from Cobra, there does
not appear to be a way to update the argument ordering in its usage
string, so it maintains the non-standard ordering (positional arguments
before optional flags).
2020-08-28 02:16:22 +00:00
Michael Eischer
b25978a53c backup: Fix reporting of directory count in summary
Previously the directory stats were reported immediately after calling
`SaveDir`. However, as the latter method saves the tree asynchronously
the stats were still initialized to their nil value. The stats are now
reported via a callback similar to the one used for the fileSaver.
2020-08-27 22:43:51 +02:00
Michael Eischer
b0a8c4ad6c copy: Only process each tree once
This speeds up copying multiple overlapping snapshots from one
repository to another, as we only have to copy the changed parts of
later snapshots.
2020-08-26 22:17:52 +02:00
Michael Eischer
908b23fda0 copy: Update for modernized repository interface 2020-08-26 22:17:52 +02:00
greatroar
4508d406ef copy: Remove separate SaveIndex in restic copy
Flush does this now.
2020-08-26 22:17:52 +02:00
Pauline Middelink
7048cc3e58 Add copy functionality
Add a copy command to copy snapshots between repositories. It allows the user
to specify a destination repository, password, password-file, password-command
or key-hint to supply the necessary details to open the destination repository.

You need to supply a list of snapshots to copy, snapshots which already exist
in the destination repository will be skipped.

Note, when using the network this becomes rather slow, as it needs to read the
blocks, decrypt them using the source key, then encrypt them again using the
destination key before finally writing them out to the destination repository.
2020-08-26 22:17:52 +02:00
MichaelEischer
eb7c00387c Merge pull request #2899 from MichaelEischer/fix-check-progress-crash
Unify progress bar of check and prune commands
2020-08-26 00:00:50 +02:00
Michael Eischer
bc0501d72c Add changelog 2020-08-25 23:33:10 +02:00
Michael Eischer
17995dec7a Unify progress bar of check and prune commands 2020-08-25 22:47:38 +02:00
MichaelEischer
e915cedc3d Merge pull request #2897 from restic/rawtaz-github-issue-feature
github: Add question to feature issue template
2020-08-24 23:52:38 +02:00
MichaelEischer
cdcaecd27d Merge pull request #2896 from restic/rawtaz-key-hint
doc: Add missing environment variables
2020-08-24 23:51:05 +02:00
rawtaz
b43ab67a22 doc: Add missing environment variables 2020-08-24 00:49:34 +02:00
rawtaz
7ddfd6cabe github: Add question to feature issue template 2020-08-24 00:22:44 +02:00
Alexander Neumann
b1b3f1ecb6 Merge pull request #2674 from MichaelEischer/prune-strict-checks
prune: Stricter error checks
2020-08-23 10:30:41 +02:00
rawtaz
fa135f72bf Merge pull request #2439 from htrendev/ignore-sync-on-macos
Ignore not supported error on sync() when using local backend
2020-08-22 01:38:49 +02:00
Hristo Trendev
51c22f4223 local backend: ignore not supported error on sync()
Closes #2395
2020-08-22 01:27:07 +02:00
rawtaz
1a5b66f33b Merge pull request #2879 from 0xMH/master
Added more explantation for --one-file-system option
2020-08-21 19:21:01 +02:00
NoNE
da6a34e044 doc: Add more explanation for --one-file-system option 2020-08-21 19:19:13 +02:00
Bruce Dillahunty
fe69b83074 Update key password prompt (#2847) 2020-08-19 21:42:08 +02:00
Michael Eischer
08d24ff99e prune: Include ID of all missing blobs in error message 2020-08-16 11:36:14 +02:00
Michael Eischer
d8b80e9862 Add changelog 2020-08-16 11:36:14 +02:00
Michael Eischer
1c84aceb39 prune: Test for abort on damaged repositories 2020-08-16 11:36:12 +02:00
Michael Eischer
575ed9a47e Test that rebuild-index errors when old index cannot be removed 2020-08-16 11:34:01 +02:00
Michael Eischer
8f811642c3 Add support for integration tests to wrap the backend 2020-08-16 11:34:01 +02:00
Michael Eischer
f4b9544ab2 prune: Add test that repack aborts on wrong blob 2020-08-16 11:34:01 +02:00
Michael Eischer
367449dede prune: Reduce memory allocations while repacking
The slicing operator `slice[low:high]` default to 0 for the lower bound and
len(slice) for the upper bound when either or both are not specified.
Fix the code to use `cap(slice)` to check for the slice capacity.
2020-08-16 11:34:01 +02:00
Michael Eischer
7042bafea5 prune: Abort repacking when a pack contains a wrong blob
If a blob in a pack file can be decrypted successfully but contains data
that results in a different hash than stated in the header pack, then
abort repacking. As both the pack header and the blob are
cryptographically verified this either means than a malicious entity
tampered with the backup or indicates hardware problems on the client.
prune should fail with an error in both cases.
2020-08-16 11:34:01 +02:00
Michael Eischer
744a15247d prune/rebuild-index: Fail if an old index cannot be removed
The old behavior was problematic in the context of rebuild-index as it
could leave old, possibly invalid index files behind without returning a
fatal error.

Prune calls rebuildIndex before removing any data from the repository.
For this use case failing to delete an old index MUST be treated as a
fatal error. Otherwise the index could still contain an old index file
that refers to blobs/packs that were later on deleted by prune. Later
backup runs will assume that the affected blobs already exist in the
repository which results in a backup which misses data.
2020-08-16 11:34:01 +02:00
Michael Eischer
3ba19869be prune: Abort if any used blobs are missing
The previous check only approximately verified whether all required
blobs were found. However, after forgetting a few snapshots the
repository contains lots of unused blobs whose number can be sufficient
to make up for missing packs.

When coupled with a malfunctioning backend that temporarily returns broken
data this could cause restic to regard the corresponding packs as
invalid and thereby delete data that's still in use. This change lets
restic play it safe and refuse to delete anything if data is missing.
2020-08-16 11:34:01 +02:00
aawsome
0fed6a8dfc Use "pack file" instead of "data file" (#2885)
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
rawtaz
643bbbe156 doc: Correct wording for repository (#2884)
Replaces "backend" with "repository" to match restic output.
2020-08-11 22:30:17 +02:00
MichaelEischer
eca0f0ad24 Merge pull request #2863 from aawsome/index-no-duplicates
Don't save exact duplicates in merged index
2020-08-08 18:24:14 +02:00
MichaelEischer
08dee8a52b Merge pull request #2865 from greatroar/unused
Dead code removal
2020-08-08 16:15:28 +02:00
Alexander Weiss
b112533812 Don't save exact duplicates when merging indexes 2020-08-05 06:32:02 +02:00
Alexander Weiss
5e63294355 Add benchmark MasterIndexAlloc 2020-08-05 06:32:02 +02:00
MichaelEischer
84b6f1ec53 Merge pull request #2874 from MichaelEischer/fix-verbose
Adjust description of --verbose=n parameter
2020-08-04 23:21:05 +02:00
MichaelEischer
06fb4ea3f0 Merge pull request #2805 from jwilk-forks/verbose
Fix examples of --verbose with argument
2020-08-04 23:11:35 +02:00
Michael Eischer
e38d415173 Adjust description of --verbose=n parameter 2020-08-04 23:07:53 +02:00
greatroar
d81a396944 Dead code removal
Found by running golangci-lint on the entire code base.
2020-08-04 08:38:57 +02:00
rawtaz
0b21ec44b7 Merge pull request #2869 from josephrocca/patch-1
doc: --verbose 2 to --verbose=2
2020-08-04 01:06:58 +02:00
josephrocca
bd36731119 --verbose 2 to --verbose=2
`--verbose 2` seems to be incorrect here (gives an error/warning that "the `2` directory does not exist, skipping")
2020-08-04 08:39:43 +10:00
MichaelEischer
38a2f9c07b Merge pull request #2864 from greatroar/dump-error
Fix error handling in dump (err != err)
2020-08-03 20:09:42 +02:00
rawtaz
5af2815627 Merge pull request #2821 from renard/mount-nolock
cmd/mount: honur --no-lock flag
2020-08-03 20:09:26 +02:00
MichaelEischer
b55de2260d Merge pull request #2868 from MichaelEischer/fix-getusedblobs
prune: Stop progress bar after searching used blobs
2020-08-03 19:58:59 +02:00
Sébastien Gross
9be4fe3e84 cmd/mount: honour --no-lock flag
Do not lock the repository if --no-lock global flag is set. This allows
to mount repositories which are archived on a read only system.

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2020-08-03 19:45:39 +02:00
Michael Eischer
05116e4787 prune: Cleanup progress bar handling while repacking 2020-08-03 19:32:46 +02:00
Michael Eischer
04f79b9642 prune: Stop progress bar after searching used blobs 2020-08-03 19:31:49 +02:00
greatroar
8b358935a0 Fix error handling in dump (err != err) 2020-08-03 09:43:44 +02:00
MichaelEischer
66d089e239 Merge pull request #2841 from aawsome/optimize-getUsedBlobs
Extract get used blobs in prune as separate function
2020-08-01 22:40:33 +02:00
MichaelEischer
49d3efe547 Merge pull request #2840 from aawsome/delete-parallel
Make delete parallel
2020-08-01 22:26:15 +02:00
Alexander Weiss
9762bec091 Use optimized getUsedBlobs in prune 2020-08-01 21:07:31 +02:00
Alexander Weiss
0eb8553c87 add changelog for #2840 2020-08-01 20:43:18 +02:00
Alexander Weiss
d3692f5b81 Delete files in parallel in rebuild-index 2020-08-01 20:43:18 +02:00
Alexander Weiss
1c0b61204b Delete files in parallel in forget 2020-08-01 20:43:18 +02:00
Alexander Weiss
2ee654763b Delete files in parallel 2020-08-01 20:39:24 +02:00
Alexander Neumann
b7b479b668 Merge pull request #2599 from MichaelEischer/tweak-mem-usage
Reduce memory usage when searching for used blobs
2020-08-01 13:49:02 +02:00
Alexander Neumann
4cf9656f12 Merge pull request #2861 from MichaelEischer/fix-rclone-crash
rclone: Don't panic after unexpected subprocess exit
2020-08-01 13:08:50 +02:00
Alexander Neumann
2580eef2aa Merge pull request #2318 from classmarkets/2175-named-keys
Allow specifying user and host when adding keys
2020-08-01 13:06:31 +02:00
Michael Eischer
2d7ab9115f Add changelog entry 2020-08-01 12:29:16 +02:00
Michael Eischer
a178e5628e Remove duplicate TreeLoader interface 2020-08-01 12:29:16 +02:00
Michael Eischer
af66a62c04 FindUsedBlobs: Test that seen blobs are skipped
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
2020-08-01 12:29:16 +02:00
Michael Eischer
9ea1a78bd4 FindUsedBlobs: Check for seen blobs before loading trees
The only effective change in behavior is that that toplevel nodes can
also be skipped.
2020-08-01 12:29:16 +02:00
Michael Eischer
184103647a FindUsedBlobs: merge seen into blobs BlobSet
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.

Suggested-by: Alexander Weiss <alex@weissfam.de>
2020-08-01 12:29:16 +02:00
Michael Eischer
c81b122374 rclone: Don't panic after unexpected subprocess exit
As the connection to the rclone child process is now closed after an
unexpected subprocess exit, later requests will cause the http2
transport to try to reestablish a new connection. As previously this never
should have happened, the connection called panic in that case. This
panic is now replaced with a simple error message, as it no longer
indicates an internal problem.
2020-08-01 12:17:40 +02:00
rawtaz
48f97f3567 Merge pull request #2857 from MichaelEischer/incomplete-backup-error
Don't print a stacktrace if some files could not be read
2020-07-29 00:06:25 +02:00
Michael Eischer
3ce9893e0b Don't print a stacktrace if some files could not be read 2020-07-28 23:52:48 +02:00
MichaelEischer
248c7c3828 Merge pull request #2587 from aawsome/optimize-fuse
Make `restic mount` faster and consume less memory
2020-07-28 23:19:17 +02:00
Alexander Weiss
f8316948d1 Optimize FUSE - make command restic mount faster and consume less memory
- Add Open() functionality to dir
- only access index for blobs when file is read
- Implement NodeOpener and put one-time file stuff there
- Add comment about locking as suggested by bazil.org/fuse

=> Thanks at Michael Eischer for suggesting the last two improvements
2020-07-28 23:01:18 +02:00
MichaelEischer
be54ceff66 Merge pull request #2855 from MichaelEischer/rclone-exit
Fix rclone subprocess handling
2020-07-26 22:55:25 +02:00
Michael Eischer
ea97ff1ba4 rclone: Skip crash test when rclone is not found 2020-07-26 12:06:18 +02:00
Michael Eischer
01b9581453 rclone: Better field names for stdio conn 2020-07-26 00:29:25 +02:00
Michael Eischer
3cd927d180 rclone: Give rclone time to finish before closing stdin pipe
Calling `Close()` on the rclone backend sometimes failed during test
execution with 'signal: Broken pipe'. The stdio connection closed both
the stdin and stdout file descriptors at the same moment, therefore
giving rclone no chance to properly send any final http2 data frames.

Now the stdin connection to rclone is closed first and will only be
forcefully closed after a timeout. In case rclone exits before the
timeout then the stdio connection will be closed normally.
2020-07-26 00:29:25 +02:00
greatroar
bf7b1f12ea rclone: Add test for pipe handling when rclone exits 2020-07-26 00:29:25 +02:00
Michael Eischer
8554332894 rclone: Close rclone side of stdio_conn pipes
restic did not notice when the rclone subprocess exited unexpectedly.

restic manually created pipes for stdin and stdout and used these for the
connection to the rclone subprocess. The process creating a pipe gets
file descriptors for the sender and receiver side of a pipe and passes
them on to the subprocess. The expected behavior would be that reads or
writes in the parent process fail / return once the child process dies
as a pipe would now just have a reader or writer but not both.

However, this never happened as restic kept the reader and writer
file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe`
close the subprocess side of pipes once the child process was started
and close the parent process side of pipes once wait has finished. We
can't use these functions as we need access to the raw `os.File` so just
replicate that behavior.
2020-07-26 00:29:25 +02:00
greatroar
3e93b36ca4 Make rclone.New private 2020-07-26 00:28:45 +02:00
MichaelEischer
573a2fb240 Merge pull request #2789 from aawsome/fix-lookup
make Lookup() return all blobs
2020-07-25 21:32:23 +02:00
Michael Eischer
c847aace35 Rename Index interface to MasterIndex
The interface is now only implemented by repository.MasterIndex.
2020-07-25 21:19:46 +02:00
Alexander Weiss
9d1fb94c6c make Lookup() return all blobs
+ simplify syntax
2020-07-25 21:18:34 +02:00
MichaelEischer
020cab8e08 Merge pull request #2787 from greatroar/no-blobsize-cache
Remove blob size cache from restic mount
2020-07-25 20:46:35 +02:00
greatroar
07da61baee Remove blob size cache from restic mount 2020-07-25 19:39:14 +02:00
MichaelEischer
37c95bf5da Merge pull request #2835 from J0WI/patch-1
Update Go version to 1.14.6 in Docker build script
2020-07-25 17:23:54 +02:00
rawtaz
c86d2f23aa Merge pull request #2854 from restic/rawtaz-patch-1
Clarify questions and text in PR template
2020-07-25 15:41:27 +02:00
rawtaz
96ec04d74d Clarify questions and text in PR template 2020-07-25 15:38:00 +02:00
J0WI
9c3414374a Update Go version to 1.14.6 in Docker build script 2020-07-25 14:45:36 +02:00
MichaelEischer
3d530dfc91 Merge pull request #2827 from aawsome/archiver-test-contents
Make self-healing work when backing up with parent snapshot
2020-07-25 13:13:18 +02:00
MichaelEischer
c43f5b2664 Merge pull request #2733 from hibbert/support_irsa
Enable support for EKS IRSA
2020-07-25 12:58:43 +02:00
Andy Hibbert
38087e40d9 Update minio-go version so it supports EKS IRSA 2020-07-25 12:41:18 +02:00
MichaelEischer
bbc960f957 Merge pull request #2635 from greatroar/optimize-sortbycached
Optimize sorting blobs by cache status
2020-07-25 12:35:42 +02:00
greatroar
309598c237 Simplify sortCachedPacksFirst test in internal/repository
The test now uses the fact that the sort is stable. It's not guaranteed
to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst
no longer needs a return value.
2020-07-25 12:12:59 +02:00
greatroar
03d23e6faa Speed up blob sorting in internal/repository
name                    old time/op    new time/op    delta
SortCachedPacksFirst-8     208µs ± 3%     186µs ± 3%  -10.74%  (p=0.000 n=10+8)

name                    old alloc/op   new alloc/op   delta
SortCachedPacksFirst-8     213kB ± 0%     139kB ± 0%  -34.62%  (p=0.000 n=10+10)

name                    old allocs/op  new allocs/op  delta
SortCachedPacksFirst-8     1.03k ± 0%     1.03k ± 0%   -0.19%  (p=0.000 n=10+10)
2020-07-25 12:12:59 +02:00
greatroar
b10acd2af7 Test and benchmark blob sorting in internal/repository 2020-07-25 12:12:58 +02:00
Alexander Weiss
9175795fdb Check contents in archiver
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
2020-07-25 08:18:28 +02:00
rawtaz
5d8d70542f Merge pull request #2852 from MichaelEischer/drop-go-1.11
Drop support for Go version 1.11
2020-07-25 01:17:36 +02:00
Michael Eischer
7c23381a2b Drop support for Go version 1.11 2020-07-24 18:52:39 +02:00
MichaelEischer
34181b13a2 Merge pull request #2328 from MichaelEischer/no-repeated-checks
Fix duplicate tree checks within `restic check`
2020-07-22 22:08:02 +02:00
MichaelEischer
bcd47ec3a2 Merge pull request #2818 from aawsome/merge-index-chaining
Merge index (based on chaining index implementation)
2020-07-22 22:05:36 +02:00
Alexander Weiss
a666a6d576 Add tests and merge indexes in index benchmarks 2020-07-22 21:54:02 +02:00
Alexander Weiss
e388d962a5 Merge final indexes together for faster index access 2020-07-22 21:54:02 +02:00
Peter Schultz
758b44b9c0 gs: support authentication with access token
In the Google Cloud Storage backend, support specifying access tokens
directly, as an alternative to a credentials file. This is useful when
restic is used non-interactively by some other program that is already
authenticated and eliminates the need to store long lived credentials.

The access token is specified in the GOOGLE_ACCESS_TOKEN environment
variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.
2020-07-22 16:23:03 +02:00
Alexander Weiss
3b7a3711e6 Add more realistic index benchmarks
+ reduce test size of BenchmarkMasterIndexLookupParallel
2020-07-21 07:18:20 +02:00
Michael Eischer
9b0e718852 checker: Test that blob types are not confused 2020-07-20 23:43:47 +02:00
MichaelEischer
82c908871d Merge pull request #2812 from greatroar/chaining
Chaining hash table for repository.Index
2020-07-20 23:29:36 +02:00
Michael Eischer
ddf0b8cd0b checker: Properly distinguish between data and tree blobs
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
2020-07-20 22:58:39 +02:00
Michael Eischer
2d0c138c9b checker: Test that check only decodes trees once
The `DuplicateTree` flag is necessary to ensure that failures cannot be
swallowed. The old checker implementation ignores errors from LoadTree
if the corresponding tree was already checked.
2020-07-20 22:51:53 +02:00
Michael Eischer
ef325ffc02 checker: Cleanup error handling code
This change only moves code around but does not result in any change in
behavior.
2020-07-20 22:51:53 +02:00
Michael Eischer
0f67ae813a Add changelog 2020-07-20 22:51:53 +02:00
Michael Eischer
7a165f32a9 checker: Traverse trees in depth-first order
Backups traverse the file tree in depth-first order and saves trees on
the way back up. This results in tree packs filled in a way comparable
to the reverse Polish notation.  In order to check tree blobs in that
order, the treeFilter would have to delay the forwarding of tree nodes
until all children of it are processed which would complicate the
implementation.

Therefore do the next similar thing and traverse the tree in depth-first
order, but process trees already on the way down. The tree blob ids are
added in reverse order to the backlog, which is once again reverted when
removing the ids from the back of the backlog.
2020-07-20 22:51:53 +02:00
Michael Eischer
36c69e3ca7 checker: Unify blobs, processed trees and referenced blobs map
The blobRefs map and the processedTrees IDSet are merged to reduce the
memory usage. The blobRefs map now uses separate flags to track blob
usage as data or tree blob. This prevents skipping of trees whose
content is identical to an already processed data blob. A third flag
tracks whether a blob exists or not, which removes the need for the
blobs IDSet.
2020-07-20 22:51:47 +02:00
Michael Eischer
35d8413639 checker: Remove dead index map 2020-07-20 22:37:31 +02:00
Michael Eischer
c66a0e408c checker: Reduce cost of debug log
Avoid duplicate allocation of the Subtree list.
2020-07-20 22:37:31 +02:00
Michael Eischer
70f4c014ef checker: Decode identical tree nodes only once
Even though the checkTreeWorker skips already processed chunks,
filterTrees did queue the same tree blob on every occurence. This
becomes a serious performance bottleneck for larger number of snapshots
that cover mostly the same directories. Therefore decode a tree blob
exactly once.
2020-07-20 22:37:31 +02:00
Michael Eischer
f0d8710611 Add benchmark for checker scaling with snapshot count 2020-07-20 22:37:31 +02:00
MichaelEischer
bd3e280f6d Merge pull request #2546 from Summerdave/fix-backup-error-code
Return an error when errors occured during backup
2020-07-20 22:31:46 +02:00
MichaelEischer
2746dcdb5f Merge pull request #2845 from restic/update-chunker
Update chunker
2020-07-20 22:20:28 +02:00
David Sommer
5729d967f5 backup: Return exit status code 3 when failing to read source data
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).

This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:

 - 0 when the command was successful
 - 1 when there was a fatal error (no snapshot created)
 - 3 when some source data could not be read (incomplete snapshot created)
2020-07-20 22:19:01 +02:00
Alexander Neumann
f9f6124558 Update chunker 2020-07-20 22:07:02 +02:00
Alexander Neumann
8074879c5f Remove 'go generate' 2020-07-19 17:28:42 +02:00
greatroar
7bda28f31f Chaining hash table for repository.Index
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.

Benchmark results on linux/amd64, -benchtime=3s -count=20:

name                                            old time/op    new time/op    delta
PackerManager-8                                    178ms ± 0%     178ms ± 0%     ~     (p=0.231 n=20+20)
DecodeIndex-8                                      4.54s ± 0%     4.30s ± 0%   -5.20%  (p=0.000 n=18+17)
DecodeIndexParallel-8                              4.54s ± 0%     4.30s ± 0%   -5.22%  (p=0.000 n=19+18)
IndexHasUnknown-8                                 44.4ns ± 5%    50.5ns ±11%  +13.82%  (p=0.000 n=19+17)
IndexHasKnown-8                                   48.3ns ± 0%    51.5ns ±12%   +6.68%  (p=0.001 n=16+20)
IndexAlloc-8                                       758ms ± 1%     616ms ± 1%  -18.69%  (p=0.000 n=19+19)
IndexAllocParallel-8                               234ms ± 3%     204ms ± 2%  -12.60%  (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8                     122ns ± 0%     145ns ± 9%  +18.44%  (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8                   369ns ± 2%     429ns ± 8%  +16.27%  (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8             68.4ns ± 5%    74.9ns ±13%   +9.47%  (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8            315ns ± 3%     369ns ±11%  +17.14%  (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8        743ns ± 1%     816ns ± 2%   +9.87%  (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8      238ns ± 1%     260ns ± 2%   +9.14%  (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8      1.01µs ± 3%    1.11µs ± 2%   +9.79%  (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8     222ns ± 0%     269ns ± 2%  +20.83%  (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8      1.06µs ± 2%    1.19µs ± 2%  +12.95%  (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8     413ns ± 1%     530ns ± 1%  +28.19%  (p=0.000 n=18+20)
SaveAndEncrypt-8                                  30.2ms ± 1%    30.4ms ± 0%   +0.71%  (p=0.000 n=19+19)
LoadTree-8                                         540µs ± 1%     576µs ± 1%   +6.73%  (p=0.000 n=20+20)
LoadBlob-8                                        5.64ms ± 0%    5.64ms ± 0%     ~     (p=0.883 n=18+17)
LoadAndDecrypt-8                                  5.93ms ± 0%    5.95ms ± 1%     ~     (p=0.247 n=20+19)
LoadIndex-8                                       25.1ms ± 0%    24.5ms ± 1%   -2.54%  (p=0.000 n=18+17)

name                                            old speed      new speed      delta
PackerManager-8                                  296MB/s ± 0%   296MB/s ± 0%     ~     (p=0.229 n=20+20)
SaveAndEncrypt-8                                 139MB/s ± 1%   138MB/s ± 0%   -0.71%  (p=0.000 n=19+19)
LoadBlob-8                                       177MB/s ± 0%   177MB/s ± 0%     ~     (p=0.890 n=18+17)
LoadAndDecrypt-8                                 169MB/s ± 0%   168MB/s ± 1%     ~     (p=0.227 n=20+19)

name                                            old alloc/op   new alloc/op   delta
PackerManager-8                                   91.8kB ± 0%    91.8kB ± 0%     ~     (p=0.772 n=12+19)
IndexAlloc-8                                       786MB ± 0%     400MB ± 0%  -49.04%  (p=0.000 n=20+18)
IndexAllocParallel-8                               786MB ± 0%     401MB ± 0%  -49.04%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                  21.0MB ± 0%    21.0MB ± 0%   +0.00%  (p=0.000 n=19+19)

name                                            old allocs/op  new allocs/op  delta
PackerManager-8                                    1.41k ± 0%     1.41k ± 0%     ~     (all equal)
IndexAlloc-8                                        977k ± 0%      907k ± 0%   -7.18%  (p=0.000 n=20+20)
IndexAllocParallel-8                                977k ± 0%      907k ± 0%   -7.17%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                    73.0 ± 0%      73.0 ± 0%     ~     (all equal)
2020-07-19 13:58:22 +02:00
greatroar
255ba83c4b Parallel index benchmarks + benchmark optimizations
createRandomIndex was using the global RNG, which locks on every call
It was also using twice as many random numbers as necessary and doing
a float division in every iteration of the inner loop.

BenchmarkDecodeIndex was using too short an input, especially for a
parallel version. (It may now be using one that is a bit large.)

Results on linux/amd64, -benchtime=3s -count=20:

name                                     old time/op    new time/op       delta
PackerManager-8                             178ms ± 0%        178ms ± 0%           ~     (p=0.165 n=20+20)
DecodeIndex-8                              13.6µs ± 2%  4539886.8µs ± 0%  +33293901.38%  (p=0.000 n=20+18)
IndexHasUnknown-8                          44.4ns ± 7%       44.4ns ± 5%           ~     (p=0.873 n=20+19)
IndexHasKnown-8                            49.2ns ± 3%       48.3ns ± 0%         -1.86%  (p=0.000 n=20+16)
IndexAlloc-8                                802ms ± 1%        758ms ± 1%         -5.51%  (p=0.000 n=20+19)
MasterIndexLookupSingleIndex-8              124ns ± 1%        122ns ± 0%         -1.41%  (p=0.000 n=20+14)
MasterIndexLookupMultipleIndex-8            373ns ± 2%        369ns ± 2%         -1.13%  (p=0.001 n=20+20)
MasterIndexLookupSingleIndexUnknown-8      67.8ns ± 3%       68.4ns ± 5%           ~     (p=0.753 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8     316ns ± 3%        315ns ± 3%           ~     (p=0.846 n=20+20)
SaveAndEncrypt-8                           30.5ms ± 1%       30.2ms ± 1%         -1.09%  (p=0.000 n=19+19)
LoadTree-8                                  527µs ± 1%        540µs ± 1%         +2.37%  (p=0.000 n=19+20)
LoadBlob-8                                 5.65ms ± 0%       5.64ms ± 0%         -0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                           7.07ms ± 2%       5.93ms ± 0%        -16.15%  (p=0.000 n=19+20)
LoadIndex-8                                32.1ms ± 2%       25.1ms ± 0%        -21.64%  (p=0.000 n=20+18)

name                                     old speed      new speed         delta
PackerManager-8                           296MB/s ± 0%      296MB/s ± 0%           ~     (p=0.159 n=20+20)
SaveAndEncrypt-8                          138MB/s ± 1%      139MB/s ± 1%         +1.10%  (p=0.000 n=19+19)
LoadBlob-8                                177MB/s ± 0%      177MB/s ± 0%         +0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                          141MB/s ± 2%      169MB/s ± 0%        +19.24%  (p=0.000 n=19+20)

name                                     old alloc/op   new alloc/op      delta
PackerManager-8                            91.8kB ± 0%       91.8kB ± 0%           ~     (p=0.826 n=19+12)
IndexAlloc-8                                786MB ± 0%        786MB ± 0%         +0.01%  (p=0.000 n=20+20)
SaveAndEncrypt-8                           21.0MB ± 0%       21.0MB ± 0%         -0.00%  (p=0.012 n=20+19)

name                                     old allocs/op  new allocs/op     delta
PackerManager-8                             1.41k ± 0%        1.41k ± 0%           ~     (all equal)
IndexAlloc-8                                 977k ± 0%         977k ± 0%         +0.01%  (p=0.022 n=20+20)
SaveAndEncrypt-8                             73.0 ± 0%         73.0 ± 0%           ~     (all equal)
2020-07-19 13:58:05 +02:00
MichaelEischer
7dc200c593 Merge pull request #2832 from alrs/fix-fuse-test-err
internal/fuse: fix dropped test error
2020-07-16 22:28:20 +02:00
Lars Lehtonen
9ac90cf5cd internal/fuse: fix dropped test error 2020-07-12 21:42:31 -07:00
MichaelEischer
b84f5177cb Merge pull request #2790 from greatroar/fix-quadratic-read
Fix quadratic file reading in restic mount
2020-07-12 18:42:14 +02:00
greatroar
4cf1c8e8da Changelog entry for subquadratic reading in mount 2020-07-12 18:27:16 +02:00
greatroar
58719e1f47 Replace mount's per-file cache by a global LRU cache 2020-07-12 18:27:16 +02:00
greatroar
d42c169458 Fix quadratic file reading in restic mount 2020-07-12 18:27:16 +02:00
rawtaz
8598bb042b Merge pull request #2769 from darkdragon-001/patch-1
Update documentation for environment variables
2020-07-12 00:40:21 +02:00
darkdragon-001
c6b74962df Update 040_backup.rst
Changes proposed in #2763:

- Adding `RESTIC_CACHE_DIR` environment variables (introduced in #2425 for Unix and #2607 for Mac, Win).
- Adding used system-wide environment variables with links to the corresponding section.
2020-07-12 00:31:56 +02:00
MichaelEischer
2c72924ffb Merge pull request #2813 from greatroar/encrypt-benchmark
Fix repository_test.BenchmarkSaveAndEncrypt
2020-07-05 17:52:10 +02:00
greatroar
02bec13ef2 Fix repository_test.BenchmarkSaveAndEncrypt
The benchmark was actually testing the speed of index lookups.

name              old time/op    new time/op        delta
SaveAndEncrypt-8     101ns ± 2%    31505824ns ± 1%    +31311591.31%  (p=0.000 n=10+10)

name              old speed      new speed          delta
SaveAndEncrypt-8  41.7TB/s ± 2%       0.0TB/s ± 1%         -100.00%  (p=0.000 n=10+10)

name              old alloc/op   new alloc/op       delta
SaveAndEncrypt-8     1.00B ± 0%  20989508.40B ± 0%  +2098950740.00%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op      delta
SaveAndEncrypt-8      0.00             123.00 ± 0%            +Inf%  (p=0.000 n=10+9)

(The actual speed is ca. 131MiB/s.)
2020-07-05 17:41:42 +02:00
MichaelEischer
64976b1a4d Merge pull request #2815 from scelfo/patch-1
--hostname flag is deprecated
2020-06-29 22:04:56 +02:00
Tony Scelfo
6a607d6ded --hostname flag is deprecated
Update documentation to follow the deprecation warning "Flag --hostname has been deprecated, use --host" that is given on the latest build(s).
2020-06-29 12:53:57 -06:00
rawtaz
6fedf1a7f4 Merge pull request #2809 from lessfoobar/patch-2
Update 080_examples.rst
2020-06-26 00:12:05 +02:00
@lessfoobar
df946fd9f8 Update 080_examples.rst 2020-06-26 00:01:30 +02:00
Jakub Wilk
4e6a9767de Fix examples of --verbose with argument 2020-06-23 20:22:14 +02:00
rawtaz
1bc80c3c8d Merge pull request #2802 from andreaso/cachedir-https-links
Use https:// for linking CACHEDIR.TAG specs
2020-06-21 20:01:42 +02:00
Andreas Olsson
0fcef2ec23 Use https:// for linking CACHEDIR.TAG specs
That site might not have supported https:// when those links were
originally added. It does now.

Also dropping the _spec.html_ ending of the url, there being a `<link
rel="canonical" ...>` tag suggesting that that no longer being the
preferred address.
2020-06-21 19:36:26 +02:00
MichaelEischer
212607dc8a Merge pull request #2760 from greatroar/backend-benchmark
Fix backend benchmarks + a micro-optimization
2020-06-17 23:17:05 +02:00
greatroar
55c3a90a0d Clarify rclone-over-SSH docs
Also added a link to S. Ruderich's blog post explaining append-only
repos using rclone and SSH.
2020-06-17 15:22:20 +02:00
greatroar
190d8e2f51 Flatten backend.LimitedReadCloser structure
This inlines the io.LimitedReader into the LimitedReadCloser body to
achieve fewer allocations. Results on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadPartialFile-8           412µs ± 4%     413µs ± 4%    ~     (p=0.634 n=17+17)
Backend/BenchmarkLoadPartialFileOffset-8     455µs ±13%     441µs ±10%    ~     (p=0.072 n=20+18)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadPartialFile-8        10.2GB/s ± 3%  10.2GB/s ± 4%    ~     (p=0.817 n=16+17)
Backend/BenchmarkLoadPartialFileOffset-8  9.25GB/s ±12%  9.54GB/s ± 9%    ~     (p=0.072 n=20+18)

name                                      old alloc/op   new alloc/op   delta
Backend/BenchmarkLoadPartialFile-8            888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)

name                                      old allocs/op  new allocs/op  delta
Backend/BenchmarkLoadPartialFile-8            18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
2020-06-17 13:11:45 +02:00
greatroar
f4cd2a7120 Make backend benchmarks fairer by removing checks
Checking whether the right data is returned takes up half the time in
some benchmarks. Results for local backend benchmarks on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadFile-8                 4.89ms ± 0%    2.72ms ± 1%   -44.26%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8           936µs ± 6%     439µs ±15%   -53.07%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8     940µs ± 1%     456µs ±10%   -51.50%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                     23.9ms ±14%    24.8ms ±41%      ~     (p=0.690 n=5+5)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadFile-8               3.43GB/s ± 0%  6.16GB/s ± 1%   +79.40%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8        4.48GB/s ± 6%  9.63GB/s ±14%  +114.78%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8  4.46GB/s ± 1%  9.22GB/s ±10%  +106.74%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                    706MB/s ±13%   698MB/s ±31%      ~     (p=0.690 n=5+5)
2020-06-17 13:11:45 +02:00
MichaelEischer
aba270df7e Merge pull request #2786 from aawsome/memory-list-blobs
Loop over index files for 'list blobs'
2020-06-15 22:27:29 +02:00
Alexander Weiss
b5543cff5d Loop over index files for 'list blobs'
=> reduces memory consumption a lot!
2020-06-14 17:25:34 +02:00
Alexander Neumann
285b5236c2 Merge pull request #2684 from MichaelEischer/cleanup-output
Cleanup command output code
2020-06-14 11:25:40 +02:00
Michael Eischer
bb1e258bb7 cat/stats: Get stdout from globalOptions 2020-06-14 11:08:11 +02:00
Michael Eischer
182655bc88 Replace fmt.Printf/Println/Fprintf with wrapper functions
cmd/restic/globals.go already provides Printf, Println and Warnf wrapper
which get their output streams from the globalOptions object. This
allows for stream replacements when testing.
2020-06-14 11:08:11 +02:00
MichaelEischer
74bc7141c1 Merge pull request #2781 from aawsome/reduce-index-memory
Reduce index memory
2020-06-14 10:58:28 +02:00
Alexander Weiss
1361341c58 don't save duplicate packIDs when using internal/repository/Index.Store 2020-06-14 07:56:24 +02:00
Alexander Weiss
ce4a2f4ca6 save packIDs and duplicates separately
A side remark to the definition of Index.blob:

Another possibility would have been to use:
blob       map[restic.BlobHandle]*indexEntry

This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry:  8 + 4 + 4 = 16 bytes
each packID: 32 bytes

To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes

More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
2020-06-14 07:56:21 +02:00
Alexander Weiss
cf979e2b81 make offset and length uint32 2020-06-14 07:50:19 +02:00
Michael Eischer
d92e2c5769 simplify index code 2020-06-14 07:50:19 +02:00
Alexander Weiss
7419844885 add changelog, benchmark, memory calculation 2020-06-14 07:50:15 +02:00
rawtaz
1d66bb9e62 Merge pull request #2784 from MichaelEischer/minio-sha256-changelog
Add changelog for #2709
2020-06-13 18:33:54 +02:00
Michael Eischer
0b2c31b05b Add changelog for #2709 2020-06-13 18:13:36 +02:00
Alexander Weiss
d3c59d18e5 Fix inconsistency of saving/loading config file
Fix saving/loading config file: Always set ID to a zero ID.
2020-06-13 16:30:23 +02:00
MichaelEischer
dd7b4f54f5 Merge pull request #2709 from greatroar/minio-sha256
Use Minio's optimized SHA-256
2020-06-12 23:32:58 +02:00
MichaelEischer
6896c6449b Merge pull request #2779 from greatroar/archiver-comment
Fix up comment on archiver.BlobSaver.Save
2020-06-12 23:04:21 +02:00
MichaelEischer
735a8074d5 Merge pull request #2773 from aawsome/index-uploads+knownblobs
Fix non-intuitive repo behavior
2020-06-12 22:41:04 +02:00
Alexander Weiss
70347e95d5 disable index uploads for prune command
+ modifications of changelog
2020-06-12 09:24:38 +02:00
greatroar
0fa3091c78 Fix up comment on archiver.BlobSaver.Save 2020-06-11 13:40:30 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
2020-06-11 13:05:23 +02:00
MichaelEischer
fae7f78057 Merge pull request #2755 from greatroar/solaris-workaround
Remove Solaris build workaround in integration tests
2020-06-10 23:30:51 +02:00
MichaelEischer
ac9ec4b990 Merge pull request #2778 from MichaelEischer/fix-unlock-hint
Revive hint to the unlock command if a repository is locked
2020-06-10 21:26:38 +02:00
Michael Eischer
087c770161 Revive hint to the unlock command if a repository is locked
errors.Fatalf wraps a error and just keeps an error message as a string.
This prevents the `restic.IsAlreadyLocked(err)` check from working as
the error is no longer an ErrAlreadyLocked.

Just add an additional remark to the error using `errors.WithMessage`.
2020-06-10 20:58:56 +02:00
MichaelEischer
6856d1e422 Merge pull request #2749 from aawsome/fix-fullindex
Change condition for full index
2020-06-10 20:40:19 +02:00
Alexander Weiss
8c1261ff02 changed condition for full index 2020-06-07 22:00:49 +02:00
rawtaz
26704be17f Merge pull request #2776 from nunoperalta/patch-1
Fix #2774 - make unable to umount message clearer
2020-06-07 17:19:50 +02:00
nunoperalta
2c3360db98 Fix #2774 - make unable to umount message clearer 2020-06-07 15:57:39 +01:00
MichaelEischer
cba6ad8d8e Merge pull request #2711 from greatroar/update-x-net
Update golang.org/x/net for fewer allocations in http2
2020-05-25 20:43:29 +02:00
greatroar
2a3312ac35 Remove Solaris build workaround in integration tests
This was introduced in #1821 because of issues with Go 1.9, but that
compiler version is no longer supported.
2020-05-25 10:33:36 +02:00
greatroar
c35c4e0cbf Update golang.org/x/net for fewer allocations in http2
name                                          old allocs/op  new allocs/op  delta
BackendREST/BenchmarkLoadFile-8                  2.15k ± 0%     2.14k ± 0%  -0.27%  (p=0.002 n=20+20)
BackendREST/BenchmarkLoadPartialFile-8             587 ± 0%       582 ± 0%  -0.88%  (p=0.000 n=20+20)
BackendREST/BenchmarkLoadPartialFileOffset-8       584 ± 0%       579 ± 0%  -0.90%  (p=0.000 n=19+16)

The throughput isn't significantly higher or lower.
2020-05-24 16:39:17 +02:00
MichaelEischer
84475aa3a8 Merge pull request #2730 from greatroar/mount-build-tags
Simplify build tags for restic mount
2020-05-23 20:33:43 +02:00
MichaelEischer
f12f9ae240 Merge pull request #2741 from tbm/typos
Fix typos
2020-05-21 23:40:12 +02:00
Martin Michlmayr
5cc1760fdf Fix typos 2020-05-16 14:05:26 +08:00
MichaelEischer
32ac5486e9 Merge pull request #2732 from restic/rawtaz-update-contributing
Reword parts of Providing Patches in CONTRIBUTING.md
2020-05-12 21:37:08 +02:00
rawtaz
c4336978eb Reword parts of Providing Patches in CONTRIBUTION.md
Primarily makes the request that contributors create a new branch for their changes stand out more.
2020-05-12 14:03:57 +02:00
greatroar
649cbec6c5 Simplify build tags for restic mount
This command can only be built on Darwin, FreeBSD and Linux
(and if we upgrade bazil.org/fuse, only FreeBSD and Linux:
https://github.com/bazil/fuse/issues/224).

Listing the few supported operating systems explicitly here makes
porting restic to new platforms easier.
2020-05-12 11:30:41 +02:00
MichaelEischer
b17bd7f860 Merge pull request #2719 from greatroar/update-x-crypto
Update x/crypto to stop poly1305 segfaulting on ARM
2020-05-09 00:02:18 +02:00
greatroar
68f1e9c524 Update x/crypto to fix poly1305 on ARM
The broken poly1305 implementation for arm64 was removed, fixing the
segfault issue on arm64 (gh-2618). The version for amd64 has improved
performance, which shows up in the internal/repository benchmark:

name              old speed      new speed      delta
SaveAndEncrypt-8   110MB/s ± 2%   113MB/s ± 1%  +2.24%  (p=0.000 n=20+20)
2020-05-05 11:05:35 +02:00
MichaelEischer
1ee2306033 Merge pull request #2717 from MichaelEischer/fix-flaky-password-prompt
backup: Avoid race between password prompt and open repository message
2020-05-01 23:58:55 +02:00
Michael Eischer
c882a92cd6 backup: Avoid race between password prompt and open repository message
`term.Print` sends the output via a channel to a goroutine which
actually prints the message. This may race with the password prompt
printed by `OpenRepository` resulting in a missing prompt.
2020-05-01 23:42:16 +02:00
MichaelEischer
f54db5d796 Merge pull request #2713 from greatroar/unused
Move Index.FindBlob to tests
2020-05-01 20:41:59 +02:00
MichaelEischer
843e7f404e Merge pull request #2716 from MichaelEischer/revert-apfs-test-workaround
Revert "Darwin test fix: allow 1μs timestamp difference"
2020-05-01 18:13:38 +02:00
Michael Eischer
d465b5b9ad Revert "Darwin test fix: allow 1μs timestamp difference"
This reverts commit f3016a9096.
2020-05-01 17:11:30 +02:00
greatroar
9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
greatroar
f97a680887 Fix repository benchmarks
BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of
their time measuring SHA-256 performance in their checks.
2020-04-28 07:57:29 +02:00
greatroar
42a3db05b0 Use Minio's optimized SHA-256
internal/repository benchmarks on an Intel i7-3770k:

name               old speed      new speed       delta
PackerManager-8     209MB/s ± 1%    291MB/s ± 1%  +38.94%  (p=0.008 n=5+5)
SaveAndEncrypt-8    112MB/s ± 1%    135MB/s ± 1%  +20.25%  (p=0.008 n=5+5)
2020-04-28 07:57:18 +02:00
Peter Schultz
90fc639a67 Allow specifying user and host when adding keys
The username and hostname for new keys can be specified with the new
--user and --host flags, respectively. The flags are used only by the
`key add` command and are otherwise ignored.

This allows adding keys with for a desired user and host without having
to run restic as that particular user on that particular host, making
automated key management easier.

Co-authored-by: James TD Smith <ahktenzero@mohorovi.cc>
2020-03-23 13:11:10 +01:00
524 changed files with 26683 additions and 9659 deletions

12
.dockerignore Normal file
View File

@@ -0,0 +1,12 @@
# Folders
.git/
.github/
changelog/
doc/
docker/
helpers/
# Files
.gitignore
.golangci.yml
*.md

View File

@@ -39,8 +39,8 @@ Please describe the feature you'd like us to add here.
-->
What are you trying to do?
--------------------------
What are you trying to do? What problem would this solve?
---------------------------------------------------------
<!--
This section should contain a brief description what you're trying to do, which

View File

@@ -1,40 +1,40 @@
<!--
Thank you very much for contributing code or documentation to restic! Please
fill out the following questions to make it easier for us to review your
changes.
You do not need to check all the boxes below all at once, feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box.
-->
What is the purpose of this change? What does it change?
--------------------------------------------------------
What does this PR change? What problem does it solve?
-----------------------------------------------------
<!--
Describe the changes here, as detailed as needed.
Describe the changes and their purpose here, as detailed as needed.
-->
Was the change discussed in an issue or in the forum before?
------------------------------------------------------------
Was the change previously discussed in an issue or on the forum?
----------------------------------------------------------------
<!--
Link issues and relevant forum posts here.
If this PR resolves an issue on GitHub, use "closes #1234" so that the issue is
closed automatically when this PR is merged.
If this PR resolves an issue on GitHub, use "Closes #1234" so that the issue
is closed automatically when this PR is merged.
-->
Checklist
---------
- [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
- [ ] I have enabled [maintainer edits for this PR](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
- [ ] I have added tests for all changes in this PR
- [ ] I have added documentation for the changes (in the manual)
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/TEMPLATE))
- [ ] I have run `gofmt` on the code in all commits
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits)
- [ ] I'm done, this Pull Request is ready for review
<!--
You do not need to check all the boxes below all at once. Feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box. Enable a checkbox by replacing [ ] with [x].
-->
- [ ] I have read the [contribution guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches).
- [ ] I have [enabled maintainer edits](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
- [ ] I have added tests for all code changes.
- [ ] I have added documentation for relevant changes (in the manual).
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (see [template](https://github.com/restic/restic/blob/master/changelog/TEMPLATE)).
- [ ] I have run `gofmt` on the code in all commits.
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits).
- [ ] I'm done! This pull request is ready for review.

325
.github/workflows/tests.yml vendored Normal file
View File

@@ -0,0 +1,325 @@
name: test
on:
# run tests on push to master, but not when other branches are pushed to
push:
branches:
- master
# run tests for all pull requests
pull_request:
env:
latest_go: "1.18.x"
GO111MODULE: on
jobs:
test:
strategy:
matrix:
# list of jobs to run:
include:
- job_name: Windows
go: 1.18.x
os: windows-latest
install_verb: install
- job_name: macOS
go: 1.18.x
os: macOS-latest
test_fuse: false
install_verb: install
- job_name: Linux
go: 1.18.x
os: ubuntu-latest
test_cloud_backends: true
test_fuse: true
check_changelog: true
install_verb: install
- job_name: Linux
go: 1.17.x
os: ubuntu-latest
test_fuse: true
install_verb: install
- job_name: Linux
go: 1.16.x
os: ubuntu-latest
test_fuse: true
install_verb: get
- job_name: Linux
go: 1.15.x
os: ubuntu-latest
test_fuse: true
install_verb: get
- job_name: Linux
go: 1.14.x
os: ubuntu-latest
test_fuse: true
install_verb: get
name: ${{ matrix.job_name }} Go ${{ matrix.go }}
runs-on: ${{ matrix.os }}
env:
GOPROXY: https://proxy.golang.org
steps:
- name: Set up Go ${{ matrix.go }}
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go }}
- name: Get programs (Linux/macOS)
run: |
echo "build Go tools"
go ${{ matrix.install_verb }} github.com/restic/rest-server/cmd/rest-server@latest
echo "install minio server"
mkdir $HOME/bin
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/darwin-amd64/minio
else
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
fi
chmod 755 $HOME/bin/minio
echo "install rclone"
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-osx-amd64.zip
else
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-linux-amd64.zip
fi
unzip rclone.zip
cp rclone*/rclone $HOME/bin
chmod 755 $HOME/bin/rclone
rm -rf rclone*
# add $HOME/bin to path ($GOBIN was already added to the path by setup-go@v2)
echo $HOME/bin >> $GITHUB_PATH
if: matrix.os == 'ubuntu-latest' || matrix.os == 'macOS-latest'
- name: Get programs (Windows)
shell: powershell
run: |
$ProgressPreference = 'SilentlyContinue'
echo "build Go tools"
go ${{ matrix.install_verb }} github.com/restic/rest-server/...
echo "install minio server"
mkdir $Env:USERPROFILE/bin
Invoke-WebRequest https://dl.minio.io/server/minio/release/windows-amd64/minio.exe -OutFile $Env:USERPROFILE/bin/minio.exe
echo "install rclone"
Invoke-WebRequest https://downloads.rclone.org/rclone-current-windows-amd64.zip -OutFile rclone.zip
unzip rclone.zip
copy rclone*/rclone.exe $Env:USERPROFILE/bin
# add $USERPROFILE/bin to path ($GOBIN was already added to the path by setup-go@v2)
echo $Env:USERPROFILE\bin >> $Env:GITHUB_PATH
echo "install tar"
cd $env:USERPROFILE
mkdir tar
cd tar
# install exactly these versions of tar and the libraries, other combinations might not work!
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/tar-1.13-1-bin.zip -OutFile tar.zip
unzip tar.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libintl-0.11.5-2-bin.zip -OutFile libintl.zip
unzip libintl.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libiconv-1.8-1-bin.zip -OutFile libiconv.zip
unzip libiconv.zip
# add $USERPROFILE/tar/bin to path
echo $Env:USERPROFILE\tar\bin >> $Env:GITHUB_PATH
if: matrix.os == 'windows-latest'
- name: Check out code
uses: actions/checkout@v2
- name: Build with build.go
run: |
go run build.go
- name: Run local Tests
env:
RESTIC_TEST_FUSE: ${{ matrix.test_fuse }}
run: |
go test -cover ./...
- name: Test cloud backends
env:
RESTIC_TEST_S3_KEY: ${{ secrets.RESTIC_TEST_S3_KEY }}
RESTIC_TEST_S3_SECRET: ${{ secrets.RESTIC_TEST_S3_SECRET }}
RESTIC_TEST_S3_REPOSITORY: ${{ secrets.RESTIC_TEST_S3_REPOSITORY }}
RESTIC_TEST_AZURE_ACCOUNT_NAME: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_NAME }}
RESTIC_TEST_AZURE_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_KEY }}
RESTIC_TEST_AZURE_REPOSITORY: ${{ secrets.RESTIC_TEST_AZURE_REPOSITORY }}
RESTIC_TEST_B2_ACCOUNT_ID: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_ID }}
RESTIC_TEST_B2_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_KEY }}
RESTIC_TEST_B2_REPOSITORY: ${{ secrets.RESTIC_TEST_B2_REPOSITORY }}
RESTIC_TEST_GS_REPOSITORY: ${{ secrets.RESTIC_TEST_GS_REPOSITORY }}
RESTIC_TEST_GS_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
GOOGLE_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64: ${{ secrets.RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 }}
RESTIC_TEST_OS_AUTH_URL: ${{ secrets.RESTIC_TEST_OS_AUTH_URL }}
RESTIC_TEST_OS_TENANT_NAME: ${{ secrets.RESTIC_TEST_OS_TENANT_NAME }}
RESTIC_TEST_OS_USERNAME: ${{ secrets.RESTIC_TEST_OS_USERNAME }}
RESTIC_TEST_OS_PASSWORD: ${{ secrets.RESTIC_TEST_OS_PASSWORD }}
RESTIC_TEST_OS_REGION_NAME: ${{ secrets.RESTIC_TEST_OS_REGION_NAME }}
RESTIC_TEST_SWIFT: ${{ secrets.RESTIC_TEST_SWIFT }}
# fail if any of the following tests cannot be run
RESTIC_TEST_DISALLOW_SKIP: "restic/backend/rest.TestBackendREST,\
restic/backend/sftp.TestBackendSFTP,\
restic/backend/s3.TestBackendMinio,\
restic/backend/rclone.TestBackendRclone,\
restic/backend/s3.TestBackendS3,\
restic/backend/swift.TestBackendSwift,\
restic/backend/b2.TestBackendB2,\
restic/backend/gs.TestBackendGS,\
restic/backend/azure.TestBackendAzure"
run: |
# prepare credentials for Google Cloud Storage tests in a temp file
export GOOGLE_APPLICATION_CREDENTIALS=$(mktemp --tmpdir restic-gcs-auth-XXXXXXX)
echo $RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
go test -cover -parallel 4 ./internal/backend/...
# only run cloud backend tests for pull requests from and pushes to our
# own repo, otherwise the secrets are not available
if: (github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository) && matrix.test_cloud_backends
- name: Check changelog files with calens
run: |
echo "install calens"
go install github.com/restic/calens@latest
echo "check changelog files"
calens
if: matrix.check_changelog
cross_compile:
strategy:
# ATTENTION: the list of architectures must be in sync with helpers/build-release-binaries/main.go!
matrix:
# run cross-compile in two batches parallel so the overall tests run faster
targets:
- "linux/386 linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/mips linux/mipsle linux/mips64 linux/mips64le linux/s390x \
openbsd/386 openbsd/amd64"
- "freebsd/386 freebsd/amd64 freebsd/arm \
aix/ppc64 \
darwin/amd64 darwin/arm64 \
netbsd/386 netbsd/amd64 \
windows/386 windows/amd64 \
solaris/amd64"
env:
GOPROXY: https://proxy.golang.org
runs-on: ubuntu-latest
name: Cross Compile for ${{ matrix.targets }}
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v2
with:
go-version: ${{ env.latest_go }}
- name: Install gox
run: |
go install github.com/mitchellh/gox@latest
- name: Check out code
uses: actions/checkout@v2
- name: Cross-compile with gox for ${{ matrix.targets }}
env:
GOFLAGS: "-trimpath"
GOX_ARCHS: "${{ matrix.targets }}"
run: |
mkdir build-output
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}" ./cmd/restic
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -tags debug -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}_debug" ./cmd/restic
lint:
name: lint
runs-on: ubuntu-latest
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v2
with:
go-version: ${{ env.latest_go }}
- name: Check out code
uses: actions/checkout@v2
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.45
# Optional: show only new issues if it's a pull request. The default value is `false`.
only-new-issues: true
args: --verbose --timeout 5m
skip-go-installation: true
# only run golangci-lint for pull requests, otherwise ALL hints get
# reported. We need to slowly address all issues until we can enable
# linting the master branch :)
if: github.event_name == 'pull_request'
- name: Check go.mod/go.sum
run: |
echo "check if go.mod and go.sum are up to date"
go mod tidy
git diff --exit-code go.mod go.sum
docker:
name: docker
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v2
- name: Docker meta
id: meta
uses: docker/metadata-action@v3
with:
# list of Docker images to use as base name for tags
images: |
restic/restic
# generate Docker tags based on the following events/attributes
tags: |
type=schedule
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Build and push
id: docker_build
uses: docker/build-push-action@v2
with:
push: false
context: .
file: docker/Dockerfile
pull: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

1
.gitignore vendored
View File

@@ -1,2 +1,3 @@
/restic
/.vagrant
/.vscode

57
.golangci.yml Normal file
View File

@@ -0,0 +1,57 @@
# This is the configuration for golangci-lint for the restic project.
#
# A sample config with all settings is here:
# https://github.com/golangci/golangci-lint/blob/master/.golangci.example.yml
linters:
# only enable the linters listed below
disable-all: true
enable:
# make sure all errors returned by functions are handled
- errcheck
# find unused code
- deadcode
# show how code can be simplified
- gosimple
# # make sure code is formatted
- gofmt
# examine code and report suspicious constructs, such as Printf calls whose
# arguments do not align with the format string
- govet
# make sure names and comments are used according to the conventions
- golint
# detect when assignments to existing variables are not used
- ineffassign
# run static analysis and find errors
- staticcheck
# find unused variables, functions, structs, types, etc.
- unused
# find unused struct fields
- structcheck
# find unused global variables
- varcheck
# parse and typecheck code
- typecheck
issues:
# don't use the default exclude rules, this hides (among others) ignored
# errors from Close() calls
exclude-use-default: false
# list of things to not warn about
exclude:
# golint: do not warn about missing comments for exported stuff
- exported (function|method|var|type|const) `.*` should have comment or be unexported
# golint: ignore constants in all caps
- don't use ALL_CAPS in Go names; use CamelCase

View File

@@ -1,2 +0,0 @@
go:
enabled: true

View File

@@ -1,66 +0,0 @@
language: go
sudo: false
matrix:
include:
- os: linux
go: "1.11.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: linux
go: "1.12.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: linux
go: "1.13.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
# only run fuse and cloud backends tests on Travis for the latest Go on Linux
- os: linux
go: "1.14.x"
sudo: true
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: osx
go: "1.14.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/Library/Caches/go-build
- $HOME/gopath/pkg/mod
branches:
only:
- master
notifications:
irc:
channels:
- "chat.freenode.net#restic"
on_success: change
on_failure: change
skip_join: true
install:
- go version
- export GOBIN="$GOPATH/bin"
- export PATH="$PATH:$GOBIN"
- go env
script:
- go run run_integration_tests.go

File diff suppressed because it is too large Load Diff

View File

@@ -13,10 +13,10 @@ bug fixes are most welcome. However even "minor" details as fixing spelling
errors, improving documentation or pointing out usability issues are a great
help also.
The restic project uses the GitHub infrastructure (see the
[project page](https://github.com/restic/restic)) for all related discussions
as well as the `#restic` channel on `irc.freenode.net`.
as well as the [forum](https://forum.restic.net/) and the `#restic` channel
on [irc.libera.chat](https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:6697/#restic).
If you want to find an area that currently needs improving have a look at the
open issues listed at the
@@ -25,7 +25,10 @@ for discussing enhancement to the restic tools.
If you are unsure what to do, please have a look at the issues, especially
those tagged
[minor complexity](https://github.com/restic/restic/labels/minor%20complexity).
[minor complexity](https://github.com/restic/restic/labels/help%3A%20minor%20complexity)
or [good first issue](https://github.com/restic/restic/labels/help%3A%20good%20first%20issue).
If you are already a bit experienced with the restic internals, take a look
at the issues tagged as [help wanted](https://github.com/restic/restic/labels/help%3A%20wanted).
Reporting Bugs
@@ -60,16 +63,11 @@ uploading it somewhere or post only the parts that are really relevant.
Development Environment
=======================
The repository contains several sets of directories with code: `cmd/` and
`internal/` contain the code written for restic, whereas `vendor/` contains
copies of libraries restic depends on. The libraries are managed with the
command `go mod vendor`.
The repository contains the code written for restic in the directories
`cmd/` and `internal/`.
Go >= 1.11
----------
For Go version 1.11 or later, you should clone the repo (without having
`$GOPATH` set) and `cd` into the directory:
Restic requires Go version 1.14 or later for compiling. Clone the repo (without
having `$GOPATH` set) and `cd` into the directory:
$ unset GOPATH
$ git clone https://github.com/restic/restic
@@ -79,40 +77,12 @@ Then use the `go` tool to build restic:
$ go build ./cmd/restic
$ ./restic version
restic 0.9.2-dev (compiled manually) compiled with go1.11 on linux/amd64
restic 0.10.0-dev (compiled manually) compiled with go1.15.2 on linux/amd64
You can run all tests with the following command:
$ go test ./...
Go < 1.11
---------
In order to compile restic with Go before 1.11, it needs to be checked out at
the right path within a `GOPATH`. The concept of a `GOPATH` is explained in
["How to write Go code"](https://golang.org/doc/code.html).
If you do not have a directory with Go code yet, executing the following
instructions in your shell will create one for you and check out the restic
repo:
$ export GOPATH="$HOME/go"
$ mkdir -p "$GOPATH/src/github.com/restic"
$ cd "$GOPATH/src/github.com/restic"
$ git clone https://github.com/restic/restic
$ cd restic
You can then build restic as follows:
$ go build ./cmd/restic
$ ./restic version
restic compiled manually
compiled with go1.8.3 on linux/amd64
The following commands can be used to run all the tests:
$ go test ./...
Providing Patches
=================
@@ -125,15 +95,14 @@ down to the following steps:
GitHub. For a new feature, please add an issue before starting to work on
it, so that duplicate work is prevented.
1. First we would kindly ask you to fork our project on GitHub if you haven't
done so already.
1. Next, fork our project on GitHub if you haven't done so already.
2. Clone the repository locally and create a new branch. If you are working on
the code itself, please set up the development environment as described in
the previous section.
2. Clone your fork of the repository locally and **create a new branch** for
your changes. If you are working on the code itself, please set up the
development environment as described in the previous section.
3. Then commit your changes as fine grained as possible, as smaller patches,
that handle one and only one issue are easier to discuss and merge.
3. Commit your changes to the new branch as fine grained as possible, as
smaller patches, for individual changes, are easier to discuss and merge.
4. Push the new branch with your changes to your fork of the repository.
@@ -146,20 +115,22 @@ down to the following steps:
existing commit, use common sense to decide which is better), they will be
automatically added to the pull request.
7. If your pull request changes anything that users should be aware
of (a bugfix, a new feature, ...) please add an entry as a new
file in `changelog/unreleased` including the issue number in the
filename (e.g. `issue-8756`). Use the template in
`changelog/TEMPLATE` for the content. It will be used in the
announcement of the next stable release. While writing, ask
yourself: If I were the user, what would I need to be aware of
with this change.
7. If your pull request changes anything that users should be aware of
(a bugfix, a new feature, ...) please add an entry as a new file in
`changelog/unreleased` including the issue number in the filename (e.g.
`issue-8756`). Use the template in `changelog/TEMPLATE` for the content.
It will be used in the announcement of the next stable release. While
writing, ask yourself: If I were the user, what would I need to be aware
of with this change?
8. Once your code looks good and passes all the tests, we'll merge it. Thanks
8. Do not edit the man pages under `doc/man` or `doc/manual_rest.rst` -
these are autogenerated before new releases.
9. Once your code looks good and passes all the tests, we'll merge it. Thanks
a lot for your contribution!
Please provide the patches for each bug or feature in a separate branch and
open up a pull request for each.
open up a pull request for each, as this simplifies discussion and merging.
The restic project uses the `gofmt` tool for Go source indentation, so please
run
@@ -172,6 +143,14 @@ Installing the script `fmt-check` from https://github.com/edsrzf/gofmt-git-hook
locally as a pre-commit hook checks formatting before committing automatically,
just copy this script to `.git/hooks/pre-commit`.
The project is using the program
[`golangci-lint`](https://github.com/golangci/golangci-lint) to run a list of
linters and checkers. It will be run on the code when you submit a PR. In order
to check your code beforehand, you can run `golangci-lint run` manually.
Eventually, we will enable `golangci-lint` for the whole code base. For now,
you can ignore warnings printed for lines you did not modify, those will be
ignored by the CI.
For each pull request, several different systems run the integration tests on
Linux, macOS and Windows. We won't merge any code that does not pass all tests
for all systems, so when a tests fails, try to find out what's wrong and fix

113
README.md Normal file
View File

@@ -0,0 +1,113 @@
[![Documentation](https://readthedocs.org/projects/restic/badge/?version=latest)](https://restic.readthedocs.io/en/latest/?badge=latest)
[![Build Status](https://github.com/restic/restic/workflows/test/badge.svg)](https://github.com/restic/restic/actions?query=workflow%3Atest)
[![Go Report Card](https://goreportcard.com/badge/github.com/restic/restic)](https://goreportcard.com/report/github.com/restic/restic)
# Introduction
restic is a backup program that is fast, efficient and secure. It supports the three major operating systems (Linux, macOS, Windows) and a few smaller ones (FreeBSD, OpenBSD).
For detailed usage and installation instructions check out the [documentation](https://restic.readthedocs.io/en/latest).
You can ask questions in our [Discourse forum](https://forum.restic.net).
Quick start
-----------
Once you've [installed](https://restic.readthedocs.io/en/latest/020_installation.html) restic, start
off with creating a repository for your backups:
$ restic init --repo /tmp/backup
enter password for new backend:
enter password again:
created restic backend 085b3c76b9 at /tmp/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
and add some data:
$ restic --repo /tmp/backup backup ~/work
enter password for repository:
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
[0:29] 100.00% 54.732 MiB/s 1.582 GiB / 1.582 GiB 2580 / 2580 items 0 errors ETA 0:00
duration: 0:29, 54.47MiB/s
snapshot 40dc1520 saved
Next you can either use `restic restore` to restore files or use `restic
mount` to mount the repository via fuse and browse the files from previous
snapshots.
For more options check out the [online documentation](https://restic.readthedocs.io/en/latest/).
# Backends
Saving a backup on the same machine is nice but not a real backup strategy.
Therefore, restic supports the following backends for storing backups natively:
- [Local directory](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local)
- [sftp server (via SSH)](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp)
- [HTTP REST server](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) ([protocol](https://restic.readthedocs.io/en/latest/100_references.html#rest-backend), [rest-server](https://github.com/restic/rest-server))
- [Amazon S3](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) (either from Amazon or using the [Minio](https://minio.io) server)
- [OpenStack Swift](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift)
- [BackBlaze B2](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2)
- [Microsoft Azure Blob Storage](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage)
- [Google Cloud Storage](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage)
- And many other services via the [rclone](https://rclone.org) [Backend](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone)
# Design Principles
Restic is a program that does backups right and was designed with the
following principles in mind:
- **Easy:** Doing backups should be a frictionless process, otherwise
you might be tempted to skip it. Restic should be easy to configure
and use, so that, in the event of a data loss, you can just restore
it. Likewise, restoring data should not be complicated.
- **Fast**: Backing up your data with restic should only be limited by
your network or hard disk bandwidth so that you can backup your files
every day. Nobody does backups if it takes too much time. Restoring
backups should only transfer data that is needed for the files that
are to be restored, so that this process is also fast.
- **Verifiable**: Much more important than backup is restore, so restic
enables you to easily verify that all data can be restored.
- **Secure**: Restic uses cryptography to guarantee confidentiality and
integrity of your data. The location the backup data is stored is
assumed not to be a trusted environment (e.g. a shared space where
others like system administrators are able to access your backups).
Restic is built to secure your data against such attackers.
- **Efficient**: With the growth of data, additional snapshots should
only take the storage of the actual increment. Even more, duplicate
data should be de-duplicated before it is actually written to the
storage back end to save precious backup space.
# Reproducible Builds
The binaries released with each restic version starting at 0.6.1 are
[reproducible](https://reproducible-builds.org/), which means that you can
reproduce a byte identical version from the source code for that
release. Instructions on how to do that are contained in the
[builder repository](https://github.com/restic/builder).
News
----
You can follow the restic project on Twitter [@resticbackup](https://twitter.com/resticbackup) or by subscribing to
the [project blog](https://restic.net/blog/).
License
-------
Restic is licensed under [BSD 2-Clause License](https://opensource.org/licenses/BSD-2-Clause). You can find the
complete text in [``LICENSE``](LICENSE).
Sponsorship
-----------
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by [AppsCode](https://appscode.com)!
[![Sponsored by AppsCode](https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png)](https://appscode.com)

View File

@@ -1,135 +0,0 @@
|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |Reviewed by Hound|
Introduction
------------
restic is a backup program that is fast, efficient and secure. It supports the three major operating systems (Linux, macOS, Windows) and a few smaller ones (FreeBSD, OpenBSD).
For detailed usage and installation instructions check out the `documentation <https://restic.readthedocs.io/en/latest>`__.
You can ask questions in our `Discourse forum <https://forum.restic.net>`__.
Quick start
-----------
Once you've `installed
<https://restic.readthedocs.io/en/latest/020_installation.html>`__ restic, start
off with creating a repository for your backups:
.. code-block:: console
$ restic init --repo /tmp/backup
enter password for new backend:
enter password again:
created restic backend 085b3c76b9 at /tmp/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
and add some data:
.. code-block:: console
$ restic --repo /tmp/backup backup ~/work
enter password for repository:
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
[0:29] 100.00% 54.732 MiB/s 1.582 GiB / 1.582 GiB 2580 / 2580 items 0 errors ETA 0:00
duration: 0:29, 54.47MiB/s
snapshot 40dc1520 saved
Next you can either use ``restic restore`` to restore files or use ``restic
mount`` to mount the repository via fuse and browse the files from previous
snapshots.
For more options check out the `online documentation <https://restic.readthedocs.io/en/latest/>`__.
Backends
--------
Saving a backup on the same machine is nice but not a real backup strategy.
Therefore, restic supports the following backends for storing backups natively:
- `Local directory <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local>`__
- `sftp server (via SSH) <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp>`__
- `HTTP REST server <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server>`__ (`protocol <doc/100_references.rst#rest-backend>`__ `rest-server <https://github.com/restic/rest-server>`__)
- `AWS S3 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3>`__ (either from Amazon or using the `Minio <https://minio.io>`__ server)
- `OpenStack Swift <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift>`__
- `BackBlaze B2 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2>`__
- `Microsoft Azure Blob Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage>`__
- `Google Cloud Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage>`__
- And many other services via the `rclone <https://rclone.org>`__ `Backend <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone>`__
Design Principles
-----------------
Restic is a program that does backups right and was designed with the
following principles in mind:
- **Easy:** Doing backups should be a frictionless process, otherwise
you might be tempted to skip it. Restic should be easy to configure
and use, so that, in the event of a data loss, you can just restore
it. Likewise, restoring data should not be complicated.
- **Fast**: Backing up your data with restic should only be limited by
your network or hard disk bandwidth so that you can backup your files
every day. Nobody does backups if it takes too much time. Restoring
backups should only transfer data that is needed for the files that
are to be restored, so that this process is also fast.
- **Verifiable**: Much more important than backup is restore, so restic
enables you to easily verify that all data can be restored.
- **Secure**: Restic uses cryptography to guarantee confidentiality and
integrity of your data. The location the backup data is stored is
assumed not to be a trusted environment (e.g. a shared space where
others like system administrators are able to access your backups).
Restic is built to secure your data against such attackers.
- **Efficient**: With the growth of data, additional snapshots should
only take the storage of the actual increment. Even more, duplicate
data should be de-duplicated before it is actually written to the
storage back end to save precious backup space.
Reproducible Builds
-------------------
The binaries released with each restic version starting at 0.6.1 are
`reproducible <https://reproducible-builds.org/>`__, which means that you can
easily reproduce a byte identical version from the source code for that
release. Instructions on how to do that are contained in the
`builder repository <https://github.com/restic/builder>`__.
News
----
You can follow the restic project on Twitter `@resticbackup <https://twitter.com/resticbackup>`__ or by subscribing to
the `development blog <https://restic.net/blog/>`__.
License
-------
Restic is licensed under `BSD 2-Clause License <https://opensource.org/licenses/BSD-2-Clause>`__. You can find the
complete text in ``LICENSE``.
Sponsorship
-----------
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by `AppsCode <https://appscode.com>`__!
|AppsCode|
.. |Documentation| image:: https://readthedocs.org/projects/restic/badge/?version=latest
:target: https://restic.readthedocs.io/en/latest/?badge=latest
.. |Build Status| image:: https://travis-ci.com/restic/restic.svg?branch=master
:target: https://travis-ci.com/restic/restic
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true
:target: https://ci.appveyor.com/project/fd0/restic/branch/master
.. |Report Card| image:: https://goreportcard.com/badge/github.com/restic/restic
:target: https://goreportcard.com/report/github.com/restic/restic
.. |Say Thanks| image:: https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg
:target: https://saythanks.io/to/restic
.. |AppsCode| image:: https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png
:target: https://appscode.com
.. |Reviewed by Hound| image:: https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg
:target: https://houndci.com

View File

@@ -1 +1 @@
0.9.6
0.13.0

View File

@@ -1,32 +0,0 @@
clone_folder: c:\restic
environment:
GOPATH: c:\gopath
branches:
only:
- master
cache:
- '%LocalAppData%\go-build'
init:
- ps: >-
$app = Get-WmiObject -Class Win32_Product -Filter "Vendor = 'http://golang.org'"
if ($app) {
$app.Uninstall()
}
install:
- rmdir c:\go /s /q
- appveyor DownloadFile https://dl.google.com/go/go1.14.windows-amd64.msi
- msiexec /i go1.14.windows-amd64.msi /q
- go version
- go env
- appveyor DownloadFile http://sourceforge.netcologne.de/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip
- 7z x tar.zip bin/tar.exe
- set PATH=bin/;%PATH%
build_script:
- go run run_integration_tests.go

View File

@@ -3,7 +3,7 @@
// This program aims to make building Go programs for end users easier by just
// calling it with `go run`, without having to setup a GOPATH.
//
// This program needs Go >= 1.11. It'll use Go modules for compilation. It
// This program needs Go >= 1.12. It'll use Go modules for compilation. It
// builds the package configured as Main in the Config struct.
// BSD 2-Clause License
@@ -35,6 +35,7 @@
// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//go:build ignore_build_go
// +build ignore_build_go
package main
@@ -58,7 +59,7 @@ var config = Config{
Main: "./cmd/restic", // package name for the main package
DefaultBuildTags: []string{"selfupdate"}, // specify build tags which are always used
Tests: []string{"./..."}, // tests to run
MinVersion: GoVersion{Major: 1, Minor: 11, Patch: 0}, // minimum Go version supported
MinVersion: GoVersion{Major: 1, Minor: 14, Patch: 0}, // minimum Go version supported
}
// Config configures the build.
@@ -123,17 +124,8 @@ func printEnv(env []string) {
// build runs "go build args..." with GOPATH set to gopath.
func build(cwd string, env map[string]string, args ...string) error {
a := []string{"build"}
// try to remove all absolute paths from resulting binary
if goVersion.AtLeast(GoVersion{1, 13, 0}) {
// use the new flag introduced by Go 1.13
a = append(a, "-trimpath")
} else {
// otherwise try to trim as many paths as possible
a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", cwd))
a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", cwd))
}
// -trimpath removes all absolute paths from the binary.
a := []string{"build", "-trimpath"}
if enablePIE {
a = append(a, "-buildmode=pie")
@@ -327,8 +319,8 @@ func (v GoVersion) String() string {
}
func main() {
if !goVersion.AtLeast(GoVersion{1, 11, 0}) {
die("Go version (%v) is too old, Go <= 1.11 does not support Go Modules\n", goVersion)
if !goVersion.AtLeast(GoVersion{1, 12, 0}) {
die("Go version (%v) is too old, restic requires Go >= 1.12\n", goVersion)
}
if !goVersion.AtLeast(config.MinVersion) {

View File

@@ -0,0 +1,8 @@
Enhancement: Optimize `restic mount`
We've optimized the FUSE implementation used within restic.
`restic mount` is now more responsive and uses less memory.
https://github.com/restic/restic/issues/1680
https://github.com/restic/restic/pull/2587
https://github.com/restic/restic/pull/2787

View File

@@ -0,0 +1,6 @@
Bugfix: Report correct number of directories processed by backup
The directory statistics calculation was fixed to report the actual number
of processed directories instead of always zero.
https://github.com/restic/restic/issues/1863

View File

@@ -0,0 +1,9 @@
Enhancement: Allow specifying user and host when creating keys
When adding a new key to the repository, the username and hostname for the new
key can be specified on the command line. This allows overriding the defaults,
for example if you would prefer to use the FQDN to identify the host or if you
want to add keys for several different hosts without having to run the key add
command on those hosts.
https://github.com/restic/restic/issues/2175

View File

@@ -0,0 +1,9 @@
Bugfix: Fix tar issues when dumping `/`
We've fixed an issue with dumping either `/` or files on the first sublevel
e.g. `/foo` to tar. This also fixes tar dumping issues on Windows where this
issue could also happen.
https://github.com/restic/restic/issues/2254
https://github.com/restic/restic/issues/2357
https://github.com/restic/restic/pull/2255

View File

@@ -0,0 +1,12 @@
Enhancement: Ignore sync errors when operation not supported by local filesystem
The local backend has been modified to work with filesystems which doesn't support
the `sync` operation. This operation is normally used by restic to ensure that data
files are fully written to disk before continuing.
For these limited filesystems, saving a file in the backend would previously fail with
an "operation not supported" error. This error is now ignored, which means that e.g.
an SMB mount on macOS can now be used as storage location for a repository.
https://github.com/restic/restic/issues/2395
https://forum.restic.net/t/sync-errors-on-mac-over-smb/1859

View File

@@ -0,0 +1,7 @@
Enhancement: Add flag `--iexclude-file` to backup command
The backup command now supports the flag `--iexclude-file` which is a
case-insensitive version of `--exclude-file`.
https://github.com/restic/restic/issues/2427
https://github.com/restic/restic/pull/2898

View File

@@ -0,0 +1,8 @@
Enhancement: Support excluding files by their size
The `backup` command now supports the `--exclude-larger-than` option to exclude files which are
larger than the specified maximum size. This can for example be useful to exclude unimportant
files with a large file size.
https://github.com/restic/restic/issues/2569
https://github.com/restic/restic/pull/2914

View File

@@ -0,0 +1,16 @@
Enhancement: Self-heal missing file parts during backup of unchanged files
We've improved the resilience of restic to certain types of repository corruption.
For files that are unchanged since the parent snapshot, the backup command now
verifies that all parts of the files still exist in the repository. Parts that are
missing, e.g. from a damaged repository, are backed up again. This verification
was already run for files that were modified since the parent snapshot, but is
now also done for unchanged files.
Note that restic will not backup file parts that are referenced in the index but
where the actual data is not present on disk, as this situation can only be
detected by restic check. Please ensure that you run `restic check` regularly.
https://github.com/restic/restic/issues/2571
https://github.com/restic/restic/pull/2827

View File

@@ -0,0 +1,9 @@
Enhancement: Support filtering snapshots by tag and path in the stats command
We've added filtering snapshots by `--tag tagList` and by `--path path` to
the `stats` command. This includes filtering of only 'latest' snapshots or
all snapshots in a repository.
https://github.com/restic/restic/issues/2858
https://github.com/restic/restic/pull/2859
https://forum.restic.net/t/stats-for-a-host-and-filtered-snapshots/3020

View File

@@ -0,0 +1,20 @@
Enhancement: Add command for copying snapshots between repositories
We've added a copy command, allowing you to copy snapshots from one
repository to another.
Note that this process will have to read (download) and write (upload) the
entire snapshot(s) due to the different encryption keys used on the source
and destination repository. Also, the transferred files are not re-chunked,
which may break deduplication between files already stored in the
destination repo and files copied there using this command.
To fully support deduplication between repositories when the copy command is
used, the init command now supports the `--copy-chunker-params` option,
which initializes the new repository with identical parameters for splitting
files into chunks as an already existing repository. This allows copied
snapshots to be equally deduplicated in both repositories.
https://github.com/restic/restic/issues/323
https://github.com/restic/restic/pull/2606
https://github.com/restic/restic/pull/2928

View File

@@ -0,0 +1,10 @@
Enhancement: Use optimized library for hash calculation of file chunks
We've switched the library used to calculate the hashes of file chunks, which
are used for deduplication, to the optimized Minio SHA-256 implementation.
Depending on the CPU it improves the hashing throughput by 10-30%. Modern x86
CPUs with the SHA Extension should be about two to three times faster.
https://github.com/restic/restic/issues/551
https://github.com/restic/restic/pull/2709

View File

@@ -14,4 +14,10 @@ file can be written to the file before any of the preceeding file blobs.
It is therefore possible to have gaps in the data written to the target
files if restore fails or interrupted by the user.
The implementation will try to preallocate space for the restored files
on the filesystem to prevent file fragmentation. This ensures good read
performance for large files, like for example VM images. If preallocating
space is not supported by the filesystem, then this step is silently skipped.
https://github.com/restic/restic/pull/2195
https://github.com/restic/restic/pull/2893

View File

@@ -0,0 +1,8 @@
Enhancement: Improve speed of check command
We've improved the check command to traverse trees only once independent of
whether they are contained in multiple snapshots. The check command is now much
faster for repositories with a large number of snapshots.
https://github.com/restic/restic/pull/2328
https://github.com/restic/restic/issues/2284

View File

@@ -0,0 +1,19 @@
Change: Return exit code 3 when failing to backup all source data
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).
This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:
- 0 when the command was successful
- 1 when there was a fatal error (no snapshot created)
- 3 when some source data could not be read (incomplete snapshot created)
https://github.com/restic/restic/pull/2546
https://github.com/restic/restic/issues/956
https://github.com/restic/restic/issues/2064
https://github.com/restic/restic/issues/2526
https://github.com/restic/restic/issues/2364

View File

@@ -1,7 +1,9 @@
Enhancement: Improve the chunking algorithm
We've updated the chunker library responsible for splitting files into smaller
blocks. It should improve the chunking throughput by 5-10% depending on the
blocks. It should improve the chunking throughput by 5-15% depending on the
CPU.
https://github.com/restic/restic/pull/2576
https://github.com/restic/restic/pull/2845
https://github.com/restic/restic/issues/2820

View File

@@ -0,0 +1,6 @@
Enhancement: Improve speed of diff command
We've improved the performance of the diff command when comparing snapshots
with similar content. It should run up to twice as fast as before.
https://github.com/restic/restic/pull/2598

View File

@@ -0,0 +1,6 @@
Enhancement: Slightly reduce memory usage of prune and stats commands
The prune and the stats command kept directory identifiers in memory twice
while searching for used blobs.
https://github.com/restic/restic/pull/2599

View File

@@ -0,0 +1,14 @@
Change: Update dependencies, require Go >= 1.13
Restic now requires Go to be at least 1.13. This allows simplifications in the
build process and removing workarounds.
This is also probably the last version of restic still supporting mounting
repositories via fuse on macOS. The library we're using for fuse does not
support macOS any more and osxfuse is not open source any more.
https://github.com/restic/restic/pull/2600
https://github.com/restic/restic/pull/2852
https://github.com/restic/restic/pull/2927
https://github.com/bazil/fuse/issues/224
https://github.com/osxfuse/osxfuse/issues/590

View File

@@ -0,0 +1,8 @@
Bugfix: Add stricter prune error checks
Additional checks were added to the prune command in order to improve
resiliency to backend, hardware and/or networking issues. The checks now
detect a few more cases where such outside factors could potentially cause
data loss.
https://github.com/restic/restic/pull/2674

View File

@@ -0,0 +1,6 @@
Enhancement: S3 backend: Add support for WebIdentityTokenFile
We've added support for EKS IAM roles for service accounts feature to the S3 backend.
https://github.com/restic/restic/pull/2733
https://github.com/restic/restic/issues/2703

View File

@@ -0,0 +1,6 @@
Enhancement: Optimize handling of new index entries
Restic now uses less memory for backups which add a lot of data, e.g. large initial backups.
In addition, we've improved the stability in some edge cases.
https://github.com/restic/restic/pull/2773

View File

@@ -0,0 +1,8 @@
Enhancement: Reduce memory consumption of in-memory index
We've improved how the index is stored in memory.
This change can reduce memory usage for large repositories by up to 50%
(depending on the operation).
https://github.com/restic/restic/pull/2781
https://github.com/restic/restic/pull/2812

View File

@@ -0,0 +1,6 @@
Enhancement: Optimize `list blobs` command
We've changed the implementation of `list blobs` which should be now a bit faster
and consume almost no memory even for large repositories.
https://github.com/restic/restic/pull/2786

View File

@@ -0,0 +1,6 @@
Enhancement: Optimized file access in restic mount
Reading large (> 100GiB) files from restic mountpoints is now faster,
and the speedup is greater for larger files.
https://github.com/restic/restic/pull/2790

View File

@@ -0,0 +1,8 @@
Change: Honor the --no-lock flag in the mount command
The mount command now does not lock the repository if given the
--no-lock flag. This allows to mount repositories which are archived
on a read only backend/filesystem.
https://github.com/restic/restic/issues/1597
https://github.com/restic/restic/pull/2821

View File

@@ -0,0 +1,7 @@
Enhancement: Speed-up file deletion in forget, prune and rebuild-index
We've sped up the file deletion for the commands forget, prune and
rebuild-index, especially for remote repositories.
Deletion was sequential before and is now run in parallel.
https://github.com/restic/restic/pull/2840

View File

@@ -0,0 +1,9 @@
Bugfix: Fix possible crash in the progress bar of check --read-data
We've fixed a possible crash while displaying the progress bar for the
check --read-data command. The crash occurred when the length of the
progress bar status exceeded the terminal width, which only happened for
very narrow terminal windows.
https://github.com/restic/restic/pull/2899
https://forum.restic.net/t/restic-rclone-pcloud-connection-issues/2963/15

View File

@@ -0,0 +1,11 @@
Bugfix: Restore timestamps and permissions on intermediate directories
When using the `--include` option of the restore command, restic restored
timestamps and permissions only on directories selected by the include pattern.
Intermediate directories, which are necessary to restore files located in sub-
directories, were created with default permissions. We've fixed the restore
command to restore timestamps and permissions for these directories as well.
https://github.com/restic/restic/issues/1212
https://github.com/restic/restic/issues/1402
https://github.com/restic/restic/pull/2906

View File

@@ -0,0 +1,15 @@
Bugfix: Mark repository files as read-only when using the local backend
Files stored in a local repository were marked as writeable on the
filesystem for non-Windows systems, which did not prevent accidental file
modifications outside of restic. In addition, the local backend did not work
with certain filesystems and network mounts which do not permit modifications
of file permissions.
restic now marks files stored in a local repository as read-only on the
filesystem on non-Windows systems. The error handling is improved to support
more filesystems.
https://github.com/restic/restic/issues/1756
https://github.com/restic/restic/issues/2157
https://github.com/restic/restic/pull/2989

View File

@@ -0,0 +1,10 @@
Bugfix: Hide password in REST backend repository URLs
When using a password in the REST backend repository URL,
the password could in some cases be included in the output
from restic, e.g. when initializing a repo or during an error.
The password is now replaced with "***" where applicable.
https://github.com/restic/restic/issues/2241
https://github.com/restic/restic/pull/2658

View File

@@ -0,0 +1,12 @@
Bugfix: Correctly dump directories into tar files
The dump command previously wrote directories in a tar file in a way which
can cause compatibility problems. This caused, for example, 7zip on Windows
to not open tar files containing directories. In addition it was not possible
to dump directories with extended attributes. These compatibility problems
are now corrected.
In addition, a tar file now includes the name of the owner and group of a file.
https://github.com/restic/restic/issues/2319
https://github.com/restic/restic/pull/3039

View File

@@ -0,0 +1,9 @@
Bugfix: Don't require `self-update --output` placeholder file
`restic self-update --output /path/to/new-restic` used to require that
new-restic was an existing file, to be overwritten. Now it's possible
to download an updated restic binary to a new path, without first
having to create a placeholder file.
https://github.com/restic/restic/issues/2491
https://github.com/restic/restic/pull/2937

View File

@@ -0,0 +1,7 @@
Bugfix: Fix rare cases of backup command hanging forever
We've fixed an issue with the backup progress reporting which could cause
restic to hang forever right before finishing a backup.
https://github.com/restic/restic/issues/2834
https://github.com/restic/restic/pull/2963

View File

@@ -0,0 +1,6 @@
Bugfix: Fix manpage formatting
The manpage formatting in restic v0.10.0 was garbled, which is fixed now.
https://github.com/restic/restic/issues/2938
https://github.com/restic/restic/pull/2977

View File

@@ -0,0 +1,7 @@
Bugfix: Make --exclude-larger-than handle disappearing files
There was a small bug in the backup command's --exclude-larger-than
option where files that disappeared between scanning and actually
backing them up to the repository caused a panic. This is now fixed.
https://github.com/restic/restic/issues/2942

View File

@@ -0,0 +1,9 @@
Bugfix: restic generate, help and self-update no longer check passwords
The commands `restic cache`, `generate`, `help` and `self-update` don't need
passwords, but they previously did run the RESTIC_PASSWORD_COMMAND (if set in
the environment), prompting users to authenticate for no reason. They now skip
running the password command.
https://github.com/restic/restic/issues/2951
https://github.com/restic/restic/pull/2987

View File

@@ -0,0 +1,9 @@
Enhancement: Optimize check for unchanged files during backup
During a backup restic skips processing files which have not changed since the last backup run.
Previously this required opening each file once which can be slow on network filesystems. The
backup command now checks for file changes before opening a file. This considerably reduces
the time to create a backup on network filesystems.
https://github.com/restic/restic/issues/2969
https://github.com/restic/restic/pull/2970

View File

@@ -0,0 +1,9 @@
Bugfix: Make snapshots --json output [] instead of null when no snapshots
Restic previously output `null` instead of `[]` for the `--json snapshots`
command, when there were no snapshots in the repository. This caused some
minor problems when parsing the output, but is now fixed such that `[]` is
output when the list of snapshots is empty.
https://github.com/restic/restic/issues/2979
https://github.com/restic/restic/pull/2984

View File

@@ -0,0 +1,12 @@
Enhancement: Add support for Volume Shadow Copy Service (VSS) on Windows
Volume Shadow Copy Service allows read access to files that are locked by
another process using an exclusive lock through a filesystem snapshot. Restic
was unable to backup those files before. This update enables backing up these
files.
This needs to be enabled explicitely using the --use-fs-snapshot option of the
backup command.
https://github.com/restic/restic/issues/340
https://github.com/restic/restic/pull/2274

View File

@@ -0,0 +1,7 @@
Enhancement: Authenticate to Google Cloud Storage with access token
When using the GCS backend, it is now possible to authenticate with OAuth2
access tokens instead of a credentials file by setting the GOOGLE_ACCESS_TOKEN
environment variable.
https://github.com/restic/restic/pull/2849

View File

@@ -0,0 +1,10 @@
Enhancement: New option --repository-file
We've added a new command-line option --repository-file as an alternative
to -r. This allows to read the repository URL from a file in order to
prevent certain types of information leaks, especially for URLs containing
credentials.
https://github.com/restic/restic/issues/1458
https://github.com/restic/restic/issues/2900
https://github.com/restic/restic/pull/2910

View File

@@ -0,0 +1,8 @@
Enhancement: Warn if parent snapshot cannot be loaded during backup
During a backup restic uses the parent snapshot to check whether a file was
changed and has to be backed up again. For this check the backup has to read
the directories contained in the old snapshot. If a tree blob cannot be
loaded, restic now warns about this problem with the backup repository.
https://github.com/restic/restic/pull/2978

View File

@@ -0,0 +1,10 @@
Bugfix: Make `mount` not create missing mount point directory
When specifying a non-existent directory as mount point for the `mount`
command, restic used to create the specified directory automatically.
This has now changed such that restic instead gives an error when the
specified directory for the mount point does not exist.
https://github.com/restic/restic/issues/1681
https://github.com/restic/restic/pull/3008

View File

@@ -0,0 +1,8 @@
Bugfix: Ignore `no data available` filesystem error during backup
Restic was unable to backup files on some filesystems, for example certain
configurations of CIFS on Linux which return a `no data available` error
when reading extended attributes. These errors are now ignored.
https://github.com/restic/restic/issues/1800
https://github.com/restic/restic/pull/3034

View File

@@ -0,0 +1,8 @@
Enhancement: Allow specifying percentage in `check --read-data-subset`
We've enhanced the `check` command's `--read-data-subset` option to also accept
a percentage (e.g. `2.5%` or `10%`). This will check the given percentage of
pack files (which are randomly selected on each run).
https://github.com/restic/restic/issues/2186
https://github.com/restic/restic/pull/3038

View File

@@ -0,0 +1,14 @@
Enhancement: Report permanent/fatal backend errors earlier
When encountering errors in reading from or writing to storage backends,
restic retries the failing operation up to nine times (for a total of ten
attempts). It used to retry all backend operations, but now detects some
permanent error conditions so that it can report fatal errors earlier.
Permanent failures include local disks being full, SSH connections
dropping and permission errors.
https://github.com/restic/restic/issues/2453
https://github.com/restic/restic/pull/3170
https://github.com/restic/restic/issues/3180
https://github.com/restic/restic/pull/3181

View File

@@ -0,0 +1,21 @@
Enhancement: Add Alibaba/Aliyun OSS support in the `s3` backend
A new extended option `s3.bucket-lookup` has been added to support
Alibaba/Aliyun OSS in the `s3` backend. The option can be set to one
of the following values:
- `auto` - Existing behaviour
- `dns` - Use DNS style bucket access
- `path` - Use path style bucket access
To make the `s3` backend work with Alibaba/Aliyun OSS you must set
`s3.bucket-lookup` to `dns` and set the `s3.region` parameter. For
example:
restic -o s3.bucket-lookup=dns -o s3.region=oss-eu-west-1 -r s3:https://oss-eu-west-1.aliyuncs.com/bucketname init
Note that `s3.region` must be set, otherwise the MinIO SDK tries to
look it up and it seems that Alibaba doesn't support that properly.
https://github.com/restic/restic/issues/2528
https://github.com/restic/restic/pull/2535

View File

@@ -0,0 +1,9 @@
Bugfix: Report the correct owner of directories in FUSE mounts
Restic 0.10.0 changed the FUSE mount to always report the current user
as the owner of directories within the FUSE mount, which is incorrect.
This is now changed back to reporting the correct owner of a directory.
https://github.com/restic/restic/issues/2563
https://github.com/restic/restic/pull/3141

View File

@@ -0,0 +1,31 @@
Bugfix: Make `backup` and `tag` commands separate tags by comma
Running `restic backup --tag foo,bar` previously created snapshots with one
single tag containing a comma (`foo,bar`) instead of two tags (`foo`, `bar`).
Similarly, the `tag` command's `--set`, `--add` and `--remove` options would
treat `foo,bar` as one tag instead of two tags. This was inconsistent with
other commands and often unexpected when one intended `foo,bar` to mean two
tags.
To be consistent in all commands, restic now interprets `foo,bar` to mean two
separate tags (`foo` and `bar`) instead of one tag (`foo,bar`) everywhere,
including in the `backup` and `tag` commands.
NOTE: This change might result in unexpected behavior in cases where you use
the `forget` command and filter on tags like `foo,bar`. Snapshots previously
backed up with `--tag foo,bar` will still not match that filter, but snapshots
saved from now on will match that filter.
To replace `foo,bar` tags with `foo` and `bar` tags in old snapshots, you can
first generate a list of the relevant snapshots using a command like:
restic snapshots --json --quiet | jq '.[] | select(contains({tags: ["foo,bar"]})) | .id'
and then use `restic tag --set foo --set bar snapshotID [...]` to set the new
tags. Please adjust the commands to include real tag names and any additional
tags, as well as the list of snapshots to process.
https://github.com/restic/restic/issues/2688
https://github.com/restic/restic/pull/2690
https://github.com/restic/restic/pull/3197

View File

@@ -0,0 +1,17 @@
Enhancement: Configurable progress reports for non-interactive terminals
The `backup`, `check` and `prune` commands never printed any progress
reports on non-interactive terminals. This behavior is now configurable
using the `RESTIC_PROGRESS_FPS` environment variable. Use for example a
value of `1` for an update every second, or `0.01666` for an update every
minute.
The `backup` command now also prints the current progress when restic
receives a `SIGUSR1` signal.
Setting the `RESTIC_PROGRESS_FPS` environment variable or sending a `SIGUSR1`
signal prints a status report even when `--quiet` was specified.
https://github.com/restic/restic/issues/2706
https://github.com/restic/restic/issues/3194
https://github.com/restic/restic/pull/3199

View File

@@ -0,0 +1,5 @@
Bugfix: Make the `cat` command respect the `--no-lock` option
The `cat` command would not respect the `--no-lock` flag. This is now fixed.
https://github.com/restic/restic/issues/2739

View File

@@ -0,0 +1,13 @@
Enhancement: Add `backup` options `--files-from-{verbatim,raw}`
The new `backup` options `--files-from-verbatim` and `--files-from-raw` read a
list of files to back up from a file. Unlike the existing `--files-from`
option, these options do not interpret the listed filenames as glob patterns;
instead, whitespace in filenames is preserved as-is and no pattern expansion is
done. Please see the documentation for specifics.
These new options are highly recommended over `--files-from`, when using a
script to generate the list of files to back up.
https://github.com/restic/restic/issues/2944
https://github.com/restic/restic/issues/3013

View File

@@ -0,0 +1,18 @@
Enhancement: Allow usage of deprecated S3 `ListObjects` API
Some S3 API implementations, e.g. Ceph before version 14.2.5, have a broken
`ListObjectsV2` implementation which causes problems for restic when using
their API endpoints. When a broken server implementation is used, restic prints
errors similar to the following:
List() returned error: Truncated response should have continuation token set
As a temporary workaround, restic now allows using the older `ListObjects`
endpoint by setting the `s3.list-objects-v1` extended option, for instance:
restic -o s3.list-objects-v1=true snapshots
Please note that this option may be removed in future versions of restic.
https://github.com/restic/restic/issues/3083
https://github.com/restic/restic/pull/3085

View File

@@ -0,0 +1,10 @@
Bugfix: The `--use-fs-snapshot` option now works on windows/386
Restic failed to create VSS snapshots on windows/386 with the following error:
GetSnapshotProperties() failed: E_INVALIDARG (0x80070057)
This is now fixed.
https://github.com/restic/restic/issues/3087
https://github.com/restic/restic/pull/3090

View File

@@ -0,0 +1,17 @@
Change: Deleting files on Google Drive now moves them to the trash
When deleting files on Google Drive via the `rclone` backend, restic used to
bypass the trash folder required that one used the `-o rclone.args` option to
enable usage of the trash folder. This ensured that deleted files in Google
Drive were not kept indefinitely in the trash folder. However, since Google
Drive's trash retention policy changed to deleting trashed files after 30 days,
this is no longer needed.
Restic now leaves it up to rclone and its configuration to use or not use the
trash folder when deleting files. The default is to use the trash folder, as
of rclone 1.53.2. To re-enable the restic 0.11 behavior, set the
`RCLONE_DRIVE_USE_TRASH` environment variable or change the rclone
configuration. See the rclone documentation for more details.
https://github.com/restic/restic/issues/3095
https://github.com/restic/restic/pull/3102

View File

@@ -0,0 +1,10 @@
Bugfix: Do not require gs bucket permissions when running `init`
Restic used to require bucket level permissions for the `gs` backend
in order to initialize a restic repository.
It now allows a `gs` service account to initialize a repository if the
bucket does exist and the service account has permissions to write/read
to that bucket.
https://github.com/restic/restic/issues/3100

View File

@@ -0,0 +1,9 @@
Bugfix: Correctly detect output redirection for `backup` command on Windows
On Windows, since restic 0.10.0 the `backup` command did not properly detect
when the output was redirected to a file. This caused restic to output
terminal control characters. This has been fixed by correcting the terminal
detection.
https://github.com/restic/restic/issues/3111
https://github.com/restic/restic/pull/3150

View File

@@ -0,0 +1,11 @@
Enhancement: Support additional environment variables for Swift authentication
The `swift` backend now supports the following additional environment variables
for passing authentication details to restic:
`OS_USER_ID`, `OS_USER_DOMAIN_ID`, `OS_PROJECT_DOMAIN_ID` and `OS_TRUST_ID`
Depending on the `openrc` configuration file these might be required when the
user and project domains differ from one another.
https://github.com/restic/restic/issues/3147
https://github.com/restic/restic/pull/3158

View File

@@ -0,0 +1,9 @@
Bugfix: Don't create invalid snapshots when `backup` is interrupted
When canceling a backup run at a certain moment it was possible that
restic created a snapshot with an invalid "null" tree. This caused
`check` and other operations to fail. The `backup` command now properly
handles interruptions and never saves a snapshot when interrupted.
https://github.com/restic/restic/issues/3151
https://github.com/restic/restic/pull/3164

View File

@@ -0,0 +1,9 @@
Bugfix: Improve error handling in the `restore` command
The `restore` command used to not print errors while downloading file contents
from the repository. It also incorrectly exited with a zero error code even
when there were errors during the restore process. This has all been fixed and
`restore` now returns with a non-zero exit code when there's an error.
https://github.com/restic/restic/issues/3166
https://github.com/restic/restic/pull/3207

View File

@@ -0,0 +1,8 @@
Enhancement: Add release binaries for MIPS architectures
We've added a few new architectures for Linux to the release binaries: `mips`,
`mipsle`, `mips64`, and `mip64le`. MIPS is mostly used for low-end embedded
systems.
https://github.com/restic/restic/issues/3191
https://github.com/restic/restic/pull/3208

View File

@@ -0,0 +1,11 @@
Bugfix: Correct statistics for overlapping targets
A user reported that restic's statistics and progress information during backup
was not correctly calculated when the backup targets (files/dirs to save)
overlap. For example, consider a directory `foo` which contains (among others)
a file `foo/bar`. When `restic backup foo foo/bar` was run, restic counted the
size of the file `foo/bar` twice, so the completeness percentage as well as the
number of files was wrong. This is now corrected.
https://github.com/restic/restic/issues/3232
https://github.com/restic/restic/pull/3243

View File

@@ -0,0 +1,12 @@
Enhancement: Back up mountpoints as empty directories
When the `--one-file-system` option is specified to `restic backup`, it
ignores all file systems mounted below one of the target directories. This
means that when a snapshot is restored, users needed to manually recreate
the mountpoint directories.
Restic now backs up mountpoints as empty directories and therefore implements
the same approach as `tar`.
https://github.com/restic/restic/issues/909
https://github.com/restic/restic/pull/3119

View File

@@ -0,0 +1,6 @@
Enhancement: Add several more error checks
We've added a lot more error checks in places where errors were previously
ignored (as hinted by the static analysis program `errcheck` via `golangci-lint`).
https://github.com/restic/restic/pull/3250

View File

@@ -0,0 +1,29 @@
Enhancement: Improve `prune` performance and make it more customizable
The `prune` command is now much faster. This is especially the case for remote
repositories or repositories with not much data to remove. Also the memory
usage of the `prune` command is now reduced.
Restic used to rebuild the index from scratch after pruning. This could lead
to missing packs in the index in some cases for eventually consistent backends
such as e.g. AWS S3. This behavior is now changed and the index rebuilding
uses the information already known by `prune`.
By default, the `prune` command no longer removes all unused data. This
behavior can be fine-tuned by new options, like the acceptable amount of
unused space or the maximum size of data to reorganize. For more details,
please see https://restic.readthedocs.io/en/stable/060_forget.html .
Moreover, `prune` now accepts the `--dry-run` option and also running
`forget --dry-run --prune` will show what `prune` would do.
This enhancement also fixes several open issues, e.g.:
- https://github.com/restic/restic/issues/1140
- https://github.com/restic/restic/issues/1599
- https://github.com/restic/restic/issues/1985
- https://github.com/restic/restic/issues/2112
- https://github.com/restic/restic/issues/2227
- https://github.com/restic/restic/issues/2305
https://github.com/restic/restic/pull/2718
https://github.com/restic/restic/pull/2842

View File

@@ -0,0 +1,27 @@
Enhancement: Add option to let `backup` trust mtime without checking ctime
The `backup` command used to require that both `ctime` and `mtime` of a file
matched with a previously backed up version to determine that the file was
unchanged. In other words, if either `ctime` or `mtime` of the file had
changed, it would be considered changed and restic would read the file's
content again to back up the relevant (changed) parts of it.
The new option `--ignore-ctime` makes restic look at `mtime` only, such that
`ctime` changes for a file does not cause restic to read the file's contents
again.
The check for both `ctime` and `mtime` was introduced in restic 0.9.6 to make
backups more reliable in the face of programs that reset `mtime` (some Unix
archivers do that), but it turned out to often be expensive because it made
restic read file contents even if only the metadata (owner, permissions) of
a file had changed. The new `--ignore-ctime` option lets the user restore the
0.9.5 behavior when needed. The existing `--ignore-inode` option already
turned off this behavior, but also removed a different check.
Please note that changes in files' metadata are still recorded, regardless of
the command line options provided to the backup command.
https://github.com/restic/restic/issues/2495
https://github.com/restic/restic/issues/2558
https://github.com/restic/restic/issues/2819
https://github.com/restic/restic/pull/2823

View File

@@ -0,0 +1,8 @@
Enhancement: Speed up the repacking step of the `prune` command
The repack step of the `prune` command, which moves still used file parts into
new pack files such that the old ones can be garbage collected later on, now
processes multiple pack files in parallel. This is especially beneficial for
high latency backends or when using a fast network connection.
https://github.com/restic/restic/pull/2941

View File

@@ -0,0 +1,11 @@
Enhancement: Speed up the `rebuild-index` command
We've optimized the `rebuild-index` command. Now, existing index entries are used
to minimize the number of pack files that must be read. This speeds up the index
rebuild a lot.
Additionally, the option `--read-all-packs` has been added, implementing the
previous behavior.
https://github.com/restic/restic/issue/2547
https://github.com/restic/restic/pull/3006

Some files were not shown because too many files have changed in this diff Show More