Compare commits

...

1049 Commits

Author SHA1 Message Date
Michael Eischer
e1bb2129ad report result of delete operation after failed upload 2021-03-08 21:54:46 +01:00
Michael Eischer
95b7f8dd81 s3: Fix sanity check
The sanity check shouldn't replace the error message if there is already
one.
2021-03-08 21:54:46 +01:00
Michael Eischer
29e39e247a stat file after retrying an upload 2021-03-08 21:54:46 +01:00
Alexander Neumann
27f241334e Add version for 0.12.0 2021-02-14 11:44:20 +01:00
Alexander Neumann
4e99a3d650 Update manpages and auto-completion 2021-02-14 11:44:20 +01:00
Alexander Neumann
1cb1cd6f44 Generate CHANGELOG.md for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann
1a34260cf0 Prepare changelog for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann
13d52c88fb Merge pull request #3282 from restic/changelog-unreleased
changelog: Correct and slightly polish some unreleased changelog files
2021-02-14 10:49:33 +01:00
Leo R. Lundgren
4b5ca1e914 changelog: Correct and slightly polish some unreleased changelog files 2021-02-14 00:57:54 +01:00
MichaelEischer
917f5b910a Merge pull request #3270 from restic/fix-ls-json
ls: Check for non-nil error before calling panic()
2021-02-09 22:05:04 +01:00
Alexander Neumann
c0f2c1d871 ls: Check for non-nil error before calling panic() 2021-02-07 21:12:54 +01:00
Alexander Neumann
9985368d46 Merge pull request #3255 from MichaelEischer/restorer-check-error
restorer: Check dropped error
2021-02-03 16:10:46 +01:00
Alexander Neumann
2dd592a06c Merge pull request #3254 from lorenz/allow-http2
Allow HTTP/2
2021-02-02 21:08:20 +01:00
Lorenz Brun
362338dd60 Add changelog entry 2021-02-01 20:20:34 +01:00
MichaelEischer
6ac032be64 Merge pull request #3256 from aawsome/rebuild-index-bar-done
rebuild-index: Add missing bar.Done()
2021-01-31 18:48:14 +01:00
MichaelEischer
0ce05d5725 Merge pull request #3236 from Wouter0100/patch-1
doc: Note only path-style URL support for S3
2021-01-31 18:44:58 +01:00
Michael Eischer
0aed8d47d7 docs: Fix typo in S3 URL support 2021-01-31 18:33:15 +01:00
Alexander Weiss
39a26066f7 rebuild-index: add missing bar.Done() 2021-01-31 18:28:02 +01:00
Michael Eischer
47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
MichaelEischer
b3dc127af5 Merge pull request #3207 from aawsome/filerestorer-fix-errorhandling
Fix error handling in filerestorer and fix #3166
2021-01-31 17:47:46 +01:00
Wouter van Os
8442c43209 doc: Update wording for S3 path-only requirement 2021-01-31 15:10:03 +01:00
Alexander Weiss
6e942693ba Fix #3166 2021-01-31 14:22:57 +01:00
Alexander Weiss
5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss
573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
MichaelEischer
b8550a21f2 Merge pull request #3253 from restic/improve-prune-error
prune: Improve error message for missing files
2021-01-31 12:03:20 +01:00
Alexander Neumann
027a51529d prune: Improve error message for missing files
This commit changes the error message so that a list of file names is
printed. Before, just the raw map was printed, which is not a great user
interface.
2021-01-31 11:31:27 +01:00
Lorenz Brun
5427119205 Allow HTTP/2 2021-01-31 02:44:30 +01:00
Alexander Neumann
f647614e24 Merge pull request #3248 from MichaelEischer/backend-cleanups
Backend code and test cleanups
2021-01-30 21:37:33 +01:00
Michael Eischer
e0867c9682 backend: try to cleanup test leftovers 2021-01-30 21:23:20 +01:00
Michael Eischer
f740b2fb23 mem: check upload length before storing upload 2021-01-30 21:23:20 +01:00
Alexander Neumann
0e5f2fff71 Merge pull request #3243 from restic/fix-scanner-overlap
backup: Fix total size for overlapping targets
2021-01-30 21:17:21 +01:00
MichaelEischer
99228be623 Merge pull request #3250 from restic/add-golangci-lint-config
Add golangci lint config, add many error checks
2021-01-30 21:09:38 +01:00
Alexander Neumann
04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann
f867e65bcd Fix issues reported by staticcheck 2021-01-30 20:43:53 +01:00
Alexander Neumann
a00e27adf6 Add entry to changelog 2021-01-30 20:22:25 +01:00
Alexander Neumann
0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
Alexander Neumann
aef3658a5f Address review comments 2021-01-30 20:02:37 +01:00
Alexander Neumann
200f09522d Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann
cbd88c457a backup: Improve error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann
1a0eb05bfa errcheck: Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann
3c753c071c errcheck: More error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann
16313bfcc9 errcheck: Add error check for MergeFinalIndexes() 2021-01-30 20:02:37 +01:00
Alexander Neumann
75f53955ee errcheck: Add error checks
Most added checks are straight forward.
2021-01-30 20:02:37 +01:00
Alexander Neumann
1632a84e7b Add a section explaining golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann
b3d5bf7c99 Update golangci-lint version 2021-01-30 20:02:37 +01:00
Alexander Neumann
57627a307f Add config for golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann
6ab7d49a03 Merge pull request #3251 from MichaelEischer/rest-dropped-error
rest: handle dropped error in save operation
2021-01-30 19:44:50 +01:00
Michael Eischer
a53778cd83 rest: handle dropped error in save operation 2021-01-30 19:25:04 +01:00
Alexander Neumann
dd94efb307 Merge pull request #3249 from MichaelEischer/fix-dropped-gs-error
gs: Don't drop error when finishing upload
2021-01-30 16:24:57 +01:00
Michael Eischer
8a486eafed gs: Don't drop error when finishing upload
The error returned when finishing the upload of an object was dropped.
This could cause silent upload failures and thus data loss in certain
cases. When a MD5 hash for the uploaded blob is specified, a wrong
hash/damaged upload would return its error via the Close() whose error
was dropped.
2021-01-30 13:31:32 +01:00
MichaelEischer
4d576c2f79 Merge pull request #3217 from M4a1x/read-data-subset-fix
Fix missing rand seed for restic check --read-data-subset=x%
2021-01-29 23:34:13 +01:00
Max Stabel
f9e1fa26ff Fix missing rand seed for restic check --read-data-subset=x% 2021-01-29 23:18:35 +01:00
MichaelEischer
fb3cf3f885 Merge pull request #3245 from aawsome/prune-fix-statistics-cacheable
prune: Fix statistics for --repack-cacheable-only
2021-01-29 23:14:49 +01:00
Alexander Weiss
e08e65dc30 prune: Simplify logic selecting packs to repack 2021-01-29 22:27:22 +01:00
Alexander Weiss
daeb4cdf8f prune: Fix statistics for --repack-cacheable-only 2021-01-29 22:27:22 +01:00
Alexander Neumann
cdd704920d azure: Pass data length to Azure libray
The azureAdapter was used directly without a pointer, but the Len()
method was only defined with a pointer receiver (which means Len() is
not present on a azureAdapter{}, only on a pointer to it).
2021-01-29 21:08:41 +01:00
Alexander Neumann
bbdf18c4a2 Merge pull request #3176 from MichaelEischer/backend-content-length
Pass upload size to backends and sanity check it
2021-01-29 20:33:44 +01:00
Michael Eischer
1f583b3d8e backend: test that incomplete uploads fail 2021-01-29 13:51:53 +01:00
Michael Eischer
c73316a111 backends: add sanity check for the uploaded file size
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.

Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
2021-01-29 13:51:51 +01:00
Michael Eischer
4526d5d197 swift: explicitly pass upload size to library
This allows properly setting the content-length which could help the
server-side to detect incomplete uploads.
2021-01-29 13:50:46 +01:00
Michael Eischer
dca9b6f5db azure: explicitly pass upload size
Previously the fallback from the azure library was to read the whole
blob into memory and use that to determine the upload size.
2021-01-29 13:50:46 +01:00
Alexander Neumann
a16ce65295 Merge pull request #3244 from MichaelEischer/better-damage-reports
Print more details about possible repository damages
2021-01-29 11:45:45 +01:00
Alexander Neumann
5c41120c70 Add entry to changelog 2021-01-29 11:31:36 +01:00
Alexander Neumann
5c617859ab backup/scanner: Fix total size for overlapping targets
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.

This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
2021-01-29 11:31:36 +01:00
Alexander Neumann
81211750ba archiver/tree: Introduce functions Leaf() and NodeNames() 2021-01-29 11:11:28 +01:00
rawtaz
de7e3a0648 Merge pull request #2823 from greatroar/trust-mtime
Add --ignore-ctime flag to backup and document change detection
2021-01-29 00:02:19 +01:00
greatroar
6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
Michael Eischer
58b5679f14 prune: reword missing blobs error
The previous wording could be understood such that the prune run did
damage the repository.
2021-01-28 21:48:24 +01:00
Michael Eischer
7b8886c052 prune: report missing but unneeded pack files
This indicates a damaged repository so add some output to help with
debugging.
2021-01-28 21:46:01 +01:00
Michael Eischer
ff95999246 rebuild-index: report added/removed/reindexed files
This should help with investigating missing pack files.
2021-01-28 21:46:01 +01:00
Michael Eischer
b71c52797a find: correctly expand multiple blob ids
For example `restic find --show-pack-id --blob f78dc991 5b9e4366 ddd8c7d4`
would previously only expand one blob if all of them belong to the same
file.
2021-01-28 21:21:54 +01:00
MichaelEischer
82140967d3 Merge pull request #3228 from aawsome/prune-all-trees
prune: Remove all unused trees
2021-01-28 21:04:27 +01:00
MichaelEischer
43cb26010a Merge pull request #3242 from greatroar/fprintln
internal/ui/termstatus: Use Fprintln to get a newline
2021-01-28 20:34:06 +01:00
MichaelEischer
35033d9b79 Merge pull request #3177 from MichaelEischer/fix-2759
prune: don't print stacktrace on console
2021-01-28 20:28:45 +01:00
Alexander Neumann
84822d44d4 Merge pull request #2536 from MatthewVance/threat-model
Update threat model
2021-01-28 14:26:03 +01:00
Matthew Vance
58c7f4694d Update threat model 2021-01-28 14:20:42 +01:00
Alexander Neumann
4d40c70214 Merge pull request #3211 from MichaelEischer/sftp-speedup
Speed-up caching/pack download via SFTP
2021-01-28 14:16:55 +01:00
Alexander Neumann
44169d0dc4 Merge pull request #3205 from MichaelEischer/fix-quiet-verbose
Properly check that --quiet and --verbose are not combined
2021-01-28 13:53:31 +01:00
Alexander Neumann
6aa7e9f9c6 Merge pull request #3174 from MichaelEischer/parallelize-lock-loading
Parallelize lock file loading
2021-01-28 13:52:12 +01:00
Alexander Neumann
bdfedf1f5b Merge pull request #3173 from MichaelEischer/unify-index-loading
Unify index loading
2021-01-28 13:50:42 +01:00
greatroar
b9cfe6f68a internal/ui/termstatus: Use Fprintln to get a newline 2021-01-28 13:30:10 +01:00
Alexander Neumann
72eec8c0c4 Merge pull request #3106 from MichaelEischer/parallel-tree-walk
Parallelize tree walk in prune and copy and add progress bar to check
2021-01-28 12:06:42 +01:00
Michael Eischer
68608a89ad restic: add comment about StreamTrees shutdown 2021-01-28 11:10:50 +01:00
Michael Eischer
1e306be000 Add changelog entry 2021-01-28 11:10:50 +01:00
Michael Eischer
ddb7697d29 restic: Test progress reporting of StreamTrees 2021-01-28 11:10:50 +01:00
Michael Eischer
313ad0e32f progress/counter: Fix test for final report call 2021-01-28 11:10:50 +01:00
Michael Eischer
e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer
505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer
eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer
258ce0c1e5 parallel: report progress for StreamTrees
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
2021-01-28 11:08:43 +01:00
Michael Eischer
3d6a3e2555 copy: Remove treeCloner struct 2021-01-28 11:08:43 +01:00
Michael Eischer
0caad1e890 copy: parallelize tree walk 2021-01-28 11:08:43 +01:00
Michael Eischer
f2a1b125cb restic: Actually parallelize FindUsedBlobs 2021-01-28 11:08:43 +01:00
Michael Eischer
6e03f80ca2 check: Split the parallelized tree loader into a reusable component
The actual code change is minimal
2021-01-28 11:08:43 +01:00
Michael Eischer
1d7bb01a6b check: Cleanup tree loading and switch to use errgroup
The helper methods are now wired up in the Structure method.
2021-01-28 11:08:43 +01:00
Alexander Neumann
a4689eb3b9 Merge pull request #3199 from MichaelEischer/non-interactive-counter
Don't print progress on non-interactive terminals
2021-01-28 10:53:38 +01:00
Alexander Neumann
c5a66e9181 ui: Simlify channel receive 2021-01-28 10:42:02 +01:00
Wouter van Os
b5972f184c doc: Note only path-style URL support for S3
This adds a node to mention that currently only path-style URL's are supported for S3, as per code at:
- aa0faa8c7d/internal/backend/s3/config.go (L42-L45)
- aa0faa8c7d/internal/backend/s3/config.go (L48-L62)
2021-01-23 16:54:08 +01:00
Alexander Weiss
d7dc19a496 prune: Always repack packs containing tree blobs 2021-01-15 16:42:04 +01:00
Michael Eischer
f3442ce8a5 Test that WriteTo of a backend's Load remains accessible 2021-01-03 22:23:53 +01:00
Michael Eischer
678e75e1c2 sftp: enforce use of optimized upload method
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
2021-01-03 22:23:53 +01:00
Michael Eischer
6b5b29dbee limiter: add unit tests 2021-01-03 22:23:53 +01:00
Michael Eischer
f35f2c48cd limiter: support WriteTo in LimitBackend for read rate limiting 2021-01-03 22:23:53 +01:00
Michael Eischer
bcb852a8d0 hashing: support WriteTo in the reader 2021-01-03 22:23:53 +01:00
MichaelEischer
aa0faa8c7d Merge pull request #3208 from restic/add-mips
Add mips* architectures to CI and release
2021-01-03 16:06:10 +01:00
MichaelEischer
f7ec263a22 Merge pull request #3109 from aawsome/optimize-filerestorer
restore: Don't save pack content in memory
2021-01-03 14:53:41 +01:00
Alexander Neumann
7d665fa1f4 Add entry to changelog 2021-01-03 14:41:11 +01:00
Michael Eischer
69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Neumann
36db248e30 Split cross compilation targets into two jobs 2021-01-03 10:50:54 +01:00
Alexander Neumann
eb72b10f55 Add mips* architectures to CI and release 2021-01-03 10:41:54 +01:00
Alexander Neumann
622f4c7daa Run cloud backends also on pushes 2021-01-03 10:38:10 +01:00
Alexander Neumann
f8c50394d6 Revert "Run cloud tests if secrets are available"
This reverts commit aa648bdcac.
2021-01-03 10:28:01 +01:00
Alexander Neumann
aa648bdcac Run cloud tests if secrets are available 2021-01-03 10:26:55 +01:00
Alexander Neumann
e8abc79ce9 Add hints for keeping the list of architectures in sync 2021-01-01 10:09:04 +01:00
Alexander Weiss
34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss
7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss
07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss
3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Michael Eischer
97388b3504 Properly check that --quiet and --verbose are not combined
If --verbose is specified once, then globalOptions.Verbose == 1.
Previously --quiet --verbose would silently ignore the --verbose flag.
2020-12-30 21:24:18 +01:00
Alexander Neumann
8b84c96d9d Merge pull request #3204 from MichaelEischer/archiver-tomb-race
archiver: fix race condition during worker startup
2020-12-30 20:04:20 +01:00
Michael Eischer
debc4a3a99 archiver: fix race condition during worker startup
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.

To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
2020-12-30 17:31:22 +01:00
MichaelEischer
e1efc193e1 Merge pull request #3139 from aawsome/prune-healing
prune: Add healing of repository in some situations
2020-12-29 22:17:27 +01:00
Alexander Weiss
f0113139ea prune: Correct error message 2020-12-29 20:20:05 +01:00
Alexander Weiss
f6df94a50e prune: Add self-healing
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
2020-12-29 20:20:05 +01:00
MichaelEischer
31e56f1ad5 Merge pull request #3197 from SkYNewZ/fix/3183
Fix tag handling for multiple tag lists
2020-12-29 18:44:38 +01:00
MichaelEischer
7fda2f2ad8 Merge pull request #3134 from greatroar/unlock-warn
Warn when unlock fails instead of returning an error
2020-12-29 18:30:08 +01:00
greatroar
dec5008369 Warn when unlock fails instead of returning an error
Only one caller was checking the error.
2020-12-29 17:48:20 +01:00
Quentin Lemaire
873505ed3b Update related changelog 2020-12-29 17:12:46 +01:00
Quentin Lemaire
25ecf9eafb fix(cmd_tag): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Quentin Lemaire
e88f3fb80c fix(cmd_backup): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Alexander Neumann
b2efa0af39 Merge pull request #3164 from MichaelEischer/improve-context-cancel
Improve context cancel handling in archiver and backends
2020-12-29 17:03:42 +01:00
Michael Eischer
25f4acdaa8 Add changelog entry 2020-12-29 16:32:18 +01:00
Michael Eischer
cff4955a48 ui: remove dead struct member 2020-12-29 16:32:18 +01:00
Michael Eischer
05a987b07c Support values less than 1 for RESTIC_PROGRESS_FPS
For example set the variable to 0.016666 to print the progress once per
minute.
2020-12-29 16:32:18 +01:00
Michael Eischer
92da5168e1 ui: force backup progress update on signal 2020-12-29 16:32:18 +01:00
Michael Eischer
34afc93ddc ui/progress: extract signal handling into own package 2020-12-29 16:32:18 +01:00
Michael Eischer
023eea6463 ui: don't shorten non-interactive progress output 2020-12-29 16:32:18 +01:00
Michael Eischer
684600cf42 ui: update status for the backup command on non-interactive terminals
Allow the backup command to print status on non-interactive terminals.
The output is disabled by setting a MinUpdatePause == 0.
2020-12-29 16:03:43 +01:00
Michael Eischer
85fe5feadb Unify progress report frequency calculation 2020-12-29 16:03:43 +01:00
Michael Eischer
969141b5e9 Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals
This makes it possible to use the environment variable to also get
regular progress updates on non-interactive terminals.
2020-12-29 16:03:43 +01:00
Michael Eischer
13ce981794 ui: cleanup backup status shutdown 2020-12-29 16:03:43 +01:00
Michael Eischer
c2ef049f1b ui/progress: don't print progress on non-interactive terminals
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
2020-12-29 16:03:43 +01:00
MichaelEischer
a488d4c847 Merge pull request #2833 from greatroar/aix
AIX port
2020-12-29 12:32:50 +01:00
Alexander Neumann
4133b1ea65 Synchronize OS and architectures for testing 2020-12-29 11:11:50 +01:00
Alexander Neumann
46d2ca5095 Remove old test script 2020-12-29 11:02:48 +01:00
Quentin Lemaire
334d8ce724 feat(tags): Create Flatten() method 2020-12-29 10:59:46 +01:00
Alexander Neumann
c661518df9 Merge pull request #2793 from greatroar/rclone-doc
Clarify rclone-over-SSH docs
2020-12-29 10:46:34 +01:00
Michael Eischer
0d81f16343 Add AIX as cross-compile target to CI 2020-12-29 01:35:01 +01:00
greatroar
3b09ae9074 AIX port 2020-12-29 01:35:01 +01:00
greatroar
18531e3d6f Portability fixes to internal/restic
syscall.Mknod is not available on AIX.
2020-12-29 01:35:01 +01:00
Michael Eischer
ca07317815 add changelog 2020-12-28 21:06:47 +01:00
Michael Eischer
d0ca8fb0b8 backend: test that a canceled context prevents RetryBackend operations 2020-12-28 21:06:47 +01:00
Michael Eischer
08b7f2b58d archiver: test that context canceled error is not dropped 2020-12-28 21:06:47 +01:00
Michael Eischer
e483b63c40 retrybackend: Fail operations when context is already canceled
Depending on the used backend, operations started with a canceled
context may fail or not. For example the local backend still works in
large parts when called with a canceled context. Backends transfering
data via http don't work. It is also not possible to retry failed
operations in that state as the RetryBackend will abort with a 'context
canceled' error.

Ensure uniform behavior of all backends by checking for a canceled
context by checking for a canceled context as a first step in the
RetryBackend. This ensures uniform behavior across all backends, as
backends are always wrapped in a RetryBackend.
2020-12-28 21:06:47 +01:00
Michael Eischer
fc60b560ba archiver: Let saveTree report a canceled context as an error
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
2020-12-28 21:06:47 +01:00
Michael Eischer
736e964317 archiver: Don't loose error if background context is canceled
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
2020-12-28 21:06:47 +01:00
MichaelEischer
9c41e4a343 Merge pull request #3192 from DRON-666/fix-options
Fix formatting of `restic options` command
2020-12-23 22:53:40 +01:00
DRON-666
332b1896d1 Some options fixes
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
2020-12-23 23:26:04 +03:00
MichaelEischer
cb6b0f6255 Merge pull request #3181 from tWido/feature-no-retry-permission
Don't retry when "Permission denied" occurs in local backend
2020-12-23 20:24:29 +01:00
MichaelEischer
1e73aac610 Merge pull request #3179 from aawsome/check-filesize
check: Remove filesize counter
2020-12-23 20:22:02 +01:00
Alexander Weiss
2a1add7538 check: remove file size counter 2020-12-23 02:34:31 +01:00
tWido
7dab113035 Don't retry when "Permission denied" occurs in local backend 2020-12-22 23:41:12 +01:00
MichaelEischer
8efb874f48 Merge pull request #3148 from aawsome/simplify-index-rebuild
rebuild-index: code simplification
2020-12-22 23:27:53 +01:00
Michael Eischer
de99207046 repository: tweak comment for packs method 2020-12-22 23:01:58 +01:00
Alexander Weiss
d8d2cc6dd9 Simplify rebuild-index code 2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer
b9f5d3fe13 repository: Add test for ForAllIndexes 2020-12-22 22:36:18 +01:00
Michael Eischer
a12c5f1d37 repository: move otherwise unused LoadIndex to tests 2020-12-22 22:36:18 +01:00
Michael Eischer
24474a36f4 repository: deduplicate index loading implementation 2020-12-22 22:36:18 +01:00
Michael Eischer
ccc84af73d debug/list: parallelize index loading 2020-12-22 22:36:18 +01:00
Michael Eischer
96904f8972 check: extract parallel index loading 2020-12-22 22:36:18 +01:00
MichaelEischer
69f9d269eb Merge pull request #3189 from restic/fix-counter-segfault
ui/progress: Use mutex instead of atomic
2020-12-22 22:14:08 +01:00
rawtaz
ec59c73489 Merge pull request #3188 from restic/forget-prune-dry-run
forget: Enable --dry-run together with --prune
2020-12-22 21:09:15 +01:00
Alexander Neumann
6c514adb8a ui/progress: Use mutex instead of atomic
The counter value needs to be aligned to 64 bit in memory for the
atomic functions to work on some platform (such as 32 bit ARM).

The atomic package says in its documentation:

> These functions require great care to be used correctly. Except for
> special, low-level applications, synchronization is better done with
> channels or the facilities of the sync package.

This commit replaces the atomic functions with a simple sync.Mutex, so
we don't have to care about alignment.
2020-12-22 21:03:27 +01:00
Alexander Neumann
edf89e1c74 forget: Enable --dry-run together with --prune 2020-12-22 20:58:02 +01:00
Alexander Neumann
f7c7c2f730 Merge pull request #3175 from MichaelEischer/fix-3099-changelog
Fix changelog for #3099
2020-12-20 11:29:57 +01:00
Michael Eischer
cfea79d0c5 prune: don't print stacktrace on console 2020-12-19 14:28:48 +01:00
Michael Eischer
5cd40f8b58 Add changelog 2020-12-19 11:38:38 +01:00
MichaelEischer
d32949ee54 Merge pull request #3081 from DRON-666/dump-zip
Add zip support to the `dump` command
2020-12-19 11:33:33 +01:00
DRON-666
83b10dbb12 Deduplicate dumper closing logic 2020-12-19 02:42:46 +03:00
DRON-666
e136dd8696 Deduplicate test code 2020-12-19 02:06:54 +03:00
DRON-666
33adb58817 Minor fixes and linter suggestions 2020-12-19 02:04:17 +03:00
DRON-666
da9053b184 Some gramma fixes in documentation 2020-12-19 01:16:15 +03:00
DRON-666
ef1aeb8724 dump: Update docs and changelog 2020-12-19 01:09:47 +03:00
DRON-666
2ca76afc2b dump: Add new option --archive 2020-12-19 01:09:47 +03:00
DRON-666
89ab6d557e dump: Add zip dumper 2020-12-19 01:09:47 +03:00
DRON-666
0256f95994 dump: Split tar and walk logic 2020-12-19 01:09:47 +03:00
Michael Eischer
bfadc82a20 Fix changelog for #3099 2020-12-18 20:58:24 +01:00
Michael Eischer
34b6130a0e restic: parallelize lock file loading 2020-12-18 20:46:16 +01:00
MichaelEischer
22260d130d Merge pull request #3170 from greatroar/dont-retry
Don't retry permanent errors in backends
2020-12-17 23:29:56 +01:00
greatroar
9341a83b05 Changelog entry for #2453 and #3170 2020-12-17 23:15:48 +01:00
greatroar
66d904c905 Make invalid handles permanent errors 2020-12-17 12:47:53 +01:00
greatroar
746dbda413 Mark "ssh exited" errors in SFTP as permanent 2020-12-17 12:43:09 +01:00
greatroar
f7784bddb3 Don't retry when "no space left on device" in local backend
Also adds relevant documentation to the restic.Backend interface.
2020-12-17 12:43:09 +01:00
Alexander Neumann
1cdd38d9e0 Merge pull request #3165 from restic/2747-doc-forget
doc: Clarify calendar boundaries for --keep-* options
2020-12-16 08:38:03 +01:00
Leo R. Lundgren
b3c0d2f45b doc: Clarify calendar boundaries for --keep-* options 2020-12-15 12:43:06 +01:00
Alexander Neumann
e96677cafb Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables
swift: Add support for id based keystone v3 auth parameters
2020-12-12 16:27:38 +01:00
Michael Eischer
1d69341e88 swift: Add support for id based keystone v3 auth parameters
This adds support for the following environment variables, which were
previously missing:

OS_USER_ID            User ID for keystone v3 authentication
OS_USER_DOMAIN_ID     User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID  Project domain ID for keystone v3 authentication
OS_TRUST_ID           Trust ID for keystone v3 authentication
2020-12-11 19:22:34 +01:00
Alexander Neumann
36c5d39c2c Fix issues reported by semgrep 2020-12-11 09:41:59 +01:00
Alexander Neumann
7facc8ccc1 Merge pull request #2505 from aawsome/fix-repo-configfile
Fix repo configfile
2020-12-07 07:52:37 +01:00
Alexander Neumann
ba31c6fdaa Merge pull request #3150 from MichaelEischer/fix-windows-redir-output
termstatus: Fix canUpdateStatus detection for redirected output on windows
2020-12-06 21:17:27 +01:00
Alexander Neumann
b58799d83a Merge pull request #3152 from MichaelEischer/fix-backup-background-hang
backup: Fix shutdown hang when running in the background on linux
2020-12-06 21:16:00 +01:00
Alexander Neumann
0d5b764f90 Merge pull request #3130 from aawsome/snapshots-parallel
Make loading snapshots parallel
2020-12-06 21:08:18 +01:00
Alexander Weiss
d6b3859e48 Add changelog 2020-12-06 19:29:18 +01:00
Michael Eischer
b48f579530 termstatus: Fix canUpdateStatus detection for redirected output
The canUpdateStatus check was simplified in #2608, but it accidentally flipped
the condition. The correct check is as follows: If the output is a pipe then
restic probably runs in mintty/cygwin. In that case it's possible to
update the output status. In all other cases it isn't.

This commit inverts to condition again to offer the previous and correct
behavior.
2020-12-06 19:02:42 +01:00
Michael Eischer
401ef92c5f backup: Fix shutdown hang when running in the background
On shutdown the backup commands waits for the terminal output goroutine
to stop. However while running in the background the goroutine ignored
the canceled context.
2020-12-06 18:53:41 +01:00
Alexander Weiss
e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss
26f85779be Parallelize ForAllSnapshots 2020-12-06 05:09:58 +01:00
Alexander Weiss
5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
MichaelEischer
3264eae9f6 Merge pull request #3149 from aawsome/fix-rebuild-index
Bugfix for rebuild-index
2020-12-05 22:18:22 +01:00
Alexander Weiss
83c8a9b058 Bugfix: packSizeFromList should save size from List() 2020-12-05 20:58:36 +01:00
Alexander Neumann
43cf301450 Merge pull request #3141 from MichaelEischer/fix-fuse-dir-owner
fuse: Properly set uid/gid for directories
2020-12-01 16:06:24 +01:00
Michael Eischer
d05c88a5d6 fuse: Properly set uid/gid for directories
In #2584 this was changed to use the uid/gid of the root node. This
would be okay for the top-level directory of a snapshot, however, this
change also applied to normal directories within a snapshot. This
change reverts the problematic part and adds a test that directory
attributes are represented correctly.
2020-11-30 23:41:49 +01:00
Alexander Neumann
058b102db0 Merge pull request #3138 from MichaelEischer/fix-debug-build 2020-11-30 12:27:51 +01:00
rawtaz
fcebc7d250 Merge pull request #3135 from restic/manual
doc: Fix misc missing/incorrect text in manual
2020-11-29 19:20:51 +01:00
MichaelEischer
f2959127b6 Merge pull request #3065 from greatroar/local-subdirs
Don't recurse in local backend's List if not required
2020-11-29 19:03:59 +01:00
Leo R. Lundgren
61460dee52 doc: Fix misc missing/incorrect text in manual 2020-11-29 18:59:24 +01:00
Michael Eischer
54a6d98945 Enable debug builds for CI 2020-11-29 18:47:00 +01:00
Michael Eischer
f72f6c9c80 Fix debug build 2020-11-29 18:44:36 +01:00
MichaelEischer
52b98f7f95 Merge pull request #3017 from greatroar/files-from0
Add backup options --files-from-verbatim and --files-from-raw
2020-11-29 18:15:21 +01:00
Alexander Neumann
04d856e601 Merge pull request #3136 from restic/rawtaz-copy-doc
doc: Emphasize double transfer and duplication in copy command
2020-11-29 13:57:10 +01:00
Alexander Neumann
a7b49c4889 Merge pull request #3119 from restic/keep-mountpoints
Keep mountpoints as empty directories for --one-file-system
2020-11-29 11:22:12 +01:00
rawtaz
a568211b98 Merge pull request #3128 from vrenaville/tzdata
[FIX] Timezone in docker image
2020-11-28 22:43:18 +01:00
Leo R. Lundgren
f70b10d0ee doc: Emphasize double transfer and duplication in copy command 2020-11-28 19:49:34 +01:00
greatroar
55bf76ba0c backup: Add --files-from-{verbatim,raw} options 2020-11-28 18:22:31 +01:00
Alexander Neumann
162117c42c Add changelog 2020-11-28 17:00:31 +01:00
Alexander Neumann
82ae942965 backup: Keep mountpoints for --one-file-system
When a file system is mounted at a directory, lstat() returns attributes
of the root node of the mounted file system, including the device ID of
the other file system. The previous code used when --one-file-system is
specified excluded the directory itself because of that.

This commit changes the code so that mountpoints are kept as empty
directories, its attributes set to the root note of the mounted file
system. The behavior mimics `tar`, which does the same.
2020-11-28 17:00:31 +01:00
Alexander Neumann
f576d3d826 Add tests 2020-11-28 17:00:31 +01:00
Alexander Neumann
037f0a4c91 Refactor device ID checking 2020-11-28 17:00:31 +01:00
Alexander Neumann
2f9346a5af Merge pull request #3125 from metalsp0rk/cat-respect-no-lock
Make restic cat respect --no-lock
2020-11-28 12:45:59 +01:00
Alexander Neumann
a3105799c9 Print warning when unlocking the repo fails 2020-11-28 11:31:25 +01:00
Kyle Brennan
666768cd17 cat: Respect --no-lock flag 2020-11-28 11:30:02 +01:00
vrenaville
adc7a6555f [FIX] Timezone in docker image 2020-11-27 08:05:19 +01:00
Alexander Neumann
9a97095a4c Merge pull request #3120 from aawsome/blob-implementation
Blob implementation
2020-11-22 20:59:30 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss
e3013271a6 Harmonize naming 2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Neumann
c844580e0f Merge pull request #3101 from aawsome/packsizes
Compute packsizes in MasterIndex
2020-11-22 15:49:19 +01:00
Alexander Weiss
67c938f232 Use PackSize in prune 2020-11-21 22:13:54 +01:00
Alexander Weiss
a851c53cbe Use PackSize in checker 2020-11-21 22:13:54 +01:00
Alexander Weiss
4960b841e6 Use PackSize in rebuild-index 2020-11-21 22:13:54 +01:00
Alexander Weiss
ce5d630681 Add MasterIndex.PackSize() 2020-11-21 22:13:54 +01:00
Alexander Weiss
c3ddde9e7d Return hdrSize in ListPack 2020-11-21 22:13:54 +01:00
rawtaz
cac481634c Merge pull request #3115 from johanbove/patch-1
Docs: Update 04_backup.rst
2020-11-20 11:23:03 +01:00
Johan Bové
c23b1a4cba Update 04_backup.rst
Fixed typo - _files_ are included from _folders_, not other _files_.
2020-11-20 07:52:23 +01:00
Alexander Neumann
110a32a08b Merge pull request #3113 from aawsome/fix-prune-stats-duplicates
Fix statistics in prune for duplicates
2020-11-19 20:34:11 +01:00
greatroar
8e213e82fc backend/local: replace fs.Walk with custom walker
This code is more strict in what it expects to find in the backend:
depending on the layout, either a directory full of files or a directory
full of such directories.
2020-11-19 16:46:42 +01:00
Alexander Neumann
8a150ee91f Merge pull request #3112 from MichaelEischer/reduce-progress-log-spam
Limit progress bar updates to once per second on non-terminal outputs
2020-11-19 07:54:36 +01:00
Alexander Weiss
cb5ec7ea6b Fix statistics in prune for duplicates
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.

If not all duplicates are marked for repacking, we lack the
information which

The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
2020-11-18 22:30:22 +01:00
Michael Eischer
625410f003 Limit progress bar updates to once per second on non-terminal outputs
The code accidentally checked whether stdin is a terminal instead of
stdout, the former is not relevant here as the output is printed on
stdout.
2020-11-18 22:12:07 +01:00
Alexander Neumann
75eff92b56 Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init
do not require gs bucket permissions to init repository
2020-11-18 16:53:45 +01:00
eleith
a24e986b2b do not require gs bucket permissions to init repository
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.

these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.

this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.

this way, restic can still initialize a repository if the service
account does have object permissions

fixes: https://github.com/restic/restic/issues/3100
2020-11-18 06:14:11 -08:00
rawtaz
6822ce8479 Merge pull request #3105 from restic/fix-init-repo-file
Allow using --repository-file in init
2020-11-17 21:54:44 +01:00
Alexander Neumann
d857fb6e59 Allow using --repository-file in init 2020-11-17 20:43:46 +01:00
rawtaz
342520b648 Merge pull request #3103 from greatroar/rephrase-pr3102
Rephrase #3095/#3102 changelog entry
2020-11-17 12:52:12 +01:00
greatroar
028f2b8c0e Rephrase #3095/#3102 changelog entry 2020-11-17 12:37:06 +01:00
rawtaz
1b6e8c888f Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param
Remove `--drive-use-trash=false` from rclone param
2020-11-17 11:21:20 +01:00
Alexander Neumann
5f3b802ee7 Merge pull request #3099 from MichaelEischer/check-less-memory 2020-11-16 10:11:25 +01:00
Michael Eischer
022dc35be9 Add changelog 2020-11-15 19:02:51 +01:00
Michael Eischer
1f43cac12d check: Only track data blobs when unused blobs should be reported
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
Michael Eischer
6da66c15d8 check: Simplify referenced blob tracking
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
2020-11-15 18:42:55 +01:00
Michael Eischer
3500f9490c check: Simplify blob status tracking
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
Michael Eischer
b8c7543a55 check: Merge 'size could not be found' and 'not found in index' errors
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.

The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
2020-11-15 18:41:50 +01:00
MichaelEischer
45ba456291 Merge pull request #3038 from fgma/check-data-subset-percentage
Check data subset: check random percentage subset
2020-11-15 18:24:13 +01:00
Michael Eischer
caac38ed27 check: Tweak subset percentage over 100% error message 2020-11-15 18:13:50 +01:00
Michael Eischer
15c537f9db Rename changelog entry to contain issue id 2020-11-15 18:13:50 +01:00
fgma
8f9cea8cc0 Check data subset: check random percentage subset 2020-11-15 18:13:50 +01:00
Alexander Neumann
3c0c0c132b Merge pull request #3006 from aawsome/new-rebuild-index
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
Alexander Neumann
9968220652 Merge pull request #2850 from greatroar/packer-malloc
Decrease allocation rate in internal/pack
2020-11-15 17:19:49 +01:00
kitone
0649828555 Add changelog 2020-11-15 17:09:30 +01:00
kitone
3c03b35212 Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error 2020-11-15 17:09:30 +01:00
greatroar
ab2b7d7f9a Decrease allocation rate in internal/pack
internal/repository benchmark results:

name             old time/op    new time/op    delta
PackerManager-8     179ms ± 1%     181ms ± 1%   +0.78%  (p=0.009 n=10+10)

name             old speed      new speed      delta
PackerManager-8   294MB/s ± 1%   292MB/s ± 1%   -0.77%  (p=0.009 n=10+10)

name             old alloc/op   new alloc/op   delta
PackerManager-8    91.3kB ± 0%    72.2kB ± 0%  -20.92%  (p=0.000 n=9+7)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.38k ± 0%     0.76k ± 0%  -45.20%  (p=0.000 n=10+7)
2020-11-15 16:51:47 +01:00
greatroar
9a8a2cae4c Move pack testing logic to test file 2020-11-15 16:45:49 +01:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss
3d1d5295cc Add changelog 2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss
187c8fb259 Parallelize MasterIndex.Save() 2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
Alexander Weiss
5898cb341f Use CreateIndexFromPacks() in test 2020-11-15 07:05:05 +01:00
Alexander Weiss
43732bb885 Add CreateIndexFromPacks() 2020-11-15 07:04:51 +01:00
MichaelEischer
6feaf6bd1f Merge pull request #3030 from greatroar/concrete-statt
Replace restic.statT interface by concrete types
2020-11-14 23:32:17 +01:00
greatroar
c45f8ee075 Replace restic.statT interface by concrete types
name                old time/op    new time/op    delta
NodeFillUser-8        1.81µs ± 9%    1.50µs ± 5%  -17.07%  (p=0.000 n=19+20)
NodeFromFileInfo-8    1.76µs ± 4%    1.49µs ± 6%  -15.63%  (p=0.000 n=20+19)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)
NodeFromFileInfo-8      496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
NodeFromFileInfo-8      3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
MichaelEischer
3601a9b6cd Merge pull request #2690 from SkYNewZ/master
Fix #2688: Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
Michael Eischer
fdec8051ab tags: Tweak description of --add/set/remove options 2020-11-14 22:55:30 +01:00
MichaelEischer
333c5a19d4 Merge pull request #3082 from aawsome/check-sizes
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
Quentin Lemaire
a8ad6b9a4b fix(tags): Change tags list type according to restic.TagList 2020-11-14 16:24:58 +00:00
Quentin Lemaire
b0882b3f3c fix(snapshots): Update help message to better match the 'forget' command one 2020-11-14 15:48:57 +00:00
Quentin Lemaire
e74110a833 docs: Write new entry to changelog/unreleased 2020-11-14 15:48:56 +00:00
Quentin Lemaire
ae441d3134 fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Neumann
913a34f568 Merge pull request #3094 from restic/fix-ci-tests
Fix ci tests
2020-11-14 11:24:28 +01:00
Alexander Neumann
468612b108 Disable fuse test for macOS 2020-11-14 11:02:49 +01:00
Alexander Weiss
7eabcabf68 Adjust changelog 2020-11-14 00:42:49 +01:00
Alexander Weiss
17bb77b1f9 check: Also check blob length and offset 2020-11-14 00:42:49 +01:00
Alexander Weiss
80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
tofran
94a154c7ca Remove --drive-use-trash=false from rclone param
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
Alexander Neumann
219d9a62f2 Download assets from repo 2020-11-13 21:56:14 +01:00
rawtaz
e8713bc209 Merge pull request #3093 from restic/fix-diff-metadata
diff: Correctly count top-level blobs
2020-11-13 21:50:29 +01:00
Alexander Neumann
04d1983800 Merge pull request #3090 from fgma/vss-fix-386
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
fgma
88208c3db2 vss: improved changelog entry 2020-11-13 21:13:17 +01:00
Alexander Neumann
59ea5a4208 diff: Correctly count top-level blobs 2020-11-13 21:11:21 +01:00
Alexander Neumann
145830005b Use old versions, otherwise tar doesn't work 2020-11-12 21:35:01 +01:00
Alexander Neumann
8ad9f88993 helpers: Improve error message 2020-11-12 20:38:31 +01:00
fgma
859d89b032 vss: add changelog file for issue 3090 2020-11-12 19:38:22 +01:00
fgma
f9223cd827 vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386 2020-11-12 19:31:00 +01:00
Alexander Neumann
0be906a92f CI: Use netcologne mirror 2020-11-11 20:59:23 +01:00
rawtaz
dfb9326b1b Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
Alexander Neumann
e4e0ce09ad CI: Update links for Windows binaries 2020-11-11 20:51:01 +01:00
Alexander Neumann
0334114865 CI: Hardcode mirror URL 2020-11-11 20:42:47 +01:00
Alexander Neumann
b1bbdcb637 Improve changelog 2020-11-11 20:30:30 +01:00
Alexander Neumann
4a0b7328ec s3: Remove dots for config description 2020-11-11 20:20:35 +01:00
Alexander Neumann
3e0456d88b Highlight that s3.list-objects-v1 is a temporary 2020-11-11 20:11:35 +01:00
Alexander Neumann
dd94174379 Merge pull request #3086 from greatroar/diff-errors
Improve error reporting from restic diff
2020-11-11 19:48:03 +01:00
greatroar
63e32c44c0 Improve error reporting from restic diff
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
Nick Douma
f013662e3f Remove separate section on Ceph, and move s3.list-objects-v1 note to S3 section 2020-11-11 15:11:14 +01:00
Nick Douma
4320ff2bbf Add changelog entry for s3.list-objects-v1 2020-11-11 12:33:22 +01:00
Nick Douma
354b7e89cc Document the extended s3.list-objects-v1 flag in a new Ceph section 2020-11-11 12:32:46 +01:00
Nick Douma
829959390a Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1 2020-11-11 11:54:38 +01:00
Nick Douma
ccd55d529d Add s3.list-objects-v1 extended option and default to false 2020-11-11 11:54:36 +01:00
Nick Douma
4ddcc17135 Add support for boolean extended options 2020-11-11 11:54:27 +01:00
MichaelEischer
407843c5f9 Merge pull request #3034 from hoyho/gh_master
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
MichaelEischer
46d31ab86d Merge pull request #3058 from greatroar/counter
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
Alexander Neumann
c986823d3f Merge pull request #3048 from aawsome/check-pack-index
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
Alexander Weiss
239931578c check: check index for packs that are read 2020-11-09 17:28:14 +01:00
hoyho
9df52327cc bugfix: omit ENOTDATA for extended attributes
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
rawtaz
605db3b389 Merge pull request #3077 from MichaelEischer/tame-golangci-lint
Switch to the default set of golangci-lint linters
2020-11-08 22:34:27 +01:00
Michael Eischer
8de129e12f Switch to the default set of golangci-lint linters 2020-11-08 22:17:50 +01:00
Alexander Neumann
2072f0a481 Merge pull request #3076 from restic/check-go-mod-tidy
Check that go.mod/go.sum are up to date
2020-11-08 21:37:22 +01:00
Alexander Neumann
5731e391f8 Merge pull request #3075 from restic/warn-prune-no-cache
UI: Add several hints
2020-11-08 20:37:10 +01:00
Alexander Neumann
6a0a1d1f1c Check that go.mod/go.sum are up to date 2020-11-08 20:32:25 +01:00
Alexander Neumann
a8d21b5dcf prune: Warn if no cache is present 2020-11-08 20:25:35 +01:00
Alexander Neumann
823d0afd6e backup: Always print parent snapshot info 2020-11-08 20:25:35 +01:00
Alexander Neumann
a5989707ac Move test for golang-ci 2020-11-08 19:58:16 +01:00
Alexander Neumann
3a0cfafeb5 Fix quotes 2020-11-08 19:52:37 +01:00
Alexander Neumann
c923bd957d Fix test for pull request 2020-11-08 19:51:11 +01:00
Alexander Neumann
1a3f885d3d Only run golangci-lint for PRs 2020-11-08 17:34:07 +01:00
Alexander Neumann
3bf43d7951 Merge pull request #2475 from restic/use-github-actions
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
Alexander Neumann
561da92396 Replace badges in README 2020-11-08 17:26:59 +01:00
Alexander Neumann
5cf42884c8 Run tests on GitHub Actions 2020-11-08 16:04:57 +01:00
MichaelEischer
9e4e0077fb Merge pull request #3069 from greatroar/xattr-eintr
Update pkg/xattr to handle EINTR on Linux
2020-11-08 12:52:23 +01:00
MichaelEischer
1758da855f Merge pull request #3068 from aawsome/context-mem-backend
Return context error in mem backend
2020-11-08 12:24:12 +01:00
greatroar
15ea90feed Update pkg/xattr to handle EINTR on Linux
Updates #2659. This is a case where the stdlib will not handle EINTR for
us, even with Go 1.16. That xattr calls are directly affected can be
seen in the report for issue #2968.
2020-11-08 09:34:24 +01:00
Alexander Weiss
826cfa0533 fix context in archiver tests 2020-11-08 08:24:24 +01:00
Alexander Weiss
fef408a8bd Return context error in mem backend 2020-11-08 00:05:53 +01:00
greatroar
a2d4209322 Don't recurse in local backend's List if not required
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.

Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9).
2020-11-07 08:54:13 +01:00
Alexander Neumann
275f713211 Remove Travis and AppVeyor 2020-11-06 21:49:52 +01:00
MichaelEischer
4707bdb204 Merge pull request #2842 from aawsome/rebuild-index-inmem
Rebuild index in prune by using in-memory index
2020-11-06 20:51:20 +01:00
Alexander Neumann
47277c4b4c Add comments, clarify computation 2020-11-06 20:23:30 +01:00
Alexander Weiss
d2e53730d6 Add test that repo.List is only called once 2020-11-06 20:23:30 +01:00
Alexander Weiss
fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Weiss
38cc4393f6 Add Masterindex.Save(); Add Index.Packs() 2020-11-06 20:23:30 +01:00
Alexander Neumann
7f6f31c34b Remove Hound 2020-11-06 11:43:58 +01:00
Alexander Neumann
164b4cb2f6 Add changelog for #3014 2020-11-06 10:05:42 +01:00
Alexander Neumann
4a9b05aff1 Merge pull request #3063 from aawsome/fix-3062
Fix #3062
2020-11-05 19:36:40 +01:00
Alexander Weiss
aaf1c44362 Fix #3062 2020-11-05 17:05:42 +01:00
Alexander Neumann
a5592e83f7 Merge pull request #3014 from ivandeex/pr-stream-reset
Fix sporadic stream reset between rclone and restic
2020-11-05 16:11:50 +01:00
Ivan Andreev
ab2790d9de Fix http2 stream reset between restic and rest backends #3014 2020-11-05 15:57:40 +03:00
Alexander Neumann
8a2a326189 Merge pull request #2535 from ncw/fix-2528
s3: add bucket-lookup parameter to select path or dns style bucket lookup
2020-11-05 12:39:34 +01:00
Nick Craig-Wood
86b5d8ffaa s3: add bucket-lookup parameter to select path or dns style bucket lookup
This is to enable restic working with Alibaba cloud

Fixes #2528
2020-11-05 12:20:10 +01:00
Alexander Neumann
636b2f2e94 Merge pull request #2941 from MichaelEischer/parallel-repack
prune: Parallelize repack step
2020-11-05 11:00:41 +01:00
Alexander Neumann
ae5302c7a8 Add comment that keepBlobs is modified 2020-11-05 10:33:38 +01:00
Alexander Neumann
866a52ad4e Remove unneeded seek
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
Alexander Neumann
a4507610a0 Fix typo 2020-11-05 10:31:49 +01:00
Alexander Neumann
7def2d8ea7 Use a non-constant seed 2020-11-05 10:31:49 +01:00
Alexander Neumann
ee0112ab3b Clarify message about expected error 2020-11-05 10:31:49 +01:00
Michael Eischer
b373f164fe prune: Parallelize repack command 2020-11-05 10:31:49 +01:00
Alexander Neumann
8a0dbe7c1a helpers: Remove old changelog files for release 2020-11-05 10:26:00 +01:00
Alexander Neumann
4e3ad8b3b1 Remove changelog files from 0.11.0 release 2020-11-05 10:22:39 +01:00
Alexander Neumann
5144141321 Merge pull request #2718 from aawsome/new-cleanup-command
Reimplementation of prune
2020-11-05 10:12:19 +01:00
Alexander Neumann
d35d279455 Set development version for 0.11.0 2020-11-05 09:41:40 +01:00
Alexander Neumann
b7e1ece1e0 Add version for 0.11.0 2020-11-05 09:41:35 +01:00
Alexander Neumann
c5300a2c56 Update manpages and auto-completion 2020-11-05 09:41:35 +01:00
Alexander Neumann
1bd92896d7 Generate CHANGELOG.md for 0.11.0 2020-11-05 09:41:22 +01:00
Alexander Neumann
9dba01021e Prepare changelog for 0.11.0 2020-11-05 09:41:22 +01:00
Alexander Neumann
23f6b8c3fd Add .vscode to gitignore 2020-11-05 09:40:56 +01:00
Alexander Neumann
79a50e3b1f helpers: Create changelog subdir for release 2020-11-05 09:39:06 +01:00
Alexander Neumann
b8a5ca2d10 Add gitignore to changelog/unreleased 2020-11-05 09:01:32 +01:00
fgma
916b2d303b vss: fix potential crash (not reachable in restic) (#3045)
HasSufficientPrivilegesForVSS() now returns an error
2020-11-04 22:14:18 +01:00
Alexander Neumann
a06f5c28c0 Merge pull request #3055 from greatroar/cleanup
Don't separately allocate sync.{Mutex,Once} if not necessary
2020-11-04 11:11:21 +01:00
greatroar
c2f3eee5af Don't separately allocate sync.{Mutex,Once} if not necessary
Separate allocation of synchronization devices suggests they're shared
between objects, but they're not.
2020-11-04 09:55:43 +01:00
rawtaz
62345abe4a Merge pull request #3053 from maikell/patch-1
Update README.md
2020-11-03 19:06:59 +01:00
Maikel van Leeuwen
e024fc6d4d Update README.md
Typo in link. 
Links to  https://forum.restic.net%3E/ should be https://forum.restic.net
2020-11-03 18:51:11 +01:00
Alexander Neumann
1ca60bccfb Refactor condition for MaxRepackBytes
Don't depend on the string (opts.MaxRepackSize) for the condition,
instead check if there's a (positive) limit configured.
2020-11-03 16:42:21 +01:00
Alexander Neumann
7f86eb4ec0 Move helper function 2020-11-03 16:42:21 +01:00
Alexander Neumann
c1a3de4a6e Refactor max-unused calculation, add unlimited option
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.

Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.

Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.

Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.

I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:

      x ≤ p * (u+x)
    ⇔ x ≤ p*u + p*x
    ⇔ x - p*x ≤ p*u
    ⇔ x * (1-p) ≤ p*u
    ⇔ x ≤ p/(1-p) * u
2020-11-03 16:42:21 +01:00
Alexander Neumann
f8c4dd7b1a Split packe rewrite logic into two case branches
The comma is too sublte, let's split this into two separate branches.
2020-11-03 16:42:21 +01:00
Alexander Neumann
a5b80452fe Add comment that usedBlobs is modified 2020-11-03 16:42:21 +01:00
Alexander Neumann
aff1e220f5 Split struct members, add comments 2020-11-03 16:42:21 +01:00
Alexander Neumann
095155d9ce Remove RepackSmall 2020-11-03 16:42:21 +01:00
Alexander Neumann
1dd9fdce74 Reword changelog slightly 2020-11-03 16:42:21 +01:00
Alexander Weiss
b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss
7f9a0a5907 Reimplementation of prune 2020-11-03 16:42:21 +01:00
Alexander Weiss
3b591ed987 Add Verboseff 2020-11-03 16:42:21 +01:00
Alexander Weiss
ce7d613749 Add Blob.Handle() 2020-11-03 16:42:21 +01:00
Alexander Weiss
581d90cf91 Make some pack parameters public 2020-11-03 16:42:21 +01:00
Alexander Neumann
0db9024aad Merge pull request #3052 from restic/error-on-invalid-size
Return an error for invalid sizes
2020-11-03 12:13:49 +01:00
Alexander Neumann
21ba15577e Return an error for invalid sizes 2020-11-03 11:53:49 +01:00
Alexander Neumann
2fb1957ca4 Rewrite README as Markdown 2020-11-03 09:20:09 +01:00
Alexander Neumann
9a88fb253b Merge pull request #3051 from greatroar/sanitize-env
Sanitize environment before starting backend processes (rclone, ssh)
2020-11-02 21:18:57 +01:00
Alexander Neumann
f14436953a Merge pull request #3050 from greatroar/widechars
Fix string truncation in ui/termstatus
2020-11-02 16:58:51 +01:00
greatroar
11fbaaae9a Sanitize environment before starting backend processes (rclone, ssh)
The restic security model includes full trust of the local machine, so
this should not fix any actual security problems, but it's better to be
safe than sorry.

Fixes #2192.
2020-11-02 16:41:23 +01:00
greatroar
3ed84ff0c6 Fix string truncation in ui/termstatus
Fixes #3046.
2020-11-02 12:50:49 +01:00
Alexander Neumann
8e965ed4eb Merge pull request #3039 from MichaelEischer/tar-proper-dirs
Properly dump directories to tar
2020-11-02 12:30:26 +01:00
Alexander Neumann
5f0fa2129e Improve readability
It's time to use a switch statement.
2020-11-02 11:24:26 +01:00
Alexander Neumann
04dfa19c7e Improve changelog 2020-11-02 11:23:09 +01:00
Alexander Neumann
6509c207f4 Merge pull request #2997 from MichaelEischer/faster-excludes
Speedup exclude/include checking
2020-11-02 11:07:21 +01:00
Alexander Neumann
445b845267 Merge pull request #2978 from MichaelEischer/warn-tree-error
Warn if backup failed to read tree blob
2020-11-02 10:14:12 +01:00
Alexander Neumann
3ff37215df Merge pull request #2935 from MichaelEischer/upgrade-minio
Upgrade minio SDK to version 7
2020-11-02 09:09:10 +01:00
MichaelEischer
5d379b5359 Merge pull request #3042 from rubiojr/remove-unused-function
Remove unused function
2020-11-01 21:55:22 +01:00
Sergio Rubio
e708628cfd Remove unused function
Not currently used, and it'd need to be added to the MasterIndex interface first.
2020-10-28 13:24:49 +01:00
MichaelEischer
bb4b3481a6 Merge pull request #3031 from greatroar/debug-no-seek
Allow debug logging to pipes and terminals
2020-10-25 17:59:06 +01:00
MichaelEischer
ad3a52e6f0 Merge pull request #3026 from greatroar/refactor-ui
internal/ui refactoring
2020-10-25 17:51:39 +01:00
rawtaz
e8b4d8d8bc Merge pull request #2998 from greatroar/env-docs
Systematize documentation of environment variables
2020-10-24 22:49:31 +02:00
Michael Eischer
1aa61e6def Add changelog 2020-10-24 22:49:29 +02:00
Michael Eischer
8d7d6ad2d5 dump: include username in tar 2020-10-24 22:42:01 +02:00
Michael Eischer
fe09e6f865 dump: test proper permissions and directory name 2020-10-24 22:42:01 +02:00
Michael Eischer
1e3c9a2c11 dump: Fix file permission to tar mapping
The file permissions included a go specific directory bit which
accidentially forced the usage of the GNU header format. This leads
to problems with 7zip on Windows or when extended attributes are
used.
2020-10-24 22:42:01 +02:00
Michael Eischer
e21dcb0eea dump: Additional ACL tests 2020-10-24 22:41:01 +02:00
MichaelEischer
31b8d7a639 Merge pull request #2274 from fgma/master
Support for Volume Shadow Copy Service (VSS) on windows
2020-10-24 15:43:34 +02:00
fgma
5695f9ebd2 vss: Implement VSS support for Windows
The VSS support works for 32 and 64-bit windows, this includes a check that
the restic version matches the OS architecture as required by VSS. The backup
operation will fail the user has not sufficient permissions to use VSS.

Snapshotting volumes also covers mountpoints but skips UNC paths.
2020-10-24 11:35:57 +02:00
greatroar
8091151638 doc: Update restic mount availability and requirements
NetBSD doesn't support restic mount either, so it's easier to list the
positive cases. Also noted that FUSE for macOS is required on the Mac.
2020-10-22 09:57:30 +02:00
greatroar
ae179ee63e Systematize documentation of environment variables
Cache locations were documented inconsistently in three places.

The backup docs mentioned PATH being used to find fusermount, which is
never run by restic backup. It now mentions ssh and rclone, which are
used by backends.

The notion of a "system-wide" environment variable makes no sense.
TMPDIR is now mentioned because it allows for optimization and may
have security implications.
2020-10-22 09:57:30 +02:00
MichaelEischer
0590e3e12d Merge pull request #3035 from greatroar/no-unmount-on-failure
restic mount: don't umount unless we actually tried to mount
2020-10-21 22:10:42 +02:00
greatroar
3807d13bdc restic mount: don't umount unless we actually tried to mount
Previously, failure to open a repository would result in an "unable to
umount" message.
2020-10-21 12:15:05 +02:00
greatroar
63be3704d9 Allow debug logging to pipes and terminals 2020-10-19 22:30:30 +02:00
greatroar
35419de232 Simplify ui.StdioWrapper.Write
Instead of looping to find line breaks, make it look for the last one.
2020-10-17 20:30:46 +02:00
greatroar
863a590a81 Refactor termstatus.Term.{Print,Error} methods 2020-10-17 20:30:46 +02:00
greatroar
7c0b6a82db Remove unused public method ui.linesWriter.Flush 2020-10-17 20:30:24 +02:00
MichaelEischer
96a912b65a Merge pull request #3025 from aawsome/fix-index-ids
Fix setting of ID in DecodeIndex
2020-10-17 09:54:45 +02:00
Alexander Weiss
b44ecde8b0 Fix setting of ID in DecodeIndex 2020-10-17 09:12:58 +02:00
MichaelEischer
39fe1e96fe Merge pull request #3018 from greatroar/background-check
Fix IsProcessBackground on Linux with stdin redirection
2020-10-16 22:50:19 +02:00
MichaelEischer
4ba237bb93 Merge pull request #3019 from greatroar/refactor-decodeindex
Refactor index decoding
2020-10-15 23:22:33 +02:00
MichaelEischer
ce87fbd7dc Merge pull request #3022 from greatroar/refactor-runworkers
Defer channel closing outside repository.RunWorkers
2020-10-14 22:44:42 +02:00
greatroar
b27375f5ce defer close(ch) outside repository.RunWorkers 2020-10-14 15:50:16 +02:00
greatroar
720e0ee0c7 if cond { return true }; return false => return cond 2020-10-13 20:56:43 +02:00
greatroar
27db3ec262 Refactor index decoding
Decoding old-format indices no longer requires loading and decrypting
twice.
2020-10-13 20:47:50 +02:00
greatroar
f80b07b2c8 Fix IsProcessBackground on Linux with stdin redirection
The previous implementation assumed that stdin was a terminal.
It now checks the terminal's fd.
2020-10-13 13:12:02 +02:00
MichaelEischer
6003dada14 Merge pull request #3011 from restic/doc-alpine
doc: Add restic install instructions for Alpine Linux
2020-10-12 23:29:40 +02:00
MichaelEischer
41a45ae908 Merge pull request #3016 from greatroar/uid-gid-lookup
Simplify/optimize cached UID/GID lookups
2020-10-12 23:24:55 +02:00
MichaelEischer
8299c5559c Merge pull request #3015 from greatroar/changelog-template
Improved changelog template
2020-10-12 23:12:29 +02:00
Alexander Neumann
56883817d8 Merge pull request #2990 from MichaelEischer/fix-goreport-warnings
Fix some goreport warnings
2020-10-12 20:44:56 +02:00
greatroar
f0cd16e5ea Cache uint32-typed ids in lookup{Username,Group}
NodeFillUser-8        1.92µs ± 5%    1.75µs ± 3%   -8.89%  (p=0.000 n=10+10)
NodeFromFileInfo-8    1.89µs ± 7%    1.76µs ± 4%   -6.69%  (p=0.001 n=10+10)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)
NodeFromFileInfo-8      504B ± 0%      496B ± 0%   -1.59%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
NodeFromFileInfo-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
2020-10-12 15:08:15 +02:00
greatroar
a03fe4562e Remove unused error return in lookup{Username,Group} 2020-10-12 14:48:23 +02:00
greatroar
efc075df87 Improved changelog template 2020-10-12 12:15:15 +02:00
Leo R. Lundgren
f500b0d90e doc: Add restic install instructions for Alpine Linux 2020-10-11 18:31:07 +02:00
Alexander Neumann
c193cea119 Pass Context to NewRestorer() 2020-10-10 15:24:26 +02:00
Alexander Neumann
f0d49ca600 Merge pull request #2933 from MichaelEischer/less-context-todo
Replace most usages of context.TODO()
2020-10-10 15:03:44 +02:00
MichaelEischer
164d8af3dd Merge pull request #2906 from kitone/fix-inconsistent-timestamps-permissions
Restore inconsistent timestamps permissions
2020-10-10 14:56:43 +02:00
kitone
052564007a Add changelog 2020-10-10 14:37:51 +02:00
kitone
6099f81692 Fix #1212 restore code produces inconsistent timestamps/permissions.
Keep track of restored child status so parent and root directory not selected by filter will also restore metadata when traversing tree.
2020-10-10 13:46:44 +02:00
kitone
9d7f616190 Improve restorer debug log information 2020-10-10 13:46:43 +02:00
kitone
295ddb9e57 Add test case for inconsistent timestamps and permissions restoration
Reproduce from https://github.com/restic/restic/issues/1212
2020-10-10 13:46:43 +02:00
Michael Eischer
e638b46a13 Embed context into ReaderAt
The io.Reader interface does not support contexts, such that it is
necessary to embed the context into the backendReaderAt struct. This has
the problem that a reader might suddenly stop working when it's
contained context is canceled. However, this is now problem here as the
reader instances never escape the calling function.
2020-10-09 22:39:07 +02:00
Michael Eischer
d6cfe857b7 pass proper context into MasterIndex.RebuildIndex 2020-10-09 22:39:07 +02:00
Michael Eischer
2964d2ad15 Skip unlocking for nil locks
Now that lockRepo receives a context, it is possible that it is canceled
before a lock was created. Thus `unlockRepo` must be able to handle this
case.
2020-10-09 22:39:06 +02:00
Michael Eischer
0c9efa9c2a Pass context to lockRepo 2020-10-09 22:39:06 +02:00
Michael Eischer
37a5e2d681 rest: use global context on repository creation 2020-10-09 22:39:06 +02:00
Michael Eischer
4b0fcaed45 unlock: use proper context for locks cleanup
The list operation used by RemoveStaleLocks or RemoveAllLocks will
already be canceled by the passed in context. Therefore we can also just
cancel the remove operation as the unlock command won't process all lock
files anyways.
2020-10-09 22:37:56 +02:00
Michael Eischer
645a6efaf2 restorer: remove redundant type specification 2020-10-09 22:37:56 +02:00
Michael Eischer
dc2e664209 integration_fuse_test: use global context
No need to use the TODO context.
2020-10-09 22:37:56 +02:00
Michael Eischer
a449450021 init: pass proper context to master key generation
This is no change in behavior as a canceled context did later on cause
the config file creation to fail. Therefore this change just lets the
repository initialization fail a bit earlier.
2020-10-09 22:37:56 +02:00
Michael Eischer
603bb0e309 restore: Use proper context while loading snapshot 2020-10-09 22:37:56 +02:00
Michael Eischer
27456f6545 debug: use proper context
This allows the debug commands to be properly interrupted.
2020-10-09 22:37:56 +02:00
Michael Eischer
c458e114d4 pass context to Find / FindSnapshot
This allows proper interruption of restic while it searches for
snapshots or key files.
2020-10-09 22:37:56 +02:00
Michael Eischer
45e9a55c62 Wire context into backend layout detection 2020-10-09 22:37:24 +02:00
Michael Eischer
307a6ba3a3 Upgrade minio sdk to v7
This changes are primarily straightforward modifications to pass the
parameters in the now expected way.
2020-10-09 22:37:24 +02:00
Michael Eischer
50da20d93d Warn if backup failed to read tree blob 2020-10-09 22:36:27 +02:00
Alexander Neumann
5fd3dbccb7 Merge pull request #2963 from MichaelEischer/fix-status-deadlock
backup: Fix possible deadlock of scanner goroutine
2020-10-09 21:35:50 +02:00
Alexander Neumann
35655a481b Merge pull request #2931 from MichaelEischer/reduce-debug-overhead
Reduce debug log overhead
2020-10-09 21:35:11 +02:00
Alexander Neumann
30cb553c8d Merge pull request #2932 from MichaelEischer/proper-rclone-create
Call rclone.Create to create a new repository for the rclone backend
2020-10-09 21:29:15 +02:00
Alexander Neumann
2f3eeff2e7 Merge pull request #2964 from MichaelEischer/fix-repro-instructions
Fix instructions to reproduce the release binaries
2020-10-09 21:28:24 +02:00
MichaelEischer
356ac404cd Merge pull request #2996 from greatroar/setconsolecursorposition
Use windows.SetConsoleCursorPosition in ui/termstatus
2020-10-09 18:19:02 +02:00
rawtaz
ab769abeaf Merge pull request #3002 from MichaelEischer/update-contributing
Update links to help wanted labels in CONTRIBUTING.md and link to forum
2020-10-09 17:24:25 +02:00
Michael Eischer
4036991c91 Update links to help wanted labels in CONTRIBUTING.md and link to forum 2020-10-09 16:15:43 +02:00
Michael Eischer
88c8e903d2 filter: Fix glob matching on absolute path marker on windows
A pattern part containing "/" is used to mark a path or a pattern as
absolute. However, on Windows the path separator is "\" such that glob
patterns like "?" could match the marker. The code now explicitly skips
the marker when the pattern does not represent an absolute path.
2020-10-09 16:11:05 +02:00
greatroar
740758a5fa Optimize filter pattern matching
By replacing "**" with "", checking for this special path component can
be reduced to a length-zero check.

name                          old time/op    new time/op    delta
FilterLines-8                   44.7ms ± 5%    44.9ms ± 5%     ~     (p=0.631 n=10+10)
FilterPatterns/Relative-8       13.6ms ± 4%    13.4ms ± 5%     ~     (p=0.165 n=10+10)
FilterPatterns/Absolute-8       10.9ms ± 5%    10.7ms ± 4%     ~     (p=0.052 n=10+10)
FilterPatterns/Wildcard-8       53.7ms ± 5%    50.4ms ± 5%   -6.00%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-8     128ms ± 2%      95ms ± 1%  -25.54%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=1.000 n=9+8)
FilterPatterns/Absolute-8       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.903 n=9+8)
FilterPatterns/Wildcard-8       19.7MB ± 0%    19.7MB ± 0%   -0.00%  (p=0.022 n=10+9)
FilterPatterns/ManyNoMatch-8    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-8        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-8        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-8     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-09 15:51:14 +02:00
MichaelEischer
862ee4b2c9 Merge pull request #2970 from labkode/fopenskip
Skip fopen for file change check to avoid network penalty
2020-10-09 11:43:54 +02:00
Hugo Gonzalez Labrador
958dc6aafc Skip fopen for file change check to avoid network penalty 2020-10-09 10:53:28 +02:00
MichaelEischer
88cc444779 Merge pull request #2934 from MichaelEischer/upgrade-backoff
Upgrade github.com/cenkalti/backoff module
2020-10-08 20:33:30 +02:00
MichaelEischer
4a424af1d5 Merge pull request #2989 from MichaelEischer/remove-local-chmod
local: mark repository files as read-only and handle chmod errors
2020-10-08 19:04:13 +02:00
Michael Eischer
bc65da2baf Add changelog 2020-10-08 14:45:26 +02:00
Michael Eischer
b79f18209f Upgrade github.com/cenkalti/backoff module
We now use v4 of the module. `backoff.WithMaxRetries` no longer repeats
an operation endlessly when a retry count of 0 is specified. This
required a few fixes for the tests.
2020-10-07 22:04:59 +02:00
Michael Eischer
8388e67c4b filter: cleanup path separator conversion 2020-10-07 21:14:07 +02:00
Michael Eischer
0acc3c5923 filter: special case patterns without globbing characters
In case a part of a path is a simple string, we can just check for
equality without complex parsing in filepath.Match.

name                          old time/op    new time/op    delta
FilterLines-4                   34.8ms ±17%    41.2ms ±23%  +18.36%  (p=0.000 n=10+10)
FilterPatterns/Relative-4       21.7ms ± 6%    12.1ms ±23%  -44.46%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       10.0ms ± 5%     9.1ms ±11%   -9.80%  (p=0.006 n=10+9)
FilterPatterns/Wildcard-4       47.0ms ± 7%    42.2ms ± 5%  -10.19%  (p=0.000 n=9+10)
FilterPatterns/ManyNoMatch-4     190ms ± 1%     131ms ±20%  -31.47%  (p=0.000 n=8+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.870 n=9+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.145 n=10+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    19.7MB ± 0%  +37.91%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.421 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:55:43 +02:00
Michael Eischer
54a124de3b filter: explicitly test separate ListWithChild function 2020-10-07 20:47:52 +02:00
Michael Eischer
bcc3bddcf4 filter: only check whether a child path could match when necessary
When checking excludes there is no need to test whether a child path
could also match the pattern, as it is by definition excluded.
Previously childMayMatch was calculated but then discarded. For simple
absolute paths this can account for half the time spent for checking
pattern matches.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.3ms ± 9%    21.7ms ± 6%   -6.68%  (p=0.004 n=10+10)
FilterPatterns/Absolute-4       13.9ms ± 7%    10.0ms ± 5%  -27.61%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4       51.4ms ± 7%    47.0ms ± 7%   -8.51%  (p=0.001 n=9+9)
FilterPatterns/ManyNoMatch-4     551ms ± 9%     190ms ± 1%  -65.41%  (p=0.000 n=10+8)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.665 n=10+9)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.480 n=9+10)
FilterPatterns/Wildcard-4       14.3MB ± 0%    14.3MB ± 0%     ~     (p=0.431 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        88.7k ± 0%     88.7k ± 0%     ~     (all equal)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:52 +02:00
Michael Eischer
17c53efb0d filter: Optimize double wildcard expansion
This only allocates a single slice to expand the double wildcard and
only copies the pattern prefix once.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       22.7ms ± 5%    23.3ms ± 9%     ~     (p=0.353 n=10+10)
FilterPatterns/Absolute-4       14.2ms ±13%    13.9ms ± 7%     ~     (p=0.853 n=10+10)
FilterPatterns/Wildcard-4        266ms ±16%      51ms ± 7%  -80.67%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4     554ms ± 6%     551ms ± 9%     ~     (p=0.436 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.349 n=10+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.073 n=10+9)
FilterPatterns/Wildcard-4        141MB ± 0%      14MB ± 0%  -89.89%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     0.09M ± 0%  -94.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:48 +02:00
Michael Eischer
7959796269 filter: Special case for absolute paths
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       23.6ms ±20%    22.7ms ± 5%     ~     (p=0.684 n=10+10)
FilterPatterns/Absolute-4       32.3ms ± 8%    14.2ms ±13%  -56.01%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        334ms ±17%     266ms ±16%  -20.56%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     709ms ± 7%     554ms ± 6%  -21.89%  (p=0.000 n=10+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       3.57MB ± 0%    3.57MB ± 0%   +0.00%  (p=0.046 n=9+10)
FilterPatterns/Absolute-4       3.57MB ± 0%    3.57MB ± 0%     ~     (p=0.464 n=10+10)
FilterPatterns/Wildcard-4        141MB ± 0%     141MB ± 0%     ~     (p=0.163 n=9+10)
FilterPatterns/ManyNoMatch-4    3.57MB ± 0%    3.57MB ± 0%     ~     (all equal)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Absolute-4        22.2k ± 0%     22.2k ± 0%     ~     (all equal)
FilterPatterns/Wildcard-4        1.63M ± 0%     1.63M ± 0%     ~     (p=0.072 n=10+10)
FilterPatterns/ManyNoMatch-4     22.2k ± 0%     22.2k ± 0%     ~     (all equal)
2020-10-07 20:47:29 +02:00
Michael Eischer
375c2a56de filter: Parse filter patterns only once
name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       30.3ms ±10%    23.6ms ±20%  -22.12%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4       49.0ms ± 3%    32.3ms ± 8%  -33.94%  (p=0.000 n=8+10)
FilterPatterns/Wildcard-4        345ms ± 9%     334ms ±17%     ~     (p=0.315 n=10+10)
FilterPatterns/ManyNoMatch-4     3.93s ± 2%     0.71s ± 7%  -81.98%  (p=0.000 n=9+10)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       4.63MB ± 0%    3.57MB ± 0%  -22.98%  (p=0.000 n=9+9)
FilterPatterns/Absolute-4       8.54MB ± 0%    3.57MB ± 0%  -58.20%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        146MB ± 0%     141MB ± 0%   -2.93%  (p=0.000 n=9+9)
FilterPatterns/ManyNoMatch-4     907MB ± 0%       4MB ± 0%  -99.61%  (p=0.000 n=9+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4        66.6k ± 0%     22.2k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4        88.7k ± 0%     22.2k ± 0%  -75.00%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.70M ± 0%     1.63M ± 0%   -3.92%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     4.46M ± 0%     0.02M ± 0%  -99.50%  (p=0.000 n=10+10)
2020-10-07 20:47:27 +02:00
Michael Eischer
b8eacd1364 filter: Reduce redundant path and pattern splitting
A single call to filter.List will split the path only once and also
split each search pattern only once and use it for both match and
childMatch.

name                          old time/op    new time/op    delta
FilterPatterns/Relative-4       62.1ms ±15%    30.3ms ±10%  -51.22%  (p=0.000 n=9+10)
FilterPatterns/Absolute-4        111ms ±10%      49ms ± 3%  -56.08%  (p=0.000 n=10+8)
FilterPatterns/Wildcard-4        393ms ±15%     345ms ± 9%  -12.30%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     10.0s ± 3%      3.9s ± 2%  -60.53%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
FilterPatterns/Relative-4       16.4MB ± 0%     4.6MB ± 0%  -71.76%  (p=0.000 n=10+9)
FilterPatterns/Absolute-4       31.4MB ± 0%     8.5MB ± 0%  -72.77%  (p=0.000 n=9+10)
FilterPatterns/Wildcard-4        168MB ± 0%     146MB ± 0%  -13.19%  (p=0.000 n=10+9)
FilterPatterns/ManyNoMatch-4    3.23GB ± 0%    0.91GB ± 0%  -71.96%  (p=0.000 n=10+9)

name                          old allocs/op  new allocs/op  delta
FilterPatterns/Relative-4         178k ± 0%       67k ± 0%  -62.50%  (p=0.000 n=10+10)
FilterPatterns/Absolute-4         266k ± 0%       89k ± 0%  -66.67%  (p=0.000 n=10+10)
FilterPatterns/Wildcard-4        1.87M ± 0%     1.70M ± 0%   -9.47%  (p=0.000 n=10+10)
FilterPatterns/ManyNoMatch-4     17.7M ± 0%      4.5M ± 0%  -74.87%  (p=0.000 n=9+10)
2020-10-07 18:13:19 +02:00
Michael Eischer
e73c281142 filter: Benchmark absolute paths, wildcards and long filter lists 2020-10-07 17:54:36 +02:00
Michael Eischer
fdd3b14db3 filter: test some corner cases 2020-10-07 17:09:44 +02:00
Michael Eischer
f4282aa6fd local: mark repository files as read-only
This is intended to prevent accidental modifications of data files.
Marking the files as read-only was accidentally removed in #1258.
2020-10-07 12:29:37 +02:00
Michael Eischer
40ee17167e local: Ignore permission errors on chmod call in Save/Remove operation
The file is already created with the proper permissions, thus the chmod
call is not critical. However, some file systems have don't allow
modifications of the file permissions. Similarly the chmod call in the Remove
operation should not prevent it from working.
2020-10-07 12:29:37 +02:00
greatroar
f3e933f0c1 Use windows.SetConsoleCursorPosition in ui/termstatus 2020-10-07 11:18:54 +02:00
MichaelEischer
0ae02f3030 Merge pull request #2844 from aawsome/prune-integration-tests
Add prune integration tests for many edge cases
2020-10-06 23:16:41 +02:00
rawtaz
e401859afb Merge pull request #2992 from lbausch/repo_wording
Replace repo/repos with repository/repositories in `--no-lock` help text
2020-10-06 23:03:55 +02:00
Lorenz Bausch
af4100e07d Replace repo/repos with repository/repositories in --no-lock help text 2020-10-06 22:49:48 +02:00
Alexander Neumann
ea5bbe0857 Remove dysfunctional link to saythanks 2020-10-06 20:43:44 +02:00
Alexander Weiss
6822a58413 Add prune integration tests for many edge cases 2020-10-06 20:20:05 +02:00
MichaelEischer
09d39e260d Merge pull request #2980 from greatroar/error-wrapping
Remove repetitive error wrapping from internal/cache
2020-10-06 15:19:00 +02:00
Michael Eischer
1579d2a8ec Remove some unused assignments 2020-10-06 14:55:13 +02:00
Michael Eischer
eba5dd831f Fix typos reported by misspell 2020-10-06 14:55:13 +02:00
Michael Eischer
9ffb698c8d Modernize context import in internal/fuse
Migrated using `go fix`.
2020-10-06 14:55:13 +02:00
Michael Eischer
efbb850d92 Remove a few redundant type specifiers
This is the result of running `gofmt -s -w **/*.go`
2020-10-06 14:55:13 +02:00
MichaelEischer
cfd57c480a Merge pull request #2991 from restic/nonsense
cache/check: Clarify error message when given arguments
2020-10-06 00:40:12 +02:00
Leo R. Lundgren
e105a3f391 cache/check: Clarify error message when given arguments 2020-10-06 00:08:59 +02:00
rawtaz
5d8cfff3f2 Merge pull request #2988 from greatroar/snapshots-json
Regression test for #2979
2020-10-05 21:02:52 +02:00
greatroar
673dda77c0 Less repetitive error messages in internal/cache
Many instances of errors.Wrap in this package would produce messages
like "Open: open <filename>: no such file or directory"; those now omit
the first "Open:" (or "Stat:", or "MkdirAll"). The function readVersion
now appends its own name to the error message, rather than the function
that failed, to make it easier to spot. Other function names (e.g.,
Load) are already added further up in the call chain.
2020-10-05 20:28:54 +02:00
MichaelEischer
1ab4c710e1 Merge pull request #2987 from greatroar/no-password-for-help
Don't read password for generate, help or self-update
2020-10-05 20:16:19 +02:00
greatroar
feedf0ebce Don't read password for generate, help or self-update
Fixes #2951.
2020-10-05 19:54:07 +02:00
MichaelEischer
1a490acd67 Merge pull request #2982 from greatroar/archiver-error-handling
Check error in archiver before calling Select
2020-10-05 19:19:47 +02:00
Michael Eischer
187518a8a3 docs: Properly reproduce zip archives for windows binaries 2020-10-05 18:24:46 +02:00
Michael Eischer
a232c833dc docs: Update binary reproduction instructions and refer to older instructions 2020-10-05 18:24:46 +02:00
MichaelEischer
c84643c6a9 Merge pull request #2981 from greatroar/mkdircachedir
Simplify cache directory creation
2020-10-05 18:15:23 +02:00
MichaelEischer
8390a8aaf3 Merge pull request #2986 from greatroar/upgrade-xxhash
Upgrade cespare/xxhash to 2.1.1
2020-10-05 17:49:08 +02:00
greatroar
1e1a1f3078 Upgrade cespare/xxhash to 2.1.1 2020-10-05 17:25:39 +02:00
MichaelEischer
abe9fa261f Merge pull request #2983 from greatroar/upgrade-siphash
Upgrade siphash to 1.2.2
2020-10-05 17:18:07 +02:00
MichaelEischer
6001b45bf7 Merge pull request #2984 from restic/snapshots-json
snapshots: Make --json output [] instead of null when no snapshots
2020-10-05 17:14:23 +02:00
greatroar
6bee0aafc2 Regression test for #2979 2020-10-05 15:08:09 +02:00
greatroar
8252ea8e3d Upgrade siphash to 1.2.2
New version is potentially faster on 32-bit ARM.
2020-10-05 14:56:13 +02:00
Leo R. Lundgren
137d20a06a snapshots: Make --json output [] instead of null when no snapshots 2020-10-05 13:14:10 +02:00
greatroar
c4e2203e45 Check error in archiver before calling Select
The archiver first called the Select function for a path before checking
whether the Lstat on that path actually worked. The RejectFuncs in
exclude.go worked around this by checking whether they received a nil
os.FileInfo. Checking first is more obvious and requires less code.
2020-10-05 11:11:04 +02:00
greatroar
7d0fa1a686 Simplify cache directory creation 2020-10-05 10:46:45 +02:00
rawtaz
a23d90d270 Merge pull request #2975 from restic/rawtaz-doc-resume
Add FAQ on whether restic can resume backups
2020-10-03 23:12:34 +02:00
MichaelEischer
2630411530 Merge pull request #2974 from gottwald/swap-gcs-lib
Swap deprecated GCS lib with replacement
2020-10-03 21:58:11 +02:00
rawtaz
b9b82d878d Add FAQ on whether restic can resume backups
This new FAQ entry explains that restic can resume interrupted backups.
2020-10-03 21:39:45 +02:00
Ingo Gottwald
8b8e230771 Swap deprecated GCS lib with replacement 2020-10-03 18:55:56 +02:00
rawtaz
7c3c6fa431 Merge pull request #2977 from MichaelEischer/downgrade-cobra
Downgrade cobra to v0.0.5
2020-10-03 18:23:13 +02:00
Michael Eischer
29908906b7 Downgrade cobra to v0.0.5 2020-10-03 18:13:07 +02:00
rawtaz
bbeb439f41 Merge pull request #2937 from andreaso/self-update-output-path-fix
Don't require `self-update --output` placeholder file
2020-10-03 13:35:20 +02:00
Andreas Olsson
ce14df303b Don't require self-update --output placeholder file
This removes the requirement on `restic self-update --output` to point
to a path of an existing file, to overwrite. In case the specified
path does exist we still want to verify that it's a regular file,
rather than a directory or a device, which gets overwritten.

We also want to verify that a path to a new file exists within an
existing directory. The alternative being running into that issue
after the actual download, etc has completed.

While at it I also replace `errors.Errorf` with the more appropriately
verbose `errors.Fatalf`.

Resolves #2491
2020-10-03 07:02:23 +02:00
MichaelEischer
3c6671b18b Merge pull request #2973 from restic/rawtaz-repo2
Correct data type for --repo2 option
2020-10-02 22:22:41 +02:00
MichaelEischer
86ff1f2bf6 Merge pull request #2972 from restic/rawtaz-verbose
Clarify max verbose level in option description
2020-10-02 22:21:45 +02:00
rawtaz
0cce6dc31c Correct data type for --repo2 option 2020-10-02 21:10:52 +02:00
rawtaz
6253ff0187 Clarify max verbose level in option description
Clarifies that the max verbosity level is 3, in the description of the --verbose option.
2020-10-02 20:25:34 +02:00
rawtaz
e9943e864f Merge pull request #2955 from jtagcat/clarify--compact
help: --compact: clarify description
2020-10-02 15:59:08 +02:00
jtagcat
a687261804 --help: --compact: clarify description 2020-10-02 16:55:56 +03:00
MichaelEischer
4df8861e09 Merge pull request #2967 from gottwald/fix-deprctd-gs-constructor
Replace deprecated method in gs backend
2020-10-01 11:01:49 +02:00
Ingo Gottwald
00cedd22aa Replace deprecated method in gs backend 2020-10-01 10:02:42 +02:00
MichaelEischer
f361ed66de Merge pull request #2910 from mtdcr/repository-file
Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
2020-10-01 01:04:23 +02:00
Michael Eischer
7b50a65492 Update backup help output in documentation 2020-10-01 00:50:27 +02:00
Michael Eischer
c18b119a9b Document new option --repository-file 2020-10-01 00:50:27 +02:00
Michael Eischer
61035d68bc Add test for --repository-file 2020-10-01 00:50:27 +02:00
Andreas Oberritter
97f7855de3 Add new option --repository-file (default: $RESTIC_REPOSITORY_FILE)
As an alternative to -r, this allows to read the repository URL
from a file in order to prevent certain types of information leaks,
especially for URLs containing credentials.

Fixes #1458, fixes #2900.
2020-10-01 00:50:26 +02:00
Michael Eischer
d44df9d00d backup: Always remove the status lines once a backup ends 2020-09-30 23:13:10 +02:00
Michael Eischer
8d0ba55ecd backup: Fix possible deadlock of scanner goroutine
When the backup is interrupted for some reason while the scanner is
still active this could lead to a deadlock. Interruptions are triggered
by canceling the context object used by both the backup progress UI and
the scanner. It is possible that a context is canceled between the
respective check in the scanner and it calling the `ReportTotal` method
of the UI. The latter method sends a message to the UI goroutine.
However, a canceled context will also stop that goroutine, which can
cause the channel send operation to block indefinitely.

This is resolved by adding a `closed` channel which is closed once the
UI goroutine is stopped and serves as an escape hatch for reported UI
updates.

This change covers not just the ReportTotal method but all potentially
affected methods of the progress UI implementation.
2020-09-30 23:13:10 +02:00
rawtaz
34ea960559 Merge pull request #2966 from MichaelEischer/recover-help-typo
recover: Fix typo in the command help
2020-09-30 18:16:26 +02:00
rawtaz
5ea01b00df Merge pull request #2965 from MichaelEischer/rework-password-prompt
Clarify verbose password prompt
2020-09-30 18:15:05 +02:00
MichaelEischer
48d0ab5276 Merge pull request #2959 from restic/keep-text
backup: Correct keep policy text
2020-09-30 17:57:00 +02:00
Michael Eischer
2a79c1a44d recover: Fix typo in the command help
Reported-by: Steve Divskinsy <stevesbrain@users.noreply.github.com>
2020-09-30 17:44:34 +02:00
MichaelEischer
fd02407863 Merge pull request #2849 from classmarkets/gcs-access-token
gs: support authentication with access token
2020-09-30 17:42:56 +02:00
Michael Eischer
aea9f7d286 clarify verbose password prompt 2020-09-30 17:25:54 +02:00
Leo R. Lundgren
028c9a5343 backup: Correct keep policy text
Makes the following corrections to the "Applying Policy:" output:

- keep the last 1 snapshots snapshots => keep 1 latest snapshots
- keep the last 1 snapshots, 3 hourly, 5 yearly snapshots => keep 1 latest, 3 hourly, 5 yearly snapshots
2020-09-28 14:26:53 +02:00
MichaelEischer
94136132e3 Merge pull request #2957 from plumbeo/patch-3
Don’t print excessively detailed debug messages on object deletion when —verbose is used
2020-09-27 22:08:07 +02:00
plumbeo
009bd907f2 Don’t print too many messages when deleting files
Print detailed debug messages on file deletions only when --verbose=2 is used
2020-09-27 14:24:04 +02:00
MichaelEischer
14b312f00d Merge pull request #2658 from creativeprojects/issue-2241
Don't echo authentication passwords (rest backend)
2020-09-22 22:17:53 +02:00
Fred
206cadfab4 Hide password from repository URLs 2020-09-22 22:00:51 +02:00
MichaelEischer
4875f7b659 Merge pull request #2614 from greatroar/simplify-fs
Simplify internal/fs
2020-09-21 22:05:10 +02:00
MichaelEischer
6f2093e491 Merge pull request #2940 from andreaso/appveyor-tar-https
Use https:// to download appveyor tar
2020-09-21 21:50:26 +02:00
MichaelEischer
16f31b2f73 Merge pull request #2939 from J0WI/patch-1
Update Go version to 1.15 in Docker build script
2020-09-21 21:46:33 +02:00
greatroar
0d65b78168 Simplify os.ModeType|os.ModeCharDevice => os.ModeType
Since Go 1.12, ModeCharDevice is included in ModeType:
golang/go@a2a3dd00c9
2020-09-21 14:21:32 +02:00
greatroar
95ebba85ff Remove stray Printf from internal/fs 2020-09-21 14:21:32 +02:00
greatroar
59b343a9bf Remove OS-specific versions of fs.MkdirAll
Go has supported Windows paths correctly since 1.11, see
https://github.com/golang/go/issues/10900 and the commit referenced
there.
2020-09-21 14:21:32 +02:00
greatroar
1557c58eef Fix and simplify fs.Reader
fakeDir.{Readdir,Readdirnames} weren't handling the case n == 0
correctly. fakeFileInfo.sys is always nil, so omit the field.
2020-09-21 14:21:32 +02:00
greatroar
c504aa505c Remove unused fs.Rename 2020-09-21 14:21:32 +02:00
greatroar
1b20f6beec Remove io.Writer from fs.File
It was only used in a single test, which now uses plain *os.File instead.
2020-09-21 14:21:32 +02:00
rawtaz
10e3340863 Merge pull request #2945 from YoshieraHuang/max-file-size
Fix nil check in rejectBySize
2020-09-21 14:18:45 +02:00
yoshiera
3cf29a777d Fix nil check in rejectBySize 2020-09-21 19:20:24 +08:00
Andreas Olsson
fd1f7b7268 Use https:// to download appveyor tar
When supported one might as well use https://.
2020-09-20 18:49:46 +02:00
J0WI
090a73f7c5 Update Go version to 1.15 in Docker build script 2020-09-20 15:42:12 +00:00
rawtaz
1cfb01a8a6 Merge pull request #2936 from andreaso/contrib-update-version
Update versions in contribution documentation
2020-09-20 13:27:13 +02:00
Andreas Olsson
ce62d3d689 Update versions in contribution documentation
Reflects 3c44598 and 429f97b.
2020-09-20 10:32:27 +02:00
Michael Eischer
8c36317b71 rclone: use configured number of connections during create 2020-09-19 19:11:43 +02:00
Michael Eischer
60a5c35de9 rclone: close connection to rclone if open fails 2020-09-19 19:11:43 +02:00
Michael Eischer
9333f707fa init: use Create method for rclone backend
This properly issues the initial repository creation command

Fixes #1896
2020-09-19 19:11:43 +02:00
Alexander Neumann
429f97b887 Set development version for 0.10.0 2020-09-19 17:38:47 +02:00
Alexander Neumann
40832b2927 Add version for 0.10.0 2020-09-19 17:38:26 +02:00
Alexander Neumann
c8a94eced7 Update manpages and auto-completion 2020-09-19 17:38:26 +02:00
Alexander Neumann
ee6e981b4e Generate CHANGELOG.md for 0.10.0 2020-09-19 17:38:10 +02:00
Alexander Neumann
96fd982f6a Prepare changelog for 0.10.0 2020-09-19 17:37:59 +02:00
Alexander Neumann
6ff0082c02 Merge pull request #2928 from MichaelEischer/init-copy-chunker
init: Add `--copy-chunker-parameters` option
2020-09-19 17:32:13 +02:00
Alexander Neumann
95c1d7d959 Merge pull request #2927 from restic/update-deps
Update dependencies
2020-09-19 17:29:35 +02:00
Michael Eischer
07f4e7d10b Only log HTTP requests when a debug log is configured
The logged HTTP requests are only visible when a debug log is
configured.
2020-09-19 17:07:53 +02:00
Michael Eischer
f003410402 init: Add --copy-chunker-params option
This allows creating multiple repositories with identical chunker
parameters which is required for working deduplication when copying
snapshots between different repositories.
2020-09-19 16:53:05 +02:00
Michael Eischer
9ad8250a78 Reduce overhead of debug calls if no log is enabled
In case no debug log is configured, then calls to debug.Log only incur
the costs to check a single boolean flag making the call really cheap.
2020-09-19 16:45:01 +02:00
Michael Eischer
655430550b Extract parameters for second repository from copy command 2020-09-19 16:07:55 +02:00
MichaelEischer
1823b8195c Merge pull request #2930 from rawtaz/password-file-text
Improve wording for --password-file and related options
2020-09-19 16:06:15 +02:00
Leo R. Lundgren
311ad2d2d0 Improve wording for --password-file and related options 2020-09-19 15:47:32 +02:00
Alexander Neumann
a10b44a265 Add note about fuse 2020-09-19 14:35:55 +02:00
Alexander Neumann
baf3a9aa3b Merge pull request #2929 from rawtaz/password-command-text
Improve wording for --password-command and --password-command2 options
2020-09-19 14:30:39 +02:00
Leo R. Lundgren
ffe6dce7e7 Improve wording for --password-command and --password-command2 options 2020-09-19 14:23:25 +02:00
Alexander Neumann
8ce0ce387f Add changelog 2020-09-19 14:22:31 +02:00
Alexander Neumann
3c44598bf6 Drop Go 1.12, require at least 1.13 2020-09-19 14:20:04 +02:00
Alexander Neumann
3bb55fd6bf Merge pull request #2914 from YoshieraHuang/max-file-size
Support excluding files by size
2020-09-19 14:12:15 +02:00
Alexander Neumann
36efefa7bd Update dependencies, pin fuse library
This updates all dependencies which causes the following changes:
 * The fuse library is set to the last version supporting macOS
 * The minimal version of Go required to build restic is now 1.13
2020-09-19 14:06:37 +02:00
yoshiera
ac4b8c98ac Support excluding files by size 2020-09-19 19:52:09 +08:00
MichaelEischer
4dcd6abf37 Merge pull request #2616 from greatroar/no-cache-interface
Remove practically unused cache interface
2020-09-18 19:50:47 +02:00
greatroar
cb3f531050 Make Cache.Path private
It's only used inside the package.
2020-09-18 11:17:29 +02:00
greatroar
23fcbb275a Remove restic.Cache interface
It was used in one code path, which then asserted its concrete type as
*cache.Cache.

Privatised some of the interface methods.
2020-09-18 10:48:13 +02:00
greatroar
6e3215a80d Remove unused error type from internal/cache 2020-09-18 10:38:14 +02:00
greatroar
9abef3bf1a Move internal/fs.TestChdir to internal/test.Chdir 2020-09-17 10:43:33 +02:00
Alexander Neumann
b10dce541e Merge pull request #2598 from MichaelEischer/diff-speedup
Diff speedup
2020-09-13 16:42:05 +02:00
Michael Eischer
4f221c4022 Add changelog entry 2020-09-13 16:28:46 +02:00
Michael Eischer
f5c448aa65 diff: Optimize diff calculation for shared subtrees
When the diff calculation compares two trees with identical id then no
differences between them can ever show up. Optimize for that case by
simply traversing the tree only once to collect all referenced blobs for
a proper calculation of added and removed blobs.

Just skipping the common subtrees is not possible as this would skew the
results if the added or removed blobs are shared with one of the
subtrees.
2020-09-13 16:28:46 +02:00
Michael Eischer
c0fc85d303 diff: Add integration test 2020-09-13 16:28:46 +02:00
Alexander Neumann
0c48e515f0 Merge pull request #2630 from MichaelEischer/fix-staticcheck
Fix lots of small issues reported by staticcheck
2020-09-13 16:19:22 +02:00
Alexander Neumann
97950ab81a options: Fix test for Go >= 1.15 2020-09-12 17:36:44 +02:00
Alexander Neumann
59fca85844 Remove darwin/386 from integration tests 2020-09-12 17:09:33 +02:00
Alexander Neumann
e207257714 Fix Appveyor 2020-09-12 16:59:03 +02:00
Alexander Neumann
82e1cbed4f Update Go versions 2020-09-12 16:52:21 +02:00
Alexander Neumann
8903b6c88a helpers: Rework list of architectures
This also removes darwin/386 which is not supported on Go 1.15 any more.
2020-09-12 16:39:41 +02:00
Alexander Neumann
93583c01b1 helpers: Fetch modules for release binaries 2020-09-12 16:25:36 +02:00
MichaelEischer
88664ba222 Merge pull request #2893 from MichaelEischer/restore-preallocate
restorer: Preallocate files
2020-09-08 22:43:05 +02:00
Michael Eischer
121233e1b3 Add preallocate tests 2020-09-07 21:41:47 +02:00
Michael Eischer
8cc9514879 restorer: pre-allocate files before loading chunks 2020-09-07 21:41:47 +02:00
Michael Eischer
2e7d475029 Process packs in order of first appearance 2020-09-06 21:25:35 +02:00
rawtaz
d3a286928a Merge pull request #2905 from MichaelEischer/fix-ls-help
ls: Explicitly enforce that the user specifies a snapshot ID
2020-09-05 10:44:01 +02:00
Michael Eischer
c46edcd9d6 error strings should not end with punctuation 2020-09-05 10:07:17 +02:00
Michael Eischer
1ede018ea6 error variable names should start with 'Err' 2020-09-05 10:07:17 +02:00
Michael Eischer
b77e933d80 Convert underscore variable names to camelCase 2020-09-05 10:07:16 +02:00
Michael Eischer
d0329cf3eb Adjust comments to match name of exported methods 2020-09-05 10:07:16 +02:00
Michael Eischer
dc31529fc3 Unindent else block after if block ending with a return statement 2020-09-05 10:07:16 +02:00
Michael Eischer
4784540f04 repository: Simplify worker group code 2020-09-05 10:07:16 +02:00
Michael Eischer
84ea2389ae archiver_test: Only skip symlinks on windows 2020-09-05 10:07:16 +02:00
Michael Eischer
b4a7ce86cf uint cannot be less than zero 2020-09-05 10:07:16 +02:00
Michael Eischer
7ee0964880 crypto: Remove unused error 2020-09-05 10:07:16 +02:00
Michael Eischer
f6f11400c2 restic: Remove unused variable value 2020-09-05 10:07:16 +02:00
Michael Eischer
e2dc5034d3 restic/file: Use FileType for all File constants
The data type of a variable is only carried on when using iota.
2020-09-05 10:07:16 +02:00
Michael Eischer
9a1b3cb5d9 restorer: Remove unused treeID struct member from test 2020-09-05 10:07:16 +02:00
Michael Eischer
b22655367c integration_test: Replace fprintf without format string 2020-09-05 10:07:16 +02:00
Michael Eischer
068a3ce23f Remove redundant return 2020-09-05 10:07:16 +02:00
Michael Eischer
ee05501ce7 archiver: Remove unused chmod function 2020-09-05 10:07:16 +02:00
Michael Eischer
014600bee6 archiver: Remove unused variable from test 2020-09-05 10:07:16 +02:00
Michael Eischer
d9a80e07b9 repository: Simplify index age calculation 2020-09-05 10:07:16 +02:00
Michael Eischer
d19f05c960 Use buf.String() instead of string(buf.Bytes()) 2020-09-05 10:07:16 +02:00
Michael Eischer
460e2ffbf6 Collapse a few boolean operations 2020-09-05 10:07:14 +02:00
Michael Eischer
49b6aac3fa Use !bytes.Equal instead of bytes.Compare !=0 to check for inequality 2020-09-05 10:06:23 +02:00
Michael Eischer
2f8335554c Remove a few unused variables 2020-09-05 10:06:23 +02:00
Michael Eischer
37113282ca Merge a few variable declaration and initializations 2020-09-05 10:05:34 +02:00
Michael Eischer
337725c354 Use sort.Strings 2020-09-05 10:05:34 +02:00
MichaelEischer
2ddb7ffb7e Merge pull request #2255 from Kidswiss/tar
Fix dumping issues with / and the first sub level
2020-09-01 21:52:17 +02:00
Michael Eischer
81dcfea11a dump: Mention tar output in command help text 2020-08-31 22:43:10 +02:00
MichaelEischer
55071ee367 Merge pull request #2859 from buschjost/stats-filter-by-tag-and-path
Add filter by tag and path to stats command
2020-08-31 22:11:01 +02:00
Michael Eischer
dcf9ded977 dump: Use slashes as path separators in tar on windows
This conversion is necessary as paths returned by filepath.Rel use the
operating-system native path separator
2020-08-30 18:26:50 +02:00
Michael Eischer
bcf44a9c3f dump: Properly handle file paths without / prefix
filepath.Rel failed if the requested path did not start with a `/` e.g.
`restic` instead of `/restic`.
2020-08-30 18:25:42 +02:00
Oliver Buschjost
a7b4c19abf Fix display of warning about ignored filters 2020-08-30 17:14:56 +02:00
Oliver Buschjost
d3fcfeba3a Add filter by tag and path to stats command 2020-08-30 17:14:48 +02:00
Michael Eischer
e69449bf2c dump: Add test for splitPath 2020-08-30 16:00:48 +02:00
Simon Beck
da4193c3ef Add changelog 2020-08-30 16:00:48 +02:00
Simon Beck
fe6445e0f4 dump: Added unit tests for tar output 2020-08-30 16:00:48 +02:00
MichaelEischer
ea81a0e282 Merge pull request #2606 from middelink/fix-323
Add copy functionality.
2020-08-30 10:18:24 +02:00
Simon Beck
80a11960dd dump: Always dump relative paths into tarballs
Tarballs should only contain relative paths.
2020-08-29 21:14:34 +02:00
Simon Beck
d6f739ec22 dump: Extract tar code to internal/dump 2020-08-29 21:14:34 +02:00
Simon Beck
b98598e55f dump: Fix dumping issues with / and the first sub level
There was an issue that prevented the dump command from working
correctly when either:

* `/` contained multiple nodes (e.g. `restic backup /`)
* dumping a file in the first sublevel was attempted (e.g. `/foo`)
2020-08-29 21:14:29 +02:00
Michael Eischer
d5f86effa1 ls: Explicitly enforce that the user specifies a snapshot ID
The help messages suggested that the `ls` command work without
explicitly passing a snapshot ID. However, this was never the case:
without a snapshot ID the command just failed with the error
`Ignoring "", it is not a snapshot id`.

Fixes #2299
2020-08-29 14:28:53 +02:00
MichaelEischer
c34c731698 Merge pull request #2702 from MichaelEischer/fix-backup-dir-count
backup: Fix reporting of directory count in summary
2020-08-29 11:29:21 +02:00
Michael Eischer
412623b848 copy: Reuse buffer for downloaded blobs 2020-08-29 10:48:44 +02:00
Michael Eischer
bbe8b73f03 Update help text of backup command in docs 2020-08-29 10:48:44 +02:00
Michael Eischer
91e8d998cd Add documentation for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer
9a4796594a integration tests: Fix checking of wrong snapshot 2020-08-29 10:48:44 +02:00
Michael Eischer
15374d22e9 integration tests: Add basic tests for copy command 2020-08-29 10:48:44 +02:00
Michael Eischer
88ad58d6cd integration tests: Redirect directory diff into intermediate buffer 2020-08-29 10:48:44 +02:00
Michael Eischer
591a8c4cdf integration tests: Deduplicate backup test-data setup code 2020-08-29 10:48:44 +02:00
Michael Eischer
ec9a53b7e8 copy: Mark and skip previously copied snapshots
Use the `Original` field of the copied snapshot to store a persistent
snapshot ID. This can either be the ID of the source snapshot if
`Original` was not yet set or the previous value stored in the
`Original` field. In order to still copy snapshots modified using the
tags command the source snapshot is compared to all snapshots in the
destination repository which have the same persistent ID. Snapshots are
only considered equal if all fields except `Original` and `Parent`
match. That way modified snapshots are still copied while avoiding
duplicate copies at the same time.
2020-08-29 10:48:44 +02:00
MichaelEischer
34a3adfd8d Merge pull request #2898 from aawsome/add-iexclude-file
backup: Add --iexclude-file option
2020-08-28 21:59:16 +02:00
Alexander Weiss
9867c4bbb4 Add flag --iexclude-file to backup 2020-08-28 21:04:41 +02:00
MichaelEischer
efb4a981cf Merge pull request #2904 from jasonccox/consistent-usage-strings
Update usage strings to put flags before args
2020-08-28 20:39:14 +02:00
Jason Cox
2447f3f110 Update usage strings to put flags before args
The standard UNIX-style ordering of command-line arguments places
optional flags before other positional arguments. All of restic's
commands support this ordering, but some of the usage strings showed the
flags after the positional arguments (which restic also parses just
fine). This change updates the doc strings to reflect the standard
ordering.

Because the `restic help` command comes directly from Cobra, there does
not appear to be a way to update the argument ordering in its usage
string, so it maintains the non-standard ordering (positional arguments
before optional flags).
2020-08-28 02:16:22 +00:00
Michael Eischer
b25978a53c backup: Fix reporting of directory count in summary
Previously the directory stats were reported immediately after calling
`SaveDir`. However, as the latter method saves the tree asynchronously
the stats were still initialized to their nil value. The stats are now
reported via a callback similar to the one used for the fileSaver.
2020-08-27 22:43:51 +02:00
Michael Eischer
b0a8c4ad6c copy: Only process each tree once
This speeds up copying multiple overlapping snapshots from one
repository to another, as we only have to copy the changed parts of
later snapshots.
2020-08-26 22:17:52 +02:00
Michael Eischer
908b23fda0 copy: Update for modernized repository interface 2020-08-26 22:17:52 +02:00
greatroar
4508d406ef copy: Remove separate SaveIndex in restic copy
Flush does this now.
2020-08-26 22:17:52 +02:00
Pauline Middelink
7048cc3e58 Add copy functionality
Add a copy command to copy snapshots between repositories. It allows the user
to specify a destination repository, password, password-file, password-command
or key-hint to supply the necessary details to open the destination repository.

You need to supply a list of snapshots to copy, snapshots which already exist
in the destination repository will be skipped.

Note, when using the network this becomes rather slow, as it needs to read the
blocks, decrypt them using the source key, then encrypt them again using the
destination key before finally writing them out to the destination repository.
2020-08-26 22:17:52 +02:00
MichaelEischer
eb7c00387c Merge pull request #2899 from MichaelEischer/fix-check-progress-crash
Unify progress bar of check and prune commands
2020-08-26 00:00:50 +02:00
Michael Eischer
bc0501d72c Add changelog 2020-08-25 23:33:10 +02:00
Michael Eischer
17995dec7a Unify progress bar of check and prune commands 2020-08-25 22:47:38 +02:00
MichaelEischer
e915cedc3d Merge pull request #2897 from restic/rawtaz-github-issue-feature
github: Add question to feature issue template
2020-08-24 23:52:38 +02:00
MichaelEischer
cdcaecd27d Merge pull request #2896 from restic/rawtaz-key-hint
doc: Add missing environment variables
2020-08-24 23:51:05 +02:00
rawtaz
b43ab67a22 doc: Add missing environment variables 2020-08-24 00:49:34 +02:00
rawtaz
7ddfd6cabe github: Add question to feature issue template 2020-08-24 00:22:44 +02:00
Alexander Neumann
b1b3f1ecb6 Merge pull request #2674 from MichaelEischer/prune-strict-checks
prune: Stricter error checks
2020-08-23 10:30:41 +02:00
rawtaz
fa135f72bf Merge pull request #2439 from htrendev/ignore-sync-on-macos
Ignore not supported error on sync() when using local backend
2020-08-22 01:38:49 +02:00
Hristo Trendev
51c22f4223 local backend: ignore not supported error on sync()
Closes #2395
2020-08-22 01:27:07 +02:00
rawtaz
1a5b66f33b Merge pull request #2879 from 0xMH/master
Added more explantation for --one-file-system option
2020-08-21 19:21:01 +02:00
NoNE
da6a34e044 doc: Add more explanation for --one-file-system option 2020-08-21 19:19:13 +02:00
Bruce Dillahunty
fe69b83074 Update key password prompt (#2847) 2020-08-19 21:42:08 +02:00
Michael Eischer
08d24ff99e prune: Include ID of all missing blobs in error message 2020-08-16 11:36:14 +02:00
Michael Eischer
d8b80e9862 Add changelog 2020-08-16 11:36:14 +02:00
Michael Eischer
1c84aceb39 prune: Test for abort on damaged repositories 2020-08-16 11:36:12 +02:00
Michael Eischer
575ed9a47e Test that rebuild-index errors when old index cannot be removed 2020-08-16 11:34:01 +02:00
Michael Eischer
8f811642c3 Add support for integration tests to wrap the backend 2020-08-16 11:34:01 +02:00
Michael Eischer
f4b9544ab2 prune: Add test that repack aborts on wrong blob 2020-08-16 11:34:01 +02:00
Michael Eischer
367449dede prune: Reduce memory allocations while repacking
The slicing operator `slice[low:high]` default to 0 for the lower bound and
len(slice) for the upper bound when either or both are not specified.
Fix the code to use `cap(slice)` to check for the slice capacity.
2020-08-16 11:34:01 +02:00
Michael Eischer
7042bafea5 prune: Abort repacking when a pack contains a wrong blob
If a blob in a pack file can be decrypted successfully but contains data
that results in a different hash than stated in the header pack, then
abort repacking. As both the pack header and the blob are
cryptographically verified this either means than a malicious entity
tampered with the backup or indicates hardware problems on the client.
prune should fail with an error in both cases.
2020-08-16 11:34:01 +02:00
Michael Eischer
744a15247d prune/rebuild-index: Fail if an old index cannot be removed
The old behavior was problematic in the context of rebuild-index as it
could leave old, possibly invalid index files behind without returning a
fatal error.

Prune calls rebuildIndex before removing any data from the repository.
For this use case failing to delete an old index MUST be treated as a
fatal error. Otherwise the index could still contain an old index file
that refers to blobs/packs that were later on deleted by prune. Later
backup runs will assume that the affected blobs already exist in the
repository which results in a backup which misses data.
2020-08-16 11:34:01 +02:00
Michael Eischer
3ba19869be prune: Abort if any used blobs are missing
The previous check only approximately verified whether all required
blobs were found. However, after forgetting a few snapshots the
repository contains lots of unused blobs whose number can be sufficient
to make up for missing packs.

When coupled with a malfunctioning backend that temporarily returns broken
data this could cause restic to regard the corresponding packs as
invalid and thereby delete data that's still in use. This change lets
restic play it safe and refuse to delete anything if data is missing.
2020-08-16 11:34:01 +02:00
aawsome
0fed6a8dfc Use "pack file" instead of "data file" (#2885)
- changed variable names, especially changed DataFile into PackFile
- changed in some comments
- always use "pack file" in docu
2020-08-16 11:16:38 +02:00
rawtaz
643bbbe156 doc: Correct wording for repository (#2884)
Replaces "backend" with "repository" to match restic output.
2020-08-11 22:30:17 +02:00
MichaelEischer
eca0f0ad24 Merge pull request #2863 from aawsome/index-no-duplicates
Don't save exact duplicates in merged index
2020-08-08 18:24:14 +02:00
MichaelEischer
08dee8a52b Merge pull request #2865 from greatroar/unused
Dead code removal
2020-08-08 16:15:28 +02:00
Alexander Weiss
b112533812 Don't save exact duplicates when merging indexes 2020-08-05 06:32:02 +02:00
Alexander Weiss
5e63294355 Add benchmark MasterIndexAlloc 2020-08-05 06:32:02 +02:00
MichaelEischer
84b6f1ec53 Merge pull request #2874 from MichaelEischer/fix-verbose
Adjust description of --verbose=n parameter
2020-08-04 23:21:05 +02:00
MichaelEischer
06fb4ea3f0 Merge pull request #2805 from jwilk-forks/verbose
Fix examples of --verbose with argument
2020-08-04 23:11:35 +02:00
Michael Eischer
e38d415173 Adjust description of --verbose=n parameter 2020-08-04 23:07:53 +02:00
greatroar
d81a396944 Dead code removal
Found by running golangci-lint on the entire code base.
2020-08-04 08:38:57 +02:00
rawtaz
0b21ec44b7 Merge pull request #2869 from josephrocca/patch-1
doc: --verbose 2 to --verbose=2
2020-08-04 01:06:58 +02:00
josephrocca
bd36731119 --verbose 2 to --verbose=2
`--verbose 2` seems to be incorrect here (gives an error/warning that "the `2` directory does not exist, skipping")
2020-08-04 08:39:43 +10:00
MichaelEischer
38a2f9c07b Merge pull request #2864 from greatroar/dump-error
Fix error handling in dump (err != err)
2020-08-03 20:09:42 +02:00
rawtaz
5af2815627 Merge pull request #2821 from renard/mount-nolock
cmd/mount: honur --no-lock flag
2020-08-03 20:09:26 +02:00
MichaelEischer
b55de2260d Merge pull request #2868 from MichaelEischer/fix-getusedblobs
prune: Stop progress bar after searching used blobs
2020-08-03 19:58:59 +02:00
Sébastien Gross
9be4fe3e84 cmd/mount: honour --no-lock flag
Do not lock the repository if --no-lock global flag is set. This allows
to mount repositories which are archived on a read only system.

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2020-08-03 19:45:39 +02:00
Michael Eischer
05116e4787 prune: Cleanup progress bar handling while repacking 2020-08-03 19:32:46 +02:00
Michael Eischer
04f79b9642 prune: Stop progress bar after searching used blobs 2020-08-03 19:31:49 +02:00
greatroar
8b358935a0 Fix error handling in dump (err != err) 2020-08-03 09:43:44 +02:00
MichaelEischer
66d089e239 Merge pull request #2841 from aawsome/optimize-getUsedBlobs
Extract get used blobs in prune as separate function
2020-08-01 22:40:33 +02:00
MichaelEischer
49d3efe547 Merge pull request #2840 from aawsome/delete-parallel
Make delete parallel
2020-08-01 22:26:15 +02:00
Alexander Weiss
9762bec091 Use optimized getUsedBlobs in prune 2020-08-01 21:07:31 +02:00
Alexander Weiss
0eb8553c87 add changelog for #2840 2020-08-01 20:43:18 +02:00
Alexander Weiss
d3692f5b81 Delete files in parallel in rebuild-index 2020-08-01 20:43:18 +02:00
Alexander Weiss
1c0b61204b Delete files in parallel in forget 2020-08-01 20:43:18 +02:00
Alexander Weiss
2ee654763b Delete files in parallel 2020-08-01 20:39:24 +02:00
Alexander Neumann
b7b479b668 Merge pull request #2599 from MichaelEischer/tweak-mem-usage
Reduce memory usage when searching for used blobs
2020-08-01 13:49:02 +02:00
Alexander Neumann
4cf9656f12 Merge pull request #2861 from MichaelEischer/fix-rclone-crash
rclone: Don't panic after unexpected subprocess exit
2020-08-01 13:08:50 +02:00
Alexander Neumann
2580eef2aa Merge pull request #2318 from classmarkets/2175-named-keys
Allow specifying user and host when adding keys
2020-08-01 13:06:31 +02:00
Michael Eischer
2d7ab9115f Add changelog entry 2020-08-01 12:29:16 +02:00
Michael Eischer
a178e5628e Remove duplicate TreeLoader interface 2020-08-01 12:29:16 +02:00
Michael Eischer
af66a62c04 FindUsedBlobs: Test that seen blobs are skipped
This also copies the TreeLoader interface from internal/walker to allow
stubbing the repository in the call to `FindUsedBlobs`.
2020-08-01 12:29:16 +02:00
Michael Eischer
9ea1a78bd4 FindUsedBlobs: Check for seen blobs before loading trees
The only effective change in behavior is that that toplevel nodes can
also be skipped.
2020-08-01 12:29:16 +02:00
Michael Eischer
184103647a FindUsedBlobs: merge seen into blobs BlobSet
The seen BlobSet always contained a subset of the entries in blobs.
Thus use blobs instead and avoid the memory overhead of the second set.

Suggested-by: Alexander Weiss <alex@weissfam.de>
2020-08-01 12:29:16 +02:00
Michael Eischer
c81b122374 rclone: Don't panic after unexpected subprocess exit
As the connection to the rclone child process is now closed after an
unexpected subprocess exit, later requests will cause the http2
transport to try to reestablish a new connection. As previously this never
should have happened, the connection called panic in that case. This
panic is now replaced with a simple error message, as it no longer
indicates an internal problem.
2020-08-01 12:17:40 +02:00
rawtaz
48f97f3567 Merge pull request #2857 from MichaelEischer/incomplete-backup-error
Don't print a stacktrace if some files could not be read
2020-07-29 00:06:25 +02:00
Michael Eischer
3ce9893e0b Don't print a stacktrace if some files could not be read 2020-07-28 23:52:48 +02:00
MichaelEischer
248c7c3828 Merge pull request #2587 from aawsome/optimize-fuse
Make `restic mount` faster and consume less memory
2020-07-28 23:19:17 +02:00
Alexander Weiss
f8316948d1 Optimize FUSE - make command restic mount faster and consume less memory
- Add Open() functionality to dir
- only access index for blobs when file is read
- Implement NodeOpener and put one-time file stuff there
- Add comment about locking as suggested by bazil.org/fuse

=> Thanks at Michael Eischer for suggesting the last two improvements
2020-07-28 23:01:18 +02:00
MichaelEischer
be54ceff66 Merge pull request #2855 from MichaelEischer/rclone-exit
Fix rclone subprocess handling
2020-07-26 22:55:25 +02:00
Michael Eischer
ea97ff1ba4 rclone: Skip crash test when rclone is not found 2020-07-26 12:06:18 +02:00
Michael Eischer
01b9581453 rclone: Better field names for stdio conn 2020-07-26 00:29:25 +02:00
Michael Eischer
3cd927d180 rclone: Give rclone time to finish before closing stdin pipe
Calling `Close()` on the rclone backend sometimes failed during test
execution with 'signal: Broken pipe'. The stdio connection closed both
the stdin and stdout file descriptors at the same moment, therefore
giving rclone no chance to properly send any final http2 data frames.

Now the stdin connection to rclone is closed first and will only be
forcefully closed after a timeout. In case rclone exits before the
timeout then the stdio connection will be closed normally.
2020-07-26 00:29:25 +02:00
greatroar
bf7b1f12ea rclone: Add test for pipe handling when rclone exits 2020-07-26 00:29:25 +02:00
Michael Eischer
8554332894 rclone: Close rclone side of stdio_conn pipes
restic did not notice when the rclone subprocess exited unexpectedly.

restic manually created pipes for stdin and stdout and used these for the
connection to the rclone subprocess. The process creating a pipe gets
file descriptors for the sender and receiver side of a pipe and passes
them on to the subprocess. The expected behavior would be that reads or
writes in the parent process fail / return once the child process dies
as a pipe would now just have a reader or writer but not both.

However, this never happened as restic kept the reader and writer
file descriptors of the pipes. `cmd.StdinPipe` and `cmd.StdoutPipe`
close the subprocess side of pipes once the child process was started
and close the parent process side of pipes once wait has finished. We
can't use these functions as we need access to the raw `os.File` so just
replicate that behavior.
2020-07-26 00:29:25 +02:00
greatroar
3e93b36ca4 Make rclone.New private 2020-07-26 00:28:45 +02:00
MichaelEischer
573a2fb240 Merge pull request #2789 from aawsome/fix-lookup
make Lookup() return all blobs
2020-07-25 21:32:23 +02:00
Michael Eischer
c847aace35 Rename Index interface to MasterIndex
The interface is now only implemented by repository.MasterIndex.
2020-07-25 21:19:46 +02:00
Alexander Weiss
9d1fb94c6c make Lookup() return all blobs
+ simplify syntax
2020-07-25 21:18:34 +02:00
MichaelEischer
020cab8e08 Merge pull request #2787 from greatroar/no-blobsize-cache
Remove blob size cache from restic mount
2020-07-25 20:46:35 +02:00
greatroar
07da61baee Remove blob size cache from restic mount 2020-07-25 19:39:14 +02:00
MichaelEischer
37c95bf5da Merge pull request #2835 from J0WI/patch-1
Update Go version to 1.14.6 in Docker build script
2020-07-25 17:23:54 +02:00
rawtaz
c86d2f23aa Merge pull request #2854 from restic/rawtaz-patch-1
Clarify questions and text in PR template
2020-07-25 15:41:27 +02:00
rawtaz
96ec04d74d Clarify questions and text in PR template 2020-07-25 15:38:00 +02:00
J0WI
9c3414374a Update Go version to 1.14.6 in Docker build script 2020-07-25 14:45:36 +02:00
MichaelEischer
3d530dfc91 Merge pull request #2827 from aawsome/archiver-test-contents
Make self-healing work when backing up with parent snapshot
2020-07-25 13:13:18 +02:00
MichaelEischer
c43f5b2664 Merge pull request #2733 from hibbert/support_irsa
Enable support for EKS IRSA
2020-07-25 12:58:43 +02:00
Andy Hibbert
38087e40d9 Update minio-go version so it supports EKS IRSA 2020-07-25 12:41:18 +02:00
MichaelEischer
bbc960f957 Merge pull request #2635 from greatroar/optimize-sortbycached
Optimize sorting blobs by cache status
2020-07-25 12:35:42 +02:00
greatroar
309598c237 Simplify sortCachedPacksFirst test in internal/repository
The test now uses the fact that the sort is stable. It's not guaranteed
to be, but the test is cleaner and more exhaustive. sortCachedPacksFirst
no longer needs a return value.
2020-07-25 12:12:59 +02:00
greatroar
03d23e6faa Speed up blob sorting in internal/repository
name                    old time/op    new time/op    delta
SortCachedPacksFirst-8     208µs ± 3%     186µs ± 3%  -10.74%  (p=0.000 n=10+8)

name                    old alloc/op   new alloc/op   delta
SortCachedPacksFirst-8     213kB ± 0%     139kB ± 0%  -34.62%  (p=0.000 n=10+10)

name                    old allocs/op  new allocs/op  delta
SortCachedPacksFirst-8     1.03k ± 0%     1.03k ± 0%   -0.19%  (p=0.000 n=10+10)
2020-07-25 12:12:59 +02:00
greatroar
b10acd2af7 Test and benchmark blob sorting in internal/repository 2020-07-25 12:12:58 +02:00
Alexander Weiss
9175795fdb Check contents in archiver
When backing up with a parent snapshot and the file is not changed, also
check if contents are still available in index.
2020-07-25 08:18:28 +02:00
rawtaz
5d8d70542f Merge pull request #2852 from MichaelEischer/drop-go-1.11
Drop support for Go version 1.11
2020-07-25 01:17:36 +02:00
Michael Eischer
7c23381a2b Drop support for Go version 1.11 2020-07-24 18:52:39 +02:00
MichaelEischer
34181b13a2 Merge pull request #2328 from MichaelEischer/no-repeated-checks
Fix duplicate tree checks within `restic check`
2020-07-22 22:08:02 +02:00
MichaelEischer
bcd47ec3a2 Merge pull request #2818 from aawsome/merge-index-chaining
Merge index (based on chaining index implementation)
2020-07-22 22:05:36 +02:00
Alexander Weiss
a666a6d576 Add tests and merge indexes in index benchmarks 2020-07-22 21:54:02 +02:00
Alexander Weiss
e388d962a5 Merge final indexes together for faster index access 2020-07-22 21:54:02 +02:00
Peter Schultz
758b44b9c0 gs: support authentication with access token
In the Google Cloud Storage backend, support specifying access tokens
directly, as an alternative to a credentials file. This is useful when
restic is used non-interactively by some other program that is already
authenticated and eliminates the need to store long lived credentials.

The access token is specified in the GOOGLE_ACCESS_TOKEN environment
variable and takes precedence over GOOGLE_APPLICATION_CREDENTIALS.
2020-07-22 16:23:03 +02:00
Alexander Weiss
3b7a3711e6 Add more realistic index benchmarks
+ reduce test size of BenchmarkMasterIndexLookupParallel
2020-07-21 07:18:20 +02:00
Michael Eischer
9b0e718852 checker: Test that blob types are not confused 2020-07-20 23:43:47 +02:00
MichaelEischer
82c908871d Merge pull request #2812 from greatroar/chaining
Chaining hash table for repository.Index
2020-07-20 23:29:36 +02:00
Michael Eischer
ddf0b8cd0b checker: Properly distinguish between data and tree blobs
If a data blob and a tree blob with the same ID (= same content) exist,
then the checker did not report a data or tree blob as unused when the
blob of the other type was still in use.
2020-07-20 22:58:39 +02:00
Michael Eischer
2d0c138c9b checker: Test that check only decodes trees once
The `DuplicateTree` flag is necessary to ensure that failures cannot be
swallowed. The old checker implementation ignores errors from LoadTree
if the corresponding tree was already checked.
2020-07-20 22:51:53 +02:00
Michael Eischer
ef325ffc02 checker: Cleanup error handling code
This change only moves code around but does not result in any change in
behavior.
2020-07-20 22:51:53 +02:00
Michael Eischer
0f67ae813a Add changelog 2020-07-20 22:51:53 +02:00
Michael Eischer
7a165f32a9 checker: Traverse trees in depth-first order
Backups traverse the file tree in depth-first order and saves trees on
the way back up. This results in tree packs filled in a way comparable
to the reverse Polish notation.  In order to check tree blobs in that
order, the treeFilter would have to delay the forwarding of tree nodes
until all children of it are processed which would complicate the
implementation.

Therefore do the next similar thing and traverse the tree in depth-first
order, but process trees already on the way down. The tree blob ids are
added in reverse order to the backlog, which is once again reverted when
removing the ids from the back of the backlog.
2020-07-20 22:51:53 +02:00
Michael Eischer
36c69e3ca7 checker: Unify blobs, processed trees and referenced blobs map
The blobRefs map and the processedTrees IDSet are merged to reduce the
memory usage. The blobRefs map now uses separate flags to track blob
usage as data or tree blob. This prevents skipping of trees whose
content is identical to an already processed data blob. A third flag
tracks whether a blob exists or not, which removes the need for the
blobs IDSet.
2020-07-20 22:51:47 +02:00
Michael Eischer
35d8413639 checker: Remove dead index map 2020-07-20 22:37:31 +02:00
Michael Eischer
c66a0e408c checker: Reduce cost of debug log
Avoid duplicate allocation of the Subtree list.
2020-07-20 22:37:31 +02:00
Michael Eischer
70f4c014ef checker: Decode identical tree nodes only once
Even though the checkTreeWorker skips already processed chunks,
filterTrees did queue the same tree blob on every occurence. This
becomes a serious performance bottleneck for larger number of snapshots
that cover mostly the same directories. Therefore decode a tree blob
exactly once.
2020-07-20 22:37:31 +02:00
Michael Eischer
f0d8710611 Add benchmark for checker scaling with snapshot count 2020-07-20 22:37:31 +02:00
MichaelEischer
bd3e280f6d Merge pull request #2546 from Summerdave/fix-backup-error-code
Return an error when errors occured during backup
2020-07-20 22:31:46 +02:00
MichaelEischer
2746dcdb5f Merge pull request #2845 from restic/update-chunker
Update chunker
2020-07-20 22:20:28 +02:00
David Sommer
5729d967f5 backup: Return exit status code 3 when failing to read source data
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).

This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:

 - 0 when the command was successful
 - 1 when there was a fatal error (no snapshot created)
 - 3 when some source data could not be read (incomplete snapshot created)
2020-07-20 22:19:01 +02:00
Alexander Neumann
f9f6124558 Update chunker 2020-07-20 22:07:02 +02:00
Alexander Neumann
8074879c5f Remove 'go generate' 2020-07-19 17:28:42 +02:00
greatroar
7bda28f31f Chaining hash table for repository.Index
These are faster to construct but slower to access. The allocation rate
is halved, the peak memory usage almost halved compared to standard map.

Benchmark results on linux/amd64, -benchtime=3s -count=20:

name                                            old time/op    new time/op    delta
PackerManager-8                                    178ms ± 0%     178ms ± 0%     ~     (p=0.231 n=20+20)
DecodeIndex-8                                      4.54s ± 0%     4.30s ± 0%   -5.20%  (p=0.000 n=18+17)
DecodeIndexParallel-8                              4.54s ± 0%     4.30s ± 0%   -5.22%  (p=0.000 n=19+18)
IndexHasUnknown-8                                 44.4ns ± 5%    50.5ns ±11%  +13.82%  (p=0.000 n=19+17)
IndexHasKnown-8                                   48.3ns ± 0%    51.5ns ±12%   +6.68%  (p=0.001 n=16+20)
IndexAlloc-8                                       758ms ± 1%     616ms ± 1%  -18.69%  (p=0.000 n=19+19)
IndexAllocParallel-8                               234ms ± 3%     204ms ± 2%  -12.60%  (p=0.000 n=20+18)
MasterIndexLookupSingleIndex-8                     122ns ± 0%     145ns ± 9%  +18.44%  (p=0.000 n=14+20)
MasterIndexLookupMultipleIndex-8                   369ns ± 2%     429ns ± 8%  +16.27%  (p=0.000 n=20+20)
MasterIndexLookupSingleIndexUnknown-8             68.4ns ± 5%    74.9ns ±13%   +9.47%  (p=0.000 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8            315ns ± 3%     369ns ±11%  +17.14%  (p=0.000 n=20+20)
MasterIndexLookupParallel/known,indices=5-8        743ns ± 1%     816ns ± 2%   +9.87%  (p=0.000 n=17+17)
MasterIndexLookupParallel/unknown,indices=5-8      238ns ± 1%     260ns ± 2%   +9.14%  (p=0.000 n=19+20)
MasterIndexLookupParallel/known,indices=10-8      1.01µs ± 3%    1.11µs ± 2%   +9.79%  (p=0.000 n=19+20)
MasterIndexLookupParallel/unknown,indices=10-8     222ns ± 0%     269ns ± 2%  +20.83%  (p=0.000 n=16+20)
MasterIndexLookupParallel/known,indices=20-8      1.06µs ± 2%    1.19µs ± 2%  +12.95%  (p=0.000 n=19+18)
MasterIndexLookupParallel/unknown,indices=20-8     413ns ± 1%     530ns ± 1%  +28.19%  (p=0.000 n=18+20)
SaveAndEncrypt-8                                  30.2ms ± 1%    30.4ms ± 0%   +0.71%  (p=0.000 n=19+19)
LoadTree-8                                         540µs ± 1%     576µs ± 1%   +6.73%  (p=0.000 n=20+20)
LoadBlob-8                                        5.64ms ± 0%    5.64ms ± 0%     ~     (p=0.883 n=18+17)
LoadAndDecrypt-8                                  5.93ms ± 0%    5.95ms ± 1%     ~     (p=0.247 n=20+19)
LoadIndex-8                                       25.1ms ± 0%    24.5ms ± 1%   -2.54%  (p=0.000 n=18+17)

name                                            old speed      new speed      delta
PackerManager-8                                  296MB/s ± 0%   296MB/s ± 0%     ~     (p=0.229 n=20+20)
SaveAndEncrypt-8                                 139MB/s ± 1%   138MB/s ± 0%   -0.71%  (p=0.000 n=19+19)
LoadBlob-8                                       177MB/s ± 0%   177MB/s ± 0%     ~     (p=0.890 n=18+17)
LoadAndDecrypt-8                                 169MB/s ± 0%   168MB/s ± 1%     ~     (p=0.227 n=20+19)

name                                            old alloc/op   new alloc/op   delta
PackerManager-8                                   91.8kB ± 0%    91.8kB ± 0%     ~     (p=0.772 n=12+19)
IndexAlloc-8                                       786MB ± 0%     400MB ± 0%  -49.04%  (p=0.000 n=20+18)
IndexAllocParallel-8                               786MB ± 0%     401MB ± 0%  -49.04%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                  21.0MB ± 0%    21.0MB ± 0%   +0.00%  (p=0.000 n=19+19)

name                                            old allocs/op  new allocs/op  delta
PackerManager-8                                    1.41k ± 0%     1.41k ± 0%     ~     (all equal)
IndexAlloc-8                                        977k ± 0%      907k ± 0%   -7.18%  (p=0.000 n=20+20)
IndexAllocParallel-8                                977k ± 0%      907k ± 0%   -7.17%  (p=0.000 n=19+15)
SaveAndEncrypt-8                                    73.0 ± 0%      73.0 ± 0%     ~     (all equal)
2020-07-19 13:58:22 +02:00
greatroar
255ba83c4b Parallel index benchmarks + benchmark optimizations
createRandomIndex was using the global RNG, which locks on every call
It was also using twice as many random numbers as necessary and doing
a float division in every iteration of the inner loop.

BenchmarkDecodeIndex was using too short an input, especially for a
parallel version. (It may now be using one that is a bit large.)

Results on linux/amd64, -benchtime=3s -count=20:

name                                     old time/op    new time/op       delta
PackerManager-8                             178ms ± 0%        178ms ± 0%           ~     (p=0.165 n=20+20)
DecodeIndex-8                              13.6µs ± 2%  4539886.8µs ± 0%  +33293901.38%  (p=0.000 n=20+18)
IndexHasUnknown-8                          44.4ns ± 7%       44.4ns ± 5%           ~     (p=0.873 n=20+19)
IndexHasKnown-8                            49.2ns ± 3%       48.3ns ± 0%         -1.86%  (p=0.000 n=20+16)
IndexAlloc-8                                802ms ± 1%        758ms ± 1%         -5.51%  (p=0.000 n=20+19)
MasterIndexLookupSingleIndex-8              124ns ± 1%        122ns ± 0%         -1.41%  (p=0.000 n=20+14)
MasterIndexLookupMultipleIndex-8            373ns ± 2%        369ns ± 2%         -1.13%  (p=0.001 n=20+20)
MasterIndexLookupSingleIndexUnknown-8      67.8ns ± 3%       68.4ns ± 5%           ~     (p=0.753 n=20+20)
MasterIndexLookupMultipleIndexUnknown-8     316ns ± 3%        315ns ± 3%           ~     (p=0.846 n=20+20)
SaveAndEncrypt-8                           30.5ms ± 1%       30.2ms ± 1%         -1.09%  (p=0.000 n=19+19)
LoadTree-8                                  527µs ± 1%        540µs ± 1%         +2.37%  (p=0.000 n=19+20)
LoadBlob-8                                 5.65ms ± 0%       5.64ms ± 0%         -0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                           7.07ms ± 2%       5.93ms ± 0%        -16.15%  (p=0.000 n=19+20)
LoadIndex-8                                32.1ms ± 2%       25.1ms ± 0%        -21.64%  (p=0.000 n=20+18)

name                                     old speed      new speed         delta
PackerManager-8                           296MB/s ± 0%      296MB/s ± 0%           ~     (p=0.159 n=20+20)
SaveAndEncrypt-8                          138MB/s ± 1%      139MB/s ± 1%         +1.10%  (p=0.000 n=19+19)
LoadBlob-8                                177MB/s ± 0%      177MB/s ± 0%         +0.21%  (p=0.000 n=19+18)
LoadAndDecrypt-8                          141MB/s ± 2%      169MB/s ± 0%        +19.24%  (p=0.000 n=19+20)

name                                     old alloc/op   new alloc/op      delta
PackerManager-8                            91.8kB ± 0%       91.8kB ± 0%           ~     (p=0.826 n=19+12)
IndexAlloc-8                                786MB ± 0%        786MB ± 0%         +0.01%  (p=0.000 n=20+20)
SaveAndEncrypt-8                           21.0MB ± 0%       21.0MB ± 0%         -0.00%  (p=0.012 n=20+19)

name                                     old allocs/op  new allocs/op     delta
PackerManager-8                             1.41k ± 0%        1.41k ± 0%           ~     (all equal)
IndexAlloc-8                                 977k ± 0%         977k ± 0%         +0.01%  (p=0.022 n=20+20)
SaveAndEncrypt-8                             73.0 ± 0%         73.0 ± 0%           ~     (all equal)
2020-07-19 13:58:05 +02:00
MichaelEischer
7dc200c593 Merge pull request #2832 from alrs/fix-fuse-test-err
internal/fuse: fix dropped test error
2020-07-16 22:28:20 +02:00
Lars Lehtonen
9ac90cf5cd internal/fuse: fix dropped test error 2020-07-12 21:42:31 -07:00
MichaelEischer
b84f5177cb Merge pull request #2790 from greatroar/fix-quadratic-read
Fix quadratic file reading in restic mount
2020-07-12 18:42:14 +02:00
greatroar
4cf1c8e8da Changelog entry for subquadratic reading in mount 2020-07-12 18:27:16 +02:00
greatroar
58719e1f47 Replace mount's per-file cache by a global LRU cache 2020-07-12 18:27:16 +02:00
greatroar
d42c169458 Fix quadratic file reading in restic mount 2020-07-12 18:27:16 +02:00
rawtaz
8598bb042b Merge pull request #2769 from darkdragon-001/patch-1
Update documentation for environment variables
2020-07-12 00:40:21 +02:00
darkdragon-001
c6b74962df Update 040_backup.rst
Changes proposed in #2763:

- Adding `RESTIC_CACHE_DIR` environment variables (introduced in #2425 for Unix and #2607 for Mac, Win).
- Adding used system-wide environment variables with links to the corresponding section.
2020-07-12 00:31:56 +02:00
MichaelEischer
2c72924ffb Merge pull request #2813 from greatroar/encrypt-benchmark
Fix repository_test.BenchmarkSaveAndEncrypt
2020-07-05 17:52:10 +02:00
greatroar
02bec13ef2 Fix repository_test.BenchmarkSaveAndEncrypt
The benchmark was actually testing the speed of index lookups.

name              old time/op    new time/op        delta
SaveAndEncrypt-8     101ns ± 2%    31505824ns ± 1%    +31311591.31%  (p=0.000 n=10+10)

name              old speed      new speed          delta
SaveAndEncrypt-8  41.7TB/s ± 2%       0.0TB/s ± 1%         -100.00%  (p=0.000 n=10+10)

name              old alloc/op   new alloc/op       delta
SaveAndEncrypt-8     1.00B ± 0%  20989508.40B ± 0%  +2098950740.00%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op      delta
SaveAndEncrypt-8      0.00             123.00 ± 0%            +Inf%  (p=0.000 n=10+9)

(The actual speed is ca. 131MiB/s.)
2020-07-05 17:41:42 +02:00
MichaelEischer
64976b1a4d Merge pull request #2815 from scelfo/patch-1
--hostname flag is deprecated
2020-06-29 22:04:56 +02:00
Tony Scelfo
6a607d6ded --hostname flag is deprecated
Update documentation to follow the deprecation warning "Flag --hostname has been deprecated, use --host" that is given on the latest build(s).
2020-06-29 12:53:57 -06:00
rawtaz
6fedf1a7f4 Merge pull request #2809 from lessfoobar/patch-2
Update 080_examples.rst
2020-06-26 00:12:05 +02:00
@lessfoobar
df946fd9f8 Update 080_examples.rst 2020-06-26 00:01:30 +02:00
Jakub Wilk
4e6a9767de Fix examples of --verbose with argument 2020-06-23 20:22:14 +02:00
rawtaz
1bc80c3c8d Merge pull request #2802 from andreaso/cachedir-https-links
Use https:// for linking CACHEDIR.TAG specs
2020-06-21 20:01:42 +02:00
Andreas Olsson
0fcef2ec23 Use https:// for linking CACHEDIR.TAG specs
That site might not have supported https:// when those links were
originally added. It does now.

Also dropping the _spec.html_ ending of the url, there being a `<link
rel="canonical" ...>` tag suggesting that that no longer being the
preferred address.
2020-06-21 19:36:26 +02:00
MichaelEischer
212607dc8a Merge pull request #2760 from greatroar/backend-benchmark
Fix backend benchmarks + a micro-optimization
2020-06-17 23:17:05 +02:00
greatroar
55c3a90a0d Clarify rclone-over-SSH docs
Also added a link to S. Ruderich's blog post explaining append-only
repos using rclone and SSH.
2020-06-17 15:22:20 +02:00
greatroar
190d8e2f51 Flatten backend.LimitedReadCloser structure
This inlines the io.LimitedReader into the LimitedReadCloser body to
achieve fewer allocations. Results on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadPartialFile-8           412µs ± 4%     413µs ± 4%    ~     (p=0.634 n=17+17)
Backend/BenchmarkLoadPartialFileOffset-8     455µs ±13%     441µs ±10%    ~     (p=0.072 n=20+18)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadPartialFile-8        10.2GB/s ± 3%  10.2GB/s ± 4%    ~     (p=0.817 n=16+17)
Backend/BenchmarkLoadPartialFileOffset-8  9.25GB/s ±12%  9.54GB/s ± 9%    ~     (p=0.072 n=20+18)

name                                      old alloc/op   new alloc/op   delta
Backend/BenchmarkLoadPartialFile-8            888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      888B ± 0%      872B ± 0%  -1.80%  (p=0.000 n=15+15)

name                                      old allocs/op  new allocs/op  delta
Backend/BenchmarkLoadPartialFile-8            18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
Backend/BenchmarkLoadPartialFileOffset-8      18.0 ± 0%      17.0 ± 0%  -5.56%  (p=0.000 n=15+15)
2020-06-17 13:11:45 +02:00
greatroar
f4cd2a7120 Make backend benchmarks fairer by removing checks
Checking whether the right data is returned takes up half the time in
some benchmarks. Results for local backend benchmarks on linux/amd64:

name                                      old time/op    new time/op    delta
Backend/BenchmarkLoadFile-8                 4.89ms ± 0%    2.72ms ± 1%   -44.26%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8           936µs ± 6%     439µs ±15%   -53.07%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8     940µs ± 1%     456µs ±10%   -51.50%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                     23.9ms ±14%    24.8ms ±41%      ~     (p=0.690 n=5+5)

name                                      old speed      new speed      delta
Backend/BenchmarkLoadFile-8               3.43GB/s ± 0%  6.16GB/s ± 1%   +79.40%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFile-8        4.48GB/s ± 6%  9.63GB/s ±14%  +114.78%  (p=0.008 n=5+5)
Backend/BenchmarkLoadPartialFileOffset-8  4.46GB/s ± 1%  9.22GB/s ±10%  +106.74%  (p=0.008 n=5+5)
Backend/BenchmarkSave-8                    706MB/s ±13%   698MB/s ±31%      ~     (p=0.690 n=5+5)
2020-06-17 13:11:45 +02:00
MichaelEischer
aba270df7e Merge pull request #2786 from aawsome/memory-list-blobs
Loop over index files for 'list blobs'
2020-06-15 22:27:29 +02:00
Alexander Weiss
b5543cff5d Loop over index files for 'list blobs'
=> reduces memory consumption a lot!
2020-06-14 17:25:34 +02:00
Alexander Neumann
285b5236c2 Merge pull request #2684 from MichaelEischer/cleanup-output
Cleanup command output code
2020-06-14 11:25:40 +02:00
Michael Eischer
bb1e258bb7 cat/stats: Get stdout from globalOptions 2020-06-14 11:08:11 +02:00
Michael Eischer
182655bc88 Replace fmt.Printf/Println/Fprintf with wrapper functions
cmd/restic/globals.go already provides Printf, Println and Warnf wrapper
which get their output streams from the globalOptions object. This
allows for stream replacements when testing.
2020-06-14 11:08:11 +02:00
MichaelEischer
74bc7141c1 Merge pull request #2781 from aawsome/reduce-index-memory
Reduce index memory
2020-06-14 10:58:28 +02:00
Alexander Weiss
1361341c58 don't save duplicate packIDs when using internal/repository/Index.Store 2020-06-14 07:56:24 +02:00
Alexander Weiss
ce4a2f4ca6 save packIDs and duplicates separately
A side remark to the definition of Index.blob:

Another possibility would have been to use:
blob       map[restic.BlobHandle]*indexEntry

This would have led to the following sizes:
key: 32 + 1 = 33 bytes
value: 8 bytes
indexEntry:  8 + 4 + 4 = 16 bytes
each packID: 32 bytes

To save N index entries, we would therefore have needed:
N * OF * (33 + 8) bytes + N * 16 + N * 32 bytes / BP = N * 82 bytes

More precicely, using a pointer instead of a direct entry is the better memory choice if:
OF * 8 bytes + entrysize < OF * entrysize <=> entrysize > 8 bytes * OF/(OF-1)
Under the assumption of OF=1.5, this means using pointers would have been the better choice
if sizeof(indexEntry) > 24 bytes.
2020-06-14 07:56:21 +02:00
Alexander Weiss
cf979e2b81 make offset and length uint32 2020-06-14 07:50:19 +02:00
Michael Eischer
d92e2c5769 simplify index code 2020-06-14 07:50:19 +02:00
Alexander Weiss
7419844885 add changelog, benchmark, memory calculation 2020-06-14 07:50:15 +02:00
rawtaz
1d66bb9e62 Merge pull request #2784 from MichaelEischer/minio-sha256-changelog
Add changelog for #2709
2020-06-13 18:33:54 +02:00
Michael Eischer
0b2c31b05b Add changelog for #2709 2020-06-13 18:13:36 +02:00
Alexander Weiss
d3c59d18e5 Fix inconsistency of saving/loading config file
Fix saving/loading config file: Always set ID to a zero ID.
2020-06-13 16:30:23 +02:00
MichaelEischer
dd7b4f54f5 Merge pull request #2709 from greatroar/minio-sha256
Use Minio's optimized SHA-256
2020-06-12 23:32:58 +02:00
MichaelEischer
6896c6449b Merge pull request #2779 from greatroar/archiver-comment
Fix up comment on archiver.BlobSaver.Save
2020-06-12 23:04:21 +02:00
MichaelEischer
735a8074d5 Merge pull request #2773 from aawsome/index-uploads+knownblobs
Fix non-intuitive repo behavior
2020-06-12 22:41:04 +02:00
Alexander Weiss
70347e95d5 disable index uploads for prune command
+ modifications of changelog
2020-06-12 09:24:38 +02:00
greatroar
0fa3091c78 Fix up comment on archiver.BlobSaver.Save 2020-06-11 13:40:30 +02:00
Alexander Weiss
91906911b0 Fix non-intuitive repository behavior
- The SaveBlob method now checks for duplicates.
- Moves handling of pending blobs to MasterIndex.
  -> also cleans up pending index entries when they are saved in the index
  -> when using SaveBlob no need to care about index any longer
- Always check for full index and save it when storing packs.
  -> removes the need of an index uploader
  -> also removes the verbose "uploaded intermediate index" messages
- The Flush method now also saves the index
- Fix race condition when checking and saving full/non-finalized indexes
2020-06-11 13:05:23 +02:00
MichaelEischer
fae7f78057 Merge pull request #2755 from greatroar/solaris-workaround
Remove Solaris build workaround in integration tests
2020-06-10 23:30:51 +02:00
MichaelEischer
ac9ec4b990 Merge pull request #2778 from MichaelEischer/fix-unlock-hint
Revive hint to the unlock command if a repository is locked
2020-06-10 21:26:38 +02:00
Michael Eischer
087c770161 Revive hint to the unlock command if a repository is locked
errors.Fatalf wraps a error and just keeps an error message as a string.
This prevents the `restic.IsAlreadyLocked(err)` check from working as
the error is no longer an ErrAlreadyLocked.

Just add an additional remark to the error using `errors.WithMessage`.
2020-06-10 20:58:56 +02:00
MichaelEischer
6856d1e422 Merge pull request #2749 from aawsome/fix-fullindex
Change condition for full index
2020-06-10 20:40:19 +02:00
Alexander Weiss
8c1261ff02 changed condition for full index 2020-06-07 22:00:49 +02:00
rawtaz
26704be17f Merge pull request #2776 from nunoperalta/patch-1
Fix #2774 - make unable to umount message clearer
2020-06-07 17:19:50 +02:00
nunoperalta
2c3360db98 Fix #2774 - make unable to umount message clearer 2020-06-07 15:57:39 +01:00
MichaelEischer
cba6ad8d8e Merge pull request #2711 from greatroar/update-x-net
Update golang.org/x/net for fewer allocations in http2
2020-05-25 20:43:29 +02:00
greatroar
2a3312ac35 Remove Solaris build workaround in integration tests
This was introduced in #1821 because of issues with Go 1.9, but that
compiler version is no longer supported.
2020-05-25 10:33:36 +02:00
greatroar
c35c4e0cbf Update golang.org/x/net for fewer allocations in http2
name                                          old allocs/op  new allocs/op  delta
BackendREST/BenchmarkLoadFile-8                  2.15k ± 0%     2.14k ± 0%  -0.27%  (p=0.002 n=20+20)
BackendREST/BenchmarkLoadPartialFile-8             587 ± 0%       582 ± 0%  -0.88%  (p=0.000 n=20+20)
BackendREST/BenchmarkLoadPartialFileOffset-8       584 ± 0%       579 ± 0%  -0.90%  (p=0.000 n=19+16)

The throughput isn't significantly higher or lower.
2020-05-24 16:39:17 +02:00
MichaelEischer
84475aa3a8 Merge pull request #2730 from greatroar/mount-build-tags
Simplify build tags for restic mount
2020-05-23 20:33:43 +02:00
MichaelEischer
f12f9ae240 Merge pull request #2741 from tbm/typos
Fix typos
2020-05-21 23:40:12 +02:00
Martin Michlmayr
5cc1760fdf Fix typos 2020-05-16 14:05:26 +08:00
MichaelEischer
32ac5486e9 Merge pull request #2732 from restic/rawtaz-update-contributing
Reword parts of Providing Patches in CONTRIBUTING.md
2020-05-12 21:37:08 +02:00
rawtaz
c4336978eb Reword parts of Providing Patches in CONTRIBUTION.md
Primarily makes the request that contributors create a new branch for their changes stand out more.
2020-05-12 14:03:57 +02:00
greatroar
649cbec6c5 Simplify build tags for restic mount
This command can only be built on Darwin, FreeBSD and Linux
(and if we upgrade bazil.org/fuse, only FreeBSD and Linux:
https://github.com/bazil/fuse/issues/224).

Listing the few supported operating systems explicitly here makes
porting restic to new platforms easier.
2020-05-12 11:30:41 +02:00
MichaelEischer
b17bd7f860 Merge pull request #2719 from greatroar/update-x-crypto
Update x/crypto to stop poly1305 segfaulting on ARM
2020-05-09 00:02:18 +02:00
greatroar
68f1e9c524 Update x/crypto to fix poly1305 on ARM
The broken poly1305 implementation for arm64 was removed, fixing the
segfault issue on arm64 (gh-2618). The version for amd64 has improved
performance, which shows up in the internal/repository benchmark:

name              old speed      new speed      delta
SaveAndEncrypt-8   110MB/s ± 2%   113MB/s ± 1%  +2.24%  (p=0.000 n=20+20)
2020-05-05 11:05:35 +02:00
MichaelEischer
1ee2306033 Merge pull request #2717 from MichaelEischer/fix-flaky-password-prompt
backup: Avoid race between password prompt and open repository message
2020-05-01 23:58:55 +02:00
Michael Eischer
c882a92cd6 backup: Avoid race between password prompt and open repository message
`term.Print` sends the output via a channel to a goroutine which
actually prints the message. This may race with the password prompt
printed by `OpenRepository` resulting in a missing prompt.
2020-05-01 23:42:16 +02:00
MichaelEischer
f54db5d796 Merge pull request #2713 from greatroar/unused
Move Index.FindBlob to tests
2020-05-01 20:41:59 +02:00
MichaelEischer
843e7f404e Merge pull request #2716 from MichaelEischer/revert-apfs-test-workaround
Revert "Darwin test fix: allow 1μs timestamp difference"
2020-05-01 18:13:38 +02:00
Michael Eischer
d465b5b9ad Revert "Darwin test fix: allow 1μs timestamp difference"
This reverts commit f3016a9096.
2020-05-01 17:11:30 +02:00
greatroar
9f7cd69f13 Move Index.FindBlob to tests 2020-04-29 10:57:01 +02:00
greatroar
f97a680887 Fix repository benchmarks
BenchmarkLoad{AndDecrypt,Blob} were spending between 38% and 50% of
their time measuring SHA-256 performance in their checks.
2020-04-28 07:57:29 +02:00
greatroar
42a3db05b0 Use Minio's optimized SHA-256
internal/repository benchmarks on an Intel i7-3770k:

name               old speed      new speed       delta
PackerManager-8     209MB/s ± 1%    291MB/s ± 1%  +38.94%  (p=0.008 n=5+5)
SaveAndEncrypt-8    112MB/s ± 1%    135MB/s ± 1%  +20.25%  (p=0.008 n=5+5)
2020-04-28 07:57:18 +02:00
MichaelEischer
070d43e290 Merge pull request #2704 from azak-azkaran/patch-1
Update 030_preparing_a_new_repo.rst
2020-04-23 20:48:34 +02:00
MichaelEischer
d4bd32a37e Merge pull request #2640 from greatroar/simplify-loadblob-usage
Simplify Repository.LoadBlob usage
2020-04-23 20:23:35 +02:00
greatroar
e7d7b85d59 Merge Repository.{LoadBlob,loadBlob}
Pushing the allocation logic down into the former loadBlob body means
that fewer allocations have to be performed:

name              old time/op    new time/op    delta
LoadTree-8           478µs ± 1%     481µs ± 2%    ~     (p=0.315 n=9+10)
LoadBlob-8          11.6ms ± 1%    11.6ms ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8    13.3ms ± 3%    13.3ms ± 3%    ~     (p=0.905 n=10+9)
LoadIndex-8         33.6ms ± 2%    33.2ms ± 1%  -1.15%  (p=0.028 n=10+9)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.1kB ± 0%  -0.23%  (p=0.000 n=10+10)
LoadBlob-8          2.28kB ± 0%    2.18kB ± 0%  -4.21%  (p=0.000 n=10+10)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%    ~     (all equal)
LoadIndex-8         5.22MB ± 0%    5.22MB ± 0%    ~     (p=0.631 n=10+10)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       651 ± 0%  -0.15%  (p=0.000 n=10+10)
LoadBlob-8            24.0 ± 0%      23.0 ± 0%  -4.17%  (p=0.000 n=10+10)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%    ~     (all equal)
LoadIndex-8          30.2k ± 0%     30.2k ± 0%    ~     (p=0.610 n=10+10)

name              old speed      new speed      delta
LoadBlob-8        86.4MB/s ± 1%  85.9MB/s ± 2%    ~     (p=0.393 n=10+10)
LoadAndDecrypt-8  75.4MB/s ± 3%  75.4MB/s ± 3%    ~     (p=0.858 n=10+9)
2020-04-23 10:04:20 +02:00
greatroar
be5a0ff59f Centralize buffer allocation and size checking in Repository.LoadBlob
Benchmark results for internal/repository:

name              old time/op    new time/op    delta
LoadTree-8           479µs ± 2%     478µs ± 1%   ~     (p=0.780 n=10+9)
LoadBlob-8          11.6ms ± 2%    11.6ms ± 1%   ~     (p=0.631 n=10+10)
LoadAndDecrypt-8    13.2ms ± 2%    13.3ms ± 3%   ~     (p=0.631 n=10+10)

name              old alloc/op   new alloc/op   delta
LoadTree-8          41.2kB ± 0%    41.2kB ± 0%   ~     (all equal)
LoadBlob-8          2.28kB ± 0%    2.28kB ± 0%   ~     (all equal)
LoadAndDecrypt-8    2.10MB ± 0%    2.10MB ± 0%   ~     (all equal)

name              old allocs/op  new allocs/op  delta
LoadTree-8             652 ± 0%       652 ± 0%   ~     (all equal)
LoadBlob-8            24.0 ± 0%      24.0 ± 0%   ~     (all equal)
LoadAndDecrypt-8      30.0 ± 0%      30.0 ± 0%   ~     (all equal)

name              old speed      new speed      delta
LoadBlob-8        86.2MB/s ± 2%  86.4MB/s ± 1%   ~     (p=0.594 n=10+10)
LoadAndDecrypt-8  75.7MB/s ± 2%  75.4MB/s ± 3%   ~     (p=0.617 n=10+10)
2020-04-23 10:04:20 +02:00
azak-azkaran
c5100d5632 Update 030_preparing_a_new_repo.rst
fixed markdown for wasabi region link
2020-04-23 09:24:19 +02:00
MichaelEischer
956a1b0f96 Merge pull request #2294 from BenWiederhake/debugsyms
build: Preserve debug symbols in debug build
2020-04-19 17:26:03 +02:00
Ben Wiederhake
fab626a3df build: Preserve debug symbols in debug and profile build
Signed-off-by: Ben Wiederhake <BenWiederhake.GitHub@gmx.de>
2020-04-19 14:48:40 +02:00
MichaelEischer
4f00564574 Merge pull request #2621 from greatroar/fixes
Some small fixes
2020-04-18 18:07:08 +02:00
MichaelEischer
f77477129f Merge pull request #2584 from greatroar/mount-cache-uid-gid
Cache uid and gid for top directories in internal/fuse
2020-04-18 17:45:14 +02:00
greatroar
2e31120f89 Remove unused argument to restic.fakeFile 2020-04-18 17:40:13 +02:00
greatroar
8fb2c0d3c1 Typo in crypto test name 2020-04-18 17:39:06 +02:00
greatroar
072cf7b02d Fix debug messages in internal/fuse 2020-04-18 17:39:06 +02:00
greatroar
df66daa5c9 Fix context usage in backend tests
Found by go vet. This is also the only complaint is has.
2020-04-18 17:39:06 +02:00
MichaelEischer
9790d8ce1c Merge pull request #2668 from MichaelEischer/fix-stats-blobs-crash
stats: Fix crash in blobs-per-file mode on missing blob
2020-04-18 17:21:26 +02:00
MichaelEischer
16710454f4 Merge pull request #2628 from MichaelEischer/one-element-pack-lists
cache: Don't sort one element pack lists
2020-04-18 17:09:06 +02:00
MichaelEischer
08ec6c9f17 Merge pull request #2677 from MichaelEischer/complain-about-invalid-indexes
rebuild_index: Report invalid packs that were ignored
2020-04-18 16:57:50 +02:00
MichaelEischer
7910ff4c0e Merge pull request #2648 from nairb774/iowritestring
termstatus: Use io.WriteString to output messages.
2020-04-18 13:48:42 +02:00
MichaelEischer
c4da9d1e90 Merge pull request #2638 from greatroar/no-close-in-packer
Don't Close in Packer.Finalize
2020-04-18 13:07:21 +02:00
rawtaz
3ed61987a2 Merge pull request #2695 from restic/rawtaz-doc-exclude-file-tilde
doc: Add note on tilde expansion in exclude files
2020-04-18 13:01:17 +02:00
rawtaz
9dba7a2577 doc: Add note on tilde expansion in exclude files
Explains to the reader that tilde expansion does not work in exclude files, and that they should instead use the $HOME variable.
2020-04-18 12:47:47 +02:00
MichaelEischer
a1352906e2 Merge pull request #2622 from greatroar/optimize-packer-manager
Fix PackerManager benchmark and optimize hashing.Writer
2020-04-18 12:46:34 +02:00
MichaelEischer
b7c0d4d8bf Merge pull request #2644 from greatroar/signal-notify-buffered
Make all signal.Notify channels buffered
2020-04-18 11:31:02 +02:00
MichaelEischer
f033850aa0 Merge pull request #2692 from MichaelEischer/fix-archiveraborttest
archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError
2020-04-13 18:36:48 +02:00
Michael Eischer
bdf7ba20cb archiver: Fix race condition triggered by TestArchiverAbortEarlyOnError
The Save methods of the BlobSaver, FileSaver and TreeSaver return early
on when the archiver is stopped due to an error. For that they select on
both the tomb.Dying() and context.Done() channels, which can lead to a
race condition when the tomb is killed due to an error: The tomb first
closes its Dying channel before canceling all child contexts.
Archiver.SaveDir only aborts its execution once the context was
canceled. When the tomb killing is paused between closing its Dying
channel and canceling the child contexts, this lets the
FileSaver/TreeSaver.Save methods return immediately, however, ScanDir
still reads further files causing the test case to fail.

As a killed tomb always cancels all child contexts and as the Savers
always use a context bound to the tomb, it is sufficient to just use
context.Done() as escape hatch in the Save functions. This fixes the
mismatch between SaveDir and Save.

Adjust the tests to use contexts bound to the tomb for all interactions
with the Savers.
2020-04-13 18:23:17 +02:00
rawtaz
3ee6b8ec63 Merge pull request #2689 from MichaelEischer/fix-background-hang
Fix shutdown hang when restic is started as background job
2020-04-12 22:39:44 +02:00
Michael Eischer
4a400f94bb Add changelog 2020-04-12 22:27:09 +02:00
Michael Eischer
1a1c572bac Fix shutdown hang when restic is started as background job
restic uses a cleanup hook to ensure that it restores the terminal
configuration to a sane state, when restic is interrupted while reading
a password from the terminal. However, this causes a problem, when
restic runs in a background job, as reconfiguring a terminal will cause
a SIGTTOU to be sent to restic pausing it. Therefore, restic seems to
hang on shutdown when it was running in the background.

This commit changes the behavior to only restore the terminal
configuration if restic was interrupted while reading a password from
the terminal. As reading a password from the terminal requires that
restic is in the foreground, this should avoid restic getting stopped.

Fixes #2298
Issue introduced in #402
2020-04-12 22:27:09 +02:00
rawtaz
5a7c27ddb6 Merge pull request #2681 from MichaelEischer/optimize-debug
Reduce memory usage and startup time of debug command
2020-04-04 00:08:14 +02:00
Michael Eischer
fb842759fc debug: don't load the repository index
The existing commands don't need a loaded repository index which can
take several minutes to load on larger repositories.
2020-04-04 00:01:01 +02:00
Michael Eischer
7aa2f8a61e debug: get stdout/stderr from gopts/globalOptions 2020-04-04 00:01:01 +02:00
Michael Eischer
08bf3bae79 debug: explicitly pass stdout to dump functions 2020-04-03 23:32:44 +02:00
rawtaz
e7b741b2d7 Merge pull request #2682 from MichaelEischer/cleanup-cli-paramter-names
Cleanup CLI parameter names for backup / global flags
2020-04-03 21:35:34 +02:00
Michael Eischer
6bee62e346 Update doc excerpts for --help
This adds some previously missing changes and the new paramters names
from the previous commit.
2020-04-03 19:49:06 +02:00
Michael Eischer
bc74cd3ae5 backup/global: Use proper name for command line argument parameters
Several paramters printed a generic "string" or "stringArray" name.
2020-04-03 19:49:04 +02:00
Michael Eischer
90243ed1c4 rebuild_index: Report invalid packs that were ignored 2020-04-02 22:38:31 +02:00
Michael Eischer
0ce81d88b6 stats: Fix crash in blobs-per-file mode on missing blob
In a damaged repository with a missing blob, the error message tried to
dereference the subtreeID field of the current node, which is a file
however. Said field is set to nil for a file thus causing a segfault
when dereferenced.

Fix this by using the actual parentTreeID.
2020-03-27 22:17:54 +01:00
rawtaz
c03bc88b29 Merge pull request #2669 from MichaelEischer/doc-cifs-linux-bug
doc: Warn about compatibility issues with CIFS and restic
2020-03-26 23:01:06 +01:00
Brian Atkinson
b8da7b1f4d termstatus: Use io.WriteString to output messages.
The previous implementation was repeating the implementation that is
found inside of io.WriteString. Simplify by making use of the stdlib's
implementation.
2020-03-26 14:55:00 -07:00
Michael Eischer
f1b4d97945 doc: Warn about compatibility issues with CIFS and restic
On Linux CIFS (SMB) seems to be incompatible with the async preemption
implementation of Go 1.14. CIFS seems not to restart syscalls (open,
read, chmod, readdir, ...) as expected by Go, which sets SA_RESTART for
its signal handler to have syscalls restarted automatically. This leads
to Go passing up lots of EINTR return codes to restic.

See https://github.com/restic/restic/issues/2659 for a detailed explanation.
2020-03-26 21:52:37 +01:00
Peter Schultz
90fc639a67 Allow specifying user and host when adding keys
The username and hostname for new keys can be specified with the new
--user and --host flags, respectively. The flags are used only by the
`key add` command and are otherwise ignored.

This allows adding keys with for a desired user and host without having
to run restic as that particular user on that particular host, making
automated key management easier.

Co-authored-by: James TD Smith <ahktenzero@mohorovi.cc>
2020-03-23 13:11:10 +01:00
rawtaz
2b5a6d255a Merge pull request #2660 from restic/rawtaz-delete-old-issue-template
Delete ISSUE_TEMPLATE.md (not used anymore)
2020-03-21 20:48:39 +01:00
rawtaz
f004dbe605 Delete ISSUE_TEMPLATE.md (not used anymore)
Nowadays the ISSUE_TEMPLATE/ directory and its files are used for this feature.
2020-03-21 20:30:19 +01:00
rawtaz
9efbe98879 Merge pull request #2623 from alrs/internal-restic-err-before-close
internal/restic: close os.File after checking for error
2020-03-18 22:23:20 +01:00
rawtaz
7d9300efca Merge pull request #2637 from greatroar/unused
Remove Go 1.5 compatibility code
2020-03-18 22:22:15 +01:00
rawtaz
c38aaaa768 Merge pull request #2652 from greatroar/upgrade-fuse
Upgrade bazil.org/fuse and remove restic mount --allow-root
2020-03-18 22:21:50 +01:00
greatroar
8941041355 Upgrade bazil.org/fuse to version that fixes FreeBSD opBmap bug 2020-03-18 10:00:39 +01:00
greatroar
18fee4806f Remove broken --allow-root from restic mount 2020-03-17 23:35:06 +01:00
greatroar
47d4d5bf1b Make all signal.Notify channels buffered 2020-03-12 20:59:39 +01:00
greatroar
74a64c47e4 Move testing logic to test file in internal/pack 2020-03-09 14:32:28 +01:00
greatroar
a23e9c86ba Remove closing logic from Packer.Finalize
The method only ever receives *hashing.Writers, which don't implement
io.Closer. These come from packerManager.findPacker and have their
actual writers closed in Repository.savePacker. Moving the closing logic
to hashing.Writer results in "file already closed" errors.
2020-03-09 14:31:45 +01:00
greatroar
4de12bf593 Remove restic.RandReader
math/rand.Rand has implemented Reader since Go 1.6. The repacking tests
are not deterministic, but they weren't before, either.
2020-03-09 10:00:28 +01:00
rawtaz
c542a509f0 Merge pull request #2633 from greatroar/fix-ssh-commandline
Revert "Put host last in SSH command line"
2020-03-08 17:29:41 +01:00
greatroar
8cf3bb8737 Revert "Put host last in SSH command line"
This reverts commit e1969d1e33.
2020-03-08 16:45:33 +01:00
Michael Eischer
b46cc6d57e repository: Don't sort one element pack lists
When loading a blob, restic first looks up pack files containing the
blob. To avoid unnecessary work an already cached pack file is preferred.
However, if there is only a single pack file to choose from (which is
the normal case) sorting the one-element list won't change anything.
Therefore avoid the unnecessary cache check in that case.
2020-03-07 10:26:06 +01:00
Lars Lehtonen
4a2156d3f0 internal/restic: close os.File after checking for error 2020-03-05 16:22:46 -08:00
greatroar
41fee11f66 Micro-optimization for hashing.Writer/PackerManager
name             old time/op    new time/op    delta
PackerManager-8     247ms ± 1%     246ms ± 1%  -0.43%  (p=0.001 n=18+18)

name             old speed      new speed      delta
PackerManager-8   213MB/s ± 1%   214MB/s ± 1%  +0.43%  (p=0.001 n=18+18)

name             old alloc/op   new alloc/op   delta
PackerManager-8    92.2kB ± 0%    91.5kB ± 0%  -0.82%  (p=0.000 n=19+20)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.43k ± 0%     1.41k ± 0%  -1.67%  (p=0.000 n=20+20)
2020-03-05 22:30:04 +01:00
greatroar
b592614061 Improve PackerManager benchmark
The previous benchmark spent much of its time allocating RNGs and
generating too many random numbers. It now spends 90% of its time
hashing and half of the rest writing to files.

name             old time/op    new time/op    delta
PackerManager-8     319ms ± 1%     247ms ± 1%  -22.48%  (p=0.000 n=20+18)

name             old speed      new speed      delta
PackerManager-8   143MB/s ± 1%   213MB/s ± 1%  +48.63%  (p=0.000 n=10+18)

name             old alloc/op   new alloc/op   delta
PackerManager-8     635kB ± 0%      92kB ± 0%  -85.48%  (p=0.000 n=10+19)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.64k ± 0%     1.43k ± 0%  -12.76%  (p=0.000 n=10+20)
2020-03-05 22:30:03 +01:00
greatroar
b7c3039eb2 Remove Go 1.5 compatibility code from PackerManager benchmark
This alone is enough to speed up the benchmark by ~10%.
2020-03-05 22:29:06 +01:00
Alexander Neumann
a307797c11 Merge pull request #2612 from restic/rawtaz-maintainer-pr
Add notes about maintainer edit access in PRs
2020-03-01 21:24:34 +01:00
rawtaz
a03f107144 Add note about allowing maintainers edits in PRs. 2020-03-01 21:12:11 +01:00
rawtaz
a141ab1bda Add maintaner edit checkbox to PR template 2020-03-01 21:05:52 +01:00
Alexander Neumann
52abec967f Merge pull request #2605 from middelink/fix-2604
Fix running tests on a SELinux enabled system
2020-03-01 20:25:28 +01:00
Pauline Middelink
2828a9c2b0 Fix running tests on a SELinux enabled system
Archivers TestMetadataChanged incorrectly clears the Extended Attributes
from the expected metadata of the temporary file. This is incorrect as on
SELinux enabled filesystem, as the kernel will automaticly add a SElinux
label. However, since ExtendedAttributes{} != ExtendedAttributes{nil} we
still need to clear them if there are no attributes found.
2020-03-01 20:23:22 +01:00
rawtaz
cec7d581f3 Merge pull request #2610 from restic/remove-vendor
Remove vendored dependencies
2020-03-01 20:22:45 +01:00
Alexander Neumann
12aa1e61da CI: Enable Go Module Proxy 2020-03-01 19:58:20 +01:00
Alexander Neumann
95da6c1c1d Merge pull request #2589 from greatroar/no-stable-sort
Replace sort.Stable by sort.Strings
2020-03-01 19:40:28 +01:00
Alexander Neumann
0c03a80fc4 Merge pull request #2592 from greatroar/sftp-ipv6
Support IPv6 in SFTP backend
2020-03-01 19:38:44 +01:00
Alexander Neumann
b1c77172c2 Add entry to changelog 2020-03-01 19:32:35 +01:00
Alexander Neumann
c0373cd307 Remove -mod=vendor from all documentation and code 2020-03-01 19:32:35 +01:00
Alexander Neumann
28121090c2 Remove vendor from CI tests 2020-03-01 19:32:35 +01:00
Alexander Neumann
44a57d66c3 Remove vendor from build scripts 2020-03-01 11:30:02 +01:00
Alexander Neumann
266f9dbe16 Remove vendor dir 2020-03-01 11:20:42 +01:00
greatroar
60e4a88f17 Changelog entry for SFTP w/ IPv6 addresses 2020-03-01 11:04:06 +01:00
Alexander Neumann
c50f91b7f9 Merge pull request #2602 from greatroar/no-bufpool
Remove sync.Pool from internal/repository
2020-03-01 10:50:57 +01:00
Alexander Neumann
e851d29565 Merge pull request #2608 from greatroar/simplify-termstatus
Simplify termstatus
2020-03-01 10:48:29 +01:00
Alexander Neumann
b67b7ebfe6 Merge pull request #2583 from greatroar/unused
Remove some unused or duplicated code
2020-03-01 10:46:17 +01:00
greatroar
751eba0e68 Remove unnecessary pipe checks in termstatus
canUpdateStatus has already determined that the file descriptor is not a
pipe.
2020-02-29 18:03:49 +01:00
greatroar
7447c44484 Use golang.org/x/sys/windows in termstatus
Some functionality is missing, but at least the types are all defined.
Replaced short, word, dword by their Go names to match the x/sys
convention.
2020-02-29 18:03:49 +01:00
greatroar
c8a672fa29 Remove code copy-pasted from x/crypto/ssh/terminal 2020-02-29 18:03:49 +01:00
greatroar
863ba76494 Drop mattn/go-isatty in favor of crypto/ssh/terminal 2020-02-29 18:03:47 +01:00
greatroar
8526cc6647 Remove sync.Pool from internal/repository
The pool was used improperly, causing more allocations to be
performed than without it.

name              old time/op    new time/op    delta
SaveAndEncrypt-8    36.8ms ± 2%    36.9ms ± 2%    ~     (p=0.218 n=10+10)

name              old speed      new speed      delta
SaveAndEncrypt-8   114MB/s ± 2%   114MB/s ± 2%    ~     (p=0.218 n=10+10)

name              old alloc/op   new alloc/op   delta
SaveAndEncrypt-8    21.1MB ± 0%    21.0MB ± 0%  -0.44%  (p=0.000 n=10+10)

name              old allocs/op  new allocs/op  delta
SaveAndEncrypt-8      79.0 ± 0%      77.0 ± 0%  -2.53%  (p=0.000 n=10+10)
2020-02-29 17:54:46 +01:00
rawtaz
694b7a17e7 Merge pull request #2607 from greatroar/cachedir
Honor RESTIC_CACHE_DIR on Mac and Windows
2020-02-28 20:41:32 +01:00
greatroar
5cd0bce452 Honor RESTIC_CACHE_DIR on Mac and Windows 2020-02-28 15:44:32 +01:00
rawtaz
58bd165253 Merge pull request #2581 from aawsome/multiple-hostnames
Allow multiple hostnames tags
2020-02-27 08:35:23 +01:00
rawtaz
65d3fb6b33 Merge pull request #2603 from greatroar/restorer-waitgroup
Fix unsafe sync.WaitGroup usage in restorer.fileRestorer
2020-02-27 00:30:59 +01:00
rawtaz
f165048172 Merge pull request #2582 from greatroar/no-go1.9
Remove Go 1.9 compatibility code
2020-02-27 00:15:06 +01:00
greatroar
de5516a90e Fix sync.WaitGroup usage in restorer.fileRestorer 2020-02-27 00:07:49 +01:00
greatroar
4f6fd9fb98 Remove remnant of Go 1.9 compatibility code from tests 2020-02-26 22:23:38 +01:00
Alexander Weiss
9a9101d144 Support specifying multiple host flags for various commands
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`,
`stats` and `tag` commands will now take into account multiple
`--host` and `-H` flags.
2020-02-26 22:17:59 +01:00
rawtaz
616f9499ae Merge pull request #2600 from restic/update-go
Update Go version to >= 1.11, add Go 1.14
2020-02-26 21:29:30 +01:00
Alexander Neumann
a40ac37550 Add changelog file 2020-02-26 20:55:12 +01:00
Alexander Neumann
99fd80a585 Remove all workarounds for Go < 1.11 2020-02-26 20:35:13 +01:00
Alexander Neumann
2464f7c4d1 Update Go versions, drop Go 1.10 2020-02-26 20:35:13 +01:00
rawtaz
b5c7778428 Merge pull request #2195 from ifedorenko/out-of-order-restore-no-progress
restorer: allow writing target file blobs out of order
2020-02-26 20:07:51 +01:00
Igor Fedorenko
c52198d12c restorer: go mod vendor; go mod tidy
Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2020-02-26 16:15:04 +01:00
Igor Fedorenko
f17ffa0283 restorer: Allow writing target file blobs out of order
Much simpler implementation that guarantees each required pack
is downloaded only once (and hence does not need to manage
pack cache). Also improves large file restore performance.

Signed-off-by: Igor Fedorenko <igor@ifedorenko.com>
2020-02-26 16:14:45 +01:00
greatroar
5e2afd91e7 Assert that archiver.Tree implements fmt.Stringer 2020-02-26 11:05:38 +01:00
greatroar
79b882e901 Merge duplicated readdir functionality
internal/archiver.readdir and internal/fs.ReadDir were unused.

internal/fs.ReadDirNames and internal/archiver.readdirnames were doing
nearly the same thing, except one sorted its output and opened with
fs.O_NOFOLLOW. Both were only used in internal/archiver.
2020-02-26 11:05:38 +01:00
greatroar
1b502fa9ef Cache uid and gid for top directories in internal/fuse 2020-02-24 10:46:09 +01:00
greatroar
e1969d1e33 Put host last in SSH command line
This is how the SSH manpage says the command line should look, and the
"--" prevents mistakes in hostnames from being interpreted as options.
2020-02-19 15:53:20 +01:00
greatroar
6ac6bca7a1 Support IPv6 in SFTP backend
The previous code was doing its own hostname:port splitting, which
caused IPv6 addresses to be misinterpreted.
2020-02-19 15:42:12 +01:00
greatroar
3a6feb0596 Replace sort.Stable by sort.Strings
Calling the slow, O(n lg² n) sort.Stable is equivalent to sort.Strings
for a slice of unique strings.
2020-02-18 19:41:06 +01:00
greatroar
2f8aa2ce30 Remove unused fs.FS from archiver.FileSaver 2020-02-18 10:39:14 +01:00
rawtaz
f2bf06a419 Merge pull request #2540 from dp-github/issue-2531
Count hard-linked files correctly in stats command
2020-02-15 21:06:46 +01:00
David Potter
71900248ab Correct #2531 (stats restore-size calculates hard links incorrectly) 2020-02-15 19:34:22 +01:00
David Potter
23055aaadf Correct #2537 (cmd_stats file counting issue) 2020-02-15 19:34:16 +01:00
rawtaz
8bf6a3af97 Merge pull request #2285 from curiousleo/bugfix/2281-json-output
Fix JSON Printf issues with format verbs
2020-02-15 02:00:57 +01:00
curiousleo
27d6e5e186 Add changelog file. 2020-02-15 01:32:43 +01:00
curiousleo
4214b1746e Ensure Print{,f,ln} use global stdout 2020-02-15 01:32:43 +01:00
curiousleo
f6f240573a Don't Printf already formatted output
Fixes #2281.
2020-02-15 01:32:43 +01:00
rawtaz
ecdf49679e Merge pull request #2579 from MichaelEischer/fix-flaky-archiver-test
Fix flaky TestArchiverAbortEarlyOnError
2020-02-15 01:29:37 +01:00
Michael Eischer
e1f722d266 archiver: Fix flaky TestArchiverAbortEarlyOnError
Each of the random test files was split into the same five blobs. The
test fails once the fifth blob is passed on to `SaveBlob`. That is for
certain interleavings of goroutine execution it would be possible for
the test to trigger the testErr just after storing the first file.

The fixed test uses a different file content for each of the nine files
and fails after writing the fourth blob. The file content is also small
enough to ensure that for each file only a single blob is saved. This
guarantees that the test cannot fail before reading the first four
files. FileReadConcurrency = 2 allows up to two files queued for
processing. Therefore the test can at most open the sixth file before it
has to save the fourth file / blob which triggers the testErr.
2020-02-14 23:16:13 +01:00
rawtaz
2d47381b1d Merge pull request #2545 from MichaelEischer/fix-racy-backup-json
Fix racy json output for backup command
2020-02-13 22:33:17 +01:00
Michael Eischer
294ca55967 Add changelog entry 2020-02-13 21:14:20 +01:00
Michael Eischer
78c518ccac Print backup summary after status output is shutdown 2020-02-13 21:14:20 +01:00
Michael Eischer
42a3292bcf Better name for jsonstatus package
internal/ui/jsonstatus and termstatus sound similar but are not related
in any way. Instead `internal/ui/backup` and `internal/ui/jsonstatus/status`
are the counterparts. Rename the latter to `internal/ui/json/backup` to
make this clear.
2020-02-13 21:14:20 +01:00
Michael Eischer
ef70a2fcb3 Fix mangled JSON output by backup command
jsonstatus wrote the JSON output without synchronization to the
stdio_wrapper which caused mangling between different status lines.

Use the Print and Error methods of termstatus instead which use a
central goroutine to synchronize output.
2020-02-13 21:14:20 +01:00
rawtaz
af20015725 Merge pull request #2287 from benchti/patch-1
doc: Correct parameter order in dump example
2020-02-13 01:54:10 +01:00
rawtaz
680a14afa1 Merge pull request #2530 from restic/fix-sftp-mkdirall
sftp: Use MkdirAll provided by the client
2020-02-13 01:12:59 +01:00
rawtaz
bf199e5ca7 Minor speling corection. 2020-02-13 00:47:09 +01:00
rawtaz
ae127a412d Merge pull request #2337 from rfjakob/favicon
docs: switch to gopher favicon
2020-02-13 00:14:33 +01:00
Jakob Unterwurzacher
5ae7e6f945 docs: switch to gopher favicon
The favicon on restic.readthedocs.io still contained the old
superman-style logo.

This changes it to the 32x32 gopher icon also used on https://forum.restic.net/ ,
just converted to .ico using Gimp.

Tested by building the documentation and opening index.html in Chrome.
The new favicon looks fine.
2020-02-13 00:09:37 +01:00
rawtaz
d8da9c4401 Merge pull request #2577 from alrs/fix-internal-errs
internal: Fix code and test dropped errors
2020-02-12 23:41:54 +01:00
rawtaz
daf3bfc882 Merge pull request #2461 from rigtorp/man-exit-status
Add documentation on exit status codes to man pages
2020-02-12 23:17:49 +01:00
Erik Rigtorp
94f4f13388 Add documentation on exit status codes to man pages
This is step one to start defining useful exit codes for all the commands.
2020-02-12 23:09:26 +01:00
rawtaz
4615bdfd70 Merge pull request #2576 from restic/update-chunker
Update the chunker
2020-02-12 22:41:36 +01:00
rawtaz
299f5971f2 Merge pull request #2560 from brualan/master
Two small improvements to code quality
2020-02-12 22:40:20 +01:00
rawtaz
7a1352ae87 Merge pull request #2570 from MichaelEischer/close-file-on-type-change
Close file if file type has changed after initial stat
2020-02-12 22:39:22 +01:00
Lars Lehtonen
72734d59b5 internal/archiver: fix dropped error 2020-02-12 13:37:37 -08:00
rawtaz
3679669aae Merge pull request #2574 from alrs/fix-dropped-cmd-err
cmd/restic: fix dropped error
2020-02-12 22:36:45 +01:00
Lars Lehtonen
3ed54e762e internal/backend/sftp: fix dropped test error 2020-02-12 13:36:21 -08:00
Lars Lehtonen
fea835b4e2 internal/repository: fix dropped test error 2020-02-12 13:33:54 -08:00
Lars Lehtonen
16b321b140 internal/restic: fix dropped test error 2020-02-12 13:32:45 -08:00
Alexander Neumann
b58dfda212 Add entry to changelog 2020-02-12 21:29:26 +01:00
Alexander Neumann
b229aa5cf1 Update the chunker 2020-02-12 21:29:26 +01:00
Alexander Neumann
81c0b031f9 Add link to the forum in 'new issue' dialog 2020-02-12 21:25:48 +01:00
Michael Eischer
760863e7f9 archiver: Fix TestRacyFileSwap on windows 2020-02-11 21:09:47 +01:00
Michael Eischer
a135699397 Close file if file type has changed after initial stat 2020-02-11 21:09:47 +01:00
Lars Lehtonen
94a4d45dfb cmd/restic: fix a dropped error 2020-02-11 09:19:03 -08:00
rawtaz
4de384087d Merge pull request #2575 from alrs/fix-travis
restic/internal/ui: fix gofmt nit that popped up in Go 1.13
2020-02-11 13:49:13 +01:00
Lars Lehtonen
57815d9cd6 restic/internal/ui: fix gofmt nit that popped up in Go 1.13 2020-02-10 11:06:47 -08:00
rawtaz
644673bcf2 Merge pull request #2573 from dhoffend/fix-processed-bytes
fix backup --json total_bytes_processed output
2020-02-10 13:35:55 +01:00
Daniel Hoffend
e7cdf2acbb fix backup --json total_bytes_processed output
Closes #2429
2020-02-09 02:02:45 +01:00
rawtaz
0f22f008ed Merge pull request #2552 from iambryancs/update-to-go1.13
Update Go version to 1.13.6 in Docker build script
2020-02-07 15:24:37 +01:00
rawtaz
c184da21d0 Merge pull request #2568 from flofeld/patch-1
corrected option s3.storage-class in documentation
2020-02-06 23:34:36 +01:00
Florian Feldmann
ef5efc6a82 corrected option s3.storage-class
Took me some time to figure it out
2020-02-06 22:53:47 +01:00
Alexander Bruyako
688014487b avoid "index out of range" error
in case of len(format) == 0, we will get an
"index out of range" error
Usage of strings.HasSuffix is allowing to avoid that
2020-01-27 19:24:42 +03:00
Alexander Bruyako
38ddfbc4d3 simpler error return 2020-01-27 18:41:46 +03:00
Alexander Bruyako
da48b925ff remove unnecessary error return
I was running "golangci-lint" and found this two warnings

internal/checker/checker.go:135:18: (*Checker).LoadIndex$3 - result 0 (error) is always nil (unparam)
        final := func() error {
                        ^
internal/repository/repository.go:457:18: (*Repository).LoadIndex$3 - result 0 (error) is always nil (unparam)
        final := func() error {
                        ^

It turns out that these functions are used only in "RunWorkers(...)",
which is used only two times in whole project right after this "final"
functions.
And because these "final" functions always return "nil", I've
descided, that it would be better to remove requriments for "final" func
to return error to avoid magick "return nil" at their end.
2020-01-27 18:28:21 +03:00
iambryancs
4bcd56fbae Update Go version to 1.13.6 in Docker build script 2020-01-17 02:13:11 +08:00
Alexander Neumann
6e9778ae0e Add changelog file 2020-01-06 21:46:22 +01:00
Alexander Neumann
63c67be908 Add hint abouth absolute sftp paths for Synology NAS 2020-01-06 21:37:57 +01:00
rawtaz
d70a4a9350 Merge pull request #2373 from vrusinov/issue-2372
Ignore username difference in TestMetadataChanged
2020-01-05 21:07:34 +01:00
Alexander Neumann
2cd9c7ef16 sftp: Use MkdirAll provided by the client
Closes #2518
2020-01-01 17:26:38 +01:00
Alexander Neumann
6ec5dc8016 Fix template for new version of calens
The `wrap` function has been renamed to `wrapIndent`
2020-01-01 11:58:21 +01:00
rawtaz
fe430a680a Merge pull request #2524 from cure/patch-1
Fix typos in documentation of FixTime
2019-12-22 16:36:23 +01:00
Ward Vandewege
69a0d0ee90 fix typos 2019-12-21 21:00:28 -05:00
rawtaz
4557881066 Merge pull request #2423 from streambinder/master
sftp: support user@domain parsing as user
2019-12-19 22:39:36 +01:00
streambinder
9b5d069ade doc: 030: sfpt: hint for domain-confined users 2019-12-19 13:15:37 +01:00
streambinder
c56cbfe95b changelog: add pull-2423 2019-12-19 13:15:37 +01:00
streambinder
97e5ce4344 internal: backend: sftp: support user@domain parsing as user 2019-12-19 13:15:37 +01:00
rawtaz
72bd467e23 Merge pull request #2271 from lorenzbausch/feature/find_timestamp
Display snapshot date when using `restic find`
2019-12-17 10:35:59 +01:00
Lorenz Bausch
d818b7618b Display respective snapshot date when using restic find 2019-12-17 07:51:12 +01:00
Matt Holt
1c3812a6f6 Merge pull request #2342 from rpsene/master
Add support for ppc64le
2019-12-16 11:50:52 -07:00
Rafael Peria de Sene
ec91b80f09 Add support for ppc64le
This commit adds support for cross-compiling Restic for ppc64le
as part of the release process.

Add a new file:   changelog/unreleased/issue-2277
Modifies:   helpers/build-release-binaries/main.go
Modifies:   run_integration_tests.go
2019-12-15 22:52:13 -03:00
rawtaz
18a336c154 Merge pull request #2510 from rawtaz/kastenhq-changelog-2390
Add changelog for #2390.
2019-12-10 23:31:20 +01:00
Eric Hamilton
d21a13f1ee Create issue-2390 2019-12-10 23:20:18 +01:00
rawtaz
e0eac30ee5 Merge pull request #2509 from restic/improve-check-docs
doc: Improve check documentation
2019-12-10 23:07:23 +01:00
rawtaz
fccb579471 doc: Improve check documentation
Make it clearer what the difference is between regular check and check with --read-data.
2019-12-10 21:15:59 +01:00
Alexander Neumann
6c700f95b5 Merge pull request #2358 from filippobottega/master 2019-11-29 20:27:30 +01:00
Filippo Bottega
95d070c147 FAQ: Add hint for antivirus setup
Add a tip to configure antivirus to exclude restic process.
2019-11-29 20:26:20 +01:00
rawtaz
da4473aba6 Merge pull request #2391 from kastenhq/refresh-lock-time
Update Lock.Time in lock.Refresh()
2019-11-26 21:44:29 +01:00
rawtaz
6e85a58045 Merge pull request #2465 from whs-dot-hk/fix-test-metadata-changed
Fix test metadata changed
2019-11-26 21:42:42 +01:00
rawtaz
476e2e8762 Merge pull request #2354 from HugoReeves/master
Doc Addition: Wasabi repo setup instructions
2019-11-25 13:56:49 +01:00
Alexander Neumann
246bb46e82 Correct CHANEGLOG.md 2019-11-24 15:18:25 +01:00
Alexander Neumann
fe0361ffcb Move changelog entry for #2179
For background on how this happened see
https://forum.restic.net/t/psa-attention-anyone-not-using-a-recent-master-ie-0-9-5-version-restic-will-not-detect-changed-files-if-mtime-is-reset-by-the-application-eg-ms-excel-and-xls-xlsx-files-on-macos/1772/8?u=fd0
2019-11-24 15:15:19 +01:00
rawtaz
952469473f Merge pull request #2486 from rawtaz/changelog-gitkeep
changelog: Add .gitkeep to persist unreleased/ folder
2019-11-24 14:59:39 +01:00
rawtaz
02108f202e Merge pull request #2487 from rawtaz/2469-fix-diff-stats
diff: Fix wrong bytes reported in diff stats (#2469)
2019-11-23 12:15:28 +01:00
Alexander Neumann
b02357f542 Fix message 2019-11-23 10:31:40 +01:00
Alexander Neumann
f2aeaef8f1 Improve questions 2019-11-23 10:19:08 +01:00
Alexander Neumann
547702d285 Remove coverage badge 2019-11-23 10:19:08 +01:00
rawtaz
599831f874 Merge pull request #2489 from restic/improve-forget-docs
Improve example for forget --keep-daily
2019-11-22 21:20:35 +01:00
Alexander Neumann
14c90d9e85 Improve example for forget --keep-daily
Following up on https://github.com/restic/restic/pull/2406
2019-11-22 20:44:50 +01:00
Alexander Neumann
c41bbb3761 Re-formulate the question in the issue template
Closes #2410
2019-11-22 20:39:47 +01:00
rawtaz
8a54a0f5ac Merge pull request #2488 from rawtaz/changelog-template
changelog: Fix typo in TEMPLATE
2019-11-22 18:23:02 +01:00
Leo R. Lundgren
c0a5530950 changelog: Fix typo in TEMPLATE. 2019-11-22 17:32:56 +01:00
Leo R. Lundgren
77ef92b95c diff: Fix wrong bytes reported in diff stats (#2469) 2019-11-22 17:26:01 +01:00
Leo R. Lundgren
69f75bbf70 changelog: Add .gitkeep to persist the unreleased/ folder. 2019-11-22 17:14:17 +01:00
Alexander Neumann
30519f01ff Set development version for 0.9.6 2019-11-22 16:19:03 +01:00
whs
7cacba0394 Assume WithAtime default to false 2019-11-06 16:38:46 +08:00
Eric Hamilton
1596d06f8e Update Lock.Time in lock.Refresh() 2019-09-04 11:38:35 -07:00
Vladimir Rusinov
db20c0b8d0 Ignore username difference in TestMetadataChanged.
In some (rare) cases "fake" UID 51234 may exist in a system running a
test. When this is the case, `cmp.Equal(want, node3)` will fail based on
difference between empty string and an actual username present in a
system.

Fixes github issue #2372
2019-08-13 22:25:00 +01:00
Hugo Reeves
3ca306d69a Doc Addition: Added Wasabi repo setup 2019-07-26 22:00:48 +12:00
Ben
5d272e5c08 Dump : parameter order in sample
Usage:
  restic dump [flags] snapshotID file
2019-05-28 18:20:52 +02:00
1805 changed files with 19783 additions and 513207 deletions

View File

@@ -1,27 +0,0 @@
<!--
Welcome! If you have a question or are unsure if you should open an issue,
please use the forum instead!
https://forum.restic.net
The forum is a better place for questions about restic or general suggestions
and topics, e.g. usage or documentation questions! This issue tracker is mainly
for tracking bugs and feature requests directly relating to the development of
the software itself, rather than the project.
Thanks for understanding, and for contributing to the project!
-->
Output of `restic version`
--------------------------
<!--
Please add the version of restic you're currently using here, this helps us
later to see what has changed in restic when we revisit this issue after some
time.
-->
Describe the issue
------------------

View File

@@ -83,8 +83,8 @@ Do you have an idea how to solve the issue?
Did restic help you or made you happy in any way?
-------------------------------------------------
Did restic help you today? Did it make you happy in any way?
------------------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!

View File

@@ -39,16 +39,16 @@ Please describe the feature you'd like us to add here.
-->
What are you trying to do?
--------------------------
What are you trying to do? What problem would this solve?
---------------------------------------------------------
<!--
This section should contain a brief description what you're trying to do, which
would be possible after implementing the new feature.
-->
Did restic help you or made you happy in any way?
-------------------------------------------------
Did restic help you today? Did it make you happy in any way?
------------------------------------------------------------
<!--
Answering this question is not required, but if you have anything positive to share, please do so here!

4
.github/ISSUE_TEMPLATE/config.yml vendored Normal file
View File

@@ -0,0 +1,4 @@
contact_links:
- name: restic forum
url: https://forum.restic.net
about: Please ask questions about using restic here, do not open an issue for questions.

View File

@@ -10,11 +10,11 @@ your time and add more commits. If you're done and ready for review, please
check the last box.
-->
What is the purpose of this change? What does it change?
--------------------------------------------------------
What does this PR change? What problem does it solve?
-----------------------------------------------------
<!--
Describe the changes here, as detailed as needed.
Describe the changes and their purpose here, as detailed as needed.
-->
Was the change discussed in an issue or in the forum before?
@@ -23,14 +23,15 @@ Was the change discussed in an issue or in the forum before?
<!--
Link issues and relevant forum posts here.
If this PR resolves an issue on GitHub, use "closes #1234" so that the issue is
closed automatically when this PR is merged.
If this PR resolves an issue on GitHub, use "Closes #1234" so that the issue
is closed automatically when this PR is merged.
-->
Checklist
---------
- [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches)
- [ ] I have enabled [maintainer edits for this PR](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork)
- [ ] I have added tests for all changes in this PR
- [ ] I have added documentation for the changes (in the manual)
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/TEMPLATE))

258
.github/workflows/tests.yml vendored Normal file
View File

@@ -0,0 +1,258 @@
name: test
on:
# run tests on push to master, but not when other branches are pushed to
push:
branches:
- master
# run tests for all pull requests
pull_request:
jobs:
test:
strategy:
matrix:
# list of jobs to run:
include:
- job_name: Windows
go: 1.15.x
os: windows-latest
- job_name: macOS
go: 1.15.x
os: macOS-latest
test_fuse: false
- job_name: Linux
go: 1.15.x
os: ubuntu-latest
test_cloud_backends: true
test_fuse: true
check_changelog: true
- job_name: Linux
go: 1.14.x
os: ubuntu-latest
test_fuse: true
- job_name: Linux
go: 1.13.x
os: ubuntu-latest
test_fuse: true
name: ${{ matrix.job_name }} Go ${{ matrix.go }}
runs-on: ${{ matrix.os }}
env:
GOPROXY: https://proxy.golang.org
steps:
- name: Set up Go ${{ matrix.go }}
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go }}
- name: Get programs (Linux/macOS)
run: |
echo "build Go tools"
go get github.com/restic/rest-server/...
echo "install minio server"
mkdir $HOME/bin
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/darwin-amd64/minio
else
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
fi
chmod 755 $HOME/bin/minio
echo "install rclone"
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-osx-amd64.zip
else
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-linux-amd64.zip
fi
unzip rclone.zip
cp rclone*/rclone $HOME/bin
chmod 755 $HOME/bin/rclone
rm -rf rclone*
# add $HOME/bin to path ($GOBIN was already added to the path by setup-go@v2)
echo $HOME/bin >> $GITHUB_PATH
if: matrix.os == 'ubuntu-latest' || matrix.os == 'macOS-latest'
- name: Get programs (Windows)
shell: powershell
run: |
$ProgressPreference = 'SilentlyContinue'
echo "build Go tools"
go get github.com/restic/rest-server/...
echo "install minio server"
mkdir $Env:USERPROFILE/bin
Invoke-WebRequest https://dl.minio.io/server/minio/release/windows-amd64/minio.exe -OutFile $Env:USERPROFILE/bin/minio.exe
echo "install rclone"
Invoke-WebRequest https://downloads.rclone.org/rclone-current-windows-amd64.zip -OutFile rclone.zip
unzip rclone.zip
copy rclone*/rclone.exe $Env:USERPROFILE/bin
# add $USERPROFILE/bin to path ($GOBIN was already added to the path by setup-go@v2)
echo $Env:USERPROFILE\bin >> $Env:GITHUB_PATH
echo "install tar"
cd $env:USERPROFILE
mkdir tar
cd tar
# install exactly these versions of tar and the libraries, other combinations might not work!
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/tar-1.13-1-bin.zip -OutFile tar.zip
unzip tar.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libintl-0.11.5-2-bin.zip -OutFile libintl.zip
unzip libintl.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libiconv-1.8-1-bin.zip -OutFile libiconv.zip
unzip libiconv.zip
# add $USERPROFILE/tar/bin to path
echo $Env:USERPROFILE\tar\bin >> $Env:GITHUB_PATH
if: matrix.os == 'windows-latest'
- name: Check out code
uses: actions/checkout@v2
- name: Build with build.go
run: |
go run build.go
- name: Run local Tests
env:
RESTIC_TEST_FUSE: ${{ matrix.test_fuse }}
run: |
go test -cover ./...
- name: Test cloud backends
env:
RESTIC_TEST_S3_KEY: ${{ secrets.RESTIC_TEST_S3_KEY }}
RESTIC_TEST_S3_SECRET: ${{ secrets.RESTIC_TEST_S3_SECRET }}
RESTIC_TEST_S3_REPOSITORY: ${{ secrets.RESTIC_TEST_S3_REPOSITORY }}
RESTIC_TEST_AZURE_ACCOUNT_NAME: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_NAME }}
RESTIC_TEST_AZURE_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_KEY }}
RESTIC_TEST_AZURE_REPOSITORY: ${{ secrets.RESTIC_TEST_AZURE_REPOSITORY }}
RESTIC_TEST_B2_ACCOUNT_ID: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_ID }}
RESTIC_TEST_B2_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_KEY }}
RESTIC_TEST_B2_REPOSITORY: ${{ secrets.RESTIC_TEST_B2_REPOSITORY }}
RESTIC_TEST_GS_REPOSITORY: ${{ secrets.RESTIC_TEST_GS_REPOSITORY }}
RESTIC_TEST_GS_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
GOOGLE_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64: ${{ secrets.RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 }}
RESTIC_TEST_OS_AUTH_URL: ${{ secrets.RESTIC_TEST_OS_AUTH_URL }}
RESTIC_TEST_OS_TENANT_NAME: ${{ secrets.RESTIC_TEST_OS_TENANT_NAME }}
RESTIC_TEST_OS_USERNAME: ${{ secrets.RESTIC_TEST_OS_USERNAME }}
RESTIC_TEST_OS_PASSWORD: ${{ secrets.RESTIC_TEST_OS_PASSWORD }}
RESTIC_TEST_OS_REGION_NAME: ${{ secrets.RESTIC_TEST_OS_REGION_NAME }}
RESTIC_TEST_SWIFT: ${{ secrets.RESTIC_TEST_SWIFT }}
# fail if any of the following tests cannot be run
RESTIC_TEST_DISALLOW_SKIP: "restic/backend/rest.TestBackendREST,\
restic/backend/sftp.TestBackendSFTP,\
restic/backend/s3.TestBackendMinio,\
restic/backend/rclone.TestBackendRclone,\
restic/backend/s3.TestBackendS3,\
restic/backend/swift.TestBackendSwift,\
restic/backend/b2.TestBackendB2,\
restic/backend/gs.TestBackendGS,\
restic/backend/azure.TestBackendAzure"
run: |
# prepare credentials for Google Cloud Storage tests in a temp file
export GOOGLE_APPLICATION_CREDENTIALS=$(mktemp --tmpdir restic-gcs-auth-XXXXXXX)
echo $RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
go test -cover -parallel 4 ./internal/backend/...
# only run cloud backend tests for pull requests from and pushes to our
# own repo, otherwise the secrets are not available
if: (github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository) && matrix.test_cloud_backends
- name: Check changelog files with calens
run: |
echo "install calens"
go get github.com/restic/calens
echo "check changelog files"
calens
if: matrix.check_changelog
cross_compile:
strategy:
# ATTENTION: the list of architectures must be in sync with helpers/build-release-binaries/main.go!
matrix:
# run cross-compile in two batches parallel so the overall tests run faster
targets:
- "linux/386 linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/mips linux/mipsle linux/mips64 linux/mips64le \
openbsd/386 openbsd/amd64"
- "freebsd/386 freebsd/amd64 freebsd/arm \
aix/ppc64 \
darwin/amd64 \
netbsd/386 netbsd/amd64 \
windows/386 windows/amd64 \
solaris/amd64"
env:
go: 1.15.x
GOPROXY: https://proxy.golang.org
runs-on: ubuntu-latest
name: Cross Compile for ${{ matrix.targets }}
steps:
- name: Set up Go ${{ env.go }}
uses: actions/setup-go@v2
with:
go-version: ${{ env.go }}
- name: Check out code
uses: actions/checkout@v2
- name: Install gox
run: |
go get github.com/mitchellh/gox
- name: Cross-compile with gox for ${{ matrix.targets }}
env:
GOFLAGS: "-trimpath"
GOX_ARCHS: "${{ matrix.targets }}"
run: |
mkdir build-output
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}" ./cmd/restic
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -tags debug -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}_debug" ./cmd/restic
lint:
name: lint
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v2
- name: golangci-lint
uses: golangci/golangci-lint-action@v2
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.36
# Optional: show only new issues if it's a pull request. The default value is `false`.
only-new-issues: true
args: --verbose --timeout 5m
# only run golangci-lint for pull requests, otherwise ALL hints get
# reported. We need to slowly address all issues until we can enable
# linting the master branch :)
if: github.event_name == 'pull_request'
- name: Check go.mod/go.sum
run: |
echo "check if go.mod and go.sum are up to date"
go mod tidy
git diff --exit-code go.mod go.sum

1
.gitignore vendored
View File

@@ -1,2 +1,3 @@
/restic
/.vagrant
/.vscode

57
.golangci.yml Normal file
View File

@@ -0,0 +1,57 @@
# This is the configuration for golangci-lint for the restic project.
#
# A sample config with all settings is here:
# https://github.com/golangci/golangci-lint/blob/master/.golangci.example.yml
linters:
# only enable the linters listed below
disable-all: true
enable:
# make sure all errors returned by functions are handled
- errcheck
# find unused code
- deadcode
# show how code can be simplified
- gosimple
# # make sure code is formatted
- gofmt
# examine code and report suspicious constructs, such as Printf calls whose
# arguments do not align with the format string
- govet
# make sure names and comments are used according to the conventions
- golint
# detect when assignments to existing variables are not used
- ineffassign
# run static analysis and find errors
- staticcheck
# find unused variables, functions, structs, types, etc.
- unused
# find unused struct fields
- structcheck
# find unused global variables
- varcheck
# parse and typecheck code
- typecheck
issues:
# don't use the default exclude rules, this hides (among others) ignored
# errors from Close() calls
exclude-use-default: false
# list of things to not warn about
exclude:
# golint: do not warn about missing comments for exported stuff
- exported (function|method|var|type|const) `.*` should have comment or be unexported
# golint: ignore constants in all caps
- don't use ALL_CAPS in Go names; use CamelCase

View File

@@ -1,2 +0,0 @@
go:
enabled: true

View File

@@ -1,66 +0,0 @@
language: go
sudo: false
matrix:
include:
- os: linux
go: "1.10.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: linux
go: "1.11.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: linux
go: "1.12.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
# only run fuse and cloud backends tests on Travis for the latest Go on Linux
- os: linux
go: "1.13.x"
sudo: true
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: osx
go: "1.13.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/Library/Caches/go-build
- $HOME/gopath/pkg/mod
branches:
only:
- master
notifications:
irc:
channels:
- "chat.freenode.net#restic"
on_success: change
on_failure: change
skip_join: true
install:
- go version
- export GOBIN="$GOPATH/bin"
- export PATH="$PATH:$GOBIN"
- go env
script:
- go run run_integration_tests.go

File diff suppressed because it is too large Load Diff

View File

@@ -16,7 +16,8 @@ help also.
The restic project uses the GitHub infrastructure (see the
[project page](https://github.com/restic/restic)) for all related discussions
as well as the `#restic` channel on `irc.freenode.net`.
as well as the [forum](https://forum.restic.net/) and the `#restic` channel
on [irc.freenode.net](https://kiwiirc.com/nextclient/irc.freenode.net/restic).
If you want to find an area that currently needs improving have a look at the
open issues listed at the
@@ -25,7 +26,10 @@ for discussing enhancement to the restic tools.
If you are unsure what to do, please have a look at the issues, especially
those tagged
[minor complexity](https://github.com/restic/restic/labels/minor%20complexity).
[minor complexity](https://github.com/restic/restic/labels/help%3A%20minor%20complexity)
or [good first issue](https://github.com/restic/restic/labels/help%3A%20good%20first%20issue).
If you are already a bit experienced with the restic internals, take a look
at the issues tagged as [help wanted](https://github.com/restic/restic/labels/help%3A%20wanted).
Reporting Bugs
@@ -46,15 +50,12 @@ Remember, the easier it is for us to reproduce the bug, the earlier it will be
corrected!
In addition, you can compile restic with debug support by running
`go run -mod=vendor build.go -tags debug` and instructing it to create a debug
`go run build.go -tags debug` and instructing it to create a debug
log by setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
$ export DEBUG_LOG=/tmp/restic-debug.log
$ restic backup ~/work
For Go < 1.11, you need to remove the `-mod=vendor` option from the build
command.
Please be aware that the debug log file will contain potentially sensitive
things like file and directory names, so please either redact it before
uploading it somewhere or post only the parts that are really relevant.
@@ -63,16 +64,11 @@ uploading it somewhere or post only the parts that are really relevant.
Development Environment
=======================
The repository contains several sets of directories with code: `cmd/` and
`internal/` contain the code written for restic, whereas `vendor/` contains
copies of libraries restic depends on. The libraries are managed with the
command `go mod vendor`.
The repository contains the code written for restic in the directories
`cmd/` and `internal/`.
Go >= 1.11
----------
For Go version 1.11 or later, you should clone the repo (without having
`$GOPATH` set) and `cd` into the directory:
Restic requires Go version 1.13 or later for compiling. Clone the repo (without
having `$GOPATH` set) and `cd` into the directory:
$ unset GOPATH
$ git clone https://github.com/restic/restic
@@ -82,40 +78,12 @@ Then use the `go` tool to build restic:
$ go build ./cmd/restic
$ ./restic version
restic 0.9.2-dev (compiled manually) compiled with go1.11 on linux/amd64
restic 0.10.0-dev (compiled manually) compiled with go1.15.2 on linux/amd64
You can run all tests with the following command:
$ go test ./...
Go < 1.11
---------
In order to compile restic with Go before 1.11, it needs to be checked out at
the right path within a `GOPATH`. The concept of a `GOPATH` is explained in
["How to write Go code"](https://golang.org/doc/code.html).
If you do not have a directory with Go code yet, executing the following
instructions in your shell will create one for you and check out the restic
repo:
$ export GOPATH="$HOME/go"
$ mkdir -p "$GOPATH/src/github.com/restic"
$ cd "$GOPATH/src/github.com/restic"
$ git clone https://github.com/restic/restic
$ cd restic
You can then build restic as follows:
$ go build ./cmd/restic
$ ./restic version
restic compiled manually
compiled with go1.8.3 on linux/amd64
The following commands can be used to run all the tests:
$ go test ./...
Providing Patches
=================
@@ -128,20 +96,19 @@ down to the following steps:
GitHub. For a new feature, please add an issue before starting to work on
it, so that duplicate work is prevented.
1. First we would kindly ask you to fork our project on GitHub if you haven't
done so already.
1. Next, fork our project on GitHub if you haven't done so already.
2. Clone the repository locally and create a new branch. If you are working on
the code itself, please set up the development environment as described in
the previous section.
2. Clone your fork of the repository locally and **create a new branch** for
your changes. If you are working on the code itself, please set up the
development environment as described in the previous section.
3. Then commit your changes as fine grained as possible, as smaller patches,
that handle one and only one issue are easier to discuss and merge.
3. Commit your changes to the new branch as fine grained as possible, as
smaller patches, for individual changes, are easier to discuss and merge.
4. Push the new branch with your changes to your fork of the repository.
5. Create a pull request by visiting the GitHub website, it will guide you
through the process.
through the process. Please [allow edits from maintainers](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
6. You will receive comments on your code and the feature or bug that they
address. Maybe you need to rework some minor things, in this case push new
@@ -149,20 +116,19 @@ down to the following steps:
existing commit, use common sense to decide which is better), they will be
automatically added to the pull request.
7. If your pull request changes anything that users should be aware
of (a bugfix, a new feature, ...) please add an entry as a new
file in `changelog/unreleased` including the issue number in the
filename (e.g. `issue-8756`). Use the template in
`changelog/TEMPLATE` for the content. It will be used in the
announcement of the next stable release. While writing, ask
yourself: If I were the user, what would I need to be aware of
with this change.
7. If your pull request changes anything that users should be aware of
(a bugfix, a new feature, ...) please add an entry as a new file in
`changelog/unreleased` including the issue number in the filename (e.g.
`issue-8756`). Use the template in `changelog/TEMPLATE` for the content.
It will be used in the announcement of the next stable release. While
writing, ask yourself: If I were the user, what would I need to be aware
of with this change?
8. Once your code looks good and passes all the tests, we'll merge it. Thanks
a lot for your contribution!
Please provide the patches for each bug or feature in a separate branch and
open up a pull request for each.
open up a pull request for each, as this simplifies discussion and merging.
The restic project uses the `gofmt` tool for Go source indentation, so please
run
@@ -175,6 +141,14 @@ Installing the script `fmt-check` from https://github.com/edsrzf/gofmt-git-hook
locally as a pre-commit hook checks formatting before committing automatically,
just copy this script to `.git/hooks/pre-commit`.
The project is using the program
[`golangci-lint`](https://github.com/golangci/golangci-lint) to run a list of
linters and checkers. It will be run on the code when you submit a PR. In order
to check your code beforehand, you can run `golangci-lint run` manually.
Eventually, we will enable `golangci-lint` for the whole code base. For now,
you can ignore warnings printed for lines you did not modify, those will be
ignored by the CI.
For each pull request, several different systems run the integration tests on
Linux, macOS and Windows. We won't merge any code that does not pass all tests
for all systems, so when a tests fails, try to find out what's wrong and fix

View File

@@ -3,7 +3,7 @@
all: restic
restic:
go run -mod=vendor build.go || go run build.go
go run build.go
clean:
rm -f restic

113
README.md Normal file
View File

@@ -0,0 +1,113 @@
[![Documentation](https://readthedocs.org/projects/restic/badge/?version=latest)](https://restic.readthedocs.io/en/latest/?badge=latest)
[![Build Status](https://github.com/restic/restic/workflows/test/badge.svg)](https://github.com/restic/restic/actions?query=workflow%3Atest)
[![Go Report Card](https://goreportcard.com/badge/github.com/restic/restic)](https://goreportcard.com/report/github.com/restic/restic)
# Introduction
restic is a backup program that is fast, efficient and secure. It supports the three major operating systems (Linux, macOS, Windows) and a few smaller ones (FreeBSD, OpenBSD).
For detailed usage and installation instructions check out the [documentation](https://restic.readthedocs.io/en/latest).
You can ask questions in our [Discourse forum](https://forum.restic.net).
Quick start
-----------
Once you've [installed](https://restic.readthedocs.io/en/latest/020_installation.html) restic, start
off with creating a repository for your backups:
$ restic init --repo /tmp/backup
enter password for new backend:
enter password again:
created restic backend 085b3c76b9 at /tmp/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
and add some data:
$ restic --repo /tmp/backup backup ~/work
enter password for repository:
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
[0:29] 100.00% 54.732 MiB/s 1.582 GiB / 1.582 GiB 2580 / 2580 items 0 errors ETA 0:00
duration: 0:29, 54.47MiB/s
snapshot 40dc1520 saved
Next you can either use `restic restore` to restore files or use `restic
mount` to mount the repository via fuse and browse the files from previous
snapshots.
For more options check out the [online documentation](https://restic.readthedocs.io/en/latest/).
# Backends
Saving a backup on the same machine is nice but not a real backup strategy.
Therefore, restic supports the following backends for storing backups natively:
- [Local directory](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local)
- [sftp server (via SSH)](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp)
- [HTTP REST server](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) ([protocol](doc/100_references.rst#rest-backend), [rest-server](https://github.com/restic/rest-server))
- [AWS S3](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) (either from Amazon or using the [Minio](https://minio.io) server)
- [OpenStack Swift](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift)
- [BackBlaze B2](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2)
- [Microsoft Azure Blob Storage](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage)
- [Google Cloud Storage](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage)
- And many other services via the [rclone](https://rclone.org) [Backend](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone)
# Design Principles
Restic is a program that does backups right and was designed with the
following principles in mind:
- **Easy:** Doing backups should be a frictionless process, otherwise
you might be tempted to skip it. Restic should be easy to configure
and use, so that, in the event of a data loss, you can just restore
it. Likewise, restoring data should not be complicated.
- **Fast**: Backing up your data with restic should only be limited by
your network or hard disk bandwidth so that you can backup your files
every day. Nobody does backups if it takes too much time. Restoring
backups should only transfer data that is needed for the files that
are to be restored, so that this process is also fast.
- **Verifiable**: Much more important than backup is restore, so restic
enables you to easily verify that all data can be restored.
- **Secure**: Restic uses cryptography to guarantee confidentiality and
integrity of your data. The location the backup data is stored is
assumed not to be a trusted environment (e.g. a shared space where
others like system administrators are able to access your backups).
Restic is built to secure your data against such attackers.
- **Efficient**: With the growth of data, additional snapshots should
only take the storage of the actual increment. Even more, duplicate
data should be de-duplicated before it is actually written to the
storage back end to save precious backup space.
# Reproducible Builds
The binaries released with each restic version starting at 0.6.1 are
[reproducible](https://reproducible-builds.org/), which means that you can
reproduce a byte identical version from the source code for that
release. Instructions on how to do that are contained in the
[builder repository](https://github.com/restic/builder).
News
----
You can follow the restic project on Twitter [@resticbackup](https://twitter.com/resticbackup) or by subscribing to
the [project blog](https://restic.net/blog/).
License
-------
Restic is licensed under [BSD 2-Clause License](https://opensource.org/licenses/BSD-2-Clause). You can find the
complete text in [``LICENSE``](LICENSE).
Sponsorship
-----------
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by [AppsCode](https://appscode.com)!
[![Sponsored by AppsCode](https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png)](https://appscode.com)

View File

@@ -1,135 +0,0 @@
|Documentation| |Build Status| |Build status| |Report Card| |Say Thanks| |TestCoverage| |Reviewed by Hound|
Introduction
------------
restic is a backup program that is fast, efficient and secure. It supports the three major operating systems (Linux, macOS, Windows) and a few smaller ones (FreeBSD, OpenBSD).
For detailed usage and installation instructions check out the `documentation <https://restic.readthedocs.io/en/latest>`__.
You can ask questions in our `Discourse forum <https://forum.restic.net>`__.
Quick start
-----------
Once you've `installed
<https://restic.readthedocs.io/en/latest/020_installation.html>`__ restic, start
off with creating a repository for your backups:
.. code-block:: console
$ restic init --repo /tmp/backup
enter password for new backend:
enter password again:
created restic backend 085b3c76b9 at /tmp/backup
Please note that knowledge of your password is required to access the repository.
Losing your password means that your data is irrecoverably lost.
and add some data:
.. code-block:: console
$ restic --repo /tmp/backup backup ~/work
enter password for repository:
scan [/home/user/work]
scanned 764 directories, 1816 files in 0:00
[0:29] 100.00% 54.732 MiB/s 1.582 GiB / 1.582 GiB 2580 / 2580 items 0 errors ETA 0:00
duration: 0:29, 54.47MiB/s
snapshot 40dc1520 saved
Next you can either use ``restic restore`` to restore files or use ``restic
mount`` to mount the repository via fuse and browse the files from previous
snapshots.
For more options check out the `online documentation <https://restic.readthedocs.io/en/latest/>`__.
Backends
--------
Saving a backup on the same machine is nice but not a real backup strategy.
Therefore, restic supports the following backends for storing backups natively:
- `Local directory <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local>`__
- `sftp server (via SSH) <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp>`__
- `HTTP REST server <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server>`__ (`protocol <doc/100_references.rst#rest-backend>`__ `rest-server <https://github.com/restic/rest-server>`__)
- `AWS S3 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3>`__ (either from Amazon or using the `Minio <https://minio.io>`__ server)
- `OpenStack Swift <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift>`__
- `BackBlaze B2 <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2>`__
- `Microsoft Azure Blob Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage>`__
- `Google Cloud Storage <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#google-cloud-storage>`__
- And many other services via the `rclone <https://rclone.org>`__ `Backend <https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#other-services-via-rclone>`__
Design Principles
-----------------
Restic is a program that does backups right and was designed with the
following principles in mind:
- **Easy:** Doing backups should be a frictionless process, otherwise
you might be tempted to skip it. Restic should be easy to configure
and use, so that, in the event of a data loss, you can just restore
it. Likewise, restoring data should not be complicated.
- **Fast**: Backing up your data with restic should only be limited by
your network or hard disk bandwidth so that you can backup your files
every day. Nobody does backups if it takes too much time. Restoring
backups should only transfer data that is needed for the files that
are to be restored, so that this process is also fast.
- **Verifiable**: Much more important than backup is restore, so restic
enables you to easily verify that all data can be restored.
- **Secure**: Restic uses cryptography to guarantee confidentiality and
integrity of your data. The location the backup data is stored is
assumed not to be a trusted environment (e.g. a shared space where
others like system administrators are able to access your backups).
Restic is built to secure your data against such attackers.
- **Efficient**: With the growth of data, additional snapshots should
only take the storage of the actual increment. Even more, duplicate
data should be de-duplicated before it is actually written to the
storage back end to save precious backup space.
Reproducible Builds
-------------------
The binaries released with each restic version starting at 0.6.1 are
`reproducible <https://reproducible-builds.org/>`__, which means that you can
easily reproduce a byte identical version from the source code for that
release. Instructions on how to do that are contained in the
`builder repository <https://github.com/restic/builder>`__.
News
----
You can follow the restic project on Twitter `@resticbackup <https://twitter.com/resticbackup>`__ or by subscribing to
the `development blog <https://restic.net/blog/>`__.
License
-------
Restic is licensed under `BSD 2-Clause License <https://opensource.org/licenses/BSD-2-Clause>`__. You can find the
complete text in ``LICENSE``.
Sponsorship
-----------
Backend integration tests for Google Cloud Storage and Microsoft Azure Blob
Storage are sponsored by `AppsCode <https://appscode.com>`__!
|AppsCode|
.. |Documentation| image:: https://readthedocs.org/projects/restic/badge/?version=latest
:target: https://restic.readthedocs.io/en/latest/?badge=latest
.. |Build Status| image:: https://travis-ci.com/restic/restic.svg?branch=master
:target: https://travis-ci.com/restic/restic
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true
:target: https://ci.appveyor.com/project/fd0/restic/branch/master
.. |Report Card| image:: https://goreportcard.com/badge/github.com/restic/restic
:target: https://goreportcard.com/report/github.com/restic/restic
.. |Say Thanks| image:: https://img.shields.io/badge/Say%20Thanks-!-1EAEDB.svg
:target: https://saythanks.io/to/restic
.. |AppsCode| image:: https://cdn.appscode.com/images/logo/appscode/ac-logo-color.png
:target: https://appscode.com
.. |Reviewed by Hound| image:: https://img.shields.io/badge/Reviewed_by-Hound-8E64B0.svg
:target: https://houndci.com

View File

@@ -1 +1 @@
0.9.6
0.12.0

View File

@@ -1,32 +0,0 @@
clone_folder: c:\restic
environment:
GOPATH: c:\gopath
branches:
only:
- master
cache:
- '%LocalAppData%\go-build'
init:
- ps: >-
$app = Get-WmiObject -Class Win32_Product -Filter "Vendor = 'http://golang.org'"
if ($app) {
$app.Uninstall()
}
install:
- rmdir c:\go /s /q
- appveyor DownloadFile https://dl.google.com/go/go1.13.4.windows-amd64.msi
- msiexec /i go1.13.4.windows-amd64.msi /q
- go version
- go env
- appveyor DownloadFile http://sourceforge.netcologne.de/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip
- 7z x tar.zip bin/tar.exe
- set PATH=bin/;%PATH%
build_script:
- go run -mod=vendor run_integration_tests.go

273
build.go
View File

@@ -3,22 +3,15 @@
// This program aims to make building Go programs for end users easier by just
// calling it with `go run`, without having to setup a GOPATH.
//
// For Go < 1.11, it'll create a new GOPATH in a temporary directory, then run
// `go build` on the package configured as Main in the Config struct.
//
// For Go >= 1.11 if the file go.mod is present, it'll use Go modules and not
// setup a GOPATH. It builds the package configured as Main in the Config
// struct with `go build -mod=vendor` to use the vendored dependencies.
// The variable GOPROXY is set to `off` so that no network calls are made. All
// files are copied to a temporary directory before `go build` is called within
// that directory.
// This program needs Go >= 1.12. It'll use Go modules for compilation. It
// builds the package configured as Main in the Config struct.
// BSD 2-Clause License
//
// Copyright (c) 2016-2018, Alexander Neumann <alexander@bumpern.de>
// All rights reserved.
//
// This file has been copied from the repository at:
// This file has been derived from the repository at:
// https://github.com/fd0/build-go
//
// Redistribution and use in source and binary forms, with or without
@@ -65,7 +58,7 @@ var config = Config{
Main: "./cmd/restic", // package name for the main package
DefaultBuildTags: []string{"selfupdate"}, // specify build tags which are always used
Tests: []string{"./..."}, // tests to run
MinVersion: GoVersion{Major: 1, Minor: 10, Patch: 0}, // minimum Go version supported
MinVersion: GoVersion{Major: 1, Minor: 11, Patch: 0}, // minimum Go version supported
}
// Config configures the build.
@@ -79,125 +72,13 @@ type Config struct {
}
var (
verbose bool
keepGopath bool
runTests bool
enableCGO bool
enablePIE bool
goVersion = ParseGoVersion(runtime.Version())
verbose bool
runTests bool
enableCGO bool
enablePIE bool
goVersion = ParseGoVersion(runtime.Version())
)
// copy all Go files in src to dst, creating directories on the fly, so calling
//
// copy("/tmp/gopath/src/github.com/restic/restic", "/home/u/restic")
//
// with "/home/u/restic" containing the file "foo.go" yields the following tree
// at "/tmp/gopath":
//
// /tmp/gopath
// └── src
// └── github.com
// └── restic
// └── restic
// └── foo.go
func copy(dst, src string) error {
verbosePrintf("copy contents of %v to %v\n", src, dst)
return filepath.Walk(src, func(name string, fi os.FileInfo, err error) error {
if name == src {
return err
}
if name == ".git" {
return filepath.SkipDir
}
if err != nil {
return err
}
if fi.IsDir() {
return nil
}
intermediatePath, err := filepath.Rel(src, name)
if err != nil {
return err
}
fileSrc := filepath.Join(src, intermediatePath)
fileDst := filepath.Join(dst, intermediatePath)
return copyFile(fileDst, fileSrc)
})
}
func directoryExists(dirname string) bool {
stat, err := os.Stat(dirname)
if err != nil && os.IsNotExist(err) {
return false
}
return stat.IsDir()
}
func fileExists(filename string) bool {
stat, err := os.Stat(filename)
if err != nil && os.IsNotExist(err) {
return false
}
return stat.Mode().IsRegular()
}
// copyFile creates dst from src, preserving file attributes and timestamps.
func copyFile(dst, src string) error {
fi, err := os.Stat(src)
if err != nil {
return err
}
fsrc, err := os.Open(src)
if err != nil {
return err
}
if err = os.MkdirAll(filepath.Dir(dst), 0755); err != nil {
fmt.Printf("MkdirAll(%v)\n", filepath.Dir(dst))
return err
}
fdst, err := os.Create(dst)
if err != nil {
_ = fsrc.Close()
return err
}
_, err = io.Copy(fdst, fsrc)
if err != nil {
_ = fsrc.Close()
_ = fdst.Close()
return err
}
err = fdst.Close()
if err != nil {
_ = fsrc.Close()
return err
}
err = fsrc.Close()
if err != nil {
return err
}
err = os.Chmod(dst, fi.Mode())
if err != nil {
return err
}
return os.Chtimes(dst, fi.ModTime(), fi.ModTime())
}
// die prints the message with fmt.Fprintf() to stderr and exits with an error
// code.
func die(message string, args ...interface{}) {
@@ -211,7 +92,6 @@ func showUsage(output io.Writer) {
fmt.Fprintf(output, "OPTIONS:\n")
fmt.Fprintf(output, " -v --verbose output more messages\n")
fmt.Fprintf(output, " -t --tags specify additional build tags\n")
fmt.Fprintf(output, " -k --keep-tempdir do not remove the temporary directory after build\n")
fmt.Fprintf(output, " -T --test run tests\n")
fmt.Fprintf(output, " -o --output set output file name\n")
fmt.Fprintf(output, " --enable-cgo use CGO to link against libc\n")
@@ -219,7 +99,6 @@ func showUsage(output io.Writer) {
fmt.Fprintf(output, " --goos value set GOOS for cross-compilation\n")
fmt.Fprintf(output, " --goarch value set GOARCH for cross-compilation\n")
fmt.Fprintf(output, " --goarm value set GOARM for cross-compilation\n")
fmt.Fprintf(output, " --tempdir dir use a specific directory for compilation\n")
}
func verbosePrintf(message string, args ...interface{}) {
@@ -230,49 +109,39 @@ func verbosePrintf(message string, args ...interface{}) {
fmt.Printf("build: "+message, args...)
}
// cleanEnv returns a clean environment with GOPATH, GOBIN and GO111MODULE
// removed (if present).
func cleanEnv() (env []string) {
removeKeys := map[string]struct{}{
"GOPATH": struct{}{},
"GOBIN": struct{}{},
"GO111MODULE": struct{}{},
}
for _, v := range os.Environ() {
data := strings.SplitN(v, "=", 2)
name := data[0]
if _, ok := removeKeys[name]; ok {
// printEnv prints Go-relevant environment variables in a nice way using verbosePrintf.
func printEnv(env []string) {
verbosePrintf("environment (GO*):\n")
for _, v := range env {
// ignore environment variables which do not start with GO*.
if !strings.HasPrefix(v, "GO") {
continue
}
env = append(env, v)
verbosePrintf(" %s\n", v)
}
return env
}
// build runs "go build args..." with GOPATH set to gopath.
func build(cwd string, env map[string]string, args ...string) error {
a := []string{"build"}
if goVersion.AtLeast(GoVersion{1, 10, 0}) {
verbosePrintf("Go version is at least 1.10, using new syntax for -gcflags\n")
// use new prefix
// try to remove all absolute paths from resulting binary
if goVersion.AtLeast(GoVersion{1, 13, 0}) {
// use the new flag introduced by Go 1.13
a = append(a, "-trimpath")
} else {
// otherwise try to trim as many paths as possible
a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", cwd))
a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", cwd))
} else {
a = append(a, "-asmflags", fmt.Sprintf("-trimpath=%s", cwd))
a = append(a, "-gcflags", fmt.Sprintf("-trimpath=%s", cwd))
}
if enablePIE {
a = append(a, "-buildmode=pie")
}
a = append(a, args...)
cmd := exec.Command("go", a...)
cmd.Env = append(cleanEnv(), "GOPROXY=off")
cmd.Env = os.Environ()
for k, v := range env {
cmd.Env = append(cmd.Env, k+"="+v)
}
@@ -280,6 +149,8 @@ func build(cwd string, env map[string]string, args ...string) error {
cmd.Env = append(cmd.Env, "CGO_ENABLED=0")
}
printEnv(cmd.Env)
cmd.Dir = cwd
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
@@ -294,7 +165,7 @@ func build(cwd string, env map[string]string, args ...string) error {
func test(cwd string, env map[string]string, args ...string) error {
args = append([]string{"test", "-count", "1"}, args...)
cmd := exec.Command("go", args...)
cmd.Env = append(cleanEnv(), "GOPROXY=off")
cmd.Env = os.Environ()
for k, v := range env {
cmd.Env = append(cmd.Env, k+"="+v)
}
@@ -305,6 +176,8 @@ func test(cwd string, env map[string]string, args ...string) error {
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
printEnv(cmd.Env)
verbosePrintf("chdir %q\n", cwd)
verbosePrintf("go %q\n", args)
@@ -454,6 +327,10 @@ func (v GoVersion) String() string {
}
func main() {
if !goVersion.AtLeast(GoVersion{1, 12, 0}) {
die("Go version (%v) is too old, restic requires Go >= 1.12\n", goVersion)
}
if !goVersion.AtLeast(config.MinVersion) {
fmt.Fprintf(os.Stderr, "%s detected, this program requires at least %s\n", goVersion, config.MinVersion)
os.Exit(1)
@@ -464,15 +341,13 @@ func main() {
skipNext := false
params := os.Args[1:]
goEnv := map[string]string{}
buildEnv := map[string]string{
"GOOS": runtime.GOOS,
"GOARCH": runtime.GOARCH,
"GOARM": "",
env := map[string]string{
"GO111MODULE": "on", // make sure we build in Module mode
"GOOS": runtime.GOOS,
"GOARCH": runtime.GOARCH,
"GOARM": "",
}
tempdir := ""
var outputFilename string
for i, arg := range params {
@@ -484,8 +359,6 @@ func main() {
switch arg {
case "-v", "--verbose":
verbose = true
case "-k", "--keep-gopath":
keepGopath = true
case "-t", "-tags", "--tags":
if i+1 >= len(params) {
die("-t given but no tag specified")
@@ -495,9 +368,6 @@ func main() {
case "-o", "--output":
skipNext = true
outputFilename = params[i+1]
case "--tempdir":
skipNext = true
tempdir = params[i+1]
case "-T", "--test":
runTests = true
case "--enable-cgo":
@@ -506,13 +376,13 @@ func main() {
enablePIE = true
case "--goos":
skipNext = true
buildEnv["GOOS"] = params[i+1]
env["GOOS"] = params[i+1]
case "--goarch":
skipNext = true
buildEnv["GOARCH"] = params[i+1]
env["GOARCH"] = params[i+1]
case "--goarm":
skipNext = true
buildEnv["GOARM"] = params[i+1]
env["GOARM"] = params[i+1]
case "-h":
showUsage(os.Stdout)
return
@@ -525,8 +395,12 @@ func main() {
verbosePrintf("detected Go version %v\n", goVersion)
preserveSymbols := false
for i := range buildTags {
buildTags[i] = strings.TrimSpace(buildTags[i])
if buildTags[i] == "debug" || buildTags[i] == "profile" {
preserveSymbols = true
}
}
verbosePrintf("build tags: %s\n", buildTags)
@@ -538,7 +412,7 @@ func main() {
if outputFilename == "" {
outputFilename = config.Name
if buildEnv["GOOS"] == "windows" {
if env["GOOS"] == "windows" {
outputFilename += ".exe"
}
}
@@ -553,7 +427,11 @@ func main() {
if version != "" {
constants["main.version"] = version
}
ldflags := "-s -w " + constants.LDFlags()
ldflags := constants.LDFlags()
if !preserveSymbols {
// Strip debug symbols.
ldflags = "-s -w " + ldflags
}
verbosePrintf("ldflags: %s\n", ldflags)
var (
@@ -567,57 +445,18 @@ func main() {
}
buildTarget := filepath.FromSlash(mainPackage)
buildCWD := ""
if goVersion.AtLeast(GoVersion{1, 11, 0}) && fileExists("go.mod") {
verbosePrintf("Go >= 1.11 and 'go.mod' found, building with modules\n")
buildCWD = root
buildArgs = append(buildArgs, "-mod=vendor")
testArgs = append(testArgs, "-mod=vendor")
goEnv["GO111MODULE"] = "on"
buildEnv["GO111MODULE"] = "on"
} else {
if tempdir == "" {
tempdir, err = ioutil.TempDir("", fmt.Sprintf("%v-build-", config.Name))
if err != nil {
die("TempDir(): %v\n", err)
}
}
verbosePrintf("Go < 1.11 or 'go.mod' not found, create GOPATH at %v\n", tempdir)
targetdir := filepath.Join(tempdir, "src", filepath.FromSlash(config.Namespace))
if err = copy(targetdir, root); err != nil {
die("copying files from %v to %v/src failed: %v\n", root, tempdir, err)
}
defer func() {
if !keepGopath {
verbosePrintf("remove %v\n", tempdir)
if err = os.RemoveAll(tempdir); err != nil {
die("remove GOPATH at %s failed: %v\n", tempdir, err)
}
} else {
verbosePrintf("leaving temporary GOPATH at %v\n", tempdir)
}
}()
buildCWD = targetdir
goEnv["GOPATH"] = tempdir
buildEnv["GOPATH"] = tempdir
buildCWD, err := os.Getwd()
if err != nil {
die("unable to determine current working directory: %v\n", err)
}
verbosePrintf("environment:\n go: %v\n build: %v\n", goEnv, buildEnv)
buildArgs = append(buildArgs,
"-tags", strings.Join(buildTags, " "),
"-ldflags", ldflags,
"-o", output, buildTarget,
)
err = build(buildCWD, buildEnv, buildArgs...)
err = build(buildCWD, env, buildArgs...)
if err != nil {
die("build failed: %v\n", err)
}
@@ -627,7 +466,7 @@ func main() {
testArgs = append(testArgs, config.Tests...)
err = test(buildCWD, goEnv, testArgs...)
err = test(buildCWD, env, testArgs...)
if err != nil {
die("running tests failed: %v\n", err)
}

View File

@@ -0,0 +1,9 @@
Enhancement: Support specifying multiple host flags for various commands
Previously commands didn't take more than one `--host` or `-H` argument into account, which could be limiting with e.g.
the `forget` command.
The `dump`, `find`, `forget`, `ls`, `mount`, `restore`, `snapshots`, `stats` and `tag` commands will now take into account
multiple `--host` and `-H` flags.
https://github.com/restic/restic/issues/1570

View File

@@ -0,0 +1,8 @@
Enhancement: Optimize `restic mount`
We've optimized the FUSE implementation used within restic.
`restic mount` is now more responsive and uses less memory.
https://github.com/restic/restic/issues/1680
https://github.com/restic/restic/pull/2587
https://github.com/restic/restic/pull/2787

View File

@@ -0,0 +1,6 @@
Bugfix: Report correct number of directories processed by backup
The directory statistics calculation was fixed to report the actual number
of processed directories instead of always zero.
https://github.com/restic/restic/issues/1863

View File

@@ -0,0 +1,5 @@
Enhancement: Display snapshot date when using `restic find`
Added the respective snapshot date to the output of `restic find`.
https://github.com/restic/restic/issues/2072

View File

@@ -0,0 +1,9 @@
Enhancement: Allow specifying user and host when creating keys
When adding a new key to the repository, the username and hostname for the new
key can be specified on the command line. This allows overriding the defaults,
for example if you would prefer to use the FQDN to identify the host or if you
want to add keys for several different hosts without having to run the key add
command on those hosts.
https://github.com/restic/restic/issues/2175

View File

@@ -0,0 +1,9 @@
Bugfix: Fix tar issues when dumping `/`
We've fixed an issue with dumping either `/` or files on the first sublevel
e.g. `/foo` to tar. This also fixes tar dumping issues on Windows where this
issue could also happen.
https://github.com/restic/restic/issues/2254
https://github.com/restic/restic/issues/2357
https://github.com/restic/restic/pull/2255

View File

@@ -0,0 +1,5 @@
Enhancement: Add support for ppc64le
Adds support for ppc64le, the processor architecture from IBM.
https://github.com/restic/restic/issues/2277

View File

@@ -0,0 +1,7 @@
Bugfix: Handle format verbs like '%' properly in `find` output
The JSON or "normal" output of the `find` command can now deal with file names
that contain substrings which the Golang `fmt` package considers "format verbs"
like `%s`.
https://github.com/restic/restic/issues/2281

View File

@@ -0,0 +1,8 @@
Bugfix: Do not hang when run as a background job
Restic did hang on exit while restoring the terminal configuration when it was
started as a background job, for example using `restic ... &`. This has been
fixed by only restoring the terminal configuration when restic is interrupted
while reading a password from the terminal.
https://github.com/restic/restic/issues/2298

View File

@@ -0,0 +1,8 @@
Bugfix: Fix mangled json output of backup command
We've fixed a race condition in the json output of the backup command
that could cause multiple lines to get mixed up. We've also ensured that
the backup summary is printed last.
https://github.com/restic/restic/issues/2389
https://github.com/restic/restic/pull/2545

View File

@@ -0,0 +1,5 @@
Bugfix: Refresh lock timestamp
Long-running operations did not refresh lock timestamp, resulting in locks becoming stale. This is now fixed.
https://github.com/restic/restic/issues/2390

View File

@@ -0,0 +1,12 @@
Enhancement: Ignore sync errors when operation not supported by local filesystem
The local backend has been modified to work with filesystems which doesn't support
the `sync` operation. This operation is normally used by restic to ensure that data
files are fully written to disk before continuing.
For these limited filesystems, saving a file in the backend would previously fail with
an "operation not supported" error. This error is now ignored, which means that e.g.
an SMB mount on macOS can now be used as storage location for a repository.
https://github.com/restic/restic/issues/2395
https://forum.restic.net/t/sync-errors-on-mac-over-smb/1859

View File

@@ -0,0 +1,7 @@
Enhancement: Add flag `--iexclude-file` to backup command
The backup command now supports the flag `--iexclude-file` which is a
case-insensitive version of `--exclude-file`.
https://github.com/restic/restic/issues/2427
https://github.com/restic/restic/pull/2898

View File

@@ -0,0 +1,6 @@
Bugfix: backup --json reports total_bytes_processed as 0
We've fixed the json output of total_bytes_processed. The non-json output
was already fixed with pull request #2138 but left the json output untouched.
https://github.com/restic/restic/issues/2429

View File

@@ -0,0 +1,5 @@
Bugfix: Fix incorrect bytes stats in `diff` command
In some cases, the wrong number of bytes (e.g. 16777215.998 TiB) were reported by the `diff` command. This is now fixed.
https://github.com/restic/restic/issues/2469

View File

@@ -0,0 +1,9 @@
Change: Remove vendored dependencies
We've removed the vendored dependencies (in the subdir `vendor/`). When
building restic, the Go compiler automatically fetches the dependencies. It
will also cryptographically verify that the correct code has been fetched by
using the hashes in `go.sum` (see the link to the documentation below).
https://github.com/restic/restic/issues/2482
https://golang.org/cmd/go/#hdr-Module_downloading_and_verification

View File

@@ -0,0 +1,16 @@
Bugfix: Do not crash with Synology NAS sftp server
It was found that when restic is used to store data on an sftp server on a
Synology NAS with a relative path (one which does not start with a slash), it
may go into an endless loop trying to create directories on the server. We've
fixed this bug by using a function in the sftp library instead of our own
implementation.
The bug was discovered because the Synology sftp server behaves erratic with
non-absolute path (e.g. `home/restic-repo`). This can be resolved by just using
an absolute path instead (`/home/restic-repo`). We've also added a paragraph in
the FAQ.
https://github.com/restic/restic/issues/2518
https://github.com/restic/restic/issues/2363
https://github.com/restic/restic/pull/2530

View File

@@ -0,0 +1,5 @@
Bugfix: Fix incorrect size calculation in `stats --mode restore-size`
The restore-size mode of stats was counting hard-linked files as if they were independent.
https://github.com/restic/restic/issues/2531

View File

@@ -0,0 +1,5 @@
Bugfix: Fix incorrect file counts in `stats --mode restore-size`
The restore-size mode of stats was failing to count empty directories and some files with hard links.
https://github.com/restic/restic/issues/2537

View File

@@ -0,0 +1,8 @@
Enhancement: Support excluding files by their size
The `backup` command now supports the `--exclude-larger-than` option to exclude files which are
larger than the specified maximum size. This can for example be useful to exclude unimportant
files with a large file size.
https://github.com/restic/restic/issues/2569
https://github.com/restic/restic/pull/2914

View File

@@ -0,0 +1,16 @@
Enhancement: Self-heal missing file parts during backup of unchanged files
We've improved the resilience of restic to certain types of repository corruption.
For files that are unchanged since the parent snapshot, the backup command now
verifies that all parts of the files still exist in the repository. Parts that are
missing, e.g. from a damaged repository, are backed up again. This verification
was already run for files that were modified since the parent snapshot, but is
now also done for unchanged files.
Note that restic will not backup file parts that are referenced in the index but
where the actual data is not present on disk, as this situation can only be
detected by restic check. Please ensure that you run `restic check` regularly.
https://github.com/restic/restic/issues/2571
https://github.com/restic/restic/pull/2827

View File

@@ -0,0 +1,9 @@
Enhancement: Support filtering snapshots by tag and path in the stats command
We've added filtering snapshots by `--tag tagList` and by `--path path` to
the `stats` command. This includes filtering of only 'latest' snapshots or
all snapshots in a repository.
https://github.com/restic/restic/issues/2858
https://github.com/restic/restic/pull/2859
https://forum.restic.net/t/stats-for-a-host-and-filtered-snapshots/3020

View File

@@ -0,0 +1,20 @@
Enhancement: Add command for copying snapshots between repositories
We've added a copy command, allowing you to copy snapshots from one
repository to another.
Note that this process will have to read (download) and write (upload) the
entire snapshot(s) due to the different encryption keys used on the source
and destination repository. Also, the transferred files are not re-chunked,
which may break deduplication between files already stored in the
destination repo and files copied there using this command.
To fully support deduplication between repositories when the copy command is
used, the init command now supports the `--copy-chunker-params` option,
which initializes the new repository with identical parameters for splitting
files into chunks as an already existing repository. This allows copied
snapshots to be equally deduplicated in both repositories.
https://github.com/restic/restic/issues/323
https://github.com/restic/restic/pull/2606
https://github.com/restic/restic/pull/2928

View File

@@ -0,0 +1,10 @@
Enhancement: Use optimized library for hash calculation of file chunks
We've switched the library used to calculate the hashes of file chunks, which
are used for deduplication, to the optimized Minio SHA-256 implementation.
Depending on the CPU it improves the hashing throughput by 10-30%. Modern x86
CPUs with the SHA Extension should be about two to three times faster.
https://github.com/restic/restic/issues/551
https://github.com/restic/restic/pull/2709

View File

@@ -0,0 +1,23 @@
Enhancement: Simplify and improve restore performance
Significantly improves restore performance of large files (i.e. 50M+):
https://github.com/restic/restic/issues/2074
https://forum.restic.net/t/restore-using-rclone-gdrive-backend-is-slow/1112/8
https://forum.restic.net/t/degraded-restore-performance-s3-backend/1400
Fixes "not enough cache capacity" error during restore:
https://github.com/restic/restic/issues/2244
NOTE: This new implementation does not guarantee order in which blobs
are written to the target files and, for example, the last blob of a
file can be written to the file before any of the preceeding file blobs.
It is therefore possible to have gaps in the data written to the target
files if restore fails or interrupted by the user.
The implementation will try to preallocate space for the restored files
on the filesystem to prevent file fragmentation. This ensures good read
performance for large files, like for example VM images. If preallocating
space is not supported by the filesystem, then this step is silently skipped.
https://github.com/restic/restic/pull/2195
https://github.com/restic/restic/pull/2893

View File

@@ -0,0 +1,8 @@
Enhancement: Improve speed of check command
We've improved the check command to traverse trees only once independent of
whether they are contained in multiple snapshots. The check command is now much
faster for repositories with a large number of snapshots.
https://github.com/restic/restic/pull/2328
https://github.com/restic/restic/issues/2284

View File

@@ -0,0 +1,5 @@
Enhancement: support user@domain parsing as user
Added the ability for user@domain-like users to be authenticated over SFTP servers.
https://github.com/restic/restic/pull/2423

View File

@@ -0,0 +1,19 @@
Change: Return exit code 3 when failing to backup all source data
The backup command used to return a zero exit code as long as a snapshot
could be created successfully, even if some of the source files could not
be read (in which case the snapshot would contain the rest of the files).
This made it hard for automation/scripts to detect failures/incomplete
backups by looking at the exit code. Restic now returns the following exit
codes for the backup command:
- 0 when the command was successful
- 1 when there was a fatal error (no snapshot created)
- 3 when some source data could not be read (incomplete snapshot created)
https://github.com/restic/restic/pull/2546
https://github.com/restic/restic/issues/956
https://github.com/restic/restic/issues/2064
https://github.com/restic/restic/issues/2526
https://github.com/restic/restic/issues/2364

View File

@@ -0,0 +1,9 @@
Enhancement: Improve the chunking algorithm
We've updated the chunker library responsible for splitting files into smaller
blocks. It should improve the chunking throughput by 5-15% depending on the
CPU.
https://github.com/restic/restic/pull/2576
https://github.com/restic/restic/pull/2845
https://github.com/restic/restic/issues/2820

View File

@@ -0,0 +1,6 @@
Bugfix: SFTP backend supports IPv6 addresses
The SFTP backend now supports IPv6 addresses natively, without relying on
aliases in the external SSH configuration.
https://github.com/restic/restic/pull/2592

View File

@@ -0,0 +1,6 @@
Enhancement: Improve speed of diff command
We've improved the performance of the diff command when comparing snapshots
with similar content. It should run up to twice as fast as before.
https://github.com/restic/restic/pull/2598

View File

@@ -0,0 +1,6 @@
Enhancement: Slightly reduce memory usage of prune and stats commands
The prune and the stats command kept directory identifiers in memory twice
while searching for used blobs.
https://github.com/restic/restic/pull/2599

View File

@@ -0,0 +1,14 @@
Change: Update dependencies, require Go >= 1.13
Restic now requires Go to be at least 1.13. This allows simplifications in the
build process and removing workarounds.
This is also probably the last version of restic still supporting mounting
repositories via fuse on macOS. The library we're using for fuse does not
support macOS any more and osxfuse is not open source any more.
https://github.com/restic/restic/pull/2600
https://github.com/restic/restic/pull/2852
https://github.com/restic/restic/pull/2927
https://github.com/bazil/fuse/issues/224
https://github.com/osxfuse/osxfuse/issues/590

View File

@@ -0,0 +1,7 @@
Bugfix: Honor RESTIC_CACHE_DIR environment variable on Mac and Windows
On Mac and Windows, the RESTIC_CACHE_DIR environment variable was ignored.
This variable can now be used on all platforms to set the directory where
restic stores caches.
https://github.com/restic/restic/pull/2607

View File

@@ -0,0 +1,6 @@
Bugfix: Don't abort the stats command when data blobs are missing
Runing the stats command in the blobs-per-file mode on a repository with
missing data blobs previously resulted in a crash.
https://github.com/restic/restic/pull/2668

View File

@@ -0,0 +1,8 @@
Bugfix: Add stricter prune error checks
Additional checks were added to the prune command in order to improve
resiliency to backend, hardware and/or networking issues. The checks now
detect a few more cases where such outside factors could potentially cause
data loss.
https://github.com/restic/restic/pull/2674

View File

@@ -0,0 +1,6 @@
Enhancement: S3 backend: Add support for WebIdentityTokenFile
We've added support for EKS IAM roles for service accounts feature to the S3 backend.
https://github.com/restic/restic/pull/2733
https://github.com/restic/restic/issues/2703

View File

@@ -0,0 +1,6 @@
Enhancement: Optimize handling of new index entries
Restic now uses less memory for backups which add a lot of data, e.g. large initial backups.
In addition, we've improved the stability in some edge cases.
https://github.com/restic/restic/pull/2773

View File

@@ -0,0 +1,8 @@
Enhancement: Reduce memory consumption of in-memory index
We've improved how the index is stored in memory.
This change can reduce memory usage for large repositories by up to 50%
(depending on the operation).
https://github.com/restic/restic/pull/2781
https://github.com/restic/restic/pull/2812

View File

@@ -0,0 +1,6 @@
Enhancement: Optimize `list blobs` command
We've changed the implementation of `list blobs` which should be now a bit faster
and consume almost no memory even for large repositories.
https://github.com/restic/restic/pull/2786

View File

@@ -0,0 +1,6 @@
Enhancement: Optimized file access in restic mount
Reading large (> 100GiB) files from restic mountpoints is now faster,
and the speedup is greater for larger files.
https://github.com/restic/restic/pull/2790

View File

@@ -0,0 +1,8 @@
Change: Honor the --no-lock flag in the mount command
The mount command now does not lock the repository if given the
--no-lock flag. This allows to mount repositories which are archived
on a read only backend/filesystem.
https://github.com/restic/restic/issues/1597
https://github.com/restic/restic/pull/2821

View File

@@ -0,0 +1,7 @@
Enhancement: Speed-up file deletion in forget, prune and rebuild-index
We've sped up the file deletion for the commands forget, prune and
rebuild-index, especially for remote repositories.
Deletion was sequential before and is now run in parallel.
https://github.com/restic/restic/pull/2840

View File

@@ -0,0 +1,9 @@
Bugfix: Fix possible crash in the progress bar of check --read-data
We've fixed a possible crash while displaying the progress bar for the
check --read-data command. The crash occurred when the length of the
progress bar status exceeded the terminal width, which only happened for
very narrow terminal windows.
https://github.com/restic/restic/pull/2899
https://forum.restic.net/t/restic-rclone-pcloud-connection-issues/2963/15

View File

@@ -0,0 +1,11 @@
Bugfix: Restore timestamps and permissions on intermediate directories
When using the `--include` option of the restore command, restic restored
timestamps and permissions only on directories selected by the include pattern.
Intermediate directories, which are necessary to restore files located in sub-
directories, were created with default permissions. We've fixed the restore
command to restore timestamps and permissions for these directories as well.
https://github.com/restic/restic/issues/1212
https://github.com/restic/restic/issues/1402
https://github.com/restic/restic/pull/2906

View File

@@ -0,0 +1,15 @@
Bugfix: Mark repository files as read-only when using the local backend
Files stored in a local repository were marked as writeable on the
filesystem for non-Windows systems, which did not prevent accidental file
modifications outside of restic. In addition, the local backend did not work
with certain filesystems and network mounts which do not permit modifications
of file permissions.
restic now marks files stored in a local repository as read-only on the
filesystem on non-Windows systems. The error handling is improved to support
more filesystems.
https://github.com/restic/restic/issues/1756
https://github.com/restic/restic/issues/2157
https://github.com/restic/restic/pull/2989

View File

@@ -0,0 +1,10 @@
Bugfix: Hide password in REST backend repository URLs
When using a password in the REST backend repository URL,
the password could in some cases be included in the output
from restic, e.g. when initializing a repo or during an error.
The password is now replaced with "***" where applicable.
https://github.com/restic/restic/issues/2241
https://github.com/restic/restic/pull/2658

View File

@@ -0,0 +1,12 @@
Bugfix: Correctly dump directories into tar files
The dump command previously wrote directories in a tar file in a way which
can cause compatibility problems. This caused, for example, 7zip on Windows
to not open tar files containing directories. In addition it was not possible
to dump directories with extended attributes. These compatibility problems
are now corrected.
In addition, a tar file now includes the name of the owner and group of a file.
https://github.com/restic/restic/issues/2319
https://github.com/restic/restic/pull/3039

View File

@@ -0,0 +1,9 @@
Bugfix: Don't require `self-update --output` placeholder file
`restic self-update --output /path/to/new-restic` used to require that
new-restic was an existing file, to be overwritten. Now it's possible
to download an updated restic binary to a new path, without first
having to create a placeholder file.
https://github.com/restic/restic/issues/2491
https://github.com/restic/restic/pull/2937

View File

@@ -0,0 +1,7 @@
Bugfix: Fix rare cases of backup command hanging forever
We've fixed an issue with the backup progress reporting which could cause
restic to hang forever right before finishing a backup.
https://github.com/restic/restic/issues/2834
https://github.com/restic/restic/pull/2963

View File

@@ -0,0 +1,6 @@
Bugfix: Fix manpage formatting
The manpage formatting in restic v0.10.0 was garbled, which is fixed now.
https://github.com/restic/restic/issues/2938
https://github.com/restic/restic/pull/2977

View File

@@ -0,0 +1,7 @@
Bugfix: Make --exclude-larger-than handle disappearing files
There was a small bug in the backup command's --exclude-larger-than
option where files that disappeared between scanning and actually
backing them up to the repository caused a panic. This is now fixed.
https://github.com/restic/restic/issues/2942

View File

@@ -0,0 +1,9 @@
Bugfix: restic generate, help and self-update no longer check passwords
The commands `restic cache`, `generate`, `help` and `self-update` don't need
passwords, but they previously did run the RESTIC_PASSWORD_COMMAND (if set in
the environment), prompting users to authenticate for no reason. They now skip
running the password command.
https://github.com/restic/restic/issues/2951
https://github.com/restic/restic/pull/2987

View File

@@ -0,0 +1,9 @@
Enhancement: Optimize check for unchanged files during backup
During a backup restic skips processing files which have not changed since the last backup run.
Previously this required opening each file once which can be slow on network filesystems. The
backup command now checks for file changes before opening a file. This considerably reduces
the time to create a backup on network filesystems.
https://github.com/restic/restic/issues/2969
https://github.com/restic/restic/pull/2970

View File

@@ -0,0 +1,9 @@
Bugfix: Make snapshots --json output [] instead of null when no snapshots
Restic previously output `null` instead of `[]` for the `--json snapshots`
command, when there were no snapshots in the repository. This caused some
minor problems when parsing the output, but is now fixed such that `[]` is
output when the list of snapshots is empty.
https://github.com/restic/restic/issues/2979
https://github.com/restic/restic/pull/2984

View File

@@ -0,0 +1,12 @@
Enhancement: Add support for Volume Shadow Copy Service (VSS) on Windows
Volume Shadow Copy Service allows read access to files that are locked by
another process using an exclusive lock through a filesystem snapshot. Restic
was unable to backup those files before. This update enables backing up these
files.
This needs to be enabled explicitely using the --use-fs-snapshot option of the
backup command.
https://github.com/restic/restic/issues/340
https://github.com/restic/restic/pull/2274

View File

@@ -0,0 +1,7 @@
Enhancement: Authenticate to Google Cloud Storage with access token
When using the GCS backend, it is now possible to authenticate with OAuth2
access tokens instead of a credentials file by setting the GOOGLE_ACCESS_TOKEN
environment variable.
https://github.com/restic/restic/pull/2849

View File

@@ -0,0 +1,10 @@
Enhancement: New option --repository-file
We've added a new command-line option --repository-file as an alternative
to -r. This allows to read the repository URL from a file in order to
prevent certain types of information leaks, especially for URLs containing
credentials.
https://github.com/restic/restic/issues/1458
https://github.com/restic/restic/issues/2900
https://github.com/restic/restic/pull/2910

View File

@@ -0,0 +1,8 @@
Enhancement: Warn if parent snapshot cannot be loaded during backup
During a backup restic uses the parent snapshot to check whether a file was
changed and has to be backed up again. For this check the backup has to read
the directories contained in the old snapshot. If a tree blob cannot be
loaded, restic now warns about this problem with the backup repository.
https://github.com/restic/restic/pull/2978

View File

@@ -0,0 +1,10 @@
Bugfix: Make `mount` not create missing mount point directory
When specifying a non-existent directory as mount point for the `mount`
command, restic used to create the specified directory automatically.
This has now changed such that restic instead gives an error when the
specified directory for the mount point does not exist.
https://github.com/restic/restic/issues/1681
https://github.com/restic/restic/pull/3008

View File

@@ -0,0 +1,8 @@
Bugfix: Ignore `no data available` filesystem error during backup
Restic was unable to backup files on some filesystems, for example certain
configurations of CIFS on Linux which return a `no data available` error
when reading extended attributes. These errors are now ignored.
https://github.com/restic/restic/issues/1800
https://github.com/restic/restic/pull/3034

View File

@@ -0,0 +1,8 @@
Enhancement: Allow specifying percentage in `check --read-data-subset`
We've enhanced the `check` command's `--read-data-subset` option to also accept
a percentage (e.g. `2.5%` or `10%`). This will check the given percentage of
pack files (which are randomly selected on each run).
https://github.com/restic/restic/issues/2186
https://github.com/restic/restic/pull/3038

View File

@@ -0,0 +1,14 @@
Enhancement: Report permanent/fatal backend errors earlier
When encountering errors in reading from or writing to storage backends,
restic retries the failing operation up to nine times (for a total of ten
attempts). It used to retry all backend operations, but now detects some
permanent error conditions so that it can report fatal errors earlier.
Permanent failures include local disks being full, SSH connections
dropping and permission errors.
https://github.com/restic/restic/issues/2453
https://github.com/restic/restic/pull/3170
https://github.com/restic/restic/issues/3180
https://github.com/restic/restic/pull/3181

View File

@@ -0,0 +1,21 @@
Enhancement: Add Alibaba/Aliyun OSS support in the `s3` backend
A new extended option `s3.bucket-lookup` has been added to support
Alibaba/Aliyun OSS in the `s3` backend. The option can be set to one
of the following values:
- `auto` - Existing behaviour
- `dns` - Use DNS style bucket access
- `path` - Use path style bucket access
To make the `s3` backend work with Alibaba/Aliyun OSS you must set
`s3.bucket-lookup` to `dns` and set the `s3.region` parameter. For
example:
restic -o s3.bucket-lookup=dns -o s3.region=oss-eu-west-1 -r s3:https://oss-eu-west-1.aliyuncs.com/bucketname init
Note that `s3.region` must be set, otherwise the MinIO SDK tries to
look it up and it seems that Alibaba doesn't support that properly.
https://github.com/restic/restic/issues/2528
https://github.com/restic/restic/pull/2535

View File

@@ -0,0 +1,9 @@
Bugfix: Report the correct owner of directories in FUSE mounts
Restic 0.10.0 changed the FUSE mount to always report the current user
as the owner of directories within the FUSE mount, which is incorrect.
This is now changed back to reporting the correct owner of a directory.
https://github.com/restic/restic/issues/2563
https://github.com/restic/restic/pull/3141

View File

@@ -0,0 +1,31 @@
Bugfix: Make `backup` and `tag` commands separate tags by comma
Running `restic backup --tag foo,bar` previously created snapshots with one
single tag containing a comma (`foo,bar`) instead of two tags (`foo`, `bar`).
Similarly, the `tag` command's `--set`, `--add` and `--remove` options would
treat `foo,bar` as one tag instead of two tags. This was inconsistent with
other commands and often unexpected when one intended `foo,bar` to mean two
tags.
To be consistent in all commands, restic now interprets `foo,bar` to mean two
separate tags (`foo` and `bar`) instead of one tag (`foo,bar`) everywhere,
including in the `backup` and `tag` commands.
NOTE: This change might result in unexpected behavior in cases where you use
the `forget` command and filter on tags like `foo,bar`. Snapshots previously
backed up with `--tag foo,bar` will still not match that filter, but snapshots
saved from now on will match that filter.
To replace `foo,bar` tags with `foo` and `bar` tags in old snapshots, you can
first generate a list of the relevant snapshots using a command like:
restic snapshots --json --quiet | jq '.[] | select(contains({tags: ["foo,bar"]})) | .id'
and then use `restic tag --set foo --set bar snapshotID [...]` to set the new
tags. Please adjust the commands to include real tag names and any additional
tags, as well as the list of snapshots to process.
https://github.com/restic/restic/issues/2688
https://github.com/restic/restic/pull/2690
https://github.com/restic/restic/pull/3197

View File

@@ -0,0 +1,17 @@
Enhancement: Configurable progress reports for non-interactive terminals
The `backup`, `check` and `prune` commands never printed any progress
reports on non-interactive terminals. This behavior is now configurable
using the `RESTIC_PROGRESS_FPS` environment variable. Use for example a
value of `1` for an update every second, or `0.01666` for an update every
minute.
The `backup` command now also prints the current progress when restic
receives a `SIGUSR1` signal.
Setting the `RESTIC_PROGRESS_FPS` environment variable or sending a `SIGUSR1`
signal prints a status report even when `--quiet` was specified.
https://github.com/restic/restic/issues/2706
https://github.com/restic/restic/issues/3194
https://github.com/restic/restic/pull/3199

View File

@@ -0,0 +1,5 @@
Bugfix: Make the `cat` command respect the `--no-lock` option
The `cat` command would not respect the `--no-lock` flag. This is now fixed.
https://github.com/restic/restic/issues/2739

View File

@@ -0,0 +1,13 @@
Enhancement: Add `backup` options `--files-from-{verbatim,raw}`
The new `backup` options `--files-from-verbatim` and `--files-from-raw` read a
list of files to back up from a file. Unlike the existing `--files-from`
option, these options do not interpret the listed filenames as glob patterns;
instead, whitespace in filenames is preserved as-is and no pattern expansion is
done. Please see the documentation for specifics.
These new options are highly recommended over `--files-from`, when using a
script to generate the list of files to back up.
https://github.com/restic/restic/issues/2944
https://github.com/restic/restic/issues/3013

View File

@@ -0,0 +1,18 @@
Enhancement: Allow usage of deprecated S3 `ListObjects` API
Some S3 API implementations, e.g. Ceph before version 14.2.5, have a broken
`ListObjectsV2` implementation which causes problems for restic when using
their API endpoints. When a broken server implementation is used, restic prints
errors similar to the following:
List() returned error: Truncated response should have continuation token set
As a temporary workaround, restic now allows using the older `ListObjects`
endpoint by setting the `s3.list-objects-v1` extended option, for instance:
restic -o s3.list-objects-v1=true snapshots
Please note that this option may be removed in future versions of restic.
https://github.com/restic/restic/issues/3083
https://github.com/restic/restic/pull/3085

View File

@@ -0,0 +1,10 @@
Bugfix: The `--use-fs-snapshot` option now works on windows/386
Restic failed to create VSS snapshots on windows/386 with the following error:
GetSnapshotProperties() failed: E_INVALIDARG (0x80070057)
This is now fixed.
https://github.com/restic/restic/issues/3087
https://github.com/restic/restic/pull/3090

View File

@@ -0,0 +1,17 @@
Change: Deleting files on Google Drive now moves them to the trash
When deleting files on Google Drive via the `rclone` backend, restic used to
bypass the trash folder required that one used the `-o rclone.args` option to
enable usage of the trash folder. This ensured that deleted files in Google
Drive were not kept indefinitely in the trash folder. However, since Google
Drive's trash retention policy changed to deleting trashed files after 30 days,
this is no longer needed.
Restic now leaves it up to rclone and its configuration to use or not use the
trash folder when deleting files. The default is to use the trash folder, as
of rclone 1.53.2. To re-enable the restic 0.11 behavior, set the
`RCLONE_DRIVE_USE_TRASH` environment variable or change the rclone
configuration. See the rclone documentation for more details.
https://github.com/restic/restic/issues/3095
https://github.com/restic/restic/pull/3102

View File

@@ -0,0 +1,10 @@
Bugfix: Do not require gs bucket permissions when running `init`
Restic used to require bucket level permissions for the `gs` backend
in order to initialize a restic repository.
It now allows a `gs` service account to initialize a repository if the
bucket does exist and the service account has permissions to write/read
to that bucket.
https://github.com/restic/restic/issues/3100

View File

@@ -0,0 +1,9 @@
Bugfix: Correctly detect output redirection for `backup` command on Windows
On Windows, since restic 0.10.0 the `backup` command did not properly detect
when the output was redirected to a file. This caused restic to output
terminal control characters. This has been fixed by correcting the terminal
detection.
https://github.com/restic/restic/issues/3111
https://github.com/restic/restic/pull/3150

View File

@@ -0,0 +1,11 @@
Enhancement: Support additional environment variables for Swift authentication
The `swift` backend now supports the following additional environment variables
for passing authentication details to restic:
`OS_USER_ID`, `OS_USER_DOMAIN_ID`, `OS_PROJECT_DOMAIN_ID` and `OS_TRUST_ID`
Depending on the `openrc` configuration file these might be required when the
user and project domains differ from one another.
https://github.com/restic/restic/issues/3147
https://github.com/restic/restic/pull/3158

View File

@@ -0,0 +1,9 @@
Bugfix: Don't create invalid snapshots when `backup` is interrupted
When canceling a backup run at a certain moment it was possible that
restic created a snapshot with an invalid "null" tree. This caused
`check` and other operations to fail. The `backup` command now properly
handles interruptions and never saves a snapshot when interrupted.
https://github.com/restic/restic/issues/3151
https://github.com/restic/restic/pull/3164

View File

@@ -0,0 +1,9 @@
Bugfix: Improve error handling in the `restore` command
The `restore` command used to not print errors while downloading file contents
from the repository. It also incorrectly exited with a zero error code even
when there were errors during the restore process. This has all been fixed and
`restore` now returns with a non-zero exit code when there's an error.
https://github.com/restic/restic/issues/3166
https://github.com/restic/restic/pull/3207

View File

@@ -0,0 +1,8 @@
Enhancement: Add release binaries for MIPS architectures
We've added a few new architectures for Linux to the release binaries: `mips`,
`mipsle`, `mips64`, and `mip64le`. MIPS is mostly used for low-end embedded
systems.
https://github.com/restic/restic/issues/3191
https://github.com/restic/restic/pull/3208

View File

@@ -0,0 +1,11 @@
Bugfix: Correct statistics for overlapping targets
A user reported that restic's statistics and progress information during backup
was not correctly calculated when the backup targets (files/dirs to save)
overlap. For example, consider a directory `foo` which contains (among others)
a file `foo/bar`. When `restic backup foo foo/bar` was run, restic counted the
size of the file `foo/bar` twice, so the completeness percentage as well as the
number of files was wrong. This is now corrected.
https://github.com/restic/restic/issues/3232
https://github.com/restic/restic/pull/3243

View File

@@ -0,0 +1,12 @@
Enhancement: Back up mountpoints as empty directories
When the `--one-file-system` option is specified to `restic backup`, it
ignores all file systems mounted below one of the target directories. This
means that when a snapshot is restored, users needed to manually recreate
the mountpoint directories.
Restic now backs up mountpoints as empty directories and therefore implements
the same approach as `tar`.
https://github.com/restic/restic/issues/909
https://github.com/restic/restic/pull/3119

View File

@@ -0,0 +1,6 @@
Enhancement: Add several more error checks
We've added a lot more error checks in places where errors were previously
ignored (as hinted by the static analysis program `errcheck` via `golangci-lint`).
https://github.com/restic/restic/pull/3250

View File

@@ -0,0 +1,29 @@
Enhancement: Improve `prune` performance and make it more customizable
The `prune` command is now much faster. This is especially the case for remote
repositories or repositories with not much data to remove. Also the memory
usage of the `prune` command is now reduced.
Restic used to rebuild the index from scratch after pruning. This could lead
to missing packs in the index in some cases for eventually consistent backends
such as e.g. AWS S3. This behavior is now changed and the index rebuilding
uses the information already known by `prune`.
By default, the `prune` command no longer removes all unused data. This
behavior can be fine-tuned by new options, like the acceptable amount of
unused space or the maximum size of data to reorganize. For more details,
please see https://restic.readthedocs.io/en/stable/060_forget.html .
Moreover, `prune` now accepts the `--dry-run` option and also running
`forget --dry-run --prune` will show what `prune` would do.
This enhancement also fixes several open issues, e.g.:
- https://github.com/restic/restic/issues/1140
- https://github.com/restic/restic/issues/1599
- https://github.com/restic/restic/issues/1985
- https://github.com/restic/restic/issues/2112
- https://github.com/restic/restic/issues/2227
- https://github.com/restic/restic/issues/2305
https://github.com/restic/restic/pull/2718
https://github.com/restic/restic/pull/2842

Some files were not shown because too many files have changed in this diff Show More