Compare commits

..

1720 Commits

Author SHA1 Message Date
Alexander Neumann
be8be3397c Add version for 0.15.2 2023-04-24 20:28:37 +02:00
Alexander Neumann
db6b4f8912 Update manpages and auto-completion 2023-04-24 20:28:37 +02:00
Alexander Neumann
1f3f042f32 Generate CHANGELOG.md for 0.15.2 2023-04-24 20:28:24 +02:00
Alexander Neumann
0aaa4e6cbe Prepare changelog for 0.15.2 2023-04-24 20:28:24 +02:00
Leo R. Lundgren
0bac935dac doc: Polish changelogs 2023-04-23 22:10:04 +02:00
Michael Eischer
1e6e9f9bd0 tweak changelogs 2023-04-23 12:46:55 +02:00
greatroar
f342db7666 ui/termstatus: Quote funny filenames
Fixes #2260, #4191.
2023-04-23 12:45:40 +02:00
Michael Eischer
07a44a88f2 Fix snapshot filtering for relative paths in the backup command
The snapshot filtering internally converts relative paths to absolute
ones to ensure that the parent snapshots selection works for backups of
relative paths.
2023-04-14 22:58:34 +02:00
Michael Eischer
48e065d971 Sync dependency upgrades from master
github.com/Azure/azure-sdk-for-go/sdk/storage/azblob still uses v0.5.1
as upgrading it would increase the minimum Go version on Solaris to
1.20.
2023-04-14 22:58:34 +02:00
greatroar
3b24c15c3d fuse: Mix inode hashes in a non-symmetric way
Since 0.15 (#4020), inodes are generated as hashes of names, xor'd with
the parent inode. That means that the inode of a/b/b is

	h(a/b/b) = h(a) ^ h(b) ^ h(b) = h(a).

I.e., the grandchild has the same inode as the grandparent. GNU find
trips over this because it thinks it has encountered a loop in the
filesystem, and fails to search a/b/b. This happens more generally when
the same name occurs an even number of times.

Fix this by multiplying the parent by a large prime, so the combining
operation is not longer symmetric in its arguments. This is what the FNV
hash does, which we used prior to 0.15. The hash is now

	h(a/b/b) = h(b) ^ p*(h(b) ^ p*h(a))

Note that we already ensure that h(x) is never zero.

Collisions can still occur, but they should be much less likely to occur
within a single path.

Fixes #4253.
2023-04-14 20:50:39 +02:00
greatroar
4304e01ca2 fuse: Report fuse.Attr.Blocks correctly
Fixes #4239.
2023-04-14 20:50:08 +02:00
Ian Muge
593eb710b4 added changelog 2023-04-13 22:54:36 +02:00
greatroar
97274ecabd cmd, restic: Refactor and fix snapshot filtering
This turns snapshotFilterOptions from cmd into a restic.SnapshotFilter
type and makes restic.FindFilteredSnapshot and FindFilteredSnapshots
methods on that type. This fixes #4211 by ensuring that hosts and paths
are named struct fields instead of unnamed function arguments in long
lists of such.

Timestamp limits are also included in the new type. To avoid too much
pointer handling, the convention is that time zero means no limit.
That's January 1st, year 1, 00:00 UTC, which is so unlikely a date that
we can sacrifice it for simpler code.
2023-04-13 22:51:45 +02:00
Michael Eischer
74f7dd0b38 Make help for --verbose less confusing
The output is now
```
-v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
instead of
```
-v, --verbose n                  be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
```
2023-04-13 22:46:37 +02:00
Michael Eischer
21ad357c10 add linux/riscv64 builds 2023-04-13 22:45:35 +02:00
Alexander Neumann
7d4b7ad9cb Add version for 0.15.1 2023-01-30 20:43:47 +01:00
Alexander Neumann
a883bb6596 Generate CHANGELOG.md for 0.15.1 2023-01-30 20:43:15 +01:00
Alexander Neumann
91acef90b2 Prepare changelog for 0.15.1 2023-01-30 20:43:15 +01:00
Michael Eischer
b2b7727b31 Merge pull request #4178 from rawtaz/changelogs
Polish changelogs
2023-01-27 22:46:46 +01:00
Leo R. Lundgren
0e4c9a5421 Polish changelogs 2023-01-27 22:27:57 +01:00
Michael Eischer
49fa8fe6dd Merge pull request #4175 from fergus-dall/deadlock-fix
Fix deadlock in Lock.Stale
2023-01-26 22:45:13 +01:00
Michael Eischer
12f167ee79 Merge pull request #4167 from aneesh-n/progress-percent-eta-fix
ui/backup: Fix percent and eta in backup progress
2023-01-26 22:42:10 +01:00
Michael Eischer
bb018fbc3e Merge pull request #4163 from MichaelEischer/fix-windows-self-upgrade
self-upgrade: Fix handling of `--output` on windows
2023-01-26 22:40:36 +01:00
Michael Eischer
3b24e0ac55 Merge pull request #4154 from MichaelEischer/cleanup-prune-tests
deduplicate prune tests a bit
2023-01-26 22:39:15 +01:00
Fergus Dall
04da31af2b Fix deadlock in Lock.Stale
With debug logging enabled this method would take a lock and then
format the lock as a string. Since PR #4022 landed the string
formatting method has also taken the lock, so this deadlocks.

Instead just record the lock ID, as is done elsewhere.
2023-01-27 03:21:37 +11:00
Alexander Neumann
65923e9c26 Merge pull request #4152 from MichaelEischer/invalid-locks
Improve handling of invalid locks
2023-01-25 08:19:53 +01:00
Michael Eischer
b903081804 Merge pull request #4169 from restic/dependabot/go_modules/github.com/klauspost/compress-1.15.15
build(deps): bump github.com/klauspost/compress from 1.15.14 to 1.15.15
2023-01-23 22:35:52 +01:00
Michael Eischer
beb1e872cc Merge pull request #4170 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.29.0
build(deps): bump cloud.google.com/go/storage from 1.28.1 to 1.29.0
2023-01-23 22:25:09 +01:00
Michael Eischer
db350c0430 Merge pull request #4171 from restic/dependabot/go_modules/google.golang.org/api-0.108.0
build(deps): bump google.golang.org/api from 0.107.0 to 0.108.0
2023-01-23 22:23:51 +01:00
dependabot[bot]
716a5dd20d build(deps): bump google.golang.org/api from 0.107.0 to 0.108.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.107.0 to 0.108.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.107.0...v0.108.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-23 02:23:51 +00:00
dependabot[bot]
dbd07ade98 build(deps): bump cloud.google.com/go/storage from 1.28.1 to 1.29.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.28.1 to 1.29.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/storage/v1.28.1...spanner/v1.29.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-23 02:23:33 +00:00
dependabot[bot]
7adf1e5d37 build(deps): bump github.com/klauspost/compress from 1.15.14 to 1.15.15
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.14 to 1.15.15.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.14...v1.15.15)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-23 02:23:15 +00:00
Michael Eischer
8f94eb5420 add changelog for less strict lock handling 2023-01-22 15:54:07 +01:00
Michael Eischer
8aaba83719 add changelog for self-upgrade bug on windows 2023-01-22 15:39:42 +01:00
Michael Eischer
e16a6d4c50 self-update: add basic test for extractToFile 2023-01-22 15:39:42 +01:00
Michael Eischer
34e67e3510 self-update: Fix handling of --output on windows
The code always assumed that the upgrade happens in place. Thus writing
the upgrade to a separate file fails, when trying to remove the file
stored at that location.
2023-01-22 15:39:42 +01:00
Michael Eischer
c527c05590 add changelog for missing backup ETA 2023-01-22 15:32:55 +01:00
Aneesh Nireshwalia
ed23edeb62 ui/backup: Fix percent and eta in backup progress
Added missing call to scanFinished=true.
This was causing the percent and eta to never get
printed for backup progress even after the scan was finished.
2023-01-21 13:25:08 -07:00
Michael Eischer
0f398b82e3 Merge pull request #4145 from greatroar/index-encode
index: Optimize generatePackList
2023-01-21 01:01:12 +01:00
greatroar
99755c634b index: Optimize generatePackList
name                 old time/op    new time/op    delta
EncodeIndex/100-8      1.56ms ± 2%    1.48ms ± 3%   -5.37%  (p=0.000 n=10+10)
EncodeIndex/1000-8     14.5ms ± 2%    13.1ms ± 2%   -9.49%  (p=0.000 n=9+10)
EncodeIndex/10000-8     120ms ± 2%     116ms ± 2%   -3.58%  (p=0.000 n=10+10)

name                 old alloc/op   new alloc/op   delta
EncodeIndex/100-8       306kB ± 1%     275kB ± 1%  -10.28%  (p=0.000 n=10+10)
EncodeIndex/1000-8     3.69MB ±11%    2.88MB ± 5%  -22.07%  (p=0.000 n=10+9)
EncodeIndex/10000-8    35.9MB ±11%    31.9MB ±10%  -11.13%  (p=0.005 n=10+10)

name                 old allocs/op  new allocs/op  delta
EncodeIndex/100-8       3.39k ± 0%     2.39k ± 0%  -29.61%  (p=0.000 n=10+10)
EncodeIndex/1000-8      32.6k ± 0%     22.9k ± 0%  -29.63%  (p=0.000 n=10+9)
EncodeIndex/10000-8      326k ± 0%      229k ± 0%  -29.71%  (p=0.000 n=10+10)

The bulk of the allocation rate improvement comes from just removing the
debug.Log calls: every one of those copied a restic.ID to the heap.
2023-01-14 20:41:07 +01:00
Michael Eischer
f5f13f6648 Merge pull request #4153 from MichaelEischer/fix-quiet-prune
prune: Don't show "packs processed" for quiet runs
2023-01-14 20:11:34 +01:00
Michael Eischer
00216d54a1 deduplicate prune tests a bit 2023-01-14 19:05:45 +01:00
Michael Eischer
1f3f68b2c0 prune: Don't show "packs processed" for quiet runs 2023-01-14 18:48:32 +01:00
Michael Eischer
57acc769b4 lock: Ignore empty lock files
These may be left behind by backends which do not guarantee atomic
uploads.
2023-01-14 18:15:46 +01:00
Michael Eischer
20ad14e362 lock: add help message how to recover from invalid locks 2023-01-14 18:04:22 +01:00
Michael Eischer
c995b5be52 lock: cleanup error message
The error message is now `Fatal: unable to create lock in backend:
[...]` instead of `unable to create lock in backend: Fatal: [...]`.
2023-01-14 17:57:02 +01:00
Michael Eischer
1adf28a2b5 repository: properly return invalid data error in LoadUnpacked
The retry backend does not return the original error, if its execution
is interrupted by canceling the context. Thus, we have to manually
ensure that the invalid data error gets returned.

Additionally, use the retry backend for some of the repository tests, as
this is the configuration which will be used by restic.
2023-01-14 17:57:02 +01:00
Michael Eischer
6d9675c323 repository: cleanup error message on invalid data
The retry printed the filename twice:
```
Load(<lock/04804cba82>, 0, 0) returned error, retrying after 720.254544ms: load(<lock/04804cba82>): invalid data returned
```
now the warning has changed to
```
Load(<lock/04804cba82>, 0, 0) returned error, retrying after 720.254544ms: invalid data returned
```
2023-01-14 17:57:02 +01:00
Michael Eischer
551b31ce3c Merge pull request #4146 from MichaelEischer/update-blazer
Update blazer to fix `b2_download_file_by_name: 404`
2023-01-14 15:23:32 +01:00
Michael Eischer
ec99507e4c update blazer
This removes a stray warning, that was printed when checking the status
of non-existent files.
2023-01-14 14:56:25 +01:00
Michael Eischer
5f97f534b1 Merge pull request #4150 from restic/dependabot/go_modules/google.golang.org/api-0.107.0
build(deps): bump google.golang.org/api from 0.106.0 to 0.107.0
2023-01-14 14:55:15 +01:00
Michael Eischer
ed11bbd0e2 Merge pull request #4151 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.47
build(deps): bump github.com/minio/minio-go/v7 from 7.0.46 to 7.0.47
2023-01-14 14:50:01 +01:00
dependabot[bot]
5bb9cb056d build(deps): bump github.com/minio/minio-go/v7 from 7.0.46 to 7.0.47
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.46 to 7.0.47.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.46...v7.0.47)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-14 13:38:58 +00:00
dependabot[bot]
cd9bd22563 build(deps): bump google.golang.org/api from 0.106.0 to 0.107.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.106.0 to 0.107.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.106.0...v0.107.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2023-01-14 13:38:42 +00:00
Michael Eischer
ecc826ef7d Merge pull request #4149 from MichaelEischer/more-frequent-dependabot
CI: let dependabot check for updates weekly
2023-01-14 14:37:30 +01:00
Michael Eischer
fb43cbab49 Merge pull request #4129 from greatroar/cleanup
cache: Replace readCloser+LimitedReader by backend.LimitedReadCloser
2023-01-14 12:17:25 +01:00
Michael Eischer
41d31b1e27 Merge pull request #4116 from MichaelEischer/extract-progress-updater
ui/progress: Extract progress updater
2023-01-14 12:07:26 +01:00
Michael Eischer
f6ea5c5865 CI: let dependabot check for updates weekly 2023-01-14 12:02:42 +01:00
Michael Eischer
4a7a6b06af ui/backup: Use progress.Updater for progress updates 2023-01-14 01:20:43 +01:00
Michael Eischer
e499bbe3ae progress: extract progress updating into Updater struct
This allows reusing the code to create periodic progress updates.
2023-01-14 01:13:08 +01:00
Michael Eischer
52682b1c7b Merge pull request #4111 from MichaelEischer/extract-stdio-wrapper
backup: extract StdioWrapper from ProgressPrinters
2023-01-14 01:12:23 +01:00
Michael Eischer
c15b4bceae backup: extract StdioWrapper from ProgressPrinters
The StdioWrapper is not used at all by the ProgressPrinters. It is
called a bit earlier than previously. However, as the password prompt
directly accessed stdin/stdout this doesn't cause problems.
2023-01-14 00:58:13 +01:00
Alexander Neumann
74348be3fa Set development version for 0.15.0 2023-01-12 20:51:23 +01:00
Alexander Neumann
37d0e1fe58 Add version for 0.15.0 2023-01-12 20:51:19 +01:00
Alexander Neumann
da196aa43e Update manpages and auto-completion 2023-01-12 20:51:19 +01:00
Alexander Neumann
099774c2aa Generate CHANGELOG.md for 0.15.0 2023-01-12 20:50:45 +01:00
Alexander Neumann
cd2f53e3f9 Prepare changelog for 0.15.0 2023-01-12 20:50:44 +01:00
Alexander Neumann
0c5a55d1bd Merge pull request #4142 from restic/update-deps
Upgrade dependencies
2023-01-12 20:23:25 +01:00
Michael Eischer
9ddca65f6d update dependencies 2023-01-11 23:22:10 +01:00
Michael Eischer
06fee601bc Merge pull request #4141 from restic/doc-no-scan-correction
doc: Correct heading level for --no-scan
2023-01-11 22:01:05 +01:00
Leo R. Lundgren
1cb920cc57 doc: Correct heading level for --no-scan 2023-01-11 21:47:38 +01:00
Michael Eischer
8f53ffb921 Merge pull request #4140 from restic/doc-manual
doc: Update manual page with --no-scan
2023-01-11 21:42:42 +01:00
Michael Eischer
351cbb4f94 Merge pull request #4139 from restic/doc-no-scan
doc: Move and update documentation for --no-scan
2023-01-11 21:41:20 +01:00
Leo R. Lundgren
b8b5508d15 doc: Update manual page with --no-scan 2023-01-11 02:27:43 +01:00
Leo R. Lundgren
c5542ddcd2 doc: Move and update documentation for --no-scan 2023-01-11 00:09:24 +01:00
rawtaz
dffb8e0c14 Merge pull request #4138 from MichaelEischer/doc-sparse
doc: add description for restore --sparse
2023-01-10 23:39:51 +01:00
Michael Eischer
b151fa498a doc: add description for restore --sparse 2023-01-10 23:27:42 +01:00
Michael Eischer
c354b55e62 Merge pull request #4136 from restic/doc-small-files
doc: Clarify text about tuning backups for small files
2023-01-08 22:00:09 +01:00
Michael Eischer
375953a001 Merge pull request #4088 from MichaelEischer/doc-cifs-backup-source
doc: reading from CIFS can be a problem on linux
2023-01-08 21:56:48 +01:00
Leo R. Lundgren
6306797238 doc: Clarify text about tuning backups for small files 2023-01-08 21:49:55 +01:00
Michael Eischer
ef9164fcbb doc: reading from CIFS can be a problem on linux 2023-01-08 21:47:34 +01:00
rawtaz
e2bcfd68dd Merge pull request #4135 from restic/changelogs
Polish changelogs
2023-01-08 20:04:24 +01:00
Leo R. Lundgren
33fb351386 Polish changelogs 2023-01-08 19:48:51 +01:00
Michael Eischer
c9840da4f8 Merge pull request #4134 from MichaelEischer/fix-verbose-help-text
Correct maximum verbosity level in help message
2023-01-07 22:24:04 +01:00
Michael Eischer
732184a849 Correct maximum verbosity level in help message
The maximum for `--verbose=n` is n=2. Internally it is translated into a
scale from 0 to 3. However, the default (without verbose) is 1, thus the
verbosity level can only be increased two times.
2023-01-07 22:02:13 +01:00
Michael Eischer
24178c97e9 Merge pull request #4117 from MichaelEischer/prune-dry-run-help
prune: make it clearer when prune is used in dry-run mode
2023-01-04 23:18:53 +01:00
Michael Eischer
7a36306901 forget: Clarify log message for --dry-run --prune 2023-01-04 00:44:46 +01:00
Michael Eischer
b404ad4eaa prune: make it clearer when prune is used in dry-run mode 2023-01-04 00:44:46 +01:00
Michael Eischer
e02a10c58a Merge pull request #4109 from MichaelEischer/fix-prune-uncompressed-accounting
prune: Fix calculation of remaining uncompressed data
2023-01-03 23:28:10 +01:00
Michael Eischer
81dc8c8d13 prune: Fix calculation of remaining uncompressed data
Only the repacking of *un*compressed packs reduces the amount of
uncompressed data. Previously the counter even overflowed for fully
compressed repositories.
2023-01-03 22:34:36 +01:00
greatroar
72922a79ed cache: Replace readCloser+LimitedReader by backend.LimitedReadCloser 2023-01-03 19:03:36 +01:00
Michael Eischer
89a8006578 Merge pull request #4104 from philaris/fix_max_uint32_uid_gid_to_zero
in tar dump, convert uid, gid of value -1 to zero
2023-01-02 22:28:28 +01:00
Panagiotis Cheilaris
3b516d4b70 convert uid/gid -1 to 0 only in 32-bit tar dump
Only for a 32-bit build of restic, convert a uid or gid value of -1 to 0.
2022-12-30 18:12:12 +01:00
Michael Eischer
0de3b24756 Merge pull request #4110 from MichaelEischer/remove-exitf
Remove Exitf function
2022-12-29 12:07:51 +01:00
Michael Eischer
0fbff39ae8 Merge pull request #4108 from MichaelEischer/cleanup-check-output
Cleanup check output
2022-12-29 11:59:18 +01:00
Michael Eischer
68b1f30733 Remove Exitf function
Commands should use the normal shutdown path. In addition, the Exitf
function was only used by `dump` and `restore` but not any other command
which introduces the risk of inconsistent behavior.
2022-12-28 21:42:38 +01:00
Panagiotis Cheilaris
a86a56cf3b fix lint issue with function name 'tarId'
See https://github.com/golang/lint/issues/89 and
https://github.com/golang/lint/issues/124
2022-12-28 18:46:58 +01:00
Panagiotis Cheilaris
050ed616ae be more explicit with uid or gid of value -1 2022-12-28 18:44:36 +01:00
Michael Eischer
8430399fce check: Partially fix garbled output
When reporting an error for a tree, the output message can overlap with
the progress bar output, e.g. `error for tree e91ef6fb:napshots`.

The fix only applies for this specific message and does not work on
Windows.
2022-12-28 17:47:27 +01:00
Michael Eischer
aea96b7d86 check: Slightly improve help message
If a repository has both pack/index related warnings and errors, then
the help message is quite misleading. Reword it slightly to be more
clear.
2022-12-28 17:46:06 +01:00
Michael Eischer
bcae28afb4 Merge pull request #4100 from klemensn/tag-self-update
Reinstate selfupdate tag to make builds without self-update work
2022-12-28 15:49:11 +01:00
Panagiotis Cheilaris
10fa5cde0a in tar dump, convert uid, gid of value -1 to zero 2022-12-27 16:36:04 +01:00
Klemens Nanni
61e7386384 Bugfix: Make distribution package builds without self-update work 2022-12-26 21:52:24 +04:00
Klemens Nanni
94f6e7d4a6 Reinstate selfupdate tag to make builds without self-update work
Revert what seems to be a typo introduced as part of the fix for #2041
in 2018 7d0f2eaf24.

`xbuild` does not look like a go build/tag keyword to me, I failed to
find documentation for it and using `go install -tags '!selfupdate' ...`
has no effect, i.e. self-update code is still compiled.

`+build` however works;  updating the OpenBSD port/binary package
security/restic to apply this PR works as expected:

```
	$ restic help | grep self
	$ restic self-update
	unknown command "self-update" for "restic"
```

(Using `go:build` now as per restic's style and gofmt.)

Previously, using `restic-0.14.0p1` on OpenBSD/amd64 7.2-current would
check for a newer version and probably attempt replacing the system wide
root-owned executable (on a read-only filesystem) as unprivileged user:

```
	$ restic version
	restic 0.14.0 compiled with go1.19.2 on openbsd/amd64
	$ restic help | grep self
	  self-update   Update the restic binary
	$ restic self-update
	writing restic to /usr/local/bin/restic
	find latest release of restic at GitHub
	restic is up to date
```

(It never tried to actually write besaid path;  doing so would fail, so
the current message can be considered misleading.)
2022-12-26 21:46:22 +04:00
Michael Eischer
90fb6f70b4 Merge pull request #4089 from greatroar/errors
Clean up error handling further
2022-12-24 10:41:56 +01:00
Michael Eischer
29b8500254 Merge pull request #4090 from restic/upgrade-dependencies
Upgrade dependencies
2022-12-23 22:34:39 +01:00
Michael Eischer
705cabb304 Merge pull request #3981 from MichaelEischer/prune-uncompressed-stats
prune: report how much data must be repacked to compressed the repo
2022-12-23 22:34:04 +01:00
Michael Eischer
a6f3ae5790 Merge pull request #4094 from googol42/master
remove duplicated init
2022-12-23 22:33:01 +01:00
Andreas Dominik Preikschat
ea37240597 remove duplicated init
the documentation contained the `init` command twice
2022-12-20 17:24:56 +01:00
Michael Eischer
bd2f6aaac3 azure: downgrade azblob dependency due to build breakages on Solaris 2022-12-17 23:35:07 +01:00
Michael Eischer
583372956b Upgrade dependencies
Nothing special has changed.
2022-12-17 15:23:11 +01:00
greatroar
1678392a6d checker: Make ErrLegacyLayout a value, not a type 2022-12-17 09:41:07 +01:00
greatroar
d9002f050e backend: Don't Wrap errors from url.Parse
The messages from url.Error.Error already start with the word "parse".
2022-12-17 09:41:07 +01:00
greatroar
b150dd0235 all: Replace some errors.Wrap calls by errors.WithStack
Mostly changed the ones that repeat the name of a system call, which is
already contained in os.PathError.Op. internal/fs.Reader had to be
changed to actually return such errors.
2022-12-17 09:41:07 +01:00
Michael Eischer
cccc17e4e9 Merge pull request #4086 from blackpiglet/modify_access_denied_code
Fix: change error code in function isAccessDenied to AccessDenied
2022-12-16 21:55:52 +01:00
Michael Eischer
2723159ed4 Merge pull request #3931 from kjetilho/feature/optional_scanner
add --no-scan to backup command
2022-12-16 21:42:22 +01:00
Xun Jiang/Bruce Jiang
ecc62c8be2 Update changelog/unreleased/issue-4085
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
Signed-off-by: Xun Jiang <blackpiglet@gmail.com>
2022-12-16 21:41:16 +01:00
Xun Jiang
cc5325d22b Fix: change error code in function isAccessDenied to AccessDenied
Signed-off-by: Xun Jiang <blackpiglet@gmail.com>
2022-12-16 21:41:16 +01:00
Michael Eischer
da0e45cf40 Merge pull request #4083 from greatroar/cleanup
repository: Remove empty cleanup functions in tests
2022-12-16 21:39:30 +01:00
Kjetil Torgrim Homme
14aa6f2a00 add --disable-scanner to backup command
The scanner process has only cosmetic effect for the progress printer,
and can be disabled without impacting functionality when the user does
not need an estimate of completion.

In many cases the scanner process can provide beneficial priming of
the file system cache, so as general advice it should not be disabled.
However, tests have shown that backup of NFS and fuse based filesystems,
where stat(2) is relatively expensive, can be significantly faster
without the scanner.
2022-12-16 21:29:59 +01:00
Michael Eischer
7bdb985dde Merge pull request #4079 from MichaelEischer/rewrite-set-original
rewrite: Always set the Original field in a rewritten snapshot
2022-12-13 22:56:20 +01:00
Michael Eischer
1bfe98bdc0 Merge pull request #2398 from DanielG/b2-hide-file
b2: Fallback to b2_hide_file when delete returns unauthorized
2022-12-13 22:52:23 +01:00
Michael Eischer
1c071a462e Merge pull request #4084 from ekarlso/azure-stat-fix
fix: Make create not error out when ContainerNotFound
2022-12-13 22:49:05 +01:00
Michael Eischer
25d22d5241 Merge pull request #4082 from MichaelEischer/unbuffered-logger-for-testing
Don't buffer the golang `log` package output when running tests
2022-12-13 22:45:50 +01:00
Endre Karlson
7dd33c0ecc azure: Make create not error out when ContainerNotFound 2022-12-11 22:57:23 +01:00
greatroar
c0b5ec55ab repository: Remove empty cleanup functions in tests
TestRepository and its variants always returned no-op cleanup functions.
If they ever do need to do cleanup, using testing.T.Cleanup is easier
than passing these functions around.
2022-12-11 11:06:25 +01:00
Michael Eischer
2e3d4640be Don't buffer the golang log output when running tests 2022-12-10 16:08:27 +01:00
Michael Eischer
38b2e9b42c rewrite: Always set the Original field in a rewritten snapshot
The Original field is meant to remember the original snapshot id if e.g.
changing its tags. It was only set by the `rewrite` command if it was
not set previously. However, a rewritten snapshot is potentially rather
different from the original snapshot. Thus just always set the Original
field. This also makes it easier to later on detect and potentially
remove the original snapshots.
2022-12-10 12:47:00 +01:00
Michael Eischer
049a105ba5 Merge pull request #4077 from greatroar/cleanup
test: Use testing.T.Cleanup to remove tempdirs
2022-12-09 22:17:46 +01:00
Michael Eischer
4b98b5562d Merge pull request #4075 from greatroar/sftp-enospc
sftp: Fix ENOSPC check
2022-12-09 22:00:13 +01:00
greatroar
f90bf84ba7 test: Use testing.T.Cleanup to remove tempdirs 2022-12-09 14:23:55 +01:00
greatroar
83d23b3ae8 Changelog for ENOSPC handling bug 2022-12-09 08:50:30 +01:00
Michael Eischer
eae7366563 Merge pull request #4028 from ekarlso/use-az-blob-sdk
Switch to azblob sdk
2022-12-07 21:58:03 +01:00
Endre Karlson
25648e2501 azure: Switch to azblob sdk 2022-12-07 21:46:07 +01:00
greatroar
62520bb7b4 sftp: Fix ENOSPC check
We now check for space that is not reserved for the root user on the
remote, and the check is no longer in a defer block because it wouldn't
fire. Some change in the surrounding code may have led the deferred
function to capture the wrong err variable.

Fixes #3336.
2022-12-07 21:06:46 +01:00
rawtaz
4ba31df08f Merge pull request #4074 from greatroar/lobaro-docker
doc: Remove ref to Lobaro's Docker image
2022-12-04 18:00:08 +01:00
greatroar
5efcbe143c doc: Remove ref to Lobaro's Docker image
It hasn't been updated for a while and has restic 0.12.0. Fixes #4002.
2022-12-04 16:20:42 +01:00
Michael Eischer
0df585dd99 Merge pull request #4066 from sedlund/fix#4033
fix#4033 cmd: copy no longer lists skipped existing snapshots by default
2022-12-03 19:22:11 +01:00
Michael Eischer
223da7344e Merge pull request #4070 from restic/fix-cloud-tests
Fix cloud tests
2022-12-03 19:21:25 +01:00
Michael Eischer
2b67862420 backend/test: check that IsNotExist actually works 2022-12-03 18:56:55 +01:00
Michael Eischer
2f934f5803 gs: check against the correct error in IsNotExist 2022-12-03 18:49:54 +01:00
Michael Eischer
04d101fa94 gs/s3: remove useless os.IsNotExist check 2022-12-03 18:49:54 +01:00
Michael Eischer
579cd6dc64 azure: fix totally broken IsNotExist 2022-12-03 18:49:54 +01:00
Michael Eischer
3ebdadc58f Merge pull request #4069 from greatroar/cleanup
cache, prune, restic: Cleanup
2022-12-03 17:50:52 +01:00
Michael Eischer
bc8b2455b9 Merge pull request #4064 from MichaelEischer/flaky-abort-early-on-error
archiver: Fix flaky TestArchiverAbortEarlyOnError
2022-12-03 17:43:44 +01:00
Michael Eischer
60c6a09324 Merge pull request #4065 from MichaelEischer/flaky-rclone-failed-start
rclone: treat "file already closed" as command startup error
2022-12-03 17:42:56 +01:00
Michael Eischer
8bf6b2b80d Merge pull request #4067 from MichaelEischer/remove-backend-test-method
Remove `Test()` method from Backend
2022-12-03 17:40:55 +01:00
Michael Eischer
78ea69082a Merge pull request #4068 from MichaelEischer/debug-lock-refresh-test
Add more debug logging to `TestLockSuccessfulRefresh`
2022-12-03 17:38:21 +01:00
Scott Edlund
cbe73ace3f Update changelog/unreleased/issue-4033
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-12-03 20:07:37 +08:00
greatroar
63bed34608 restic: Clean up restic.IDs type
IDs.Less can be rewritten as

	string(list[i][:]) < string(list[j][:])

Note that this does not copy the ID's.

The Uniq method was no longer used.

The String method has been reimplemented without first copying into a
separate slice of a custom type.
2022-12-03 12:38:20 +01:00
greatroar
0c749dd358 prune: Pass fewer options around 2022-12-03 12:14:04 +01:00
greatroar
d45a2475e1 cache: Rewrite unnecessary if-else 2022-12-03 12:13:54 +01:00
Michael Eischer
6b5d6b9f2c Add more debug logging to TestLockSuccessfulRefresh
The test fails from time to time. Add some more logging to hopefully get
an idea where things go wrong.
2022-12-03 12:05:38 +01:00
Michael Eischer
648edeca40 retry: Do not retry Stat() if file does not exist
In non test/debug code, Stat() is used exclusively to check whether a
file exists. Thus, do not retry if a file is reported as not existing.
2022-12-03 11:42:48 +01:00
Michael Eischer
40ac678252 backend: remove Test method
The Test method was only used in exactly one place, namely when trying
to create a new repository it was used to check whether a config file
already exists.

Use a combination of Stat() and IsNotExist() instead.
2022-12-03 11:28:10 +01:00
sedlund
06ee0339aa fix#4033 cmd: copy no longer lists skipped existing snapshots by default 2022-12-03 09:55:39 +08:00
Michael Eischer
57d8eedb88 Merge pull request #4020 from greatroar/fuse-inode
fuse: Better inode generation
2022-12-02 22:28:15 +01:00
Michael Eischer
ca1803cacb Merge pull request #4063 from MichaelEischer/replace-ioutil-usage
Replace ioutil usage
2022-12-02 21:49:40 +01:00
Michael Eischer
0af89a5738 Merge pull request #3132 from metalsp0rk/init-json
Init command JSON output
2022-12-02 21:49:22 +01:00
Michael Eischer
364a396fd6 init: use standard name message_type to distinguish JSON messages 2022-12-02 21:33:03 +01:00
Michael Eischer
9a9f559806 init: cleanup json print code 2022-12-02 21:33:03 +01:00
Kyle Brennan
933c9af328 create changelog entry for issue-3124 and pull-3132 2022-12-02 21:32:30 +01:00
Kyle Brennan
a6ae79b39e support json output for init command 2022-12-02 21:32:30 +01:00
Michael Eischer
f3d964a8c1 rclone: treat "file already closed" as command startup error
Since #3940 the rclone backend returns the commands exit code if it
fails to start. The list of expected errors was missing the "file
already closed"-error which can occur if the http test request first
learns about the closed pipe to rclone before noticing the canceled
context.

Go internally makes sure that a file descriptor is unusable once it was
closed, thus this cannot have unintended side effects (like accidentally
reading from the wrong file due to a reused file descriptor).
2022-12-02 20:46:02 +01:00
Michael Eischer
a9972dbe7d archiver: Fix flaky TestArchiverAbortEarlyOnError
Saving the blobs of a file by now happens asynchronously to the
processing in the FileSaver. Thus we have to account for the blobs
queued for saving.
2022-12-02 20:07:34 +01:00
Michael Eischer
f755233210 Replace usages of ioutil.ReadDir
This changes the return type to []fs.DirEntry. However, as we only use
the filenames anyways, this doesn't make a difference.
2022-12-02 19:54:27 +01:00
Michael Eischer
fa20a78bb6 Merge pull request #4056 from greatroar/cleanup
backend, fs, options: Minor cleanup
2022-12-02 19:44:54 +01:00
Michael Eischer
ff7ef5007e Replace most usages of ioutil with the underlying function
The ioutil functions are deprecated since Go 1.17 and only wrap another
library function. Thus directly call the underlying function.

This commit only mechanically replaces the function calls.
2022-12-02 19:36:43 +01:00
greatroar
65612d797c backend, options: Prefer strings.Cut to SplitN
Also realigned the various "split into host🪣prefix"
implementations.
2022-12-02 19:19:14 +01:00
Michael Eischer
2d5e28e777 Merge pull request #4059 from restic/dependabot/go_modules/github.com/minio/minio-go/v7-7.0.45
build(deps): bump github.com/minio/minio-go/v7 from 7.0.44 to 7.0.45
2022-12-01 21:20:57 +01:00
dependabot[bot]
4fefa2ade2 build(deps): bump github.com/minio/minio-go/v7 from 7.0.44 to 7.0.45
Bumps [github.com/minio/minio-go/v7](https://github.com/minio/minio-go) from 7.0.44 to 7.0.45.
- [Release notes](https://github.com/minio/minio-go/releases)
- [Commits](https://github.com/minio/minio-go/compare/v7.0.44...v7.0.45)

---
updated-dependencies:
- dependency-name: github.com/minio/minio-go/v7
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-12-01 01:03:42 +00:00
Michael Eischer
3c5d1eabe9 Merge pull request #4051 from restic/dependabot/go_modules/github.com/klauspost/compress-1.15.12
build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.15.12
2022-11-28 21:47:55 +01:00
Michael Eischer
bec391ee26 Merge pull request #4053 from greatroar/xattr
Upgrade pkg/xattr to version with Solaris FIFO fix
2022-11-28 21:06:19 +01:00
greatroar
daafcaf380 Upgrade pkg/xattr to version with Solaris FIFO fix
This version doesn't have a release tag yet, but it's 0.4.9 + one patch.

Fixes #4003.
2022-11-28 20:43:51 +01:00
Alexander Neumann
1d7e7fcd6b Merge pull request #4049 from MichaelEischer/fix-rewrite-docs
rewrite: fix link anchors in documentation
2022-11-28 19:39:38 +01:00
dependabot[bot]
57d59c71e3 build(deps): bump github.com/klauspost/compress from 1.15.9 to 1.15.12
Bumps [github.com/klauspost/compress](https://github.com/klauspost/compress) from 1.15.9 to 1.15.12.
- [Release notes](https://github.com/klauspost/compress/releases)
- [Changelog](https://github.com/klauspost/compress/blob/master/.goreleaser.yml)
- [Commits](https://github.com/klauspost/compress/compare/v1.15.9...v1.15.12)

---
updated-dependencies:
- dependency-name: github.com/klauspost/compress
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-28 18:39:35 +00:00
Alexander Neumann
bb83c78ee5 Merge pull request #4047 from MichaelEischer/clean-ci-configuration
Cleanup CI configuration
2022-11-28 19:38:33 +01:00
greatroar
60aa87bbab fs: Remove explicit type check in extendedStat
Without comma-ok, the runtime inserts the same check with a similar
enough panic message:

    interface conversion: interface {} is nil, not *syscall.Stat_t
2022-11-27 19:58:06 +01:00
Michael Eischer
34609bca0e Merge pull request #4050 from greatroar/lruv2
bloblru: Upgrade to hashicorp/golang-lru/v2
2022-11-27 17:37:14 +01:00
greatroar
e5d597fd22 bloblru: Upgrade to hashicorp/golang-lru/v2
The new genericized LRU cache no longer needs to have the IDs separately
allocated:

name   old time/op    new time/op    delta
Add-8     494ns ± 2%     388ns ± 2%  -21.46%  (p=0.000 n=10+9)

name   old alloc/op   new alloc/op   delta
Add-8      176B ± 0%      152B ± 0%  -13.64%  (p=0.000 n=10+10)

name   old allocs/op  new allocs/op  delta
Add-8      5.00 ± 0%      3.00 ± 0%  -40.00%  (p=0.000 n=10+10)
2022-11-27 17:18:13 +01:00
Michael Eischer
0eddc89e98 doc: design.rst: Fix highlighting for index snippet
JSON does not support comments. As JSON is a subset of Javascript, use
the latter instead.
2022-11-27 17:01:27 +01:00
Michael Eischer
41b0f1d43a doc: fix link to amazon s3 section 2022-11-27 17:01:22 +01:00
Michael Eischer
6a793db9ca rewrite: fix link anchors in documentation 2022-11-27 16:38:10 +01:00
Michael Eischer
05cebc1c4b Merge pull request #4044 from restic/dependabot/go_modules/cloud.google.com/go/storage-1.28.0
build(deps): bump cloud.google.com/go/storage from 1.25.0 to 1.28.0
2022-11-27 15:23:06 +01:00
Michael Eischer
ce39727846 Merge pull request #4036 from restic/dependabot/go_modules/github.com/pkg/profile-1.7.0
build(deps): bump github.com/pkg/profile from 1.6.0 to 1.7.0
2022-11-27 15:22:16 +01:00
Michael Eischer
9aa06ce959 CI: remove option to configure command used to install go tools
With the minimum required go version of 1.18, we always use `go
install`.
2022-11-27 15:07:29 +01:00
Michael Eischer
5968971313 CI: remove dependabot ignore for bazil.org/fuse
We've switched to a fork of the original library, thus the ignore is no
longer necessary.
2022-11-27 15:06:30 +01:00
dependabot[bot]
95374767de build(deps): bump github.com/pkg/profile from 1.6.0 to 1.7.0
Bumps [github.com/pkg/profile](https://github.com/pkg/profile) from 1.6.0 to 1.7.0.
- [Release notes](https://github.com/pkg/profile/releases)
- [Commits](https://github.com/pkg/profile/compare/v1.6.0...v1.7.0)

---
updated-dependencies:
- dependency-name: github.com/pkg/profile
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 14:02:32 +00:00
dependabot[bot]
c100a62ebf build(deps): bump cloud.google.com/go/storage from 1.25.0 to 1.28.0
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.25.0 to 1.28.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.25.0...spanner/v1.28.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 14:01:36 +00:00
Michael Eischer
c41a1b66e1 Merge pull request #4037 from restic/dependabot/go_modules/google.golang.org/api-0.103.0
build(deps): bump google.golang.org/api from 0.93.0 to 0.103.0
2022-11-27 15:00:16 +01:00
dependabot[bot]
705aed0ecb build(deps): bump google.golang.org/api from 0.93.0 to 0.103.0
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.93.0 to 0.103.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.93.0...v0.103.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 13:45:08 +00:00
Michael Eischer
28d6de648c Merge pull request #4040 from restic/dependabot/go_modules/github.com/spf13/cobra-1.6.1
build(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.1
2022-11-27 14:44:25 +01:00
Michael Eischer
bb40b55d1c Merge pull request #4038 from restic/dependabot/go_modules/github.com/cenkalti/backoff/v4-4.2.0
build(deps): bump github.com/cenkalti/backoff/v4 from 4.1.3 to 4.2.0
2022-11-27 14:13:39 +01:00
dependabot[bot]
a24c1e99a6 build(deps): bump github.com/cenkalti/backoff/v4 from 4.1.3 to 4.2.0
Bumps [github.com/cenkalti/backoff/v4](https://github.com/cenkalti/backoff) from 4.1.3 to 4.2.0.
- [Release notes](https://github.com/cenkalti/backoff/releases)
- [Commits](https://github.com/cenkalti/backoff/compare/v4.1.3...v4.2.0)

---
updated-dependencies:
- dependency-name: github.com/cenkalti/backoff/v4
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 12:57:41 +00:00
dependabot[bot]
fd56ead4a8 build(deps): bump github.com/spf13/cobra from 1.5.0 to 1.6.1
Bumps [github.com/spf13/cobra](https://github.com/spf13/cobra) from 1.5.0 to 1.6.1.
- [Release notes](https://github.com/spf13/cobra/releases)
- [Commits](https://github.com/spf13/cobra/compare/v1.5.0...v1.6.1)

---
updated-dependencies:
- dependency-name: github.com/spf13/cobra
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-27 12:57:23 +00:00
Michael Eischer
cc679c6494 Merge pull request #4041 from MichaelEischer/require-go-1.18
Require go 1.18
2022-11-27 13:56:43 +01:00
greatroar
c9c7671c58 fuse: Clean up inode generation 2022-11-27 13:53:42 +01:00
Michael Eischer
530f129a39 rest: remove workaround for content-length handling bug 2022-11-27 13:18:44 +01:00
Michael Eischer
8ad231bcad bump version numbers in instructions to reproduce binaries 2022-11-27 13:18:44 +01:00
Michael Eischer
a1eb923876 remove no longer necessary conditional compiles 2022-11-27 13:18:44 +01:00
Michael Eischer
bcdfc2a8ea CI: allow dependabot update of oauth2
Our minimum go version is new enough to allow updating the library.
2022-11-27 13:18:44 +01:00
Michael Eischer
686b0b2a3e update the minimum required go version to 1.18 2022-11-27 13:18:43 +01:00
Michael Eischer
69a2e81bd3 Merge pull request #4039 from restic/dependabot/go_modules/github.com/google/go-cmp-0.5.9
build(deps): bump github.com/google/go-cmp from 0.5.8 to 0.5.9
2022-11-26 17:39:27 +01:00
dependabot[bot]
278e93f738 build(deps): bump github.com/google/go-cmp from 0.5.8 to 0.5.9
Bumps [github.com/google/go-cmp](https://github.com/google/go-cmp) from 0.5.8 to 0.5.9.
- [Release notes](https://github.com/google/go-cmp/releases)
- [Commits](https://github.com/google/go-cmp/compare/v0.5.8...v0.5.9)

---
updated-dependencies:
- dependency-name: github.com/google/go-cmp
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
2022-11-26 11:35:05 +00:00
Michael Eischer
747d2ecd7b Merge pull request #4042 from restic/skip-cloud-test-for-dependabot
CI: skip cloud tests for dependabot pull requests
2022-11-26 12:34:16 +01:00
Michael Eischer
98c6ca9d8f CI: skip cloud tests for dependabot pull requests 2022-11-26 12:23:55 +01:00
Michael Eischer
9113b2620f Merge pull request #4024 from MichaelEischer/macos-fuse
mount: switch to anacrolix fork of bazil/fuse
2022-11-26 12:15:14 +01:00
Michael Eischer
f115d64634 Merge pull request #4022 from MichaelEischer/race-checker
CI: Run the golang race checker
2022-11-26 12:13:50 +01:00
Michael Eischer
923c06cea0 Merge pull request #4025 from MichaelEischer/update-minio
Update minio library to add `credential_process` support
2022-11-25 23:21:57 +01:00
Michael Eischer
f4d3ed77c4 update minio library 2022-11-25 22:36:21 +01:00
greatroar
189e0fe5a9 fuse: Better inode generation
Hard links to the same file now get the same inode within the FUSE
mount. Also, inode generation is faster and, more importantly, no longer
allocates.

Benchmarked on Linux/amd64. Old means the benchmark with

        sink = fs.GenerateDynamicInode(1, sub.node.Name)

instead of calling inodeFromNode. Results:

name                   old time/op    new time/op    delta
Inode/no_hard_links-8     137ns ± 4%      34ns ± 1%   -75.20%  (p=0.000 n=10+10)
Inode/hard_link-8        33.6ns ± 1%     9.5ns ± 0%   -71.82%  (p=0.000 n=9+8)

name                   old alloc/op   new alloc/op   delta
Inode/no_hard_links-8     48.0B ± 0%      0.0B       -100.00%  (p=0.000 n=10+10)
Inode/hard_link-8         0.00B          0.00B           ~     (all equal)

name                   old allocs/op  new allocs/op  delta
Inode/no_hard_links-8      1.00 ± 0%      0.00       -100.00%  (p=0.000 n=10+10)
Inode/hard_link-8          0.00           0.00           ~     (all equal)
2022-11-16 08:35:01 +01:00
Michael Eischer
32ffcd86a2 Merge pull request #3993 from MichaelEischer/backup-json-full-snapshot-id
backup: print full snapshot id in JSON summary
2022-11-12 20:42:35 +01:00
Michael Eischer
f032a9d0ad prune: report how much data must be repacked to compressed the repo
prune now reports the remaining size of pack files containing
uncompressed blobs. The displayed value is suitable for use with `--max-repack-size`.
2022-11-12 20:20:23 +01:00
Michael Eischer
66818a8f98 Merge pull request #3980 from MichaelEischer/prune-compression-stats
prune: Correctly count used/duplicate blobs for partially compressed repos
2022-11-12 20:06:56 +01:00
Michael Eischer
4b5234924b Merge pull request #2875 from fgma/issue2699
issue2699: restore symlinks on windows when run as admin user
2022-11-12 20:06:45 +01:00
Michael Eischer
726a1969cd Merge pull request #2731 from dionorgua/rewrite-snapshot
Implement 'rewrite' command to exclude files from existing snapshots
2022-11-12 20:06:35 +01:00
Michael Eischer
bb0fa76c06 Cleanup exclude pattern collection 2022-11-12 19:55:22 +01:00
Michael Eischer
537cfe2e4c rewrite: Fix check that an exclude pattern was passed
The old check did not consider files containing case insensitive
excludes. The check is now implemented as a function of the
excludePatternOptions struct to improve cohesion.
2022-11-12 19:55:22 +01:00
Leo R. Lundgren
f175da2756 rewrite: Polish documentation 2022-11-12 19:55:22 +01:00
Leo R. Lundgren
f86ef4d3dd rewrite: Polish code and add missing messages 2022-11-12 19:55:22 +01:00
Leo R. Lundgren
c15bedccc0 rewrite: Revert unrelated documentation change 2022-11-12 19:55:22 +01:00
Michael Eischer
f88acd4503 rewrite: Fail if a tree contains an unknown field
In principle, the JSON format of Tree objects is extensible without
requiring a format change. In order to not loose information just play
it safe and reject rewriting trees for which we could loose data.
2022-11-12 19:55:22 +01:00
Michael Eischer
11b8c3a158 rewrite: add documentation 2022-11-12 19:55:22 +01:00
Michael Eischer
ec0c91e233 rewrite: Add tests for further ways to use the command 2022-11-12 19:55:22 +01:00
Michael Eischer
0224e276ec walker: Add tests for FilterTree 2022-11-12 19:55:22 +01:00
Michael Eischer
73f54cc5ea rewrite: rename --inplace to --forget 2022-11-12 19:55:22 +01:00
Michael Eischer
a47d9a1c40 rewrite: use unified snapshot filter options 2022-11-12 19:55:22 +01:00
Michael Eischer
b044649118 rewrite: add minimal test 2022-11-12 19:55:22 +01:00
Michael Eischer
375a3db64d rewrite: non-exclusive lock if snapshots are only added 2022-11-12 19:55:22 +01:00
Michael Eischer
327f418a9c rewrite: cleanup err handling and output 2022-11-12 19:55:22 +01:00
Michael Eischer
ad14d6e4ac rewrite: use SelectByName like in the backup command 2022-11-12 19:55:22 +01:00
Michael Eischer
7ebaf6e899 rewrite: start repository uploader goroutines 2022-11-12 19:55:22 +01:00
Michael Eischer
559acea0d8 unify exclude pattern options 2022-11-12 19:55:22 +01:00
Michael Eischer
4cace1ffe9 unify exclude patterns with backup command 2022-11-12 19:55:22 +01:00
Michael Eischer
2b69a1c53b rewrite: filter all snapshots if none are specified 2022-11-12 19:55:22 +01:00
Michael Eischer
f6339b88af rewrite: extract tree filtering 2022-11-12 19:55:22 +01:00
Michael Eischer
c0f7ba2388 rewrite: simplify dryrun 2022-11-12 19:55:22 +01:00
Michael Eischer
4d6ab83019 rewrite: use treejsonbuilder 2022-11-12 19:55:22 +01:00
Michael Eischer
82592b88b5 rewrite: address most review comments 2022-11-12 19:55:22 +01:00
Michael Eischer
b922774343 rewrite: fix compilation 2022-11-12 19:55:22 +01:00
Dmitry Nezhevenko
dc29709742 Implement 'rewrite' command to exclude files from existing snapshots 2022-11-12 19:55:22 +01:00
Michael Eischer
220eaee76b mount: switch to anacrolix fork of bazil/fuse
The anacrolix fork contains the latest changes from bazil/fuse and
additionally provides support for recent versions of macFUSE.
2022-11-12 19:22:31 +01:00
Michael Eischer
6fa45d0d39 Merge pull request #4011 from greatroar/backup-stdin-password
cmd: Don't read password from stdin for backup --stdin
2022-11-12 19:18:56 +01:00
Michael Eischer
bbd180ae21 Merge pull request #4017 from Rajpratik71/Rajpratik71-patch-1
feat: dependabot workflow automation for updating dependency
2022-11-12 15:48:48 +01:00
Pratik Raj
bef1064b8e chore: ignore upgrade for 'bazil/fuse' and 'golang.org/x/oauth2' 2022-11-12 19:39:16 +05:30
Michael Eischer
7b4fe7bad5 Merge pull request #4021 from greatroar/mac-fsync
backend/local: Ignore ENOTTY for fsync on Mac
2022-11-11 23:10:37 +01:00
greatroar
348e966daa backend/local: Ignore ENOTTY for fsync on Mac
Fixes #4016.
2022-11-11 22:51:51 +01:00
Michael Eischer
0e5fe4c6ab CI: run golang race checker 2022-11-11 22:15:22 +01:00
Michael Eischer
13fbc96ed3 lock: Synchronize Refresh() and Stale()
The lock test creates a lock and checks that it is not stale. However,
it is possible that the lock is refreshed concurrently, which updates
the lock timestamp. Checking the timestamp in `Stale()` without
synchronization results in a data race. Thus add a lock to prevent
concurrent accesses.
2022-11-11 21:52:53 +01:00
Michael Eischer
e1ba7ab684 lock: Don't copy the lock when checking for process existence
The lock test creates a lock and checks that it is not stale. This also
tests whether the corresponding process still exists. However, it is
possible that the lock is refreshed concurrently, which updates the lock
timestamp. Calling `processExists()` with a value receiver, however,
creates an unsynchronized copy of this field. Thus call the method using
a pointer receiver.
2022-11-11 21:45:55 +01:00
Michael Eischer
dc060356c2 mount: only start next test after mount command cleanup is complete
The test did not wait for the mount command to fully shutdown all
running goroutines. This caused the go race detector to report a data
race related to lock refreshes.

==================
WARNING: DATA RACE
Write at 0x0000021bdfdb by goroutine 667:
  github.com/restic/restic/internal/backend/retry.TestFastRetries()
      /restic/restic/internal/backend/retry/testing.go:7 +0x18f
  github.com/restic/restic/cmd/restic.withTestEnvironment()
      /restic/restic/cmd/restic/integration_helpers_test.go:175 +0x183
  github.com/restic/restic/cmd/restic.TestMountSameTimestamps()
      /restic/restic/cmd/restic/integration_fuse_test.go:202 +0xac
  testing.tRunner()
      /usr/lib/go/src/testing/testing.go:1446 +0x216
  testing.(*T).Run.func1()
      /usr/lib/go/src/testing/testing.go:1493 +0x47

Previous read at 0x0000021bdfdb by goroutine 609:
  github.com/restic/restic/internal/backend/retry.(*Backend).retry()
      /restic/restic/internal/backend/retry/backend_retry.go:72 +0x9e
  github.com/restic/restic/internal/backend/retry.(*Backend).Remove()
      /restic/restic/internal/backend/retry/backend_retry.go:149 +0x17d
  github.com/restic/restic/internal/cache.(*Backend).Remove()
      /restic/restic/internal/cache/backend.go:38 +0x11d
  github.com/restic/restic/internal/restic.(*Lock).Unlock()
      /restic/restic/internal/restic/lock.go:190 +0x249
  github.com/restic/restic/cmd/restic.refreshLocks.func1()
      /restic/restic/cmd/restic/lock.go:86 +0xae
  runtime.deferreturn()
      /usr/lib/go/src/runtime/panic.go:476 +0x32
  github.com/restic/restic/cmd/restic.lockRepository.func2()
      /restic/restic/cmd/restic/lock.go:61 +0x71

[...]

Goroutine 609 (finished) created at:
  github.com/restic/restic/cmd/restic.lockRepository()
      /restic/restic/cmd/restic/lock.go:61 +0x488
  github.com/restic/restic/cmd/restic.lockRepo()
      /restic/restic/cmd/restic/lock.go:25 +0x219
  github.com/restic/restic/cmd/restic.runMount()
      /restic/restic/cmd/restic/cmd_mount.go:126 +0x1f8
  github.com/restic/restic/cmd/restic.testRunMount()
      /restic/restic/cmd/restic/integration_fuse_test.go:61 +0x1ce
  github.com/restic/restic/cmd/restic.checkSnapshots.func1()
      /restic/restic/cmd/restic/integration_fuse_test.go:90 +0x124
==================
2022-11-11 21:43:01 +01:00
Michael Eischer
32c9667990 Merge pull request #4019 from MichaelEischer/fix-file-saver-race
archiver: Fix race condition resulting in files containing null IDs
2022-11-11 20:52:33 +01:00
Michael Eischer
d268552a0a Merge pull request #4014 from MichaelEischer/fix-debug-examine
debug: fix crash in `debug examine --reupload-blobs`
2022-11-10 20:37:32 +01:00
Michael Eischer
5756c96c9f archiver: Fix race condition resulting in files containing null IDs
In some rare cases files could be created which contain null IDs (all
zero) in their content list. This was caused by a race condition between
growing the `Content` slice and inserting the blob IDs into it. In some
cases the blob ID was written to the old slice, which a short time
afterwards was replaced with a larger copy, that did not yet contain the
blob ID.
2022-11-10 20:19:37 +01:00
Pratik Raj
df614fff26 feat: dependabot workflow automation for updating dependency
Signed-off-by: Pratik Raj <Rajpratik71@gmail.com>
2022-11-10 16:02:03 +05:30
Michael Eischer
11a4bb051e debug: fix crash in debug examine --reupload-blobs 2022-11-09 22:13:17 +01:00
Michael Eischer
5f9ac2b165 Merge pull request #4010 from MichaelEischer/file-saver-sanity-check
archiver: Check that saved file does not have null IDs in content
2022-11-08 23:07:32 +01:00
Michael Eischer
b1d1202b1d archiver: Check that saved file does not have null IDs in content
Null IDs in the file content indicate that something went wrong. Thus
fails before saving the affected file.
2022-11-08 22:57:41 +01:00
greatroar
5dceadeb72 cmd: Don't read password from stdin for backup --stdin 2022-11-06 14:55:57 +01:00
Michael Eischer
1ccab95bc4 b2: Support file hiding instead of deleting them permanently
Automatically fall back to hiding files if not authorized to permanently
delete files. This allows using restic with an append-only application
key with B2.  Thus, an attacker cannot directly delete backups with the
API key used by restic.

To use this feature create an application key without the deleteFiles
capability. It is recommended to restrict the key to just one bucket.
For example using the b2 command line tool:

    b2 create-key --bucket <bucketName> <keyName> listBuckets,readFiles,writeFiles,listFiles

Suggested-by: Daniel Gröber <dxld@darkboxed.org>
2022-11-05 20:10:45 +01:00
Michael Eischer
24a2e5cab9 Merge pull request #4008 from MichaelEischer/tweak-lock-refresh-test
lock: Tweak timeouts for lock refresh test
2022-11-05 10:53:13 +01:00
Michael Eischer
403390479c Merge pull request #3997 from greatroar/fuse-hash
fuse: Better check for whether snapshots changed
2022-11-05 10:52:11 +01:00
Michael Eischer
d29abc1a31 Merge pull request #4007 from MichaelEischer/hide-compression-level-for-v1-repo
Only print compression level starting from repository version 2
2022-11-05 10:33:25 +01:00
greatroar
c091e43b33 fuse: Better check for whether snapshots changed
We previously checked whether the set of snapshots might have changed
based only on their number, which fails when as many snapshots are
forgotten as are added. Check for the SHA-256 of their id's instead.
2022-11-05 09:32:45 +01:00
Michael Eischer
aaac63da8d lock: Tweak timeouts for lock refresh test
For some reason the test fails from time to time. Increase the timeouts
to hopefully avoid this issue.
2022-11-04 22:48:18 +01:00
Michael Eischer
fd4d23460f only print compression level starting from repository version 2 2022-11-04 22:40:07 +01:00
Alexander Neumann
8dd95b710e Merge pull request #3992 from MichaelEischer/err-on-invalid-compression
Return error if RESTIC_COMPRESSION env variable is invalid
2022-11-04 19:41:34 +01:00
Alexander Neumann
783b8781a7 Merge pull request #4000 from restic/min-go-version
build: Correct checks for minimum Go version
2022-11-04 10:31:02 +01:00
Alexander Neumann
543649f2f2 Merge pull request #4001 from restic/docker-go-version
docker: Increase Go version to 1.19
2022-11-04 10:30:11 +01:00
Leo R. Lundgren
0a4cddb34d docker: Increase Go version to 1.19 2022-11-03 22:59:59 +01:00
Leo R. Lundgren
333c2c6ed4 build: Correct checks for minimum Go version 2022-11-03 22:50:07 +01:00
rawtaz
92df039e5d Merge pull request #3996 from MichaelEischer/fix-ui-progress
backup: fix stuck status bar
2022-11-02 21:48:16 +01:00
Michael Eischer
9354262b1b backup: fix stuck status bar
The status bar got stuck once the first error was reported, the scanner
completed or some file was backed up. Either case sets a flag that the
scanner has started.

This flag is used to hide the progress bar until the flag is set. Due to
an inverted condition, the opposite happened and the status stopped
refreshing once the flag was set.

In addition, the scannerStarted flag was not set when the scanner just
reported progress information.
2022-11-02 21:31:13 +01:00
Michael Eischer
06141ce1f4 backup: print full snapshot id in JSON summary 2022-10-31 19:03:42 +01:00
Michael Eischer
59a90943bb Merge pull request #3983 from greatroar/formatting
Centralize and fix formatting of bytes, percentages, durations
2022-10-31 18:52:24 +01:00
greatroar
5ab3e6276a ui: Fix FormatBytes at exactly 1024 time a unit
1024 would be displayed as "1024 bytes" instead of "1.000 KiB", etc.
2022-10-31 18:39:28 +01:00
rawtaz
4f1fae9c98 Merge pull request #3982 from MichaelEischer/show-compression-mode
Show selected compression level when opening repository
2022-10-30 21:29:42 +01:00
Michael Eischer
8fe159cc5a enable ysmlink tests for windows 2022-10-30 18:43:04 +01:00
Michael Eischer
3499c6354e Merge pull request #3955 from MichaelEischer/async-futurefile-completion
Improve archiver performance for small files
2022-10-30 18:38:04 +01:00
Michael Eischer
144257f8bd restore symlink timestamps on windows 2022-10-30 11:04:04 +01:00
Michael Eischer
c0f34af9db backup: hide files from status which are read completely but not saved
As the FileSaver is asynchronously waiting for all blobs of a file to be
stored, the number of active files is higher than the number of files
from which restic is reading concurrently. Thus to not confuse users,
only display files in the status from which restic is currently reading.
2022-10-30 10:29:12 +01:00
Michael Eischer
a571fc4aa1 add changelog for faster backups with small files 2022-10-30 10:29:12 +01:00
Michael Eischer
b52a8ff05c ui: Properly clear lines no longer used for status
Previously, the old status text remained until it was overwritten.
2022-10-30 10:29:12 +01:00
Michael Eischer
b4de902596 archiver: Asynchronously complete FutureFile
After reading and chunking all data in a file, the FutureFile still has
to wait until the FutureBlobs are completed. This was done synchronously
which results in blocking the file saver and prevents the next file from
being read.

By replacing the FutureBlob with a callback, it becomes possible to
complete the FutureFile asynchronously.
2022-10-30 10:29:11 +01:00
Michael Eischer
47e05080a9 Merge pull request #3990 from MichaelEischer/lock-refresh-test
lock: add test to check that refreshing works
2022-10-30 10:15:44 +01:00
Michael Eischer
c7ace314f6 Merge pull request #3989 from greatroar/eachbypack
More compact data structure for Index.EachByPack
2022-10-30 00:02:55 +02:00
greatroar
0e8893dae9 index: Compact data structure for Index.EachByPack 2022-10-29 23:09:17 +02:00
greatroar
137f0bc944 repository: Fix benchmarkSaveAndEncrypt 2022-10-29 23:09:17 +02:00
Michael Eischer
01f0db4e56 return error if RESTIC_COMPRESSION env variable is invalid 2022-10-29 22:03:39 +02:00
Michael Eischer
7c87fb941c Merge pull request #3986 from greatroar/counter
ui/progress: Load both values in a single Lock/Unlock
2022-10-29 21:50:55 +02:00
Michael Eischer
3b0bb02a68 Merge pull request #3977 from greatroar/progress
ui/backup: Replace channels with a mutex
2022-10-29 21:33:04 +02:00
Michael Eischer
0d260cfd82 enable symlink test on windows 2022-10-29 21:26:34 +02:00
fgma
8e5eb1090c issue2699: restore symlinks on windows when run as admin user 2022-10-29 21:19:33 +02:00
rawtaz
af3f7c866f Merge pull request #3988 from FelixBurkhard/FelixBurkhard-patch-1
Clearify what Azure account name means
2022-10-29 13:32:44 +02:00
Michael Eischer
24267e9a9d lock: add test to check that refreshing works 2022-10-29 11:26:00 +02:00
Michael Eischer
8e51e1e605 shorten 'repository opened' output 2022-10-29 11:22:00 +02:00
FelixBurkhard
575d26ec87 Clearify what Azure account name means
When reading it first it was not clear to me the 'account name' meant the name of the
Azure Storage Account and not the Azure account itself.
2022-10-29 00:27:43 +02:00
greatroar
2dafda9164 ui/progress: Load both values in a single Lock/Unlock
We always need both values, except in a test, so we don't need to lock
twice and risk scheduling in between.

Also, removed the resetting in Done. This copied a mutex, which isn't
allowed. Static analyzers tend to trip over that.
2022-10-25 07:55:24 +02:00
Michael Eischer
f8910bc4ff Merge pull request #3985 from saltsa/fix_lock_refresh
Fix bug in lock refresh monitoring
2022-10-24 22:59:18 +02:00
Joonas Aunola
b06427c9f6 fix Unix to UnixNano 2022-10-23 23:40:21 +03:00
greatroar
006380199e cmd, ui: Deduplicate formatting utilities 2022-10-23 13:40:07 +02:00
greatroar
04216eb9aa ui/backup: Replace channels with a mutex
The channel-based algorithm had grown quite complicated. This is easier
to reason about and likely to be more performant with very many
CompleteBlob calls.
2022-10-23 13:28:41 +02:00
Michael Eischer
4fea3a413d show selected compression level when opening repository 2022-10-22 20:18:46 +02:00
Michael Eischer
ba58ccbe07 prune: add remark about non-deterministic blob selection 2022-10-22 19:46:10 +02:00
Michael Eischer
05651d6d4f prune: Correctly count used/duplicate blobs for partially compressed repos
Counting the first occurrence of a duplicate blob as used and counting
all other as duplicates, independent of which instance of the blob is
kept, is only accurate if all copies of the blob have the same size. This
is no longer the case for a repository containing both compressed and
uncompressed blobs.

Thus for duplicated blobs first count all instances as duplicates and
then subtract the actually used instance later on.
2022-10-22 19:24:36 +02:00
Michael Eischer
b57d42905c Merge pull request #3899 from MichaelEischer/less-prune-mem
Optimize prune memory usage
2022-10-22 18:56:02 +02:00
Michael Eischer
d966c52707 prune: allow gc of set of repacked blobs before index rebuild 2022-10-22 18:45:12 +02:00
Michael Eischer
1e2794fa55 add prune memory optimization changelog 2022-10-22 18:45:12 +02:00
Michael Eischer
68c9cb9c6a prune: Shrink keepBlobs set if possible
As long as only a small fraction of the data in a repository is
rewritten, the keepBlobs set will be rather small after cleaning it up.
As golang maps do not shrink their memory usage, just copy the contents
over to a new map. However, only copy the map if the cleanup removed at
least half the entries.
2022-10-22 18:45:12 +02:00
Michael Eischer
c4fc5c97f9 prune: Use a single CountedBlobSet to track blobs
The set covers necessary, existing and duplicate blobs. This removes the
duplicate sets used to track whether all necessary blobs also exist.
This reduces the memory usage of prune by about 20-30%.
2022-10-22 18:45:12 +02:00
Michael Eischer
b21241ec1c restic: Add CountedBlobSet type
This allows maintaining a usage counter for each blob.
2022-10-22 18:45:12 +02:00
Michael Eischer
ee6688a9f6 Merge pull request #3915 from plumbeo/compression-stats
restic stats: print uncompressed size in mode raw-data
2022-10-21 22:10:29 +02:00
Michael Eischer
27634a1a68 Merge pull request #3978 from MichaelEischer/fix-negative-pattern-example
Remove misleading wildcard from negative exclude pattern example
2022-10-21 22:04:30 +02:00
Michael Eischer
aa77702e49 Merge pull request #3971 from MichaelEischer/parallel-list
Unify ForAllIndex/Snapshot/Lock functions
2022-10-21 21:58:33 +02:00
Michael Eischer
6877aaa8aa Merge pull request #3967 from MichaelEischer/archiver-extract-exclude-options
backup: extract exclude pattern options
2022-10-21 21:50:00 +02:00
Michael Eischer
2e9ee8577a Merge pull request #3970 from MichaelEischer/split-retry-backend
Split backend package into smaller parts
2022-10-21 21:49:46 +02:00
Michael Eischer
59d46bb3f5 backup: extract exclude pattern options
This is a preparation to make the exclude options usable for the
upcoming `rewrite` command.
2022-10-21 21:40:59 +02:00
Michael Eischer
5c7a9a739a backend: Split RetryBackend into own package
The RetryBackend tests depend on the mock backend. When the Backend
interface is eventually split from the restic package, this will lead to
a dependency cycle between backend and backend/mock. Thus split the
RetryBackend into a separate package to avoid this problem.
2022-10-21 21:38:17 +02:00
Michael Eischer
32603d49c4 backend: remove unused ErrorBackend 2022-10-21 21:36:05 +02:00
Michael Eischer
8c18c65b3b backend: remove unused Paths variable 2022-10-21 21:36:05 +02:00
Michael Eischer
4ccd5e806b backend: split layout code into own subpackage 2022-10-21 21:36:05 +02:00
Michael Eischer
b361284f28 Merge pull request #3979 from MichaelEischer/backup-less-time-now
backup: reduce calls to time.Now
2022-10-21 21:33:34 +02:00
Michael Eischer
738b2a0445 parallelize more List usages 2022-10-21 21:26:45 +02:00
Michael Eischer
ae45f3b04f restic: Unify code to load Index/Lock/Snapshot 2022-10-21 21:25:11 +02:00
Michael Eischer
8e2695be0b Merge pull request #3973 from MichaelEischer/speedup-integration-tests
speed-up integration tests by reducing the RetryBackend timeout
2022-10-21 21:17:35 +02:00
Michael Eischer
35d968bcde Merge pull request #3969 from MichaelEischer/key-by-id
Port restic.Find to return IDs and identify keys by restic.ID
2022-10-21 21:15:40 +02:00
Michael Eischer
4133fee6f9 Merge pull request #3972 from MichaelEischer/fix-flaky-lock-cancel-test
lock: fix flaky TestLockFailedRefresh
2022-10-21 21:12:34 +02:00
Michael Eischer
c8c8391b21 Merge pull request #3974 from greatroar/cleanup
More cleanups and a micro-optimization
2022-10-21 21:11:37 +02:00
Michael Eischer
ee7c28f5e6 backup: reduce calls to time.Now
Archiver.Save queries the current time multiple times. This commit
removes one of these calls as they showed up while profiling a backup of
a nearly unchanged dataset containing 3 million files.
2022-10-21 20:55:01 +02:00
Michael Eischer
3e60d38a23 Remove misleading wildcard from negative exclude pattern example
There is no need to use a special wildcard `**` to demonstrate negative
patterns. Actually, it is both slower than the simpler variant and seems
to confuse users.
2022-10-21 20:48:45 +02:00
greatroar
9adae5521d cache: Call interface method once 2022-10-21 14:32:46 +02:00
greatroar
201e5c7e74 backup: Clean up progress reporting code 2022-10-21 13:48:30 +02:00
plumbeo
a6f83e0011 Add changelog 2022-10-17 15:38:42 +02:00
plumbeo
bc945d0bf0 restic stats: add more compression statistics
Calculate and display compression ratio, space saving and progress
2022-10-17 15:38:38 +02:00
greatroar
b513597546 internal/restic: Make FileType a uint8 instead of a string
The string form was presumably useful before the introduction of
layouts, but right now it just makes call sequences and garbage
collection more expensive (the latter because every string contains
a pointer to be scanned).
2022-10-16 10:59:01 +02:00
greatroar
22147e1e02 all: Minor cleanups
if x { return true } return false => return x

	fmt.Sprintf("%v", x) => fmt.Sprint(x) or x.String()

The fmt.Sprintf idiom is still used in the SecretString tests, where it
serves security hardening.
2022-10-16 10:50:39 +02:00
greatroar
d03460010f internal/restic: Fix ID.UnmarshalJSON, ParseID
ID.UnmarshalJSON accepted non-JSON input with ' as the string delimiter.
Also, the error message for non-hex input was less informative than it
could be and it performed too many checks.

Changed ParseID to keep the error messages consistent.
2022-10-16 10:39:52 +02:00
Michael Eischer
aa39bf3cf6 backend/test: remove duplicate test
The test is identical to the tests for the mem backend.
2022-10-15 23:15:07 +02:00
Michael Eischer
28e1c4574b mem: use cheaper hash for backend 2022-10-15 23:14:33 +02:00
Michael Eischer
c3400d3c55 backend: speedup RetryBackend tests 2022-10-15 23:13:44 +02:00
Michael Eischer
99547518cd lock: fix flaky TestLockFailedRefresh
The comparison of the current time and the last lock refresh were using
seconds represented as integers. As the test only waits for up to one
second, the associated number truncation can cause the test to take
longer than once second and thus to fail.

Switch to nanoseconds to avoid this problem. This also slightly speeds
up the test.
2022-10-15 22:36:32 +02:00
Michael Eischer
e10420553b speed-up integration tests by reducing the RetryBackend timeout
On my machine this decreases the runtime for `./cmd/restic` from 9.5s to
6.5s.
2022-10-15 22:29:58 +02:00
Michael Eischer
367f35db27 Merge pull request #3968 from MichaelEischer/cleanup-complete-blob
backup: Remove unused filename parameter from CompleteBlob callback
2022-10-15 16:11:16 +02:00
Michael Eischer
8d62a7adb4 identify keys by ID and not name 2022-10-15 16:07:43 +02:00
Michael Eischer
02634dce7a restic: change Find to return ids
That way consumers no longer have to manually convert the returned name
to an id.
2022-10-15 16:06:54 +02:00
Michael Eischer
964977677f backup: Remove unused filename parameter from CompleteBlob callback 2022-10-15 15:21:17 +02:00
Michael Eischer
258b487d8f Merge pull request #3951 from MichaelEischer/rework-snapshot-filter
Rework snapshot filtering
2022-10-15 14:47:47 +02:00
Michael Eischer
de9bc031df add changelog for ls handling of missing snapshots 2022-10-15 13:34:50 +02:00
Michael Eischer
246d3032ae restic: Don't list snapshots if FindSnapshot gets full id 2022-10-15 13:34:34 +02:00
Michael Eischer
d8c00b9726 add comment 2022-10-15 13:34:21 +02:00
Michael Eischer
a3113c6097 restic: Change FindSnapshot functions to return the snapshot 2022-10-15 13:34:04 +02:00
Michael Eischer
b50f48594d restic: cleanup arguments of findLatestSnapshot 2022-10-15 13:33:48 +02:00
Michael Eischer
61e827ae4f restic: hide findLatestSnapshot 2022-10-15 13:33:32 +02:00
Michael Eischer
fcad5e6f5d backup: use unified FindFilteredSnapshot 2022-10-15 13:33:29 +02:00
Michael Eischer
0aa73bbd39 ls: proper error handling for non-existent snapshot
Use restic.FindFilteredSnapshot to resolve the snapshot ID. This ensures
consistent behavior for all commands using initSingleSnapshotFilterOptions.
2022-10-15 13:32:00 +02:00
Michael Eischer
a81f0432e9 restic: Add unified method to resolve a single snapshot 2022-10-15 13:31:45 +02:00
Michael Eischer
95a1bb4261 restic: Rework error handling of FindFilteredSnapshots and handle snapshotIDs
FindFilteredSnapshots no longer prints errors during snapshot loading on
stderr, but instead passes the error to the callback to allow the caller
to decide on what to do.

In addition, it moves the logic to handle an explicit snapshot list from
the main package to restic.
2022-10-15 13:31:26 +02:00
Michael Eischer
cff22a5f01 dump: use correct help text for filter options 2022-10-15 13:31:10 +02:00
Michael Eischer
7a6dcb4831 Merge pull request #3966 from MichaelEischer/cleanup-walker-test
walker: Convert tests to use TreeJSONBuilder
2022-10-15 11:25:11 +02:00
Michael Eischer
7cf042118f walker: Convert tests to use TreeJSONBuilder
The old code marshalled the tree blobs different than other places in
restic. The hashed tree blob did not contain a final newline character.
2022-10-15 11:04:13 +02:00
Michael Eischer
cea7191995 Merge pull request #3959 from MichaelEischer/buffered-backup-progress
backup: Use buffered channels to collect backup status
2022-10-15 10:57:19 +02:00
Michael Eischer
ba688aad20 Merge pull request #3961 from greatroar/cleanup
Misc. cleanup
2022-10-14 21:49:35 +02:00
Michael Eischer
9c290a8093 Merge pull request #3960 from greatroar/errors
errors: Drop WithMessage
2022-10-14 21:41:28 +02:00
greatroar
0e155fd9a6 internal/restic: Fix UID/GID parsing
The helper function uidGidInt used strconv.ParseInt instead of
ParseUint, so it silently ignored some invalid user/group IDs.

Also, improve the error message. "Invalid UID" is more informative than
having "ParseInt" twice (*strconv.NumError displays the function name).

Finally, the user.User struct can be passed by pointer to get reduce
code size.
2022-10-14 18:21:00 +02:00
greatroar
e0b743c64d internal/restic: Remove unused ID.EqualString 2022-10-14 18:20:11 +02:00
greatroar
6922360179 ui/backup: Remove unused ProgressReporter type, Progress field 2022-10-14 14:36:19 +02:00
greatroar
d4aadfa389 all: Drop ctxhttp
This package is no longer needed, since we can use the stdlib's
http.NewRequestWithContext.

backend/rclone already did, but it needed a different error check due to
a difference between net/http and ctxhttp.

Also, store the http.Client by value in the REST backend (changed to a
pointer when ctxhttp was introduced) and use errors.WithStack instead
of errors.Wrap where the message was no longer accurate. Errors from
http.NewRequestWithContext will start with "net/http" or "net/url", so
they're easy to identify.
2022-10-14 14:33:49 +02:00
greatroar
16849d5361 internal/archiver: Missing argument to errors.Errorf 2022-10-14 14:18:52 +02:00
greatroar
09c14f33c8 internal/checker: Pass Error.Error pointer receiver 2022-10-14 14:13:32 +02:00
greatroar
feb790f497 internal/restic: Use errors.New when no formatting is needed 2022-10-14 14:07:20 +02:00
greatroar
ba44666704 errors: Drop WithMessage 2022-10-14 14:06:47 +02:00
Michael Eischer
1a6160d152 Merge pull request #3880 from MichaelEischer/archiver-savedir-cleanup
archiver: Improve handling of "file xxx already present" error
2022-10-08 21:48:14 +02:00
Michael Eischer
21b1d7a880 Merge pull request #3948 from MichaelEischer/split-index
repository: split index into a separate package
2022-10-08 21:41:57 +02:00
Michael Eischer
5278ab51c8 archiver: Check that duplicates are only ignored if identical 2022-10-08 21:38:36 +02:00
Michael Eischer
403b01b788 backup: Only return a warning for duplicate directory entries
The backup command failed if a directory contains duplicate entries.
Downgrade the severity of this problem from fatal error to a warning.
This allows users to still create a backup.
2022-10-08 21:38:21 +02:00
Michael Eischer
d7d7b4ab27 archiver: refactor TreeSaverTest 2022-10-08 21:29:32 +02:00
Michael Eischer
8e38c43c27 archiver: let FutureNode.Take return an error if no data is available
This ensures that we cannot accidentally store an invalid node.
2022-10-08 21:28:39 +02:00
Michael Eischer
2b88cd6eab archiver: Restructure SaveTree to work like SaveDir
SaveTree did not use the TreeSaver but rather managed the tree
collection and upload itself. This prevents using the parallelism
offered by the TreeSaver and duplicates all related code. Using the
TreeSaver can provide some speed-ups as all steps within the backup tree
now rely on FutureNodes. This can be especially relevant for backups
with large amounts of explicitly specified files.

The main difference between SaveTree and SaveDir is, that only the
former can save tree blobs in which nodes have a different name than the
actual file on disk. This is the result of resolving name conflicts
between multiple files with the same name. The filename that must be
used within the snapshot is now passed directly to
restic.NodeFromFileInfo. This ensures that a FutureNode already contains
the correct filename.
2022-10-08 21:28:39 +02:00
Michael Eischer
2e3f1c08c5 repository: split index into a separate package 2022-10-08 21:15:34 +02:00
Michael Eischer
5760ba6989 Merge pull request #3949 from MichaelEischer/simplify-mixedpacks
repository: remove IsMixedPack and add replacement for checker
2022-10-08 21:14:14 +02:00
Michael Eischer
5ee25e669a Merge pull request #3940 from MichaelEischer/better-rclone-error
Better error message if connection to rclone fails
2022-10-08 21:14:00 +02:00
Michael Eischer
5600f11696 rclone: Fix stderr handling if command exits unexpectedly
According to the documentation of exec.Cmd Wait() must not be called
before completing all reads from the pipe returned by StdErrPipe(). Thus
return a context that is canceled once rclone has exited and use that as
a precondition to calling Wait(). This should ensure that all errors
printed to stderr have been copied first.
2022-10-08 20:16:06 +02:00
Michael Eischer
b8acad4da0 rclone: return rclone error instead of canceled context
When rclone fails during the connection setup this currently often
results in a context canceled error. Replace this error with the exit
code from rclone.
2022-10-08 20:15:24 +02:00
Michael Eischer
d3ebec8f21 backup: Use buffered channels to collect backup status
When backing up many small files, the unbuffered channels frequently
cause the FileSaver to block when reporting progress information. Thus,
add buffers to these channels to avoid unnecessary scheduling.

As the status information is purely informational, it doesn't matter
that the status reporting shutdown is somewhat racy and could miss a few
final updates.
2022-10-08 18:20:41 +02:00
Michael Eischer
f9d4e0c2af Merge pull request #3958 from greatroar/errors
errors: Drop Cause in favor of Go 1.13 error handling
2022-10-08 18:06:35 +02:00
Michael Eischer
119e6aee01 Merge pull request #3957 from greatroar/typo
cmd: Typo in --read-concurrency description
2022-10-08 14:41:35 +02:00
greatroar
07e5c38361 errors: Drop Cause in favor of Go 1.13 error handling
The only use cases in the code were in errors.IsFatal, backend/b2,
which needs a workaround, and backend.ParseLayout. The last of these
requires all backends to implement error unwrapping in IsNotExist.
All backends except gs already did that.
2022-10-08 13:08:08 +02:00
greatroar
4eae4d3e1a cmd: Typo in --read-concurrency description 2022-10-08 11:27:39 +02:00
Michael Eischer
83cb58b4f3 Merge pull request #3956 from MichaelEischer/fix-lock-refresh
lock: Use the correct duration to check for expired locks
2022-10-07 22:58:10 +02:00
Michael Eischer
7c5d63a794 lock: Use the correct duration to check for expired locks 2022-10-07 22:39:53 +02:00
Michael Eischer
8b7c952f17 Merge pull request #3953 from keachi/typo
Fix typo
2022-10-07 22:18:32 +02:00
Michael Eischer
e43d2d45f7 Merge pull request #3952 from hoelzro/master
Update copy documentation to use --from-repo option
2022-10-07 22:18:06 +02:00
Rob Hoelz
03e9a26018 Update copy documentation to use --from-repo option
Removing the last references to the deprecated --repo2 option
2022-10-07 22:00:12 +02:00
tr
43cc01d63e doc: Fix typo 2022-10-05 21:03:14 +02:00
Michael Eischer
7112a132c3 Merge pull request #3950 from MichaelEischer/misc-cleanups
Cleanups for cmd_debug/repository and remove dead code from restic package
2022-10-03 12:46:32 +02:00
Michael Eischer
4bb5240720 repository: remove unused PrefixLength 2022-10-03 12:15:53 +02:00
Michael Eischer
999fe29976 repository: hide prepareCache 2022-10-03 12:15:53 +02:00
Michael Eischer
9197c63007 debug: use repository.ListPack wrapper 2022-10-03 12:09:08 +02:00
Michael Eischer
ddcf549eba repository: remove IsMixedPack and add replacement for checker
Repositories with mixed packs are probably quite rare by now. When
loading data blobs from a mixed pack file, this will no longer trigger
caching that file. However, usually tree blobs are accessed first such
that this shouldn't make much of a difference.

The checker gets a simpler replacement.
2022-10-03 12:03:59 +02:00
Michael Eischer
a61fbd287a Merge pull request #3569 from MichaelEischer/strict-locking
Strict repository lock handling
2022-10-03 00:44:44 +02:00
Michael Eischer
6d2d297215 pass global context through cobra 2022-10-03 00:19:46 +02:00
Michael Eischer
49126796d0 lock: fix timer expiry monitoring during standby
Monotonic timers are paused during standby. Thus these timers won't fire
after waking up. Fall back to periodic polling to detect too large clock
jumps. See https://github.com/golang/go/issues/35012 for a discussion of
go timers during standby.
2022-10-03 00:19:46 +02:00
Michael Eischer
401e432e9d lock: Do not ignore invalid lock files
While searching for lock file from concurrently running restic
instances, restic ignored unreadable lock files. These can either be
in fact invalid or just be temporarily unreadable. As it is not really
possible to differentiate between both cases, just err on the side of
caution and consider the repository as already locked.

The code retries searching for other locks up to three times to smooth
out temporarily unreadable lock files.
2022-10-03 00:19:46 +02:00
Michael Eischer
aeed420e1a add changelog 2022-10-03 00:19:46 +02:00
Michael Eischer
9959190e39 lock: Add integration test
The tests check that the wrapped context is properly canceled whenever
the repository is unlock or when the lock refresh fails.
2022-10-03 00:19:46 +02:00
Michael Eischer
c3538b063a lock: Use repository interface instead of struct 2022-10-03 00:19:46 +02:00
Michael Eischer
d92957dd78 lock: Implement strict lock expiry monitoring
Restic continued e.g. a backup task even when it failed to renew the
lock or failed to do so in time. For example if a backup client enters
standby during the backup this can allow other operations like `prune`
to run in the meantime (after calling `unlock`). After leaving standby
the backup client will continue its backup and upload indexes which
refer pack files that were removed in the meantime.

This commit introduces a goroutine explicitly monitoring for locks that
are not refreshed in time. To simplify the implementation there's now a
separate goroutine to refresh the lock and monitor for timeouts for each
lock. The monitoring goroutine would now cause the backup to fail as the
client has lost it's lock in the meantime.

The lock refresh goroutines are bound to the context used to lock the
repository initially. The context returned by `lockRepo` is also
cancelled when any of the goroutines exits. This ensures that the
context is cancelled whenever for any reason the lock is no longer
refreshed.
2022-10-03 00:19:46 +02:00
Michael Eischer
928914f821 Prepare for context bound to lock lifetime 2022-10-03 00:19:46 +02:00
Michael Eischer
985722b102 Remove ctx from globalOptions
Previously the global context was either accessed via gopts.ctx,
stored in a local variable and then used within that function or
sometimes both. This makes it very hard to follow which ctx or a wrapped
version of it reaches which method.

Thus just drop the context from the globalOptions struct and pass it
explicitly to every command line handler method.
2022-10-03 00:19:46 +02:00
Michael Eischer
ab819b2344 key: Cleanup method signatures 2022-10-03 00:19:46 +02:00
Michael Eischer
d0668b695d Remove unnecessary context.WithCancel calls
The gopts.ctx is cancelled when the main() method of restic exits.
2022-10-03 00:19:46 +02:00
Michael Eischer
7ce4cb7908 Merge pull request #3947 from MichaelEischer/fix-cache-verify-test
cache: Fix file descriptor leak in TestBackendRemoveBroken
2022-10-03 00:19:26 +02:00
Michael Eischer
430ab32941 cache: Fix file descriptor leak in TestBackendRemoveBroken 2022-10-03 00:06:44 +02:00
Michael Eischer
e99ad39b34 Merge pull request #2750 from metalsp0rk/min-packsize
Add `backup --file-read-concurrency` flag
2022-10-02 23:11:47 +02:00
Michael Eischer
2e606ca70b backup: rework read concurrency 2022-10-02 22:55:14 +02:00
Kyle Brennan
4a501d7118 backup: add option for file read concurrency 2022-10-02 22:51:45 +02:00
Michael Eischer
9ec7eee803 Merge pull request #3521 from MichaelEischer/redownload-broken-files
Redownload files with wrong hash
2022-10-02 22:50:03 +02:00
Michael Eischer
b25d0773b6 Merge pull request #3944 from MichaelEischer/fix-linter-errors
CI: ignore warning about missing package comment
2022-09-27 21:41:55 +02:00
Michael Eischer
5265550ff3 CI: ignore warning about missing package comment 2022-09-27 21:31:37 +02:00
Michael Eischer
e89fc2a29d Merge pull request #3943 from MichaelEischer/find-match-only-valid-ids
ignore filenames which are not IDs when expanding a prefix
2022-09-27 20:56:48 +02:00
Michael Eischer
67e4620cd6 Merge pull request #3938 from restic/errdot
rclone/sftp: Improve handling of ErrDot errors
2022-09-27 20:33:42 +02:00
Michael Eischer
5d3c5b9e50 restic: ignore filenames which are not IDs when expanding a prefix
Some backends generate additional files for each existing file, e.g.

1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef
1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef.sha256

For some commands this leads to an "multiple IDs with prefix" error when
trying to reference a snapshot.
2022-09-27 20:30:40 +02:00
Leo R. Lundgren
ebe9f2c969 rclone/sftp: Improve handling of ErrDot errors
Restic now yields a more informative error message when exec.ErrDot occurs.
2022-09-25 16:19:03 +02:00
Michael Eischer
d114e483c4 Add changelog for corrupt data downloads 2022-09-25 11:55:09 +02:00
Michael Eischer
34c1a83340 cache: Drop cache entry if it cannot be processed
Failing to process data requested from the cache usually indicates a
problem with the returned data. Assume that the cache entry is somehow
damaged and retry downloading it once.
2022-09-25 11:55:09 +02:00
Michael Eischer
aa3b1925b4 cache: Simplify loadFromCacheOrDelegate 2022-09-25 11:35:35 +02:00
Michael Eischer
5c6b6edefe retry index, lock and snapshot loading on hash mismatch 2022-09-25 11:35:35 +02:00
Michael Eischer
822422ef03 retry key loading on hash mismatch 2022-09-25 11:35:35 +02:00
Michael Eischer
d6575f53ca Merge pull request #3942 from MichaelEischer/split-cross-compile-test
Split cross compile test
2022-09-24 22:27:08 +02:00
Michael Eischer
78d2312ee9 Merge pull request #3854 from MichaelEischer/sparsefiles
restore: Add support for sparse files
2022-09-24 22:04:02 +02:00
Michael Eischer
46b30b9826 split cross compilation into three parts
The cross compilation tasks are currently the slowest part of the CI
runs. Splitting it into three parts should reduce its time to roughly
that of the windows CI run.
2022-09-24 22:00:25 +02:00
Michael Eischer
bd191ec60b update golang-ci to version 1.49 2022-09-24 22:00:08 +02:00
Michael Eischer
519059cca4 update ci actions 2022-09-24 21:59:36 +02:00
Michael Eischer
19afad8a09 restore: support sparse restores also on windows 2022-09-24 21:39:39 +02:00
Michael Eischer
0f89f443c7 update sparse restore changelog 2022-09-24 21:39:39 +02:00
Michael Eischer
c147422ba5 repository: special case SaveBlob for all zero chunks
Sparse files contain large regions containing only zero bytes. Checking
that a blob only contains zeros is possible with over 100GB/s for modern
x86 CPUs. Calculating sha256 hashes is only possible with 500MB/s (or
2GB/s using hardware acceleration). Thus we can speed up the hash
calculation for all zero blobs (which always have length
chunker.MinSize) by checking for zero bytes and then using the
precomputed hash.

The all zeros check is only performed for blobs with the minimal chunk
size, and thus should add no overhead most of the time. For chunks which
are not all zero but have the minimal chunks size, the overhead will be
below 2% based on the above performance numbers.

This allows reading sparse sections of files as fast as the kernel can
return data to us. On my system using BTRFS this resulted in about
4GB/s.
2022-09-24 21:39:39 +02:00
Michael Eischer
34fe1362da restorer: move zeroPrefixLen to restic package 2022-09-24 21:39:39 +02:00
Michael Eischer
a5ebd5de4b restorer: Fix race condition in partialFile.WriteAt
The restorer can issue multiple calls to WriteAt in parallel. This can
result in unexpected orderings of the Truncate and WriteAt calls and
sometimes too short restored files.
2022-09-24 21:39:39 +02:00
Michael Eischer
5b6a77058a Enable sparseness only conditionally
We can either preallocate storage for a file or sparsify it. This
detects a pack file as sparse if it contains an all zero block or
consists of only one block. As the file sparsification is just an
approximation, hide it behind a `--sparse` parameter.
2022-09-24 21:20:00 +02:00
greatroar
3047bf611c Changelog entry for sparse file restoring 2022-09-24 21:18:48 +02:00
greatroar
5d4568d393 Write sparse files in restorer
This writes files by using (*os.File).Truncate, which resolves to the
truncate system call on Unix.

Compared to the naive loop,

	for _, b := range p {
		if b != 0 {
			return false
		}
	}

the optimized allZero is about 10× faster:

name       old time/op    new time/op     delta
AllZero-8    1.09ms ± 1%     0.09ms ± 1%    -92.10%  (p=0.000 n=10+10)

name       old speed      new speed       delta
AllZero-8  3.84GB/s ± 1%  48.59GB/s ± 1%  +1166.51%  (p=0.000 n=10+10)
2022-09-24 21:18:48 +02:00
Michael Eischer
eb83402d39 Merge pull request #3935 from miles170/master
Only display the message if there were locks to be removed
2022-09-24 20:53:13 +02:00
Michael Eischer
ef58ddd7b1 Merge pull request #3923 from MichaelEischer/fix-flaky-cache-test
cache: fix flaky TestFileSaveConcurrent on windows
2022-09-24 20:52:55 +02:00
Michael Eischer
7fc178aaf4 internal/cache: extend description of cache sharing test failure 2022-09-24 13:07:01 +02:00
Miles Liu
1acbda18f8 Only display the message if there were locks to be removed
`restic unlock` now only shows `successfully removed locks` if there were locks to be removed.
In addition, it also reports the number of the removed lock files.
2022-09-24 19:02:24 +08:00
Michael Eischer
da1a359c8b Merge pull request #3927 from MichaelEischer/faster-index-each
Speed up MasterIndex.Each
2022-09-24 12:35:23 +02:00
Michael Eischer
041a51512a Merge pull request #3780 from jkmw/fix/2578
Remove existing path before restoring a symlink
2022-09-24 12:34:42 +02:00
Michael Eischer
1ebd57247a repository: optimize MasterIndex.Each
Sending data through a channel at very high frequency is extremely
inefficient. Thus use simple callbacks instead of channels.

> name                old time/op  new time/op  delta
> MasterIndexEach-16   6.68s ±24%   0.96s ± 2%  -85.64%  (p=0.008 n=5+5)
2022-09-24 12:21:59 +02:00
Michael Eischer
825b95e313 repository: add benchmark for MasterIndex.Each 2022-09-24 12:21:59 +02:00
greatroar
1220fe9650 internal/cache: Concurrent use of cache not working on Windows 2022-09-17 19:49:44 +02:00
Jerome Küttner
ef618bdd3f use os.Remove if path already exists on symlink restore 2022-09-14 08:14:31 +02:00
Michael Eischer
b48766d7b8 Merge pull request #3928 from restic/rawtaz-doc-b2-s3
doc: Clarify S3 recommendation for B2 slightly
2022-09-13 20:59:50 +02:00
rawtaz
20f1913ef7 doc: Clarify S3 recommendation for B2 slightly
This gives slightly more background to the recommendation for using restic's S3 backend with Backblaze B2.
2022-09-12 17:48:59 +02:00
rawtaz
d79e61ce5d Merge pull request #3925 from hgraeber/add-powershell-completion
Add powershell completion
2022-09-11 01:04:57 +02:00
Herbert Graeber
988b386e8b Add powershell completion
- Add code for powersehll complition available in cobra
- Add documentation for powershell completion
- Add changelog for pr3925
2022-09-11 00:44:12 +02:00
rawtaz
14d09a6081 Merge pull request #3912 from MichaelEischer/cleanup-snapshot-filter-options
Clean up snapshot filter options
2022-09-11 00:18:42 +02:00
Michael Eischer
381da0443a tweak snapshot filter descriptions 2022-09-10 23:50:20 +02:00
Michael Eischer
8b9778d537 Merge pull request #3900 from MichaelEischer/b2-init-timeout
Add timeout for the initial connection to B2
2022-09-10 23:28:59 +02:00
Michael Eischer
17c27400f8 Merge pull request #3921 from MichaelEischer/filter-cleanup-error-handling
filter: deduplicate error handling for pattern validation
2022-09-10 23:24:50 +02:00
Michael Eischer
f76643bd2e Merge pull request #3894 from MichaelEischer/filter-mount-exit-code
Mount should return exit code 0 after pressing Ctrl-C
2022-09-10 23:22:01 +02:00
Michael Eischer
be9ccc186e Merge pull request #3875 from MichaelEischer/fix-fuse-context-cancel
mount: Fix input/output errors for canceled syscalls
2022-09-10 23:20:29 +02:00
Michael Eischer
2363e5c083 Merge pull request #3913 from MichaelEischer/better-migrate-error-message
migrate: Report why an migration cannot be applied
2022-09-09 23:37:25 +02:00
Michael Eischer
8e0ca80547 filter: deduplicate error handling for pattern validation 2022-09-09 23:12:41 +02:00
plumbeo
d66e755ac7 Change uncompressed size calculation to account for the encryption overhead 2022-09-08 10:15:19 +02:00
plumbeo
837b816358 restic stats: print uncompressed size in mode raw-data 2022-09-05 17:38:32 +02:00
Michael Eischer
d6309961c5 deduplicate the snapshot filter cli option setup 2022-09-04 10:27:33 +02:00
Michael Eischer
8b4dd70013 migrate: Report why an migration cannot be applied
Just returning that `Migration upgrade cannot be applied: check failed`
is not too useful when running `migrate upgrade_repo_v2`.
2022-09-03 11:49:31 +02:00
Michael Eischer
7689d6c679 normalize help text for host, tag and path options 2022-09-03 00:06:38 +02:00
Michael Eischer
6c69f08a7b Merge pull request #3905 from DRON-666/haspaths-linear
Reduce quadratic time complexity of `Snapshot.HasPaths`
2022-08-30 20:35:56 +02:00
Michael Eischer
3e70bac56e Merge pull request #3898 from MichaelEischer/fix-copy-hang
don't hang when `copy` uses a single connection
2022-08-30 20:23:39 +02:00
DRON-666
2a630c51c1 Add changelog 2022-08-30 20:22:07 +02:00
DRON-666
d0f1060df7 Fix quadratic time complexity of Snapshot.HasPaths 2022-08-30 04:38:17 +03:00
Michael Eischer
f481ad64c8 Merge pull request #3904 from lbausch/add-newline
Add newline to keep prompt intact
2022-08-29 21:43:18 +02:00
Lorenz Bausch
7ddd803e46 Add newline to keep prompt intact 2022-08-29 17:37:49 +02:00
Michael Eischer
e5b2c4d571 b2: sniff the error that caused init retry loops 2022-08-28 17:46:03 +02:00
Michael Eischer
dc2db2de5e b2: cancel connection setup after a minute
If the connection to B2 fails, the library enters an endless loop.
2022-08-28 14:56:17 +02:00
Michael Eischer
7682149c9d repository: cleanup copy connection count check 2022-08-28 11:40:56 +02:00
Michael Eischer
b03277ead5 repository: don't hang when copying using a single connection 2022-08-28 11:40:31 +02:00
Michael Eischer
1b233c4e2e Merge pull request #2661 from creativeprojects/issue-1734
"Everything is ok" message after retrying
2022-08-28 11:04:59 +02:00
Fred
4042db5169 Add changelog 2022-08-27 22:36:19 +02:00
Fred
be6baaec12 Add success callback to the backend 2022-08-27 22:27:15 +02:00
Fred
baf58fbaa8 Add unit tests 2022-08-27 22:21:06 +02:00
Fred
d629333efe Add function to notify of success after retrying 2022-08-27 22:21:06 +02:00
Alexander Neumann
c169e37139 Merge pull request #3895 from MichaelEischer/refactor-cat-key
cat: Simplify implementation of 'cat key'
2022-08-27 18:40:46 +02:00
Michael Eischer
1b4af0c6e5 cat: Simplify implementation of 'cat key' 2022-08-26 23:21:51 +02:00
Michael Eischer
3174641ca4 add changelog for mount exit code filtering 2022-08-26 23:17:04 +02:00
Michael Eischer
5478ab22c5 mount: return exit code 0 after receiving a SIGINT 2022-08-26 23:07:07 +02:00
Michael Eischer
d768c1c3e4 Allow cleanup handlers to filter the exit code 2022-08-26 23:04:59 +02:00
Michael Eischer
908f7441fe Merge pull request #3885 from MichaelEischer/delete-fixes
Improve reliability of upload retries and B2 file deletions
2022-08-26 22:30:50 +02:00
Michael Eischer
4c90d91d4d backend: Test that failed uploads are not removed for backends with atomic replace 2022-08-26 21:20:52 +02:00
Michael Eischer
694dfa026a add changelog for reliable B2 deletes 2022-08-26 21:20:46 +02:00
MichaelEischer
582167d671 Merge pull request #3882 from MichaelEischer/sftp-init-single-connection
sftp: Only connect once to server during `init`
2022-08-26 21:13:28 +02:00
MichaelEischer
3822ded0b3 Merge pull request #3877 from MichaelEischer/no-env-in-help
Do not include the actual values of environment variables in help output
2022-08-26 20:59:54 +02:00
Michael Eischer
cf0a8d7758 sftp: Only connect once for repository creation
This is especially useful if ssh asks for a password or if closing the
initial connection could return an error due to a problematic server
implementation.
2022-08-26 20:50:40 +02:00
Michael Eischer
dd7cd5b9b3 fuse: remove unused context parameter 2022-08-26 20:48:48 +02:00
Michael Eischer
a0c1ae9f90 mount: Correctly return context.Canceled for interrupted syscalls
bazil/fuse expects us to return context.Canceled to signal that a
syscall was successfully interrupted. Returning a wrapped version of
that error however causes the fuse library to signal an EIO (input/output
error). Thus unwrap context.Canceled errors before returning them.
2022-08-26 20:48:48 +02:00
Michael Eischer
5d0649faaf Update help output in docs 2022-08-26 20:44:01 +02:00
Michael Eischer
faa4597af1 Set name for option values of cli 2022-08-26 20:42:34 +02:00
Michael Eischer
6ed157aee6 Do not include the actual values of environment variables in help output
This results in printing a `(default: $ENV) (default: value)` suffix for
the corresponding options which looks strange. In addition, some of the
environment variables might contain secrets which should not be
displayed.
2022-08-26 20:39:54 +02:00
MichaelEischer
f7808245aa Merge pull request #3878 from MichaelEischer/cheaper-cache-load
cache: Just try to open cache entry without calling stat first
2022-08-26 20:33:36 +02:00
MichaelEischer
bee15dd555 Merge pull request #3879 from MichaelEischer/mem-optimize
Some random (minor) memory-allocation optimizations
2022-08-26 20:33:02 +02:00
MichaelEischer
0e1d082b12 Merge pull request #3886 from MichaelEischer/recommend-s3-over-b2
doc: recommend usage of B2's S3 API
2022-08-26 20:29:05 +02:00
Alexander Neumann
d464543171 Update repo version table 2022-08-25 21:30:25 +02:00
Alexander Neumann
6b40456db7 Set development version for 0.14.0 2022-08-25 19:55:05 +02:00
Alexander Neumann
1bc87e1718 Add version for 0.14.0 2022-08-25 19:55:00 +02:00
Alexander Neumann
c1a5da56e3 Update manpages and auto-completion 2022-08-25 19:55:00 +02:00
Alexander Neumann
193c62dfc3 Generate CHANGELOG.md for 0.14.0 2022-08-25 19:54:02 +02:00
Alexander Neumann
a825e0d409 Prepare changelog for 0.14.0 2022-08-25 19:54:01 +02:00
rawtaz
b824d8cdcc Merge pull request #3891 from MichaelEischer/fix-secondary-repo-typo
Fix typo in the environment variable name for `--from-password-file`
2022-08-24 22:38:47 +02:00
Michael Eischer
a2e89234fc Fix typo in the environment variable name for --from-password-file 2022-08-24 22:25:18 +02:00
MichaelEischer
b4ae05627f Merge pull request #3876 from MichaelEischer/document-aws-session-token
doc: document aws session token
2022-08-24 22:20:26 +02:00
MichaelEischer
bd7bca2b51 Merge pull request #3889 from restic/prepare-0-14
Polish changelog entries
2022-08-24 21:28:44 +02:00
MichaelEischer
be90a565cc Merge pull request #3887 from MichaelEischer/rclone-permanent-error
rclone: Return a permanent error if rclone already exited
2022-08-24 21:19:00 +02:00
Michael Eischer
7a5d29ce24 Further changelog polishing 2022-08-24 21:13:14 +02:00
MichaelEischer
d198a77d86 Merge pull request #3888 from restic/doc-prepare-repo
doc: Improve/clarify preparing and versions of repositories
2022-08-24 20:40:49 +02:00
Leo R. Lundgren
cb5a61c46e doc: Improve/clarify preparing and versions of repositories 2022-08-24 02:08:46 +02:00
Leo R. Lundgren
ee6b9dc492 Polish changelog entries 2022-08-24 01:23:51 +02:00
Michael Eischer
506d92e87c rclone: Return a permanent error if rclone already exited
rclone can exit early for example when the connection to rclone is
relayed for example via ssh: `-o rclone.program='ssh user@example.org
forced-command'`
2022-08-23 22:05:04 +02:00
MichaelEischer
ad6eabbfa5 Merge pull request #3883 from MichaelEischer/update-dependencies
Update dependencies
2022-08-23 21:44:38 +02:00
Michael Eischer
7681a63fdb restic: Cleanup xattr error handling for Solaris
Since xattr 0.4.8 (https://github.com/pkg/xattr/pull/68) returns ENOTSUP
similar to Linux.
2022-08-23 21:25:15 +02:00
Michael Eischer
99e4ccbd94 remain compatible with go 1.15 2022-08-23 21:25:14 +02:00
Michael Eischer
22f46c18f9 downgrade bazil/fuse again to retain macOS support 2022-08-23 21:24:47 +02:00
Michael Eischer
6db979b3a6 update dependencies 2022-08-23 21:24:46 +02:00
MichaelEischer
9cdc8da10f Merge pull request #3881 from MichaelEischer/cleanup-release-helper
Cleanup release helper
2022-08-23 21:10:57 +02:00
MichaelEischer
b51e73e78f Merge pull request #3884 from MichaelEischer/fix-index-saving-progress
repository: Do not report ignored packs in EachByPack
2022-08-23 21:10:07 +02:00
MichaelEischer
98dcd0a887 Merge pull request #3874 from MichaelEischer/disk-wear-note
Add note that larger packs increase disk wear
2022-08-21 22:18:08 +02:00
Michael Eischer
e0d6bf525c doc: fix typo 2022-08-21 19:12:19 +02:00
Michael Eischer
5d0523e2f1 Add note that larger packs increase disk wear 2022-08-21 19:12:19 +02:00
Michael Eischer
c586a5e20f doc: recommend usage of B2's S3 API 2022-08-21 11:39:03 +02:00
Michael Eischer
623556bab6 b2: Increase list size to maximum
Just request as many files as possible in one call to reduce the number
of network roundtrips.
2022-08-21 11:20:03 +02:00
Michael Eischer
de0162ea76 backend/retry: Overwrite failed uploads instead of deleting them
For backends which are able to atomically replace files, we just can
overwrite the old copy, if it is necessary to retry an upload. This has
the benefit of issuing one operation less and might be beneficial if a
backend storage, due to bugs or similar, could mix up the order of the
upload and delete calls.
2022-08-21 11:14:53 +02:00
Michael Eischer
fc506f8538 b2: Repeat deleting until all file versions are removed
When hard deleting the latest file version on B2, this uncovers earlier
versions. If an upload required retries, multiple version might exist
for a file. Thus to reliably delete a file, we have to remove all
versions of it.
2022-08-21 11:11:00 +02:00
Michael Eischer
cc4728d287 repository: Do not report ignored packs in EachByPack
Ignored packs were reported as an empty pack by EachByPack. The most
immediate effect of this is that the progress bar for rebuilding the
index reports processing more packs than actually exist.
2022-08-21 10:38:40 +02:00
Michael Eischer
c3374b3ea5 helper: download modules as first step
There's no use in running that step in parallel.
2022-08-20 12:11:54 +02:00
Michael Eischer
7f0929e519 helper: Reduce number of parallel builds a bit
The go compiler is already parallelized. The high concurrency caused my
podman container to hit a resource limit.
2022-08-20 12:10:48 +02:00
Michael Eischer
ed94678820 helper: cleanups 2022-08-20 12:10:29 +02:00
Michael Eischer
e530d422a0 helper: don't setup cmd paths twice 2022-08-20 12:09:42 +02:00
Michael Eischer
7a992fc794 repository: Reduce buffer reallocations in ForAllIndexes
Previously the buffer was grown incrementally inside `repo.LoadUnpacked`.
But we can do better as we already know how large the index will be.
Allocate a bit more memory to increase the chance that the buffer can be
reused in the future.
2022-08-19 21:13:40 +02:00
Michael Eischer
77b1980d8e repository: MasterIndex.Packs: reduce allocations 2022-08-19 21:10:43 +02:00
Michael Eischer
6ff9517e45 repository: MasterIndex.ListPacks / Index.EachByPack allow earlier GC
Allow earlier garbage collection of some of the intermediate data
structures.
2022-08-19 21:06:33 +02:00
Michael Eischer
ce902aac67 cache: Just try to open cache entry without calling stat first
Instead of first checking whether a file is in the repository cache and
then opening it, we just can open the file. This saves one stat call. If
the file is in the cache, everything is fine and otherwise the code
follows its normal fallback path.
2022-08-19 20:59:06 +02:00
Michael Eischer
b6c86ababe doc: document aws session token 2022-08-19 20:41:15 +02:00
MichaelEischer
0d9ac78437 Merge pull request #3873 from MichaelEischer/gofmt-comments
gofmt comments
2022-08-19 19:54:30 +02:00
MichaelEischer
7e96a5af62 Merge pull request #3872 from MichaelEischer/fuse-fix
mount: Only remember successful snapshot refreshes
2022-08-19 19:21:29 +02:00
Michael Eischer
f414db987d gofmt all files
Apparently the rules for comment formatting have changed with go 1.19.
2022-08-19 19:12:26 +02:00
Michael Eischer
522406b4f0 mount: Only remember successful snapshot refreshes
If the context provided by the fuse library is canceled before the index
was loaded this could lead to missing snapshots.
2022-08-19 19:07:07 +02:00
MichaelEischer
dbca93da28 Merge pull request #3742 from MichaelEischer/from-repo
copy: replace --repo2 with --from-repo
2022-08-19 19:01:04 +02:00
MichaelEischer
b4dfab002a Merge pull request #3691 from greatroar/tag-filenames
Sanitize tags when used as filenames by restic mount
2022-08-19 18:33:44 +02:00
Michael Eischer
2758d76b77 copy: replace --repo2 with --from-repo
`init` and `copy` use `--repo2` with two different meaning which has
proven to be confusing for users. `--from-repo` now consistently marks a
source repository from which data is read. `--repo` is now always the
target/destination repository.
2022-08-19 18:33:26 +02:00
Michael Eischer
af50fe9ac0 mount: Map slashes in tags to underscores
Suggested-by: greatroar <>
2022-08-19 18:17:57 +02:00
MichaelEischer
4cccffab58 Merge pull request #3862 from restic/3861-forget-invalid-policy
forget: Error when invalid unit is given in duration policy
2022-08-18 20:45:26 +02:00
Michael Eischer
2ea6c82cf6 comment cleanup
gofmt reformatted the comment
2022-08-18 20:15:38 +02:00
Michael Eischer
bb27f7408c forget: Fail test if duration parsing error is missing 2022-08-18 20:14:09 +02:00
MichaelEischer
c4b3a154ba Merge pull request #3865 from restic/rawtaz-doc-comp-typo
doc: Fix typo in compression section
2022-08-18 19:37:44 +02:00
rawtaz
2cb2aa31cd doc: Fix typo in compression section 2022-08-12 22:17:38 +02:00
Leo R. Lundgren
6f517858e8 forget: Error when invalid unit is given in duration policy 2022-08-10 13:37:26 +02:00
rawtaz
f0bb4f8708 Merge pull request #3857 from restic/rawtaz-gcs-account
doc: Update links to GCS documentation
2022-08-08 22:52:30 +02:00
Michael Eischer
40c8755b13 doc: Update more links to GCS documentation 2022-08-08 20:00:44 +02:00
rawtaz
f673068dbb doc: Update link to GCS documentation
Updates the link to Google Cloud Storage documentation about creating a service account key.
2022-08-08 12:51:43 +02:00
rawtaz
f26231c9e6 Merge pull request #3852 from MichaelEischer/snapshots-processed
stats: Add snapshots count to json output
2022-08-08 00:05:15 +02:00
MichaelEischer
80e3efffef Merge pull request #3855 from mattxtaz/master
Fix typo with double percentage in help text
2022-08-07 22:44:36 +02:00
mattxtaz
01ab36336f Fix typo with double percentage in help text 2022-08-07 20:21:05 +01:00
Michael Eischer
6e92d852a8 stats: Add snapshots count to json output 2022-08-07 15:44:09 +02:00
MichaelEischer
9ad3ad5972 Merge pull request #3850 from lbausch/go1.19
Update tests to Go 1.19
2022-08-07 14:56:17 +02:00
MichaelEischer
2930a102de Merge pull request #3731 from metalsp0rk/feature/min-packsize-flag
Feature: min packsize flag
2022-08-07 14:54:45 +02:00
MichaelEischer
8fa64a8f99 Merge pull request #3036 from greatroar/refactor-fuse
Clean up internal/fuse
2022-08-07 14:46:30 +02:00
Michael Eischer
f3fdc66b32 restic: Use stable sorting in snapshot policy
sort.Sort is not guaranteed to be stable. Go 1.19 has changed the
sorting algorithm which resulted in changes of the sort order. When
comparing snapshots with identical timestamp but different paths and
tags lists, there is not meaningful order among them. So just keep their
order stable.
2022-08-07 14:10:40 +02:00
Lorenz Bausch
b82f4824f0 Bump golangci-lint version 2022-08-07 14:10:39 +02:00
Lorenz Bausch
0b9b4c52ad Update tests to Go 1.19 2022-08-07 14:10:39 +02:00
Michael Eischer
0b7291b8b2 mount: Fix parent inode used by snapshots dir 2022-08-07 13:03:32 +02:00
greatroar
cfa80e2c6b mount: remove unused inode field from root node 2022-08-07 13:03:26 +02:00
MichaelEischer
74ae76036f Merge pull request #2913 from aawsome/mount-snapshot-slashes
mount: Make snapshots dir structure customizable
2022-08-07 12:27:59 +02:00
MichaelEischer
09497aec02 Merge pull request #3826 from MichaelEischer/debug-log-for-release
Debug log for release build
2022-08-07 12:15:27 +02:00
Michael Eischer
83b4c50ee3 Mention --snapshot-template and --time-template in changelog 2022-08-07 12:13:06 +02:00
Michael Eischer
caa17988a3 fuse: Redesign snapshot dirstruct
Cleanly separate the directory presentation and the snapshot directory
structure. SnapshotsDir now translates the dirStruct into a format
usable by the fuse library and contains only minimal special case rules.
All decisions have moved into SnapshotsDirStructure which now creates a
fully preassembled tree data structure.
2022-08-07 12:13:06 +02:00
Michael Eischer
a3e48da3a3 Add changelog for DEBUG_LOG available in release builds 2022-08-05 23:49:39 +02:00
Michael Eischer
b3cdee66a9 update documentation to reflect DEBUG_LOG for release builds 2022-08-05 23:49:39 +02:00
Michael Eischer
1ed775e3a8 debug: support roundtripper logging also for release builds
Different from debug builds do not use the eofDetectRoundTripper if
logging is disabled.
2022-08-05 23:49:39 +02:00
Michael Eischer
38becfc436 debug: enable debug support for release builds 2022-08-05 23:49:39 +02:00
Michael Eischer
82c268c917 Remove unused hooks mechanism 2022-08-05 23:49:39 +02:00
Michael Eischer
7266f07c87 repository: StreamPack in parts if there are too large gaps
For large pack sizes we might be only interested in the first and last
blob of a pack file. Thus stream a pack file in multiple parts if the
gaps between requested blobs grow too large.
2022-08-05 23:48:36 +02:00
Michael Eischer
55a11c1396 Reword prune --repack-small description 2022-08-05 23:48:36 +02:00
Michael Eischer
eaf43607f9 Add note that pack-size is not an exact limit 2022-08-05 23:48:36 +02:00
Michael Eischer
7f3b2be1e8 s3: Disable multipart uploads below 200MB 2022-08-05 23:48:36 +02:00
Michael Eischer
176b387d98 Always repack very small pack files 2022-08-05 23:48:36 +02:00
Michael Eischer
324935cb80 Only repack small files if there are multiple of them 2022-08-05 23:48:34 +02:00
Michael Eischer
1b076cda97 rename option to --pack-size 2022-08-05 23:47:43 +02:00
Michael Eischer
d7e2892048 Add changelog for packsize option 2022-08-05 23:47:43 +02:00
Michael Eischer
8a44258b6f update restic help snippets in documentation 2022-08-05 23:47:43 +02:00
Michael Eischer
420ddc03c9 rework pack size parameter documentation 2022-08-05 23:47:43 +02:00
Kyle Brennan
e43be84eb8 document minPackSize 2022-08-05 23:47:41 +02:00
Kyle Brennan
1e3f05c3f1 repository: prevent header overfill 2022-08-05 23:47:12 +02:00
Michael Eischer
6a6d313c9a prune: reduce priority of repacking small packs 2022-08-05 23:47:12 +02:00
Kyle Brennan
0269381b8d prune: add repack-small parameter 2022-08-05 23:47:12 +02:00
Michael Eischer
0a6fa602c8 add option for setting min pack size 2022-08-05 23:47:12 +02:00
Michael Eischer
2db7733ee3 fuse: remove unused MetaDir 2022-08-05 23:46:46 +02:00
Michael Eischer
f678f7cb04 fuse: cleanup test 2022-08-05 23:46:46 +02:00
Alexander Weiss
1751afae26 Make snapshots dirs in mount command customizable 2022-08-05 23:46:46 +02:00
Alexander Weiss
57f4003f2f Generalize fuse snapshot dirs implemetation
+ allow "/" in tags and snapshot template
2022-08-05 23:46:46 +02:00
Alexander Weiss
696c18e031 Add possibility to set snapshot ID (used in test) 2022-08-05 23:46:46 +02:00
MichaelEischer
04a8ee80fb Merge pull request #3829 from MichaelEischer/prune-refactor
Split prune into slightly small functions
2022-08-05 23:29:52 +02:00
MichaelEischer
15679be858 Merge pull request #3841 from MichaelEischer/compression-env
Add environment variable RESTIC_COMPRESSION
2022-08-05 23:20:04 +02:00
MichaelEischer
7d14b1baf1 Merge pull request #3845 from greatroar/solaris-xattr
internal/restic: Handle EINVAL for xattr on Solaris
2022-08-04 22:04:52 +02:00
greatroar
ad6ac680af internal/restic: Handle EINVAL for xattr on Solaris
Also make the errors a bit less verbose by not prepending the operation,
since pkg/xattr already does that. Old errors looked like

    Listxattr: xattr.list /myfiles/.zfs/snapshot: invalid argument
2022-08-01 12:45:17 +02:00
MichaelEischer
846d021db5 Merge pull request #3840 from greatroar/sftp-init
Speed up restic init over slow SFTP links
2022-07-31 19:54:59 +02:00
greatroar
b9fa6e05bd Add changelog for #3837/#3840 2022-07-30 22:53:52 +02:00
Michael Eischer
73053674d9 repository: Test fallback to existing blobs 2022-07-30 17:37:07 +02:00
Michael Eischer
e85a21eda2 prune: move code 2022-07-30 17:37:07 +02:00
Michael Eischer
623770eebb repository: try to recover from invalid blob while repacking
If a blob that should be kept is invalid, Repack will now try to request
the blob using LoadBlob. Only return an error if that fails.
2022-07-30 17:37:07 +02:00
Michael Eischer
d0590b7841 prune: Add internal integrity check
After repacking every blob that should be kept must have been repacked.
We have seen a few cases in which a single blob went missing, which
could have been caused by a bitflip somewhere. This sanity check might
help catch some of these cases.
2022-07-30 17:37:07 +02:00
Michael Eischer
5cbde03eae prune: split into smaller functions 2022-07-30 17:37:07 +02:00
Alexander Weiss
7643237da5 prune: separate collecting/printing/pruning 2022-07-30 17:37:07 +02:00
Michael Eischer
5723c72eb1 Add environment variable RESTIC_COMPRESSION 2022-07-30 16:21:53 +02:00
greatroar
2bdc40e612 Speed up restic init over slow SFTP links
pkg/sftp.Client.MkdirAll(d) does a Stat to determine if d exists and is
a directory, then a recursive call to create the parent, so the calls
for data/?? each take three round trips. Doing a Mkdir first should
eliminate two round trips for 255/256 data directories as well as all
but one of the top-level directories.

Also, we can do all of the calls concurrently. This may reintroduce some
of the Stat calls when multiple goroutines try to create the same
parent, but at the default number of connections, that should not be
much of a problem.
2022-07-30 13:09:08 +02:00
greatroar
23ebec717c Remove stale comments from backend/sftp
The preExec and postExec functions were removed in
0bdb131521 from 2018.
2022-07-30 13:07:25 +02:00
MichaelEischer
4ffd479ba4 Merge pull request #3773 from MichaelEischer/efficient-dir-json
Reduce memory usage for large directories/files
2022-07-23 17:47:32 +02:00
Michael Eischer
2ba14160de Add changelog for the optimized tree serialization 2022-07-23 14:49:08 +02:00
Michael Eischer
4a10ebed15 archiver: reduce memory usage for large files
FutureBlob now uses a Take() method as a more memory-efficient way to
retrieve the futures result. In addition, futures are now collected
while saving the file. As only a limited number of blobs can be queued
for uploading, for a large file nearly all FutureBlobs already have
their result ready, such that the FutureBlob object just consumes
memory.
2022-07-23 14:45:07 +02:00
Michael Eischer
b817681a11 archiver: Incrementally serialize tree nodes
That way it is not necessary to keep both the Nodes forming a Tree and
the serialized JSON version in memory.
2022-07-23 14:45:07 +02:00
Michael Eischer
c206a101a3 archiver: unify FutureTree/File into futureNode
There is no real difference between the FutureTree and FutureFile
structs. However, differentiating both increases the size of the
FutureNode struct.

The FutureNode struct is now only 16 bytes large on 64bit platforms.
That way is has a very low overhead if the corresponding file/directory
was not processed yet.

There is a special case for nodes that were reused from the parent
snapshot, as a go channel seems to have 96 bytes overhead which would
result in a memory usage regression.
2022-07-23 14:45:07 +02:00
Michael Eischer
32f4997733 archiver: remove unused fileInfo from progress callback 2022-07-23 14:16:23 +02:00
Michael Eischer
dcb00fd2d1 archiver: cleanup Saver interface 2022-07-23 14:16:23 +02:00
Michael Eischer
79321a195c archiver: remove dead attribute from FutureNode 2022-07-23 14:16:23 +02:00
MichaelEischer
049f4c4144 Merge pull request #3730 from MichaelEischer/stricter-check
Let `check` warn about legacy variants of the repo format version 1
2022-07-23 14:14:50 +02:00
Michael Eischer
3bf53da672 Add changelog for stricter checks 2022-07-23 11:21:26 +02:00
Michael Eischer
5a6f2f9fa0 Fix S3 legacy layout migration 2022-07-23 11:19:32 +02:00
Michael Eischer
04e49924fb checker: Fix S3 legacy layout detection 2022-07-23 11:19:32 +02:00
Michael Eischer
768c890fcb check: Deprecate --check-unused
Unused blobs are not a problem but rather expected to exist now that
prune by default does not remove every unused blob. However, the option
has caused questions from users whether a repository is damaged or not,
so just remove that option.

Note that the remaining code is left intact as it is still useful for
our test cases.
2022-07-23 11:19:32 +02:00
Michael Eischer
fcb3ddf181 check: Complain about usage of s3 legacy layout 2022-07-23 11:19:32 +02:00
Michael Eischer
8b8bd4e8ac check: complain about mixed pack files 2022-07-23 11:19:32 +02:00
MichaelEischer
443cc49afd Merge pull request #3830 from MichaelEischer/cleanup-repo
Extract Load/SaveTree/JSONUnpacked from repository
2022-07-23 10:46:13 +02:00
MichaelEischer
1f5369e072 Merge pull request #3831 from MichaelEischer/move-code
Move code out of the restic package and consolidate backend specific code
2022-07-23 10:33:05 +02:00
MichaelEischer
827ab02eea Merge pull request #3661 from rgammans/azure_sas_support
add SAS authentication option for Azure repos
2022-07-23 10:32:03 +02:00
MichaelEischer
e9c39442fb Merge pull request #3827 from MichaelEischer/backup-doc
backup: clarify usage string
2022-07-23 10:31:33 +02:00
Michael Eischer
9729e6d7ef backend: extract readerat from restic package 2022-07-17 15:29:09 +02:00
Michael Eischer
c44b21d366 restorer: extract hardlinks index from restic package 2022-07-17 13:45:42 +02:00
Michael Eischer
8c11fc3ec9 crypto: move crypto buffer helpers 2022-07-17 13:42:23 +02:00
Michael Eischer
a0cef9f247 limiter: move to internal/backend 2022-07-17 13:40:15 +02:00
Michael Eischer
163ab9c025 mock: move to internal/backend 2022-07-17 13:40:06 +02:00
Michael Eischer
89d3ce852b repository: extract Load/StoreJSONUnpacked
A Load/Store method for each data type is much clearer. As a result the
repository no longer needs a method to load / store json.
2022-07-17 13:22:00 +02:00
Michael Eischer
fbcbd5318c repository: extract LoadTree/SaveTree
The repository has no real idea what a Tree is. So these methods never
belonged there.
2022-07-17 13:11:28 +02:00
MichaelEischer
d9ea1e9ee2 Merge pull request #3290 from aawsome/prune-handle-duplicates
prune: Handle duplicate blobs more efficiently
2022-07-17 11:51:54 +02:00
Michael Eischer
715d457aad prune: code cleanups 2022-07-17 11:41:56 +02:00
Michael Eischer
9be1bd2acc prune: handle very high duplication of some blobs
Suggested-By: Alexander Weiss <alex@weissfam.de>
2022-07-17 11:39:56 +02:00
Alexander Weiss
7478cbf70e prune: Enhance treatment of duplicates 2022-07-17 00:22:23 +02:00
Michael Eischer
b2043e8198 backup: clarify usage string
Using the `--files-from` options it is possible to run `backup` without
specifying any source paths directly on the command line.
2022-07-17 00:03:22 +02:00
Michael Eischer
5639c41b6a azure: Strip ? prefix from sas token 2022-07-16 23:55:18 +02:00
Roger Gammans
64a7ec5341 azure: add SAS authentication option 2022-07-16 23:55:18 +02:00
MichaelEischer
6cbeb4a9f9 Merge pull request #3825 from restic/rawtaz-doc-gdrive
doc: Add note about using rclone for Google Drive
2022-07-16 19:21:59 +02:00
rawtaz
f5c219f5a2 doc: Add note about using rclone for Google Drive
It wasn't clear that Google Cloud Storage and Google Drive are two different services and that one should use the rclone backend for the latter. This commit adds a note with this information.
2022-07-16 13:22:38 +02:00
MichaelEischer
d71b29221b Merge pull request #3822 from JsBergbau/doc-max-compression
Added hint for --compression max in migration process
2022-07-12 23:02:45 +02:00
Michael Eischer
71ff6b77f0 doc: Rework hint to repack with max compression 2022-07-12 21:24:40 +02:00
MichaelEischer
6970d05d47 Merge pull request #3741 from lbausch/repository-wording
Wording: change repo to repository
2022-07-12 21:15:33 +02:00
MichaelEischer
3934480da4 Merge pull request #3819 from lbausch/restore-validate-patterns
restore: validate include/exclude patterns
2022-07-12 20:58:41 +02:00
MichaelEischer
71a0157c2c Merge pull request #3820 from lbausch/patch-1
Fix wording in changelog template
2022-07-12 20:51:52 +02:00
Michael Eischer
2aad6f24b5 doc: update sample help output 2022-07-12 20:49:01 +02:00
Michael Eischer
ec4dfa3c66 Wording: replace further repo occurrences with repository 2022-07-12 20:48:01 +02:00
JsBergbau
8d3f04aefa Added hint for --compression max in migration process
Added hint for --compression max in migration process. Since this is a onetime process users should be aware of this and consider this step.
2022-07-12 00:08:26 +02:00
Lorenz Bausch
b609523582 Add changelog entry 2022-07-09 22:24:56 +02:00
lbausch
ac96a4138d Fix wording in changelog template 2022-07-09 22:13:54 +02:00
Lorenz Bausch
7e36ec279d Test restore fails when using invalid patterns 2022-07-08 20:09:26 +02:00
Lorenz Bausch
be524f0b78 Add testRunRestoreAssumeFailure function 2022-07-08 20:09:25 +02:00
Lorenz Bausch
9a7db6675c Restore: validate provided patterns 2022-07-08 20:09:25 +02:00
Lorenz Bausch
d6e3c7f28e Wording: change repo to repository 2022-07-08 20:05:35 +02:00
MichaelEischer
98a3125ce4 Merge pull request #3816 from mattxtaz/master
Fix minor typo in docs
2022-07-05 21:30:26 +02:00
mattxtaz
87d899c099 Fix minor typo in docs 2022-07-05 20:06:27 +01:00
Jerome Küttner
6f3883c9d2 add changelog 2022-07-05 08:47:48 +02:00
Alexander Neumann
545220803b Merge pull request #3813 from MichaelEischer/fix-blob-saver-data-race
Fix data race in blob_saver
2022-07-04 08:10:01 +02:00
Michael Eischer
ce89018902 Fix data race in blob_saver
After the `BlobSaver` job is submitted, the buffer can be released and
reused by another `FileSaver` even before `BlobSaver.Save` returns. That
FileSaver will the change `buf.Data` leading to wrong backup statistics.

Found by `go test -race ./...`:

WARNING: DATA RACE
Write at 0x00c0000784a0 by goroutine 41:
  github.com/restic/restic/internal/archiver.(*FileSaver).saveFile()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:176 +0x789
  github.com/restic/restic/internal/archiver.(*FileSaver).worker()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:242 +0x2af
  github.com/restic/restic/internal/archiver.NewFileSaver.func2()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:88 +0x5d
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/michael/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x91

Previous read at 0x00c0000784a0 by goroutine 29:
  github.com/restic/restic/internal/archiver.(*BlobSaver).Save()
      /home/michael/Projekte/restic/restic/internal/archiver/blob_saver.go:57 +0x1dd
  github.com/restic/restic/internal/archiver.(*BlobSaver).Save-fm()
      <autogenerated>:1 +0xac
  github.com/restic/restic/internal/archiver.(*FileSaver).saveFile()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:191 +0x855
  github.com/restic/restic/internal/archiver.(*FileSaver).worker()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:242 +0x2af
  github.com/restic/restic/internal/archiver.NewFileSaver.func2()
      /home/michael/Projekte/restic/restic/internal/archiver/file_saver.go:88 +0x5d
  golang.org/x/sync/errgroup.(*Group).Go.func1()
      /home/michael/go/pkg/mod/golang.org/x/sync@v0.0.0-20210220032951-036812b2e83c/errgroup/errgroup.go:57 +0x91
2022-07-03 14:47:53 +02:00
MichaelEischer
b6a38d43b3 Merge pull request #3611 from MichaelEischer/auto-concurrency
Adjust worker goroutines to number of backend connections
2022-07-03 12:33:32 +02:00
Michael Eischer
3af9c2cc58 Document automatic CPU/IO-concurrency 2022-07-03 12:19:26 +02:00
Michael Eischer
6f53ecc1ae adapt workers based on whether an operation is CPU or IO-bound
Use runtime.GOMAXPROCS(0) as worker count for CPU-bound tasks,
repo.Connections() for IO-bound task and a combination if a task can be
both. Streaming packs is treated as IO-bound as adding more worker
cannot provide a speedup.

Typical IO-bound tasks are download / uploading / deleting files.
Decoding / Encoding / Verifying are usually CPU-bound. Several tasks are
a combination of both, e.g. for combined download and decode functions.
In the latter case add both limits together. As the backends have their
own concurrency limits restic still won't download more than
repo.Connections() files in parallel, but the additional workers can
decode already downloaded data in parallel.
2022-07-03 12:19:26 +02:00
MichaelEischer
cd50feb66f Merge pull request #3489 from MichaelEischer/async-pack-uploads
Asynchronously upload pack files
2022-07-03 11:56:05 +02:00
Michael Eischer
74df9d5998 Add changelog for async pack uploads 2022-07-03 11:34:01 +02:00
Michael Eischer
dbb5860dc9 Document connections and compression option 2022-07-03 11:19:24 +02:00
Michael Eischer
753e56ee29 repository: Limit to a single pending pack file
Use only a single not completed pack file to keep the number of open and
active pack files low. The main change here is to defer hashing the pack
file to the upload step. This prevents the pack assembly step to become
a bottleneck as the only task is now to write data to the temporary pack
file.

The tests are cleaned up to no longer reimplement packer manager
functions.
2022-07-02 22:42:34 +02:00
Michael Eischer
fa25d6118e archiver: Reduce tree saver concurrency
Large amount of tree savers have no obvious benefit, however they can
increase the amount of (potentially large) trees kept in memory.
2022-07-02 22:42:34 +02:00
Michael Eischer
bba1e81719 archiver: Limit blob saver count to GOMAXPROCS
Now with the asynchronous uploaders there's no more benefit from using
more blob savers than we have CPUs. Thus use just one blob saver for
each CPU we are allowed to use.
2022-07-02 22:42:34 +02:00
Michael Eischer
120ccc8754 repository: Rework blob saving to use an async pack uploader
Previously, SaveAndEncrypt would assemble blobs into packs and either
return immediately if the pack is not yet full or upload the pack file
otherwise. The upload will block the current goroutine until it
finishes.

Now, the upload is done using separate goroutines. This requires changes
to the error handling. As uploads are no longer tied to a SaveAndEncrypt
call, failed uploads are signaled using an errgroup.

To count the uploaded amount of data, the pack header overhead is no
longer returned by `packer.Finalize` but rather by
`packer.HeaderOverhead`. This helper method is necessary to continue
returning the pack header overhead directly to the responsible call to
`repository.SaveBlob`. Without the method this would not be possible,
as packs are finalized asynchronously.
2022-07-02 22:42:34 +02:00
Michael Eischer
bba4c69a2a tag: Remove unnecessary flush call 2022-07-02 22:42:23 +02:00
MichaelEischer
3e1de52e0a Merge pull request #3805 from greatroar/global
cmd/restic, limiter: Move config knowledge to internal packages
2022-07-02 21:56:35 +02:00
MichaelEischer
621023a50b Merge pull request #3772 from MichaelEischer/fix-mixed-index
rebuild-index: correctly rebuild index for mixed packs
2022-07-02 20:10:02 +02:00
MichaelEischer
90e9c5c4cc Merge pull request #3729 from MichaelEischer/full-ids-in-check
Include full IDs in check output
2022-07-02 20:09:39 +02:00
MichaelEischer
7137034517 Merge pull request #3811 from restic/fix-secret-string-crash
Don't crash if SecretString is uninitialized
2022-07-02 19:56:36 +02:00
Michael Eischer
cdaf9b4f26 Don't crash if SecretString is uninitialized 2022-07-02 19:44:28 +02:00
Michael Eischer
5e0f1c3cef check: remove dead code 2022-07-02 19:28:57 +02:00
Michael Eischer
0df022fa6d check: Print full ids
The short ids are not always unique. In addition, recovering from
damages is easier when having the full ids as that makes it easier to
access the corresponding files.
2022-07-02 19:28:57 +02:00
Michael Eischer
04c23fa95d rebuild-index: correctly rebuild index for mixed packs
For mixed packs, data and tree blobs were stored in separate index
entries. This results in warning from the check command and maybe other
problems.
2022-07-02 19:24:02 +02:00
MichaelEischer
bb5f196b09 Merge pull request #3733 from restic/improve-stats
Improve stats
2022-07-02 19:07:31 +02:00
MichaelEischer
c16f989d4a Merge pull request #3470 from MichaelEischer/sanitize-debug-log
Sanitize debug log
2022-07-02 19:00:54 +02:00
Michael Eischer
00d7fcff96 extend compression feature changelog entry 2022-07-02 18:55:59 +02:00
Michael Eischer
a6e9e08034 Account for pack header overhead at each entry
This will miss the pack header crypto overhead and the length field,
which only amount to a few bytes per pack file.
2022-07-02 18:55:58 +02:00
Michael Eischer
856d5e4303 stats: return storage size for raw-data mode
raw-data summed up the size of the blob plaintexts. However, with
compression this makes little sense as the storage size in the
repository is lower due to compression. Thus sum up the actual size each
blob takes in the repository.
2022-07-02 18:55:12 +02:00
Alexander Neumann
6c4ceaf1e7 Print number of bytes added to the repo
This includes optional compression and crypto overhead.
2022-07-02 18:55:12 +02:00
Alexander Neumann
99634c0936 Return real size from SaveBlob 2022-07-02 18:55:12 +02:00
MichaelEischer
fdc53a9d32 Merge pull request #3787 from MichaelEischer/refactor-repository
repository: (Mostly) index-related cleanups
2022-07-02 18:54:04 +02:00
Michael Eischer
6923353c43 redact swift auth token in debug output 2022-07-02 18:47:35 +02:00
Michael Eischer
5a11d14082 redacted keys/token in backend config debug log 2022-07-02 18:47:35 +02:00
Michael Eischer
0936d864a4 redact http authorization header in debug log output 2022-07-02 18:47:35 +02:00
Michael Eischer
ec7c9ce88b drop unused repository.Loader interface 2022-07-02 18:39:59 +02:00
Michael Eischer
2cd7e90ad1 repository: cleanup 2022-07-02 18:39:59 +02:00
Michael Eischer
c1a8fa4290 repository: remove unused packIDToIndex field 2022-07-02 18:39:59 +02:00
Michael Eischer
e68c3a4e62 repository: simplify CreateIndexFromPacks 2022-07-02 18:39:59 +02:00
Michael Eischer
1974ad7ce2 repository: hide MasterIndex.FinalizeFullIndexes / FinalizeNotFinalIndexes 2022-07-02 18:39:59 +02:00
Michael Eischer
ef53ca4a5a repository: remove MasterIndex.All() 2022-07-02 18:39:59 +02:00
Michael Eischer
bf81bf0795 repository: Properly set id for finalized index
As MergeFinalIndex and index uploads can occur concurrently, it is
necessary for MergeFinalIndex to check whether the IDs for an index were
already set before merging it. Otherwise, we'd loose the ID of an index
which is set _after_ uploading it.
2022-07-02 18:39:59 +02:00
Michael Eischer
e0a7852b8b repository: remove unused (Master)Index.Count 2022-07-02 18:39:58 +02:00
Michael Eischer
8ef2968f28 repository: remove unused index.ListPack 2022-07-02 18:39:12 +02:00
Michael Eischer
e4f20dea61 repository: inline index.encode 2022-07-02 18:39:12 +02:00
Michael Eischer
fe5a8e137a repository: remove unused index.Store 2022-07-02 18:39:12 +02:00
Michael Eischer
628ae799ca repository: make flushPacks private 2022-07-02 18:39:12 +02:00
Michael Eischer
ed8aa15376 repository: add Save method to MasterIndex interface 2022-07-02 18:38:56 +02:00
Michael Eischer
a77d5c4d11 repository: index saving belongs into the MasterIndex 2022-07-02 18:38:56 +02:00
MichaelEischer
19641bf828 Merge pull request #3810 from greatroar/revert-3786
Revert "restic prune: Merge three loops over the index"
2022-07-01 23:13:39 +02:00
greatroar
a0fa9c6e9f Revert "restic prune: Merge three loops over the index"
This reverts commit 8bdfcf779f.
Should fix #3809. Also needed to make #3290 apply cleanly.
2022-06-30 15:27:34 +02:00
greatroar
90d2c0502b cmd/restic, limiter: Move config knowledge to internal packages
The GlobalOptions struct now embeds a backend.TransportOptions, so it
doesn't need to construct one in open and create. The upload and
download limits are similarly now a struct in internal/limiter that is
embedded in GlobalOptions.
2022-06-22 18:29:58 +02:00
MichaelEischer
bc96879d41 Merge pull request #3785 from MichaelEischer/replace-tomb-usage
Remove usage of tomb package
2022-06-19 14:42:48 +02:00
MichaelEischer
307f14604f Merge pull request #3795 from greatroar/sema
backend: Move semaphores to a dedicated package
2022-06-18 17:12:01 +02:00
MichaelEischer
19581dbc18 Merge pull request #3786 from greatroar/prune
restic prune: Merge three loops over the index
2022-06-18 16:54:50 +02:00
greatroar
8bdfcf779f restic prune: Merge three loops over the index
There were three loops over the index in restic prune, to find
duplicates, to determine sizes (in pack.Size) and to generate packInfos.
These three are now one loop. This way, prune doesn't need to construct
a set of duplicate blobs, pack.Size doesn't need to contain special
logic for prune's use case (the onlyHdr argument) and pack.Size doesn't
need to construct a map only to have it immediately transformed into a
different map.

Some quick testing on a 160GiB local repo doesn't show running time or
memory use of restic prune --dry-run changing significantly.
2022-06-18 10:40:33 +02:00
greatroar
910d917b71 backend: Move semaphores to a dedicated package
... called backend/sema. I resisted the temptation to call the main
type sema.Phore. Also, semaphores are now passed by value to skip a
level of indirection when using them.
2022-06-18 10:01:58 +02:00
MichaelEischer
2c893fe43c Merge pull request #3798 from greatroar/errors
all: Move away from pkg/errors, easy cases
2022-06-17 19:01:40 +02:00
greatroar
f92ecf13c9 all: Move away from pkg/errors, easy cases
github.com/pkg/errors is no longer getting updates, because Go 1.13
went with the more flexible errors.{As,Is} function. Use those instead:
errors from pkg/errors already support the Unwrap interface used by 1.13
error handling. Also:

* check for io.EOF with a straight ==. That value should not be wrapped,
  and the chunker (whose error is checked in the cases changed) does not
  wrap it.
* Give custom Error methods pointer receivers, so there's no ambiguity
  when type-switching since the value type will no longer implement error.
* Make restic.ErrAlreadyLocked private, and rename it to
  alreadyLockedError to match the stdlib convention that error type
  names end in Error.
* Same with rest.ErrIsNotExist => rest.notExistError.
* Make s3.Backend.IsAccessDenied a private function.
2022-06-14 08:36:38 +02:00
MichaelEischer
0c0e7b6957 Merge pull request #3776 from wjiec/bugfix/maxkeys-in-search
Limit number of key files tested while opening a repository
2022-06-12 15:51:10 +02:00
Michael Eischer
c9ef873192 tweak password test count changelog 2022-06-12 15:39:06 +02:00
Michael Eischer
d08549f0ae fix flaky key test 2022-06-12 14:19:06 +02:00
Jayson Wang
f144920ed5 fix handling of maxKeys in SearchKey 2022-06-12 14:19:06 +02:00
Alexander Neumann
1dd4b9b60e Merge pull request #3788 from greatroar/sftp-posix-rename
backend/sftp: Support atomic rename
2022-06-06 19:39:48 +02:00
Alexander Neumann
07114ccb21 Merge pull request #3789 from greatroar/fix-loadblob
internal/repository: Fix LoadBlob + fuzz test
2022-06-06 19:33:42 +02:00
greatroar
c9557b2822 internal/repository: Fix LoadBlob + fuzz test
When given a buf that is big enough for a compressed blob but not its
decompressed contents, the copy at the end of LoadBlob would skip the
last part of the contents.

Fixes #3783.
2022-06-06 17:02:28 +02:00
greatroar
fa8f02292e backend/sftp: Support atomic rename
... if the server has posix-rename@openssh.com.
OpenSSH introduced this extension in 2008:
7c29661471
2022-06-06 13:40:42 +02:00
Alexander Neumann
7d64aa7f57 Merge pull request #3784 from MichaelEischer/cleanup-migrate-check
migrate: Cleanup option to request repository check
2022-06-05 17:37:45 +02:00
Michael Eischer
853ceb3bec get rid of tomb package 2022-06-05 15:54:57 +02:00
Michael Eischer
e002b09d57 archiver: free workers once finished 2022-06-05 15:48:10 +02:00
Michael Eischer
408ac1a0c2 archiver: remove tomb usage 2022-06-05 15:47:52 +02:00
Michael Eischer
5eba1217e7 migrate: Cleanup option to request repository check 2022-06-04 23:45:00 +02:00
MichaelEischer
0cb6b3d80a Merge pull request #3778 from greatroar/ellipsis
cmd/restic: Remove trailing "..." from progress messages
2022-06-04 19:16:00 +02:00
MichaelEischer
60ca6b1418 Merge pull request #3774 from greatroar/archiver-pool
archiver: Remove cleanup goroutine from BufferPool
2022-06-04 18:50:24 +02:00
Jerome Küttner
9adaa6e240 Delete existing path before restoring a symlink 2022-06-01 17:26:25 +02:00
greatroar
b7c990871f cmd/restic: Remove trailing "..." from progress messages
These were added after message since the last refactor of the progress
printing code. Also skips an allocation in the common case.
2022-05-31 19:06:26 +02:00
greatroar
0db1d11b2e archiver: Remove cleanup goroutine from BufferPool
This isn't doing anything. Channels should get cleaned up by the GC when
the last reference to them disappears, just like all other data
structures. Also inlined BufferPool.Put in Buffer.Release, its only
caller.
2022-05-29 17:09:16 +02:00
Alexander Neumann
74f7fe2b98 Merge pull request #3767 from MichaelEischer/fix-prune-empty-snapshot
prune: Fix crash on snapshot loading error
2022-05-29 16:52:45 +02:00
Alexander Neumann
d2c5843c68 Merge pull request #3704 from MichaelEischer/compression-migrations
Support migration to repository format with compression
2022-05-29 15:52:21 +02:00
Alexander Neumann
78a21bbccf Merge pull request #3752 from MichaelEischer/fix-dir-sync-errors
local: Ignore additional errors for directory syncing
2022-05-29 12:54:51 +02:00
MichaelEischer
2ce8587598 Merge pull request #3771 from greatroar/id-marshaljson
internal/restic: Custom ID.MarshalJSON
2022-05-28 16:24:51 +02:00
Michael Eischer
a73fc31b50 Fix linter check 2022-05-28 16:13:46 +02:00
greatroar
dde8e9e296 internal/restic: Custom ID.MarshalJSON
This skips an allocation. internal/archiver benchmarks, Linux/amd64:

name                     old time/op    new time/op    delta
ArchiverSaveFileSmall-8    3.94ms ± 6%    3.91ms ± 6%    ~     (p=0.947 n=20+20)
ArchiverSaveFileLarge-8     304ms ± 3%     301ms ± 4%    ~     (p=0.265 n=18+18)

name                     old speed      new speed      delta
ArchiverSaveFileSmall-8  1.04MB/s ± 6%  1.05MB/s ± 6%    ~     (p=0.803 n=20+20)
ArchiverSaveFileLarge-8   142MB/s ± 3%   143MB/s ± 4%    ~     (p=0.421 n=18+19)

name                     old alloc/op   new alloc/op   delta
ArchiverSaveFileSmall-8    17.9MB ± 0%    17.9MB ± 0%  -0.01%  (p=0.000 n=19+19)
ArchiverSaveFileLarge-8     382MB ± 2%     382MB ± 1%    ~     (p=0.687 n=20+19)

name                     old allocs/op  new allocs/op  delta
ArchiverSaveFileSmall-8       540 ± 1%       528 ± 0%  -2.19%  (p=0.000 n=19+19)
ArchiverSaveFileLarge-8     1.93k ± 3%     1.79k ± 4%  -7.06%  (p=0.000 n=20+20)
2022-05-27 12:26:37 +02:00
Alexander Neumann
bc27c370e7 Update gopkg.in/yaml
This fixes a panic in invalid input, but I think we aren't affected.
2022-05-26 14:23:49 +02:00
Alexander Neumann
9e30152f3c Merge pull request #3770 from lbausch/update-minio-go
Update github.com/minio/minio-go/v7 to v7.0.27
2022-05-26 14:21:19 +02:00
Lorenz Bausch
cc3f8d3732 Update github.com/minio/minio-go/v7 to v7.0.27
This version adds support for Cloudflare R2, as discussed in #3757
2022-05-26 13:05:13 +02:00
Michael Eischer
c8e1ac4049 prune: Don't print stack trace if snapshot can't be loaded 2022-05-23 22:38:45 +02:00
Michael Eischer
173695104c prune: Fix crash on empty snapshot 2022-05-23 22:32:59 +02:00
MichaelEischer
ded783dd61 Merge pull request #3762 from marigbede/marigbede-gs-documentation-fix
Update 030_preparing_a_new_repo.rst
2022-05-23 22:10:38 +02:00
Arigbede Moses
2aa75d6272 Update 030_preparing_a_new_repo.rst 2022-05-18 22:03:59 +01:00
MichaelEischer
88a8701fb5 Merge pull request #3734 from lbausch/validate-patterns
Validate exclude patterns
2022-05-14 16:20:15 +02:00
MichaelEischer
b2a2e5f727 Merge pull request #3753 from greatroar/indexmap-alloc
repository: Re-tune indexmap allocation strategy
2022-05-14 15:44:08 +02:00
MichaelEischer
b52c631bd3 Merge pull request #3754 from greatroar/simplify-hashing
hashing: Fix up comments
2022-05-14 15:33:51 +02:00
Lorenz Bausch
e7df66cc91 Add changelog entry for validating exclude patterns 2022-05-11 22:41:05 +02:00
Lorenz Bausch
36bd464e8c Add tests for validating exclude patterns 2022-05-11 22:41:00 +02:00
greatroar
39a335e690 hashing: Fix up comments 2022-05-11 21:36:10 +02:00
greatroar
5141228e0c repository: Re-tune indexmap allocation strategy
fd05037e1a changed the allocation batch
size from 256 to 128 under the assumption that an indexEntry is 60 bytes
on amd64, but it's 64: structs are padded out to a multiple of 8 for
alignment reasons. That means we'd waste no space in malloc even without
the batch allocation, at least on 64-bit machines. While that strategy
cuts the overallocation down dramatically for many small indexes, it also
seems to slow allocation down (Go 1.18, Linux, amd64, -benchtime=2s):

    name                   old time/op    new time/op    delta
    DecodeIndex-8             4.67s ± 5%     4.60s ± 1%      ~     (p=0.953 n=10+5)
    DecodeIndexParallel-8     4.67s ± 3%     4.60s ± 1%      ~     (p=0.953 n=10+5)
    IndexHasUnknown-8        37.8ns ± 8%    36.5ns ±14%      ~     (p=0.841 n=5+5)
    IndexHasKnown-8          38.5ns ±12%    37.7ns ±10%      ~     (p=0.968 n=5+5)
    IndexAlloc-8              615ms ±18%     607ms ± 1%      ~     (p=1.000 n=10+5)
    IndexAllocParallel-8      245ms ±11%     285ms ± 6%   +16.40%  (p=0.001 n=10+5)
    MasterIndexAlloc-8        286ms ± 9%     275ms ± 2%      ~     (p=1.000 n=10+5)
    LoadIndex/v1-8           27.0ms ± 4%    26.8ms ± 1%      ~     (p=0.690 n=5+5)
    LoadIndex/v2-8           22.4ms ± 1%    22.8ms ± 2%    +1.48%  (p=0.016 n=5+5)

    name                   old alloc/op   new alloc/op   delta
    IndexAlloc-8              446MB ± 0%     446MB ± 0%    -0.00%  (p=0.000 n=8+4)
    IndexAllocParallel-8      446MB ± 0%     446MB ± 0%    -0.00%  (p=0.008 n=8+5)
    MasterIndexAlloc-8        213MB ± 0%     159MB ± 0%   -25.47%  (p=0.000 n=10+5)

    name                   old allocs/op  new allocs/op  delta
    IndexAlloc-8               913k ± 0%     2632k ± 0%  +188.19%  (p=0.008 n=5+5)
    IndexAllocParallel-8       913k ± 0%     2632k ± 0%  +188.21%  (p=0.008 n=5+5)
    MasterIndexAlloc-8         318k ± 0%     1172k ± 0%  +267.86%  (p=0.008 n=5+5)

Instead, this patch sets a batch size of 4, which means no space is
wasted by malloc on 64-bit and very little on 32-bit. It still gets very
close to the savings from not allocating in batches, without requiring
special code for bits.UintSize==64. Benchmark results, again for
Linux/amd64:

    name                   old time/op    new time/op    delta
    DecodeIndex-8             4.67s ± 5%     4.83s ± 9%     ~     (p=0.315 n=10+10)
    DecodeIndexParallel-8     4.67s ± 3%     4.68s ± 4%     ~     (p=0.315 n=10+10)
    IndexHasUnknown-8        37.8ns ± 8%    44.5ns ±19%     ~     (p=0.095 n=5+5)
    IndexHasKnown-8          38.5ns ±12%    36.9ns ± 8%     ~     (p=0.690 n=5+5)
    IndexAlloc-8              615ms ±18%     628ms ±18%     ~     (p=0.218 n=10+10)
    IndexAllocParallel-8      245ms ±11%     262ms ± 9%   +7.02%  (p=0.043 n=10+10)
    MasterIndexAlloc-8        286ms ± 9%     287ms ±13%     ~     (p=1.000 n=10+10)
    LoadIndex/v1-8           27.0ms ± 4%    26.8ms ± 0%     ~     (p=1.000 n=5+5)
    LoadIndex/v2-8           22.4ms ± 1%    22.5ms ± 0%     ~     (p=0.056 n=5+5)

    name                   old alloc/op   new alloc/op   delta
    IndexAlloc-8              446MB ± 0%     446MB ± 0%     ~     (p=1.000 n=8+10)
    IndexAllocParallel-8      446MB ± 0%     446MB ± 0%   -0.00%  (p=0.000 n=8+8)
    MasterIndexAlloc-8        213MB ± 0%     160MB ± 0%  -25.02%  (p=0.000 n=10+9)

    name                   old allocs/op  new allocs/op  delta
    IndexAlloc-8               913k ± 0%     1333k ± 0%  +45.94%  (p=0.000 n=8+10)
    IndexAllocParallel-8       913k ± 0%     1333k ± 0%  +45.94%  (p=0.000 n=8+8)
    MasterIndexAlloc-8         318k ± 0%      525k ± 0%  +64.99%  (p=0.000 n=10+10)

The allocation method indexmap.newEntry has also been rewritten in a
form that is a few instructions shorter.
2022-05-11 21:22:14 +02:00
Michael Eischer
48a0d83143 local: Ignore additional errors for directory syncing
Apparently SMB/CIFS on Linux/macOS returns somewhat random errnos when
trying to sync a windows share which does not support calling fsync for
a directory.
2022-05-11 20:37:59 +02:00
MichaelEischer
ac36fda155 Merge pull request #3749 from greatroar/simplify-hashing
hashing: Remove io.WriterTo implementation
2022-05-11 20:03:43 +02:00
MichaelEischer
df554e5f69 Merge pull request #3748 from greatroar/runworkers
repository: Remove RunWorkers, report ctx.Err()
2022-05-11 19:38:46 +02:00
greatroar
54b8337813 hashing: Remove io.WriterTo implementation
This functionality has gone unused since
4b3dc415ef changed hashing.Reader's only
client to use ioutil.ReadAll on a bufio.Reader wrapping the hashing
Reader.

Reverts bcb852a8d0.
2022-05-10 23:41:18 +02:00
greatroar
2e0f1f5113 repository: Remove RunWorkers, report ctx.Err()
This removes RunWorkers, which had become mere overhead by successive
refactors. It also ensures that each former user of that function
returns any context error that occurs, so failure to complete an
operation is always reported as an error.
2022-05-10 22:26:00 +02:00
MichaelEischer
47c56dea5c Merge pull request #3746 from greatroar/cache-lstat
cache: Don't Lstat before creating CACHEDIR.TAG
2022-05-10 20:31:15 +02:00
Alexander Neumann
c270ab1e08 Merge pull request #3744 from MichaelEischer/fix-windows-temp-file
Fix error on temp file deletion on windows
2022-05-10 19:53:25 +02:00
greatroar
2da377c582 cache: Don't Lstat before creating the tag file
The tag file is opened with O_CREATE|O_EXCL and ErrExist is handled, so
we don't need to check for existence first.
2022-05-10 18:52:39 +02:00
Michael Eischer
ae7e51382a Fix error on temp file deletion on windows
Apparently it can take a moment between closing a tempfile marked as
DELETE_ON_CLOSE and it actually being deleted. During that time the file
is inaccessible. Thus just skip deleting the temp file on windows.
2022-05-09 22:43:26 +02:00
Michael Eischer
5c6db534d4 Add compression migration support to changelog 2022-05-09 22:39:02 +02:00
Michael Eischer
c1bbbcd0dc migrate: Allow migrations to request a check run
This is currently only used by upgrade_repo_v2.
2022-05-09 22:31:30 +02:00
Michael Eischer
59eb132dcd check: Better differentiate between warnings and errors 2022-05-09 22:31:30 +02:00
Michael Eischer
5815f727ee checker: convert error type to use pointer-receivers 2022-05-09 22:31:30 +02:00
Michael Eischer
4faff0debe doc: Describe repository upgrade process 2022-05-09 22:31:30 +02:00
Michael Eischer
e36a40db10 upgrade_repo_v2: Use atomic replace for supported backends 2022-05-09 22:31:30 +02:00
Michael Eischer
7559d2f105 Document repository version and minimum restic version 2022-05-09 22:31:30 +02:00
Michael Eischer
381bd94c6c prune: Add option to repack uncompressed data 2022-05-09 22:31:30 +02:00
Michael Eischer
5406743102 prune: Automatically repack uncompressed trees for repo v2
Tree packs are cached locally at clients and thus benefit a lot from
being compressed. Ensure this be having prune always repack pack files
containing uncompressed trees.
2022-05-09 22:31:30 +02:00
Alexander Neumann
c8c0d659ec Add migration to compress all data 2022-05-09 22:31:30 +02:00
Alexander Neumann
8c244214bf Add tests for upgrade migration 2022-05-09 22:31:30 +02:00
Alexander Neumann
a5f1d318ac Try to make repo upgrade migration more failsafe 2022-05-09 22:31:30 +02:00
Alexander Neumann
82ed5a3a15 Add repo upgrade migration 2022-05-09 22:31:30 +02:00
Alexander Neumann
3af6c180e4 Improve migrate command 2022-05-09 22:31:30 +02:00
Michael Eischer
92816fa966 init: Enable compression support by default 2022-05-09 22:31:30 +02:00
MichaelEischer
ab49c14621 Merge pull request #3740 from MichaelEischer/fix-restore-size-stats
Fix restore size calculation for multiple snapshots
2022-05-09 21:39:45 +02:00
Michael Eischer
2c07f7fff3 stats: hardlinks only reduce restore within a snapshot
The `stats` command checks inodes to not count hardlinked files multiple
times into the restore size. This check applies across all snapshots and
not only within snapshots. As a result the result size was far too low
when calculating it for multiple snapshots and it would vary depending
on the order in which snapshots were listed.
2022-05-09 21:26:24 +02:00
Lorenz Bausch
9fb81c4246 Validate exclude patterns 2022-05-07 21:12:47 +02:00
Lorenz Bausch
e7fd200237 Keep original pattern for later use 2022-05-07 21:08:09 +02:00
MichaelEischer
cc8a03b1d0 Merge pull request #3735 from HenrikBengtsson/master
DOCS: Incorrectly used SI units when IEC units were meant (fix #3669)
2022-05-07 11:15:27 +02:00
Henrik Bengtsson
9bb532672a DOCS: Incorrectly used SI units when IEC units were meant (fix #3669) 2022-05-02 21:21:39 -07:00
MichaelEischer
26c333325c Merge pull request #3715 from ema/master
doc: specify AWS Region via AWS_DEFAULT_REGION
2022-05-02 21:46:56 +02:00
Emanuele Rocca
38c0531b52 doc: specify AWS Region via AWS_DEFAULT_REGION
If no specific Region is mentioned in RESTIC_REPOSITORY, AWS defaults to
us-east-1. For this reason, users that follow the tutorial and create
their S3 bucket in any other region get the following error:

"Fatal: create repository at [...] client.BucketExists"

Explicitly specifying the AWS region name fixes the issue.
2022-05-02 21:26:58 +02:00
Alexander Neumann
fb5b9370f3 Merge pull request #3728 from MichaelEischer/debug-examine-upload
Add support to reupload blobs in `debug examine`
2022-04-30 20:39:19 +02:00
Alexander Neumann
ffbd48c0c6 Merge pull request #3481 from MichaelEischer/recover-enospace
Recover from no free space errors
2022-04-30 20:22:26 +02:00
Michael Eischer
95bcc9ea31 debug: Support pack ID prefixes in debug examine 2022-04-30 20:20:31 +02:00
Michael Eischer
2d6a943911 debug: Add switch to upload blobs extracted by debug examine
This simplifies salvaging a damaged pack file. Reuploading tree blobs
was previously not possible.
2022-04-30 20:20:31 +02:00
Alexander Neumann
9af499d8a4 Tidy go.sum 2022-04-30 20:16:09 +02:00
Alexander Neumann
2e3d23c1d7 Update github.com/klauspost/compress 2022-04-30 20:03:21 +02:00
Michael Eischer
dbbeac7174 prune: Add unsafe option to recover from no free space
The new option allows prune to operate with nearly no scratch space by only removing
no longer necessary pack files and first deleting the index before
rebuilding it. By first deleting the index it becomes safe to just
delete no longer necessary pack files. However, as a downside there's
now the risk that the repository becomes inaccessible if prune fails.

To recover from that problem a user might have to manually delete the
repository index and then run (a full) `rebuild-index` again.
2022-04-30 19:21:07 +02:00
Michael Eischer
cf5cb673fb repository: Use existing method to collect pack ids 2022-04-30 19:14:21 +02:00
Michael Eischer
b335cb6285 repository: Refactor index IDs collection 2022-04-30 19:14:21 +02:00
MichaelEischer
9c047f170a Merge pull request #3419 from DanielG/relax-file-modes
Enable admin to decide group access to repository files
2022-04-30 16:39:37 +02:00
Daniel Gröber
f31b4f29c1 Use config file modes to derive new dir/file modes
Fixes #2351
2022-04-30 15:59:51 +02:00
Alexander Neumann
71c653f9e0 Merge pull request #3727 from MichaelEischer/changelog-3475
Add changelog for local/sftp connection limit
2022-04-30 14:05:21 +02:00
Michael Eischer
29a8f92967 Add changelog for local/sftp connection limit 2022-04-30 13:36:47 +02:00
MichaelEischer
ac9324aeaf Merge pull request #3666 from MichaelEischer/compression
Implement compression support
2022-04-30 11:49:05 +02:00
Alexander Neumann
dc5adef255 Add documentation for --repository-version 2022-04-30 11:34:10 +02:00
Michael Eischer
4b01b06f2f repository: Test compressed blobs in StreamPack 2022-04-30 11:34:10 +02:00
Michael Eischer
bcab548617 pack: slightly expand testing of compressed blobs 2022-04-30 11:34:10 +02:00
Michael Eischer
ec2b25565a repository: test uncompressedLength field and index example 2022-04-30 11:34:10 +02:00
Michael Eischer
9ffb8920f1 repository: run blackbox tests using old and new repo version 2022-04-30 11:34:10 +02:00
Michael Eischer
abe5935693 repository: unify repository version-specific initialization
Mark the master index as compressed also when initializing a new
repository. This is only relevant for testing.
2022-04-30 11:34:10 +02:00
Alexander Neumann
8776031f96 Leave allocating slices to the decompress code 2022-04-30 11:34:10 +02:00
Alexander Neumann
5eb05a0afe Configure zstd encoder/decoder 2022-04-30 11:34:10 +02:00
Michael Eischer
2f36e044db Cleanup pack header check 2022-04-30 11:34:10 +02:00
Alexander Neumann
94dc9a0fa7 Amend changelog 2022-04-30 11:34:10 +02:00
Alexander Neumann
8b11b86383 Add option global --compression 2022-04-30 11:34:10 +02:00
Michael Eischer
f38f457a64 Add basic changelog for compression support 2022-04-30 11:34:10 +02:00
Michael Eischer
ba27d29d58 Print repository version when opening a repo 2022-04-30 11:34:10 +02:00
Michael Eischer
7132df529e repository: Increase index size for repo version 2
A compressed index is only about one third the size of an uncompressed
one. Thus increase the number of entries in an index to avoid cluttering
the repository with small indexes.
2022-04-30 11:34:10 +02:00
Michael Eischer
2535524132 debug: Add support for compressed blobs 2022-04-30 11:34:10 +02:00
Michael Eischer
fda7bb0f09 debug: Reduce code duplication 2022-04-30 11:34:10 +02:00
Michael Eischer
66f9048bce repository: Alloc zstd encoder/decoder on demand 2022-04-30 11:34:10 +02:00
Michael Eischer
fd05037e1a repository: recalibrate index batch allocation size 2022-04-30 11:34:10 +02:00
Michael Eischer
6fb408d90e repository: implement pack compression 2022-04-30 11:34:10 +02:00
Michael Eischer
362ab06023 init: Add flag to specify created repository version 2022-04-30 10:07:42 +02:00
Michael Eischer
4b957e7373 repository: Implement index/snapshot/lock compression
The config file is not compressed as it should remain readable by older
restic versions such that these can return a proper error.

As the old format for unpacked data does not include a version header,
make use of a trick: The old data is always encoded as JSON. Thus it can
only start with '{' or '['. For any other value the first byte indicates
a versioned format. The version is set to 2 for now. Then the zstd
compressed data follows.
2022-04-30 10:07:42 +02:00
Michael Eischer
0957b74887 Misc design.rst cleanups 2022-04-30 10:07:42 +02:00
Alexander Neumann
270ed00d1f doc: Add repository compression support documentation
Co-authored-by: Michael Eischer <michael.eischer@fau.de>
2022-04-30 10:07:42 +02:00
Alexander Neumann
4e1ef7804a Merge pull request #3717 from MichaelEischer/fix-stuck-repack
Fix stuck repack step
2022-04-30 09:50:43 +02:00
Alexander Neumann
e4780d3956 Merge pull request #3718 from MichaelEischer/sftp-docs-fix
doc: sftp with password actually works
2022-04-23 20:27:27 +02:00
Alexander Neumann
c183e35b5a Merge pull request #3719 from MichaelEischer/read-write-order
Describe repository read/write order required by repository format
2022-04-23 15:23:00 +02:00
Michael Eischer
6f9e20a1bb doc: Describe repository read/write order 2022-04-23 15:21:02 +02:00
Michael Eischer
f9219e8608 doc: sftp with password actually works 2022-04-23 11:57:36 +02:00
Michael Eischer
3b630d9998 add missing streamPacks changelog 2022-04-23 11:50:19 +02:00
Michael Eischer
566ac11c65 fix changelog name 2022-04-23 11:37:00 +02:00
Michael Eischer
f5609d1d3c prune: Fail early if too few backend connections 2022-04-23 11:32:52 +02:00
Michael Eischer
e597b99b55 repository: Reduce repack workers to prevent deadlock
As repack streams packs these occupy one backend connection. Uploading a
new pack also requires a backend connection. To prevent a deadlock
during repack when reaching the backend connections limit, simply limit
the repackWorker count to always leave one connection for uploading.
2022-04-23 11:28:18 +02:00
Michael Eischer
ee627cd832 backend/mem: Actually enforce connection limit
This will allow tests to detect deadlocks related to the connections
limit.
2022-04-23 11:22:00 +02:00
Michael Eischer
4f97492d28 Backend: Expose connections parameter 2022-04-23 11:13:08 +02:00
rawtaz
07a565e6f7 Merge pull request #3716 from MichaelEischer/password-error-on-stderr
Print password error message on stderr
2022-04-21 01:40:54 +02:00
Michael Eischer
bf7da7ff10 Print password error message on stderr
The password prompt itself is already printed on stderr.
2022-04-20 22:22:09 +02:00
Alexander Neumann
dba47d29d5 Merge pull request #3711 from restic/doc-grouping
doc: Clarify and make grouping in forget more noticeable
2022-04-16 10:08:11 +02:00
Leo R. Lundgren
8ac7519fd5 doc: Clarify and make grouping in forget more noticeable 2022-04-16 01:13:13 +02:00
Alexander Neumann
edc1a24a90 Merge pull request #3707 from duracell/patch-1
doc: fix missing "init" in rest-server example
2022-04-12 20:39:25 +02:00
Michael
9563e2f75c doc: fix missing "init" in rest-server example 2022-04-12 18:21:53 +02:00
Alexander Neumann
7f133a28b2 Update VERSION file 2022-04-11 20:34:14 +02:00
Alexander Neumann
4f3b1f19cb Set development version for 0.13.1 2022-04-10 20:41:00 +02:00
Alexander Neumann
89ee1cf9ee Merge pull request #3610 from MichaelEischer/windows-temp-files
Improve handling of temporary files on windows
2022-04-10 20:29:04 +02:00
Alexander Neumann
a059ef90f8 Merge pull request #3702 from MichaelEischer/extend-config-error
Print used key name if config fails to load
2022-04-10 20:25:24 +02:00
Michael Eischer
4077a81b34 Add simple test for fs.TempFile on windows 2022-04-09 23:37:58 +02:00
Michael Eischer
9a3f1a9703 Simplify and comment TempFile implementation for windows 2022-04-09 23:37:58 +02:00
Michael Eischer
c2aabb2686 Print used key name if config fails to load 2022-04-09 22:38:18 +02:00
MichaelEischer
c60a5f00c9 Merge pull request #3675 from ItsMattL/update
Refactor file handing for self-update.
2022-04-09 21:55:56 +02:00
Matt LaPlante
0ba9d4ced7 Refactor file handing for self-update.
* Write new file payload to a temp file before touching the original
binary. Minimizes the possibility of failing mid-write and corrupting
the binary.
* On Windows, move the original binary out to a temp file rather than
removing it as the running binary is locked. Fixes issue #2248.
2022-04-09 21:40:33 +02:00
Alexander Neumann
04e054465a Merge pull request #3475 from MichaelEischer/local-sftp-conn-limit
Limit concurrent operations for local / sftp backend
2022-04-09 21:33:00 +02:00
Alexander Neumann
1519e9f911 Merge pull request #3570 from MichaelEischer/list-snapshots-before-index
List snapshots before index
2022-04-09 21:15:49 +02:00
Michael Eischer
ebab35581c Check in integration test that snapshots are listed before the index
As an exception prune is still allowed to load the index before
snapshots, as it uses exclusive locks. In case of problems with locking
it is also better to load snapshots created after loading the index, as
this will lead to a prune sanity check failure instead of a broken snapshot.
2022-04-09 12:27:27 +02:00
Michael Eischer
7b9ae91e04 copy: Load snapshots before indexes 2022-04-09 12:27:25 +02:00
Michael Eischer
47243176fa diff: list snapshots only once 2022-04-09 12:26:31 +02:00
Michael Eischer
5af828e3e6 add changelogs 2022-04-09 12:26:31 +02:00
Michael Eischer
4636c20397 test that TestFindListOnce calls List only once 2022-04-09 12:26:31 +02:00
Michael Eischer
9e12159230 Fix O(n) backend list calls in FindFilteredSnapshots
When resolving snapshotIDs in FindFilteredSnapshots either
FindLatestSnapshot or FindSnapshot is called. Both operations issue a
list operation to the backend. When for example passing a long list of
snapshot ids to `forget` this could lead to a large number of list
operations.
2022-04-09 12:26:31 +02:00
Michael Eischer
3d29083e60 copy/find/ls/recover/stats: Memorize snapshot listing before index
These commands filter the snapshots according to some criteria which
essentially requires loading the index before filtering the snapshots.
Thus create a copy of the snapshots list beforehand and use it later on.
2022-04-09 12:26:30 +02:00
Michael Eischer
2ec0f3303a backup/diff/dump/restore/stats: List snapshots before index
During a backup the index is written before the corresponding snapshots.
To ensure that a concurrent/later restic run can read a snapshot's data,
restic thus must first load the snapshots and only afterwards the index.
Otherwise it is not possible to ensure that the loaded index is recent
enough to cover all of the snapshot's data.
2022-04-09 12:24:09 +02:00
Michael Eischer
ece06f125e sftp: Limit concurrent backend operations 2022-04-09 12:21:38 +02:00
Michael Eischer
cd783358d3 local: Limit concurrent backend operations
Use a limit of 2 similar to the filereader concurrency in the archiver.
2022-04-09 12:21:38 +02:00
Michael Eischer
0b258cc054 backends: clean reader closing 2022-04-09 12:21:38 +02:00
Alex Duchesne
9e34c791c9 Better temp file cleanup on Windows. 2022-04-09 12:00:22 +02:00
Alexander Neumann
7d55b4f95e Merge pull request #3701 from restic/rawtaz-doc-exclude-cachedir
doc: Link to CACHEDIR.TAG specification
2022-04-08 09:35:08 +02:00
rawtaz
de4e3117eb doc: Link to CACHEDIR.TAG specification 2022-04-08 03:07:27 +02:00
MichaelEischer
500079d265 Merge pull request #3689 from brightdroid/patch-1
added documentation for zsh autocompletion
2022-04-03 21:49:47 +02:00
Alexander Neumann
192288bc9c Merge pull request #3696 from cqjjjzr/fix-win-procgrp
Fix rclone (scoop shim) and sftp issue due to detached console on Windows
2022-04-03 13:46:28 +02:00
Charlie Jiang
d9c9415cfd Fix rclone (scoop shim) and sftp issue due to detached console on Windows 2022-04-03 17:53:17 +08:00
Alexander Neumann
59370b6062 Merge pull request #3695 from jernej-9/typo-fix
Fix a typo in the docs
2022-04-01 20:10:40 +02:00
Jernej Debevc
6e5731bf2f Fix a typo in the docs 2022-04-01 17:16:55 +02:00
Alexander Neumann
305cd1e730 Merge pull request #3693 from greatroar/cast-btrfs-super-magic
Cast unix.Statfs_t.Type to int64 when checking for btrfs
2022-04-01 08:09:46 +02:00
greatroar
c23c0f7c14 Cast unix.Statfs_t.Type to int64 when checking for btrfs
Fixes #3687. Uses the cast suggested by @MichaelEischer, except that the
contant isn't cast along, because it's untyped and will be converted by
the compiler as necessary.
2022-03-31 22:30:45 +02:00
Alexander Neumann
774c2e75ca Merge pull request #3680 from restic/update-deps
Update all dependencies (except fuse), require Go 1.15
2022-03-30 21:36:08 +02:00
Alexander Neumann
66d50b72e3 Require Go 1.15 or later
The library github.com/golang-jwt/jwt/v4 requires the FillByte() method
of *big.Int, so we're raising the minimum Go version to 1.15.
2022-03-30 21:11:17 +02:00
Alexander Neumann
89d86a7933 Update all dependencies (except fuse) 2022-03-30 21:11:13 +02:00
Christoph Roeder
7510bdc247 added documentation for zsh autocompletion 2022-03-30 10:45:52 +02:00
Alexander Neumann
f190d2e60e Merge pull request #3686 from restic/fix-diff
Fix diff
2022-03-30 07:43:07 +02:00
Alexander Neumann
206550a246 Fix diff
Nodes in trees were always printed with a `+` in diff, regardless of
whether or not a dir was added or removed. Let's use the mode we were
passed in printDir().

Closes #3685
2022-03-29 21:05:11 +02:00
Alexander Neumann
db8a958991 Merge pull request #3683 from MichaelEischer/fix-golangci-lint-warnings
Fix golangci lint warnings
2022-03-29 11:45:10 +02:00
Alexander Neumann
724ace0e99 Merge pull request #3682 from MichaelEischer/refactor-code
Pack size calculation cleanup and misc other changes
2022-03-29 11:06:04 +02:00
Michael Eischer
af31266b7d golangci-lint: replace deprecated golint with revive 2022-03-28 22:33:17 +02:00
Michael Eischer
2f81af6afa bloblru: Fix comment for New function 2022-03-28 22:25:25 +02:00
Michael Eischer
61e179ee78 switch to golang.org/x/term 2022-03-28 22:24:15 +02:00
Michael Eischer
c60540b196 add go:build headers everywhere 2022-03-28 22:23:47 +02:00
Michael Eischer
fefe9f5c0e pack: Hide more implementation details 2022-03-28 22:12:16 +02:00
Michael Eischer
a773cb6527 pack: cleanup header size calculation 2022-03-28 22:09:49 +02:00
Michael Eischer
6408686973 repository: Simplify Blob equality check 2022-03-28 22:09:49 +02:00
Michael Eischer
243698680a crypto: Use helpers for size calculations 2022-03-28 22:09:49 +02:00
Michael Eischer
d6db5a1fc2 archiver: Fix test
The test relied on an undeocumented sideeffect of the LoadBlob implementation
2022-03-28 22:09:49 +02:00
Michael Eischer
f78bd14e28 repository: Remove pack implementation details from MasterIndex 2022-03-28 22:09:49 +02:00
Michael Eischer
dc3d77dacc repository: make saveAndEncrypt private 2022-03-28 22:09:49 +02:00
Michael Eischer
6877e7edbb repository: Rename LoadAndDecrypt to LoadUnpacked
The method is the complement for SaveUnpacked and not for
SaveAndEncrypt. The latter assembles blobs into pack files.
2022-03-28 22:09:49 +02:00
Michael Eischer
2e1613d4c6 errors: Ensure that errors.IsFatal(errors.Fatal("err")) == true
This fixes a few cases where restic output "Fatal: Fatal: [...]"
2022-03-28 22:09:49 +02:00
Alexander Neumann
a08b95c497 Merge pull request #3513 from MichaelEischer/fast-copy
Speed-up copy command
2022-03-28 20:18:43 +02:00
Michael Eischer
537b4c310a copy: Implement by reusing repack
The repack operation copies all selected blobs from a set of pack files
into new pack files. For prune the source and destination repositories
are identical. To implement copy, just use a different source and
destination repository.
2022-03-26 20:47:15 +01:00
Alexander Neumann
4d5db61bd0 Merge pull request #3484 from MichaelEischer/stream-check-repack
Stream packs in `check --read-data` and during repacking
2022-03-26 20:46:17 +01:00
Alexander Neumann
4ab12f59a5 Set development version for 0.13.0 2022-03-26 20:10:07 +01:00
Alexander Neumann
40791fff64 Add version for 0.13.0 2022-03-26 20:09:59 +01:00
Alexander Neumann
a53a4a23fd Update manpages and auto-completion 2022-03-26 20:09:59 +01:00
Alexander Neumann
b567c08e80 Generate CHANGELOG.md for 0.13.0 2022-03-26 20:09:40 +01:00
Alexander Neumann
0ca89b6fec Prepare changelog for 0.13.0 2022-03-26 20:09:39 +01:00
Alexander Neumann
d7e46c187a Merge pull request #3678 from restic/changelogs
Review, reword and polish unreleased changelog entries
2022-03-26 19:44:30 +01:00
Alexander Neumann
6aefe3e111 Merge pull request #3677 from restic/3490-polish
check: Adjust help and documentation for check --read-data-subset
2022-03-26 19:34:51 +01:00
Leo R. Lundgren
03137a34db Review, reword and polish unreleased changelog entries 2022-03-26 13:01:53 +01:00
Leo R. Lundgren
c7d637ec39 check: Adjust help and documentation for check --read-data-subset 2022-03-26 00:11:04 +01:00
Alexander Neumann
6087c4ad75 Merge pull request #3656 from lgommans/forget-security
forget: Update docs for readability and append-only considerations
2022-03-24 21:36:19 +01:00
Leo R. Lundgren
cdf478c8f4 doc: More updates to forget documentation and security considerations 2022-03-23 23:12:19 +01:00
Luc Gommans
80969a6347 Update docs according to comments from MichaelEischer in PR #3656 2022-03-23 23:12:19 +01:00
Leo R. Lundgren
676d5d498c doc: Update forget security considerations and thread model 2022-03-23 23:12:19 +01:00
Luc Gommans
9c1d49e312 Document "forget" security considerations and add references
Removing data based on a policy when the attacker had the opportunity to
add data to your repository comes with some considerations. This is
added to the 060_forget.rst documentation.

That document is also updated to reflect that restic now considers
the current system time while running "forget".

References to the security considerations section are added:
- In `restic forget --help`
- In the threat model (design.rst)
- In the (030) setup section where an append-only setup is referenced

A reference is also to be added to the `rest-server` readme's
append-only paragraph (see my fork).

This commit also resolves a typo (amount->number for countable noun),
changes a password length recommendation into the metric that
actually matters when creating passwords (entropy) since I was editing
these doc files anyway, and updates the outdated copyright year in
`conf.py`.

Some wording in 060_forget (line 21..22) was changed to clarify what
"forget" and "prune" do, to try and avoid the apparent misconception
that "forget" does not remove any data.
2022-03-23 23:12:19 +01:00
Alexander Neumann
e682f7c0d6 Add tests for StreamPack 2022-03-21 21:15:03 +01:00
Alexander Neumann
ca1e2316cf Merge pull request #3665 from MichaelEischer/sane-list-locks
list: Never lock the repository when listing lock files
2022-03-21 11:14:44 +01:00
Alexander Neumann
0b8b524f12 Merge pull request #3512 from MichaelEischer/cleaner-lock-refresh
Prevent lock refresh from leaving behind lots of stale locks
2022-03-21 11:10:37 +01:00
Alexander Neumann
a350625554 Merge pull request #3524 from MichaelEischer/atomic-sftp
sftp: Implement atomic uploads
2022-03-21 11:08:22 +01:00
Alexander Neumann
32e61f2620 Update changelog/unreleased/issue-1106
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-03-21 11:04:04 +01:00
Alexander Neumann
8388f66c4c Merge pull request #3668 from greatroar/symlink-size
Report symlink sizes from FUSE mount
2022-03-21 11:02:32 +01:00
Alexander Neumann
0937008648 Merge pull request #3654 from MichaelEischer/limit-huge-tree-streams
Limit number of large tree blobs loaded in parallel by StreamTrees
2022-03-21 11:01:04 +01:00
Alexander Neumann
3a285f91bc Merge pull request #2311 from vincentbernat/feature/negative-pattern
filter: ability to use negative patterns
2022-03-20 14:02:30 +01:00
Alexander Neumann
29a5778626 Improve wording 2022-03-20 13:46:16 +01:00
Michael Eischer
53656f019a filter: address review comments 2022-03-20 13:33:08 +01:00
Michael Eischer
cd190bee14 filter: short circuit if no negative patterns 2022-03-20 13:33:08 +01:00
Vincent Bernat
2ee07ded2b filter: ability to use negative patterns
This is quite similar to gitignore. If a pattern is suffixed by an
exclamation mark and match a file that was previously matched by a
regular pattern, the match is cancelled. Notably, this can be used
with `--exclude-file` to cancel the exclusion of some files.

Like for gitignore, once a directory is excluded, it is not possible
to include files inside the directory. For example, a user wanting to
only keep `*.c` in some directory should not use:

    ~/work
    !~/work/*.c

But:

    ~/work/*
    !~/work/*.c

I didn't write documentation or changelog entry. I would like to get
feedback if this is the right approach for excluding/including files
at will for backups. I use something like this as an exclude file to
backup my home:

    $HOME/**/*
    !$HOME/Documents
    !$HOME/code
    !$HOME/.emacs.d
    !$HOME/games
    # [...]
    node_modules
    *~
    *.o
    *.lo
    *.pyc
    # [...]
    $HOME/code/linux/*
    !$HOME/code/linux/.git
    # [...]

There are some limitations for this change:

 - Patterns are not mixed accross methods: patterns from file are
   handled first and if a file is excluded with this method, it's not
   possible to reinclude it with `--exclude !something`.

 - Patterns starting with `!` are now interpreted as a negative
   pattern. I don't think anyone was relying on that.

 - The whole list of patterns is walked for each match. We may
   optimize later by exiting early if we know no pattern is starting
   with `!`.

Fix #233
2022-03-20 13:33:08 +01:00
Michael Eischer
12606b575f filter: Cleanup variable naming 2022-03-20 13:33:08 +01:00
Michael Eischer
5f145f0c7e filter: introduce pattern struct 2022-03-20 13:33:08 +01:00
Vincent Bernat
13c40d4199 filter: additional tests for filter.List() 2022-03-20 13:33:08 +01:00
Alexander Neumann
13aae82635 Merge pull request #3673 from restic/update-go
Update go
2022-03-20 12:02:32 +01:00
Alexander Neumann
b85d035956 Fix calens install step 2022-03-20 11:36:45 +01:00
Alexander Neumann
47aa4613f7 Force Go to use Module Mode 2022-03-20 11:30:01 +01:00
Alexander Neumann
a9a5acb8ce Update golangci-lint 2022-03-20 11:26:56 +01:00
Alexander Neumann
6dee59b789 Install gox before checking out code
Otherwise newer Go versions complain that the hash for the installed
version of gox is not in the go.mod, which we don't want anyways because
the tests should use the latest version of gox.
2022-03-20 11:26:56 +01:00
Alexander Neumann
2e19d19216 Use latest Go version for cross-compile and lint 2022-03-20 11:26:56 +01:00
Alexander Neumann
18a1de0de1 Use "go get" or "go install" selectively
Go 1.18 dropped support for installing binaries via "go get", Go <= 1.16
does not support it. So we need to use the right verb depending on the
Go version.
2022-03-20 11:26:56 +01:00
Alexander Neumann
9b57fcc6b0 Fix build.go, minimum Go version is 1.14 2022-03-20 10:54:33 +01:00
Alexander Neumann
17878036d8 Update tests to Go 1.18 2022-03-20 10:54:24 +01:00
Jason Lenz
2b1932a258 Report symlink sizes from FUSE mount for snapshot dir
Fixes #3667.
2022-03-17 22:21:47 -05:00
greatroar
fdc738fb70 Report symlink sizes from FUSE mount
Fixes #3667.
2022-03-13 16:48:35 +01:00
MichaelEischer
daea461f15 Merge pull request #3663 from jimt/find-msgs
Remove period from find messages
2022-03-07 22:23:49 +01:00
Jim Tittsler
a3d99217a4 Remove period from find messages
Simplifies cut-and-paste of IDs (and makes stylistically
consistent with other messages) #3659
2022-03-07 11:16:04 +09:00
MichaelEischer
e0ab689ccd Merge pull request #3664 from DanielG/fix-doc-warning
doc: Fix block quote warning
2022-03-06 21:56:26 +01:00
Michael Eischer
7af69fd7b9 list: Never lock the repository when listing lock files
There's no point in locking the repository just to list the currently
existing lock files. This won't work for an exclusively locked
repository and is also confusing to users.
2022-03-06 21:44:51 +01:00
Daniel Gröber
49b67c8aaa doc: Fix block quote warning 2022-03-06 18:15:55 +01:00
rawtaz
44d543ede3 Merge pull request #3653 from MichaelEischer/fix-ls-option-help
ls: Fix description for --host, --tag and --path options
2022-02-19 23:06:24 +01:00
Michael Eischer
5ef4ee7760 ls: Fix description for --host, --tag and --path options 2022-02-19 22:36:02 +01:00
Michael Eischer
254c8743fc Limit number of large tree blobs loaded in parallel by StreamTrees
Load tree blobs with more than 50MB only from a single goroutine. Very
large tree blobs with for example 400 MB size can otherwise require
roughly 1GB * streamTreeParallelism memory.
2022-02-19 12:26:09 +01:00
MichaelEischer
ad4f4dbc7a Merge pull request #3645 from greatroar/stdin-parent
Don't set a parent for --stdin backups
2022-02-19 11:36:51 +01:00
greatroar
63f6a9b085 Don't set a parent for --stdin backups
Loading any parent tree for these only wastes time and memory.
Fixes #3641, where it was shown that the most recent tree will get
picked.

--parent is now implicitly ignored when --stdin is given.
2022-02-19 10:41:33 +01:00
MichaelEischer
4a2d5a146d Merge pull request #3507 from ahmgithubahm/document-AWS_PROFILE-support
Document AWS_PROFILE support
2022-02-18 23:40:25 +01:00
Michael Eischer
1efc26899d Update docs for AWS_PROFILE and AWS_SHARED_CREDENTIALS_FILE 2022-02-18 23:31:10 +01:00
MichaelEischer
8df246d0f3 Merge pull request #3628 from gum3ng/issue_3127
[#issue 3127] Add xattr support for Solaris
2022-02-17 22:07:39 +01:00
Michael Eischer
27524979e8 restorer: Remove dead code 2022-02-13 11:43:09 +01:00
gum3ng
dd30083c2b [#issue 3127] Add xattr support for Solaris 2022-02-13 14:24:37 +05:30
MichaelEischer
fb4c5af5c4 Merge pull request #3642 from gco/master
Fix test failures on Solaris
2022-02-12 22:07:14 +01:00
MichaelEischer
18ec49ddfa Merge pull request #3644 from duritong/centos-epel-repo
add a note about installation via epel for RHEL / CentOS Stream 8 & 9
2022-02-12 21:41:46 +01:00
Michael Eischer
5ec312ca06 sftp: Implement atomic uploads
Create a temporary file with a sufficiently random name to essentially
avoid any chance of conflicts. Once the upload has finished remove the
temporary suffix. Interrupted upload thus will be ignored by restic.
2022-02-12 20:30:49 +01:00
Michael Eischer
aebd24e414 Add changelog 2022-02-12 20:21:58 +01:00
Michael Eischer
d72181c8c1 Ensure that the lock cleanup handler is run after the global one
cleanup handlers run in the order in which they are added. As Go calls
init() functions in lexical order, the cleanup handler from global.go
was registered before that from lock.go, which is the correct order.

Make this order explicit to ensure that this won't break accidentally.
2022-02-12 20:21:58 +01:00
Michael Eischer
c6fd13425b remember the refreshed lock file even if removal failed
This ensures that restic won't create lots of new lock files without
deleting them later on.

In some cases a Delete operation on a backend can return a "File does
not exist" error even though the Delete operation succeeded. This can
for example be caused by request retries. This caused restic to forget
about the new lock file and continue trying to remove the old (already
deleted) lock file.
2022-02-12 20:21:58 +01:00
Michael Eischer
bba8ba7a5b repository: cancel streampack context after error 2022-02-12 20:18:25 +01:00
Michael Eischer
47554a3428 repository: Fix error handling in repack
When storing a blob fails, this is a fatal error which must not be
retried.
2022-02-12 20:18:25 +01:00
Michael Eischer
4b3dc415ef checker: cleanup header extraction 2022-02-12 20:18:25 +01:00
Michael Eischer
930a00ad54 checker: reuse bufio reader 2022-02-12 20:18:25 +01:00
Michael Eischer
34ebafb8b6 repository: don't crash if blob size is too short 2022-02-12 20:18:25 +01:00
Michael Eischer
becebf5d88 repository: remove unused DownloadAndHash 2022-02-12 20:18:25 +01:00
Michael Eischer
f1e58e7c7f checker: rewrite ReadData to stream packs 2022-02-12 20:18:25 +01:00
Michael Eischer
f40abd92fa restorer: convert to use StreamPack 2022-02-12 20:18:25 +01:00
Michael Eischer
f00f690658 repository: stream packs during repacking 2022-02-12 20:18:25 +01:00
Michael Eischer
c4a2bfcb39 repository: Add StreamPacks function
The function supports efficiently loading a specified list of blobs from
a single pack in a streaming fashion. That is there's no need for
temporary files independent of the pack size.
2022-02-12 20:18:25 +01:00
Michael Eischer
153e2ba859 repository: Implement lisiting blobs per pack file 2022-02-12 20:18:24 +01:00
MichaelEischer
cc90f2ba6b Merge pull request #2816 from greatroar/noatime
Set O_NOATIME flag on Linux
2022-02-07 21:38:31 +01:00
MichaelEischer
d8f58fb7bf Merge pull request #3592 from vgt/jsondiff
Add json output for diff command
2022-02-07 21:33:30 +01:00
duritong
a4786dda5a Update doc/020_installation.rst
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2022-02-06 21:31:18 +01:00
Michael Eischer
aaa7f94139 Add changelog for O_NOATIME 2022-02-06 15:00:37 +01:00
Michael Eischer
6b17a7110c backup: Set O_NOATIME in the right place
The archiver uses FS.OpenFile, where FS is an instance of the FS
interface. This is different from fs.OpenFile, which uses the OpenFile
method provided by the fs package.
2022-02-06 15:00:37 +01:00
greatroar
7080fed7ae Set O_NOATIME flag on Linux
Citing Kerrisk, The Linux Programming Interface:

    The O_NOATIME flag is intended for use by indexing and backup
    programs. Its use can significantly reduce the amount of disk
    activity, because repeated disk seeks back and forth across the
    disk are not required to read the contents of a file and to update
    the last access time in the file’s i-node[.]

restic used to do this, but the functionality was removed along with the
fadvise call in #670.
2022-02-06 15:00:34 +01:00
Michael Eischer
74f29ad09b diff: add basic test for json output format 2022-02-06 11:44:15 +01:00
Michael Eischer
5f34ad523f diff: fix test failure and add remark on quiet to changelog 2022-02-05 23:02:07 +01:00
MichaelEischer
58236ead12 Merge pull request #3619 from aneeshusa/avoid-time-travel-paradoxes-when-finding-parents
Avoid choosing parent snapshot newer than time of current snapshot
2022-02-05 22:51:24 +01:00
Michael Eischer
8ae4d86a84 rename snapshot timestamp filter variable 2022-02-05 22:42:38 +01:00
mh
3f0184ba2a add a note about installation via epel 2022-02-05 22:18:33 +01:00
rawtaz
90473ea9ff Merge pull request #3638 from leogott/patch-1
Documentation: Update msys2 wiki url after move
2022-01-27 17:07:45 +01:00
Leona 'leo' Gottfried
4e84e8ab3f Update msys2 wiki url after move
https://github.com/msys2/msys2/wiki/Porting was permanently moved and redirects to https://www.msys2.org/
I substituted the new location of the /wiki/Porting page in the docs
2022-01-27 11:09:50 +01:00
Greg
2e9180638e Fix test failures on Solaris
Add exceptions for symlinks, sticky bits, and device nodes in the same places where the BSDSs and/or Darwin have them.
2022-01-25 18:05:56 -08:00
Aneesh Agrawal
058dfc20da Avoid choosing parent snapshot newer than time of current snapshot
Currently, `restic backup` (if a `--parent` is not provided)
will choose the most recent matching snapshot as the parent snapshot.
This makes sense in the usual case,
where we tag the snapshot-being-created with the current time.

However, this doesn't make sense if the user has passed `--time`
and is currently creating a snapshot older than the latest snapshot.
Instead, choose the most recent snapshot
which is not newer than the snapshot-being-created's timestamp,
to avoid any time travel.

Impetus for this change:
I'm using restic for the first time!
I have a number of existing BTRFS snapshots
I am backing up via restic to serve as my initial set of backups.
I initially `restic backup`'d the most recent snapshot to test,
then started backing up each of the other snapshots.
I noticed in `restic cat snapshot <id>` output
that all the remaining snapshots have the most recent as the parent.
2022-01-23 23:55:00 -05:00
Aneesh Agrawal
502fc3281c Add CONTRIBUTING.md docs to not edit man pages
Document this code review feedback I got for other contributors.
2022-01-23 23:54:24 -05:00
David Vogt
77c850148a Add json output for diff command 2022-01-23 23:22:26 +01:00
MichaelEischer
df89aa0087 Merge pull request #3623 from invine/issue-3464
Skip lock file creation on forget with --no-lock and --dry-run
2022-01-23 18:12:34 +01:00
Pavel Frolov
792523b28b [issue 3464] skip lock creation in case of dry-run 2022-01-23 06:44:41 +03:00
MichaelEischer
f0a8182493 Merge pull request #3626 from fkusche/issue-3620-doc-unreferenced-packs
Update documentation regarding unreferenced packs
2022-01-22 15:15:25 +01:00
Florian Kusche
6183d0be53 Update output of restic check 2022-01-16 12:34:20 +01:00
rawtaz
7f6fc78f95 Merge pull request #3544 from restic/fix-b2-delete-retry
b2: Successful delete if file does not exist
2022-01-13 23:28:30 +01:00
rawtaz
abfbacf3d3 Merge pull request #3591 from MichaelEischer/prune-fix-max-repack
prune: Handle --max-repack-size=0 as expected
2022-01-13 03:53:20 +01:00
Florian Kusche
b0c1d0f9cd Update documentation regarding unreferenced packs
Also removes an unnecessary space at the end of the last line.
2022-01-09 11:45:03 +01:00
rawtaz
8b6fe845d4 Merge pull request #3618 from mattxtaz/master
Add missing colon in prune stats output and realign the fields
2022-01-06 23:07:56 +01:00
mattxtaz
6ff32ee4d3 Add missing colon in prune stats output and change padding to 14 chars to align the fields 2022-01-06 21:15:15 +00:00
rawtaz
2ff3b7d69c Merge pull request #3615 from gum3ng/issue_3558
doc: Add a FAQ section for invalid Windows filenames
2022-01-02 18:06:02 +01:00
gum3ng
9589de16db [issue 3558]: Add a FAQ section for invalid Windows filenames 2022-01-02 22:24:00 +05:30
MichaelEischer
2c3e5d943d Merge pull request #3593 from DarkKirb/parallelize-restic-copy
Parallelize blob upload/download for restic copy
2021-12-29 22:31:54 +01:00
Charlotte 🦝 Delenk
e2bb384a60 Parallelize blob upload/download for restic copy
Currently restic copy will copy each blob from every snapshot serially,
which has performance implications on high-latency backends such as b2.

This commit introduces 8x parallelism for blob downloads/uploads which
can improve restic copy operations up to 8x for repositories with many
small blobs on b2.

This commit also addresses the TODO comment in the copyTree function.

Related work:

A more thorough improvement of the restic copy performance can be found
in PR #3513
2021-12-29 18:59:09 +01:00
MichaelEischer
e5985e0d63 Merge pull request #3602 from cqjjjzr/fix-rclone-sigint
fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever (#3601)
2021-12-29 16:56:37 +01:00
MichaelEischer
8832837a8a Merge pull request #3607 from jtagcat/quieter_cleanup
logging: quiet 'removing n old cache dirs'
2021-12-29 16:06:35 +01:00
jtagcat
f92130d878 logging: quiet 'removing n old cache dirs'
Closes #3595

Choosing to include `stdoutIsTerminal()` as:
 - all other instances with `!opts.JSON` do so
 - this likely will not affect anything, especially when autorun
 - this seems to not be a meaningful enough summary
     to include in auto-backup reports

JSON is still likely not guaranteed to work and this is a suboptimal
  solution to this. Ideally, #1804 should refactor all print statements,
  and define+document(+handle) when stdoutIsTerminal() should be used.
  Else, it may end up more inconsistent and bulky
  (duplicate lines, longer files).
2021-12-29 01:08:29 +02:00
Charlie Jiang
a5b0e0bef4 fix: rclone receiving SIGINT prematurely on Windows causing restic hang forever
Co-authored-by: greatroar <61184462+greatroar@users.noreply.github.com>
2021-12-28 13:14:46 +08:00
rawtaz
e6e51b84ac Merge pull request #3605 from gum3ng/issue_3463
Improve clarity of error message in restic ls command
2021-12-27 21:45:23 +01:00
MichaelEischer
c5c3dfe10f Merge pull request #3590 from metalsp0rk/documentation-enhance
Document Safe Passwords and Clarify B2 App Key information
2021-12-27 20:21:43 +01:00
Kyle Brennan
19ec4d8f17 Document safe passwords. Fix #2238 2021-12-27 10:43:18 -08:00
Kyle Brennan
47ecd950b8 Enhance details about user application keys. Fix #2672 2021-12-27 10:43:15 -08:00
MichaelEischer
051cc7ce71 Merge pull request #3589 from metalsp0rk/copy-no-lock
Make Copy respect no lock
2021-12-27 19:11:02 +01:00
Michael Eischer
64e733f3d6 tweak copy --no-lock changelog 2021-12-27 18:22:25 +01:00
Gautam Menghani
017614c41a [#issue 3463] Improve clarity of error message in restic ls command 2021-12-27 22:42:27 +05:30
Michael Eischer
0cfdb82ea4 prune: Handle --max-repack-size=0 as expected
Previously the flag was ignored and `--max-repack-size=1` had to be
used.
2021-12-27 15:48:56 +01:00
Kyle Brennan
d5ed5da85c update changelog for copy --no-lock 2021-12-03 12:16:40 -08:00
Kyle Brennan
8eb83029a8 Make copy honor --no-lock 2021-12-03 09:50:28 -08:00
MichaelEischer
882d58abce Merge pull request #3163 from palbr/patch-1
Add PGP fingerprint to 020_installation.rst
2021-11-19 23:58:50 +01:00
Peter Albrecht
8de4401bb5 Changed URL for key-file
The keyfile provided by restic's own webserver (https://restic.net) should be
more stable than relying on public keyservers. So I changed the URL to the
GPG keyfile, as recommended by MichaelEischer.
2021-11-19 15:47:59 +01:00
MichaelEischer
f7a9b90eb9 Merge pull request #3573 from magandrez/feat/ls-report-symbolic-notation
Provide mode in symbolic notation for `restic ls --json`
2021-11-18 21:29:04 +01:00
MichaelEischer
aa214f99b4 Merge pull request #3565 from mathstuf/doc-missing-metadata
doc: mention metadata that is not backed up currently
2021-11-18 21:28:55 +01:00
MichaelEischer
4a25bbaed3 Merge pull request #3578 from adsultana/patch-1
docs: Fix link to "help wanted" issues
2021-11-18 21:20:30 +01:00
Michael Eischer
583edc39b8 doc: reorder backup metadata exceptions 2021-11-18 21:17:38 +01:00
Manuel González
212b2f651f Add file mode in symbolic notation to ls --json
This aligns `restic ls --json` with `restic find --json`, utilizing the same
naming.
2021-11-16 19:45:50 +02:00
Andrew Sultana
15ab96ecd6 docs: Fix link to "help wanted" issues 2021-11-16 00:03:34 +00:00
Ben Boeckel
d71afb3d32 doc: mention metadata that is not backed up currently
See: #3497
See: #1622
See: #2075
2021-11-13 18:40:26 -05:00
MichaelEischer
4bf05d91a1 Merge pull request #3571 from garrmcnu/blazer-unknown-authority
Update Backblaze B2 blazer module
2021-11-13 22:46:49 +01:00
MichaelEischer
de3afc1005 Merge pull request #3574 from gurjeet/rename_aws_s3_to_amazon_s3
Use S3's proper product name, Amazon S3
2021-11-13 22:30:32 +01:00
Michael Eischer
2ea998f70e Add PR link to changelog 2021-11-13 22:25:54 +01:00
Garry McNulty
e8fa3855e7 Update blazer
If a request fails with "x509: certificate signed by unknown authority",
the B2 backend now returns the error without retrying the request.

Closes #3556
Closes #2355
2021-11-13 22:25:54 +01:00
Gurjeet Singh
34a6a24544 Use S3's proper product name, Amazon S3
Per Amazon's product page [1], S3 is officially called "Amazon S3". The
restic project uses the phrase "AWS S3" in some places. This patch
corrects the product name.

[1]:https://aws.amazon.com/s3/
2021-11-13 22:21:06 +01:00
MichaelEischer
1d8a0b06cb Merge pull request #3575 from MichaelEischer/adjust-http2-canary-test
rest: Adjust http2-missing-eof test to golang >= 1.17.3
2021-11-13 22:13:24 +01:00
Michael Eischer
50053a85d3 rest: Adjust http2 missing eof test to golang >= 1.17.3, >= 1.16.10
The missing eof with http2 when a response included a content-length
header but no data, has been fixed in golang 1.17.3/1.16.10. Therefore
just drop the canary test and schedule it for removal once go 1.18 is
required as minimum version by restic.
2021-11-13 21:57:30 +01:00
MichaelEischer
f1cfb97237 Merge pull request #3514 from phcreery/rclone_timeout
rclone: extend timeout from 60s to 240s
2021-11-07 18:10:34 +01:00
MichaelEischer
cb81ee9396 Merge pull request #3474 from kitone/fix-issue-3382
Honor RESTIC_CACHE_DIR environment variable
2021-11-07 17:57:54 +01:00
Michael Eischer
b0e64deb27 rclone: Fix timeout calculation 2021-11-07 17:49:33 +01:00
phcreery
43d173b042 rclone: add timeout option and documentation 2021-11-07 17:49:21 +01:00
rawtaz
1b152a2c4d Merge pull request #3568 from MichaelEischer/fix-local-fd-leak
local: Fix fd leak when encountering files directly inside data/
2021-11-07 01:36:48 +01:00
MichaelEischer
15cc3c0e23 Merge pull request #3566 from mathstuf/check-progress-output-stomping
check: wait for progress bar output
2021-11-06 23:34:49 +01:00
kitone
5904f80cfa restic cache should display the name of the cache without shortening it in the case of the restic check 2021-11-06 20:18:51 +01:00
Ben Boeckel
4d579c4387 check: wait for progress bar output
Further code will also output to the terminal and the bar's cursor
positioning causes its output to overlap with the remaining output in a
racy way.

Fixes: #3344
2021-11-06 15:05:09 -04:00
Michael Eischer
15d7313387 local: Fix fd leak when encountering files directly inside data/ 2021-11-06 19:44:57 +01:00
MichaelEischer
6c84ea1412 Merge pull request #3548 from gum3ng/issue_3490
Support for specifying amount of data in read-data-subset
2021-11-05 23:28:11 +01:00
MichaelEischer
78c7dd53ef Merge pull request #3526 from greatroar/dump-refactor
Refactor internal/dump + concurrent load/write
2021-11-05 22:38:39 +01:00
MichaelEischer
a34bfa8269 Merge pull request #3562 from restic/rawtaz-mount-usage
mount: Improve usage information when mounted
2021-11-05 22:10:32 +01:00
kitone
0425a30420 add changelog issue-3382 2021-11-04 15:13:48 +01:00
kitone
1b23675f21 cache --cleanup should handle directories created by restic check.
Because there is no guarantee that a cleanup of these directories will occur
after the "restic check", we extend the behavior to detect and manage these
specific cache directories and allow their cleanup too.
2021-11-04 15:10:38 +01:00
Gautam Menghani
836fbb9133 [#issue 3490] Support for specifying file size in read-data-subset 2021-11-02 15:25:46 +05:30
greatroar
c71729dfc4 Refactor internal/dump + concurrent load/write
Package internal/dump has been reworked so its API consists of a single
type Dumper that handles tar and zip formats. Tree loading and node
writing happen concurrently.
2021-11-01 23:01:55 +01:00
Leo R. Lundgren
711ceb0109 mount: Improve usage information when mounted 2021-11-01 20:59:20 +01:00
MichaelEischer
829c0a67af Merge pull request #3520 from nxt-engineering/Docker
Docker Multistage build
2021-10-24 17:37:54 +02:00
Alexander Neumann
fb5d9345a7 Merge pull request #3510 from MichaelEischer/fix-archiver-early-on-abort
archiver: Fix TestArchiverAbortEarlyOnError test
2021-10-16 15:37:41 +02:00
kitone
95eb859b54 Honor RESTIC_CACHE_DIR environment variable
Fix #3382: restic check doesn't obey the RESTIC_CACHE_DIR environment variable
2021-10-10 16:00:02 +02:00
Michael Eischer
257740b0cc b2: Successful delete if file does not exist
When deleting a file, B2 sometimes returns a "500 Service Unavailable"
error but nevertheless correctly deletes the file. Due to retries in
the B2 library blazer, we sometimes also see a "400 File not present"
error. The retries of restic for the delete request then fail with
"404 File with such name does not exist.".

As we have to rely on request retries in a distributed system to handle
temporary errors, also consider a delete request to be successful if the
file is reported as not existing. This should be safe as B2 claims to
provide a strongly consistent bucket listing and thus a missing file
shouldn't mysteriously show up again later on.
2021-10-09 23:51:12 +02:00
MichaelEischer
46d08d9404 Merge pull request #3535 from jtagcat/writingverbose
Change: selfupdate: 'writing restic to' as verbose
2021-10-09 22:41:51 +02:00
jtagcat
a7853057ab Change: selfupdate: 'writing restic to' as verbose
Running restic self-update --quiet no longer
prints "writing restic to /usr/local/bin/restic".

The only output printed with -q is failures or
"successfully updated restic to version 0.12.1"

https://github.com/restic/restic/pull/3535

fix test fail: changelog title can't end with `.`

shorten changelog title
2021-10-09 23:24:18 +03:00
MichaelEischer
eb282532dc Merge pull request #3534 from jtagcat/patch-1
docs/example: useradd: restic now system account
2021-10-09 20:42:43 +02:00
MichaelEischer
f2a3b3b4a1 Merge pull request #3537 from greatroar/dump-lru
Check cap instead of len in bloblru
2021-10-09 20:00:27 +02:00
MichaelEischer
58e8b34633 Merge pull request #3532 from garrmcnu/s3-credentials-config
s3: Add warning if key ID or secret is empty
2021-10-09 19:32:32 +02:00
MichaelEischer
a02cea6e83 Merge pull request #3539 from jnoxon/fix-ec2-metadata-v2
get iam roles working with ec2 metadata v2
2021-10-07 23:00:27 +02:00
Garry McNulty
708d7a2574 s3: Add warning if key ID or secret is empty
Also add debug message if no credential types are available.

Closes #2388
2021-10-06 23:13:40 +01:00
Jeff Noxon
6f4b5ab8d1 get iam roles working with ec2 metadata v2 2021-10-03 20:06:55 -05:00
greatroar
634a9c162d Check cap instead of len in bloblru
restic dump uses bloblru.Cache to keep buffers alive, but also reuses
evicted buffers. That means large buffers may be used to store small
blobs, causing the cache to think it's using less memory than it
actually does.
2021-10-03 09:34:17 +02:00
jtagcat
632ca2ef52 docs/example: useradd: restic now system account
and: use /sbin/nologin
and: use long flags in the useradd command
and: download v0.12.1 instead of v0.9.6
2021-10-03 00:51:06 +03:00
MichaelEischer
24088f8307 Merge pull request #3528 from greatroar/tree-insert
internal/restic: Don't allocate in Tree.Insert
2021-09-27 22:22:03 +02:00
greatroar
c892c0bab9 internal/restic: Don't allocate in Tree.Insert
name         old time/op    new time/op    delta
BuildTree-8    34.6µs ± 4%     7.0µs ± 3%  -79.68%  (p=0.000 n=18+19)

name         old alloc/op   new alloc/op   delta
BuildTree-8    34.0kB ± 0%     0.9kB ± 0%  -97.37%  (p=0.000 n=20+20)

name         old allocs/op  new allocs/op  delta
BuildTree-8       108 ± 0%         1 ± 0%  -99.07%  (p=0.000 n=20+20)
2021-09-26 18:08:48 +02:00
MichaelEischer
78dac2fd48 Merge pull request #3523 from greatroar/swift-v2
Upgrade ncw/swift to v2
2021-09-24 21:41:36 +02:00
Michael Eischer
5ea8bba1a1 swift: restore context err check for list() 2021-09-24 21:19:46 +02:00
MichaelEischer
a5e103a212 Merge pull request #3522 from greatroar/dump-lru
Use LRU cache in restic dump
2021-09-24 20:33:58 +02:00
greatroar
e7ec0453b1 Upgrade ncw/swift to v2 2021-09-24 19:08:37 +02:00
Uli Martens
1ebcb1d097 Add changelog entry for PR #3508 2021-09-24 15:45:09 +02:00
greatroar
fe04d024c7 Use LRU cache in restic dump 2021-09-24 15:45:08 +02:00
Uli Martens
718966a81a Move Blobcache into dedicated internal package 2021-09-24 15:45:00 +02:00
greatroar
4f33eca634 Remove unused Writer arg to internal/dump.writeDump 2021-09-24 15:40:42 +02:00
MichaelEischer
cc110c42e6 Merge pull request #2657 from mansam/add-skip-tls-verification-flag
Add --insecure-tls flag to disable SSL cert verification
2021-09-22 21:40:01 +02:00
Sam Lucidi
897d8e662c Add --insecure-tls flag to disable SSL cert verification
Signed-off-by: Sam Lucidi <slucidi@redhat.com>
2021-09-21 10:52:40 -04:00
cimnine
4a95af5290 GitHub action for Docker build 2021-09-21 08:23:33 +02:00
cimnine
f28c8bc1c2 Multistage Docker build 2021-09-21 08:23:33 +02:00
MichaelEischer
1827b16ade Merge pull request #3519 from greatroar/maphash
Replace siphash by hash/maphash
2021-09-19 19:46:03 +02:00
greatroar
8b758c78a3 Require Go 1.14 to build 2021-09-19 16:18:19 +02:00
greatroar
8d2996eaaa Replace siphash by hash/maphash
In Go 1.17.1, maphash has become quite a bit faster than siphash, so we
can drop one third-party dependency. maphash is just an interface to the
standard Go map's hash function, which we already trust for other use
cases.

Benchmark results on linux/amd64, -benchtime=3s:

name                                             old time/op    new time/op    delta
IndexHasUnknown-8                                  50.6ns ±10%    41.0ns ±19%  -18.92%  (p=0.000 n=9+10)
IndexHasKnown-8                                    52.6ns ±12%    41.5ns ±12%  -21.13%  (p=0.000 n=9+10)
IndexMapHash-8                                     3.64µs ± 1%    2.00µs ± 0%  -45.09%  (p=0.000 n=10+9)
IndexAlloc-8                                        700ms ± 1%     601ms ± 6%  -14.18%  (p=0.000 n=8+10)
IndexAllocParallel-8                                205ms ± 5%     192ms ± 8%   -6.18%  (p=0.043 n=10+10)
MasterIndexAlloc-8                                  319ms ± 1%     279ms ± 5%  -12.58%  (p=0.000 n=10+10)
MasterIndexLookupSingleIndex-8                      156ns ± 8%     147ns ± 6%   -5.46%  (p=0.023 n=10+10)
MasterIndexLookupMultipleIndex-8                    150ns ± 7%     142ns ± 8%   -5.69%  (p=0.007 n=10+10)
MasterIndexLookupSingleIndexUnknown-8              74.4ns ± 6%    72.0ns ± 9%     ~     (p=0.175 n=10+9)
MasterIndexLookupMultipleIndexUnknown-8            67.4ns ± 9%    65.5ns ± 7%     ~     (p=0.340 n=9+9)
MasterIndexLookupParallel/known,indices=25-8        461ns ± 2%     445ns ± 2%   -3.49%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=25-8      408ns ±11%     378ns ± 5%   -7.22%  (p=0.035 n=10+9)
MasterIndexLookupParallel/known,indices=50-8        479ns ± 1%     437ns ± 4%   -8.82%  (p=0.000 n=10+10)
MasterIndexLookupParallel/unknown,indices=50-8      406ns ± 8%     343ns ±15%  -15.44%  (p=0.001 n=10+10)
MasterIndexLookupParallel/known,indices=100-8       480ns ± 1%     455ns ± 5%   -5.15%  (p=0.000 n=8+10)
MasterIndexLookupParallel/unknown,indices=100-8     391ns ±18%     382ns ± 8%     ~     (p=0.315 n=10+10)
MasterIndexLookupBlobSize-8                        71.0ns ± 8%    57.2ns ±11%  -19.36%  (p=0.000 n=9+10)
PackerManager-8                                     254ms ± 1%     254ms ± 1%     ~     (p=0.285 n=15+15)

name                                             old speed      new speed      delta
IndexMapHash-8                                   1.12GB/s ± 1%  2.05GB/s ± 0%  +82.13%  (p=0.000 n=10+9)
PackerManager-8                                   208MB/s ± 1%   207MB/s ± 1%     ~     (p=0.281 n=15+15)

name                                             old alloc/op   new alloc/op   delta
IndexMapHash-8                                      0.00B          0.00B          ~     (all equal)
IndexAlloc-8                                        400MB ± 0%     400MB ± 0%     ~     (p=1.000 n=9+10)
IndexAllocParallel-8                                401MB ± 0%     401MB ± 0%   +0.00%  (p=0.000 n=10+10)
MasterIndexAlloc-8                                  258MB ± 0%     262MB ± 0%   +1.42%  (p=0.000 n=9+10)
PackerManager-8                                    73.1kB ± 0%    73.1kB ± 0%     ~     (p=0.382 n=13+13)

name                                             old allocs/op  new allocs/op  delta
IndexMapHash-8                                       0.00           0.00          ~     (all equal)
IndexAlloc-8                                         907k ± 0%      907k ± 0%   -0.00%  (p=0.000 n=10+10)
IndexAllocParallel-8                                 907k ± 0%      907k ± 0%   +0.00%  (p=0.009 n=10+10)
MasterIndexAlloc-8                                   327k ± 0%      317k ± 0%   -3.06%  (p=0.000 n=10+10)
PackerManager-8                                       744 ± 0%       744 ± 0%     ~     (all equal)
2021-09-19 16:05:18 +02:00
MichaelEischer
58efe21eca Merge pull request #3264 from amozoss/upstream-master
Refactor backup progress
2021-09-19 14:54:42 +02:00
MichaelEischer
71fcf48533 Merge pull request #2594 from greatroar/concurrent-restore-verify
Concurrent restore --verify
2021-09-19 14:52:31 +02:00
Michael Eischer
921e328b56 restore: Fix linting error 2021-09-19 14:41:07 +02:00
Michael Eischer
e62d4f622f Reword changelog 2021-09-19 14:41:07 +02:00
Michael Eischer
2cdc0719af restorer: Sanitize verify errors 2021-09-19 14:01:26 +02:00
greatroar
bdcdfaf6b4 restore --verify: buffer reuse consistency and comment 2021-09-19 13:11:27 +02:00
greatroar
2b94742ca5 Replace no-op closures in restorer by nil check 2021-09-19 13:11:16 +02:00
greatroar
d357744104 Handle canceled context in restore --verify properly 2021-09-19 13:11:05 +02:00
greatroar
d4225ec803 Simplify buffer growing in Restorer.verifyFile
Suggested-by: Igor Fedorenko <igor@ifedorenko.com>
2021-09-19 13:10:55 +02:00
greatroar
de8521ae56 Report number of successes from Restorer.VerifyFiles
Previously, the number of attempts was reported.
2021-09-19 13:10:44 +02:00
greatroar
bb066cf7d3 Concurrent Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 9.726s to 4.645s (-52%).
2021-09-19 13:10:41 +02:00
greatroar
556424d61b Reuse buffer in Restorer.VerifyFiles
Time to verify a 2GB snapshot down from 11.568s to 9.726s (-16%).
2021-09-19 12:55:31 +02:00
greatroar
92ae951ffa Report timing from restic restore --verify 2021-09-19 12:53:09 +02:00
greatroar
973fa921cb Test and document Restorer.VerifyFiles 2021-09-19 12:52:11 +02:00
Michael Eischer
e0d615c264 archiver: Fix TestArchiverAbortEarlyOnError test
This can be caused when the test has uploaded four blobs, then queues
two blobs for upload which are delayed. Then a seventh file can be
opened which lead to a test failure.
2021-09-12 22:17:17 +02:00
MichaelEischer
ef5672a902 Merge pull request #3509 from ajaspers/patch-1
Update instructions for reproducing build
2021-09-12 21:40:37 +02:00
ajaspers
c0eddc9969 Update instructions for reproducing build
Dependencies are fetched at build time and stored in the GOPATH. These paths end up being in the final binary.

Bump restic version to latest and go version to the 1.16.6, which was used to build restic 0.12.1.
2021-09-12 09:02:57 -07:00
Michael Eischer
fbb0e6499a ui: consolidate backup ui in ui/backup package 2021-09-12 16:20:15 +02:00
Michael Eischer
503d4c3e2f Add changelog 2021-09-12 16:00:49 +02:00
Michael Eischer
cccb0d4064 ui: assert that ProgressPrinter interface is implemented 2021-09-12 15:26:40 +02:00
Michael Eischer
a144c986f2 backup: Reenable JSON status updates with redirected output
After the refactoring status updates were no longer printed in quiet
mode or when the output is not an interactive terminal. However, the
JSON output is often piped to e.g. another program. Thus, don't set the
update frequency to 0 in that case. The status updates are still
disabled for backup --quiet.

This also reduces the status update frequency to 60fps compared to a
potentially much higher value before the refactoring.
2021-09-12 15:26:40 +02:00
Michael Eischer
d62bfed65d ui: move SetDryRun to ProgressReporter 2021-09-12 15:25:58 +02:00
Michael Eischer
77b129ec74 ui: remove unused Summary field 2021-09-12 15:25:58 +02:00
Michael Eischer
3024239e40 ui/json: restore summary output 2021-09-12 15:25:58 +02:00
Michael Eischer
5ccf583b8a ui: restore a few comments 2021-09-12 15:25:58 +02:00
Michael Eischer
80cbaf6d38 ui: Simplify ReportTotal methods 2021-09-12 15:25:58 +02:00
Dan Willoughby
448419990c Refactor backup progress
Move the shared logic into the progress

Allows logic to be shared with forth coming restore status
2021-09-12 15:25:58 +02:00
Andy
7baa9a570d document AWS_PROFILE support
Since restic (or whatever library it is using) seems to respect/use AWS_PROFILE, it's worth documenting this.
2021-09-10 14:06:46 +01:00
Alexander Neumann
bf9c8771a4 Merge pull request #3482 from MichaelEischer/changelog-3429
Add changelog for #3246 and #3429
2021-09-08 09:15:40 +02:00
Michael Eischer
5e84f38f31 Add changelog for #3246 2021-09-07 21:18:11 +02:00
Alexander Neumann
8fe122d675 Merge pull request #3488 from MichaelEischer/rebuild-broken-index
Fix `rebuild-index` for damaged index
2021-09-07 17:00:23 +02:00
Alexander Neumann
74c47f1f12 Merge pull request #3502 from restic/rawtaz-issue-519
doc: Add note about scheduling to backup section
2021-09-07 16:58:17 +02:00
MichaelEischer
fa5ca8af81 Merge pull request #2856 from aawsome/remove-readahead
Simplify cache logic
2021-09-04 20:25:49 +02:00
MichaelEischer
b45d88e124 Merge pull request #3496 from juergenhoetzel/imporove-snapshot-filter-warning-message
Improve snapshot filter warning message
2021-09-03 23:42:27 +02:00
MichaelEischer
bc4cbd775b Merge pull request #2880 from aawsome/enhance-recover
Improve recover command
2021-09-03 23:40:43 +02:00
MichaelEischer
a29777f467 Merge pull request #3501 from greatroar/printprogress
Streamline progress printing in cmd/restic
2021-09-03 23:34:36 +02:00
Alexander Weiss
bce87922c0 Improve recover
- only save directories not referenced by any snapshot
- dont't write empty snapshot
- use progress bar
2021-09-03 21:36:57 +02:00
Alexander Weiss
81876d5c1b Simplify cache logic 2021-09-03 21:01:00 +02:00
greatroar
7f0aa49f45 cmd/restic: Streamline progress printing
* PrintProgress no longer does unnecessary Sprintf calls, and performs
  fewer allocations in general
* newProgressMax's callback checks whether the terminal supports
  line updates once instead of once per call
* the callback looks up the terminal width once per call instead of
  twice (on Windows)
* the status shortening now uses the Unicode-aware version from
  internal/ui/termstatus (future-proofing)
2021-09-03 11:48:22 +02:00
greatroar
5aaa3e93c1 internal/ui/termstatus: Optimize and publish Truncate
name               old time/op  new time/op  delta
TruncateASCII-8     347ns ± 1%    69ns ± 1%  -80.02%  (p=0.000 n=9+10)
TruncateUnicode-8   447ns ± 3%   348ns ± 1%  -22.04%  (p=0.000 n=10+10)
2021-09-03 11:48:22 +02:00
MichaelEischer
ec2e3b260e Merge pull request #3499 from greatroar/wrappedconn-pointer
rclone: Return one fewer value from run
2021-08-31 21:48:36 +02:00
rawtaz
26914abe62 doc: Add note about scheduling to backup section
Explains that restic doesn't have built-in scheduling
and mentions a few keywords one can search for.
2021-08-29 22:03:22 +02:00
greatroar
950b818274 rclone: Return one fewer value from run 2021-08-26 18:12:08 +02:00
Juergen Hoetzel
defe19fdf6 Quote snapshot prefix in error string 2021-08-25 16:11:28 +02:00
Juergen Hoetzel
409e4936af Improve snapshot filter warning message
Include the root-cause why the snapshot prefix is ignored.
2021-08-25 15:46:21 +02:00
MichaelEischer
10b39d7591 Merge pull request #3487 from andreaso/rest-proto-readme-link
Fix REST protocol link in README file
2021-08-22 18:47:32 +02:00
Michael Eischer
194ed19557 Add changelog 2021-08-22 18:29:58 +02:00
Michael Eischer
877fc9f352 rebuild-index: test that invalid indexes are skipped and removed 2021-08-22 18:24:19 +02:00
Michael Eischer
64258a2c2a rebuild-index: Ignore broken index files
Previously, an e.g. truncated index could prevent rebuild-index from
working.
2021-08-22 18:23:47 +02:00
Andreas Olsson
c520672982 Fix REST protocol link in README file
1. All other document links goes to the _Read the Docs_ site
2. The GitHub reStructuredText renderer doesn't appear to do includes,
   making for a rather empty read.
2021-08-22 13:16:51 +02:00
Alexander Neumann
9374c3ce81 Merge pull request #3485 from lbausch/fix_typo
Fix typo in changelog
2021-08-21 13:16:20 +02:00
Lorenz Bausch
4d56b34096 Fix typo in changelog 2021-08-21 12:16:04 +02:00
Alexander Neumann
66382b2861 Update Go to 1.17 2021-08-17 21:38:34 +02:00
Michael Eischer
1fab5892b5 Add changelog for #3429 2021-08-16 17:42:16 +02:00
MichaelEischer
c898f7a6bf Merge pull request #3479 from BUFU1610/patch-1
fixed --keep-within-X options in example
2021-08-15 20:06:26 +02:00
BUFU
7659790923 fixed --keep-within-X options in example
The order of the words was wrong in the example, i.e. fixed to --keep-within-hourly from --keep-hourly-within.
2021-08-15 19:56:22 +02:00
MichaelEischer
ecf34783ef Merge pull request #3480 from MichaelEischer/fix-rest-tests
Fix rest backend tests
2021-08-15 18:59:02 +02:00
Michael Eischer
68370feeee backends: Remove TestSaveFilenames test
Filenames are expected to match the sha256 sum of the file content. This
rule is now enforced by the rest server thus making this test useless.
2021-08-15 18:24:16 +02:00
Michael Eischer
574c83e47f rest: Fix test to use paths which are the sha256 sum of the data 2021-08-15 18:19:43 +02:00
Michael Eischer
e6a5801155 rest: Fix test backend url
The rest config normally uses prepareURL to sanitize URLs and ensures
that the URL ends with a slash. However, the test used an URL without a
trailing slash, which after the rest server changes causes test
failures.
2021-08-15 18:16:17 +02:00
Alexander Neumann
d90efd7704 Fix test 2021-08-09 10:30:10 +02:00
Alexander Neumann
9fe5a87785 Merge pull request #3429 from MichaelEischer/safe-key-change
key: Check that new key works before deleting the old one
2021-08-09 10:07:15 +02:00
Alexander Neumann
7f1608dc77 Merge pull request #3246 from restic/content-hash-for-upload
Calculate content hashes during upload
2021-08-08 17:24:08 +02:00
Michael Eischer
f4c5dec05d backend: test that a wrong hash fails an upload 2021-08-04 22:17:46 +02:00
Michael Eischer
7c1903e1ee panic if hash returns an error
Add a sanity check that the interface contract is honoured.
2021-08-04 22:17:46 +02:00
Michael Eischer
51b7e3119b mem: calculate md5 content hash for uploads
The mem backend is primarily used for testing. This ensures that the
upload hash calculation gets appropriate test coverage.
2021-08-04 22:17:46 +02:00
Michael Eischer
a009b39e4c gs/swift: calculate md5 content hash for upload 2021-08-04 22:17:46 +02:00
Michael Eischer
1d3e99f475 azure: check upload using md5 content hash
For files below 256MB this uses the md5 hash calculated while assembling
the pack file. For larger files the hash for each 100MB part is
calculated on the fly. That hash is also reused as temporary filename.
As restic only uploads encrypted data which includes among others a
random initialization vector, the file hash shouldn't be susceptible to
md5 collision attacks (even though the algorithm is broken).
2021-08-04 22:17:46 +02:00
Michael Eischer
9aa2eff384 Add plumbing to calculate backend specific file hash for upload
This enables the backends to request the calculation of a
backend-specific hash. For the currently supported backends this will
always be MD5. The hash calculation happens as early as possible, for
pack files this is during assembly of the pack file. That way the hash
would even capture corruptions of the temporary pack file on disk.
2021-08-04 22:17:46 +02:00
Michael Eischer
ee2f14eaf0 s3: enable content hash calculation for uploads 2021-08-04 22:12:12 +02:00
MichaelEischer
553ea36ca6 Merge pull request #2838 from greatroar/cache-conflicts
Make cache concurrency-safe
2021-08-04 22:11:50 +02:00
greatroar
6586e90acf Modernize internal/cache error handling 2021-08-04 22:02:42 +02:00
greatroar
ea04f40eb3 Save cached files to a temporary location first 2021-08-04 22:02:42 +02:00
greatroar
f9b6f8fd45 Replace duplicate type checking in cache with a function 2021-08-04 22:02:42 +02:00
MichaelEischer
1b1a2115fa Merge pull request #3436 from greatroar/local-save-tmp
Save files under temporary name in local backend
2021-08-04 22:01:53 +02:00
Michael Eischer
65908647e3 Add changelog 2021-08-04 21:51:53 +02:00
greatroar
81e2499d19 Sync directory to get durable writes in local backend 2021-08-04 21:51:53 +02:00
greatroar
195a5cf996 Save files under temporary name in local backend
Fixes #3435.
2021-08-04 21:51:53 +02:00
MichaelEischer
bc97a3d1f9 Merge pull request #3300 from aawsome/backup-dryrun
backup: add --dry-run/-n flag
2021-08-04 21:50:38 +02:00
Michael Eischer
702cff636f Add use case to changelog 2021-08-04 21:19:29 +02:00
Alexander Weiss
780e11b7e2 Adapt changelog 2021-08-04 21:19:29 +02:00
erin
4126435663 resolve rawtaz's review comments
make majority of suggestions from review by @rawtaz verbatim, with one clarification on my part in changelog
2021-08-04 21:19:29 +02:00
Alexander Weiss
d107a2cfdf Separate dry run tests 2021-08-04 21:19:29 +02:00
Alexander Weiss
38a8a48a25 Simplify dry run backend 2021-08-04 21:19:29 +02:00
Ryan Hitchman
77bf148460 backup: add --dry-run/-n flag to show what would happen.
This can be used to check how large a backup is or validate exclusions.
It does not actually write any data to the underlying backend. This is
implemented as a simple overlay backend that accepts writes without
forwarding them, passes through reads, and generally does the minimal
necessary to pretend that progress is actually happening.

Fixes #1542

Example usage:

$ restic -vv --dry-run . | grep add
new       /changelog/unreleased/issue-1542, saved in 0.000s (350 B added)
modified  /cmd/restic/cmd_backup.go, saved in 0.000s (16.543 KiB added)
modified  /cmd/restic/global.go, saved in 0.000s (0 B added)
new       /internal/backend/dry/dry_backend_test.go, saved in 0.000s (3.866 KiB added)
new       /internal/backend/dry/dry_backend.go, saved in 0.000s (3.744 KiB added)
modified  /internal/backend/test/tests.go, saved in 0.000s (0 B added)
modified  /internal/repository/repository.go, saved in 0.000s (20.707 KiB added)
modified  /internal/ui/backup.go, saved in 0.000s (9.110 KiB added)
modified  /internal/ui/jsonstatus/status.go, saved in 0.001s (11.055 KiB added)
modified  /restic, saved in 0.131s (25.542 MiB added)
Would add to the repo: 25.892 MiB
2021-08-04 21:19:29 +02:00
MichaelEischer
533ac4fd95 Merge pull request #3467 from greatroar/wrappedconn-pointer
Use rclone.wrappedConn by pointer
2021-08-04 21:18:53 +02:00
Alexander Neumann
7049f1cbfc Set development version for 0.12.1 2021-08-03 11:45:39 +02:00
Alexander Neumann
dc7a8aab24 Add version for 0.12.1 2021-08-03 11:45:36 +02:00
Alexander Neumann
94983a1f36 Update manpages and auto-completion 2021-08-03 11:45:36 +02:00
Alexander Neumann
a92faca10e Generate CHANGELOG.md for 0.12.1 2021-08-03 11:45:19 +02:00
Alexander Neumann
b19cd8c50f Prepare changelog for 0.12.1 2021-08-03 11:45:19 +02:00
Alexander Neumann
b862732318 Merge pull request #3468 from restic/rework-changelog
Reword changelog entries
2021-08-03 10:06:56 +02:00
Leo R. Lundgren
cb844e7136 Polish changelog entries 2021-08-03 00:01:09 +02:00
Alexander Neumann
b7fe1fe6b4 Reword changelog entries 2021-08-03 00:01:09 +02:00
MichaelEischer
c98bbdcdbe Merge pull request #3457 from systemmonkey42/feature/untagged
Feature to match untagged snapshots only when listing or forgetting
2021-08-02 23:19:39 +02:00
David le Blanc
326fefcd80 Allow --tag and --keep-tag to match untagged snapshots 2021-08-02 23:06:20 +02:00
greatroar
fa3eed1998 Use rclone.wrappedConn by pointer
This shaves a kilobyte off the Linux binary by not generating a
non-pointer interface implementation.
2021-08-01 09:11:50 +02:00
MichaelEischer
5571c3f7fd Merge pull request #3453 from MichaelEischer/http2-zero-length-workaround
rest: Workaround Http2 zero-length reply bug
2021-07-31 20:30:06 +02:00
Alexander Neumann
d8ea10db8c rest: Rework handling HTTP2 zero-length replies bug
Add comment that the check is based on the stdlib HTTP2 client. Refactor
the checks into a function. Return an error if the value in the
Content-Length header cannot be parsed.
2021-07-31 17:12:24 +02:00
Alexander Neumann
77551597b2 Merge pull request #3416 from torfason/keep-hourly-within
Keep hourly within
2021-07-30 10:36:41 +02:00
Alexander Neumann
92f293cd0b Merge pull request #3427 from MichaelEischer/find-packs-from-index
find: List missing pack files based on the index
2021-07-30 10:33:02 +02:00
Magnus Thor Torfason
2081bd12fb forget: Ensure future snapshots do not affect --keep-within-*
Ensure that only snapshots made in the past are taken into account when running restic forget with the within switches (--keep-within, --keep-within- hourly, and friends)
2021-07-24 16:14:43 +00:00
Magnus Thor Torfason
74ebc650ab forget: Add --keep-within-hourly (and friends)
Allow keeping hourly/daily/weekly/monthly/yearly snapshots for a given time period.

This adds the following flags/parameters to restic forget:
  --keep-within-hourly duration
  --keep-within-daily duration
  --keep-within-weekly duration
  --keep-within-monthly duration
  --keep-within-yearly duration

Includes following changes:
  - Add tests for --keep-within-hourly (and friends)
  - Add documentation for --keep-within-hourly (and friends)
  - Add changelog for --keep-within-hourly (and friends)
2021-07-24 16:14:43 +00:00
Alexander Neumann
c707d71b72 Merge pull request #3401 from MichaelEischer/goroutine-shutdown-cleanups
Goroutine shutdown cleanups
2021-07-11 16:32:28 +02:00
Alexander Neumann
691866ce43 Merge pull request #3452 from MichaelEischer/add-s390x-releases
Add release binaries for linux/s390x
2021-07-11 16:30:25 +02:00
Alexander Neumann
efd918c59e Merge pull request #3454 from MichaelEischer/update-dependencies
Update dependencies
2021-07-11 16:29:31 +02:00
Michael Eischer
7d28006e2e Add changelog 2021-07-10 22:39:01 +02:00
Michael Eischer
0880afe67b Use our generate command instead of cobra's complete command 2021-07-10 19:44:18 +02:00
Michael Eischer
100baf74c0 Update cobra
The most noteworthy change seems to be

https://github.com/spf13/cobra/pull/1192
Have Cobra create 'completion' command automatically
2021-07-10 19:43:14 +02:00
Michael Eischer
c733ae6b16 Readd and update indirect dependencies
The azure-sdk-for-go is the only remaining module without a go.mod file.
Thus we only need indirect dependencies for it. Remove all other
indirect dependencies.
2021-07-10 19:21:43 +02:00
Michael Eischer
989b398fee Misc library updates
The most noteworthy change is the xattr update which includes
https://github.com/pkg/xattr/pull/54 that adds xattr support for solaris
and illumos
2021-07-10 19:04:47 +02:00
Michael Eischer
bbc8146934 Update minio/sha256-simd
Apparently the standard Go sha256 implementation is now faster than the
assembly implementation. The library now only adds support for SHA
extensions available in some processors.

See https://github.com/minio/sha256-simd/pull/57 for more details.
2021-07-10 18:34:16 +02:00
Michael Eischer
aa22ebac69 Update backend dependencies
Possibly interesting changes:

https://github.com/Azure/azure-sdk-for-go/pull/14521
Retry on HTTP client errors

https://github.com/minio/minio-go/pull/1452
fix: make sure getObject returns error on truncated responses

f5854403a9
http2: close Transport connection on write errors
2021-07-10 18:27:13 +02:00
Michael Eischer
097ed659b2 rest: test that zero-length replies over HTTP2 work correctly
The first test function ensures that the workaround works as expected.
And the second test function is intended to fail as soon as the issue
has been fixed in golang to allow us to eventually remove the
workaround.
2021-07-10 17:22:42 +02:00
Michael Eischer
185a55026b rest: workaround for HTTP2 zero-length replies bug
The golang http client does not return an error when a HTTP2 reply
includes a non-zero content length but does not return any data at all.
This scenario can occur e.g. when using rclone when a file stored in a
backend seems to be accessible but then fails to download.
2021-07-10 16:59:01 +02:00
Michael Eischer
495831d53c add release binaries for linux/s390x 2021-07-10 00:10:24 +02:00
Michael Eischer
3442dc87fb find: Address review comments 2021-07-06 21:04:34 +02:00
Michael Eischer
a81f34ae47 Add changelog 2021-07-06 21:04:34 +02:00
Michael Eischer
95b44490a0 find: search blob ids for pack in index if pack is missing
If a pack file is missing try to determine the contained pack ids based
on the repository index. This helps with assessing the damage to a
repository before running `rebuild-index`.
2021-07-06 21:04:34 +02:00
Michael Eischer
3caab3c7ac find: Print not found pack files 2021-07-06 21:04:34 +02:00
Michael Eischer
40745b4f82 find: stop file listing after resolving all pack files 2021-07-06 21:04:34 +02:00
Michael Eischer
6c01078f3d find: support resolving multiple pack ids to blobs
Just passing the list of blobs to packsToBlobs would also work in most
cases, however, it could cause unexpected results when multiple pack
files have the same prefix. Forget found prefixes to prevent this.
2021-07-06 21:04:34 +02:00
Leo R. Lundgren
790294dc26 contributing: Change freenode to libera 2021-07-04 11:05:36 +02:00
Alexander Neumann
30d968b0e4 Merge pull request #3449 from MichaelEischer/fix-restore-retries
restore: Correctly handle partial pack download errors
2021-07-04 10:55:09 +02:00
Michael Eischer
43b82d69b4 Add changelog 2021-06-29 21:27:00 +02:00
Michael Eischer
bd316d3893 restore: Test partial pack downloads in filerestorer 2021-06-29 21:11:30 +02:00
Michael Eischer
e8bbb05328 restore: Correctly handle partial pack download errors
Failed pack/blob downloads should be retried. For blobs that fail
decryption assume that the pack file is really damaged and try to
restore the remaining blobs.
2021-06-29 20:54:16 +02:00
rawtaz
58be5172ff Merge pull request #3437 from MichaelEischer/fix-prune-output
prune: Add missing newlines in error descriptions
2021-06-28 00:56:51 +02:00
MichaelEischer
cb6fd281a0 Merge pull request #3442 from restic/rawtaz-doc-verbose
doc: Correct position of --verbose in backup docs
2021-06-27 18:19:37 +02:00
rawtaz
eb61de7b3a Merge pull request #3287 from tjrana/clean-up-pr-template
Clean up PR template
2021-06-25 21:06:50 +02:00
rawtaz
98a88b483d doc: Correct position of --verbose in backup docs 2021-06-25 19:26:46 +02:00
MichaelEischer
6a4c1ed50d Merge pull request #3438 from MichaelEischer/fast-fuse-changelog
Add changelog for #3426
2021-06-20 15:05:19 +02:00
Michael Eischer
409306db2b Add changelog for #3426 2021-06-20 14:46:37 +02:00
Michael Eischer
aad8864835 prune: Add missing newlines in error descriptions 2021-06-20 14:25:40 +02:00
MichaelEischer
c1eb7ac1a1 Merge pull request #3420 from greatroar/local-errs
Modernize error handling in local backend
2021-06-20 14:20:40 +02:00
greatroar
e5f0f67ba0 Modernize error handling in local backend
* Stop prepending the operation name: it's already part of os.PathError,
  leading to repetitive errors like "Chmod: chmod /foo/bar: operation not
  permitted".

* Use errors.Is to check for specific errors.
2021-06-18 11:13:27 +02:00
Alexander Neumann
45eb30388f Merge pull request #3426 from MichaelEischer/fast-fuse
mount: enable fuse readahead
2021-06-14 10:27:41 +02:00
Michael Eischer
454b6d608e key: Add test that failed key saves don't break the repository 2021-06-13 13:46:48 +02:00
TJ Rana
a61a0255a8 Modify pull request template 2021-06-12 19:34:46 -04:00
Michael Eischer
6add186867 key: Check that a new key file actually works 2021-06-12 23:09:19 +02:00
MichaelEischer
a476752962 Merge pull request #3421 from greatroar/s3-fileinfo
Return s3.fileInfos by pointer
2021-06-12 18:55:18 +02:00
MichaelEischer
e8d20ea32c Merge pull request #3409 from greatroar/lchown-mknod
Make restic.{lchown,mknod} regular functions
2021-06-12 18:22:38 +02:00
Michael Eischer
fe43f53528 mount: enable fuse readahead
Apparently readahead was disabled by default. Enable readahead with the
Linux default size of 128kB. Larger values seem to have no effect.
This can speed up reading from the fuse mount by at least factor 5.

Speedup for a 1G random file stored in a local repository:
(Only one result shown, but times were quite stable, restarted restic
after each command)
$ dd if=/dev/urandom bs=1M count=1024 of=rand
$ shasum -a 256 tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  tmp/rand

before: $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  fuse/snapshots/latest/tmp/rand

real    0m18.294s
user    0m4.522s
sys     0m3.305s

before: $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real    0m14.924s
user    0m0.000s
sys     0m4.625s

after:  $ time shasum -a 256 fuse/snapshots/latest/tmp/rand
75dd9b374e712577d64672a05b8ceee40dfc45dce6321082d2c2fd51d60c6c2d  fuse/snapshots/latest/tmp/rand

real    0m6.106s
user    0m3.115s
sys     0m0.182s

after:  $ time cat fuse/snapshots/latest/tmp/rand > /dev/null
real    0m3.096s
user    0m0.017s
sys     0m0.241s
2021-06-12 17:07:30 +02:00
greatroar
0d4f16b6ba Return s3.fileInfos by pointer
Since the fileInfos are returned in a []interface, they're already
allocated on the heap. Making them pointers explicitly means the
compiler doesn't need to generate fileInfo and *fileInfo versions of the
methods on this type. The binary becomes about 7KiB smaller on
Linux/amd64.
2021-06-07 19:48:43 +02:00
greatroar
0666c4d244 Make restic.{lchown,mknod} regular functions
This way, they can be inlined and dead code can be removed on Windows.
Also fixed some comments.
2021-05-27 22:51:40 +02:00
rawtaz
fdbd65485e Merge pull request #3402 from MichaelEischer/misc-fixes
Various small code cleanups
2021-05-24 11:30:31 +02:00
rawtaz
2daf033156 Merge pull request #3403 from MichaelEischer/fast-cat
cat: only load index if really necessary
2021-05-24 11:02:10 +02:00
MichaelEischer
5dad45f005 Merge pull request #3312 from soraxas/master
Bump cobra and add completions for fish
2021-05-23 13:58:27 +02:00
MichaelEischer
7eb6372123 Merge pull request #3257 from greatroar/ls-json-empty
ls: print "size":0 for empty files in JSON
2021-05-23 13:49:56 +02:00
Michael Eischer
61b368ddea cat: only load index if really necessary 2021-05-23 13:11:55 +02:00
Michael Eischer
fd8bce8184 backup: cleanly shutdown goroutines on error 2021-05-23 13:02:44 +02:00
greatroar
d7322a5f36 restic ls --json: print "size":0 for empty files 2021-05-21 21:06:00 +02:00
Tin Lai
9cc1ecdd45 bump cobra and add completions for fish
Signed-off-by: Tin Lai <oscar@tinyiu.com>
2021-05-21 13:47:52 +10:00
Alexander Neumann
af3de702c7 Merge pull request #3332 from restic/debug-1999
Merge `debug examine` to salvage damaged pack files
2021-05-18 09:38:40 +02:00
Alexander Neumann
226cd8d4d1 Merge pull request #3310 from MichaelEischer/copy-unstable-tree
`copy` raw tree blobs
2021-05-18 09:36:51 +02:00
Alexander Neumann
4cabad8c34 Merge pull request #3325 from MichaelEischer/fix-mintty-output
Fix windows terminal output for mintty
2021-05-18 09:29:24 +02:00
Michael Eischer
cf92c58460 Properly wrap errors in readerat helper 2021-05-17 21:08:23 +02:00
Michael Eischer
5767c65c62 delete: properly close fileChan if context is canceled 2021-05-17 21:05:54 +02:00
Michael Eischer
75c990504d azure/gs: Fix default value in connections help text 2021-05-17 20:56:51 +02:00
Michael Eischer
5a87a0ba0a find: use Str() to format short ids 2021-05-17 20:56:49 +02:00
Alexander Neumann
38ccddc84f Merge pull request #3399 from oli-h/oli-h-patch-060_forget.rst
Sudden find: fix within first "forget" example
2021-05-17 11:39:36 +02:00
Alexander Neumann
d7b5061aa5 Merge pull request #3394 from MichaelEischer/apple-m1-build
Build darwin/arm64 binaries for Apple M1
2021-05-17 09:56:44 +02:00
rawtaz
27141ae87f Merge pull request #2999 from restic/issue-2109
backup: Improve wording for --one-file-system description
2021-05-15 00:23:36 +02:00
Leo R. Lundgren
90d75651e6 backup: Improve wording for --one-file-system description 2021-05-15 00:06:27 +02:00
rawtaz
2a915069a8 Merge pull request #3393 from MichaelEischer/fix-filter-crash
filter: Fix crash for '**' pattern
2021-05-14 23:58:44 +02:00
Michael Eischer
55bea6e7a6 filter: Fix crash for '**' pattern 2021-05-14 23:50:31 +02:00
Michael Eischer
af6f6fba15 Build darwin/arm64 binaries for Apple M1 2021-05-14 23:04:45 +02:00
MichaelEischer
6d8ceefd67 Merge pull request #3373 from greatroar/simplify-limiter
Simplify internal/limiter
2021-05-14 20:27:31 +02:00
MichaelEischer
7349c6d338 Merge pull request #3167 from renard/limit-snapshots-list
Add option to limit snapshots list
2021-05-13 20:26:33 +02:00
Sébastien Gross
2a92b68e65 cmd/snapshots: Add option to limit snapshots list
This patch adds a `--latest` option to limit snapshots list to the n
last snapshots. It is very similar to the `--last` one but does not
limit to one entry. It also deprecates the `--last` flag usage in
favor of `--latest 1`

Output example:

    $ restic snapshots --latest 2
    repository 0d3eb989 opened successfully, password is correct
    ID        Time                 Host        Tags        Paths
    ------------------------------------------------------------
    5a33bdcc  2020-12-14 12:30:00  local                   /home
    73887d8e  2020-12-15 12:30:00  local                   /home
    ------------------------------------------------------------
    2 snapshots

Signed-off-by: Sébastien Gross <seb•ɑƬ•chezwam•ɖɵʈ•org>
2021-05-13 20:18:23 +02:00
MichaelEischer
64b00d28b1 Merge pull request #3345 from greatroar/sftp-enospc
Check for ENOSPC and remove broken files in SFTP
2021-05-13 20:09:38 +02:00
oli-h
23f9cb838d Sudden find: fix within first "forget" example
I think this was a typo or copy/paste problem when created the doc.
Feel free to merge. Cheers
2021-05-10 08:55:42 +02:00
rawtaz
01261770bb Merge pull request #3386 from adrian5/doc-change
doc: Add direct link to GitHub issue
2021-05-09 00:12:21 +02:00
adrian5
a0f1c74000 doc: Add direct link to GitHub issue 2021-05-08 21:13:11 +00:00
Alexander Neumann
be6fc02c04 Merge pull request #3376 from restic/rawtaz-doc-exclude
doc: Polish exclude file documentation
2021-04-27 19:25:00 +02:00
Alexander Neumann
3ce5544796 Merge pull request #3321 from restic/doc-files-from
doc: Improve docs for --files-from et al
2021-04-27 19:15:31 +02:00
rawtaz
556caa326f doc: Polish exclude file documentation 2021-04-25 20:36:11 +02:00
greatroar
ae170e2b38 Simplify internal/limiter 2021-04-24 11:54:43 +02:00
rawtaz
f7316cea07 Merge pull request #3371 from cyounkins/max_unused_default
doc: default for --max-unused
2021-04-23 03:18:05 +02:00
Craig Younkins
32a84ab3e4 doc: default for --max-unused 2021-04-22 21:12:37 -04:00
rawtaz
6c7eabf08c Merge pull request #3363 from davegallant/design-doc-grammar-fix
doc: Fix grammar in design.rst
2021-04-15 11:01:38 +02:00
Dave Gallant
4aaf10c356 doc: Fix grammar in design.rst 2021-04-14 17:25:53 -04:00
Michael Eischer
f65e1211d9 Add changelog entry 2021-04-11 20:02:09 +02:00
Michael Eischer
7cb8ea69ba Add test to mintty pipe detection 2021-04-11 20:02:09 +02:00
Michael Eischer
80564a9bc9 Properly detect mintty output redirection
mintty on windows always uses pipes to connect stdout between processes
and for the terminal output. The previous implementation always assumed
that stdout connected to a pipe means that stdout is displayed on a
mintty terminal. However, this detection breaks when using pipes to
connect processes and for powershell which uses pipes when redirecting
to a file.

Now the pipe filename is queried and matched against the pattern used by
msys / cygwin when connected to the terminal. In all other cases assume
that a pipe is just a regular pipe.
2021-04-11 20:02:09 +02:00
Michael Eischer
5e6af77b7a Unify interactive terminal detection code
Previously the progress bar / status update interval used
stdoutIsTerminal to determine whether it is possible to update the
progress bar or not. However, its implementation differed from the
detection within the backup command which included additional checks to
detect the presence of mintty on Windows. mintty behaves like a terminal
but uses pipes for communication.

This adds stdoutCanUpdateStatus() which calls the same terminal detection
code used by backup. This ensures that all commands consistently switch
between interactive and non-interactive terminal mode.

stdoutIsTerminal() now also returns true whenever stdoutCanUpdateStatus()
does so. This is required to properly handle the special case of mintty.
2021-04-11 20:02:09 +02:00
MichaelEischer
cc254dfefe Merge pull request #3362 from greatroar/darwin-preallocate
Use FcntlFstore to preallocate on Mac
2021-04-10 22:47:56 +02:00
greatroar
23531be272 Use FcntlFstore to preallocate on Mac 2021-04-10 16:54:07 +02:00
Alexander Neumann
b922fc851b Merge pull request #3356 from restic/rawtaz-doc-dollar-sign
doc: Clarify dollar sign expansion in exclude files
2021-04-09 12:30:54 +02:00
rawtaz
ccfd5f1d4a Merge pull request #3298 from jniggemann/patch-1
doc: Add note about bash completion path
2021-04-09 00:17:17 +02:00
Jan Niggemann
9a1f685179 doc: Add note about bash completion path 2021-04-08 23:53:38 +02:00
rawtaz
b5e40b370c doc: Clarify dollar sign expansion in exclude files 2021-04-08 23:46:53 +02:00
rawtaz
74c0607c92 Merge pull request #3319 from MichaelEischer/skip-prealloc-test
restorer: Skip preallocate test if not supported by the filesystem
2021-04-07 18:59:06 +02:00
Leo R. Lundgren
5861bb031c doc: Improve docs for --files-from et al 2021-04-07 18:31:46 +02:00
rawtaz
c2569ff923 Merge pull request #3354 from greatroar/filechange-docs
docs: on Windows, the filesize must match for "unchanged"
2021-04-07 15:29:46 +02:00
greatroar
ecbe7f3a99 Backup docs: on Windows, the filesize must match for "unchanged" 2021-04-04 17:01:48 +02:00
Michael Eischer
88731d8c28 Add changelog 2021-03-27 17:08:35 +01:00
greatroar
dc88ca79b6 Handle lack of space and remove broken files in SFTP backend 2021-03-27 15:02:14 +01:00
MichaelEischer
efb10b3c40 Merge pull request #3347 from jazcap53/fix_ambiguous_warning
Change ambiguous Warning message
2021-03-27 12:54:07 +01:00
MichaelEischer
d456437ad1 Merge pull request #3343 from hluup/3334
Display 'created new cache in ' message only if output is terminal
2021-03-27 12:43:37 +01:00
MichaelEischer
4c61825249 Merge pull request #3335 from cyounkins/patch-1
prune --max-unused does not limit metadata repacking
2021-03-27 12:40:35 +01:00
Andrew Jarcho
9f44129c2f Change ambiguous Warning message
Fixes #3346

On branch fix_ambiguous_warning
Changes to be committed:
    modified:   cmd_backup.go
2021-03-24 10:44:47 -04:00
Hendrik Luup
5592c17e4a Display 'created new cache in ' message only if output is terminal 2021-03-22 11:31:08 +02:00
Craig Younkins
aa0a7b78a8 doc: prune --max-unused unlimited will still repack metadata
Adding minor clarification to documentation of `prune --max-unused unlimited` to indicate metadata will still be repacked. The referenced PR indicates `max-repack-size` CAN limit metadata repacking.

Ref: https://forum.restic.net/t/max-unused-unlimited-still-repacks/3661
Ref: https://github.com/restic/restic/pull/3228
2021-03-21 13:19:09 -04:00
Alexander Neumann
88a23521dd Merge pull request #3327 from MichaelEischer/fix-s3-sanity-check
s3: Fix sanity check
2021-03-11 13:13:46 +01:00
MichaelEischer
e678acafcf Merge pull request #3331 from greatroar/upgrade-sftp
Upgrade pkg/sftp to 1.13
2021-03-10 22:27:32 +01:00
Michael Eischer
54d58edacc debug: fix usage for examine command 2021-03-10 22:22:33 +01:00
Michael Eischer
5975ed61f3 debug: fix linter warning 2021-03-10 21:22:53 +01:00
Michael Eischer
dc62ec5933 debug: use Printf/Warnf for output 2021-03-10 21:20:21 +01:00
Michael Eischer
547d9b384d debug: cleanup repair code 2021-03-10 21:15:38 +01:00
greatroar
187a77fb27 Upgrade pkg/sftp to 1.13 and simplify SFTP.IsNotExist 2021-03-10 21:05:24 +01:00
Michael Eischer
fa7b9d5dfe debug: Cleanup pack size checks 2021-03-10 20:57:14 +01:00
Michael Eischer
6774fc6454 debug: check arguments and cleanup error handling 2021-03-10 20:43:22 +01:00
Michael Eischer
096f15db5c debug: extract examinePack function 2021-03-10 20:21:05 +01:00
Michael Eischer
84491ff40b debug: store repaired and correct blobs 2021-03-10 20:03:44 +01:00
Alexander Neumann
b3c3121622 debug: Save raw decrypt (disregarding signature) 2021-03-10 20:03:44 +01:00
Alexander Neumann
ce4b6d0874 examine: add byte repair mode 2021-03-10 20:03:44 +01:00
Alexander Neumann
52061e817c debug: fix percentage 2021-03-10 20:03:44 +01:00
Alexander Neumann
133ac42a0b debug: check file content hash 2021-03-10 20:03:44 +01:00
Alexander Neumann
90f975fa1c debug: make output less verbose 2021-03-10 20:03:44 +01:00
Alexander Neumann
086993bae1 debug: check packs not in index, implement repair 2021-03-10 20:03:44 +01:00
Alexander Neumann
d6f78163d4 Add 'debug examine' command to debug #1999 2021-03-10 20:03:44 +01:00
MichaelEischer
c9b4fadd91 Merge pull request #3294 from Achilleshiel/fix-copy-repofile
Add RepositoryFile2 Option for secondary repository
2021-03-08 22:50:31 +01:00
Michael Eischer
da458a55db Cleanup comments in secondary repo test 2021-03-08 22:41:13 +01:00
Wouter Horlings
9ccdba9df6 Add repositoryFile2 option
The `init` and `copy` commands can now use `--repository-file2` flag and
the `$RESTIC_REPOSITORY_FILE2` environment variable.

This also fixes the conflict with the `--repository-file` and `--repo2`
flag.

Tests are added for the initSecondaryGlobalOpts function.

This adds a NOK function to the test helper functions. This NOK tests if
err is not nil, and otherwise fail the test.

With the NOK function a couple of sad paths are tested in the
initSecondaryGlobalOpts function.

In total the tests checks wether the following are passed correct:
   - Password
   - PasswordFile
   - Repo
   - RepositoryFile

The following situation must return an error to pass the test:
   - no Repo or RepositoryFile defined
   - Repo and RepositoryFile defined both
2021-03-08 22:41:13 +01:00
MichaelEischer
a0f9d73d44 Merge pull request #3305 from juergenhoetzel/terminal-race-condition
backup: In case of error also Wait() for terminal goroutine
2021-03-08 22:20:18 +01:00
Michael Eischer
427781928f copy: test that trees with unstable json encoding are properly copied 2021-03-08 22:16:48 +01:00
Michael Eischer
2fc7abac35 copy: copy raw bytes of tree blobs
This avoids problems when for some reason the JSON encoding changes.
This also ensures forward compatibility with future restic versions
which might e.g. add new fields to the tree metadata.
2021-03-08 22:16:48 +01:00
Michael Eischer
f9c581f219 Tweak changelog 2021-03-08 22:11:34 +01:00
Juergen Hoetzel
18fccb5995 backup: In case of error also Wait() for terminal goroutine
This prevents a race condition where the final summary message can get lost.
2021-03-08 22:11:34 +01:00
Michael Eischer
2a9f0f19b6 s3: Fix sanity check
The sanity check shouldn't replace the error message if there is already
one.
2021-03-08 20:23:57 +01:00
rawtaz
d686fa25de Merge pull request #3323 from restic/doc-setcap
doc: Clarify setcap applying only to current binary
2021-03-08 19:32:11 +01:00
rawtaz
f000f41c91 doc: Clarify setcap applying only to current binary 2021-03-08 19:21:06 +01:00
MichaelEischer
64fe733fa0 Merge pull request #3320 from MichaelEischer/fix-ci
Update to Go 1.16 and pin go version used for golangcil-lint
2021-03-08 19:18:10 +01:00
Michael Eischer
781378a65e Bump go version to 1.16 2021-03-04 23:24:26 +01:00
Michael Eischer
64a9272c9a go mod tidy for go 1.16 2021-03-04 23:18:53 +01:00
Michael Eischer
ee89e33f12 CI: Pin go version used for golint to 1.15.x
This prevents unexpected lint failures when a new go version is
released.
2021-03-04 23:12:48 +01:00
Michael Eischer
f0e9068ef2 go mod tidy for go 1.16 2021-03-04 22:56:08 +01:00
Michael Eischer
f70aca6f6f restorer: Skip preallocate test if not supported by the filesystem 2021-03-04 20:33:46 +01:00
MichaelEischer
814a399e4c Merge pull request #3283 from dennypage/master
Treat an empty password as a fatal error for repository init.
2021-02-28 00:49:21 +01:00
Michael Eischer
13730e3844 Add link to pr to changelog 2021-02-28 00:40:52 +01:00
Denny Page
0f41e99ea7 Treat an empty password as a fatal error for repository init. 2021-02-28 00:40:00 +01:00
MichaelEischer
6712c6de73 Merge pull request #3309 from MichaelEischer/fix-random-check-crash
check: Fix crash of --read-data-subset=x% on empty repository
2021-02-27 16:24:56 +01:00
Michael Eischer
9e852af5be check: Fix crash of --read-data-subset=x% on empty repository
Rounding up to at least one pack file should only be done if at least a
pack file exists.
2021-02-27 16:05:36 +01:00
MichaelEischer
4baebdc6f5 Merge pull request #3308 from MichaelEischer/fix-not-a-directory
local: Ignore files in intermediate folders
2021-02-27 15:34:36 +01:00
Michael Eischer
a293fd9aef local: Ignore files in intermediate folders
For example the data folder of a repository might contain hidden files
which caused list operations to fail.
2021-02-27 13:47:55 +01:00
TJ Rana
91e8f0e486 Clean up writing style 2021-02-16 22:19:11 -05:00
TJ Rana
d7d562b287 Clarify checklist items 2021-02-16 22:11:22 -05:00
TJ Rana
76e1ba6fd0 Clean up pull request template 2021-02-16 22:03:33 -05:00
rawtaz
8eb6a5805b Merge pull request #3286 from MichaelEischer/fix-quiet-backup
backup: Correctly handle --quiet flag
2021-02-15 22:35:04 +01:00
Michael Eischer
65bd2a9a49 backup: Correctly handle --quiet flag
The quiet flag changes the backup output to assume a non-interactive
terminal. However, the output progress interval was not set to 0 by
default.
2021-02-15 22:14:58 +01:00
Alexander Neumann
12f0ccc237 helpers: Also push versioned image to Docker 2021-02-15 20:08:40 +01:00
Alexander Neumann
bb53fcfc0d Set development version for 0.12.0 2021-02-14 11:44:24 +01:00
Alexander Neumann
27f241334e Add version for 0.12.0 2021-02-14 11:44:20 +01:00
Alexander Neumann
4e99a3d650 Update manpages and auto-completion 2021-02-14 11:44:20 +01:00
Alexander Neumann
1cb1cd6f44 Generate CHANGELOG.md for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann
1a34260cf0 Prepare changelog for 0.12.0 2021-02-14 11:44:00 +01:00
Alexander Neumann
13d52c88fb Merge pull request #3282 from restic/changelog-unreleased
changelog: Correct and slightly polish some unreleased changelog files
2021-02-14 10:49:33 +01:00
Leo R. Lundgren
4b5ca1e914 changelog: Correct and slightly polish some unreleased changelog files 2021-02-14 00:57:54 +01:00
MichaelEischer
917f5b910a Merge pull request #3270 from restic/fix-ls-json
ls: Check for non-nil error before calling panic()
2021-02-09 22:05:04 +01:00
Alexander Neumann
c0f2c1d871 ls: Check for non-nil error before calling panic() 2021-02-07 21:12:54 +01:00
Alexander Neumann
9985368d46 Merge pull request #3255 from MichaelEischer/restorer-check-error
restorer: Check dropped error
2021-02-03 16:10:46 +01:00
Alexander Neumann
2dd592a06c Merge pull request #3254 from lorenz/allow-http2
Allow HTTP/2
2021-02-02 21:08:20 +01:00
Lorenz Brun
362338dd60 Add changelog entry 2021-02-01 20:20:34 +01:00
MichaelEischer
6ac032be64 Merge pull request #3256 from aawsome/rebuild-index-bar-done
rebuild-index: Add missing bar.Done()
2021-01-31 18:48:14 +01:00
MichaelEischer
0ce05d5725 Merge pull request #3236 from Wouter0100/patch-1
doc: Note only path-style URL support for S3
2021-01-31 18:44:58 +01:00
Michael Eischer
0aed8d47d7 docs: Fix typo in S3 URL support 2021-01-31 18:33:15 +01:00
Alexander Weiss
39a26066f7 rebuild-index: add missing bar.Done() 2021-01-31 18:28:02 +01:00
Michael Eischer
47faf69230 restorer: Check dropped error 2021-01-31 18:06:28 +01:00
MichaelEischer
b3dc127af5 Merge pull request #3207 from aawsome/filerestorer-fix-errorhandling
Fix error handling in filerestorer and fix #3166
2021-01-31 17:47:46 +01:00
Wouter van Os
8442c43209 doc: Update wording for S3 path-only requirement 2021-01-31 15:10:03 +01:00
Alexander Weiss
6e942693ba Fix #3166 2021-01-31 14:22:57 +01:00
Alexander Weiss
5e22ae10f1 Add error handling for fileRestorer 2021-01-31 14:22:57 +01:00
Alexander Weiss
573221aa40 Add failing test for fileRestorer 2021-01-31 13:40:42 +01:00
MichaelEischer
b8550a21f2 Merge pull request #3253 from restic/improve-prune-error
prune: Improve error message for missing files
2021-01-31 12:03:20 +01:00
Alexander Neumann
027a51529d prune: Improve error message for missing files
This commit changes the error message so that a list of file names is
printed. Before, just the raw map was printed, which is not a great user
interface.
2021-01-31 11:31:27 +01:00
Lorenz Brun
5427119205 Allow HTTP/2 2021-01-31 02:44:30 +01:00
Alexander Neumann
f647614e24 Merge pull request #3248 from MichaelEischer/backend-cleanups
Backend code and test cleanups
2021-01-30 21:37:33 +01:00
Michael Eischer
e0867c9682 backend: try to cleanup test leftovers 2021-01-30 21:23:20 +01:00
Michael Eischer
f740b2fb23 mem: check upload length before storing upload 2021-01-30 21:23:20 +01:00
Alexander Neumann
0e5f2fff71 Merge pull request #3243 from restic/fix-scanner-overlap
backup: Fix total size for overlapping targets
2021-01-30 21:17:21 +01:00
MichaelEischer
99228be623 Merge pull request #3250 from restic/add-golangci-lint-config
Add golangci lint config, add many error checks
2021-01-30 21:09:38 +01:00
Alexander Neumann
04ca69cc78 Address issues reported by golint 2021-01-30 20:45:57 +01:00
Alexander Neumann
f867e65bcd Fix issues reported by staticcheck 2021-01-30 20:43:53 +01:00
Alexander Neumann
a00e27adf6 Add entry to changelog 2021-01-30 20:22:25 +01:00
Alexander Neumann
0858fbf6aa Add more error handling 2021-01-30 20:19:47 +01:00
Alexander Neumann
aef3658a5f Address review comments 2021-01-30 20:02:37 +01:00
Alexander Neumann
200f09522d Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann
cbd88c457a backup: Improve error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann
1a0eb05bfa errcheck: Add more error checks 2021-01-30 20:02:37 +01:00
Alexander Neumann
3c753c071c errcheck: More error handling 2021-01-30 20:02:37 +01:00
Alexander Neumann
16313bfcc9 errcheck: Add error check for MergeFinalIndexes() 2021-01-30 20:02:37 +01:00
Alexander Neumann
75f53955ee errcheck: Add error checks
Most added checks are straight forward.
2021-01-30 20:02:37 +01:00
Alexander Neumann
1632a84e7b Add a section explaining golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann
b3d5bf7c99 Update golangci-lint version 2021-01-30 20:02:37 +01:00
Alexander Neumann
57627a307f Add config for golangci-lint 2021-01-30 20:02:37 +01:00
Alexander Neumann
6ab7d49a03 Merge pull request #3251 from MichaelEischer/rest-dropped-error
rest: handle dropped error in save operation
2021-01-30 19:44:50 +01:00
Michael Eischer
a53778cd83 rest: handle dropped error in save operation 2021-01-30 19:25:04 +01:00
Alexander Neumann
dd94efb307 Merge pull request #3249 from MichaelEischer/fix-dropped-gs-error
gs: Don't drop error when finishing upload
2021-01-30 16:24:57 +01:00
Michael Eischer
8a486eafed gs: Don't drop error when finishing upload
The error returned when finishing the upload of an object was dropped.
This could cause silent upload failures and thus data loss in certain
cases. When a MD5 hash for the uploaded blob is specified, a wrong
hash/damaged upload would return its error via the Close() whose error
was dropped.
2021-01-30 13:31:32 +01:00
MichaelEischer
4d576c2f79 Merge pull request #3217 from M4a1x/read-data-subset-fix
Fix missing rand seed for restic check --read-data-subset=x%
2021-01-29 23:34:13 +01:00
Max Stabel
f9e1fa26ff Fix missing rand seed for restic check --read-data-subset=x% 2021-01-29 23:18:35 +01:00
MichaelEischer
fb3cf3f885 Merge pull request #3245 from aawsome/prune-fix-statistics-cacheable
prune: Fix statistics for --repack-cacheable-only
2021-01-29 23:14:49 +01:00
Alexander Weiss
e08e65dc30 prune: Simplify logic selecting packs to repack 2021-01-29 22:27:22 +01:00
Alexander Weiss
daeb4cdf8f prune: Fix statistics for --repack-cacheable-only 2021-01-29 22:27:22 +01:00
Alexander Neumann
cdd704920d azure: Pass data length to Azure libray
The azureAdapter was used directly without a pointer, but the Len()
method was only defined with a pointer receiver (which means Len() is
not present on a azureAdapter{}, only on a pointer to it).
2021-01-29 21:08:41 +01:00
Alexander Neumann
bbdf18c4a2 Merge pull request #3176 from MichaelEischer/backend-content-length
Pass upload size to backends and sanity check it
2021-01-29 20:33:44 +01:00
Michael Eischer
1f583b3d8e backend: test that incomplete uploads fail 2021-01-29 13:51:53 +01:00
Michael Eischer
c73316a111 backends: add sanity check for the uploaded file size
Bugs in the error handling while uploading a file to the backend could
cause incomplete files, e.g. https://github.com/golang/go/issues/42400
which could affect the local backend.

Proactively add sanity checks which will treat an upload as failed if
the reported upload size does not match the actual file size.
2021-01-29 13:51:51 +01:00
Michael Eischer
4526d5d197 swift: explicitly pass upload size to library
This allows properly setting the content-length which could help the
server-side to detect incomplete uploads.
2021-01-29 13:50:46 +01:00
Michael Eischer
dca9b6f5db azure: explicitly pass upload size
Previously the fallback from the azure library was to read the whole
blob into memory and use that to determine the upload size.
2021-01-29 13:50:46 +01:00
Alexander Neumann
a16ce65295 Merge pull request #3244 from MichaelEischer/better-damage-reports
Print more details about possible repository damages
2021-01-29 11:45:45 +01:00
Alexander Neumann
5c41120c70 Add entry to changelog 2021-01-29 11:31:36 +01:00
Alexander Neumann
5c617859ab backup/scanner: Fix total size for overlapping targets
Before, the scanner would could files twice if they were included in the
list of backup targets twice, e.g. `restic backup foo foo/bar` would
could the file `foo/bar` twice.

This commit uses the tree structure from the archiver to run the
scanner, so both parts see the same files.
2021-01-29 11:31:36 +01:00
Alexander Neumann
81211750ba archiver/tree: Introduce functions Leaf() and NodeNames() 2021-01-29 11:11:28 +01:00
rawtaz
de7e3a0648 Merge pull request #2823 from greatroar/trust-mtime
Add --ignore-ctime flag to backup and document change detection
2021-01-29 00:02:19 +01:00
greatroar
6bd8a2faaa backup: Add --ignore-ctime option and document change detection 2021-01-28 23:42:10 +01:00
Michael Eischer
58b5679f14 prune: reword missing blobs error
The previous wording could be understood such that the prune run did
damage the repository.
2021-01-28 21:48:24 +01:00
Michael Eischer
7b8886c052 prune: report missing but unneeded pack files
This indicates a damaged repository so add some output to help with
debugging.
2021-01-28 21:46:01 +01:00
Michael Eischer
ff95999246 rebuild-index: report added/removed/reindexed files
This should help with investigating missing pack files.
2021-01-28 21:46:01 +01:00
Michael Eischer
b71c52797a find: correctly expand multiple blob ids
For example `restic find --show-pack-id --blob f78dc991 5b9e4366 ddd8c7d4`
would previously only expand one blob if all of them belong to the same
file.
2021-01-28 21:21:54 +01:00
MichaelEischer
82140967d3 Merge pull request #3228 from aawsome/prune-all-trees
prune: Remove all unused trees
2021-01-28 21:04:27 +01:00
MichaelEischer
43cb26010a Merge pull request #3242 from greatroar/fprintln
internal/ui/termstatus: Use Fprintln to get a newline
2021-01-28 20:34:06 +01:00
MichaelEischer
35033d9b79 Merge pull request #3177 from MichaelEischer/fix-2759
prune: don't print stacktrace on console
2021-01-28 20:28:45 +01:00
Alexander Neumann
84822d44d4 Merge pull request #2536 from MatthewVance/threat-model
Update threat model
2021-01-28 14:26:03 +01:00
Matthew Vance
58c7f4694d Update threat model 2021-01-28 14:20:42 +01:00
Alexander Neumann
4d40c70214 Merge pull request #3211 from MichaelEischer/sftp-speedup
Speed-up caching/pack download via SFTP
2021-01-28 14:16:55 +01:00
Alexander Neumann
44169d0dc4 Merge pull request #3205 from MichaelEischer/fix-quiet-verbose
Properly check that --quiet and --verbose are not combined
2021-01-28 13:53:31 +01:00
Alexander Neumann
6aa7e9f9c6 Merge pull request #3174 from MichaelEischer/parallelize-lock-loading
Parallelize lock file loading
2021-01-28 13:52:12 +01:00
Alexander Neumann
bdfedf1f5b Merge pull request #3173 from MichaelEischer/unify-index-loading
Unify index loading
2021-01-28 13:50:42 +01:00
greatroar
b9cfe6f68a internal/ui/termstatus: Use Fprintln to get a newline 2021-01-28 13:30:10 +01:00
Alexander Neumann
72eec8c0c4 Merge pull request #3106 from MichaelEischer/parallel-tree-walk
Parallelize tree walk in prune and copy and add progress bar to check
2021-01-28 12:06:42 +01:00
Michael Eischer
68608a89ad restic: add comment about StreamTrees shutdown 2021-01-28 11:10:50 +01:00
Michael Eischer
1e306be000 Add changelog entry 2021-01-28 11:10:50 +01:00
Michael Eischer
ddb7697d29 restic: Test progress reporting of StreamTrees 2021-01-28 11:10:50 +01:00
Michael Eischer
313ad0e32f progress/counter: Fix test for final report call 2021-01-28 11:10:50 +01:00
Michael Eischer
e2b0072441 check: add progress bar to the tree structure check 2021-01-28 11:10:50 +01:00
Michael Eischer
505f8a2229 progress/counter: Support updating the progress bar maximum 2021-01-28 11:10:47 +01:00
Michael Eischer
eda8c67616 restic: let FindUsedBlobs handle multiple snapshots at once 2021-01-28 11:08:43 +01:00
Michael Eischer
258ce0c1e5 parallel: report progress for StreamTrees
This assigns an id to each tree root and then keeps track of how many
tree loads (i.e. trees referenced for the first time) are pending per
tree root. Once a tree root and its subtrees were fully processed there
are no more pending tree loads and the tree root is reported as
processed.
2021-01-28 11:08:43 +01:00
Michael Eischer
3d6a3e2555 copy: Remove treeCloner struct 2021-01-28 11:08:43 +01:00
Michael Eischer
0caad1e890 copy: parallelize tree walk 2021-01-28 11:08:43 +01:00
Michael Eischer
f2a1b125cb restic: Actually parallelize FindUsedBlobs 2021-01-28 11:08:43 +01:00
Michael Eischer
6e03f80ca2 check: Split the parallelized tree loader into a reusable component
The actual code change is minimal
2021-01-28 11:08:43 +01:00
Michael Eischer
1d7bb01a6b check: Cleanup tree loading and switch to use errgroup
The helper methods are now wired up in the Structure method.
2021-01-28 11:08:43 +01:00
Alexander Neumann
a4689eb3b9 Merge pull request #3199 from MichaelEischer/non-interactive-counter
Don't print progress on non-interactive terminals
2021-01-28 10:53:38 +01:00
Alexander Neumann
c5a66e9181 ui: Simlify channel receive 2021-01-28 10:42:02 +01:00
Wouter van Os
b5972f184c doc: Note only path-style URL support for S3
This adds a node to mention that currently only path-style URL's are supported for S3, as per code at:
- aa0faa8c7d/internal/backend/s3/config.go (L42-L45)
- aa0faa8c7d/internal/backend/s3/config.go (L48-L62)
2021-01-23 16:54:08 +01:00
Alexander Weiss
d7dc19a496 prune: Always repack packs containing tree blobs 2021-01-15 16:42:04 +01:00
Michael Eischer
f3442ce8a5 Test that WriteTo of a backend's Load remains accessible 2021-01-03 22:23:53 +01:00
Michael Eischer
678e75e1c2 sftp: enforce use of optimized upload method
ReadFrom was already used by Save before, this just ensures that this
won't accidentally change in the future.
2021-01-03 22:23:53 +01:00
Michael Eischer
6b5b29dbee limiter: add unit tests 2021-01-03 22:23:53 +01:00
Michael Eischer
f35f2c48cd limiter: support WriteTo in LimitBackend for read rate limiting 2021-01-03 22:23:53 +01:00
Michael Eischer
bcb852a8d0 hashing: support WriteTo in the reader 2021-01-03 22:23:53 +01:00
MichaelEischer
aa0faa8c7d Merge pull request #3208 from restic/add-mips
Add mips* architectures to CI and release
2021-01-03 16:06:10 +01:00
MichaelEischer
f7ec263a22 Merge pull request #3109 from aawsome/optimize-filerestorer
restore: Don't save pack content in memory
2021-01-03 14:53:41 +01:00
Alexander Neumann
7d665fa1f4 Add entry to changelog 2021-01-03 14:41:11 +01:00
Michael Eischer
69d5b4c36b restorer: lower-case variable name 2021-01-03 13:55:59 +01:00
Alexander Neumann
36db248e30 Split cross compilation targets into two jobs 2021-01-03 10:50:54 +01:00
Alexander Neumann
eb72b10f55 Add mips* architectures to CI and release 2021-01-03 10:41:54 +01:00
Alexander Neumann
622f4c7daa Run cloud backends also on pushes 2021-01-03 10:38:10 +01:00
Alexander Neumann
f8c50394d6 Revert "Run cloud tests if secrets are available"
This reverts commit aa648bdcac.
2021-01-03 10:28:01 +01:00
Alexander Neumann
aa648bdcac Run cloud tests if secrets are available 2021-01-03 10:26:55 +01:00
Alexander Neumann
e8abc79ce9 Add hints for keeping the list of architectures in sync 2021-01-01 10:09:04 +01:00
Alexander Weiss
34a33565c8 Fix loadBlob in filerestorer 2021-01-01 08:06:04 +01:00
Alexander Weiss
7409225fa8 Add filerestorer test where only parts of pack are used 2021-01-01 07:24:46 +01:00
Alexander Weiss
07b3f65a6f filesrestorer: Re-use buffer 2021-01-01 07:24:46 +01:00
Alexander Weiss
3e0acf1395 restore: Don't save (part of) pack in memory 2021-01-01 07:24:46 +01:00
Michael Eischer
97388b3504 Properly check that --quiet and --verbose are not combined
If --verbose is specified once, then globalOptions.Verbose == 1.
Previously --quiet --verbose would silently ignore the --verbose flag.
2020-12-30 21:24:18 +01:00
Alexander Neumann
8b84c96d9d Merge pull request #3204 from MichaelEischer/archiver-tomb-race
archiver: fix race condition during worker startup
2020-12-30 20:04:20 +01:00
Michael Eischer
debc4a3a99 archiver: fix race condition during worker startup
When the tomb is created with a canceled context, then the workers
started via `t.Go` exist nearly immediately. Once for the first time all
started goroutines have been stopped, it is not allowed to issue further
calls to `t.Go`. This is a problem when the started goroutines exit
immediately, as for example the first goroutine might already have
stopped before starting the second one, which is not allowed as once the
first goroutines has stopped no goroutines were running.

To fix this race condition the startup and main task of the archiver now
also run within a `t.Go` function. This also allows unifying the error
handling as it is no longer necessary to distinguish between errors
returned by the workers or the saveTree processing. The tomb now just
returns the first error encountered, which should also be the most
descriptive one.
2020-12-30 17:31:22 +01:00
MichaelEischer
e1efc193e1 Merge pull request #3139 from aawsome/prune-healing
prune: Add healing of repository in some situations
2020-12-29 22:17:27 +01:00
Alexander Weiss
f0113139ea prune: Correct error message 2020-12-29 20:20:05 +01:00
Alexander Weiss
f6df94a50e prune: Add self-healing
Allow prune to heal situations where blobs in the index are missing or
the corresponding packfiles are damaged if those blobs are not needed.
2020-12-29 20:20:05 +01:00
MichaelEischer
31e56f1ad5 Merge pull request #3197 from SkYNewZ/fix/3183
Fix tag handling for multiple tag lists
2020-12-29 18:44:38 +01:00
MichaelEischer
7fda2f2ad8 Merge pull request #3134 from greatroar/unlock-warn
Warn when unlock fails instead of returning an error
2020-12-29 18:30:08 +01:00
greatroar
dec5008369 Warn when unlock fails instead of returning an error
Only one caller was checking the error.
2020-12-29 17:48:20 +01:00
Quentin Lemaire
873505ed3b Update related changelog 2020-12-29 17:12:46 +01:00
Quentin Lemaire
25ecf9eafb fix(cmd_tag): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Quentin Lemaire
e88f3fb80c fix(cmd_backup): Use restic.TagLists 2020-12-29 17:12:46 +01:00
Alexander Neumann
b2efa0af39 Merge pull request #3164 from MichaelEischer/improve-context-cancel
Improve context cancel handling in archiver and backends
2020-12-29 17:03:42 +01:00
Michael Eischer
25f4acdaa8 Add changelog entry 2020-12-29 16:32:18 +01:00
Michael Eischer
cff4955a48 ui: remove dead struct member 2020-12-29 16:32:18 +01:00
Michael Eischer
05a987b07c Support values less than 1 for RESTIC_PROGRESS_FPS
For example set the variable to 0.016666 to print the progress once per
minute.
2020-12-29 16:32:18 +01:00
Michael Eischer
92da5168e1 ui: force backup progress update on signal 2020-12-29 16:32:18 +01:00
Michael Eischer
34afc93ddc ui/progress: extract signal handling into own package 2020-12-29 16:32:18 +01:00
Michael Eischer
023eea6463 ui: don't shorten non-interactive progress output 2020-12-29 16:32:18 +01:00
Michael Eischer
684600cf42 ui: update status for the backup command on non-interactive terminals
Allow the backup command to print status on non-interactive terminals.
The output is disabled by setting a MinUpdatePause == 0.
2020-12-29 16:03:43 +01:00
Michael Eischer
85fe5feadb Unify progress report frequency calculation 2020-12-29 16:03:43 +01:00
Michael Eischer
969141b5e9 Honor RESTIC_PROGRESS_FPS env variable on non-interactive terminals
This makes it possible to use the environment variable to also get
regular progress updates on non-interactive terminals.
2020-12-29 16:03:43 +01:00
Michael Eischer
13ce981794 ui: cleanup backup status shutdown 2020-12-29 16:03:43 +01:00
Michael Eischer
c2ef049f1b ui/progress: don't print progress on non-interactive terminals
This reverts to the old behavior of not printing progress updates on
non-interactive terminals. It was accidentally changed in #3058.
2020-12-29 16:03:43 +01:00
MichaelEischer
a488d4c847 Merge pull request #2833 from greatroar/aix
AIX port
2020-12-29 12:32:50 +01:00
Alexander Neumann
4133b1ea65 Synchronize OS and architectures for testing 2020-12-29 11:11:50 +01:00
Alexander Neumann
46d2ca5095 Remove old test script 2020-12-29 11:02:48 +01:00
Quentin Lemaire
334d8ce724 feat(tags): Create Flatten() method 2020-12-29 10:59:46 +01:00
Alexander Neumann
c661518df9 Merge pull request #2793 from greatroar/rclone-doc
Clarify rclone-over-SSH docs
2020-12-29 10:46:34 +01:00
Michael Eischer
0d81f16343 Add AIX as cross-compile target to CI 2020-12-29 01:35:01 +01:00
greatroar
3b09ae9074 AIX port 2020-12-29 01:35:01 +01:00
greatroar
18531e3d6f Portability fixes to internal/restic
syscall.Mknod is not available on AIX.
2020-12-29 01:35:01 +01:00
Michael Eischer
ca07317815 add changelog 2020-12-28 21:06:47 +01:00
Michael Eischer
d0ca8fb0b8 backend: test that a canceled context prevents RetryBackend operations 2020-12-28 21:06:47 +01:00
Michael Eischer
08b7f2b58d archiver: test that context canceled error is not dropped 2020-12-28 21:06:47 +01:00
Michael Eischer
e483b63c40 retrybackend: Fail operations when context is already canceled
Depending on the used backend, operations started with a canceled
context may fail or not. For example the local backend still works in
large parts when called with a canceled context. Backends transfering
data via http don't work. It is also not possible to retry failed
operations in that state as the RetryBackend will abort with a 'context
canceled' error.

Ensure uniform behavior of all backends by checking for a canceled
context by checking for a canceled context as a first step in the
RetryBackend. This ensures uniform behavior across all backends, as
backends are always wrapped in a RetryBackend.
2020-12-28 21:06:47 +01:00
Michael Eischer
fc60b560ba archiver: Let saveTree report a canceled context as an error
If the context was canceled then saveTree might receive a treeID or not
depending on the timing. This could cause saveTree to incorrectly return
a nil treeID as valid. Fix this always returning an error when the
context was canceled in the meantime.
2020-12-28 21:06:47 +01:00
Michael Eischer
736e964317 archiver: Don't loose error if background context is canceled
A canceled background context lets the blob/tree/fileSavers exit
without reporting an error. The error handling previously replaced
a 'context canceled' error received by the main backup method with
the error reported by the savers. However, in case of a canceled
background context that error is nil, causing restic to loose the
error and save a snapshot with a nil tree.
2020-12-28 21:06:47 +01:00
MichaelEischer
9c41e4a343 Merge pull request #3192 from DRON-666/fix-options
Fix formatting of `restic options` command
2020-12-23 22:53:40 +01:00
DRON-666
332b1896d1 Some options fixes
Add tests for bool type.
Fix subtle bug in TestOptionsApplyInvalid.
Fix options list formatting.
2020-12-23 23:26:04 +03:00
MichaelEischer
cb6b0f6255 Merge pull request #3181 from tWido/feature-no-retry-permission
Don't retry when "Permission denied" occurs in local backend
2020-12-23 20:24:29 +01:00
MichaelEischer
1e73aac610 Merge pull request #3179 from aawsome/check-filesize
check: Remove filesize counter
2020-12-23 20:22:02 +01:00
Alexander Weiss
2a1add7538 check: remove file size counter 2020-12-23 02:34:31 +01:00
tWido
7dab113035 Don't retry when "Permission denied" occurs in local backend 2020-12-22 23:41:12 +01:00
MichaelEischer
8efb874f48 Merge pull request #3148 from aawsome/simplify-index-rebuild
rebuild-index: code simplification
2020-12-22 23:27:53 +01:00
Michael Eischer
de99207046 repository: tweak comment for packs method 2020-12-22 23:01:58 +01:00
Alexander Weiss
d8d2cc6dd9 Simplify rebuild-index code 2020-12-22 23:01:58 +01:00
Alexander Weiss
68b74e359e Count packs directly in RebuildIndexFiles 2020-12-22 23:01:58 +01:00
Michael Eischer
b9f5d3fe13 repository: Add test for ForAllIndexes 2020-12-22 22:36:18 +01:00
Michael Eischer
a12c5f1d37 repository: move otherwise unused LoadIndex to tests 2020-12-22 22:36:18 +01:00
Michael Eischer
24474a36f4 repository: deduplicate index loading implementation 2020-12-22 22:36:18 +01:00
Michael Eischer
ccc84af73d debug/list: parallelize index loading 2020-12-22 22:36:18 +01:00
Michael Eischer
96904f8972 check: extract parallel index loading 2020-12-22 22:36:18 +01:00
MichaelEischer
69f9d269eb Merge pull request #3189 from restic/fix-counter-segfault
ui/progress: Use mutex instead of atomic
2020-12-22 22:14:08 +01:00
rawtaz
ec59c73489 Merge pull request #3188 from restic/forget-prune-dry-run
forget: Enable --dry-run together with --prune
2020-12-22 21:09:15 +01:00
Alexander Neumann
6c514adb8a ui/progress: Use mutex instead of atomic
The counter value needs to be aligned to 64 bit in memory for the
atomic functions to work on some platform (such as 32 bit ARM).

The atomic package says in its documentation:

> These functions require great care to be used correctly. Except for
> special, low-level applications, synchronization is better done with
> channels or the facilities of the sync package.

This commit replaces the atomic functions with a simple sync.Mutex, so
we don't have to care about alignment.
2020-12-22 21:03:27 +01:00
Alexander Neumann
edf89e1c74 forget: Enable --dry-run together with --prune 2020-12-22 20:58:02 +01:00
Alexander Neumann
f7c7c2f730 Merge pull request #3175 from MichaelEischer/fix-3099-changelog
Fix changelog for #3099
2020-12-20 11:29:57 +01:00
Michael Eischer
cfea79d0c5 prune: don't print stacktrace on console 2020-12-19 14:28:48 +01:00
Michael Eischer
5cd40f8b58 Add changelog 2020-12-19 11:38:38 +01:00
MichaelEischer
d32949ee54 Merge pull request #3081 from DRON-666/dump-zip
Add zip support to the `dump` command
2020-12-19 11:33:33 +01:00
DRON-666
83b10dbb12 Deduplicate dumper closing logic 2020-12-19 02:42:46 +03:00
DRON-666
e136dd8696 Deduplicate test code 2020-12-19 02:06:54 +03:00
DRON-666
33adb58817 Minor fixes and linter suggestions 2020-12-19 02:04:17 +03:00
DRON-666
da9053b184 Some gramma fixes in documentation 2020-12-19 01:16:15 +03:00
DRON-666
ef1aeb8724 dump: Update docs and changelog 2020-12-19 01:09:47 +03:00
DRON-666
2ca76afc2b dump: Add new option --archive 2020-12-19 01:09:47 +03:00
DRON-666
89ab6d557e dump: Add zip dumper 2020-12-19 01:09:47 +03:00
DRON-666
0256f95994 dump: Split tar and walk logic 2020-12-19 01:09:47 +03:00
Michael Eischer
bfadc82a20 Fix changelog for #3099 2020-12-18 20:58:24 +01:00
Michael Eischer
34b6130a0e restic: parallelize lock file loading 2020-12-18 20:46:16 +01:00
MichaelEischer
22260d130d Merge pull request #3170 from greatroar/dont-retry
Don't retry permanent errors in backends
2020-12-17 23:29:56 +01:00
greatroar
9341a83b05 Changelog entry for #2453 and #3170 2020-12-17 23:15:48 +01:00
greatroar
66d904c905 Make invalid handles permanent errors 2020-12-17 12:47:53 +01:00
greatroar
746dbda413 Mark "ssh exited" errors in SFTP as permanent 2020-12-17 12:43:09 +01:00
greatroar
f7784bddb3 Don't retry when "no space left on device" in local backend
Also adds relevant documentation to the restic.Backend interface.
2020-12-17 12:43:09 +01:00
Alexander Neumann
1cdd38d9e0 Merge pull request #3165 from restic/2747-doc-forget
doc: Clarify calendar boundaries for --keep-* options
2020-12-16 08:38:03 +01:00
Leo R. Lundgren
b3c0d2f45b doc: Clarify calendar boundaries for --keep-* options 2020-12-15 12:43:06 +01:00
Peter Albrecht
cd25e36811 Add PGP fingerprint to 020_installation.rst
I like the idea of verifying the integrity of applications, I download from the internet. So I was very happy to see that restic does provide SHA256-checksums which are signed with the maintainers PGP key.

The only thing I miss: I could not find a direct way to download the used PGP key and verify the keys fingerprint.

Doing some searches, I found:
* https://github.com/restic/rest-server/issues/121
* https://restic.net/blog/2015-09-16/verifying-code-archive-integrity/

To help other restic users, I think you should add information about your PGP key/fingerprint to this installation doc, too. To save you some precious time, I created a draft, how this doc might be expanded, in this pull-request. You are free to accept it or change the text to your liking.

I copied the key/fingerprint text from: ``restic/restic/master/doc/090_participating.rst``

Thank you for your work in restic!
2020-12-13 17:00:00 +01:00
Alexander Neumann
e96677cafb Merge pull request #3158 from MichaelEischer/support-swift-auth-id-variables
swift: Add support for id based keystone v3 auth parameters
2020-12-12 16:27:38 +01:00
Michael Eischer
1d69341e88 swift: Add support for id based keystone v3 auth parameters
This adds support for the following environment variables, which were
previously missing:

OS_USER_ID            User ID for keystone v3 authentication
OS_USER_DOMAIN_ID     User domain ID for keystone v3 authentication
OS_PROJECT_DOMAIN_ID  Project domain ID for keystone v3 authentication
OS_TRUST_ID           Trust ID for keystone v3 authentication
2020-12-11 19:22:34 +01:00
Alexander Neumann
36c5d39c2c Fix issues reported by semgrep 2020-12-11 09:41:59 +01:00
Alexander Neumann
7facc8ccc1 Merge pull request #2505 from aawsome/fix-repo-configfile
Fix repo configfile
2020-12-07 07:52:37 +01:00
Alexander Neumann
ba31c6fdaa Merge pull request #3150 from MichaelEischer/fix-windows-redir-output
termstatus: Fix canUpdateStatus detection for redirected output on windows
2020-12-06 21:17:27 +01:00
Alexander Neumann
b58799d83a Merge pull request #3152 from MichaelEischer/fix-backup-background-hang
backup: Fix shutdown hang when running in the background on linux
2020-12-06 21:16:00 +01:00
Alexander Neumann
0d5b764f90 Merge pull request #3130 from aawsome/snapshots-parallel
Make loading snapshots parallel
2020-12-06 21:08:18 +01:00
Alexander Weiss
d6b3859e48 Add changelog 2020-12-06 19:29:18 +01:00
Michael Eischer
b48f579530 termstatus: Fix canUpdateStatus detection for redirected output
The canUpdateStatus check was simplified in #2608, but it accidentally flipped
the condition. The correct check is as follows: If the output is a pipe then
restic probably runs in mintty/cygwin. In that case it's possible to
update the output status. In all other cases it isn't.

This commit inverts to condition again to offer the previous and correct
behavior.
2020-12-06 19:02:42 +01:00
Michael Eischer
401ef92c5f backup: Fix shutdown hang when running in the background
On shutdown the backup commands waits for the terminal output goroutine
to stop. However while running in the background the goroutine ignored
the canceled context.
2020-12-06 18:53:41 +01:00
Alexander Weiss
e329623771 Remove LoadAllSnapshots 2020-12-06 05:22:27 +01:00
Alexander Weiss
26f85779be Parallelize ForAllSnapshots 2020-12-06 05:09:58 +01:00
Alexander Weiss
5b9ee56335 Add ForAllSnapshots 2020-12-06 05:04:21 +01:00
MichaelEischer
3264eae9f6 Merge pull request #3149 from aawsome/fix-rebuild-index
Bugfix for rebuild-index
2020-12-05 22:18:22 +01:00
Alexander Weiss
83c8a9b058 Bugfix: packSizeFromList should save size from List() 2020-12-05 20:58:36 +01:00
Alexander Neumann
43cf301450 Merge pull request #3141 from MichaelEischer/fix-fuse-dir-owner
fuse: Properly set uid/gid for directories
2020-12-01 16:06:24 +01:00
Michael Eischer
d05c88a5d6 fuse: Properly set uid/gid for directories
In #2584 this was changed to use the uid/gid of the root node. This
would be okay for the top-level directory of a snapshot, however, this
change also applied to normal directories within a snapshot. This
change reverts the problematic part and adds a test that directory
attributes are represented correctly.
2020-11-30 23:41:49 +01:00
Alexander Neumann
058b102db0 Merge pull request #3138 from MichaelEischer/fix-debug-build 2020-11-30 12:27:51 +01:00
rawtaz
fcebc7d250 Merge pull request #3135 from restic/manual
doc: Fix misc missing/incorrect text in manual
2020-11-29 19:20:51 +01:00
MichaelEischer
f2959127b6 Merge pull request #3065 from greatroar/local-subdirs
Don't recurse in local backend's List if not required
2020-11-29 19:03:59 +01:00
Leo R. Lundgren
61460dee52 doc: Fix misc missing/incorrect text in manual 2020-11-29 18:59:24 +01:00
Michael Eischer
54a6d98945 Enable debug builds for CI 2020-11-29 18:47:00 +01:00
Michael Eischer
f72f6c9c80 Fix debug build 2020-11-29 18:44:36 +01:00
MichaelEischer
52b98f7f95 Merge pull request #3017 from greatroar/files-from0
Add backup options --files-from-verbatim and --files-from-raw
2020-11-29 18:15:21 +01:00
Alexander Neumann
04d856e601 Merge pull request #3136 from restic/rawtaz-copy-doc
doc: Emphasize double transfer and duplication in copy command
2020-11-29 13:57:10 +01:00
Alexander Neumann
a7b49c4889 Merge pull request #3119 from restic/keep-mountpoints
Keep mountpoints as empty directories for --one-file-system
2020-11-29 11:22:12 +01:00
rawtaz
a568211b98 Merge pull request #3128 from vrenaville/tzdata
[FIX] Timezone in docker image
2020-11-28 22:43:18 +01:00
Leo R. Lundgren
f70b10d0ee doc: Emphasize double transfer and duplication in copy command 2020-11-28 19:49:34 +01:00
greatroar
55bf76ba0c backup: Add --files-from-{verbatim,raw} options 2020-11-28 18:22:31 +01:00
Alexander Neumann
162117c42c Add changelog 2020-11-28 17:00:31 +01:00
Alexander Neumann
82ae942965 backup: Keep mountpoints for --one-file-system
When a file system is mounted at a directory, lstat() returns attributes
of the root node of the mounted file system, including the device ID of
the other file system. The previous code used when --one-file-system is
specified excluded the directory itself because of that.

This commit changes the code so that mountpoints are kept as empty
directories, its attributes set to the root note of the mounted file
system. The behavior mimics `tar`, which does the same.
2020-11-28 17:00:31 +01:00
Alexander Neumann
f576d3d826 Add tests 2020-11-28 17:00:31 +01:00
Alexander Neumann
037f0a4c91 Refactor device ID checking 2020-11-28 17:00:31 +01:00
Alexander Neumann
2f9346a5af Merge pull request #3125 from metalsp0rk/cat-respect-no-lock
Make restic cat respect --no-lock
2020-11-28 12:45:59 +01:00
Alexander Neumann
a3105799c9 Print warning when unlocking the repo fails 2020-11-28 11:31:25 +01:00
Kyle Brennan
666768cd17 cat: Respect --no-lock flag 2020-11-28 11:30:02 +01:00
vrenaville
adc7a6555f [FIX] Timezone in docker image 2020-11-27 08:05:19 +01:00
Alexander Neumann
9a97095a4c Merge pull request #3120 from aawsome/blob-implementation
Blob implementation
2020-11-22 20:59:30 +01:00
Alexander Weiss
aa7a5f19c2 Use BlobHandle in index methods 2020-11-22 20:41:12 +01:00
Alexander Weiss
e3013271a6 Harmonize naming 2020-11-22 20:41:12 +01:00
Alexander Weiss
92bd448691 Make BlobHandle substruct of Blob 2020-11-22 20:41:10 +01:00
Alexander Neumann
c844580e0f Merge pull request #3101 from aawsome/packsizes
Compute packsizes in MasterIndex
2020-11-22 15:49:19 +01:00
Alexander Weiss
67c938f232 Use PackSize in prune 2020-11-21 22:13:54 +01:00
Alexander Weiss
a851c53cbe Use PackSize in checker 2020-11-21 22:13:54 +01:00
Alexander Weiss
4960b841e6 Use PackSize in rebuild-index 2020-11-21 22:13:54 +01:00
Alexander Weiss
ce5d630681 Add MasterIndex.PackSize() 2020-11-21 22:13:54 +01:00
Alexander Weiss
c3ddde9e7d Return hdrSize in ListPack 2020-11-21 22:13:54 +01:00
rawtaz
cac481634c Merge pull request #3115 from johanbove/patch-1
Docs: Update 04_backup.rst
2020-11-20 11:23:03 +01:00
Johan Bové
c23b1a4cba Update 04_backup.rst
Fixed typo - _files_ are included from _folders_, not other _files_.
2020-11-20 07:52:23 +01:00
Alexander Neumann
110a32a08b Merge pull request #3113 from aawsome/fix-prune-stats-duplicates
Fix statistics in prune for duplicates
2020-11-19 20:34:11 +01:00
greatroar
8e213e82fc backend/local: replace fs.Walk with custom walker
This code is more strict in what it expects to find in the backend:
depending on the layout, either a directory full of files or a directory
full of such directories.
2020-11-19 16:46:42 +01:00
Alexander Neumann
8a150ee91f Merge pull request #3112 from MichaelEischer/reduce-progress-log-spam
Limit progress bar updates to once per second on non-terminal outputs
2020-11-19 07:54:36 +01:00
Alexander Weiss
cb5ec7ea6b Fix statistics in prune for duplicates
Note that this fix only solves the statistics problem, if
all duplicates are marked for repacking.

If not all duplicates are marked for repacking, we lack the
information which

The situation that not all duplicates are marked for repacking can occur
when using the `max-repack-size` option
2020-11-18 22:30:22 +01:00
Michael Eischer
625410f003 Limit progress bar updates to once per second on non-terminal outputs
The code accidentally checked whether stdin is a terminal instead of
stdout, the former is not relevant here as the output is printed on
stdout.
2020-11-18 22:12:07 +01:00
Alexander Neumann
75eff92b56 Merge pull request #3107 from eleith/do-not-require-bucket-permissions-for-init
do not require gs bucket permissions to init repository
2020-11-18 16:53:45 +01:00
eleith
a24e986b2b do not require gs bucket permissions to init repository
a gs service account may only have object permissions on an existing
bucket but no bucket create/get permissions.

these service accounts currently are blocked from initialization a
restic repository because restic can not determine if the bucket exists.

this PR updates the logic to assume the bucket exists when the bucket
attribute request results in a permissions denied error.

this way, restic can still initialize a repository if the service
account does have object permissions

fixes: https://github.com/restic/restic/issues/3100
2020-11-18 06:14:11 -08:00
rawtaz
6822ce8479 Merge pull request #3105 from restic/fix-init-repo-file
Allow using --repository-file in init
2020-11-17 21:54:44 +01:00
Alexander Neumann
d857fb6e59 Allow using --repository-file in init 2020-11-17 20:43:46 +01:00
rawtaz
342520b648 Merge pull request #3103 from greatroar/rephrase-pr3102
Rephrase #3095/#3102 changelog entry
2020-11-17 12:52:12 +01:00
greatroar
028f2b8c0e Rephrase #3095/#3102 changelog entry 2020-11-17 12:37:06 +01:00
rawtaz
1b6e8c888f Merge pull request #3102 from tofran/remove-rclone-drive-use-tras-default-param
Remove `--drive-use-trash=false` from rclone param
2020-11-17 11:21:20 +01:00
Alexander Neumann
5f3b802ee7 Merge pull request #3099 from MichaelEischer/check-less-memory 2020-11-16 10:11:25 +01:00
Michael Eischer
022dc35be9 Add changelog 2020-11-15 19:02:51 +01:00
Michael Eischer
1f43cac12d check: Only track data blobs when unused blobs should be reported
This improves the memory usage of check a lot as it now only has to
track tree blobs when run using the default parameters.
2020-11-15 18:43:07 +01:00
Michael Eischer
6da66c15d8 check: Simplify referenced blob tracking
The result is identical as long as the context in not canceled. However,
in that case the result is incomplete anyways.
2020-11-15 18:42:55 +01:00
Michael Eischer
3500f9490c check: Simplify blob status tracking
UnusedBlobs now directly reads the list of existing blobs from the
repository index. This removes the need for the blobStatusExists flag,
which in turn allows converting the blobRefs map into a BlobSet.
2020-11-15 18:42:42 +01:00
Michael Eischer
b8c7543a55 check: Merge 'size could not be found' and 'not found in index' errors
By construction these two errors always show up in pairs: 'size could
not be found' is printed when the blob is not found in the repository
index. That blob is also part of the `blobs` array. Later on, check
iterates over that array and checks whether the blob is marked as
existing. Which cannot be the case as that mark is generated by
iterating over the repository index.

The merged warning no longer reports the blob index within a file. That
information could also be derived by printing the affected tree using
`cat` and searching for the blob.
2020-11-15 18:41:50 +01:00
MichaelEischer
45ba456291 Merge pull request #3038 from fgma/check-data-subset-percentage
Check data subset: check random percentage subset
2020-11-15 18:24:13 +01:00
Michael Eischer
caac38ed27 check: Tweak subset percentage over 100% error message 2020-11-15 18:13:50 +01:00
Michael Eischer
15c537f9db Rename changelog entry to contain issue id 2020-11-15 18:13:50 +01:00
fgma
8f9cea8cc0 Check data subset: check random percentage subset 2020-11-15 18:13:50 +01:00
Alexander Neumann
3c0c0c132b Merge pull request #3006 from aawsome/new-rebuild-index
Reimplement rebuild-index and remove /internal/index
2020-11-15 17:48:43 +01:00
Alexander Neumann
9968220652 Merge pull request #2850 from greatroar/packer-malloc
Decrease allocation rate in internal/pack
2020-11-15 17:19:49 +01:00
kitone
0649828555 Add changelog 2020-11-15 17:09:30 +01:00
kitone
3c03b35212 Fix #1681 should not try to create the mount point if it doesn't exist, rather return an error 2020-11-15 17:09:30 +01:00
greatroar
ab2b7d7f9a Decrease allocation rate in internal/pack
internal/repository benchmark results:

name             old time/op    new time/op    delta
PackerManager-8     179ms ± 1%     181ms ± 1%   +0.78%  (p=0.009 n=10+10)

name             old speed      new speed      delta
PackerManager-8   294MB/s ± 1%   292MB/s ± 1%   -0.77%  (p=0.009 n=10+10)

name             old alloc/op   new alloc/op   delta
PackerManager-8    91.3kB ± 0%    72.2kB ± 0%  -20.92%  (p=0.000 n=9+7)

name             old allocs/op  new allocs/op  delta
PackerManager-8     1.38k ± 0%     0.76k ± 0%  -45.20%  (p=0.000 n=10+7)
2020-11-15 16:51:47 +01:00
greatroar
9a8a2cae4c Move pack testing logic to test file 2020-11-15 16:45:49 +01:00
Alexander Weiss
9607cad267 Remove internal/index 2020-11-15 07:05:09 +01:00
Alexander Weiss
3d1d5295cc Add changelog 2020-11-15 07:05:09 +01:00
Alexander Weiss
30b6a0878a Reimplement rebuild-index 2020-11-15 07:05:09 +01:00
Alexander Weiss
187c8fb259 Parallelize MasterIndex.Save() 2020-11-15 07:05:09 +01:00
Alexander Weiss
1ec628ddf5 Add extraObsolete to MasterIndex.Save 2020-11-15 07:05:09 +01:00
Alexander Weiss
5898cb341f Use CreateIndexFromPacks() in test 2020-11-15 07:05:05 +01:00
Alexander Weiss
43732bb885 Add CreateIndexFromPacks() 2020-11-15 07:04:51 +01:00
MichaelEischer
6feaf6bd1f Merge pull request #3030 from greatroar/concrete-statt
Replace restic.statT interface by concrete types
2020-11-14 23:32:17 +01:00
greatroar
c45f8ee075 Replace restic.statT interface by concrete types
name                old time/op    new time/op    delta
NodeFillUser-8        1.81µs ± 9%    1.50µs ± 5%  -17.07%  (p=0.000 n=19+20)
NodeFromFileInfo-8    1.76µs ± 4%    1.49µs ± 6%  -15.63%  (p=0.000 n=20+19)

name                old alloc/op   new alloc/op   delta
NodeFillUser-8          496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)
NodeFromFileInfo-8      496B ± 0%      352B ± 0%  -29.03%  (p=0.000 n=20+20)

name                old allocs/op  new allocs/op  delta
NodeFillUser-8          3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
NodeFromFileInfo-8      3.00 ± 0%      2.00 ± 0%  -33.33%  (p=0.000 n=20+20)
2020-11-14 23:23:26 +01:00
MichaelEischer
3601a9b6cd Merge pull request #2690 from SkYNewZ/master
Fix #2688: Handle comma-separated list tags when using backup command
2020-11-14 23:19:42 +01:00
Michael Eischer
fdec8051ab tags: Tweak description of --add/set/remove options 2020-11-14 22:55:30 +01:00
MichaelEischer
333c5a19d4 Merge pull request #3082 from aawsome/check-sizes
Check: check sizes of packs from index and packheader
2020-11-14 22:37:42 +01:00
Quentin Lemaire
a8ad6b9a4b fix(tags): Change tags list type according to restic.TagList 2020-11-14 16:24:58 +00:00
Quentin Lemaire
b0882b3f3c fix(snapshots): Update help message to better match the 'forget' command one 2020-11-14 15:48:57 +00:00
Quentin Lemaire
e74110a833 docs: Write new entry to changelog/unreleased 2020-11-14 15:48:56 +00:00
Quentin Lemaire
ae441d3134 fix(backup): Switch tags cobra type to handle comma-separated list 2020-11-14 15:48:56 +00:00
Alexander Neumann
913a34f568 Merge pull request #3094 from restic/fix-ci-tests
Fix ci tests
2020-11-14 11:24:28 +01:00
Alexander Neumann
468612b108 Disable fuse test for macOS 2020-11-14 11:02:49 +01:00
Alexander Weiss
7eabcabf68 Adjust changelog 2020-11-14 00:42:49 +01:00
Alexander Weiss
17bb77b1f9 check: Also check blob length and offset 2020-11-14 00:42:49 +01:00
Alexander Weiss
80dcfca191 check: Check sizes computed from index and pack header 2020-11-14 00:42:49 +01:00
tofran
94a154c7ca Remove --drive-use-trash=false from rclone param
Google drive trash retention policy changed making this
no longer a good default
a go
Issue #3095
2020-11-13 22:58:48 +00:00
Alexander Neumann
219d9a62f2 Download assets from repo 2020-11-13 21:56:14 +01:00
rawtaz
e8713bc209 Merge pull request #3093 from restic/fix-diff-metadata
diff: Correctly count top-level blobs
2020-11-13 21:50:29 +01:00
Alexander Neumann
04d1983800 Merge pull request #3090 from fgma/vss-fix-386
vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386
2020-11-13 21:30:34 +01:00
fgma
88208c3db2 vss: improved changelog entry 2020-11-13 21:13:17 +01:00
Alexander Neumann
59ea5a4208 diff: Correctly count top-level blobs 2020-11-13 21:11:21 +01:00
Alexander Neumann
145830005b Use old versions, otherwise tar doesn't work 2020-11-12 21:35:01 +01:00
Alexander Neumann
8ad9f88993 helpers: Improve error message 2020-11-12 20:38:31 +01:00
fgma
859d89b032 vss: add changelog file for issue 3090 2020-11-12 19:38:22 +01:00
fgma
f9223cd827 vss: fix DeleteSnapshots() and GetSnapshotProperties() on 386 2020-11-12 19:31:00 +01:00
Alexander Neumann
0be906a92f CI: Use netcologne mirror 2020-11-11 20:59:23 +01:00
rawtaz
dfb9326b1b Merge pull request #3085 from LordGaav/s3-list-objects-v1-flag
Extended option to select V1 API for ListObjects on S3 backend
2020-11-11 20:52:44 +01:00
Alexander Neumann
e4e0ce09ad CI: Update links for Windows binaries 2020-11-11 20:51:01 +01:00
Alexander Neumann
0334114865 CI: Hardcode mirror URL 2020-11-11 20:42:47 +01:00
Alexander Neumann
b1bbdcb637 Improve changelog 2020-11-11 20:30:30 +01:00
Alexander Neumann
4a0b7328ec s3: Remove dots for config description 2020-11-11 20:20:35 +01:00
Alexander Neumann
3e0456d88b Highlight that s3.list-objects-v1 is a temporary 2020-11-11 20:11:35 +01:00
Alexander Neumann
dd94174379 Merge pull request #3086 from greatroar/diff-errors
Improve error reporting from restic diff
2020-11-11 19:48:03 +01:00
greatroar
63e32c44c0 Improve error reporting from restic diff
Instead of a stacktrace, restic diff 111 222 now reports:

	Fatal: no matching ID found for prefix "111"
2020-11-11 16:40:40 +01:00
Nick Douma
f013662e3f Remove separate section on Ceph, and move s3.list-objects-v1 note to S3 section 2020-11-11 15:11:14 +01:00
Nick Douma
4320ff2bbf Add changelog entry for s3.list-objects-v1 2020-11-11 12:33:22 +01:00
Nick Douma
354b7e89cc Document the extended s3.list-objects-v1 flag in a new Ceph section 2020-11-11 12:32:46 +01:00
Nick Douma
829959390a Provide UseV1 parameter to minio.ListObjectsOptions based on s3.list-objects-v1 2020-11-11 11:54:38 +01:00
Nick Douma
ccd55d529d Add s3.list-objects-v1 extended option and default to false 2020-11-11 11:54:36 +01:00
Nick Douma
4ddcc17135 Add support for boolean extended options 2020-11-11 11:54:27 +01:00
MichaelEischer
407843c5f9 Merge pull request #3034 from hoyho/gh_master
bugfix: omit ENOTDATA for extended attributes
2020-11-09 22:38:49 +01:00
MichaelEischer
46d31ab86d Merge pull request #3058 from greatroar/counter
Replace restic.Progress with new progress.Counter (fixes two race conditions)
2020-11-09 22:19:09 +01:00
Alexander Neumann
c986823d3f Merge pull request #3048 from aawsome/check-pack-index
check: check index for packs that are read
2020-11-09 20:12:32 +01:00
Alexander Weiss
239931578c check: check index for packs that are read 2020-11-09 17:28:14 +01:00
hoyho
9df52327cc bugfix: omit ENOTDATA for extended attributes
Signed-off-by: hoyho <luohaihao@gmail.com>
2020-11-10 00:20:34 +08:00
greatroar
21b787a4d1 Stop Counters where they're constructed and started 2020-11-09 13:03:31 +01:00
greatroar
ddca699cd2 Replace restic.Progress with new progress.Counter
This fixes two race conditions while cleaning up the code.
2020-11-09 12:12:35 +01:00
rawtaz
605db3b389 Merge pull request #3077 from MichaelEischer/tame-golangci-lint
Switch to the default set of golangci-lint linters
2020-11-08 22:34:27 +01:00
Michael Eischer
8de129e12f Switch to the default set of golangci-lint linters 2020-11-08 22:17:50 +01:00
Alexander Neumann
2072f0a481 Merge pull request #3076 from restic/check-go-mod-tidy
Check that go.mod/go.sum are up to date
2020-11-08 21:37:22 +01:00
Alexander Neumann
5731e391f8 Merge pull request #3075 from restic/warn-prune-no-cache
UI: Add several hints
2020-11-08 20:37:10 +01:00
Alexander Neumann
6a0a1d1f1c Check that go.mod/go.sum are up to date 2020-11-08 20:32:25 +01:00
Alexander Neumann
a8d21b5dcf prune: Warn if no cache is present 2020-11-08 20:25:35 +01:00
Alexander Neumann
823d0afd6e backup: Always print parent snapshot info 2020-11-08 20:25:35 +01:00
Alexander Neumann
a5989707ac Move test for golang-ci 2020-11-08 19:58:16 +01:00
Alexander Neumann
3a0cfafeb5 Fix quotes 2020-11-08 19:52:37 +01:00
Alexander Neumann
c923bd957d Fix test for pull request 2020-11-08 19:51:11 +01:00
Alexander Neumann
1a3f885d3d Only run golangci-lint for PRs 2020-11-08 17:34:07 +01:00
Alexander Neumann
3bf43d7951 Merge pull request #2475 from restic/use-github-actions
Use GitHub Actions for CI
2020-11-08 17:30:12 +01:00
Alexander Neumann
561da92396 Replace badges in README 2020-11-08 17:26:59 +01:00
Alexander Neumann
5cf42884c8 Run tests on GitHub Actions 2020-11-08 16:04:57 +01:00
MichaelEischer
9e4e0077fb Merge pull request #3069 from greatroar/xattr-eintr
Update pkg/xattr to handle EINTR on Linux
2020-11-08 12:52:23 +01:00
MichaelEischer
1758da855f Merge pull request #3068 from aawsome/context-mem-backend
Return context error in mem backend
2020-11-08 12:24:12 +01:00
greatroar
15ea90feed Update pkg/xattr to handle EINTR on Linux
Updates #2659. This is a case where the stdlib will not handle EINTR for
us, even with Go 1.16. That xattr calls are directly affected can be
seen in the report for issue #2968.
2020-11-08 09:34:24 +01:00
Alexander Weiss
826cfa0533 fix context in archiver tests 2020-11-08 08:24:24 +01:00
Alexander Weiss
fef408a8bd Return context error in mem backend 2020-11-08 00:05:53 +01:00
greatroar
a2d4209322 Don't recurse in local backend's List if not required
Due to the return if !isFile, the IsDir branch in List was never taken
and subdirectories were traversed recursively.

Also replaced isFile by an IsRegular check, which has been equivalent
since Go 1.12 (golang/go@a2a3dd00c9).
2020-11-07 08:54:13 +01:00
Alexander Neumann
275f713211 Remove Travis and AppVeyor 2020-11-06 21:49:52 +01:00
MichaelEischer
4707bdb204 Merge pull request #2842 from aawsome/rebuild-index-inmem
Rebuild index in prune by using in-memory index
2020-11-06 20:51:20 +01:00
Alexander Neumann
47277c4b4c Add comments, clarify computation 2020-11-06 20:23:30 +01:00
Alexander Weiss
d2e53730d6 Add test that repo.List is only called once 2020-11-06 20:23:30 +01:00
Alexander Weiss
fd33030556 Use in-memory index to rebuild index in prune 2020-11-06 20:23:30 +01:00
Alexander Weiss
38cc4393f6 Add Masterindex.Save(); Add Index.Packs() 2020-11-06 20:23:30 +01:00
Alexander Neumann
7f6f31c34b Remove Hound 2020-11-06 11:43:58 +01:00
Alexander Neumann
164b4cb2f6 Add changelog for #3014 2020-11-06 10:05:42 +01:00
Alexander Neumann
4a9b05aff1 Merge pull request #3063 from aawsome/fix-3062
Fix #3062
2020-11-05 19:36:40 +01:00
Alexander Weiss
aaf1c44362 Fix #3062 2020-11-05 17:05:42 +01:00
Alexander Neumann
a5592e83f7 Merge pull request #3014 from ivandeex/pr-stream-reset
Fix sporadic stream reset between rclone and restic
2020-11-05 16:11:50 +01:00
Ivan Andreev
ab2790d9de Fix http2 stream reset between restic and rest backends #3014 2020-11-05 15:57:40 +03:00
Alexander Neumann
8a2a326189 Merge pull request #2535 from ncw/fix-2528
s3: add bucket-lookup parameter to select path or dns style bucket lookup
2020-11-05 12:39:34 +01:00
Nick Craig-Wood
86b5d8ffaa s3: add bucket-lookup parameter to select path or dns style bucket lookup
This is to enable restic working with Alibaba cloud

Fixes #2528
2020-11-05 12:20:10 +01:00
Alexander Neumann
636b2f2e94 Merge pull request #2941 from MichaelEischer/parallel-repack
prune: Parallelize repack step
2020-11-05 11:00:41 +01:00
Alexander Neumann
ae5302c7a8 Add comment that keepBlobs is modified 2020-11-05 10:33:38 +01:00
Alexander Neumann
866a52ad4e Remove unneeded seek
The file returned from DownloadAndHash() is already seeked to the start
of the file.
2020-11-05 10:31:49 +01:00
Alexander Neumann
a4507610a0 Fix typo 2020-11-05 10:31:49 +01:00
Alexander Neumann
7def2d8ea7 Use a non-constant seed 2020-11-05 10:31:49 +01:00
Alexander Neumann
ee0112ab3b Clarify message about expected error 2020-11-05 10:31:49 +01:00
Michael Eischer
b373f164fe prune: Parallelize repack command 2020-11-05 10:31:49 +01:00
Alexander Neumann
8a0dbe7c1a helpers: Remove old changelog files for release 2020-11-05 10:26:00 +01:00
Alexander Neumann
4e3ad8b3b1 Remove changelog files from 0.11.0 release 2020-11-05 10:22:39 +01:00
Alexander Neumann
5144141321 Merge pull request #2718 from aawsome/new-cleanup-command
Reimplementation of prune
2020-11-05 10:12:19 +01:00
Alexander Neumann
d35d279455 Set development version for 0.11.0 2020-11-05 09:41:40 +01:00
Alexander Neumann
1ca60bccfb Refactor condition for MaxRepackBytes
Don't depend on the string (opts.MaxRepackSize) for the condition,
instead check if there's a (positive) limit configured.
2020-11-03 16:42:21 +01:00
Alexander Neumann
7f86eb4ec0 Move helper function 2020-11-03 16:42:21 +01:00
Alexander Neumann
c1a3de4a6e Refactor max-unused calculation, add unlimited option
Add a callback to the PruneOptions struct which calculates the number of
bytes allowed to be unused after prune is done. This way, the logic is
closer to the option parsing code.

Also, add an explicit option `unlimited` for the use case when storage
does not matter but bandwidth and time do. Internally, this sets the
maximum number of unused bytes to MaxUint64.

Rework the documentation slightly so that no more "packs" are
mentioned and it talks about "files" instead.

Make it clear in the documentation that the percentage given to
`--max-unused` is relative to the whole repository size after pruning is
done. If specified, it must be below 100%, otherwise the repository
would contain 100% of unused data, which is pointless.

I had a hard time coming up with the correct formula to calculate the
maximum number of unused bytes based on the number of used bytes. For a
fraction `p` (0 ≤ p < 1), a repo with `u` bytes used, and the number of
unused bytes `x` the following holds:

      x ≤ p * (u+x)
    ⇔ x ≤ p*u + p*x
    ⇔ x - p*x ≤ p*u
    ⇔ x * (1-p) ≤ p*u
    ⇔ x ≤ p/(1-p) * u
2020-11-03 16:42:21 +01:00
Alexander Neumann
f8c4dd7b1a Split packe rewrite logic into two case branches
The comma is too sublte, let's split this into two separate branches.
2020-11-03 16:42:21 +01:00
Alexander Neumann
a5b80452fe Add comment that usedBlobs is modified 2020-11-03 16:42:21 +01:00
Alexander Neumann
aff1e220f5 Split struct members, add comments 2020-11-03 16:42:21 +01:00
Alexander Neumann
095155d9ce Remove RepackSmall 2020-11-03 16:42:21 +01:00
Alexander Neumann
1dd9fdce74 Reword changelog slightly 2020-11-03 16:42:21 +01:00
Alexander Weiss
b2f5381737 Make realistic forget --prune --dryrun 2020-11-03 16:42:21 +01:00
Alexander Weiss
7f9a0a5907 Reimplementation of prune 2020-11-03 16:42:21 +01:00
Alexander Weiss
3b591ed987 Add Verboseff 2020-11-03 16:42:21 +01:00
Alexander Weiss
ce7d613749 Add Blob.Handle() 2020-11-03 16:42:21 +01:00
Alexander Weiss
581d90cf91 Make some pack parameters public 2020-11-03 16:42:21 +01:00
greatroar
55c3a90a0d Clarify rclone-over-SSH docs
Also added a link to S. Ruderich's blog post explaining append-only
repos using rclone and SSH.
2020-06-17 15:22:20 +02:00
Alexander Weiss
d3c59d18e5 Fix inconsistency of saving/loading config file
Fix saving/loading config file: Always set ID to a zero ID.
2020-06-13 16:30:23 +02:00
651 changed files with 33971 additions and 17435 deletions

12
.dockerignore Normal file
View File

@@ -0,0 +1,12 @@
# Folders
.git/
.github/
changelog/
doc/
docker/
helpers/
# Files
.gitignore
.golangci.yml
*.md

2
.gitattributes vendored Normal file
View File

@@ -0,0 +1,2 @@
# Workaround for https://github.com/golang/go/issues/52268.
**/testdata/fuzz/*/* eol=lf

View File

@@ -1,13 +1,7 @@
<!-- <!--
Thank you very much for contributing code or documentation to restic! Please Thank you very much for contributing code or documentation to restic! Please
fill out the following questions to make it easier for us to review your fill out the following questions to make it easier for us to review your
changes. changes.
You do not need to check all the boxes below all at once, feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box.
--> -->
What does this PR change? What problem does it solve? What does this PR change? What problem does it solve?
@@ -17,8 +11,8 @@ What does this PR change? What problem does it solve?
Describe the changes and their purpose here, as detailed as needed. Describe the changes and their purpose here, as detailed as needed.
--> -->
Was the change discussed in an issue or in the forum before? Was the change previously discussed in an issue or on the forum?
------------------------------------------------------------ ----------------------------------------------------------------
<!-- <!--
Link issues and relevant forum posts here. Link issues and relevant forum posts here.
@@ -30,11 +24,17 @@ is closed automatically when this PR is merged.
Checklist Checklist
--------- ---------
- [ ] I have read the [Contribution Guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches) <!--
- [ ] I have enabled [maintainer edits for this PR](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork) You do not need to check all the boxes below all at once. Feel free to take
- [ ] I have added tests for all changes in this PR your time and add more commits. If you're done and ready for review, please
- [ ] I have added documentation for the changes (in the manual) check the last box. Enable a checkbox by replacing [ ] with [x].
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (template [here](https://github.com/restic/restic/blob/master/changelog/TEMPLATE)) -->
- [ ] I have run `gofmt` on the code in all commits
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits) - [ ] I have read the [contribution guidelines](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#providing-patches).
- [ ] I'm done, this Pull Request is ready for review - [ ] I have [enabled maintainer edits](https://help.github.com/en/github/collaborating-with-issues-and-pull-requests/allowing-changes-to-a-pull-request-branch-created-from-a-fork).
- [ ] I have added tests for all code changes.
- [ ] I have added documentation for relevant changes (in the manual).
- [ ] There's a new file in `changelog/unreleased/` that describes the changes for our users (see [template](https://github.com/restic/restic/blob/master/changelog/TEMPLATE)).
- [ ] I have run `gofmt` on the code in all commits.
- [ ] All commit messages are formatted in the same style as [the other commits in the repo](https://github.com/restic/restic/blob/master/CONTRIBUTING.md#git-commits).
- [ ] I'm done! This pull request is ready for review.

13
.github/dependabot.yml vendored Normal file
View File

@@ -0,0 +1,13 @@
version: 2
updates:
# Dependencies listed in go.mod
- package-ecosystem: "gomod"
directory: "/" # Location of package manifests
schedule:
interval: "weekly"
# Dependencies listed in .github/workflows/*.yml
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"

311
.github/workflows/tests.yml vendored Normal file
View File

@@ -0,0 +1,311 @@
name: test
on:
# run tests on push to master, but not when other branches are pushed to
push:
branches:
- master
# run tests for all pull requests
pull_request:
env:
latest_go: "1.19.x"
GO111MODULE: on
jobs:
test:
strategy:
matrix:
# list of jobs to run:
include:
- job_name: Windows
go: 1.19.x
os: windows-latest
- job_name: macOS
go: 1.19.x
os: macOS-latest
test_fuse: false
- job_name: Linux
go: 1.19.x
os: ubuntu-latest
test_cloud_backends: true
test_fuse: true
check_changelog: true
- job_name: Linux (race)
go: 1.19.x
os: ubuntu-latest
test_fuse: true
test_opts: "-race"
- job_name: Linux
go: 1.18.x
os: ubuntu-latest
test_fuse: true
name: ${{ matrix.job_name }} Go ${{ matrix.go }}
runs-on: ${{ matrix.os }}
env:
GOPROXY: https://proxy.golang.org
steps:
- name: Set up Go ${{ matrix.go }}
uses: actions/setup-go@v3
with:
go-version: ${{ matrix.go }}
- name: Get programs (Linux/macOS)
run: |
echo "build Go tools"
go install github.com/restic/rest-server/cmd/rest-server@latest
echo "install minio server"
mkdir $HOME/bin
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/darwin-amd64/minio
else
wget --no-verbose -O $HOME/bin/minio https://dl.minio.io/server/minio/release/linux-amd64/minio
fi
chmod 755 $HOME/bin/minio
echo "install rclone"
if [ "$RUNNER_OS" == "macOS" ]; then
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-osx-amd64.zip
else
wget --no-verbose -O rclone.zip https://downloads.rclone.org/rclone-current-linux-amd64.zip
fi
unzip rclone.zip
cp rclone*/rclone $HOME/bin
chmod 755 $HOME/bin/rclone
rm -rf rclone*
# add $HOME/bin to path ($GOBIN was already added to the path by setup-go@v3)
echo $HOME/bin >> $GITHUB_PATH
if: matrix.os == 'ubuntu-latest' || matrix.os == 'macOS-latest'
- name: Get programs (Windows)
shell: powershell
run: |
$ProgressPreference = 'SilentlyContinue'
echo "build Go tools"
go install github.com/restic/rest-server/...
echo "install minio server"
mkdir $Env:USERPROFILE/bin
Invoke-WebRequest https://dl.minio.io/server/minio/release/windows-amd64/minio.exe -OutFile $Env:USERPROFILE/bin/minio.exe
echo "install rclone"
Invoke-WebRequest https://downloads.rclone.org/rclone-current-windows-amd64.zip -OutFile rclone.zip
unzip rclone.zip
copy rclone*/rclone.exe $Env:USERPROFILE/bin
# add $USERPROFILE/bin to path ($GOBIN was already added to the path by setup-go@v3)
echo $Env:USERPROFILE\bin >> $Env:GITHUB_PATH
echo "install tar"
cd $env:USERPROFILE
mkdir tar
cd tar
# install exactly these versions of tar and the libraries, other combinations might not work!
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/tar-1.13-1-bin.zip -OutFile tar.zip
unzip tar.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libintl-0.11.5-2-bin.zip -OutFile libintl.zip
unzip libintl.zip
Invoke-WebRequest https://github.com/restic/test-assets/raw/master/libiconv-1.8-1-bin.zip -OutFile libiconv.zip
unzip libiconv.zip
# add $USERPROFILE/tar/bin to path
echo $Env:USERPROFILE\tar\bin >> $Env:GITHUB_PATH
if: matrix.os == 'windows-latest'
- name: Check out code
uses: actions/checkout@v3
- name: Build with build.go
run: |
go run build.go
- name: Run local Tests
env:
RESTIC_TEST_FUSE: ${{ matrix.test_fuse }}
run: |
go test -cover ${{matrix.test_opts}} ./...
- name: Test cloud backends
env:
RESTIC_TEST_S3_KEY: ${{ secrets.RESTIC_TEST_S3_KEY }}
RESTIC_TEST_S3_SECRET: ${{ secrets.RESTIC_TEST_S3_SECRET }}
RESTIC_TEST_S3_REPOSITORY: ${{ secrets.RESTIC_TEST_S3_REPOSITORY }}
RESTIC_TEST_AZURE_ACCOUNT_NAME: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_NAME }}
RESTIC_TEST_AZURE_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_AZURE_ACCOUNT_KEY }}
RESTIC_TEST_AZURE_REPOSITORY: ${{ secrets.RESTIC_TEST_AZURE_REPOSITORY }}
RESTIC_TEST_B2_ACCOUNT_ID: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_ID }}
RESTIC_TEST_B2_ACCOUNT_KEY: ${{ secrets.RESTIC_TEST_B2_ACCOUNT_KEY }}
RESTIC_TEST_B2_REPOSITORY: ${{ secrets.RESTIC_TEST_B2_REPOSITORY }}
RESTIC_TEST_GS_REPOSITORY: ${{ secrets.RESTIC_TEST_GS_REPOSITORY }}
RESTIC_TEST_GS_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
GOOGLE_PROJECT_ID: ${{ secrets.RESTIC_TEST_GS_PROJECT_ID }}
RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64: ${{ secrets.RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 }}
RESTIC_TEST_OS_AUTH_URL: ${{ secrets.RESTIC_TEST_OS_AUTH_URL }}
RESTIC_TEST_OS_TENANT_NAME: ${{ secrets.RESTIC_TEST_OS_TENANT_NAME }}
RESTIC_TEST_OS_USERNAME: ${{ secrets.RESTIC_TEST_OS_USERNAME }}
RESTIC_TEST_OS_PASSWORD: ${{ secrets.RESTIC_TEST_OS_PASSWORD }}
RESTIC_TEST_OS_REGION_NAME: ${{ secrets.RESTIC_TEST_OS_REGION_NAME }}
RESTIC_TEST_SWIFT: ${{ secrets.RESTIC_TEST_SWIFT }}
# fail if any of the following tests cannot be run
RESTIC_TEST_DISALLOW_SKIP: "restic/backend/rest.TestBackendREST,\
restic/backend/sftp.TestBackendSFTP,\
restic/backend/s3.TestBackendMinio,\
restic/backend/rclone.TestBackendRclone,\
restic/backend/s3.TestBackendS3,\
restic/backend/swift.TestBackendSwift,\
restic/backend/b2.TestBackendB2,\
restic/backend/gs.TestBackendGS,\
restic/backend/azure.TestBackendAzure"
run: |
# prepare credentials for Google Cloud Storage tests in a temp file
export GOOGLE_APPLICATION_CREDENTIALS=$(mktemp --tmpdir restic-gcs-auth-XXXXXXX)
echo $RESTIC_TEST_GS_APPLICATION_CREDENTIALS_B64 | base64 -d > $GOOGLE_APPLICATION_CREDENTIALS
go test -cover -parallel 4 ./internal/backend/...
# only run cloud backend tests for pull requests from and pushes to our
# own repo, otherwise the secrets are not available
# Skip for Dependabot pull requests as these are run without secrets
# https://docs.github.com/en/code-security/dependabot/working-with-dependabot/automating-dependabot-with-github-actions#responding-to-events
if: (github.event_name == 'push' || github.event.pull_request.head.repo.full_name == github.repository) && (github.actor != 'dependabot[bot]') && matrix.test_cloud_backends
- name: Check changelog files with calens
run: |
echo "install calens"
go install github.com/restic/calens@latest
echo "check changelog files"
calens
if: matrix.check_changelog
cross_compile:
strategy:
# ATTENTION: the list of architectures must be in sync with helpers/build-release-binaries/main.go!
matrix:
# run cross-compile in three batches parallel so the overall tests run faster
targets:
- "linux/386 linux/amd64 linux/arm linux/arm64 linux/ppc64le linux/mips linux/mipsle linux/mips64 linux/mips64le linux/riscv64 linux/s390x"
- "openbsd/386 openbsd/amd64 \
freebsd/386 freebsd/amd64 freebsd/arm \
aix/ppc64 \
darwin/amd64 darwin/arm64"
- "netbsd/386 netbsd/amd64 \
windows/386 windows/amd64 \
solaris/amd64"
env:
GOPROXY: https://proxy.golang.org
runs-on: ubuntu-latest
name: Cross Compile for ${{ matrix.targets }}
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v3
with:
go-version: ${{ env.latest_go }}
- name: Install gox
run: |
go install github.com/mitchellh/gox@latest
- name: Check out code
uses: actions/checkout@v3
- name: Cross-compile with gox for ${{ matrix.targets }}
env:
GOFLAGS: "-trimpath"
GOX_ARCHS: "${{ matrix.targets }}"
run: |
mkdir build-output
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}" ./cmd/restic
gox -parallel 2 -verbose -osarch "$GOX_ARCHS" -tags debug -output "build-output/{{.Dir}}_{{.OS}}_{{.Arch}}_debug" ./cmd/restic
lint:
name: lint
runs-on: ubuntu-latest
steps:
- name: Set up Go ${{ env.latest_go }}
uses: actions/setup-go@v3
with:
go-version: ${{ env.latest_go }}
- name: Check out code
uses: actions/checkout@v3
- name: golangci-lint
uses: golangci/golangci-lint-action@v3
with:
# Required: the version of golangci-lint is required and must be specified without patch version: we always use the latest patch version.
version: v1.49
# Optional: show only new issues if it's a pull request. The default value is `false`.
only-new-issues: true
args: --verbose --timeout 5m
# only run golangci-lint for pull requests, otherwise ALL hints get
# reported. We need to slowly address all issues until we can enable
# linting the master branch :)
if: github.event_name == 'pull_request'
- name: Check go.mod/go.sum
run: |
echo "check if go.mod and go.sum are up to date"
go mod tidy
git diff --exit-code go.mod go.sum
docker:
name: docker
runs-on: ubuntu-latest
steps:
- name: Check out code
uses: actions/checkout@v3
- name: Docker meta
id: meta
uses: docker/metadata-action@v4
with:
# list of Docker images to use as base name for tags
images: |
restic/restic
# generate Docker tags based on the following events/attributes
tags: |
type=schedule
type=ref,event=branch
type=ref,event=pr
type=semver,pattern={{version}}
type=semver,pattern={{major}}.{{minor}}
type=semver,pattern={{major}}
type=sha
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Build and push
id: docker_build
uses: docker/build-push-action@v3
with:
push: false
context: .
file: docker/Dockerfile
pull: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

59
.golangci.yml Normal file
View File

@@ -0,0 +1,59 @@
# This is the configuration for golangci-lint for the restic project.
#
# A sample config with all settings is here:
# https://github.com/golangci/golangci-lint/blob/master/.golangci.example.yml
linters:
# only enable the linters listed below
disable-all: true
enable:
# make sure all errors returned by functions are handled
- errcheck
# find unused code
- deadcode
# show how code can be simplified
- gosimple
# # make sure code is formatted
- gofmt
# examine code and report suspicious constructs, such as Printf calls whose
# arguments do not align with the format string
- govet
# make sure names and comments are used according to the conventions
- revive
# detect when assignments to existing variables are not used
- ineffassign
# run static analysis and find errors
- staticcheck
# find unused variables, functions, structs, types, etc.
- unused
# find unused struct fields
- structcheck
# find unused global variables
- varcheck
# parse and typecheck code
- typecheck
issues:
# don't use the default exclude rules, this hides (among others) ignored
# errors from Close() calls
exclude-use-default: false
# list of things to not warn about
exclude:
# revive: do not warn about missing comments for exported stuff
- exported (function|method|var|type|const) .* should have comment or be unexported
# revive: ignore constants in all caps
- don't use ALL_CAPS in Go names; use CamelCase
# revive: lots of packages don't have such a comment
- "package-comments: should have a package comment"

View File

@@ -1,2 +0,0 @@
go:
enabled: true

View File

@@ -1,58 +0,0 @@
language: go
sudo: false
matrix:
include:
- os: linux
go: "1.13.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: linux
go: "1.14.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
# only run fuse and cloud backends tests on Travis for the latest Go on Linux
- os: linux
go: "1.15.x"
sudo: true
cache:
directories:
- $HOME/.cache/go-build
- $HOME/gopath/pkg/mod
- os: osx
go: "1.15.x"
env: RESTIC_TEST_FUSE=0 RESTIC_TEST_CLOUD_BACKENDS=0
cache:
directories:
- $HOME/Library/Caches/go-build
- $HOME/gopath/pkg/mod
branches:
only:
- master
notifications:
irc:
channels:
- "chat.freenode.net#restic"
on_success: change
on_failure: change
skip_join: true
install:
- go version
- export GOBIN="$GOPATH/bin"
- export PATH="$PATH:$GOBIN"
- go env
script:
- go run run_integration_tests.go

File diff suppressed because it is too large Load Diff

View File

@@ -13,11 +13,10 @@ bug fixes are most welcome. However even "minor" details as fixing spelling
errors, improving documentation or pointing out usability issues are a great errors, improving documentation or pointing out usability issues are a great
help also. help also.
The restic project uses the GitHub infrastructure (see the The restic project uses the GitHub infrastructure (see the
[project page](https://github.com/restic/restic)) for all related discussions [project page](https://github.com/restic/restic)) for all related discussions
as well as the [forum](https://forum.restic.net/) and the `#restic` channel as well as the [forum](https://forum.restic.net/) and the `#restic` channel
on [irc.freenode.net](https://kiwiirc.com/nextclient/irc.freenode.net/restic). on [irc.libera.chat](https://kiwiirc.com/nextclient/#ircs://irc.libera.chat:6697/#restic).
If you want to find an area that currently needs improving have a look at the If you want to find an area that currently needs improving have a look at the
open issues listed at the open issues listed at the
@@ -49,9 +48,8 @@ environment was used and so on. Please tell us at least the following things:
Remember, the easier it is for us to reproduce the bug, the earlier it will be Remember, the easier it is for us to reproduce the bug, the earlier it will be
corrected! corrected!
In addition, you can compile restic with debug support by running In addition, you can instruct restic to create a debug log by setting the
`go run build.go -tags debug` and instructing it to create a debug environment variable `DEBUG_LOG` to a file, e.g. like this:
log by setting the environment variable `DEBUG_LOG` to a file, e.g. like this:
$ export DEBUG_LOG=/tmp/restic-debug.log $ export DEBUG_LOG=/tmp/restic-debug.log
$ restic backup ~/work $ restic backup ~/work
@@ -67,8 +65,8 @@ Development Environment
The repository contains the code written for restic in the directories The repository contains the code written for restic in the directories
`cmd/` and `internal/`. `cmd/` and `internal/`.
Restic requires Go version 1.13 or later for compiling. Clone the repo (without Make sure you have the minimum required Go version installed. Clone the repo
having `$GOPATH` set) and `cd` into the directory: (without having `$GOPATH` set) and `cd` into the directory:
$ unset GOPATH $ unset GOPATH
$ git clone https://github.com/restic/restic $ git clone https://github.com/restic/restic
@@ -78,7 +76,7 @@ Then use the `go` tool to build restic:
$ go build ./cmd/restic $ go build ./cmd/restic
$ ./restic version $ ./restic version
restic 0.10.0-dev (compiled manually) compiled with go1.15.2 on linux/amd64 restic 0.14.0-dev (compiled manually) compiled with go1.19 on linux/amd64
You can run all tests with the following command: You can run all tests with the following command:
@@ -124,7 +122,10 @@ down to the following steps:
writing, ask yourself: If I were the user, what would I need to be aware writing, ask yourself: If I were the user, what would I need to be aware
of with this change? of with this change?
8. Once your code looks good and passes all the tests, we'll merge it. Thanks 8. Do not edit the man pages under `doc/man` or `doc/manual_rest.rst` -
these are autogenerated before new releases.
9. Once your code looks good and passes all the tests, we'll merge it. Thanks
a lot for your contribution! a lot for your contribution!
Please provide the patches for each bug or feature in a separate branch and Please provide the patches for each bug or feature in a separate branch and
@@ -141,6 +142,14 @@ Installing the script `fmt-check` from https://github.com/edsrzf/gofmt-git-hook
locally as a pre-commit hook checks formatting before committing automatically, locally as a pre-commit hook checks formatting before committing automatically,
just copy this script to `.git/hooks/pre-commit`. just copy this script to `.git/hooks/pre-commit`.
The project is using the program
[`golangci-lint`](https://github.com/golangci/golangci-lint) to run a list of
linters and checkers. It will be run on the code when you submit a PR. In order
to check your code beforehand, you can run `golangci-lint run` manually.
Eventually, we will enable `golangci-lint` for the whole code base. For now,
you can ignore warnings printed for lines you did not modify, those will be
ignored by the CI.
For each pull request, several different systems run the integration tests on For each pull request, several different systems run the integration tests on
Linux, macOS and Windows. We won't merge any code that does not pass all tests Linux, macOS and Windows. We won't merge any code that does not pass all tests
for all systems, so when a tests fails, try to find out what's wrong and fix for all systems, so when a tests fails, try to find out what's wrong and fix

View File

@@ -1,6 +1,5 @@
[![Documentation](https://readthedocs.org/projects/restic/badge/?version=latest)](https://restic.readthedocs.io/en/latest/?badge=latest) [![Documentation](https://readthedocs.org/projects/restic/badge/?version=latest)](https://restic.readthedocs.io/en/latest/?badge=latest)
[![Build Status Travis](https://travis-ci.com/restic/restic.svg?branch=master)](https://travis-ci.com/restic/restic) [![Build Status](https://github.com/restic/restic/workflows/test/badge.svg)](https://github.com/restic/restic/actions?query=workflow%3Atest)
[![Build Status AppVeyor](https://ci.appveyor.com/api/projects/status/nuy4lfbgfbytw92q/branch/master?svg=true)](https://ci.appveyor.com/project/fd0/restic/branch/master)
[![Go Report Card](https://goreportcard.com/badge/github.com/restic/restic)](https://goreportcard.com/report/github.com/restic/restic) [![Go Report Card](https://goreportcard.com/badge/github.com/restic/restic)](https://goreportcard.com/report/github.com/restic/restic)
# Introduction # Introduction
@@ -47,8 +46,8 @@ Therefore, restic supports the following backends for storing backups natively:
- [Local directory](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local) - [Local directory](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#local)
- [sftp server (via SSH)](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp) - [sftp server (via SSH)](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#sftp)
- [HTTP REST server](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) ([protocol](doc/100_references.rst#rest-backend), [rest-server](https://github.com/restic/rest-server)) - [HTTP REST server](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) ([protocol](https://restic.readthedocs.io/en/latest/100_references.html#rest-backend), [rest-server](https://github.com/restic/rest-server))
- [AWS S3](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) (either from Amazon or using the [Minio](https://minio.io) server) - [Amazon S3](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) (either from Amazon or using the [Minio](https://minio.io) server)
- [OpenStack Swift](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift) - [OpenStack Swift](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#openstack-swift)
- [BackBlaze B2](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2) - [BackBlaze B2](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#backblaze-b2)
- [Microsoft Azure Blob Storage](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage) - [Microsoft Azure Blob Storage](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#microsoft-azure-blob-storage)

View File

@@ -1 +1 @@
0.11.0 0.15.2

View File

@@ -1,32 +0,0 @@
clone_folder: c:\restic
environment:
GOPATH: c:\gopath
branches:
only:
- master
cache:
- '%LocalAppData%\go-build'
init:
- ps: >-
$app = Get-WmiObject -Class Win32_Product -Filter "Vendor = 'http://golang.org'"
if ($app) {
$app.Uninstall()
}
install:
- rmdir c:\go /s /q
- appveyor DownloadFile https://dl.google.com/go/go1.15.2.windows-amd64.msi
- msiexec /i go1.15.2.windows-amd64.msi /q
- go version
- go env
- appveyor DownloadFile https://sourceforge.netcologne.de/project/gnuwin32/tar/1.13-1/tar-1.13-1-bin.zip -FileName tar.zip
- 7z x tar.zip bin/tar.exe
- set PATH=bin/;%PATH%
build_script:
- go run run_integration_tests.go

View File

@@ -3,8 +3,8 @@
// This program aims to make building Go programs for end users easier by just // This program aims to make building Go programs for end users easier by just
// calling it with `go run`, without having to setup a GOPATH. // calling it with `go run`, without having to setup a GOPATH.
// //
// This program needs Go >= 1.12. It'll use Go modules for compilation. It // This program checks for a minimum Go version. It will use Go modules for
// builds the package configured as Main in the Config struct. // compilation. It builds the package configured as Main in the Config struct.
// BSD 2-Clause License // BSD 2-Clause License
// //
@@ -35,6 +35,7 @@
// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE // OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. // OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
//go:build ignore_build_go
// +build ignore_build_go // +build ignore_build_go
package main package main
@@ -42,7 +43,6 @@ package main
import ( import (
"fmt" "fmt"
"io" "io"
"io/ioutil"
"os" "os"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
@@ -58,7 +58,7 @@ var config = Config{
Main: "./cmd/restic", // package name for the main package Main: "./cmd/restic", // package name for the main package
DefaultBuildTags: []string{"selfupdate"}, // specify build tags which are always used DefaultBuildTags: []string{"selfupdate"}, // specify build tags which are always used
Tests: []string{"./..."}, // tests to run Tests: []string{"./..."}, // tests to run
MinVersion: GoVersion{Major: 1, Minor: 11, Patch: 0}, // minimum Go version supported MinVersion: GoVersion{Major: 1, Minor: 18, Patch: 0}, // minimum Go version supported
} }
// Config configures the build. // Config configures the build.
@@ -123,17 +123,8 @@ func printEnv(env []string) {
// build runs "go build args..." with GOPATH set to gopath. // build runs "go build args..." with GOPATH set to gopath.
func build(cwd string, env map[string]string, args ...string) error { func build(cwd string, env map[string]string, args ...string) error {
a := []string{"build"} // -trimpath removes all absolute paths from the binary.
a := []string{"build", "-trimpath"}
// try to remove all absolute paths from resulting binary
if goVersion.AtLeast(GoVersion{1, 13, 0}) {
// use the new flag introduced by Go 1.13
a = append(a, "-trimpath")
} else {
// otherwise try to trim as many paths as possible
a = append(a, "-asmflags", fmt.Sprintf("all=-trimpath=%s", cwd))
a = append(a, "-gcflags", fmt.Sprintf("all=-trimpath=%s", cwd))
}
if enablePIE { if enablePIE {
a = append(a, "-buildmode=pie") a = append(a, "-buildmode=pie")
@@ -187,7 +178,7 @@ func test(cwd string, env map[string]string, args ...string) error {
// getVersion returns the version string from the file VERSION in the current // getVersion returns the version string from the file VERSION in the current
// directory. // directory.
func getVersionFromFile() string { func getVersionFromFile() string {
buf, err := ioutil.ReadFile("VERSION") buf, err := os.ReadFile("VERSION")
if err != nil { if err != nil {
verbosePrintf("error reading file VERSION: %v\n", err) verbosePrintf("error reading file VERSION: %v\n", err)
return "" return ""
@@ -327,12 +318,8 @@ func (v GoVersion) String() string {
} }
func main() { func main() {
if !goVersion.AtLeast(GoVersion{1, 12, 0}) {
die("Go version (%v) is too old, restic requires Go >= 1.12\n", goVersion)
}
if !goVersion.AtLeast(config.MinVersion) { if !goVersion.AtLeast(config.MinVersion) {
fmt.Fprintf(os.Stderr, "%s detected, this program requires at least %s\n", goVersion, config.MinVersion) fmt.Fprintf(os.Stderr, "Detected version %s is too old, restic requires at least %s\n", goVersion, config.MinVersion)
os.Exit(1) os.Exit(1)
} }

View File

@@ -0,0 +1,10 @@
Bugfix: Make `mount` not create missing mount point directory
When specifying a non-existent directory as mount point for the `mount`
command, restic used to create the specified directory automatically.
This has now changed such that restic instead gives an error when the
specified directory for the mount point does not exist.
https://github.com/restic/restic/issues/1681
https://github.com/restic/restic/pull/3008

View File

@@ -0,0 +1,8 @@
Bugfix: Ignore `no data available` filesystem error during backup
Restic was unable to backup files on some filesystems, for example certain
configurations of CIFS on Linux which return a `no data available` error
when reading extended attributes. These errors are now ignored.
https://github.com/restic/restic/issues/1800
https://github.com/restic/restic/pull/3034

View File

@@ -0,0 +1,8 @@
Enhancement: Allow specifying percentage in `check --read-data-subset`
We've enhanced the `check` command's `--read-data-subset` option to also accept
a percentage (e.g. `2.5%` or `10%`). This will check the given percentage of
pack files (which are randomly selected on each run).
https://github.com/restic/restic/issues/2186
https://github.com/restic/restic/pull/3038

View File

@@ -0,0 +1,14 @@
Enhancement: Report permanent/fatal backend errors earlier
When encountering errors in reading from or writing to storage backends,
restic retries the failing operation up to nine times (for a total of ten
attempts). It used to retry all backend operations, but now detects some
permanent error conditions so that it can report fatal errors earlier.
Permanent failures include local disks being full, SSH connections
dropping and permission errors.
https://github.com/restic/restic/issues/2453
https://github.com/restic/restic/pull/3170
https://github.com/restic/restic/issues/3180
https://github.com/restic/restic/pull/3181

View File

@@ -0,0 +1,21 @@
Enhancement: Add Alibaba/Aliyun OSS support in the `s3` backend
A new extended option `s3.bucket-lookup` has been added to support
Alibaba/Aliyun OSS in the `s3` backend. The option can be set to one
of the following values:
- `auto` - Existing behaviour
- `dns` - Use DNS style bucket access
- `path` - Use path style bucket access
To make the `s3` backend work with Alibaba/Aliyun OSS you must set
`s3.bucket-lookup` to `dns` and set the `s3.region` parameter. For
example:
restic -o s3.bucket-lookup=dns -o s3.region=oss-eu-west-1 -r s3:https://oss-eu-west-1.aliyuncs.com/bucketname init
Note that `s3.region` must be set, otherwise the MinIO SDK tries to
look it up and it seems that Alibaba doesn't support that properly.
https://github.com/restic/restic/issues/2528
https://github.com/restic/restic/pull/2535

View File

@@ -0,0 +1,9 @@
Bugfix: Report the correct owner of directories in FUSE mounts
Restic 0.10.0 changed the FUSE mount to always report the current user
as the owner of directories within the FUSE mount, which is incorrect.
This is now changed back to reporting the correct owner of a directory.
https://github.com/restic/restic/issues/2563
https://github.com/restic/restic/pull/3141

View File

@@ -0,0 +1,31 @@
Bugfix: Make `backup` and `tag` commands separate tags by comma
Running `restic backup --tag foo,bar` previously created snapshots with one
single tag containing a comma (`foo,bar`) instead of two tags (`foo`, `bar`).
Similarly, the `tag` command's `--set`, `--add` and `--remove` options would
treat `foo,bar` as one tag instead of two tags. This was inconsistent with
other commands and often unexpected when one intended `foo,bar` to mean two
tags.
To be consistent in all commands, restic now interprets `foo,bar` to mean two
separate tags (`foo` and `bar`) instead of one tag (`foo,bar`) everywhere,
including in the `backup` and `tag` commands.
NOTE: This change might result in unexpected behavior in cases where you use
the `forget` command and filter on tags like `foo,bar`. Snapshots previously
backed up with `--tag foo,bar` will still not match that filter, but snapshots
saved from now on will match that filter.
To replace `foo,bar` tags with `foo` and `bar` tags in old snapshots, you can
first generate a list of the relevant snapshots using a command like:
restic snapshots --json --quiet | jq '.[] | select(contains({tags: ["foo,bar"]})) | .id'
and then use `restic tag --set foo --set bar snapshotID [...]` to set the new
tags. Please adjust the commands to include real tag names and any additional
tags, as well as the list of snapshots to process.
https://github.com/restic/restic/issues/2688
https://github.com/restic/restic/pull/2690
https://github.com/restic/restic/pull/3197

View File

@@ -0,0 +1,17 @@
Enhancement: Configurable progress reports for non-interactive terminals
The `backup`, `check` and `prune` commands never printed any progress
reports on non-interactive terminals. This behavior is now configurable
using the `RESTIC_PROGRESS_FPS` environment variable. Use for example a
value of `1` for an update every second, or `0.01666` for an update every
minute.
The `backup` command now also prints the current progress when restic
receives a `SIGUSR1` signal.
Setting the `RESTIC_PROGRESS_FPS` environment variable or sending a `SIGUSR1`
signal prints a status report even when `--quiet` was specified.
https://github.com/restic/restic/issues/2706
https://github.com/restic/restic/issues/3194
https://github.com/restic/restic/pull/3199

View File

@@ -0,0 +1,5 @@
Bugfix: Make the `cat` command respect the `--no-lock` option
The `cat` command would not respect the `--no-lock` flag. This is now fixed.
https://github.com/restic/restic/issues/2739

View File

@@ -0,0 +1,13 @@
Enhancement: Add `backup` options `--files-from-{verbatim,raw}`
The new `backup` options `--files-from-verbatim` and `--files-from-raw` read a
list of files to back up from a file. Unlike the existing `--files-from`
option, these options do not interpret the listed filenames as glob patterns;
instead, whitespace in filenames is preserved as-is and no pattern expansion is
done. Please see the documentation for specifics.
These new options are highly recommended over `--files-from`, when using a
script to generate the list of files to back up.
https://github.com/restic/restic/issues/2944
https://github.com/restic/restic/issues/3013

View File

@@ -0,0 +1,18 @@
Enhancement: Allow usage of deprecated S3 `ListObjects` API
Some S3 API implementations, e.g. Ceph before version 14.2.5, have a broken
`ListObjectsV2` implementation which causes problems for restic when using
their API endpoints. When a broken server implementation is used, restic prints
errors similar to the following:
List() returned error: Truncated response should have continuation token set
As a temporary workaround, restic now allows using the older `ListObjects`
endpoint by setting the `s3.list-objects-v1` extended option, for instance:
restic -o s3.list-objects-v1=true snapshots
Please note that this option may be removed in future versions of restic.
https://github.com/restic/restic/issues/3083
https://github.com/restic/restic/pull/3085

View File

@@ -0,0 +1,10 @@
Bugfix: The `--use-fs-snapshot` option now works on windows/386
Restic failed to create VSS snapshots on windows/386 with the following error:
GetSnapshotProperties() failed: E_INVALIDARG (0x80070057)
This is now fixed.
https://github.com/restic/restic/issues/3087
https://github.com/restic/restic/pull/3090

View File

@@ -0,0 +1,17 @@
Change: Deleting files on Google Drive now moves them to the trash
When deleting files on Google Drive via the `rclone` backend, restic used to
bypass the trash folder required that one used the `-o rclone.args` option to
enable usage of the trash folder. This ensured that deleted files in Google
Drive were not kept indefinitely in the trash folder. However, since Google
Drive's trash retention policy changed to deleting trashed files after 30 days,
this is no longer needed.
Restic now leaves it up to rclone and its configuration to use or not use the
trash folder when deleting files. The default is to use the trash folder, as
of rclone 1.53.2. To re-enable the restic 0.11 behavior, set the
`RCLONE_DRIVE_USE_TRASH` environment variable or change the rclone
configuration. See the rclone documentation for more details.
https://github.com/restic/restic/issues/3095
https://github.com/restic/restic/pull/3102

View File

@@ -0,0 +1,10 @@
Bugfix: Do not require gs bucket permissions when running `init`
Restic used to require bucket level permissions for the `gs` backend
in order to initialize a restic repository.
It now allows a `gs` service account to initialize a repository if the
bucket does exist and the service account has permissions to write/read
to that bucket.
https://github.com/restic/restic/issues/3100

View File

@@ -0,0 +1,9 @@
Bugfix: Correctly detect output redirection for `backup` command on Windows
On Windows, since restic 0.10.0 the `backup` command did not properly detect
when the output was redirected to a file. This caused restic to output
terminal control characters. This has been fixed by correcting the terminal
detection.
https://github.com/restic/restic/issues/3111
https://github.com/restic/restic/pull/3150

View File

@@ -0,0 +1,11 @@
Enhancement: Support additional environment variables for Swift authentication
The `swift` backend now supports the following additional environment variables
for passing authentication details to restic:
`OS_USER_ID`, `OS_USER_DOMAIN_ID`, `OS_PROJECT_DOMAIN_ID` and `OS_TRUST_ID`
Depending on the `openrc` configuration file these might be required when the
user and project domains differ from one another.
https://github.com/restic/restic/issues/3147
https://github.com/restic/restic/pull/3158

View File

@@ -0,0 +1,9 @@
Bugfix: Don't create invalid snapshots when `backup` is interrupted
When canceling a backup run at a certain moment it was possible that
restic created a snapshot with an invalid "null" tree. This caused
`check` and other operations to fail. The `backup` command now properly
handles interruptions and never saves a snapshot when interrupted.
https://github.com/restic/restic/issues/3151
https://github.com/restic/restic/pull/3164

View File

@@ -0,0 +1,9 @@
Bugfix: Improve error handling in the `restore` command
The `restore` command used to not print errors while downloading file contents
from the repository. It also incorrectly exited with a zero error code even
when there were errors during the restore process. This has all been fixed and
`restore` now returns with a non-zero exit code when there's an error.
https://github.com/restic/restic/issues/3166
https://github.com/restic/restic/pull/3207

View File

@@ -0,0 +1,8 @@
Enhancement: Add release binaries for MIPS architectures
We've added a few new architectures for Linux to the release binaries: `mips`,
`mipsle`, `mips64`, and `mip64le`. MIPS is mostly used for low-end embedded
systems.
https://github.com/restic/restic/issues/3191
https://github.com/restic/restic/pull/3208

View File

@@ -0,0 +1,11 @@
Bugfix: Correct statistics for overlapping targets
A user reported that restic's statistics and progress information during backup
was not correctly calculated when the backup targets (files/dirs to save)
overlap. For example, consider a directory `foo` which contains (among others)
a file `foo/bar`. When `restic backup foo foo/bar` was run, restic counted the
size of the file `foo/bar` twice, so the completeness percentage as well as the
number of files was wrong. This is now corrected.
https://github.com/restic/restic/issues/3232
https://github.com/restic/restic/pull/3243

View File

@@ -0,0 +1,12 @@
Enhancement: Back up mountpoints as empty directories
When the `--one-file-system` option is specified to `restic backup`, it
ignores all file systems mounted below one of the target directories. This
means that when a snapshot is restored, users needed to manually recreate
the mountpoint directories.
Restic now backs up mountpoints as empty directories and therefore implements
the same approach as `tar`.
https://github.com/restic/restic/issues/909
https://github.com/restic/restic/pull/3119

View File

@@ -0,0 +1,6 @@
Enhancement: Add several more error checks
We've added a lot more error checks in places where errors were previously
ignored (as hinted by the static analysis program `errcheck` via `golangci-lint`).
https://github.com/restic/restic/pull/3250

View File

@@ -0,0 +1,29 @@
Enhancement: Improve `prune` performance and make it more customizable
The `prune` command is now much faster. This is especially the case for remote
repositories or repositories with not much data to remove. Also the memory
usage of the `prune` command is now reduced.
Restic used to rebuild the index from scratch after pruning. This could lead
to missing packs in the index in some cases for eventually consistent backends
such as e.g. AWS S3. This behavior is now changed and the index rebuilding
uses the information already known by `prune`.
By default, the `prune` command no longer removes all unused data. This
behavior can be fine-tuned by new options, like the acceptable amount of
unused space or the maximum size of data to reorganize. For more details,
please see https://restic.readthedocs.io/en/stable/060_forget.html .
Moreover, `prune` now accepts the `--dry-run` option and also running
`forget --dry-run --prune` will show what `prune` would do.
This enhancement also fixes several open issues, e.g.:
- https://github.com/restic/restic/issues/1140
- https://github.com/restic/restic/issues/1599
- https://github.com/restic/restic/issues/1985
- https://github.com/restic/restic/issues/2112
- https://github.com/restic/restic/issues/2227
- https://github.com/restic/restic/issues/2305
https://github.com/restic/restic/pull/2718
https://github.com/restic/restic/pull/2842

View File

@@ -0,0 +1,27 @@
Enhancement: Add option to let `backup` trust mtime without checking ctime
The `backup` command used to require that both `ctime` and `mtime` of a file
matched with a previously backed up version to determine that the file was
unchanged. In other words, if either `ctime` or `mtime` of the file had
changed, it would be considered changed and restic would read the file's
content again to back up the relevant (changed) parts of it.
The new option `--ignore-ctime` makes restic look at `mtime` only, such that
`ctime` changes for a file does not cause restic to read the file's contents
again.
The check for both `ctime` and `mtime` was introduced in restic 0.9.6 to make
backups more reliable in the face of programs that reset `mtime` (some Unix
archivers do that), but it turned out to often be expensive because it made
restic read file contents even if only the metadata (owner, permissions) of
a file had changed. The new `--ignore-ctime` option lets the user restore the
0.9.5 behavior when needed. The existing `--ignore-inode` option already
turned off this behavior, but also removed a different check.
Please note that changes in files' metadata are still recorded, regardless of
the command line options provided to the backup command.
https://github.com/restic/restic/issues/2495
https://github.com/restic/restic/issues/2558
https://github.com/restic/restic/issues/2819
https://github.com/restic/restic/pull/2823

View File

@@ -0,0 +1,8 @@
Enhancement: Speed up the repacking step of the `prune` command
The repack step of the `prune` command, which moves still used file parts into
new pack files such that the old ones can be garbage collected later on, now
processes multiple pack files in parallel. This is especially beneficial for
high latency backends or when using a fast network connection.
https://github.com/restic/restic/pull/2941

View File

@@ -0,0 +1,11 @@
Enhancement: Speed up the `rebuild-index` command
We've optimized the `rebuild-index` command. Now, existing index entries are used
to minimize the number of pack files that must be read. This speeds up the index
rebuild a lot.
Additionally, the option `--read-all-packs` has been added, implementing the
previous behavior.
https://github.com/restic/restic/issue/2547
https://github.com/restic/restic/pull/3006

View File

@@ -0,0 +1,13 @@
Bugfix: Fix sporadic stream reset between rclone and restic
Sometimes when using restic with the `rclone` backend, an error message
similar to the following would be printed:
Didn't finish writing GET request (wrote 0/xxx): http2: stream closed
It was found that this was caused by restic closing the connection to rclone
to soon when downloading data. A workaround has been added which waits for
the end of the download before closing the connection.
https://github.com/restic/restic/pull/3014
https://github.com/rclone/rclone/issues/2598

View File

@@ -0,0 +1,23 @@
Enhancement: Add more checks for index and pack files in the `check` command
The `check` command run with the `--read-data` or `--read-data-subset` options
used to only verify only the pack file content - it did not check if the blobs
within the pack are correctly contained in the index.
A check for the latter is now in place, which can print the following error:
Blob ID is not contained in index or position is incorrect
Another test is also added, which compares pack file sizes computed from the
index and the pack header with the actual file size. This test is able to
detect truncated pack files.
If the index is not correct, it can be rebuilt by using the `rebuild-index`
command.
Having added these tests, `restic check` is now able to detect non-existing
blobs which are wrongly referenced in the index. This situation could have
lead to missing data.
https://github.com/restic/restic/pull/3048
https://github.com/restic/restic/pull/3082

View File

@@ -0,0 +1,8 @@
Enhancement: Make the `dump` command support `zip` format
Previously, restic could dump the contents of a whole folder structure only
in the `tar` format. The `dump` command now has a new flag to change output
format to `zip`. Just pass `--archive zip` as an option to `restic dump`.
https://github.com/restic/restic/pull/2433
https://github.com/restic/restic/pull/3081

View File

@@ -0,0 +1,6 @@
Enhancement: Reduce memory usage of `check` command
The `check` command now requires less memory if it is run without the
`--check-unused` option.
https://github.com/restic/restic/pull/3099

View File

@@ -0,0 +1,10 @@
Enhancement: Parallelize scan of snapshot content in `copy` and `prune`
The `copy` and `prune` commands used to traverse the directories of
snapshots one by one to find used data. This snapshot traversal is
now parallized which can speed up this step several times.
In addition the `check` command now reports how many snapshots have
already been processed.
https://github.com/restic/restic/pull/3106

View File

@@ -0,0 +1,13 @@
Enhancement: Parallelize reading of locks and snapshots
Restic used to read snapshots sequentially. For repositories containing
many snapshots this slowed down commands which have to read all snapshots.
Now the reading of snapshots is parallelized. This speeds up for example
`prune`, `backup` and other commands that search for snapshots with certain
properties or which have to find the `latest` snapshot.
The speed up also applies to locks stored in the backup repository.
https://github.com/restic/restic/pull/3130
https://github.com/restic/restic/pull/3174

View File

@@ -0,0 +1,8 @@
Bugfix: Do not hang until foregrounded when completed in background
On Linux, when running in the background restic failed to stop the terminal
output of the `backup` command after it had completed. This caused restic to
hang until moved to the foreground. This has now been fixed.
https://github.com/restic/restic/pull/3152
https://forum.restic.net/t/restic-alpine-container-cron-hangs-epoll-pwait/3334

View File

@@ -0,0 +1,7 @@
Bugfix: Improve error handling in `gs` backend
The `gs` backend did not notice when the last step of completing a
file upload failed. Under rare circumstances, this could cause
missing files in the backup repository. This has now been fixed.
https://github.com/restic/restic/pull/3249

View File

@@ -0,0 +1,8 @@
Enhancement: Enable HTTP/2 for backend connections
Go's HTTP library usually automatically chooses between HTTP/1.x and HTTP/2
depending on what the server supports. But for compatibility this mechanism
is disabled if DialContext is used (which is the case for restic). This change
allows restic's HTTP client to negotiate HTTP/2 if supported by the server.
https://github.com/restic/restic/pull/3254

View File

@@ -0,0 +1,13 @@
Bugfix: Improve error handling for rclone and REST backend over HTTP2
When retrieving data from the rclone / REST backend while also using HTTP2
restic did not detect when no data was returned at all. This could cause
for example the `check` command to report the following error:
Pack ID does not match, want [...], got e3b0c442
This has been fixed by correctly detecting and retrying the incomplete download.
https://github.com/restic/restic/issues/2742
https://github.com/restic/restic/pull/3453
https://forum.restic.net/t/http2-stream-closed-connection-reset-context-canceled/3743/10

View File

@@ -0,0 +1,6 @@
Enhancement: Add release binaries for s390x architecture on Linux
We've added release binaries for Linux using the s390x architecture.
https://github.com/restic/restic/issues/2780
https://github.com/restic/restic/pull/3452

View File

@@ -0,0 +1,11 @@
Bugfix: Fix terminal output redirection for PowerShell
When redirecting the output of restic using PowerShell on Windows, the
output contained terminal escape characters. This has been fixed by
properly detecting the terminal type.
In addition, the mintty terminal now shows progress output for the backup
command.
https://github.com/restic/restic/issues/3111
https://github.com/restic/restic/pull/3325

View File

@@ -0,0 +1,9 @@
Bugfix: Treat an empty password as a fatal error for repository init
When attempting to initialize a new repository, if an empty password was
supplied, the repository would be created but the init command would return
an error with a stack trace. Now, if an empty password is provided, it is
treated as a fatal error, and no repository is created.
https://github.com/restic/restic/issues/3214
https://github.com/restic/restic/pull/3283

View File

@@ -0,0 +1,8 @@
Change: Empty files now have size of 0 in `ls --json` output
The `ls --json` command used to omit the sizes of empty files in its
output. It now reports a size of zero explicitly for regular files,
while omitting the size field for all other types.
https://github.com/restic/restic/issues/3247
https://github.com/restic/restic/pull/3257

View File

@@ -0,0 +1,11 @@
Bugfix: `copy` failed to copy snapshots in rare cases
The `copy` command could in rare cases fail with the error message `SaveTree(...)
returned unexpected id ...`. This has been fixed.
On Linux/BSDs, the error could be caused by backing up symlinks with non-UTF-8
target paths. Note that, due to limitations in the repository format, these are
not stored properly and should be avoided if possible.
https://github.com/restic/restic/issues/3267
https://github.com/restic/restic/pull/3310

View File

@@ -0,0 +1,11 @@
Bugfix: `backup --quiet` no longer prints status information
A regression in the latest restic version caused the output of `backup --quiet`
to contain large amounts of backup progress information when run using an
interactive terminal. This is fixed now.
A workaround for this bug is to run restic as follows:
`restic backup --quiet [..] | cat -`.
https://github.com/restic/restic/issues/3184
https://github.com/restic/restic/pull/3186

View File

@@ -0,0 +1,14 @@
Enhancement: Add `--repository-file2` option to `init` and `copy` command
The `init` and `copy` command can now be used with the `--repository-file2`
option or the `$RESTIC_REPOSITORY_FILE2` environment variable.
These to options are in addition to the `--repo2` flag and allow you to read
the destination repository from a file.
Using both `--repository-file` and `--repo2` options resulted in an error for
the `copy` or `init` command. The handling of this combination of options has
been fixed. A workaround for this issue is to only use `--repo` or `-r` and
`--repo2` for `init` or `copy`.
https://github.com/restic/restic/issues/3293
https://github.com/restic/restic/pull/3294

View File

@@ -0,0 +1,7 @@
Bugfix: Fix crash of `check --read-data-subset=x%` run for an empty repository
The command `restic check --read-data-subset=x%` crashed when run for an empty
repository. This has been fixed.
https://github.com/restic/restic/issues/3296
https://github.com/restic/restic/pull/3309

View File

@@ -0,0 +1,8 @@
Bugfix: Fix `fdopendir: not a directory` error for local backend
The `check`, `list packs`, `prune` and `rebuild-index` commands failed
for the local backend when the `data` folder in the repository contained
files. This has been fixed.
https://github.com/restic/restic/issues/3302
https://github.com/restic/restic/pull/3308

View File

@@ -0,0 +1,5 @@
Enhancement: Add auto-completion support for fish
The `generate` command now supports fish auto completion.
https://github.com/restic/restic/pull/3312

View File

@@ -0,0 +1,8 @@
Bugfix: Print `created new cache` message only on a terminal
The message `created new cache` was printed even when the output wasn't a
terminal. That broke piping `restic dump` output to tar or zip if cache
directory didn't exist. The message is now only printed on a terminal.
https://github.com/restic/restic/issues/3334
https://github.com/restic/restic/pull/3343

View File

@@ -0,0 +1,8 @@
Enhancement: SFTP backend now checks for disk space
Backing up over SFTP previously spewed multiple generic "failure" messages
when the remote disk was full. It now checks for disk space before writing
a file and fails immediately with a "no space left on device" message.
https://github.com/restic/restic/issues/3336
https://github.com/restic/restic/pull/3345

View File

@@ -0,0 +1,6 @@
Enhancement: Add release binaries for Apple Silicon
We've added release binaries for macOS on Apple Silicon (M1).
https://github.com/restic/restic/issues/3377
https://github.com/restic/restic/pull/3394

View File

@@ -0,0 +1,7 @@
Bugfix: Fix crash of `backup --exclude='**'`
The exclude filter `**`, which excludes all files, caused restic to crash. This
has been corrected.
https://github.com/restic/restic/issues/3380
https://github.com/restic/restic/pull/3393

View File

@@ -0,0 +1,20 @@
Enhancement: Add `--keep-within-hourly` option to restic forget
The `forget` command allowed keeping a given number of hourly
backups or to keep all backups within a given interval, but it
was not possible to specify keeping hourly backups within a given
interval.
The new `--keep-within-hourly` option now offers this functionality.
Similar options for daily/weekly/monthly/yearly are also implemented,
the new options are:
--keep-within-hourly <1y2m3d4h>
--keep-within-daily <1y2m3d4h>
--keep-within-weekly <1y2m3d4h>
--keep-within-monthly <1y2m3d4h>
--keep-within-yearly <1y2m3d4h>
https://github.com/restic/restic/issues/3414
https://github.com/restic/restic/pull/3416
https://forum.restic.net/t/forget-policy/4014/11

View File

@@ -0,0 +1,9 @@
Enhancement: Support filtering and specifying untagged snapshots
It was previously not possible to specify an empty tag with the `--tag` and
`--keep-tag` options. This has now been fixed, such that `--tag ''` and
`--keep-tag ''` now matches snapshots without tags. This allows e.g. the
`snapshots` and `forget` commands to only operate on untagged snapshots.
https://github.com/restic/restic/issues/3456
https://github.com/restic/restic/pull/3457

View File

@@ -0,0 +1,9 @@
Enhancement: Allow specifying limit of `snapshots` list
The `--last` option allowed limiting the output of the `snapshots`
command to the latest snapshot for each host. The new `--latest n`
option allows limiting the output to the latest `n` snapshots.
This change deprecates the option `--last` in favour of `--latest 1`.
https://github.com/restic/restic/pull/3167

View File

@@ -0,0 +1,6 @@
Bugfix: Fix possibly missing backup summary of JSON output in case of error
When using `--json` output it happened from time to time that the summary
output was missing in case an error occurred. This has been fixed.
https://github.com/restic/restic/pull/3305

View File

@@ -0,0 +1,7 @@
Enhancement: Optimize read performance of mount command
Reading large files in a mounted repository may be up to five times faster.
This improvement primarily applies to repositories stored at a backend that can
be accessed with low latency, like e.g. the local backend.
https://github.com/restic/restic/pull/3426

View File

@@ -0,0 +1,13 @@
Enhancement: `find --pack` fallback to index if data file is missing
When investigating a repository with missing data files, it might be useful to
determine affected snapshots before running `rebuild-index`. Previously,
`find --pack pack-id` returned no data as it required accessing the data file.
Now, if the necessary data is still available in the repository index, it gets
retrieved from there.
The command now also supports looking up multiple pack files in a single `find`
run.
https://github.com/restic/restic/pull/3427
https://forum.restic.net/t/missing-packs-not-found/2600

View File

@@ -0,0 +1,8 @@
Bugfix: Correctly handle download errors during `restore`
Due to a regression in restic 0.12.0, the `restore` command in some cases did
not retry download errors and only printed a warning. This has been fixed by
retrying incomplete data downloads.
https://github.com/restic/restic/issues/3439
https://github.com/restic/restic/pull/3449

View File

@@ -0,0 +1,10 @@
Bugfix: Never lock repository for `list locks`
The `list locks` command previously locked to the repository by default. This
had the problem that it wouldn't work for an exclusively locked repository and
that the command would also display its own lock file which can be confusing.
Now, the `list locks` command never locks the repository.
https://github.com/restic/restic/issues/1106
https://github.com/restic/restic/pull/3665

View File

@@ -0,0 +1,16 @@
Enhancement: Add `--dry-run`/`-n` option to `backup` command
Testing exclude filters and other configuration options was error prone as
wrong filters could cause files to be uploaded unintentionally. It was also
not possible to estimate beforehand how much data would be uploaded.
The `backup` command now has a `--dry-run`/`-n` option, which performs all the
normal steps of a backup without actually writing anything to the repository.
Passing -vv will log information about files that would be added, allowing for
verification of source and exclusion options before running the real backup.
https://github.com/restic/restic/issues/1542
https://github.com/restic/restic/pull/2308
https://github.com/restic/restic/pull/3210
https://github.com/restic/restic/pull/3300

View File

@@ -0,0 +1,15 @@
Enhancement: Add upload checksum for Azure, GS, S3 and Swift backends
Previously only the B2 and partially the Swift backends verified the integrity
of uploaded (encrypted) files. The verification works by informing the backend
about the expected hash of the uploaded file. The backend then verifies the
upload and thereby rules out any data corruption during upload.
We have now added upload checksums for the Azure, GS, S3 and Swift backends,
which besides integrity checking for uploads also means that restic can now be
used to store backups in S3 buckets which have Object Lock enabled.
https://github.com/restic/restic/issues/2202
https://github.com/restic/restic/issues/2700
https://github.com/restic/restic/issues/3023
https://github.com/restic/restic/pull/3246

View File

@@ -0,0 +1,29 @@
Enhancement: Support negative include/exclude patterns
If a pattern starts with an exclamation mark and it matches a file that was
previously matched by a regular pattern, the match is cancelled. Notably,
this can be used with `--exclude-file` to cancel the exclusion of some files.
It works similarly to `.gitignore`, with the same limitation; Once a directory
is excluded, it is not possible to include files inside the directory.
Example of use as an exclude pattern for the `backup` command:
$HOME/**/*
!$HOME/Documents
!$HOME/code
!$HOME/.emacs.d
!$HOME/games
# [...]
node_modules
*~
*.o
*.lo
*.pyc
# [...]
$HOME/code/linux/*
!$HOME/code/linux/.git
# [...]
https://github.com/restic/restic/issues/233
https://github.com/restic/restic/pull/2311

View File

@@ -0,0 +1,13 @@
Bugfix: Make cache crash-resistant and usable by multiple concurrent processes
The restic cache directory (`RESTIC_CACHE_DIR`) could end up in a broken state
in the event of restic (or the OS) crashing. This is now less likely to occur
as files are downloaded to a temporary location before being moved to their
proper location.
This also allows multiple concurrent restic processes to operate on a single
repository without conflicts. Previously, concurrent operations could cause
segfaults because the processes saw each other's partially downloaded files.
https://github.com/restic/restic/issues/2345
https://github.com/restic/restic/pull/2838

View File

@@ -0,0 +1,10 @@
Enhancement: Add warning for S3 if partial credentials are provided
Previously restic did not notify about incomplete credentials when using the
S3 backend, instead just reporting access denied.
Restic now checks that both the AWS key ID and secret environment variables are
set before connecting to the remote server, and reports an error if not.
https://github.com/restic/restic/issues/2388
https://github.com/restic/restic/pull/3532

View File

@@ -0,0 +1,11 @@
Bugfix: Improve error handling of repository locking
Previously, when the lock refresh failed to delete the old lock file, it forgot
about the newly created one. Instead it continued trying to delete the old
(usually no longer existing) lock file and thus over time lots of lock files
accumulated. This has now been fixed.
https://github.com/restic/restic/issues/2452
https://github.com/restic/restic/issues/2473
https://github.com/restic/restic/issues/2562
https://github.com/restic/restic/pull/3512

View File

@@ -0,0 +1,8 @@
Enhancement: Support JSON output and quiet mode for the `diff` command
The `diff` command now supports outputting machine-readable output in JSON
format. To enable this, pass the `--json` option to the command. To only print
the summary and suppress detailed output, pass the `--quiet` option.
https://github.com/restic/restic/issues/2508
https://github.com/restic/restic/pull/3592

View File

@@ -0,0 +1,8 @@
Enhancement: Add flag to disable TLS verification for self-signed certificates
There is now an `--insecure-tls` global option in restic, which disables TLS
verification for self-signed certificates in order to support some development
workflows.
https://github.com/restic/restic/issues/2656
https://github.com/restic/restic/pull/2657

View File

@@ -0,0 +1,8 @@
Bugfix: Don't print progress for `backup --json --quiet`
Unlike the text output, the `--json` output format still printed progress
information even in `--quiet` mode. This has now been fixed by always
disabling the progress output in quiet mode.
https://github.com/restic/restic/issues/2738
https://github.com/restic/restic/pull/3264

View File

@@ -0,0 +1,9 @@
Enhancement: Atomic uploads for the SFTP backend
The SFTP backend did not upload files atomically. An interrupted upload could
leave an incomplete file behind which could prevent restic from accessing the
repository. This has now been fixed and uploads in the SFTP backend are done
atomically.
https://github.com/restic/restic/issues/3003
https://github.com/restic/restic/pull/3524

View File

@@ -0,0 +1,6 @@
Enhancement: Add xattr (extended attributes) support for Solaris
Restic now supports xattr for the Solaris operating system.
https://github.com/restic/restic/issues/3127
https://github.com/restic/restic/pull/3628

View File

@@ -0,0 +1,8 @@
Bugfix: Make `check` command honor `RESTIC_CACHE_DIR` environment variable
Previously, the `check` command didn't honor the `RESTIC_CACHE_DIR` environment
variable, which caused problems in certain system/usage configurations. This
has now been fixed.
https://github.com/restic/restic/issues/3382
https://github.com/restic/restic/pull/3474

View File

@@ -0,0 +1,10 @@
Enhancement: Skip lock creation on `forget` if `--no-lock` and `--dry-run`
Restic used to silently ignore the `--no-lock` option of the `forget` command.
It now skips creation of lock file in case both `--dry-run` and `--no-lock`
are specified. If `--no-lock` option is specified without `--dry-run`, restic
prints a warning message to stderr.
https://github.com/restic/restic/issues/3464
https://github.com/restic/restic/pull/3623

View File

@@ -0,0 +1,8 @@
Enhancement: Support random subset by size in `check --read-data-subset`
The `--read-data-subset` option of the `check` command now supports a third way
of specifying the subset to check, namely `nS` where `n` is a size in bytes with
suffix `S` as k/K, m/M, g/G or t/T.
https://github.com/restic/restic/issues/3490
https://github.com/restic/restic/pull/3548

View File

@@ -0,0 +1,8 @@
Bugfix: Make `copy` command honor `--no-lock` for source repository
The `copy` command previously did not respect the `--no-lock` option for the
source repository, causing failures with read-only storage backends. This has
now been fixed such that the option is now respected.
https://github.com/restic/restic/issues/3518
https://github.com/restic/restic/pull/3589

View File

@@ -0,0 +1,9 @@
Enhancement: Improve handling of temporary B2 delete errors
Deleting files on B2 could sometimes fail temporarily, which required restic to
retry the delete operation. In some cases the file was deleted nevertheless,
causing the retries and ultimately the restic command to fail. This has now been
fixed.
https://github.com/restic/restic/issues/3541
https://github.com/restic/restic/pull/3544

View File

@@ -0,0 +1,8 @@
Enhancement: Add file mode in symbolic notation to `ls --json`
The `ls --json` command now provides the file mode in symbolic notation (using
the `permissions` key), aligned with `find --json`.
https://github.com/restic/restic/issues/3542
https://github.com/restic/restic/pull/3573
https://forum.restic.net/t/restic-ls-understanding-file-mode-with-json/4371

View File

@@ -0,0 +1,10 @@
Bugfix: Fix hang with Backblaze B2 on SSL certificate authority error
Previously, if a request failed with an SSL unknown certificate authority
error, the B2 backend retried indefinitely and restic would appear to hang.
This has now been fixed and restic instead fails with an error message.
https://github.com/restic/restic/issues/3556
https://github.com/restic/restic/issues/2355
https://github.com/restic/restic/pull/3571

View File

@@ -0,0 +1,15 @@
Bugfix: Fix rclone backend prematurely exiting when receiving SIGINT on Windows
Previously, pressing Ctrl+C in a Windows console where restic was running with
rclone as the backend would cause rclone to exit prematurely due to getting a
`SIGINT` signal at the same time as restic. Restic would then wait for a long
time for time with "unexpected EOF" and "rclone stdio connection already closed"
errors.
This has now been fixed by restic starting the rclone process detached from the
console restic runs in (similar to starting processes in a new process group on
Linux), which enables restic to gracefully clean up rclone (which now never gets
the `SIGINT`).
https://github.com/restic/restic/issues/3601
https://github.com/restic/restic/pull/3602

View File

@@ -0,0 +1,14 @@
Change: Ignore parent snapshot for `backup --stdin`
Restic uses a parent snapshot to speed up directory scanning when performing
backups, but this only wasted time and memory when the backup source is stdin
(using the `--stdin` option of the `backup` command), since no directory scanning
is performed in this case.
Snapshots made with `backup --stdin` no longer have a parent snapshot, which allows
restic to skip some startup operations and saves a bit of resources.
The `--parent` option is still available for `backup --stdin`, but is now ignored.
https://github.com/restic/restic/issues/3641
https://github.com/restic/restic/pull/3645

View File

@@ -0,0 +1,8 @@
Bugfix: The `mount` command now reports symlinks sizes
Symlinks used to have size zero in restic mountpoints, confusing some
third-party tools. They now have a size equal to the byte length of their
target path, as required by POSIX.
https://github.com/restic/restic/issues/3667
https://github.com/restic/restic/pull/3668

View File

@@ -0,0 +1,7 @@
Enhancement: Speed up the `restore --verify` command
The `--verify` option lets the `restore` command verify the file content
after it has restored a snapshot. The performance of this operation has
now been improved by up to a factor of two.
https://github.com/restic/restic/pull/2594

View File

@@ -0,0 +1,10 @@
Enhancement: The `backup` command no longer updates file access times on Linux
When reading files during backup, restic used to cause the operating system to
update the files' access times. Note that this did not apply to filesystems with
disabled file access times.
Restic now instructs the operating system not to update the file access time,
if the user running restic is the file owner or has root permissions.
https://github.com/restic/restic/pull/2816

View File

@@ -0,0 +1,9 @@
Enhancement: Make `recover` collect only unreferenced trees
Previously, the `recover` command used to generate a snapshot containing *all*
root trees, even those which were already referenced by a snapshot.
This has been improved such that it now only processes trees not already
referenced by any snapshot.
https://github.com/restic/restic/pull/2880

View File

@@ -0,0 +1,12 @@
Enhancement: Verify that new or modified keys are stored correctly
When adding a new key or changing the password of a key, restic used to just
create the new key (and remove the old one, when changing the password). There
was no verification that the new key was stored correctly and works properly.
As the repository cannot be decrypted without a valid key file, this could in
rare cases cause the repository to become inaccessible.
Restic now checks that new key files actually work before continuing. This
can protect against some (rare) cases of hardware or storage problems.
https://github.com/restic/restic/pull/3429

View File

@@ -0,0 +1,12 @@
Enhancement: Improve local backend's resilience to (system) crashes
Restic now ensures that files stored using the `local` backend are created
atomically (that is, files are either stored completely or not at all). This
ensures that no incomplete files are left behind even if restic is terminated
while writing a file.
In addition, restic now tries to ensure that the directory in the repository
which contains a newly uploaded file is also written to disk. This can prevent
missing files if the system crashes or the disk is not properly unmounted.
https://github.com/restic/restic/pull/3436

View File

@@ -0,0 +1,9 @@
Bugfix: `rebuild-index` failed if an index file was damaged
Previously, the `rebuild-index` command would fail with an error if an index
file was damaged or truncated. This has now been fixed.
On older restic versions, a (slow) workaround is to use
`rebuild-index --read-all-packs` or to manually delete the damaged index.
https://github.com/restic/restic/pull/3488

View File

@@ -0,0 +1,10 @@
Enhancement: Cache blobs read by the `dump` command
When dumping a file using the `dump` command, restic did not cache blobs in any
way, so even consecutive runs of the same blob were loaded from the repository
again and again, slowing down the dump.
Now, the caching mechanism already used by the `fuse` command is also used by
the `dump` command. This makes dumping much faster, especially for sparse files.
https://github.com/restic/restic/pull/3508

Some files were not shown because too many files have changed in this diff Show More