Add a global option --retry-lock

Fixes restic#719

If the option is passed, restic will wait the specified duration of time
and retry locking the repo every 10 seconds (or more often if the total
timeout is relatively small).

- Play nice with json output
- Reduce wait time in lock tests
- Rework timeout last attempt
- Reduce test wait time to 0.1s
- Use exponential back off for the retry lock
- Don't pass gopts to lockRepo functions
- Use global variable for retry sleep setup
- Exit retry lock on cancel
- Better wording for flag help
- Reorder debug statement
- Refactor tests
- Lower max sleep time to 1m
- Test that we cancel/timeout in time
- Use non blocking sleep function
- Refactor into minDuration func

Co-authored-by: Julian Brost <julian@0x4a42.net>

doc/design.rst

@@ -603,7 +603,10 @@ that the process is dead and considers the lock to be stale.
 When a new lock is to be created and no other conflicting locks are
 detected, restic creates a new lock, waits, and checks if other locks
 appeared in the repository. Depending on the type of the other locks and
-the lock to be created, restic either continues or fails.
+the lock to be created, restic either continues or fails. If the
+``--retry-lock`` option is specified, restic will retry
+creating the lock periodically until it succeeds or the specified
+timeout expires.
 
 Read and Write Ordering
 =======================

doc/manual_rest.rst

@@ -66,6 +66,7 @@ Usage help is available:
       -q, --quiet                      do not output comprehensive progress report
       -r, --repo repository            repository to backup to or restore from (default: $RESTIC_REPOSITORY)
           --repository-file file       file to read the repository location from (default: $RESTIC_REPOSITORY_FILE)
+          --retry-lock duration        retry to lock the repository if it is already locked, takes a value like 5m or 2h (default: no retries)
           --tls-client-cert file       path to a file containing PEM encoded TLS client certificate and private key
       -v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)
 
@@ -141,6 +142,7 @@ command:
       -q, --quiet                      do not output comprehensive progress report
       -r, --repo repository            repository to backup to or restore from (default: $RESTIC_REPOSITORY)
           --repository-file file       file to read the repository location from (default: $RESTIC_REPOSITORY_FILE)
+          --retry-lock duration        retry to lock the repository if it is already locked, takes a value like 5m or 2h (default: no retries)
           --tls-client-cert file       path to a file containing PEM encoded TLS client certificate and private key
       -v, --verbose                    be verbose (specify multiple times or a level using --verbose=n, max level/times is 2)