tailscale/cmd/tailscaled
Joe Tsai 84c99fe0d9
logtail: be less aggressive about re-uploads (#8117)
The retry logic was pathological in the following ways:

* If we restarted the logging service, any pending uploads
would be placed in a retry-loop where it depended on backoff.Backoff,
which was too aggresive. It would retry failures within milliseconds,
taking at least 10 retries to hit a delay of 1 second.

* In the event where a logstream was rate limited,
the aggressive retry logic would severely exacerbate the problem
since each retry would also log an error message.
It is by chance that the rate of log error spam
does not happen to exceed the rate limit itself.

We modify the retry logic in the following ways:

* We now respect the "Retry-After" header sent by the logging service.

* Lacking a "Retry-After" header, we retry after a hard-coded period of
30 to 60 seconds. This avoids the thundering-herd effect when all nodes
try reconnecting to the logging service at the same time after a restart.

* We do not treat a status 400 as having been uploaded.
This is simply not the behavior of the logging service.

Updates #tailscale/corp#11213

Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2023-05-11 12:52:35 -07:00
..
childproc all: update copyright and license headers 2023-01-27 15:36:29 -08:00
debug.go all: avoid repeated default interface lookups 2023-04-20 15:46:01 -07:00
depaware.txt logtail: be less aggressive about re-uploads (#8117) 2023-05-11 12:52:35 -07:00
generate.go cmd/mkmanifest, cmd/tailscale, cmd/tailscaled: remove Windows arm32 resources from OSS 2023-03-01 15:45:12 -07:00
install_darwin.go all: update copyright and license headers 2023-01-27 15:36:29 -08:00
install_windows.go all: update copyright and license headers 2023-01-27 15:36:29 -08:00
manifest_windows_386.syso cmd/{tailscale,tailscaled}: embed manifest into Windows binaries 2022-11-21 18:15:51 -05:00
manifest_windows_amd64.syso cmd/{tailscale,tailscaled}: embed manifest into Windows binaries 2022-11-21 18:15:51 -05:00
manifest_windows_arm64.syso cmd/{tailscale,tailscaled}: embed manifest into Windows binaries 2022-11-21 18:15:51 -05:00
proxy.go all: update copyright and license headers 2023-01-27 15:36:29 -08:00
required_version.go all: update to Go 1.20, use strings.CutPrefix/Suffix instead of our fork 2023-02-01 15:23:54 -08:00
ssh.go all: update copyright and license headers 2023-01-27 15:36:29 -08:00
taildrop.go various: add detection and Taildrop for Unraid 2023-05-04 13:40:13 -07:00
tailscaled_bird.go all: update copyright and license headers 2023-01-27 15:36:29 -08:00
tailscaled_notwindows.go all: update copyright and license headers 2023-01-27 15:36:29 -08:00
tailscaled_test.go all: update copyright and license headers 2023-01-27 15:36:29 -08:00
tailscaled_windows.go tsd: add package with System type to unify subsystem init, discovery 2023-05-04 14:21:59 -07:00
tailscaled.defaults cmd/tailscaled: rename relaynode reference in defaults file comment 2020-03-13 14:38:04 -07:00
tailscaled.go fix sys.Set(router) issue will crash the daemon in some OSs 2023-05-06 14:11:43 -07:00
tailscaled.openrc tstest/integration/vms: use an in-process logcatcher (#2360) 2021-07-08 14:39:45 -04:00
tailscaled.service cmd/tailscaled: use explicit equal sign in --port=$PORT in tailscaled.service 2022-09-22 11:54:22 -07:00
windows-manifest.xml cmd/{tailscale,tailscaled}: embed manifest into Windows binaries 2022-11-21 18:15:51 -05:00
with_cli.go all: update copyright and license headers 2023-01-27 15:36:29 -08:00