The retry logic was pathological in the following ways:
* If we restarted the logging service, any pending uploads
would be placed in a retry-loop where it depended on backoff.Backoff,
which was too aggresive. It would retry failures within milliseconds,
taking at least 10 retries to hit a delay of 1 second.
* In the event where a logstream was rate limited,
the aggressive retry logic would severely exacerbate the problem
since each retry would also log an error message.
It is by chance that the rate of log error spam
does not happen to exceed the rate limit itself.
We modify the retry logic in the following ways:
* We now respect the "Retry-After" header sent by the logging service.
* Lacking a "Retry-After" header, we retry after a hard-coded period of
30 to 60 seconds. This avoids the thundering-herd effect when all nodes
try reconnecting to the logging service at the same time after a restart.
* We do not treat a status 400 as having been uploaded.
This is simply not the behavior of the logging service.
Updates #tailscale/corp#11213
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
This test was either fixed by intermediate changes or was mis-flagged as
failing during #7876 triage.
Updates #7876
Signed-off-by: James Tucker <jftucker@gmail.com>
This test was either fixed in the intermediate time or mis-flagged
during the #7876 triage, but is now passing.
Updates #7876
Signed-off-by: James Tucker <jftucker@gmail.com>
The subshell in which gocross gets built cd's to the corp checkout dir
near the top, so all future references to corp repository files should
be simple relative paths, and not reference $repo_root. When $repo_root
is an absolute path, it doesn't matter and everything works out, but on
some OSes and shells and invocations, $repo_root is a completely relative
path that is invalidated by the "cd".
Fixestailscale/corp#11183
Signed-off-by: David Anderson <danderson@tailscale.com>
I noticed cmd/{cloner,viewer} didn't support structs with embedded
fields while working on a change in another repo. This adds support.
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Wait 2 minutes before we start reporting battery usage. There is always
radio activity on initial startup, which gets reported as 100% high
power usage. Let that settle before we report usage data.
Updates tailscale/corp#9230
Signed-off-by: Will Norris <will@tailscale.com>
The previous commit 58ab66e added ssh/tailssh/user.go as part of
working on #4945. So move some more user-related code over to it.
Updates #cleanup
Change-Id: I24de66df25ffb8f867e1a0a540d410f9ef16d7b0
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Current code will set the "dirty" field of VersionInfo always "true"
if vcs.modified flag is there. No matter whether the flag is "true" or
"false". It will make sense to set this field due to vcs.modified
value, not only the existence of the key.
Signed-off-by: Chenyang Gao <gps949@outlook.com>
Signed-off-by: Chenyang Gao <gps949@outlook.com>
in commit 6e96744, the tsd system type has been added.
Which will cause the daemon will crash on some OSs (Windows, darwin and so on).
The root cause is that on those OSs, handleSubnetsInNetstack() will return true and set the conf.Router with a wrapper.
Later in NewUserspaceEngine() it will do subsystem set and found that early set router mismatch to current value, then panic.
expvar can only be defined once, so running tests with a repeat counter
will fail if the variables are defined inside of the test function.
Observed failure:
```
--- FAIL: TestHandler (0.00s)
panic: Reuse of exported var name: gauge_promvarz_test_expvar
[recovered]
panic: Reuse of exported var name: gauge_promvarz_test_expvar
goroutine 9 [running]:
testing.tRunner.func1.2({0x100f267e0, 0x1400026e770})
/usr/local/go/src/testing/testing.go:1526 +0x1c8
testing.tRunner.func1()
/usr/local/go/src/testing/testing.go:1529 +0x364
panic({0x100f267e0, 0x1400026e770})
/usr/local/go/src/runtime/panic.go:884 +0x1f4
log.Panicln({0x140000b8e20?, 0x1a?, 0x1400026e750?})
/usr/local/go/src/log/log.go:398 +0x60
expvar.Publish({0x100e2b21d, 0x1a}, {0x100fd7a08?, 0x140000232c0})
/usr/local/go/src/expvar/expvar.go:284 +0xc0
expvar.NewInt(...)
/usr/local/go/src/expvar/expvar.go:304
tailscale.com/tsweb/promvarz.TestHandler(0x14000082b60)
/Users/charlotte/ts-src/tailscale/tsweb/promvarz/promvarz_test.go:18 +0x5c
testing.tRunner(0x14000082b60, 0x100fd5858)
/usr/local/go/src/testing/testing.go:1576 +0x104
created by testing.(*T).Run
/usr/local/go/src/testing/testing.go:1629 +0x370
FAIL tailscale.com/tsweb/promvarz 0.149s
```
Fixes#8065
Signed-off-by: James Tucker <james@tailscale.com>
This change bumps the capability version to 62, after support for
sending SSHEventNotificationRequests to control via noise for failure
events was introduced.
Updates tailscale/corp#9967
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
This change adds a ConnectionID field to both SSHEventNotifyRequest and
CastHeader that identifies the ID of a connection to the SSH server.
Updates tailscale/corp#9967
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
This change sends an SSHEventNotificationRequest over noise when a
SSH session is set to fail closed and the session is unable to start
because a recorder is not available or a session is terminated because
connection to the recorder is ended. Each of these scenarios have their
own event type.
Updates tailscale/corp#9967
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
This change renames SSHFailureNotifyRequest to SSHEventNotifyRequest
to better reflect the additional events we could add in the future.
This change also adds an EventType used to catagories the events.
Updates tailscale/corp#9967
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
This change introduces a NodeKey func on localbackend that returns the
public node key.
Updates tailscale/corp#9967
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
It was supposed to be best effort but in some cases (macsys at least,
per @marwan-at-work) it hangs and exhausts the whole context.Context
deadline so we fail to make the SetDNS call to the server.
Updates #8067
Updates #3273 etc
Change-Id: Ie1f04abe9689951484748aecdeae312afbafdb0f
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
`interfaces.Tailscale()` returns all zero values when it finds no
Tailscale interface and encounters no errors. The netns package was
treating no error as a signal that it would receive a non-zero pointer
value leading to nil pointer dereference.
Observed in:
```
--- FAIL: TestGetInterfaceIndex (0.00s)
--- FAIL: TestGetInterfaceIndex/IP_and_port (0.00s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x2 addr=0x0 pc=0x1029eb7d8]
goroutine 7 [running]:
testing.tRunner.func1.2({0x102a691e0, 0x102bc05c0})
/Users/raggi/.cache/tailscale-go/src/testing/testing.go:1526 +0x1c8
testing.tRunner.func1()
/Users/raggi/.cache/tailscale-go/src/testing/testing.go:1529 +0x384
panic({0x102a691e0, 0x102bc05c0})
/Users/raggi/.cache/tailscale-go/src/runtime/panic.go:884 +0x204
tailscale.com/net/netns.getInterfaceIndex(0x14000073f28, 0x1028d0284?, {0x1029ef3b7, 0xa})
/Users/raggi/src/github.com/tailscale/tailscale/net/netns/netns_darwin.go:114 +0x228
tailscale.com/net/netns.TestGetInterfaceIndex.func2(0x14000138000)
/Users/raggi/src/github.com/tailscale/tailscale/net/netns/netns_darwin_test.go:37 +0x54
testing.tRunner(0x14000138000, 0x140000551b0)
/Users/raggi/.cache/tailscale-go/src/testing/testing.go:1576 +0x10c
created by testing.(*T).Run
/Users/raggi/.cache/tailscale-go/src/testing/testing.go:1629 +0x368
FAIL tailscale.com/net/netns 0.824s
```
Fixes#8064
Signed-off-by: James Tucker <jftucker@gmail.com>
This is part of an effort to clean up tailscaled initialization between
tailscaled, tailscaled Windows service, tsnet, and the mac GUI.
Updates #8036
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Periodic update for start of cycle. goreleaser is not updated to v2 yet,
but indirects updated.
Updates #8043
Signed-off-by: James Tucker <james@tailscale.com>
Installer script relies on pkgs.tailscale.com being reachable, both for
checking what Linux distros are supported, but also for actually
downloading repo configuration files, gpg keys and packages themselves.
This change adds a simple reachability check which will print an error
message when pkgs.tailscale.com is not reachable.
Fixes https://github.com/tailscale/corp/issues/8952
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
This holds back gvisor, kubernetes, goreleaser, and esbuild, which all
had breaking API changes.
Updates #8043
Updates #7381
Updates #8042 (updates u-root which adds deps)
Change-Id: I889759bea057cd3963037d41f608c99eb7466a5b
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
We removed it earlier in 916aa782af, but we still want to support it for some time longer.
Updates tailscale/corp#9967
Signed-off-by: Maisem Ali <maisem@tailscale.com>
This change introduces address selection for wireguard only endpoints.
If a endpoint has not been used before, an address is randomly selected
to be used based on information we know about, such as if they are able
to use IPv4 or IPv6. When an address is initially selected, we also
initiate a new ICMP ping to the endpoints addresses to determine which
endpoint offers the best latency. This information is then used to
update which endpoint we should be using based on the best possible
route. If the latency is the same for a IPv4 and an IPv6 address, IPv6
will be used.
Updates #7826
Signed-off-by: Charlotte Brandhorst-Satzkorn <charlotte@tailscale.com>
In the case where the exit node requires SNAT, we would SNAT all traffic not just the
traffic meant to go through the exit node. This was a result of the default route being
added to the routing table which would match basically everything.
In this case, we need to account for all peers in the routing table not just the ones
that require NAT.
Fix and add a test.
Updates tailscale/corp#8020
Signed-off-by: Maisem Ali <maisem@tailscale.com>
Avoid selecting an endpoint as "better" than the current endpoint if the
total latency improvement is less than 1%. This adds some hysteresis to
avoid flapping between endpoints for a minimal improvement in latency.
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: If8312e1768ea65c4b4d4e13d8de284b3825d7a73
This passes the *dnscache.Resolver down from the Direct client into the
Noise client and from there into the controlhttp client. This retains
the Resolver so that it can share state across calls instead of creating
a new resolver.
Updates #4845
Updates #6110
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: Ia5d6af1870f3b5b5d7dd5685d775dcf300aec7af
Every time we change `installer.sh`, run it in a few docker
containers based on different Linux distros, just as a simple test.
Also includes a few changes to the installer script itself to make
installation work in docker:
- install dnf config-manager command before running it
- run zypper in non-interactive mode
- update pacman indexes before installing packages
Updates https://github.com/tailscale/corp/issues/8952
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
