The derp metrics got out of sync in 74eb99aed1 (2023-03).
They were fixed in 0380cbc90d (2024-05).
This adds some further guardrails (atop the previous fix) to make sure
they don't get out of sync again.
Updates #12288
Change-Id: I809061a81f8ff92f45054d0253bc13871fc71634
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Without changing behaviour, don't create a goroutine per connection that
sits and sleeps, but rather use a timer that wakes up and gathers
statistics on a regular basis.
Fixes#12127
Signed-off-by: Andrew Dunham <andrew@du.nham.ca>
Change-Id: Ibc486447e403070bdc3c2cd8ae340e7d02854f21
The netcheck client, when no UDP is available, probes distance using
HTTPS.
Several problems:
* It probes using /derp/latency-check.
* But cmd/derper serves the handler at /derp/probe
* Despite the difference, it work by accident until c8f4dfc8c0
which made netcheck's probe require a 2xx status code.
* in tests, we only use derphttp.Handler, so the cmd/derper-installed
mux routes aren't preesnt, so there's no probe. That breaks
tests in airplane mode. netcheck.Client then reports "unexpected
HTTP status 426" (Upgrade Required)
This makes derp handle both /derp/probe and /derp/latency-check
equivalently, and in both cmd/derper and derphttp.Handler standalone
modes.
I notice this when wgengine/magicsock TestActiveDiscovery was failing
in airplane mode (no wifi). It still doesn't pass, but it gets
further.
Fixes#11989
Change-Id: I45213d4bd137e0f29aac8bd4a9ac92091065113f
It's deprecated and using it gets us the old slow behavior
according to https://go.dev/blog/randv2.
> Having eliminated repeatability of the global output stream, Go 1.20
> was also able to make the global generator scale better in programs
> that don’t call rand.Seed, replacing the Go 1 generator with a very
> cheap per-thread wyrand generator already used inside the Go
> runtime. This removed the global mutex and made the top-level
> functions scale much better. Programs that do call rand.Seed fall
> back to the mutex-protected Go 1 generator.
Updates #7123
Change-Id: Ia5452e66bd16b5457d4b1c290a59294545e13291
Signed-off-by: Maisem Ali <maisem@tailscale.com>
The goal is to move more network state accessors to netmon.Monitor
where they can be cheaper/cached. But first (this change and others)
we need to make sure the one netmon.Monitor is plumbed everywhere.
Some notable bits:
* tsdial.NewDialer is added, taking a now-required netmon
* because a tsdial.Dialer always has a netmon, anything taking both
a Dialer and a NetMon is now redundant; take only the Dialer and
get the NetMon from that if/when needed.
* netmon.NewStatic is added, primarily for tests
Updates tailscale/corp#10910
Updates tailscale/corp#18960
Updates #7967
Updates #3299
Change-Id: I877f9cb87618c4eb037cee098241d18da9c01691
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This adds a health.Tracker to tsd.System, accessible via
a new tsd.System.HealthTracker method.
In the future, that new method will return a tsd.System-specific
HealthTracker, so multiple tsnet.Servers in the same process are
isolated. For now, though, it just always returns the temporary
health.Global value. That permits incremental plumbing over a number
of changes. When the second to last health.Global reference is gone,
then the tsd.System.HealthTracker implementation can return a private
Tracker.
The primary plumbing this does is adding it to LocalBackend and its
dozen and change health calls. A few misc other callers are also
plumbed. Subsequent changes will flesh out other parts of the tree
(magicsock, controlclient, etc).
Updates #11874
Updates #4136
Change-Id: Id51e73cfc8a39110425b6dc19d18b3975eac75ce
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The derphttp.Client mutex is held during connects (for up to 10
seconds) so this LocalAddr method (blocking on said mutex) could also
block for up to 10 seconds, causing a pileup upstream in
magicsock/wgengine and ultimately a watchdog timeout resulting in a
crash.
Updates #11519
Change-Id: Idd1d94ee00966be1b901f6899d8b9492f18add0f
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
This CONNECT client doesn't match what Go's net/http.Transport does
(making the two values match). This makes it match.
This is all pretty unspecified but most clients & doc examples show
these matching. And some proxy implementations (such as Zscaler) care.
Updates tailscale/corp#18716
Change-Id: I135c5facbbcec9276faa772facbde1bb0feb2d26
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
So derpers can check an external URL for whether to permit access
to a certain public key.
Updates tailscale/corp#17693
Change-Id: I8594de58f54a08be3e2dbef8bcd1ff9b728ab297
Co-authored-by: Maisem Ali <maisem@tailscale.com>
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
See the field alignment lints for more information.
Reductions are 64->24 and 64->32 respectively.
Updates #self
Signed-off-by: James Tucker <james@tailscale.com>
Observed on one busy derp node, there were 600 goroutines blocked
writing to this channel, which represents not only more blocked routines
than we need, but also excess wake-ups downstream as the latent
goroutines writes represent no new work.
Updates #self
Signed-off-by: James Tucker <james@tailscale.com>
3d7fb6c21d dropped the explicit called to (*Client).connect when
its (*Client).WatchConnectionChanges got removed+refactored.
This puts it back, but in RunWatchConnectionLoop, before the call
to the (*Client).ServerPublicKey accessor, which is documented to
return the zero value (which is what broke us) on an unconnected
connection.
Plus some tests.
Fixestailscale/corp#15604
Change-Id: I0f242816f5ee4ad3bb0bf0400abc961dbe9f5fc8
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
The derphttp client automatically reconnects upon failure.
RunWatchConnectionLoop called derphttp.Client.WatchConnectionChanges
once, but that wrapper method called the underlying
derp.Client.WatchConnectionChanges exactly once on derphttp.Client's
currently active connection. If there's a failure, we need to re-subscribe
upon all reconnections.
This removes the derphttp.Client.WatchConnectionChanges method, which
was basically impossible to use correctly, and changes it to be a
boolean field on derphttp.Client alongside MeshKey and IsProber. Then
it moves the call to the underlying derp.Client.WatchConnectionChanges
to derphttp's client connection code, so it's resubscribed on any
reconnect.
Some paranoia is then added to make sure people hold the API right,
not calling derphttp.Client.RunWatchConnectionLoop on an
already-started Client without having set the bool to true. (But still
auto-setting it to true if that's the first method that's been called
on that derphttp.Client, as is commonly the case, and prevents
existing code from breaking)
Fixestailscale/corp#9916
Supercedes tailscale/tailscale#9719
Co-authored-by: Val <valerie@tailscale.com>
Co-authored-by: Irbe Krumina <irbe@tailscale.com>
Co-authored-by: Anton Tolchanov <anton@tailscale.com>
Signed-off-by: Brad Fitzpatrick <brad@danga.com>
When trying to set up multiple derper instances meshing with each
other, it turned out that while one can specify an alternative
listening port using the -a flag, the TLS hostname gets incorrectly
determined and includes the set alternative listening port as part of
the hostname. Thus, the TLS hostname validation always fails when the
-mesh-with values have ports.
Updates #9949
Signed-off-by: Thomas Kosiewski <thomas.kosiewski@loft.sh>
This makes wsconn.Conns somewhat present reasonably when they are
the client of an http.Request, rather than just put a placeholder
in that field.
Updates tailscale/corp#13777
Signed-off-by: David Anderson <danderson@tailscale.com>
Named result meant error paths assigned that variable to nil.
But a goroutine was concurrently using that variable.
Don't use a named result for that first parameter. Then then return
paths don't overwrite it.
Fixes#9129
Change-Id: Ie57f99d40ca8110085097780686d9bd620aaf160
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Like net/http.Server.BaseContext, this lets callers specify a base
context for dials.
Updates tailscale/corp#12702
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
~97% of the log messages derper outputs are related to the normal
non-error state of a client disconnecting in some manner. Add a
verbose logging feature that only logs these messages when enabled.
Fixes#8024
Signed-off-by: Val <valerie@tailscale.com>
On some platforms (notably macOS and iOS) we look up the default
interface to bind outgoing connections to. This is both duplicated
work and results in logspam when the default interface is not available
(i.e. when a phone has no connectivity, we log an error and thus cause
more things that we will try to upload and fail).
Fixed by passing around a netmon.Monitor to more places, so that we can
use its cached interface state.
Fixes#7850
Updates #7621
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
Using log.Printf may end up being printed out to the console, which
is not desirable. I noticed this when I was investigating some client
logs with `sockstats: trace "NetcheckClient" was overwritten by another`.
That turns to be harmless/expected (the netcheck client will fall back
to the DERP client in some cases, which does its own sockstats trace).
However, the log output could be visible to users if running the
`tailscale netcheck` CLI command, which would be needlessly confusing.
Updates tailscale/corp#9230
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
Make developing derp easier by:
1. Creating an envknob telling clients to use HTTP to connect to derp
servers, so devs don't have to acquire a valid TLS cert.
2. Creating an envknob telling clients which derp server to connect
to, so devs don't have to edit the ACLs in the admin console to add a
custom DERP map.
3. Explaining how the -dev and -a command lines args to derper
interact.
To use this:
1. Run derper with -dev.
2. Run tailscaled with TS_DEBUG_USE_DERP_HTTP=1 and
TS_DEBUG_USE_DERP_ADDR=localhost
This will result in the client connecting to derp via HTTP on port
3340.
Fixes#7700
Signed-off-by: Val <valerie@tailscale.com>
* wgengine/magicsock: add envknob to send CallMeMaybe to non-existent peer
For testing older client version responses to the PeerGone packet format change.
Updates #4326
Signed-off-by: Val <valerie@tailscale.com>
* derp: remove dead sclient struct member replaceLimiter
Leftover from an previous solution to the duplicate client problem.
Updates #2751
Signed-off-by: Val <valerie@tailscale.com>
* derp, derp/derphttp, wgengine/magicsock: add new PeerGone message type Not Here
Extend the PeerGone message type by adding a reason byte. Send a
PeerGone "Not Here" message when an endpoint sends a disco message to
a peer that this server has no record of.
Fixes#4326
Signed-off-by: Val <valerie@tailscale.com>
---------
Signed-off-by: Val <valerie@tailscale.com>
This allows tracking packet flow via logs for prober clients. Note that
the new sclient.debug() function is called on every received packet, but
will do nothing for most clients.
I have adjusted sclient logging to print public keys in short format
rather than full. This takes effect even for existing non-debug logging
(mostly client disconnect messages).
Example logs for a packet being sent from client [SbsJn] (connected to
derper [dM2E3]) to client [10WOo] (connected to derper [AVxvv]):
```
derper [dM2E3]:
derp client 10.0.0.1:35470[SbsJn]: register single client mesh("10.0.1.1"): 4 peers
derp client 10.0.0.1:35470[SbsJn]: read frame type 4 len 40 err <nil>
derp client 10.0.0.1:35470[SbsJn]: SendPacket for [10WOo], forwarding via <derphttp_client.Client [AVxvv] url=https://10.0.1.1/derp>: <nil>
derp client 10.0.0.1:35470[SbsJn]: read frame type 0 len 0 err EOF
derp client 10.0.0.1:35470[SbsJn]: read EOF
derp client 10.0.0.1:35470[SbsJn]: sender failed: context canceled
derp client 10.0.0.1:35470[SbsJn]: removing connection
derper [AVxvv]:
derp client 10.0.1.1:50650[10WOo]: register single client
derp client 10.0.1.1:50650[10WOo]: received forwarded packet from [SbsJn] via [dM2E3]
derp client 10.0.1.1:50650[10WOo]: sendPkt attempt 0 enqueued
derp client 10.0.1.1:50650[10WOo]: sendPacket from [SbsJn]: <nil>
derp client 10.0.1.1:50650[10WOo]: read frame type 0 len 0 err EOF
derp client 10.0.1.1:50650[10WOo]: read EOF
derp client 10.0.1.1:50650[10WOo]: sender failed: context canceled
derp client 10.0.1.1:50650[10WOo]: removing connection
```
Signed-off-by: Anton Tolchanov <anton@tailscale.com>
Makes it cheaper/simpler to persist values, and encourages reuse of
labels as opposed to generating an arbitrary number.
Updates tailscale/corp#9230
Updates #3363
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
Uses the hooks added by tailscale/go#45 to instrument the reads and
writes on the major code paths that do network I/O in the client. The
convention is to use "<package>.<type>:<label>" as the annotation for
the responsible code path.
Enabled on iOS, macOS and Android only, since mobile platforms are the
ones we're most interested in, and we are less sensitive to any
throughput degradation due to the per-I/O callback overhead (macOS is
also enabled for ease of testing during development).
For now just exposed as counters on a /v0/sockstats PeerAPI endpoint.
We also keep track of the current interface so that we can break out
the stats by interface.
Updates tailscale/corp#9230
Updates #3363
Signed-off-by: Mihai Parparita <mihai@tailscale.com>
Updates #7123
Updates #6257 (more to do in other repos)
Change-Id: I073e2a6d81a5d7fbecc29caddb7e057ff65239d0
Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Update all code generation tools, and those that check for license
headers to use the new standard header.
Also update copyright statement in LICENSE file.
Fixes#6865
Signed-off-by: Will Norris <will@tailscale.com>
This updates all source files to use a new standard header for copyright
and license declaration. Notably, copyright no longer includes a date,
and we now use the standard SPDX-License-Identifier header.
This commit was done almost entirely mechanically with perl, and then
some minimal manual fixes.
Updates #6865
Signed-off-by: Will Norris <will@tailscale.com>
