d4bfe34ba7
The Go zstd package is not friendly for stateless zstd compression. Passing around multiple zstd.Encoder just for stateless compression is a waste of memory since the memory is never freed and seldom used if no compression operations are happening. For performance, we pool the relevant Encoder/Decoder with the specific options set. Functionally, this package is a wrapper over the Go zstd package with a more ergonomic API for stateless operations. This package can be used to cleanup various pre-existing zstd.Encoder pools or one-off handlers spread throughout our codebases. Performance: BenchmarkEncode/Best 1690 610926 ns/op 25.78 MB/s 1 B/op 0 allocs/op zstd_test.go:137: memory: 50.336 MiB zstd_test.go:138: ratio: 3.269x BenchmarkEncode/Better 10000 100939 ns/op 156.04 MB/s 0 B/op 0 allocs/op zstd_test.go:137: memory: 20.399 MiB zstd_test.go:138: ratio: 3.131x BenchmarkEncode/Default 15775 74976 ns/op 210.08 MB/s 105 B/op 0 allocs/op zstd_test.go:137: memory: 1.586 MiB zstd_test.go:138: ratio: 3.064x BenchmarkEncode/Fastest 23222 53977 ns/op 291.81 MB/s 26 B/op 0 allocs/op zstd_test.go:137: memory: 599.458 KiB zstd_test.go:138: ratio: 2.898x BenchmarkEncode/FastestLowMemory 23361 50789 ns/op 310.13 MB/s 15 B/op 0 allocs/op zstd_test.go:137: memory: 334.458 KiB zstd_test.go:138: ratio: 2.898x BenchmarkEncode/FastestNoChecksum 23086 50253 ns/op 313.44 MB/s 26 B/op 0 allocs/op zstd_test.go:137: memory: 599.458 KiB zstd_test.go:138: ratio: 2.900x BenchmarkDecode/Checksum 70794 17082 ns/op 300.96 MB/s 4 B/op 0 allocs/op zstd_test.go:163: memory: 316.438 KiB BenchmarkDecode/NoChecksum 74935 15990 ns/op 321.51 MB/s 4 B/op 0 allocs/op zstd_test.go:163: memory: 316.438 KiB BenchmarkDecode/LowMemory 71043 16739 ns/op 307.13 MB/s 0 B/op 0 allocs/op zstd_test.go:163: memory: 79.347 KiB We can see that the options are taking effect where compression ratio improves with higher levels and compression speed diminishes. We can also see that LowMemory takes effect where the pooled coder object references less memory than other cases. We can see that the pooling is taking effect as there are 0 amortized allocations. Additional performance: BenchmarkEncodeParallel/zstd-24 1857 619264 ns/op 1796 B/op 49 allocs/op BenchmarkEncodeParallel/zstdframe-24 1954 532023 ns/op 4293 B/op 49 allocs/op BenchmarkDecodeParallel/zstd-24 5288 197281 ns/op 2516 B/op 49 allocs/op BenchmarkDecodeParallel/zstdframe-24 6441 196254 ns/op 2513 B/op 49 allocs/op In concurrent usage, handling the pooling in this package has a marginal benefit over the zstd package, which relies on a Go channel as the pooling mechanism. In particular, coders can be freed by the GC when not in use. Coders can be shared throughout the program if they use this package instead of multiple independent pools doing the same thing. The allocations are unrelated to pooling as they're caused by the spawning of goroutines. Updates #cleanup Updates tailscale/corp#18514 Updates tailscale/corp#17653 Updates tailscale/corp#18005 Signed-off-by: Joe Tsai <joetsai@digital-static.net>
Tailscale
Private WireGuard® networks made easy
Overview
This repository contains the majority of Tailscale's open source code.
Notably, it includes the tailscaled daemon and
the tailscale CLI tool. The tailscaled daemon runs on Linux, Windows,
macOS, and to varying degrees
on FreeBSD and OpenBSD. The Tailscale iOS and Android apps use this repo's
code, but this repo doesn't contain the mobile GUI code.
Other Tailscale repos of note:
- the Android app is at https://github.com/tailscale/tailscale-android
- the Synology package is at https://github.com/tailscale/tailscale-synology
- the QNAP package is at https://github.com/tailscale/tailscale-qpkg
- the Chocolatey packaging is at https://github.com/tailscale/tailscale-chocolatey
For background on which parts of Tailscale are open source and why, see https://tailscale.com/opensource/.
Using
We serve packages for a variety of distros and platforms at https://pkgs.tailscale.com.
Other clients
The macOS, iOS, and Windows clients use the code in this repository but additionally include small GUI wrappers. The GUI wrappers on non-open source platforms are themselves not open source.
Building
We always require the latest Go release, currently Go 1.22. (While we build releases with our Go fork, its use is not required.)
go install tailscale.com/cmd/tailscale{,d}
If you're packaging Tailscale for distribution, use build_dist.sh
instead, to burn commit IDs and version info into the binaries:
./build_dist.sh tailscale.com/cmd/tailscale
./build_dist.sh tailscale.com/cmd/tailscaled
If your distro has conventions that preclude the use of
build_dist.sh, please do the equivalent of what it does in your
distro's way, so that bug reports contain useful version information.
Bugs
Please file any issues about this code or the hosted service on the issue tracker.
Contributing
PRs welcome! But please file bugs. Commit messages should reference bugs.
We require Developer Certificate of
Origin
Signed-off-by lines in commits.
See git log for our commit message style. It's basically the same as
Go's style.
About Us
Tailscale is primarily developed by the people at https://github.com/orgs/tailscale/people. For other contributors, see:
- https://github.com/tailscale/tailscale/graphs/contributors
- https://github.com/tailscale/tailscale-android/graphs/contributors
Legal
WireGuard is a registered trademark of Jason A. Donenfeld.