TheArchive/tailscale
1
0
Fork 0
mirror of https://github.com/tailscale/tailscale.git synced 2024-11-29 04:55:31 +00:00
Code Issues Packages Projects Releases Wiki Activity
c5b1565337
tailscale/util/deephash/deephash_test.go

998 lines
33 KiB
Go
Raw Normal View History

internal/deepprint: add missing copyright headers
2020-06-30 02:36:45 +00:00
// Copyright (c) 2020 Tailscale Inc & AUTHORS All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE file.
internal/deephash: rename from deepprint Yes, it printed, but that was an implementation detail for hashing. And coming optimization will make it print even less. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 19:09:25 +00:00
package deephash
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
import (
util/deephash: include type as part of hash for interfaces (#2476) A Go interface may hold any number of different concrete types. Just because two underlying values hash to the same thing does not mean the two values are identical if they have different concrete types. As such, include the type in the hash.
2021-07-21 17:26:04 +00:00
"archive/tar"
util/deephash: fix TestArrayAllocs Unfortunately this test fails on certain architectures. The problem comes down to inconsistencies in the Go escape analysis where specific variables are marked as escaping on certain architectures. The variables escaping to the heap are unfortunately in crypto/sha256, which makes it impossible to fixthis locally in deephash. For now, fix the test by compensating for the allocations that occur from calling sha256.digest.Sum. See golang/go#48055 Fixes #2727 Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-30 17:47:21 +00:00
"crypto/sha256"
util/deephash: use binary encoding of time.Time (#5352) Formatting a time.Time as RFC3339 is slow. See https://go.dev/issue/54093 Now that we have efficient hashing of fixed-width integers, just hash the time.Time as a binary value. Performance: Hash-24 19.0µs ± 1% 18.6µs ± 1% -2.03% (p=0.000 n=10+9) TailcfgNode-24 1.79µs ± 1% 1.40µs ± 1% -21.74% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 21:42:51 +00:00
"encoding/binary"
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
"fmt"
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
"hash"
util/deephash: hash uint{8,16,32,64} explicitly (#2502) Instead of hashing the humanly formatted forms of a number, hash the native machine bits of the integers themselves. There is a small performance gain for this: name old time/op new time/op delta Hash-8 75.7µs ± 1% 76.0µs ± 2% ~ (p=0.315 n=10+9) HashMapAcyclic-8 63.1µs ± 3% 61.3µs ± 1% -2.77% (p=0.000 n=10+10) TailcfgNode-8 10.3µs ± 1% 10.2µs ± 1% -1.48% (p=0.000 n=10+10) HashArray-8 1.07µs ± 1% 1.05µs ± 1% -1.79% (p=0.000 n=10+10) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-03 04:44:13 +00:00
"math"
util/deephash: rely on direct memory hashing for primitive kinds (#5457) Rather than separate functions to hash each kind, just rely on the fact that these are direct memory hashable, thus simplifying the code. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 03:50:56 +00:00
"math/bits"
util/deephash: fix map hashing when key & element have the same type Regression from 09afb8e35b56c1150ae9872d781c7ae3c151b848, in which the same reflect.Value scratch value was being used as the map iterator copy destination. Also: make nil and empty maps hash differently, add test. Fixes #4871 Co-authored-by: Josh Bleecher Snyder <josharian@gmail.com> Change-Id: I67f42524bc81f694c1b7259d6682200125ea4a66 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-16 20:21:32 +00:00
"math/rand"
all: use various net/netip parse funcs directly Mechanical change with perl+goimports. Changed {Must,}Parse{IP,IPPrefix,IPPort} to their netip variants, then goimports -d . Finally, removed the net/netaddr wrappers, to prevent future use. Updates #5162 Change-Id: I59c0e38b5fbca5a935d701645789cddf3d7863ad Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 03:55:44 +00:00
"net/netip"
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
"reflect"
util/deephash: fix TestArrayAllocs Unfortunately this test fails on certain architectures. The problem comes down to inconsistencies in the Go escape analysis where specific variables are marked as escaping on certain architectures. The variables escaping to the heap are unfortunately in crypto/sha256, which makes it impossible to fixthis locally in deephash. For now, fix the test by compensating for the allocations that occur from calling sha256.digest.Sum. See golang/go#48055 Fixes #2727 Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-30 17:47:21 +00:00
"runtime"
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
"testing"
util/deephash: fix map hashing when key & element have the same type Regression from 09afb8e35b56c1150ae9872d781c7ae3c151b848, in which the same reflect.Value scratch value was being used as the map iterator copy destination. Also: make nil and empty maps hash differently, add test. Fixes #4871 Co-authored-by: Josh Bleecher Snyder <josharian@gmail.com> Change-Id: I67f42524bc81f694c1b7259d6682200125ea4a66 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-16 20:21:32 +00:00
"testing/quick"
util/deephash: don't track cycles on non-recursive types name old time/op new time/op delta Hash-8 67.3µs ±20% 76.5µs ±16% ~ (p=0.143 n=10+10) HashMapAcyclic-8 63.0µs ± 2% 56.3µs ± 1% -10.65% (p=0.000 n=10+8) TailcfgNode-8 9.18µs ± 2% 6.52µs ± 3% -28.96% (p=0.000 n=9+10) HashArray-8 732ns ± 3% 709ns ± 1% -3.21% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I28642050d837dff66b2db54b2b0e6d272a930be8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
"time"
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
util/deephash: use unsafe.Pointer instead of reflect.Value (#5459) Use of reflect.Value.SetXXX panics if the provided argument was obtained from an unexported struct field. Instead, pass an unsafe.Pointer around and convert to a reflect.Value when necessary (i.e., for maps and interfaces). Converting from unsafe.Pointer to reflect.Value guarantees that none of the read-only bits will be populated. When running in race mode, we attach type information to the pointer so that we can type check every pointer operation. This also type-checks that direct memory hashing is within the valid range of a struct value. We add test cases that previously caused deephash to panic, but now pass. Performance: name old time/op new time/op delta Hash 14.1µs ± 1% 14.1µs ± 1% ~ (p=0.590 n=10+9) HashPacketFilter 2.53µs ± 2% 2.44µs ± 1% -3.79% (p=0.000 n=9+10) TailcfgNode 1.45µs ± 1% 1.43µs ± 0% -1.36% (p=0.000 n=9+9) HashArray 318ns ± 2% 318ns ± 2% ~ (p=0.541 n=10+10) HashMapAcyclic 32.9µs ± 1% 31.6µs ± 1% -4.16% (p=0.000 n=10+9) There is a slight performance gain due to the use of unsafe.Pointer over reflect.Value methods. Also, passing an unsafe.Pointer (1 word) on the stack is cheaper than passing a reflect.Value (3 words). Performance gains are diminishing since SHA-256 hashing now dominates the runtime. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 19:30:35 +00:00
qt "github.com/frankban/quicktest"
all: replace tailcfg.DiscoKey with key.DiscoPublic. Signed-off-by: David Anderson <danderson@tailscale.com>
2021-11-02 21:41:56 +00:00
"go4.org/mem"
util/deephash: specialize for netip.Addr and drop AppendTo support (#5402) There are 5 types that we care about that implement AppendTo: key.DiscoPublic key.NodePublic netip.Prefix netipx.IPRange netip.Addr The key types are thin wrappers around [32]byte and are memory hashable. The netip.Prefix and netipx.IPRange types are thin wrappers over netip.Addr and are hashable by default if netip.Addr is hashable. The netip.Addr type is the only one with a complex structure where the default behavior of deephash does not hash it correctly due to the presence of the intern.Value type. Drop support for AppendTo and instead add specialized hashing for netip.Addr that would be semantically equivalent to == on the netip.Addr values. The AppendTo support was already broken prior to this change. It was fully removed (intentionally or not) in #4870. It was partially restored in #4858 for the fast path, but still broken in the slow path. Just drop support for it altogether. This does mean we lack any ability for types to self-hash themselves. In the future we can add support for types that implement: interface { DeepHash() Sum } Test and fuzz cases were added for the relevant types that used to rely on the AppendTo method. FuzzAddr has been executed on 1 billion samples without issues. Signed-off-by: Joe Tsai joetsai@digital-static.net
2022-08-19 05:54:56 +00:00
"go4.org/netipx"
internal/deepprint: improve benchmark This more closely matches our real usage of deepprint. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-05-11 16:45:12 +00:00
"tailscale.com/tailcfg"
types/dnstype: introduce new package for Resolver So the type can be used in net/dns without introducing a tailcfg dependency. For #2596 Signed-off-by: David Crawshaw <crawshaw@tailscale.com>
2021-08-05 21:05:24 +00:00
"tailscale.com/types/dnstype"
util/deephash: don't reflect.Copy if element type is a defined uint8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-07 18:58:02 +00:00
"tailscale.com/types/ipproto"
wgengine/wgcfg: convert to use new node key type. Updates #3206 Signed-off-by: David Anderson <danderson@tailscale.com>
2021-10-28 00:42:33 +00:00
"tailscale.com/types/key"
util/deephash: fix unexported time.Time hashing Updates tailscale/corp#6311 Change-Id: I33cd7e4040966261c2f2eb3d32f29936aeb7f632 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-27 16:10:22 +00:00
"tailscale.com/util/deephash/testtype"
internal/deepprint: improve benchmark This more closely matches our real usage of deepprint. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-05-11 16:45:12 +00:00
"tailscale.com/util/dnsname"
util/deephash: skip alloc test under race detector Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-07 18:40:28 +00:00
"tailscale.com/version"
util/deephash: don't reflect.Copy if element type is a defined uint8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-07 18:58:02 +00:00
"tailscale.com/wgengine/filter"
many: gofmt. Signed-off-by: David Anderson <danderson@tailscale.com>
2021-03-26 00:41:51 +00:00
"tailscale.com/wgengine/router"
all: move wgcfg from wireguard-go This is mostly code movement from the wireguard-go repo. Most of the new wgcfg package corresponds to the wireguard-go wgcfg package. wgengine/wgcfg/device{_test}.go was device/config{_test}.go. There were substantive but simple changes to device_test.go to remove internal package device references. The API of device.Config (now wgcfg.DeviceConfig) grew an error return; we previously logged the error and threw it away. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-01-29 20:16:36 +00:00
"tailscale.com/wgengine/wgcfg"
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
)
util/deephash: disambiguate hashing of AppendTo (#2483) Prepend size to AppendTo output. Fixes #2443 Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-07-21 18:29:08 +00:00
type appendBytes []byte
func (p appendBytes) AppendTo(b []byte) []byte {
return append(b, p...)
}
util/deephash: include type as part of hash for interfaces (#2476) A Go interface may hold any number of different concrete types. Just because two underlying values hash to the same thing does not mean the two values are identical if they have different concrete types. As such, include the type in the hash.
2021-07-21 17:26:04 +00:00
func TestHash(t *testing.T) {
all: use any instead of interface{} My favorite part of generics. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2022-03-16 23:27:57 +00:00
type tuple [2]any
type iface struct{ X any }
util/deephash: hash uint{8,16,32,64} explicitly (#2502) Instead of hashing the humanly formatted forms of a number, hash the native machine bits of the integers themselves. There is a small performance gain for this: name old time/op new time/op delta Hash-8 75.7µs ± 1% 76.0µs ± 2% ~ (p=0.315 n=10+9) HashMapAcyclic-8 63.1µs ± 3% 61.3µs ± 1% -2.77% (p=0.000 n=10+10) TailcfgNode-8 10.3µs ± 1% 10.2µs ± 1% -1.48% (p=0.000 n=10+10) HashArray-8 1.07µs ± 1% 1.05µs ± 1% -1.79% (p=0.000 n=10+10) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-03 04:44:13 +00:00
type scalars struct {
I8 int8
I16 int16
I32 int32
I64 int64
I int
U8 uint8
U16 uint16
U32 uint32
U64 uint64
U uint
UP uintptr
F32 float32
F64 float64
C64 complex64
C128 complex128
}
util/deephash: include type as part of hash for interfaces (#2476) A Go interface may hold any number of different concrete types. Just because two underlying values hash to the same thing does not mean the two values are identical if they have different concrete types. As such, include the type in the hash.
2021-07-21 17:26:04 +00:00
type MyBool bool
type MyHeader tar.Header
util/deephash: fix collisions between different types Updates #4883 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-22 02:50:48 +00:00
var zeroFloat64 float64
util/deephash: include type as part of hash for interfaces (#2476) A Go interface may hold any number of different concrete types. Just because two underlying values hash to the same thing does not mean the two values are identical if they have different concrete types. As such, include the type in the hash.
2021-07-21 17:26:04 +00:00
tests := []struct {
in tuple
wantEq bool
}{
util/deephash: hash uint{8,16,32,64} explicitly (#2502) Instead of hashing the humanly formatted forms of a number, hash the native machine bits of the integers themselves. There is a small performance gain for this: name old time/op new time/op delta Hash-8 75.7µs ± 1% 76.0µs ± 2% ~ (p=0.315 n=10+9) HashMapAcyclic-8 63.1µs ± 3% 61.3µs ± 1% -2.77% (p=0.000 n=10+10) TailcfgNode-8 10.3µs ± 1% 10.2µs ± 1% -1.48% (p=0.000 n=10+10) HashArray-8 1.07µs ± 1% 1.05µs ± 1% -1.79% (p=0.000 n=10+10) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-03 04:44:13 +00:00
{in: tuple{false, true}, wantEq: false},
{in: tuple{true, true}, wantEq: true},
{in: tuple{false, false}, wantEq: true},
{
in: tuple{
scalars{-8, -16, -32, -64, -1234, 8, 16, 32, 64, 1234, 5678, 32.32, 64.64, 32 + 32i, 64 + 64i},
scalars{-8, -16, -32, -64, -1234, 8, 16, 32, 64, 1234, 5678, 32.32, 64.64, 32 + 32i, 64 + 64i},
},
wantEq: true,
},
{in: tuple{scalars{I8: math.MinInt8}, scalars{I8: math.MinInt8 / 2}}, wantEq: false},
{in: tuple{scalars{I16: math.MinInt16}, scalars{I16: math.MinInt16 / 2}}, wantEq: false},
{in: tuple{scalars{I32: math.MinInt32}, scalars{I32: math.MinInt32 / 2}}, wantEq: false},
{in: tuple{scalars{I64: math.MinInt64}, scalars{I64: math.MinInt64 / 2}}, wantEq: false},
{in: tuple{scalars{I: -1234}, scalars{I: -1234 / 2}}, wantEq: false},
{in: tuple{scalars{U8: math.MaxUint8}, scalars{U8: math.MaxUint8 / 2}}, wantEq: false},
{in: tuple{scalars{U16: math.MaxUint16}, scalars{U16: math.MaxUint16 / 2}}, wantEq: false},
{in: tuple{scalars{U32: math.MaxUint32}, scalars{U32: math.MaxUint32 / 2}}, wantEq: false},
{in: tuple{scalars{U64: math.MaxUint64}, scalars{U64: math.MaxUint64 / 2}}, wantEq: false},
{in: tuple{scalars{U: 1234}, scalars{U: 1234 / 2}}, wantEq: false},
{in: tuple{scalars{UP: 5678}, scalars{UP: 5678 / 2}}, wantEq: false},
{in: tuple{scalars{F32: 32.32}, scalars{F32: math.Nextafter32(32.32, 0)}}, wantEq: false},
{in: tuple{scalars{F64: 64.64}, scalars{F64: math.Nextafter(64.64, 0)}}, wantEq: false},
{in: tuple{scalars{F32: float32(math.NaN())}, scalars{F32: float32(math.NaN())}}, wantEq: true},
{in: tuple{scalars{F64: float64(math.NaN())}, scalars{F64: float64(math.NaN())}}, wantEq: true},
{in: tuple{scalars{C64: 32 + 32i}, scalars{C64: complex(math.Nextafter32(32, 0), 32)}}, wantEq: false},
{in: tuple{scalars{C128: 64 + 64i}, scalars{C128: complex(math.Nextafter(64, 0), 64)}}, wantEq: false},
util/deephash: disambiguate hashing of AppendTo (#2483) Prepend size to AppendTo output. Fixes #2443 Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-07-21 18:29:08 +00:00
{in: tuple{[]appendBytes{{}, {0, 0, 0, 0, 0, 0, 0, 1}}, []appendBytes{{}, {0, 0, 0, 0, 0, 0, 0, 1}}}, wantEq: true},
{in: tuple{[]appendBytes{{}, {0, 0, 0, 0, 0, 0, 0, 1}}, []appendBytes{{0, 0, 0, 0, 0, 0, 0, 1}, {}}}, wantEq: false},
util/deephash: include type as part of hash for interfaces (#2476) A Go interface may hold any number of different concrete types. Just because two underlying values hash to the same thing does not mean the two values are identical if they have different concrete types. As such, include the type in the hash.
2021-07-21 17:26:04 +00:00
{in: tuple{iface{MyBool(true)}, iface{MyBool(true)}}, wantEq: true},
{in: tuple{iface{true}, iface{MyBool(true)}}, wantEq: false},
{in: tuple{iface{MyHeader{}}, iface{MyHeader{}}}, wantEq: true},
{in: tuple{iface{MyHeader{}}, iface{tar.Header{}}}, wantEq: false},
{in: tuple{iface{&MyHeader{}}, iface{&MyHeader{}}}, wantEq: true},
{in: tuple{iface{&MyHeader{}}, iface{&tar.Header{}}}, wantEq: false},
{in: tuple{iface{[]map[string]MyBool{}}, iface{[]map[string]MyBool{}}}, wantEq: true},
{in: tuple{iface{[]map[string]bool{}}, iface{[]map[string]MyBool{}}}, wantEq: false},
util/deephash: fix collisions between different types Updates #4883 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-22 02:50:48 +00:00
{in: tuple{zeroFloat64, -zeroFloat64}, wantEq: false}, // Issue 4883 (false alarm)
{in: tuple{[]any(nil), 0.0}, wantEq: false}, // Issue 4883
{in: tuple{[]any(nil), uint8(0)}, wantEq: false}, // Issue 4883
{in: tuple{nil, nil}, wantEq: true}, // Issue 4883
util/deephash: improve cycle detection (#2470) The previous algorithm used a map of all visited pointers. The strength of this approach is that it quickly prunes any nodes that we have ever visited before. The detriment of the approach is that pruning is heavily dependent on the order that pointers were visited. This is especially relevant for hashing a map where map entries are visited in a non-deterministic manner, which would cause the map hash to be non-deterministic (which defeats the point of a hash). This new algorithm uses a stack of all visited pointers, similar to how github.com/google/go-cmp performs cycle detection. When we visit a pointer, we push it onto the stack, and when we leave a pointer, we pop it from the stack. Before visiting a pointer, we first check whether the pointer exists anywhere in the stack. If yes, then we prune the node. The detriment of this approach is that we may hash a node more often than before since we do not prune as aggressively. The set of visited pointers up until any node is only the path of nodes up to that node and not any other pointers that may have been visited elsewhere. This provides us deterministic hashing regardless of visit order. We can now delete hashMapFallback and associated complexity, which only exists because the previous approach was non-deterministic in the presence of cycles. This fixes a failure of the old algorithm where obviously different values are treated as equal because the pruning was too aggresive. See https://github.com/tailscale/tailscale/issues/2443#issuecomment-883653534 The new algorithm is slightly slower since it prunes less aggresively: name old time/op new time/op delta Hash-8 66.1µs ± 1% 68.8µs ± 1% +4.09% (p=0.000 n=19+19) HashMapAcyclic-8 63.0µs ± 1% 62.5µs ± 1% -0.76% (p=0.000 n=18+19) TailcfgNode-8 9.79µs ± 2% 9.88µs ± 1% +0.95% (p=0.000 n=19+17) HashArray-8 643ns ± 1% 653ns ± 1% +1.64% (p=0.000 n=19+19) However, a slower but more correct algorithm seems more favorable than a faster but incorrect algorithm. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-07-22 22:22:48 +00:00
{
in: func() tuple {
i1 := 1
i2 := 2
v1 := [3]*int{&i1, &i2, &i1}
v2 := [3]*int{&i1, &i2, &i2}
return tuple{v1, v2}
}(),
wantEq: false,
},
util/deephash: specialize for netip.Addr and drop AppendTo support (#5402) There are 5 types that we care about that implement AppendTo: key.DiscoPublic key.NodePublic netip.Prefix netipx.IPRange netip.Addr The key types are thin wrappers around [32]byte and are memory hashable. The netip.Prefix and netipx.IPRange types are thin wrappers over netip.Addr and are hashable by default if netip.Addr is hashable. The netip.Addr type is the only one with a complex structure where the default behavior of deephash does not hash it correctly due to the presence of the intern.Value type. Drop support for AppendTo and instead add specialized hashing for netip.Addr that would be semantically equivalent to == on the netip.Addr values. The AppendTo support was already broken prior to this change. It was fully removed (intentionally or not) in #4870. It was partially restored in #4858 for the fast path, but still broken in the slow path. Just drop support for it altogether. This does mean we lack any ability for types to self-hash themselves. In the future we can add support for types that implement: interface { DeepHash() Sum } Test and fuzz cases were added for the relevant types that used to rely on the AppendTo method. FuzzAddr has been executed on 1 billion samples without issues. Signed-off-by: Joe Tsai joetsai@digital-static.net
2022-08-19 05:54:56 +00:00
{in: tuple{netip.Addr{}, netip.Addr{}}, wantEq: true},
{in: tuple{netip.Addr{}, netip.AddrFrom4([4]byte{})}, wantEq: false},
{in: tuple{netip.AddrFrom4([4]byte{}), netip.AddrFrom4([4]byte{})}, wantEq: true},
{in: tuple{netip.AddrFrom4([4]byte{192, 168, 0, 1}), netip.AddrFrom4([4]byte{192, 168, 0, 1})}, wantEq: true},
{in: tuple{netip.AddrFrom4([4]byte{192, 168, 0, 1}), netip.AddrFrom4([4]byte{192, 168, 0, 2})}, wantEq: false},
{in: tuple{netip.AddrFrom4([4]byte{}), netip.AddrFrom16([16]byte{})}, wantEq: false},
{in: tuple{netip.AddrFrom16([16]byte{}), netip.AddrFrom16([16]byte{})}, wantEq: true},
{in: tuple{netip.AddrPort{}, netip.AddrPort{}}, wantEq: true},
{in: tuple{netip.AddrPort{}, netip.AddrPortFrom(netip.AddrFrom4([4]byte{}), 0)}, wantEq: false},
{in: tuple{netip.AddrPortFrom(netip.AddrFrom4([4]byte{}), 0), netip.AddrPortFrom(netip.AddrFrom4([4]byte{}), 0)}, wantEq: true},
{in: tuple{netip.AddrPortFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 1234), netip.AddrPortFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 1234)}, wantEq: true},
{in: tuple{netip.AddrPortFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 1234), netip.AddrPortFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 1235)}, wantEq: false},
{in: tuple{netip.AddrPortFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 1234), netip.AddrPortFrom(netip.AddrFrom4([4]byte{192, 168, 0, 2}), 1234)}, wantEq: false},
{in: tuple{netip.Prefix{}, netip.Prefix{}}, wantEq: true},
{in: tuple{netip.Prefix{}, netip.PrefixFrom(netip.Addr{}, 1)}, wantEq: false},
{in: tuple{netip.Prefix{}, netip.PrefixFrom(netip.AddrFrom4([4]byte{}), 0)}, wantEq: false},
{in: tuple{netip.PrefixFrom(netip.AddrFrom4([4]byte{}), 1), netip.PrefixFrom(netip.AddrFrom4([4]byte{}), 1)}, wantEq: true},
{in: tuple{netip.PrefixFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 1), netip.PrefixFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 1)}, wantEq: true},
{in: tuple{netip.PrefixFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 1), netip.PrefixFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 0)}, wantEq: false},
{in: tuple{netip.PrefixFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), 1), netip.PrefixFrom(netip.AddrFrom4([4]byte{192, 168, 0, 2}), 1)}, wantEq: false},
{in: tuple{netipx.IPRange{}, netipx.IPRange{}}, wantEq: true},
{in: tuple{netipx.IPRange{}, netipx.IPRangeFrom(netip.AddrFrom4([4]byte{}), netip.AddrFrom16([16]byte{}))}, wantEq: false},
{in: tuple{netipx.IPRangeFrom(netip.AddrFrom4([4]byte{}), netip.AddrFrom16([16]byte{})), netipx.IPRangeFrom(netip.AddrFrom4([4]byte{}), netip.AddrFrom16([16]byte{}))}, wantEq: true},
{in: tuple{netipx.IPRangeFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), netip.AddrFrom4([4]byte{192, 168, 0, 100})), netipx.IPRangeFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), netip.AddrFrom4([4]byte{192, 168, 0, 100}))}, wantEq: true},
{in: tuple{netipx.IPRangeFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), netip.AddrFrom4([4]byte{192, 168, 0, 100})), netipx.IPRangeFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), netip.AddrFrom4([4]byte{192, 168, 0, 101}))}, wantEq: false},
{in: tuple{netipx.IPRangeFrom(netip.AddrFrom4([4]byte{192, 168, 0, 1}), netip.AddrFrom4([4]byte{192, 168, 0, 100})), netipx.IPRangeFrom(netip.AddrFrom4([4]byte{192, 168, 0, 2}), netip.AddrFrom4([4]byte{192, 168, 0, 100}))}, wantEq: false},
{in: tuple{key.DiscoPublic{}, key.DiscoPublic{}}, wantEq: true},
{in: tuple{key.DiscoPublic{}, key.DiscoPublicFromRaw32(mem.B(func() []byte {
b := make([]byte, 32)
b[0] = 1
return b
}()))}, wantEq: false},
{in: tuple{key.NodePublic{}, key.NodePublic{}}, wantEq: true},
{in: tuple{key.NodePublic{}, key.NodePublicFromRaw32(mem.B(func() []byte {
b := make([]byte, 32)
b[0] = 1
return b
}()))}, wantEq: false},
util/deephash: include type as part of hash for interfaces (#2476) A Go interface may hold any number of different concrete types. Just because two underlying values hash to the same thing does not mean the two values are identical if they have different concrete types. As such, include the type in the hash.
2021-07-21 17:26:04 +00:00
}
for _, tt := range tests {
gotEq := Hash(tt.in[0]) == Hash(tt.in[1])
if gotEq != tt.wantEq {
util/deephash: fix collisions between different types Updates #4883 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-22 02:50:48 +00:00
t.Errorf("(Hash(%T %v) == Hash(%T %v)) = %v, want %v", tt.in[0], tt.in[0], tt.in[1], tt.in[1], gotEq, tt.wantEq)
util/deephash: include type as part of hash for interfaces (#2476) A Go interface may hold any number of different concrete types. Just because two underlying values hash to the same thing does not mean the two values are identical if they have different concrete types. As such, include the type in the hash.
2021-07-21 17:26:04 +00:00
}
}
}
internal/deephash: simplify API Reduce to just a single external endpoint. Convert from a variadic number of interfaces to a slice there. name old time/op new time/op delta Hash-8 14.4µs ± 0% 14.0µs ± 1% -3.08% (p=0.000 n=9+9) name old alloc/op new alloc/op delta Hash-8 873B ± 0% 793B ± 0% -9.16% (p=0.000 n=9+6) name old allocs/op new allocs/op delta Hash-8 18.0 ± 0% 14.0 ± 0% -22.22% (p=0.000 n=10+10) Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-05-18 20:16:37 +00:00
func TestDeepHash(t *testing.T) {
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
// v contains the types of values we care about for our current callers.
// Mostly we're just testing that we don't panic on handled types.
v := getVal()
util/deephash: export a Hash func for use by the control plane name old time/op new time/op delta Hash-6 69.4µs ± 6% 68.4µs ± 4% ~ (p=0.286 n=9+9) HashMapAcyclic-6 115µs ± 5% 115µs ± 4% ~ (p=1.000 n=10+10) name old alloc/op new alloc/op delta Hash-6 2.29kB ± 0% 1.88kB ± 0% -18.13% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (all equal) name old allocs/op new allocs/op delta Hash-6 58.0 ± 0% 54.0 ± 0% -6.90% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-05 04:25:15 +00:00
hash1 := Hash(v)
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
t.Logf("hash: %v", hash1)
for i := 0; i < 20; i++ {
util/deephash: export a Hash func for use by the control plane name old time/op new time/op delta Hash-6 69.4µs ± 6% 68.4µs ± 4% ~ (p=0.286 n=9+9) HashMapAcyclic-6 115µs ± 5% 115µs ± 4% ~ (p=1.000 n=10+10) name old alloc/op new alloc/op delta Hash-6 2.29kB ± 0% 1.88kB ± 0% -18.13% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (all equal) name old allocs/op new allocs/op delta Hash-6 58.0 ± 0% 54.0 ± 0% -6.90% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-05 04:25:15 +00:00
hash2 := Hash(getVal())
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
if hash1 != hash2 {
t.Error("second hash didn't match")
}
}
}
util/deephash: fix map hashing to actually hash elements Fixes #4868 Change-Id: I574fd139cb7f7033dd93527344e6aa0e625477c7 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-16 18:24:04 +00:00
// Tests that we actually hash map elements. Whoops.
func TestIssue4868(t *testing.T) {
m1 := map[int]string{1: "foo"}
m2 := map[int]string{1: "bar"}
if Hash(m1) == Hash(m2) {
t.Error("bogus")
}
}
util/deephash: fix map hashing when key & element have the same type Regression from 09afb8e35b56c1150ae9872d781c7ae3c151b848, in which the same reflect.Value scratch value was being used as the map iterator copy destination. Also: make nil and empty maps hash differently, add test. Fixes #4871 Co-authored-by: Josh Bleecher Snyder <josharian@gmail.com> Change-Id: I67f42524bc81f694c1b7259d6682200125ea4a66 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-16 20:21:32 +00:00
func TestIssue4871(t *testing.T) {
m1 := map[string]string{"": "", "x": "foo"}
m2 := map[string]string{}
if h1, h2 := Hash(m1), Hash(m2); h1 == h2 {
t.Errorf("bogus: h1=%x, h2=%x", h1, h2)
}
}
func TestNilVsEmptymap(t *testing.T) {
m1 := map[string]string(nil)
m2 := map[string]string{}
if h1, h2 := Hash(m1), Hash(m2); h1 == h2 {
t.Errorf("bogus: h1=%x, h2=%x", h1, h2)
}
}
func TestMapFraming(t *testing.T) {
m1 := map[string]string{"foo": "", "fo": "o"}
m2 := map[string]string{}
if h1, h2 := Hash(m1), Hash(m2); h1 == h2 {
t.Errorf("bogus: h1=%x, h2=%x", h1, h2)
}
}
func TestQuick(t *testing.T) {
initSeed()
err := quick.Check(func(v, w map[string]string) bool {
return (Hash(v) == Hash(w)) == reflect.DeepEqual(v, w)
}, &quick.Config{MaxCount: 1000, Rand: rand.New(rand.NewSource(int64(seed)))})
if err != nil {
t.Fatalf("seed=%v, err=%v", seed, err)
}
}
util/deephash: always keep values addressable (#5328) The logic of deephash is both simpler and easier to reason about if values are always addressable. In Go, the composite kinds are slices, arrays, maps, structs, interfaces, pointers, channels, and functions, where we define "composite" as a Go value that encapsulates some other Go value (e.g., a map is a collection of key-value entries). In the cases of pointers and slices, the sub-values are always addressable. In the cases of arrays and structs, the sub-values are always addressable if and only if the parent value is addressable. In the case of maps and interfaces, the sub-values are never addressable. To make them addressable, we need to copy them onto the heap. For the purposes of deephash, we do not care about channels and functions. For all non-composite kinds (e.g., strings and ints), they are only addressable if obtained from one of the composite kinds that produce addressable values (i.e., pointers, slices, addressable arrays, and addressable structs). A non-addressible, non-composite kind can be made addressable by allocating it on the heap, obtaining a pointer to it, and dereferencing it. Thus, if we can ensure that values are addressable at the entry points, and shallow copy sub-values whenever we encounter an interface or map, then we can ensure that all values are always addressable and assume such property throughout all the logic. Performance: name old time/op new time/op delta Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9) HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9) HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10) TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10) HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
func getVal() any {
return &struct {
WGConfig *wgcfg.Config
RouterConfig *router.Config
MapFQDNAddrs map[dnsname.FQDN][]netip.Addr
MapFQDNAddrPorts map[dnsname.FQDN][]netip.AddrPort
MapDiscoPublics map[key.DiscoPublic]bool
MapResponse *tailcfg.MapResponse
FilterMatch filter.Match
}{
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
&wgcfg.Config{
wgengine/wgcfg: remove Config.ListenPort We don't use the port that wireguard-go passes to us (via magicsock.connBind.Open). We ignore it entirely and use the port we selected. When we tell wireguard-go that we're changing the listen_port, it calls connBind.Close and then connBind.Open. And in the meantime, it stops calling the receive functions, which means that we stop receiving and processing UDP and DERP packets. And that is Very Bad. That was never a problem prior to b3ceca1dd7d7a1a6f9ddab136a4e12900e976333, because we passed the SkipBindUpdate flag to our wireguard-go fork, which told wireguard-go not to re-bind on listen_port changes. That commit eliminated the SkipBindUpdate flag. We could write a bunch of code to work around the gap. We could add background readers that process UDP and DERP packets when wireguard-go isn't. But it's simpler to never create the conditions in which wireguard-go rebinds. The other scenario in which wireguard-go re-binds is device.Down. Conveniently, we never call device.Down. We go from device.Up to device.Close, and the latter only when we're shutting down a magicsock.Conn completely. Rubber-ducked-by: Avery Pennarun <apenwarr@tailscale.com> Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-04-03 01:04:39 +00:00
Name: "foo",
all: migrate more code code to net/netip directly Instead of going through the tailscale.com/net/netaddr transitional wrappers. Updates #5162 Change-Id: I3dafd1c2effa1a6caa9b7151ecf6edd1a3fda3dd Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-08-02 20:38:11 +00:00
Addresses: []netip.Prefix{netip.PrefixFrom(netip.AddrFrom16([16]byte{3: 3}).Unmap(), 5)},
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
Peers: []wgcfg.Peer{
{
wgengine/wgcfg: convert to use new node key type. Updates #3206 Signed-off-by: David Anderson <danderson@tailscale.com>
2021-10-28 00:42:33 +00:00
PublicKey: key.NodePublic{},
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
},
},
},
&router.Config{
all: convert more code to use net/netip directly perl -i -npe 's,netaddr.IPPrefixFrom,netip.PrefixFrom,' $(git grep -l -F netaddr.) perl -i -npe 's,netaddr.IPPortFrom,netip.AddrPortFrom,' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPrefix,netip.Prefix,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPort,netip.AddrPort,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IP\b,netip.Addr,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPv6Raw\b,netip.AddrFrom16,g' $(git grep -l -F netaddr. ) goimports -w . Then delete some stuff from the net/netaddr shim package which is no longer neeed. Updates #5162 Change-Id: Ia7a86893fe21c7e3ee1ec823e8aba288d4566cd8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 04:14:09 +00:00
Routes: []netip.Prefix{
all: use various net/netip parse funcs directly Mechanical change with perl+goimports. Changed {Must,}Parse{IP,IPPrefix,IPPort} to their netip variants, then goimports -d . Finally, removed the net/netaddr wrappers, to prevent future use. Updates #5162 Change-Id: I59c0e38b5fbca5a935d701645789cddf3d7863ad Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 03:55:44 +00:00
netip.MustParsePrefix("1.2.3.0/24"),
netip.MustParsePrefix("1234::/64"),
wgengine/router: dns: unify on *BSD, multimode on Linux, Magic DNS (#536) Signed-off-by: Dmytro Shynkevych <dmytro@tailscale.com>
2020-07-14 13:12:00 +00:00
},
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
},
all: convert more code to use net/netip directly perl -i -npe 's,netaddr.IPPrefixFrom,netip.PrefixFrom,' $(git grep -l -F netaddr.) perl -i -npe 's,netaddr.IPPortFrom,netip.AddrPortFrom,' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPrefix,netip.Prefix,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPort,netip.AddrPort,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IP\b,netip.Addr,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPv6Raw\b,netip.AddrFrom16,g' $(git grep -l -F netaddr. ) goimports -w . Then delete some stuff from the net/netaddr shim package which is no longer neeed. Updates #5162 Change-Id: Ia7a86893fe21c7e3ee1ec823e8aba288d4566cd8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 04:14:09 +00:00
map[dnsname.FQDN][]netip.Addr{
all: use various net/netip parse funcs directly Mechanical change with perl+goimports. Changed {Must,}Parse{IP,IPPrefix,IPPort} to their netip variants, then goimports -d . Finally, removed the net/netaddr wrappers, to prevent future use. Updates #5162 Change-Id: I59c0e38b5fbca5a935d701645789cddf3d7863ad Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 03:55:44 +00:00
dnsname.FQDN("a."): {netip.MustParseAddr("1.2.3.4"), netip.MustParseAddr("4.3.2.1")},
dnsname.FQDN("b."): {netip.MustParseAddr("8.8.8.8"), netip.MustParseAddr("9.9.9.9")},
dnsname.FQDN("c."): {netip.MustParseAddr("6.6.6.6"), netip.MustParseAddr("7.7.7.7")},
dnsname.FQDN("d."): {netip.MustParseAddr("6.7.6.6"), netip.MustParseAddr("7.7.7.8")},
dnsname.FQDN("e."): {netip.MustParseAddr("6.8.6.6"), netip.MustParseAddr("7.7.7.9")},
dnsname.FQDN("f."): {netip.MustParseAddr("6.9.6.6"), netip.MustParseAddr("7.7.7.0")},
internal/deepprint: improve benchmark This more closely matches our real usage of deepprint. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-05-11 16:45:12 +00:00
},
all: convert more code to use net/netip directly perl -i -npe 's,netaddr.IPPrefixFrom,netip.PrefixFrom,' $(git grep -l -F netaddr.) perl -i -npe 's,netaddr.IPPortFrom,netip.AddrPortFrom,' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPrefix,netip.Prefix,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPort,netip.AddrPort,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IP\b,netip.Addr,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPv6Raw\b,netip.AddrFrom16,g' $(git grep -l -F netaddr. ) goimports -w . Then delete some stuff from the net/netaddr shim package which is no longer neeed. Updates #5162 Change-Id: Ia7a86893fe21c7e3ee1ec823e8aba288d4566cd8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 04:14:09 +00:00
map[dnsname.FQDN][]netip.AddrPort{
all: use various net/netip parse funcs directly Mechanical change with perl+goimports. Changed {Must,}Parse{IP,IPPrefix,IPPort} to their netip variants, then goimports -d . Finally, removed the net/netaddr wrappers, to prevent future use. Updates #5162 Change-Id: I59c0e38b5fbca5a935d701645789cddf3d7863ad Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 03:55:44 +00:00
dnsname.FQDN("a."): {netip.MustParseAddrPort("1.2.3.4:11"), netip.MustParseAddrPort("4.3.2.1:22")},
dnsname.FQDN("b."): {netip.MustParseAddrPort("8.8.8.8:11"), netip.MustParseAddrPort("9.9.9.9:22")},
dnsname.FQDN("c."): {netip.MustParseAddrPort("8.8.8.8:12"), netip.MustParseAddrPort("9.9.9.9:23")},
dnsname.FQDN("d."): {netip.MustParseAddrPort("8.8.8.8:13"), netip.MustParseAddrPort("9.9.9.9:24")},
dnsname.FQDN("e."): {netip.MustParseAddrPort("8.8.8.8:14"), netip.MustParseAddrPort("9.9.9.9:25")},
internal/deepprint: improve benchmark This more closely matches our real usage of deepprint. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-05-11 16:45:12 +00:00
},
all: replace tailcfg.DiscoKey with key.DiscoPublic. Signed-off-by: David Anderson <danderson@tailscale.com>
2021-11-02 21:41:56 +00:00
map[key.DiscoPublic]bool{
key.DiscoPublicFromRaw32(mem.B([]byte{1: 1, 31: 0})): true,
key.DiscoPublicFromRaw32(mem.B([]byte{1: 2, 31: 0})): false,
key.DiscoPublicFromRaw32(mem.B([]byte{1: 3, 31: 0})): true,
key.DiscoPublicFromRaw32(mem.B([]byte{1: 4, 31: 0})): false,
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
},
util/deephash: add a benchmark and some benchmark data No code changes. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 04:21:52 +00:00
&tailcfg.MapResponse{
DERPMap: &tailcfg.DERPMap{
Regions: map[int]*tailcfg.DERPRegion{
all: gofmt -w -s (simplify) tests And it updates the build tag style on a couple files. Change-Id: I84478d822c8de3f84b56fa1176c99d2ea5083237 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-12-15 16:42:25 +00:00
1: {
util/deephash: add a benchmark and some benchmark data No code changes. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 04:21:52 +00:00
RegionID: 1,
RegionCode: "foo",
Nodes: []*tailcfg.DERPNode{
{
Name: "n1",
RegionID: 1,
HostName: "foo.com",
},
{
Name: "n2",
RegionID: 1,
HostName: "bar.com",
},
},
},
},
},
DNSConfig: &tailcfg.DNSConfig{
tailcfg,all: change structs to []*dnstype.Resolver Signed-off-by: Maisem Ali <maisem@tailscale.com>
2022-05-03 21:41:58 +00:00
Resolvers: []*dnstype.Resolver{
util/deephash: add a benchmark and some benchmark data No code changes. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 04:21:52 +00:00
{Addr: "10.0.0.1"},
},
},
PacketFilter: []tailcfg.FilterRule{
{
SrcIPs: []string{"1.2.3.4"},
DstPorts: []tailcfg.NetPortRange{
{
IP: "1.2.3.4/32",
util/deephash: use keyed struct literal, fix vet Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 04:31:30 +00:00
Ports: tailcfg.PortRange{First: 1, Last: 2},
util/deephash: add a benchmark and some benchmark data No code changes. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 04:21:52 +00:00
},
},
},
},
Peers: []*tailcfg.Node{
{
ID: 1,
},
{
ID: 2,
},
},
UserProfiles: []tailcfg.UserProfile{
{ID: 1, LoginName: "foo@bar.com"},
{ID: 2, LoginName: "bar@foo.com"},
},
},
util/deephash: don't reflect.Copy if element type is a defined uint8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-07 18:58:02 +00:00
filter.Match{
IPProto: []ipproto.Proto{1, 2, 3},
},
wgengine, internal/deepprint: replace UAPI usage as hash func; add deepprint The new deepprint package just walks a Go data structure and writes to an io.Writer. It's not pretty like go-spew, etc. We then use it to replace the use of UAPI (which we have a TODO to remove) to generate signatures of data structures to detect whether anything changed (without retaining the old copy). This was necessary because the UAPI conversion ends up trying to do DNS lookups which an upcoming change depends on not happening.
2020-06-28 17:58:21 +00:00
}
}
internal/deepprint: add BenchmarkHash deepprint currently accounts for 15% of allocs in tailscaled. This is a useful benchmark to have. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-05-08 01:05:04 +00:00
util/deephash: add canMemHash func + typeInfo property Currently unused. (breaking up a bigger change) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-25 19:46:01 +00:00
type IntThenByte struct {
i int
b byte
}
type TwoInts struct{ a, b int }
type IntIntByteInt struct {
i1, i2 int32
b byte // padding after
i3 int32
}
util/deephash: use binary encoding of time.Time (#5352) Formatting a time.Time as RFC3339 is slow. See https://go.dev/issue/54093 Now that we have efficient hashing of fixed-width integers, just hash the time.Time as a binary value. Performance: Hash-24 19.0µs ± 1% 18.6µs ± 1% -2.03% (p=0.000 n=10+9) TailcfgNode-24 1.79µs ± 1% 1.40µs ± 1% -21.74% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 21:42:51 +00:00
func u8(n uint8) string { return string([]byte{n}) }
func u16(n uint16) string { return string(binary.LittleEndian.AppendUint16(nil, n)) }
func u32(n uint32) string { return string(binary.LittleEndian.AppendUint32(nil, n)) }
func u64(n uint64) string { return string(binary.LittleEndian.AppendUint64(nil, n)) }
util/deephash: rely on direct memory hashing for primitive kinds (#5457) Rather than separate functions to hash each kind, just rely on the fact that these are direct memory hashable, thus simplifying the code. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 03:50:56 +00:00
func ux(n uint) string {
if bits.UintSize == 32 {
return u32(uint32(n))
} else {
return u64(uint64(n))
}
}
util/deephash: use binary encoding of time.Time (#5352) Formatting a time.Time as RFC3339 is slow. See https://go.dev/issue/54093 Now that we have efficient hashing of fixed-width integers, just hash the time.Time as a binary value. Performance: Hash-24 19.0µs ± 1% 18.6µs ± 1% -2.03% (p=0.000 n=10+9) TailcfgNode-24 1.79µs ± 1% 1.40µs ± 1% -21.74% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 21:42:51 +00:00
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
func TestGetTypeHasher(t *testing.T) {
switch runtime.GOARCH {
case "amd64", "arm64", "arm", "386", "riscv64":
default:
// Test outputs below are specifically for little-endian machines.
// Just skip everything else for now. Feel free to add more above if
// you have the hardware to test and it's little-endian.
t.Skipf("skipping on %v", runtime.GOARCH)
}
type typedString string
var (
someInt = int('A')
someComplex128 = complex128(1 + 2i)
all: use various net/netip parse funcs directly Mechanical change with perl+goimports. Changed {Must,}Parse{IP,IPPrefix,IPPort} to their netip variants, then goimports -d . Finally, removed the net/netaddr wrappers, to prevent future use. Updates #5162 Change-Id: I59c0e38b5fbca5a935d701645789cddf3d7863ad Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 03:55:44 +00:00
someIP = netip.MustParseAddr("1.2.3.4")
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
)
tests := []struct {
name string
val any
out string
out32 string // overwrites out if 32-bit
}{
{
name: "int",
val: int(1),
util/deephash: rely on direct memory hashing for primitive kinds (#5457) Rather than separate functions to hash each kind, just rely on the fact that these are direct memory hashable, thus simplifying the code. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 03:50:56 +00:00
out: ux(1),
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
},
{
name: "int_negative",
val: int(-1),
util/deephash: rely on direct memory hashing for primitive kinds (#5457) Rather than separate functions to hash each kind, just rely on the fact that these are direct memory hashable, thus simplifying the code. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 03:50:56 +00:00
out: ux(math.MaxUint),
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
},
{
name: "int8",
val: int8(1),
out: "\x01",
},
{
name: "float64",
val: float64(1.0),
out: "\x00\x00\x00\x00\x00\x00\xf0?",
},
{
name: "float32",
val: float32(1.0),
out: "\x00\x00\x80?",
},
{
name: "string",
val: "foo",
out: "\x03\x00\x00\x00\x00\x00\x00\x00foo",
},
{
name: "typedString",
val: typedString("foo"),
out: "\x03\x00\x00\x00\x00\x00\x00\x00foo",
},
{
name: "string_slice",
val: []string{"foo", "bar"},
out: "\x02\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00foo\x03\x00\x00\x00\x00\x00\x00\x00bar",
},
{
name: "int_slice",
val: []int{1, 0, -1},
out: "\x03\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff",
out32: "\x03\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff",
},
{
name: "struct",
val: struct {
a, b int
c uint16
}{1, -1, 2},
util/deephash: always keep values addressable (#5328) The logic of deephash is both simpler and easier to reason about if values are always addressable. In Go, the composite kinds are slices, arrays, maps, structs, interfaces, pointers, channels, and functions, where we define "composite" as a Go value that encapsulates some other Go value (e.g., a map is a collection of key-value entries). In the cases of pointers and slices, the sub-values are always addressable. In the cases of arrays and structs, the sub-values are always addressable if and only if the parent value is addressable. In the case of maps and interfaces, the sub-values are never addressable. To make them addressable, we need to copy them onto the heap. For the purposes of deephash, we do not care about channels and functions. For all non-composite kinds (e.g., strings and ints), they are only addressable if obtained from one of the composite kinds that produce addressable values (i.e., pointers, slices, addressable arrays, and addressable structs). A non-addressible, non-composite kind can be made addressable by allocating it on the heap, obtaining a pointer to it, and dereferencing it. Thus, if we can ensure that values are addressable at the entry points, and shallow copy sub-values whenever we encounter an interface or map, then we can ensure that all values are always addressable and assume such property throughout all the logic. Performance: name old time/op new time/op delta Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9) HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9) HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10) TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10) HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
out: "\x01\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\x02\x00",
out32: "\x01\x00\x00\x00\xff\xff\xff\xff\x02\x00",
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
},
{
name: "nil_int_ptr",
val: (*int)(nil),
out: "\x00",
},
{
name: "int_ptr",
val: &someInt,
out: "\x01A\x00\x00\x00\x00\x00\x00\x00",
out32: "\x01A\x00\x00\x00",
},
{
name: "nil_uint32_ptr",
val: (*uint32)(nil),
out: "\x00",
},
{
name: "complex128_ptr",
val: &someComplex128,
out: "\x01\x00\x00\x00\x00\x00\x00\xf0?\x00\x00\x00\x00\x00\x00\x00@",
},
{
name: "packet_filter",
val: filterRules,
util/deephash: specialize for netip.Addr and drop AppendTo support (#5402) There are 5 types that we care about that implement AppendTo: key.DiscoPublic key.NodePublic netip.Prefix netipx.IPRange netip.Addr The key types are thin wrappers around [32]byte and are memory hashable. The netip.Prefix and netipx.IPRange types are thin wrappers over netip.Addr and are hashable by default if netip.Addr is hashable. The netip.Addr type is the only one with a complex structure where the default behavior of deephash does not hash it correctly due to the presence of the intern.Value type. Drop support for AppendTo and instead add specialized hashing for netip.Addr that would be semantically equivalent to == on the netip.Addr values. The AppendTo support was already broken prior to this change. It was fully removed (intentionally or not) in #4870. It was partially restored in #4858 for the fast path, but still broken in the slow path. Just drop support for it altogether. This does mean we lack any ability for types to self-hash themselves. In the future we can add support for types that implement: interface { DeepHash() Sum } Test and fuzz cases were added for the relevant types that used to rely on the AppendTo method. FuzzAddr has been executed on 1 billion samples without issues. Signed-off-by: Joe Tsai joetsai@digital-static.net
2022-08-19 05:54:56 +00:00
out: "\x04\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00*\v\x00\x00\x00\x00\x00\x00\x0010.1.3.4/32\v\x00\x00\x00\x00\x00\x00\x0010.0.0.0/24\x03\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\n\x00\x00\x00\x00\x00\x00\x001.2.3.4/32\x01 \x00\x00\x00\x00\x00\x00\x00\x01\x00\x02\x00\x04\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x01\x02\x03\x04 \x00\x01\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00foo\x01\x00\x00\x00\x00\x00\x00\x00\v\x00\x00\x00\x00\x00\x00\x00foooooooooo\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\f\x00\x00\x00\x00\x00\x00\x00baaaaaarrrrr\x00\x01\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\v\x00\x00\x00\x00\x00\x00\x00foooooooooo\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\f\x00\x00\x00\x00\x00\x00\x00baaaaaarrrrr\x00\x01\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\v\x00\x00\x00\x00\x00\x00\x00foooooooooo\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\f\x00\x00\x00\x00\x00\x00\x00baaaaaarrrrr\x00\x01\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00",
out32: "\x04\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00*\v\x00\x00\x00\x00\x00\x00\x0010.1.3.4/32\v\x00\x00\x00\x00\x00\x00\x0010.0.0.0/24\x03\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\n\x00\x00\x00\x00\x00\x00\x001.2.3.4/32\x01 \x00\x00\x00\x01\x00\x02\x00\x04\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x01\x02\x03\x04 \x00\x01\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00foo\x01\x00\x00\x00\x00\x00\x00\x00\v\x00\x00\x00\x00\x00\x00\x00foooooooooo\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\f\x00\x00\x00\x00\x00\x00\x00baaaaaarrrrr\x00\x01\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\v\x00\x00\x00\x00\x00\x00\x00foooooooooo\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\f\x00\x00\x00\x00\x00\x00\x00baaaaaarrrrr\x00\x01\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\v\x00\x00\x00\x00\x00\x00\x00foooooooooo\x00\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\f\x00\x00\x00\x00\x00\x00\x00baaaaaarrrrr\x00\x01\x00\x02\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00",
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
},
{
all: convert more code to use net/netip directly perl -i -npe 's,netaddr.IPPrefixFrom,netip.PrefixFrom,' $(git grep -l -F netaddr.) perl -i -npe 's,netaddr.IPPortFrom,netip.AddrPortFrom,' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPrefix,netip.Prefix,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPort,netip.AddrPort,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IP\b,netip.Addr,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPv6Raw\b,netip.AddrFrom16,g' $(git grep -l -F netaddr. ) goimports -w . Then delete some stuff from the net/netaddr shim package which is no longer neeed. Updates #5162 Change-Id: Ia7a86893fe21c7e3ee1ec823e8aba288d4566cd8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 04:14:09 +00:00
name: "netip.Addr",
all: use various net/netip parse funcs directly Mechanical change with perl+goimports. Changed {Must,}Parse{IP,IPPrefix,IPPort} to their netip variants, then goimports -d . Finally, removed the net/netaddr wrappers, to prevent future use. Updates #5162 Change-Id: I59c0e38b5fbca5a935d701645789cddf3d7863ad Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 03:55:44 +00:00
val: netip.MustParseAddr("fe80::123%foo"),
util/deephash: specialize for netip.Addr and drop AppendTo support (#5402) There are 5 types that we care about that implement AppendTo: key.DiscoPublic key.NodePublic netip.Prefix netipx.IPRange netip.Addr The key types are thin wrappers around [32]byte and are memory hashable. The netip.Prefix and netipx.IPRange types are thin wrappers over netip.Addr and are hashable by default if netip.Addr is hashable. The netip.Addr type is the only one with a complex structure where the default behavior of deephash does not hash it correctly due to the presence of the intern.Value type. Drop support for AppendTo and instead add specialized hashing for netip.Addr that would be semantically equivalent to == on the netip.Addr values. The AppendTo support was already broken prior to this change. It was fully removed (intentionally or not) in #4870. It was partially restored in #4858 for the fast path, but still broken in the slow path. Just drop support for it altogether. This does mean we lack any ability for types to self-hash themselves. In the future we can add support for types that implement: interface { DeepHash() Sum } Test and fuzz cases were added for the relevant types that used to rely on the AppendTo method. FuzzAddr has been executed on 1 billion samples without issues. Signed-off-by: Joe Tsai joetsai@digital-static.net
2022-08-19 05:54:56 +00:00
out: u64(16+3) + u64(0x80fe) + u64(0x2301<<48) + "foo",
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
},
{
all: convert more code to use net/netip directly perl -i -npe 's,netaddr.IPPrefixFrom,netip.PrefixFrom,' $(git grep -l -F netaddr.) perl -i -npe 's,netaddr.IPPortFrom,netip.AddrPortFrom,' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPrefix,netip.Prefix,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPort,netip.AddrPort,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IP\b,netip.Addr,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPv6Raw\b,netip.AddrFrom16,g' $(git grep -l -F netaddr. ) goimports -w . Then delete some stuff from the net/netaddr shim package which is no longer neeed. Updates #5162 Change-Id: Ia7a86893fe21c7e3ee1ec823e8aba288d4566cd8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 04:14:09 +00:00
name: "ptr-netip.Addr",
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
val: &someIP,
util/deephash: specialize for netip.Addr and drop AppendTo support (#5402) There are 5 types that we care about that implement AppendTo: key.DiscoPublic key.NodePublic netip.Prefix netipx.IPRange netip.Addr The key types are thin wrappers around [32]byte and are memory hashable. The netip.Prefix and netipx.IPRange types are thin wrappers over netip.Addr and are hashable by default if netip.Addr is hashable. The netip.Addr type is the only one with a complex structure where the default behavior of deephash does not hash it correctly due to the presence of the intern.Value type. Drop support for AppendTo and instead add specialized hashing for netip.Addr that would be semantically equivalent to == on the netip.Addr values. The AppendTo support was already broken prior to this change. It was fully removed (intentionally or not) in #4870. It was partially restored in #4858 for the fast path, but still broken in the slow path. Just drop support for it altogether. This does mean we lack any ability for types to self-hash themselves. In the future we can add support for types that implement: interface { DeepHash() Sum } Test and fuzz cases were added for the relevant types that used to rely on the AppendTo method. FuzzAddr has been executed on 1 billion samples without issues. Signed-off-by: Joe Tsai joetsai@digital-static.net
2022-08-19 05:54:56 +00:00
out: u8(1) + u64(4) + u32(0x04030201),
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
},
{
all: convert more code to use net/netip directly perl -i -npe 's,netaddr.IPPrefixFrom,netip.PrefixFrom,' $(git grep -l -F netaddr.) perl -i -npe 's,netaddr.IPPortFrom,netip.AddrPortFrom,' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPrefix,netip.Prefix,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPort,netip.AddrPort,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IP\b,netip.Addr,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPv6Raw\b,netip.AddrFrom16,g' $(git grep -l -F netaddr. ) goimports -w . Then delete some stuff from the net/netaddr shim package which is no longer neeed. Updates #5162 Change-Id: Ia7a86893fe21c7e3ee1ec823e8aba288d4566cd8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 04:14:09 +00:00
name: "ptr-nil-netip.Addr",
val: (*netip.Addr)(nil),
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
out: "\x00",
},
{
name: "time",
util/deephash: use binary encoding of time.Time (#5352) Formatting a time.Time as RFC3339 is slow. See https://go.dev/issue/54093 Now that we have efficient hashing of fixed-width integers, just hash the time.Time as a binary value. Performance: Hash-24 19.0µs ± 1% 18.6µs ± 1% -2.03% (p=0.000 n=10+9) TailcfgNode-24 1.79µs ± 1% 1.40µs ± 1% -21.74% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 21:42:51 +00:00
val: time.Unix(1234, 5678).In(time.UTC),
out: u64(1234) + u32(5678) + u32(0),
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
},
util/deephash: fix unexported time.Time hashing Updates tailscale/corp#6311 Change-Id: I33cd7e4040966261c2f2eb3d32f29936aeb7f632 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-27 16:10:22 +00:00
{
name: "time_ptr", // addressable, as opposed to "time" test above
util/deephash: use binary encoding of time.Time (#5352) Formatting a time.Time as RFC3339 is slow. See https://go.dev/issue/54093 Now that we have efficient hashing of fixed-width integers, just hash the time.Time as a binary value. Performance: Hash-24 19.0µs ± 1% 18.6µs ± 1% -2.03% (p=0.000 n=10+9) TailcfgNode-24 1.79µs ± 1% 1.40µs ± 1% -21.74% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 21:42:51 +00:00
val: ptrTo(time.Unix(1234, 5678).In(time.UTC)),
out: u8(1) + u64(1234) + u32(5678) + u32(0),
util/deephash: fix unexported time.Time hashing Updates tailscale/corp#6311 Change-Id: I33cd7e4040966261c2f2eb3d32f29936aeb7f632 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-27 16:10:22 +00:00
},
{
name: "time_ptr_via_unexported",
util/deephash: use binary encoding of time.Time (#5352) Formatting a time.Time as RFC3339 is slow. See https://go.dev/issue/54093 Now that we have efficient hashing of fixed-width integers, just hash the time.Time as a binary value. Performance: Hash-24 19.0µs ± 1% 18.6µs ± 1% -2.03% (p=0.000 n=10+9) TailcfgNode-24 1.79µs ± 1% 1.40µs ± 1% -21.74% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 21:42:51 +00:00
val: testtype.NewUnexportedAddressableTime(time.Unix(1234, 5678).In(time.UTC)),
out: u8(1) + u64(1234) + u32(5678) + u32(0),
util/deephash: fix unexported time.Time hashing Updates tailscale/corp#6311 Change-Id: I33cd7e4040966261c2f2eb3d32f29936aeb7f632 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-27 16:10:22 +00:00
},
{
name: "time_ptr_via_unexported_value",
util/deephash: use binary encoding of time.Time (#5352) Formatting a time.Time as RFC3339 is slow. See https://go.dev/issue/54093 Now that we have efficient hashing of fixed-width integers, just hash the time.Time as a binary value. Performance: Hash-24 19.0µs ± 1% 18.6µs ± 1% -2.03% (p=0.000 n=10+9) TailcfgNode-24 1.79µs ± 1% 1.40µs ± 1% -21.74% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 21:42:51 +00:00
val: *testtype.NewUnexportedAddressableTime(time.Unix(1234, 5678).In(time.UTC)),
out: u64(1234) + u32(5678) + u32(0),
util/deephash: fix unexported time.Time hashing Updates tailscale/corp#6311 Change-Id: I33cd7e4040966261c2f2eb3d32f29936aeb7f632 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-27 16:10:22 +00:00
},
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
{
name: "time_custom_zone",
val: time.Unix(1655311822, 0).In(time.FixedZone("FOO", -60*60)),
util/deephash: use binary encoding of time.Time (#5352) Formatting a time.Time as RFC3339 is slow. See https://go.dev/issue/54093 Now that we have efficient hashing of fixed-width integers, just hash the time.Time as a binary value. Performance: Hash-24 19.0µs ± 1% 18.6µs ± 1% -2.03% (p=0.000 n=10+9) TailcfgNode-24 1.79µs ± 1% 1.40µs ± 1% -21.74% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 21:42:51 +00:00
out: u64(1655311822) + u32(0) + u32(math.MaxUint32-60*60+1),
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
},
{
name: "time_nil",
val: (*time.Time)(nil),
out: "\x00",
},
{
name: "array_memhash",
val: [4]byte{1, 2, 3, 4},
out: "\x01\x02\x03\x04",
},
{
name: "array_ptr_memhash",
val: ptrTo([4]byte{1, 2, 3, 4}),
out: "\x01\x01\x02\x03\x04",
},
{
name: "ptr_to_struct_partially_memhashable",
val: &struct {
A int16
B int16
C *int
}{5, 6, nil},
out: "\x01\x05\x00\x06\x00\x00",
},
{
name: "struct_partially_memhashable_but_cant_addr",
val: struct {
A int16
B int16
C *int
}{5, 6, nil},
out: "\x05\x00\x06\x00\x00",
},
{
name: "array_elements",
val: [4]byte{1, 2, 3, 4},
out: "\x01\x02\x03\x04",
},
{
name: "bool",
val: true,
out: "\x01",
},
{
name: "IntIntByteInt",
val: IntIntByteInt{1, 2, 3, 4},
out: "\x01\x00\x00\x00\x02\x00\x00\x00\x03\x04\x00\x00\x00",
},
{
name: "IntIntByteInt-canddr",
val: &IntIntByteInt{1, 2, 3, 4},
out: "\x01\x01\x00\x00\x00\x02\x00\x00\x00\x03\x04\x00\x00\x00",
},
{
name: "array-IntIntByteInt",
val: [2]IntIntByteInt{
{1, 2, 3, 4},
{5, 6, 7, 8},
},
out: "\x01\x00\x00\x00\x02\x00\x00\x00\x03\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\a\b\x00\x00\x00",
},
{
name: "array-IntIntByteInt-canaddr",
val: &[2]IntIntByteInt{
{1, 2, 3, 4},
{5, 6, 7, 8},
},
out: "\x01\x01\x00\x00\x00\x02\x00\x00\x00\x03\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\a\b\x00\x00\x00",
},
{
name: "tailcfg.Node",
val: &tailcfg.Node{},
control/controlclient,tailcfg: [capver 40] create KeySignature field in tailcfg.Node We calve out a space to put the node-key signature (used on tailnets where network lock is enabled). Signed-off-by: Tom DNetto <tom@tailscale.com>
2022-08-12 19:34:25 +00:00
out: "\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" + u64(uint64(time.Time{}.Unix())) + u64(0) + u32(0) + u32(0) + "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00" + u64(uint64(time.Time{}.Unix())) + u32(0) + u32(0) + "\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00",
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
rv := reflect.ValueOf(tt.val)
util/deephash: use unsafe.Pointer instead of reflect.Value (#5459) Use of reflect.Value.SetXXX panics if the provided argument was obtained from an unexported struct field. Instead, pass an unsafe.Pointer around and convert to a reflect.Value when necessary (i.e., for maps and interfaces). Converting from unsafe.Pointer to reflect.Value guarantees that none of the read-only bits will be populated. When running in race mode, we attach type information to the pointer so that we can type check every pointer operation. This also type-checks that direct memory hashing is within the valid range of a struct value. We add test cases that previously caused deephash to panic, but now pass. Performance: name old time/op new time/op delta Hash 14.1µs ± 1% 14.1µs ± 1% ~ (p=0.590 n=10+9) HashPacketFilter 2.53µs ± 2% 2.44µs ± 1% -3.79% (p=0.000 n=9+10) TailcfgNode 1.45µs ± 1% 1.43µs ± 0% -1.36% (p=0.000 n=9+9) HashArray 318ns ± 2% 318ns ± 2% ~ (p=0.541 n=10+10) HashMapAcyclic 32.9µs ± 1% 31.6µs ± 1% -4.16% (p=0.000 n=10+9) There is a slight performance gain due to the use of unsafe.Pointer over reflect.Value methods. Also, passing an unsafe.Pointer (1 word) on the stack is cheaper than passing a reflect.Value (3 words). Performance gains are diminishing since SHA-256 hashing now dominates the runtime. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 19:30:35 +00:00
va := reflect.New(rv.Type()).Elem()
util/deephash: always keep values addressable (#5328) The logic of deephash is both simpler and easier to reason about if values are always addressable. In Go, the composite kinds are slices, arrays, maps, structs, interfaces, pointers, channels, and functions, where we define "composite" as a Go value that encapsulates some other Go value (e.g., a map is a collection of key-value entries). In the cases of pointers and slices, the sub-values are always addressable. In the cases of arrays and structs, the sub-values are always addressable if and only if the parent value is addressable. In the case of maps and interfaces, the sub-values are never addressable. To make them addressable, we need to copy them onto the heap. For the purposes of deephash, we do not care about channels and functions. For all non-composite kinds (e.g., strings and ints), they are only addressable if obtained from one of the composite kinds that produce addressable values (i.e., pointers, slices, addressable arrays, and addressable structs). A non-addressible, non-composite kind can be made addressable by allocating it on the heap, obtaining a pointer to it, and dereferencing it. Thus, if we can ensure that values are addressable at the entry points, and shallow copy sub-values whenever we encounter an interface or map, then we can ensure that all values are always addressable and assume such property throughout all the logic. Performance: name old time/op new time/op delta Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9) HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9) HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10) TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10) HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
va.Set(rv)
fn := getTypeInfo(va.Type()).hasher()
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
hb := &hashBuffer{Hash: sha256.New()}
h := new(hasher)
util/sha256x: rename Hash as Block512 (#5351) Rename Hash as Block512 to indicate that this is a general-purpose hash.Hash for any algorithm that operates on 512-bit block sizes. While we rename the package as hashx in this commit, a subsequent commit will move the sha256x package to hashx. This is done separately to avoid confusing git. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-16 16:49:48 +00:00
h.Block512.Hash = hb
util/deephash: use unsafe.Pointer instead of reflect.Value (#5459) Use of reflect.Value.SetXXX panics if the provided argument was obtained from an unexported struct field. Instead, pass an unsafe.Pointer around and convert to a reflect.Value when necessary (i.e., for maps and interfaces). Converting from unsafe.Pointer to reflect.Value guarantees that none of the read-only bits will be populated. When running in race mode, we attach type information to the pointer so that we can type check every pointer operation. This also type-checks that direct memory hashing is within the valid range of a struct value. We add test cases that previously caused deephash to panic, but now pass. Performance: name old time/op new time/op delta Hash 14.1µs ± 1% 14.1µs ± 1% ~ (p=0.590 n=10+9) HashPacketFilter 2.53µs ± 2% 2.44µs ± 1% -3.79% (p=0.000 n=9+10) TailcfgNode 1.45µs ± 1% 1.43µs ± 0% -1.36% (p=0.000 n=9+9) HashArray 318ns ± 2% 318ns ± 2% ~ (p=0.541 n=10+10) HashMapAcyclic 32.9µs ± 1% 31.6µs ± 1% -4.16% (p=0.000 n=10+9) There is a slight performance gain due to the use of unsafe.Pointer over reflect.Value methods. Also, passing an unsafe.Pointer (1 word) on the stack is cheaper than passing a reflect.Value (3 words). Performance gains are diminishing since SHA-256 hashing now dominates the runtime. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 19:30:35 +00:00
fn(h, pointerOf(va.Addr()))
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
const ptrSize = 32 << uintptr(^uintptr(0)>>63)
if tt.out32 != "" && ptrSize == 32 {
tt.out = tt.out32
}
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
h.sum()
if got := string(hb.B); got != tt.out {
util/deephash: generate type-specific hasher funcs name old time/op new time/op delta Hash-8 71.1µs ± 2% 71.5µs ± 1% ~ (p=0.114 n=9+8) HashPacketFilter-8 8.39µs ± 1% 4.83µs ± 2% -42.38% (p=0.000 n=8+9) HashMapAcyclic-8 56.2µs ± 1% 56.9µs ± 2% +1.17% (p=0.035 n=10+9) TailcfgNode-8 6.49µs ± 2% 3.54µs ± 1% -45.37% (p=0.000 n=9+9) HashArray-8 729ns ± 2% 566ns ± 3% -22.30% (p=0.000 n=10+10) name old alloc/op new alloc/op delta Hash-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashPacketFilter-8 24.0B ± 0% 24.0B ± 0% ~ (all equal) HashMapAcyclic-8 0.00B 0.00B ~ (all equal) TailcfgNode-8 0.00B 0.00B ~ (all equal) HashArray-8 0.00B 0.00B ~ (all equal) name old allocs/op new allocs/op delta Hash-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashPacketFilter-8 1.00 ± 0% 1.00 ± 0% ~ (all equal) HashMapAcyclic-8 0.00 0.00 ~ (all equal) TailcfgNode-8 0.00 0.00 ~ (all equal) HashArray-8 0.00 0.00 ~ (all equal) Change-Id: I34c4e786e748fe60280646d40cc63a2adb2ea6fe Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-15 05:49:11 +00:00
t.Fatalf("got %q; want %q", got, tt.out)
}
})
}
}
util/deephash: use unsafe.Pointer instead of reflect.Value (#5459) Use of reflect.Value.SetXXX panics if the provided argument was obtained from an unexported struct field. Instead, pass an unsafe.Pointer around and convert to a reflect.Value when necessary (i.e., for maps and interfaces). Converting from unsafe.Pointer to reflect.Value guarantees that none of the read-only bits will be populated. When running in race mode, we attach type information to the pointer so that we can type check every pointer operation. This also type-checks that direct memory hashing is within the valid range of a struct value. We add test cases that previously caused deephash to panic, but now pass. Performance: name old time/op new time/op delta Hash 14.1µs ± 1% 14.1µs ± 1% ~ (p=0.590 n=10+9) HashPacketFilter 2.53µs ± 2% 2.44µs ± 1% -3.79% (p=0.000 n=9+10) TailcfgNode 1.45µs ± 1% 1.43µs ± 0% -1.36% (p=0.000 n=9+9) HashArray 318ns ± 2% 318ns ± 2% ~ (p=0.541 n=10+10) HashMapAcyclic 32.9µs ± 1% 31.6µs ± 1% -4.16% (p=0.000 n=10+9) There is a slight performance gain due to the use of unsafe.Pointer over reflect.Value methods. Also, passing an unsafe.Pointer (1 word) on the stack is cheaper than passing a reflect.Value (3 words). Performance gains are diminishing since SHA-256 hashing now dominates the runtime. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 19:30:35 +00:00
func TestMapCycle(t *testing.T) {
type M map[string]M
c := qt.New(t)
a := make(M) // cylic graph of 1 node
a["self"] = a
b := make(M) // cylic graph of 1 node
b["self"] = b
ha := Hash(a)
hb := Hash(b)
c.Assert(ha, qt.Equals, hb)
c1 := make(M) // cyclic graph of 2 nodes
c2 := make(M) // cyclic graph of 2 nodes
c1["peer"] = c2
c2["peer"] = c1
hc1 := Hash(c1)
hc2 := Hash(c2)
c.Assert(hc1, qt.Equals, hc2)
c.Assert(ha, qt.Not(qt.Equals), hc1)
c.Assert(hb, qt.Not(qt.Equals), hc2)
c3 := make(M) // graph of 1 node pointing to cyclic graph of 2 nodes
c3["child"] = c1
hc3 := Hash(c3)
c.Assert(hc1, qt.Not(qt.Equals), hc3)
}
func TestPointerCycle(t *testing.T) {
type P *P
c := qt.New(t)
a := new(P) // cyclic graph of 1 node
*a = a
b := new(P) // cyclic graph of 1 node
*b = b
ha := Hash(&a)
hb := Hash(&b)
c.Assert(ha, qt.Equals, hb)
c1 := new(P) // cyclic graph of 2 nodes
c2 := new(P) // cyclic graph of 2 nodes
*c1 = c2
*c2 = c1
hc1 := Hash(&c1)
hc2 := Hash(&c2)
c.Assert(hc1, qt.Equals, hc2)
c.Assert(ha, qt.Not(qt.Equals), hc1)
c.Assert(hb, qt.Not(qt.Equals), hc2)
c3 := new(P) // graph of 1 node pointing to cyclic graph of 2 nodes
*c3 = c1
hc3 := Hash(&c3)
c.Assert(hc1, qt.Not(qt.Equals), hc3)
}
func TestInterfaceCycle(t *testing.T) {
type I struct{ v any }
c := qt.New(t)
a := new(I) // cyclic graph of 1 node
a.v = a
b := new(I) // cyclic graph of 1 node
b.v = b
ha := Hash(&a)
hb := Hash(&b)
c.Assert(ha, qt.Equals, hb)
c1 := new(I) // cyclic graph of 2 nodes
c2 := new(I) // cyclic graph of 2 nodes
c1.v = c2
c2.v = c1
hc1 := Hash(&c1)
hc2 := Hash(&c2)
c.Assert(hc1, qt.Equals, hc2)
c.Assert(ha, qt.Not(qt.Equals), hc1)
c.Assert(hb, qt.Not(qt.Equals), hc2)
c3 := new(I) // graph of 1 node pointing to cyclic graph of 2 nodes
c3.v = c1
hc3 := Hash(&c3)
c.Assert(hc1, qt.Not(qt.Equals), hc3)
}
util/deephash: always keep values addressable (#5328) The logic of deephash is both simpler and easier to reason about if values are always addressable. In Go, the composite kinds are slices, arrays, maps, structs, interfaces, pointers, channels, and functions, where we define "composite" as a Go value that encapsulates some other Go value (e.g., a map is a collection of key-value entries). In the cases of pointers and slices, the sub-values are always addressable. In the cases of arrays and structs, the sub-values are always addressable if and only if the parent value is addressable. In the case of maps and interfaces, the sub-values are never addressable. To make them addressable, we need to copy them onto the heap. For the purposes of deephash, we do not care about channels and functions. For all non-composite kinds (e.g., strings and ints), they are only addressable if obtained from one of the composite kinds that produce addressable values (i.e., pointers, slices, addressable arrays, and addressable structs). A non-addressible, non-composite kind can be made addressable by allocating it on the heap, obtaining a pointer to it, and dereferencing it. Thus, if we can ensure that values are addressable at the entry points, and shallow copy sub-values whenever we encounter an interface or map, then we can ensure that all values are always addressable and assume such property throughout all the logic. Performance: name old time/op new time/op delta Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9) HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9) HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10) TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10) HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
var sink Sum
util/deephash: export a Hash func for use by the control plane name old time/op new time/op delta Hash-6 69.4µs ± 6% 68.4µs ± 4% ~ (p=0.286 n=9+9) HashMapAcyclic-6 115µs ± 5% 115µs ± 4% ~ (p=1.000 n=10+10) name old alloc/op new alloc/op delta Hash-6 2.29kB ± 0% 1.88kB ± 0% -18.13% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (all equal) name old allocs/op new allocs/op delta Hash-6 58.0 ± 0% 54.0 ± 0% -6.90% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-05 04:25:15 +00:00
internal/deepprint: add BenchmarkHash deepprint currently accounts for 15% of allocs in tailscaled. This is a useful benchmark to have. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-05-08 01:05:04 +00:00
func BenchmarkHash(b *testing.B) {
b.ReportAllocs()
v := getVal()
for i := 0; i < b.N; i++ {
util/deephash: export a Hash func for use by the control plane name old time/op new time/op delta Hash-6 69.4µs ± 6% 68.4µs ± 4% ~ (p=0.286 n=9+9) HashMapAcyclic-6 115µs ± 5% 115µs ± 4% ~ (p=1.000 n=10+10) name old alloc/op new alloc/op delta Hash-6 2.29kB ± 0% 1.88kB ± 0% -18.13% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (all equal) name old allocs/op new allocs/op delta Hash-6 58.0 ± 0% 54.0 ± 0% -6.90% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-05 04:25:15 +00:00
sink = Hash(v)
internal/deepprint: add BenchmarkHash deepprint currently accounts for 15% of allocs in tailscaled. This is a useful benchmark to have. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-05-08 01:05:04 +00:00
}
}
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
util/deephash: add packet filter benchmark (breaking up parts of another change) This adds a PacketFilter hashing benchmark with an input that both contains every possible field, but also is somewhat representative in the shape of what real packet filters contain. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-25 19:26:54 +00:00
func ptrTo[T any](v T) *T { return &v }
// filterRules is a packet filter that has both everything populated (in its
// first element) and also a few entries that are the typical shape for regular
// packet filters as sent to clients.
var filterRules = []tailcfg.FilterRule{
{
SrcIPs: []string{"*", "10.1.3.4/32", "10.0.0.0/24"},
SrcBits: []int{1, 2, 3},
DstPorts: []tailcfg.NetPortRange{{
IP: "1.2.3.4/32",
Bits: ptrTo(32),
Ports: tailcfg.PortRange{First: 1, Last: 2},
}},
IPProto: []int{1, 2, 3, 4},
CapGrant: []tailcfg.CapGrant{{
all: convert more code to use net/netip directly perl -i -npe 's,netaddr.IPPrefixFrom,netip.PrefixFrom,' $(git grep -l -F netaddr.) perl -i -npe 's,netaddr.IPPortFrom,netip.AddrPortFrom,' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPrefix,netip.Prefix,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPPort,netip.AddrPort,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IP\b,netip.Addr,g' $(git grep -l -F netaddr. ) perl -i -npe 's,netaddr.IPv6Raw\b,netip.AddrFrom16,g' $(git grep -l -F netaddr. ) goimports -w . Then delete some stuff from the net/netaddr shim package which is no longer neeed. Updates #5162 Change-Id: Ia7a86893fe21c7e3ee1ec823e8aba288d4566cd8 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-26 04:14:09 +00:00
Dsts: []netip.Prefix{netip.MustParsePrefix("1.2.3.4/32")},
util/deephash: add packet filter benchmark (breaking up parts of another change) This adds a PacketFilter hashing benchmark with an input that both contains every possible field, but also is somewhat representative in the shape of what real packet filters contain. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-25 19:26:54 +00:00
Caps: []string{"foo"},
}},
},
{
SrcIPs: []string{"foooooooooo"},
DstPorts: []tailcfg.NetPortRange{{
IP: "baaaaaarrrrr",
Ports: tailcfg.PortRange{First: 1, Last: 2},
}},
},
{
SrcIPs: []string{"foooooooooo"},
DstPorts: []tailcfg.NetPortRange{{
IP: "baaaaaarrrrr",
Ports: tailcfg.PortRange{First: 1, Last: 2},
}},
},
{
SrcIPs: []string{"foooooooooo"},
DstPorts: []tailcfg.NetPortRange{{
IP: "baaaaaarrrrr",
Ports: tailcfg.PortRange{First: 1, Last: 2},
}},
},
}
func BenchmarkHashPacketFilter(b *testing.B) {
b.ReportAllocs()
util/deephash: always keep values addressable (#5328) The logic of deephash is both simpler and easier to reason about if values are always addressable. In Go, the composite kinds are slices, arrays, maps, structs, interfaces, pointers, channels, and functions, where we define "composite" as a Go value that encapsulates some other Go value (e.g., a map is a collection of key-value entries). In the cases of pointers and slices, the sub-values are always addressable. In the cases of arrays and structs, the sub-values are always addressable if and only if the parent value is addressable. In the case of maps and interfaces, the sub-values are never addressable. To make them addressable, we need to copy them onto the heap. For the purposes of deephash, we do not care about channels and functions. For all non-composite kinds (e.g., strings and ints), they are only addressable if obtained from one of the composite kinds that produce addressable values (i.e., pointers, slices, addressable arrays, and addressable structs). A non-addressible, non-composite kind can be made addressable by allocating it on the heap, obtaining a pointer to it, and dereferencing it. Thus, if we can ensure that values are addressable at the entry points, and shallow copy sub-values whenever we encounter an interface or map, then we can ensure that all values are always addressable and assume such property throughout all the logic. Performance: name old time/op new time/op delta Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9) HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9) HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10) TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10) HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
hash := HasherForType[*[]tailcfg.FilterRule]()
util/deephash: add packet filter benchmark (breaking up parts of another change) This adds a PacketFilter hashing benchmark with an input that both contains every possible field, but also is somewhat representative in the shape of what real packet filters contain. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-25 19:26:54 +00:00
for i := 0; i < b.N; i++ {
util/deephash: always keep values addressable (#5328) The logic of deephash is both simpler and easier to reason about if values are always addressable. In Go, the composite kinds are slices, arrays, maps, structs, interfaces, pointers, channels, and functions, where we define "composite" as a Go value that encapsulates some other Go value (e.g., a map is a collection of key-value entries). In the cases of pointers and slices, the sub-values are always addressable. In the cases of arrays and structs, the sub-values are always addressable if and only if the parent value is addressable. In the case of maps and interfaces, the sub-values are never addressable. To make them addressable, we need to copy them onto the heap. For the purposes of deephash, we do not care about channels and functions. For all non-composite kinds (e.g., strings and ints), they are only addressable if obtained from one of the composite kinds that produce addressable values (i.e., pointers, slices, addressable arrays, and addressable structs). A non-addressible, non-composite kind can be made addressable by allocating it on the heap, obtaining a pointer to it, and dereferencing it. Thus, if we can ensure that values are addressable at the entry points, and shallow copy sub-values whenever we encounter an interface or map, then we can ensure that all values are always addressable and assume such property throughout all the logic. Performance: name old time/op new time/op delta Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9) HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9) HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10) TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10) HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
sink = hash(&filterRules)
util/deephash: add packet filter benchmark (breaking up parts of another change) This adds a PacketFilter hashing benchmark with an input that both contains every possible field, but also is somewhat representative in the shape of what real packet filters contain. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-06-25 19:26:54 +00:00
}
}
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
func TestHashMapAcyclic(t *testing.T) {
m := map[int]string{}
for i := 0; i < 100; i++ {
m[i] = fmt.Sprint(i)
}
got := map[string]bool{}
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
hb := &hashBuffer{Hash: sha256.New()}
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
util/deephash: move map logic to separate function (#5464) This helps pprof better identify which Go kinds take the most time since the kind is always in the function name. There is a minor adjustment where we hash the length of the map to be more on the cautious side. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 22:49:26 +00:00
hash := getTypeInfo(reflect.TypeOf(m)).hasher()
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
for i := 0; i < 20; i++ {
util/deephash: move map logic to separate function (#5464) This helps pprof better identify which Go kinds take the most time since the kind is always in the function name. There is a minor adjustment where we hash the length of the map to be more on the cautious side. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 22:49:26 +00:00
va := reflect.ValueOf(&m).Elem()
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
hb.Reset()
h := new(hasher)
util/sha256x: rename Hash as Block512 (#5351) Rename Hash as Block512 to indicate that this is a general-purpose hash.Hash for any algorithm that operates on 512-bit block sizes. While we rename the package as hashx in this commit, a subsequent commit will move the sha256x package to hashx. This is done separately to avoid confusing git. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-16 16:49:48 +00:00
h.Block512.Hash = hb
util/deephash: move map logic to separate function (#5464) This helps pprof better identify which Go kinds take the most time since the kind is always in the function name. There is a minor adjustment where we hash the length of the map to be more on the cautious side. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 22:49:26 +00:00
hash(h, pointerOf(va.Addr()))
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
h.sum()
if got[string(hb.B)] {
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
continue
}
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
got[string(hb.B)] = true
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
}
if len(got) != 1 {
t.Errorf("got %d results; want 1", len(got))
}
}
util/deephash: optimize hashing of byte arrays, reduce allocs in Hash name old time/op new time/op delta Hash-6 173µs ± 4% 101µs ± 3% -41.69% (p=0.000 n=10+9) HashMapAcyclic-6 101µs ± 5% 105µs ± 3% +3.52% (p=0.001 n=9+10) TailcfgNode-6 29.4µs ± 2% 16.4µs ± 3% -44.25% (p=0.000 n=8+10) name old alloc/op new alloc/op delta Hash-6 3.60kB ± 0% 1.13kB ± 0% -68.70% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (p=0.137 n=10+8) TailcfgNode-6 528B ± 0% 0B -100.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Hash-6 84.0 ± 0% 40.0 ± 0% -52.38% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) TailcfgNode-6 11.0 ± 0% 0.0 -100.00% (p=0.000 n=10+10) Updates tailscale/corp#2130 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 05:13:33 +00:00
func TestPrintArray(t *testing.T) {
type T struct {
X [32]byte
}
util/deephash: improve cycle detection (#2470) The previous algorithm used a map of all visited pointers. The strength of this approach is that it quickly prunes any nodes that we have ever visited before. The detriment of the approach is that pruning is heavily dependent on the order that pointers were visited. This is especially relevant for hashing a map where map entries are visited in a non-deterministic manner, which would cause the map hash to be non-deterministic (which defeats the point of a hash). This new algorithm uses a stack of all visited pointers, similar to how github.com/google/go-cmp performs cycle detection. When we visit a pointer, we push it onto the stack, and when we leave a pointer, we pop it from the stack. Before visiting a pointer, we first check whether the pointer exists anywhere in the stack. If yes, then we prune the node. The detriment of this approach is that we may hash a node more often than before since we do not prune as aggressively. The set of visited pointers up until any node is only the path of nodes up to that node and not any other pointers that may have been visited elsewhere. This provides us deterministic hashing regardless of visit order. We can now delete hashMapFallback and associated complexity, which only exists because the previous approach was non-deterministic in the presence of cycles. This fixes a failure of the old algorithm where obviously different values are treated as equal because the pruning was too aggresive. See https://github.com/tailscale/tailscale/issues/2443#issuecomment-883653534 The new algorithm is slightly slower since it prunes less aggresively: name old time/op new time/op delta Hash-8 66.1µs ± 1% 68.8µs ± 1% +4.09% (p=0.000 n=19+19) HashMapAcyclic-8 63.0µs ± 1% 62.5µs ± 1% -0.76% (p=0.000 n=18+19) TailcfgNode-8 9.79µs ± 2% 9.88µs ± 1% +0.95% (p=0.000 n=19+17) HashArray-8 643ns ± 1% 653ns ± 1% +1.64% (p=0.000 n=19+19) However, a slower but more correct algorithm seems more favorable than a faster but incorrect algorithm. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-07-22 22:22:48 +00:00
x := T{X: [32]byte{1: 1, 31: 31}}
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
hb := &hashBuffer{Hash: sha256.New()}
h := new(hasher)
util/sha256x: rename Hash as Block512 (#5351) Rename Hash as Block512 to indicate that this is a general-purpose hash.Hash for any algorithm that operates on 512-bit block sizes. While we rename the package as hashx in this commit, a subsequent commit will move the sha256x package to hashx. This is done separately to avoid confusing git. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-16 16:49:48 +00:00
h.Block512.Hash = hb
util/deephash: use unsafe.Pointer instead of reflect.Value (#5459) Use of reflect.Value.SetXXX panics if the provided argument was obtained from an unexported struct field. Instead, pass an unsafe.Pointer around and convert to a reflect.Value when necessary (i.e., for maps and interfaces). Converting from unsafe.Pointer to reflect.Value guarantees that none of the read-only bits will be populated. When running in race mode, we attach type information to the pointer so that we can type check every pointer operation. This also type-checks that direct memory hashing is within the valid range of a struct value. We add test cases that previously caused deephash to panic, but now pass. Performance: name old time/op new time/op delta Hash 14.1µs ± 1% 14.1µs ± 1% ~ (p=0.590 n=10+9) HashPacketFilter 2.53µs ± 2% 2.44µs ± 1% -3.79% (p=0.000 n=9+10) TailcfgNode 1.45µs ± 1% 1.43µs ± 0% -1.36% (p=0.000 n=9+9) HashArray 318ns ± 2% 318ns ± 2% ~ (p=0.541 n=10+10) HashMapAcyclic 32.9µs ± 1% 31.6µs ± 1% -4.16% (p=0.000 n=10+9) There is a slight performance gain due to the use of unsafe.Pointer over reflect.Value methods. Also, passing an unsafe.Pointer (1 word) on the stack is cheaper than passing a reflect.Value (3 words). Performance gains are diminishing since SHA-256 hashing now dominates the runtime. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 19:30:35 +00:00
va := reflect.ValueOf(&x).Elem()
ti := getTypeInfo(va.Type())
ti.hasher()(h, pointerOf(va.Addr()))
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
h.sum()
util/deephash: remove unnecessary formatting for structs and slices (#2571) The index for every struct field or slice element and the number of fields for the struct is unncessary. The hashing of Go values is unambiguous because every type (except maps) encodes in a parsable manner. So long as we know the type information, we could theoretically decode every value (except for maps). At a high level: * numbers are encoded as fixed-width records according to precision. * strings (and AppendTo output) are encoded with a fixed-width length, followed by the contents of the buffer. * slices are prefixed by a fixed-width length, followed by the encoding of each value. So long as we know the type of each element, we could theoretically decode each element. * arrays are encoded just like slices, but elide the length since it is determined from the Go type. * maps are encoded first with a byte indicating whether it is a cycle. If a cycle, it is followed by a fixed-width index for the pointer, otherwise followed by the SHA-256 hash of its contents. The encoding of maps is not decodeable, but a SHA-256 hash is sufficient to avoid ambiguities. * interfaces are encoded first with a byte indicating whether it is nil. If not nil, it is followed by a fixed-width index for the type, and then the encoding for the underlying value. Having the type be encoded first ensures that the value could theoretically be decoded next. * pointers are encoded first with a byte indicating whether it is 1) nil, 2) a cycle, or 3) newly seen. If a cycle, it is followed by a fixed-width index for the pointer. If newly seen, it is followed by the encoding for the pointed-at value. Removing unnecessary details speeds up hashing: name old time/op new time/op delta Hash-8 76.0µs ± 1% 55.8µs ± 2% -26.62% (p=0.000 n=10+10) HashMapAcyclic-8 61.9µs ± 0% 62.0µs ± 0% ~ (p=0.666 n=9+9) TailcfgNode-8 10.2µs ± 1% 7.5µs ± 1% -26.90% (p=0.000 n=10+9) HashArray-8 1.07µs ± 1% 0.70µs ± 1% -34.67% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-04 03:35:57 +00:00
const want = "\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x1f"
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
if got := hb.B; string(got) != want {
util/deephash: optimize hashing of byte arrays, reduce allocs in Hash name old time/op new time/op delta Hash-6 173µs ± 4% 101µs ± 3% -41.69% (p=0.000 n=10+9) HashMapAcyclic-6 101µs ± 5% 105µs ± 3% +3.52% (p=0.001 n=9+10) TailcfgNode-6 29.4µs ± 2% 16.4µs ± 3% -44.25% (p=0.000 n=8+10) name old alloc/op new alloc/op delta Hash-6 3.60kB ± 0% 1.13kB ± 0% -68.70% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (p=0.137 n=10+8) TailcfgNode-6 528B ± 0% 0B -100.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Hash-6 84.0 ± 0% 40.0 ± 0% -52.38% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) TailcfgNode-6 11.0 ± 0% 0.0 -100.00% (p=0.000 n=10+10) Updates tailscale/corp#2130 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 05:13:33 +00:00
t.Errorf("wrong:\n got: %q\nwant: %q\n", got, want)
}
}
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
func BenchmarkHashMapAcyclic(b *testing.B) {
b.ReportAllocs()
m := map[int]string{}
for i := 0; i < 100; i++ {
m[i] = fmt.Sprint(i)
}
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
hb := &hashBuffer{Hash: sha256.New()}
util/deephash: use unsafe.Pointer instead of reflect.Value (#5459) Use of reflect.Value.SetXXX panics if the provided argument was obtained from an unexported struct field. Instead, pass an unsafe.Pointer around and convert to a reflect.Value when necessary (i.e., for maps and interfaces). Converting from unsafe.Pointer to reflect.Value guarantees that none of the read-only bits will be populated. When running in race mode, we attach type information to the pointer so that we can type check every pointer operation. This also type-checks that direct memory hashing is within the valid range of a struct value. We add test cases that previously caused deephash to panic, but now pass. Performance: name old time/op new time/op delta Hash 14.1µs ± 1% 14.1µs ± 1% ~ (p=0.590 n=10+9) HashPacketFilter 2.53µs ± 2% 2.44µs ± 1% -3.79% (p=0.000 n=9+10) TailcfgNode 1.45µs ± 1% 1.43µs ± 0% -1.36% (p=0.000 n=9+9) HashArray 318ns ± 2% 318ns ± 2% ~ (p=0.541 n=10+10) HashMapAcyclic 32.9µs ± 1% 31.6µs ± 1% -4.16% (p=0.000 n=10+9) There is a slight performance gain due to the use of unsafe.Pointer over reflect.Value methods. Also, passing an unsafe.Pointer (1 word) on the stack is cheaper than passing a reflect.Value (3 words). Performance gains are diminishing since SHA-256 hashing now dominates the runtime. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 19:30:35 +00:00
va := reflect.ValueOf(&m).Elem()
util/deephash: move map logic to separate function (#5464) This helps pprof better identify which Go kinds take the most time since the kind is always in the function name. There is a minor adjustment where we hash the length of the map to be more on the cautious side. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 22:49:26 +00:00
hash := getTypeInfo(va.Type()).hasher()
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
h := new(hasher)
util/sha256x: rename Hash as Block512 (#5351) Rename Hash as Block512 to indicate that this is a general-purpose hash.Hash for any algorithm that operates on 512-bit block sizes. While we rename the package as hashx in this commit, a subsequent commit will move the sha256x package to hashx. This is done separately to avoid confusing git. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-16 16:49:48 +00:00
h.Block512.Hash = hb
util/deephash: move funcs to methods Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-07 05:37:32 +00:00
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
for i := 0; i < b.N; i++ {
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
h.Reset()
util/deephash: move map logic to separate function (#5464) This helps pprof better identify which Go kinds take the most time since the kind is always in the function name. There is a minor adjustment where we hash the length of the map to be more on the cautious side. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-27 22:49:26 +00:00
hash(h, pointerOf(va.Addr()))
internal/deephash: hash maps without sorting in the acyclic common case Hash and xor each entry instead, then write final xor'ed result. name old time/op new time/op delta Hash-4 33.6µs ± 4% 34.6µs ± 3% +3.03% (p=0.013 n=10+9) name old alloc/op new alloc/op delta Hash-4 1.86kB ± 0% 1.77kB ± 0% -5.10% (p=0.000 n=10+9) name old allocs/op new allocs/op delta Hash-4 51.0 ± 0% 49.0 ± 0% -3.92% (p=0.000 n=10+10) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-05-11 20:17:12 +00:00
}
}
internal/deephash: 8 bits of output is not enough Running hex.Encode(b, b) is a bad idea. The first byte of input will overwrite the first two bytes of output. Subsequent bytes have no impact on the output. Not related to today's IPv6 bug, but...wh::ps. This caused us to spuriously ignore some wireguard config updates. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-07-02 20:45:23 +00:00
util/deephash: add a benchmark and some benchmark data No code changes. Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 04:21:52 +00:00
func BenchmarkTailcfgNode(b *testing.B) {
b.ReportAllocs()
node := new(tailcfg.Node)
for i := 0; i < b.N; i++ {
sink = Hash(node)
}
}
internal/deephash: 8 bits of output is not enough Running hex.Encode(b, b) is a bad idea. The first byte of input will overwrite the first two bytes of output. Subsequent bytes have no impact on the output. Not related to today's IPv6 bug, but...wh::ps. This caused us to spuriously ignore some wireguard config updates. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-07-02 20:45:23 +00:00
func TestExhaustive(t *testing.T) {
util/deephash: make hash type opaque The fact that Hash returns a [sha256.Size]byte leaks details about the underlying hash implementation. This could very well be any other hashing algorithm with a possible different block size. Abstract this implementation detail away by declaring an opaque type that is comparable. While we are changing the signature of UpdateHash, rename it to just Update to reduce stutter (e.g., deephash.Update). Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-07-20 05:49:51 +00:00
seen := make(map[Sum]bool)
internal/deephash: 8 bits of output is not enough Running hex.Encode(b, b) is a bad idea. The first byte of input will overwrite the first two bytes of output. Subsequent bytes have no impact on the output. Not related to today's IPv6 bug, but...wh::ps. This caused us to spuriously ignore some wireguard config updates. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-07-02 20:45:23 +00:00
for i := 0; i < 100000; i++ {
util/deephash: export a Hash func for use by the control plane name old time/op new time/op delta Hash-6 69.4µs ± 6% 68.4µs ± 4% ~ (p=0.286 n=9+9) HashMapAcyclic-6 115µs ± 5% 115µs ± 4% ~ (p=1.000 n=10+10) name old alloc/op new alloc/op delta Hash-6 2.29kB ± 0% 1.88kB ± 0% -18.13% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (all equal) name old allocs/op new allocs/op delta Hash-6 58.0 ± 0% 54.0 ± 0% -6.90% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-05 04:25:15 +00:00
s := Hash(i)
internal/deephash: 8 bits of output is not enough Running hex.Encode(b, b) is a bad idea. The first byte of input will overwrite the first two bytes of output. Subsequent bytes have no impact on the output. Not related to today's IPv6 bug, but...wh::ps. This caused us to spuriously ignore some wireguard config updates. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2021-07-02 20:45:23 +00:00
if seen[s] {
t.Fatalf("hash collision %v", i)
}
seen[s] = true
}
}
util/deephash: export a Hash func for use by the control plane name old time/op new time/op delta Hash-6 69.4µs ± 6% 68.4µs ± 4% ~ (p=0.286 n=9+9) HashMapAcyclic-6 115µs ± 5% 115µs ± 4% ~ (p=1.000 n=10+10) name old alloc/op new alloc/op delta Hash-6 2.29kB ± 0% 1.88kB ± 0% -18.13% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (all equal) name old allocs/op new allocs/op delta Hash-6 58.0 ± 0% 54.0 ± 0% -6.90% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-05 04:25:15 +00:00
util/deephash: prevent infinite loop on map cycle Fixes #2340 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-07 04:41:18 +00:00
// verify this doesn't loop forever, as it used to (Issue 2340)
func TestMapCyclicFallback(t *testing.T) {
type T struct {
all: use any instead of interface{} My favorite part of generics. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2022-03-16 23:27:57 +00:00
M map[string]any
util/deephash: prevent infinite loop on map cycle Fixes #2340 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-07 04:41:18 +00:00
}
v := &T{
all: use any instead of interface{} My favorite part of generics. Signed-off-by: Josh Bleecher Snyder <josh@tailscale.com>
2022-03-16 23:27:57 +00:00
M: map[string]any{},
util/deephash: prevent infinite loop on map cycle Fixes #2340 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-07 04:41:18 +00:00
}
v.M["m"] = v.M
Hash(v)
}
util/deephash: optimize hashing of byte arrays, reduce allocs in Hash name old time/op new time/op delta Hash-6 173µs ± 4% 101µs ± 3% -41.69% (p=0.000 n=10+9) HashMapAcyclic-6 101µs ± 5% 105µs ± 3% +3.52% (p=0.001 n=9+10) TailcfgNode-6 29.4µs ± 2% 16.4µs ± 3% -44.25% (p=0.000 n=8+10) name old alloc/op new alloc/op delta Hash-6 3.60kB ± 0% 1.13kB ± 0% -68.70% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (p=0.137 n=10+8) TailcfgNode-6 528B ± 0% 0B -100.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Hash-6 84.0 ± 0% 40.0 ± 0% -52.38% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) TailcfgNode-6 11.0 ± 0% 0.0 -100.00% (p=0.000 n=10+10) Updates tailscale/corp#2130 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 05:13:33 +00:00
func TestArrayAllocs(t *testing.T) {
util/deephash: skip alloc test under race detector Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-07 18:40:28 +00:00
if version.IsRace() {
t.Skip("skipping test under race detector")
}
util/deephash: fix TestArrayAllocs Unfortunately this test fails on certain architectures. The problem comes down to inconsistencies in the Go escape analysis where specific variables are marked as escaping on certain architectures. The variables escaping to the heap are unfortunately in crypto/sha256, which makes it impossible to fixthis locally in deephash. For now, fix the test by compensating for the allocations that occur from calling sha256.digest.Sum. See golang/go#48055 Fixes #2727 Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-30 17:47:21 +00:00
// In theory, there should be no allocations. However, escape analysis on
// certain architectures fails to detect that certain cases do not escape.
// This discrepency currently affects sha256.digest.Sum.
// Measure the number of allocations in sha256 to ensure that Hash does
// not allocate on top of its usage of sha256.
// See https://golang.org/issue/48055.
var b []byte
h := sha256.New()
want := int(testing.AllocsPerRun(1000, func() {
b = h.Sum(b[:0])
}))
switch runtime.GOARCH {
case "amd64", "arm64":
want = 0 // ensure no allocations on popular architectures
}
util/deephash: optimize hashing of byte arrays, reduce allocs in Hash name old time/op new time/op delta Hash-6 173µs ± 4% 101µs ± 3% -41.69% (p=0.000 n=10+9) HashMapAcyclic-6 101µs ± 5% 105µs ± 3% +3.52% (p=0.001 n=9+10) TailcfgNode-6 29.4µs ± 2% 16.4µs ± 3% -44.25% (p=0.000 n=8+10) name old alloc/op new alloc/op delta Hash-6 3.60kB ± 0% 1.13kB ± 0% -68.70% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (p=0.137 n=10+8) TailcfgNode-6 528B ± 0% 0B -100.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Hash-6 84.0 ± 0% 40.0 ± 0% -52.38% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) TailcfgNode-6 11.0 ± 0% 0.0 -100.00% (p=0.000 n=10+10) Updates tailscale/corp#2130 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 05:13:33 +00:00
type T struct {
X [32]byte
}
x := &T{X: [32]byte{1: 1, 2: 2, 3: 3, 4: 4}}
util/deephash: fix TestArrayAllocs Unfortunately this test fails on certain architectures. The problem comes down to inconsistencies in the Go escape analysis where specific variables are marked as escaping on certain architectures. The variables escaping to the heap are unfortunately in crypto/sha256, which makes it impossible to fixthis locally in deephash. For now, fix the test by compensating for the allocations that occur from calling sha256.digest.Sum. See golang/go#48055 Fixes #2727 Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-30 17:47:21 +00:00
got := int(testing.AllocsPerRun(1000, func() {
util/deephash: optimize hashing of byte arrays, reduce allocs in Hash name old time/op new time/op delta Hash-6 173µs ± 4% 101µs ± 3% -41.69% (p=0.000 n=10+9) HashMapAcyclic-6 101µs ± 5% 105µs ± 3% +3.52% (p=0.001 n=9+10) TailcfgNode-6 29.4µs ± 2% 16.4µs ± 3% -44.25% (p=0.000 n=8+10) name old alloc/op new alloc/op delta Hash-6 3.60kB ± 0% 1.13kB ± 0% -68.70% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (p=0.137 n=10+8) TailcfgNode-6 528B ± 0% 0B -100.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Hash-6 84.0 ± 0% 40.0 ± 0% -52.38% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) TailcfgNode-6 11.0 ± 0% 0.0 -100.00% (p=0.000 n=10+10) Updates tailscale/corp#2130 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 05:13:33 +00:00
sink = Hash(x)
}))
util/deephash: fix TestArrayAllocs Unfortunately this test fails on certain architectures. The problem comes down to inconsistencies in the Go escape analysis where specific variables are marked as escaping on certain architectures. The variables escaping to the heap are unfortunately in crypto/sha256, which makes it impossible to fixthis locally in deephash. For now, fix the test by compensating for the allocations that occur from calling sha256.digest.Sum. See golang/go#48055 Fixes #2727 Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-30 17:47:21 +00:00
if got > want {
t.Errorf("allocs = %v; want %v", got, want)
util/deephash: optimize hashing of byte arrays, reduce allocs in Hash name old time/op new time/op delta Hash-6 173µs ± 4% 101µs ± 3% -41.69% (p=0.000 n=10+9) HashMapAcyclic-6 101µs ± 5% 105µs ± 3% +3.52% (p=0.001 n=9+10) TailcfgNode-6 29.4µs ± 2% 16.4µs ± 3% -44.25% (p=0.000 n=8+10) name old alloc/op new alloc/op delta Hash-6 3.60kB ± 0% 1.13kB ± 0% -68.70% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (p=0.137 n=10+8) TailcfgNode-6 528B ± 0% 0B -100.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Hash-6 84.0 ± 0% 40.0 ± 0% -52.38% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) TailcfgNode-6 11.0 ± 0% 0.0 -100.00% (p=0.000 n=10+10) Updates tailscale/corp#2130 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 05:13:33 +00:00
}
}
util/deephash: fix unexported time.Time hashing Updates tailscale/corp#6311 Change-Id: I33cd7e4040966261c2f2eb3d32f29936aeb7f632 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2022-07-27 16:10:22 +00:00
// Test for http://go/corp/6311 issue.
func TestHashThroughView(t *testing.T) {
type sshPolicyOut struct {
Rules []tailcfg.SSHRuleView
}
type mapResponseOut struct {
SSHPolicy *sshPolicyOut
}
// Just test we don't panic:
_ = Hash(&mapResponseOut{
SSHPolicy: &sshPolicyOut{
Rules: []tailcfg.SSHRuleView{
(&tailcfg.SSHRule{
RuleExpires: ptrTo(time.Unix(123, 0)),
}).View(),
},
},
})
}
util/deephash: optimize hashing of byte arrays, reduce allocs in Hash name old time/op new time/op delta Hash-6 173µs ± 4% 101µs ± 3% -41.69% (p=0.000 n=10+9) HashMapAcyclic-6 101µs ± 5% 105µs ± 3% +3.52% (p=0.001 n=9+10) TailcfgNode-6 29.4µs ± 2% 16.4µs ± 3% -44.25% (p=0.000 n=8+10) name old alloc/op new alloc/op delta Hash-6 3.60kB ± 0% 1.13kB ± 0% -68.70% (p=0.000 n=10+10) HashMapAcyclic-6 2.53kB ± 0% 2.53kB ± 0% ~ (p=0.137 n=10+8) TailcfgNode-6 528B ± 0% 0B -100.00% (p=0.000 n=10+10) name old allocs/op new allocs/op delta Hash-6 84.0 ± 0% 40.0 ± 0% -52.38% (p=0.000 n=10+10) HashMapAcyclic-6 202 ± 0% 202 ± 0% ~ (all equal) TailcfgNode-6 11.0 ± 0% 0.0 -100.00% (p=0.000 n=10+10) Updates tailscale/corp#2130 Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
2021-07-06 05:13:33 +00:00
func BenchmarkHashArray(b *testing.B) {
b.ReportAllocs()
type T struct {
X [32]byte
}
x := &T{X: [32]byte{1: 1, 2: 2, 3: 3, 4: 4}}
for i := 0; i < b.N; i++ {
sink = Hash(x)
}
}
util/deephash: use sha256x (#5339) Switch deephash to use sha256x.Hash. We add sha256x.HashString to efficiently hash a string. It uses unsafe under the hood to convert a string to a []byte. We also modify sha256x.Hash to export the underlying hash.Hash for testing purposes so that we can intercept all hash.Hash calls. Performance: name old time/op new time/op delta Hash-24 19.8µs ± 1% 19.2µs ± 1% -3.01% (p=0.000 n=10+10) HashPacketFilter-24 2.61µs ± 0% 2.53µs ± 1% -3.01% (p=0.000 n=8+10) HashMapAcyclic-24 31.3µs ± 1% 29.8µs ± 0% -4.80% (p=0.000 n=10+9) TailcfgNode-24 1.83µs ± 1% 1.82µs ± 2% ~ (p=0.305 n=10+10) HashArray-24 344ns ± 2% 323ns ± 1% -6.02% (p=0.000 n=9+10) The performance gains is not as dramatic as sha256x over sha256 due to: 1. most of the hashing already occurring through the direct memory hashing logic, and 2. what does not go through direct memory hashing is slowed down by reflect. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 00:44:09 +00:00
// hashBuffer is a hash.Hash that buffers all written data.
type hashBuffer struct {
hash.Hash
B []byte
}
func (h *hashBuffer) Write(b []byte) (int, error) {
n, err := h.Hash.Write(b)
h.B = append(h.B, b[:n]...)
return n, err
}
func (h *hashBuffer) Reset() {
h.Hash.Reset()
h.B = h.B[:0]
}
util/deephash: use binary encoding of time.Time (#5352) Formatting a time.Time as RFC3339 is slow. See https://go.dev/issue/54093 Now that we have efficient hashing of fixed-width integers, just hash the time.Time as a binary value. Performance: Hash-24 19.0µs ± 1% 18.6µs ± 1% -2.03% (p=0.000 n=10+9) TailcfgNode-24 1.79µs ± 1% 1.40µs ± 1% -21.74% (p=0.000 n=10+9) Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-12 21:42:51 +00:00
func FuzzTime(f *testing.F) {
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(0), false, "", 0)
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(0), true, "", 0)
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(0), true, "hello", 0)
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(0), true, "", 1234)
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(0), true, "hello", 1234)
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(1), false, "", 0)
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(1), true, "", 0)
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(1), true, "hello", 0)
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(1), true, "", 1234)
f.Add(int64(0), int64(0), false, "", 0, int64(0), int64(1), true, "hello", 1234)
f.Add(int64(math.MaxInt64), int64(math.MaxInt64), false, "", 0, int64(math.MaxInt64), int64(math.MaxInt64), false, "", 0)
f.Add(int64(math.MaxInt64), int64(math.MaxInt64), false, "", 0, int64(math.MaxInt64), int64(math.MaxInt64), true, "", 0)
f.Add(int64(math.MaxInt64), int64(math.MaxInt64), false, "", 0, int64(math.MaxInt64), int64(math.MaxInt64), true, "hello", 0)
f.Add(int64(math.MaxInt64), int64(math.MaxInt64), false, "", 0, int64(math.MaxInt64), int64(math.MaxInt64), true, "", 1234)
f.Add(int64(math.MaxInt64), int64(math.MaxInt64), false, "", 0, int64(math.MaxInt64), int64(math.MaxInt64), true, "hello", 1234)
f.Add(int64(math.MinInt64), int64(math.MinInt64), false, "", 0, int64(math.MinInt64), int64(math.MinInt64), false, "", 0)
f.Add(int64(math.MinInt64), int64(math.MinInt64), false, "", 0, int64(math.MinInt64), int64(math.MinInt64), true, "", 0)
f.Add(int64(math.MinInt64), int64(math.MinInt64), false, "", 0, int64(math.MinInt64), int64(math.MinInt64), true, "hello", 0)
f.Add(int64(math.MinInt64), int64(math.MinInt64), false, "", 0, int64(math.MinInt64), int64(math.MinInt64), true, "", 1234)
f.Add(int64(math.MinInt64), int64(math.MinInt64), false, "", 0, int64(math.MinInt64), int64(math.MinInt64), true, "hello", 1234)
f.Fuzz(func(t *testing.T,
s1, ns1 int64, loc1 bool, name1 string, off1 int,
s2, ns2 int64, loc2 bool, name2 string, off2 int,
) {
t1 := time.Unix(s1, ns1)
if loc1 {
t1.In(time.FixedZone(name1, off1))
}
t2 := time.Unix(s2, ns2)
if loc2 {
t2.In(time.FixedZone(name2, off2))
}
got := Hash(&t1) == Hash(&t2)
want := t1.Format(time.RFC3339Nano) == t2.Format(time.RFC3339Nano)
if got != want {
t.Errorf("time.Time(%s) == time.Time(%s) mismatches hash equivalent", t1.Format(time.RFC3339Nano), t2.Format(time.RFC3339Nano))
}
})
}
util/deephash: specialize for netip.Addr and drop AppendTo support (#5402) There are 5 types that we care about that implement AppendTo: key.DiscoPublic key.NodePublic netip.Prefix netipx.IPRange netip.Addr The key types are thin wrappers around [32]byte and are memory hashable. The netip.Prefix and netipx.IPRange types are thin wrappers over netip.Addr and are hashable by default if netip.Addr is hashable. The netip.Addr type is the only one with a complex structure where the default behavior of deephash does not hash it correctly due to the presence of the intern.Value type. Drop support for AppendTo and instead add specialized hashing for netip.Addr that would be semantically equivalent to == on the netip.Addr values. The AppendTo support was already broken prior to this change. It was fully removed (intentionally or not) in #4870. It was partially restored in #4858 for the fast path, but still broken in the slow path. Just drop support for it altogether. This does mean we lack any ability for types to self-hash themselves. In the future we can add support for types that implement: interface { DeepHash() Sum } Test and fuzz cases were added for the relevant types that used to rely on the AppendTo method. FuzzAddr has been executed on 1 billion samples without issues. Signed-off-by: Joe Tsai joetsai@digital-static.net
2022-08-19 05:54:56 +00:00
func FuzzAddr(f *testing.F) {
f.Fuzz(func(t *testing.T,
u1a, u1b uint64, zone1 string,
u2a, u2b uint64, zone2 string,
) {
var b1, b2 [16]byte
binary.LittleEndian.PutUint64(b1[:8], u1a)
binary.LittleEndian.PutUint64(b1[8:], u1b)
binary.LittleEndian.PutUint64(b2[:8], u2a)
binary.LittleEndian.PutUint64(b2[8:], u2b)
var ips [4]netip.Addr
ips[0] = netip.AddrFrom4(*(*[4]byte)(b1[:]))
ips[1] = netip.AddrFrom4(*(*[4]byte)(b2[:]))
ips[2] = netip.AddrFrom16(b1)
if zone1 != "" {
ips[2] = ips[2].WithZone(zone1)
}
ips[3] = netip.AddrFrom16(b2)
if zone2 != "" {
ips[3] = ips[2].WithZone(zone2)
}
for _, ip1 := range ips[:] {
for _, ip2 := range ips[:] {
got := Hash(&ip1) == Hash(&ip2)
want := ip1 == ip2
if got != want {
t.Errorf("netip.Addr(%s) == netip.Addr(%s) mismatches hash equivalent", ip1.String(), ip2.String())
}
}
}
})
}
Reference in New Issue Copy Permalink