2020-06-30 02:36:45 +00:00
|
|
|
// Copyright (c) 2020 Tailscale Inc & AUTHORS All rights reserved.
|
|
|
|
// Use of this source code is governed by a BSD-style
|
|
|
|
// license that can be found in the LICENSE file.
|
|
|
|
|
2021-07-21 16:23:04 +00:00
|
|
|
// Package deephash hashes a Go value recursively, in a predictable order,
|
|
|
|
// without looping. The hash is only valid within the lifetime of a program.
|
|
|
|
// Users should not store the hash on disk or send it over the network.
|
|
|
|
// The hash is sufficiently strong and unique such that
|
|
|
|
// Hash(x) == Hash(y) is an appropriate replacement for x == y.
|
2021-07-03 04:30:29 +00:00
|
|
|
//
|
util/deephash: remove unnecessary formatting for structs and slices (#2571)
The index for every struct field or slice element and
the number of fields for the struct is unncessary.
The hashing of Go values is unambiguous because every type (except maps)
encodes in a parsable manner. So long as we know the type information,
we could theoretically decode every value (except for maps).
At a high level:
* numbers are encoded as fixed-width records according to precision.
* strings (and AppendTo output) are encoded with a fixed-width length,
followed by the contents of the buffer.
* slices are prefixed by a fixed-width length, followed by the encoding
of each value. So long as we know the type of each element, we could
theoretically decode each element.
* arrays are encoded just like slices, but elide the length
since it is determined from the Go type.
* maps are encoded first with a byte indicating whether it is a cycle.
If a cycle, it is followed by a fixed-width index for the pointer,
otherwise followed by the SHA-256 hash of its contents. The encoding of maps
is not decodeable, but a SHA-256 hash is sufficient to avoid ambiguities.
* interfaces are encoded first with a byte indicating whether it is nil.
If not nil, it is followed by a fixed-width index for the type,
and then the encoding for the underlying value. Having the type be encoded
first ensures that the value could theoretically be decoded next.
* pointers are encoded first with a byte indicating whether it is
1) nil, 2) a cycle, or 3) newly seen. If a cycle, it is followed by
a fixed-width index for the pointer. If newly seen, it is followed by
the encoding for the pointed-at value.
Removing unnecessary details speeds up hashing:
name old time/op new time/op delta
Hash-8 76.0µs ± 1% 55.8µs ± 2% -26.62% (p=0.000 n=10+10)
HashMapAcyclic-8 61.9µs ± 0% 62.0µs ± 0% ~ (p=0.666 n=9+9)
TailcfgNode-8 10.2µs ± 1% 7.5µs ± 1% -26.90% (p=0.000 n=10+9)
HashArray-8 1.07µs ± 1% 0.70µs ± 1% -34.67% (p=0.000 n=10+9)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-04 03:35:57 +00:00
|
|
|
// The definition of equality is identical to reflect.DeepEqual except:
|
2022-08-02 16:33:46 +00:00
|
|
|
// - Floating-point values are compared based on the raw bits,
|
|
|
|
// which means that NaNs (with the same bit pattern) are treated as equal.
|
|
|
|
// - Types which implement interface { AppendTo([]byte) []byte } use
|
|
|
|
// the AppendTo method to produce a textual representation of the value.
|
|
|
|
// Thus, two values are equal if AppendTo produces the same bytes.
|
util/deephash: remove unnecessary formatting for structs and slices (#2571)
The index for every struct field or slice element and
the number of fields for the struct is unncessary.
The hashing of Go values is unambiguous because every type (except maps)
encodes in a parsable manner. So long as we know the type information,
we could theoretically decode every value (except for maps).
At a high level:
* numbers are encoded as fixed-width records according to precision.
* strings (and AppendTo output) are encoded with a fixed-width length,
followed by the contents of the buffer.
* slices are prefixed by a fixed-width length, followed by the encoding
of each value. So long as we know the type of each element, we could
theoretically decode each element.
* arrays are encoded just like slices, but elide the length
since it is determined from the Go type.
* maps are encoded first with a byte indicating whether it is a cycle.
If a cycle, it is followed by a fixed-width index for the pointer,
otherwise followed by the SHA-256 hash of its contents. The encoding of maps
is not decodeable, but a SHA-256 hash is sufficient to avoid ambiguities.
* interfaces are encoded first with a byte indicating whether it is nil.
If not nil, it is followed by a fixed-width index for the type,
and then the encoding for the underlying value. Having the type be encoded
first ensures that the value could theoretically be decoded next.
* pointers are encoded first with a byte indicating whether it is
1) nil, 2) a cycle, or 3) newly seen. If a cycle, it is followed by
a fixed-width index for the pointer. If newly seen, it is followed by
the encoding for the pointed-at value.
Removing unnecessary details speeds up hashing:
name old time/op new time/op delta
Hash-8 76.0µs ± 1% 55.8µs ± 2% -26.62% (p=0.000 n=10+10)
HashMapAcyclic-8 61.9µs ± 0% 62.0µs ± 0% ~ (p=0.666 n=9+9)
TailcfgNode-8 10.2µs ± 1% 7.5µs ± 1% -26.90% (p=0.000 n=10+9)
HashArray-8 1.07µs ± 1% 0.70µs ± 1% -34.67% (p=0.000 n=10+9)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-04 03:35:57 +00:00
|
|
|
//
|
|
|
|
// WARNING: This package, like most of the tailscale.com Go module,
|
|
|
|
// should be considered Tailscale-internal; we make no API promises.
|
2021-05-11 19:09:25 +00:00
|
|
|
package deephash
|
2020-06-28 17:58:21 +00:00
|
|
|
|
|
|
|
import (
|
2021-05-10 20:29:56 +00:00
|
|
|
"bufio"
|
2020-06-28 17:58:21 +00:00
|
|
|
"crypto/sha256"
|
2021-07-06 05:13:33 +00:00
|
|
|
"encoding/binary"
|
2021-05-19 18:51:21 +00:00
|
|
|
"encoding/hex"
|
2020-06-28 17:58:21 +00:00
|
|
|
"fmt"
|
2021-05-11 20:17:12 +00:00
|
|
|
"hash"
|
2022-06-15 05:49:11 +00:00
|
|
|
"log"
|
2021-07-06 04:28:54 +00:00
|
|
|
"math"
|
2020-06-28 17:58:21 +00:00
|
|
|
"reflect"
|
2021-05-11 20:17:12 +00:00
|
|
|
"sync"
|
2021-07-21 16:23:04 +00:00
|
|
|
"time"
|
2021-07-22 22:22:48 +00:00
|
|
|
"unsafe"
|
2020-06-28 17:58:21 +00:00
|
|
|
)
|
|
|
|
|
util/deephash: remove unnecessary formatting for structs and slices (#2571)
The index for every struct field or slice element and
the number of fields for the struct is unncessary.
The hashing of Go values is unambiguous because every type (except maps)
encodes in a parsable manner. So long as we know the type information,
we could theoretically decode every value (except for maps).
At a high level:
* numbers are encoded as fixed-width records according to precision.
* strings (and AppendTo output) are encoded with a fixed-width length,
followed by the contents of the buffer.
* slices are prefixed by a fixed-width length, followed by the encoding
of each value. So long as we know the type of each element, we could
theoretically decode each element.
* arrays are encoded just like slices, but elide the length
since it is determined from the Go type.
* maps are encoded first with a byte indicating whether it is a cycle.
If a cycle, it is followed by a fixed-width index for the pointer,
otherwise followed by the SHA-256 hash of its contents. The encoding of maps
is not decodeable, but a SHA-256 hash is sufficient to avoid ambiguities.
* interfaces are encoded first with a byte indicating whether it is nil.
If not nil, it is followed by a fixed-width index for the type,
and then the encoding for the underlying value. Having the type be encoded
first ensures that the value could theoretically be decoded next.
* pointers are encoded first with a byte indicating whether it is
1) nil, 2) a cycle, or 3) newly seen. If a cycle, it is followed by
a fixed-width index for the pointer. If newly seen, it is followed by
the encoding for the pointed-at value.
Removing unnecessary details speeds up hashing:
name old time/op new time/op delta
Hash-8 76.0µs ± 1% 55.8µs ± 2% -26.62% (p=0.000 n=10+10)
HashMapAcyclic-8 61.9µs ± 0% 62.0µs ± 0% ~ (p=0.666 n=9+9)
TailcfgNode-8 10.2µs ± 1% 7.5µs ± 1% -26.90% (p=0.000 n=10+9)
HashArray-8 1.07µs ± 1% 0.70µs ± 1% -34.67% (p=0.000 n=10+9)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-04 03:35:57 +00:00
|
|
|
// There is much overlap between the theory of serialization and hashing.
|
2021-08-24 14:36:48 +00:00
|
|
|
// A hash (useful for determining equality) can be produced by printing a value
|
util/deephash: remove unnecessary formatting for structs and slices (#2571)
The index for every struct field or slice element and
the number of fields for the struct is unncessary.
The hashing of Go values is unambiguous because every type (except maps)
encodes in a parsable manner. So long as we know the type information,
we could theoretically decode every value (except for maps).
At a high level:
* numbers are encoded as fixed-width records according to precision.
* strings (and AppendTo output) are encoded with a fixed-width length,
followed by the contents of the buffer.
* slices are prefixed by a fixed-width length, followed by the encoding
of each value. So long as we know the type of each element, we could
theoretically decode each element.
* arrays are encoded just like slices, but elide the length
since it is determined from the Go type.
* maps are encoded first with a byte indicating whether it is a cycle.
If a cycle, it is followed by a fixed-width index for the pointer,
otherwise followed by the SHA-256 hash of its contents. The encoding of maps
is not decodeable, but a SHA-256 hash is sufficient to avoid ambiguities.
* interfaces are encoded first with a byte indicating whether it is nil.
If not nil, it is followed by a fixed-width index for the type,
and then the encoding for the underlying value. Having the type be encoded
first ensures that the value could theoretically be decoded next.
* pointers are encoded first with a byte indicating whether it is
1) nil, 2) a cycle, or 3) newly seen. If a cycle, it is followed by
a fixed-width index for the pointer. If newly seen, it is followed by
the encoding for the pointed-at value.
Removing unnecessary details speeds up hashing:
name old time/op new time/op delta
Hash-8 76.0µs ± 1% 55.8µs ± 2% -26.62% (p=0.000 n=10+10)
HashMapAcyclic-8 61.9µs ± 0% 62.0µs ± 0% ~ (p=0.666 n=9+9)
TailcfgNode-8 10.2µs ± 1% 7.5µs ± 1% -26.90% (p=0.000 n=10+9)
HashArray-8 1.07µs ± 1% 0.70µs ± 1% -34.67% (p=0.000 n=10+9)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2021-08-04 03:35:57 +00:00
|
|
|
// and hashing the output. The format must:
|
|
|
|
// * be deterministic such that the same value hashes to the same output, and
|
|
|
|
// * be parsable such that the same value can be reproduced by the output.
|
|
|
|
//
|
|
|
|
// The logic below hashes a value by printing it to a hash.Hash.
|
|
|
|
// To be parsable, it assumes that we know the Go type of each value:
|
|
|
|
// * scalar types (e.g., bool or int32) are printed as fixed-width fields.
|
|
|
|
// * list types (e.g., strings, slices, and AppendTo buffers) are prefixed
|
|
|
|
// by a fixed-width length field, followed by the contents of the list.
|
|
|
|
// * slices, arrays, and structs print each element/field consecutively.
|
|
|
|
// * interfaces print with a 1-byte prefix indicating whether it is nil.
|
|
|
|
// If non-nil, it is followed by a fixed-width field of the type index,
|
|
|
|
// followed by the format of the underlying value.
|
|
|
|
// * pointers print with a 1-byte prefix indicating whether the pointer is
|
|
|
|
// 1) nil, 2) previously seen, or 3) newly seen. Previously seen pointers are
|
|
|
|
// followed by a fixed-width field with the index of the previous pointer.
|
|
|
|
// Newly seen pointers are followed by the format of the underlying value.
|
|
|
|
// * maps print with a 1-byte prefix indicating whether the map pointer is
|
|
|
|
// 1) nil, 2) previously seen, or 3) newly seen. Previously seen pointers
|
|
|
|
// are followed by a fixed-width field of the index of the previous pointer.
|
|
|
|
// Newly seen maps are printed as a fixed-width field with the XOR of the
|
|
|
|
// hash of every map entry. With a sufficiently strong hash, this value is
|
|
|
|
// theoretically "parsable" by looking up the hash in a magical map that
|
|
|
|
// returns the set of entries for that given hash.
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
// addressableValue is a reflect.Value that is guaranteed to be addressable
|
|
|
|
// such that calling the Addr and Set methods do not panic.
|
|
|
|
//
|
|
|
|
// There is no compile magic that enforces this property,
|
|
|
|
// but rather the need to construct this type makes it easier to examine each
|
|
|
|
// construction site to ensure that this property is upheld.
|
|
|
|
type addressableValue struct{ reflect.Value }
|
|
|
|
|
|
|
|
// newAddressableValue constructs a new addressable value of type t.
|
|
|
|
func newAddressableValue(t reflect.Type) addressableValue {
|
|
|
|
return addressableValue{reflect.New(t).Elem()} // dereferenced pointer is always addressable
|
|
|
|
}
|
|
|
|
|
2021-07-06 05:13:33 +00:00
|
|
|
const scratchSize = 128
|
|
|
|
|
2021-07-05 04:25:15 +00:00
|
|
|
// hasher is reusable state for hashing a value.
|
|
|
|
// Get one via hasherPool.
|
|
|
|
type hasher struct {
|
2021-07-22 22:22:48 +00:00
|
|
|
h hash.Hash
|
|
|
|
bw *bufio.Writer
|
|
|
|
scratch [scratchSize]byte
|
|
|
|
visitStack visitStack
|
2020-06-28 17:58:21 +00:00
|
|
|
}
|
|
|
|
|
2021-08-03 04:29:14 +00:00
|
|
|
func (h *hasher) reset() {
|
|
|
|
if h.h == nil {
|
|
|
|
h.h = sha256.New()
|
|
|
|
}
|
|
|
|
if h.bw == nil {
|
|
|
|
h.bw = bufio.NewWriterSize(h.h, h.h.BlockSize())
|
|
|
|
}
|
|
|
|
h.bw.Flush()
|
|
|
|
h.h.Reset()
|
2021-07-07 05:37:32 +00:00
|
|
|
}
|
|
|
|
|
2021-07-20 05:49:51 +00:00
|
|
|
// Sum is an opaque checksum type that is comparable.
|
|
|
|
type Sum struct {
|
|
|
|
sum [sha256.Size]byte
|
|
|
|
}
|
|
|
|
|
2021-08-03 04:29:14 +00:00
|
|
|
func (s1 *Sum) xor(s2 Sum) {
|
|
|
|
for i := 0; i < sha256.Size; i++ {
|
|
|
|
s1.sum[i] ^= s2.sum[i]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-07-20 05:49:51 +00:00
|
|
|
func (s Sum) String() string {
|
|
|
|
return hex.EncodeToString(s.sum[:])
|
|
|
|
}
|
|
|
|
|
2021-07-21 16:23:04 +00:00
|
|
|
var (
|
2022-06-16 20:21:32 +00:00
|
|
|
seedOnce sync.Once
|
|
|
|
seed uint64
|
2021-07-21 16:23:04 +00:00
|
|
|
)
|
|
|
|
|
2022-06-16 20:21:32 +00:00
|
|
|
func initSeed() {
|
|
|
|
seed = uint64(time.Now().UnixNano())
|
|
|
|
}
|
|
|
|
|
2021-08-03 04:29:14 +00:00
|
|
|
func (h *hasher) sum() (s Sum) {
|
2021-07-05 04:25:15 +00:00
|
|
|
h.bw.Flush()
|
2021-07-06 05:13:33 +00:00
|
|
|
// Sum into scratch & copy out, as hash.Hash is an interface
|
|
|
|
// so the slice necessarily escapes, and there's no sha256
|
|
|
|
// concrete type exported and we don't want the 'hash' result
|
|
|
|
// parameter to escape to the heap:
|
2021-08-03 04:29:14 +00:00
|
|
|
copy(s.sum[:], h.h.Sum(h.scratch[:0]))
|
|
|
|
return s
|
2021-07-05 04:25:15 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
var hasherPool = &sync.Pool{
|
2022-03-16 23:27:57 +00:00
|
|
|
New: func() any { return new(hasher) },
|
2021-07-05 04:25:15 +00:00
|
|
|
}
|
|
|
|
|
2021-07-20 05:49:51 +00:00
|
|
|
// Hash returns the hash of v.
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
// For performance, this should be a non-nil pointer.
|
2022-03-16 23:27:57 +00:00
|
|
|
func Hash(v any) (s Sum) {
|
2021-07-07 05:37:32 +00:00
|
|
|
h := hasherPool.Get().(*hasher)
|
|
|
|
defer hasherPool.Put(h)
|
2021-08-03 04:29:14 +00:00
|
|
|
h.reset()
|
2022-06-16 20:21:32 +00:00
|
|
|
seedOnce.Do(initSeed)
|
2021-08-03 04:44:13 +00:00
|
|
|
h.hashUint64(seed)
|
2022-06-22 02:50:48 +00:00
|
|
|
|
|
|
|
rv := reflect.ValueOf(v)
|
|
|
|
if rv.IsValid() {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
var va addressableValue
|
|
|
|
if rv.Kind() == reflect.Pointer && !rv.IsNil() {
|
|
|
|
va = addressableValue{rv.Elem()} // dereferenced pointer is always addressable
|
|
|
|
} else {
|
|
|
|
va = newAddressableValue(rv.Type())
|
|
|
|
va.Set(rv)
|
|
|
|
}
|
|
|
|
|
2022-06-22 02:50:48 +00:00
|
|
|
// Always treat the Hash input as an interface (it is), including hashing
|
|
|
|
// its type, otherwise two Hash calls of different types could hash to the
|
|
|
|
// same bytes off the different types and get equivalent Sum values. This is
|
|
|
|
// the same thing that we do for reflect.Kind Interface in hashValue, but
|
|
|
|
// the initial reflect.ValueOf from an interface value effectively strips
|
|
|
|
// the interface box off so we have to do it at the top level by hand.
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
h.hashType(va.Type())
|
|
|
|
h.hashValue(va, false)
|
2022-06-22 02:50:48 +00:00
|
|
|
}
|
2021-08-03 04:29:14 +00:00
|
|
|
return h.sum()
|
2021-07-05 04:25:15 +00:00
|
|
|
}
|
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
// HasherForType is like Hash, but it returns a Hash func that's specialized for
|
|
|
|
// the provided reflect type, avoiding a map lookup per value.
|
|
|
|
func HasherForType[T any]() func(T) Sum {
|
|
|
|
var zeroT T
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
t := reflect.TypeOf(zeroT)
|
|
|
|
ti := getTypeInfo(t)
|
|
|
|
var tiElem *typeInfo
|
|
|
|
if t.Kind() == reflect.Pointer {
|
|
|
|
tiElem = getTypeInfo(t.Elem())
|
|
|
|
}
|
2022-06-15 05:49:11 +00:00
|
|
|
seedOnce.Do(initSeed)
|
|
|
|
|
|
|
|
return func(v T) Sum {
|
|
|
|
h := hasherPool.Get().(*hasher)
|
|
|
|
defer hasherPool.Put(h)
|
|
|
|
h.reset()
|
|
|
|
h.hashUint64(seed)
|
|
|
|
|
|
|
|
rv := reflect.ValueOf(v)
|
|
|
|
|
|
|
|
if rv.IsValid() {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
if rv.Kind() == reflect.Pointer && !rv.IsNil() {
|
|
|
|
va := addressableValue{rv.Elem()} // dereferenced pointer is always addressable
|
|
|
|
h.hashType(va.Type())
|
|
|
|
h.hashValueWithType(va, tiElem, false)
|
|
|
|
} else {
|
|
|
|
va := newAddressableValue(rv.Type())
|
|
|
|
va.Set(rv)
|
|
|
|
h.hashType(va.Type())
|
|
|
|
h.hashValueWithType(va, ti, false)
|
|
|
|
}
|
2022-06-15 05:49:11 +00:00
|
|
|
}
|
|
|
|
return h.sum()
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-07-20 05:49:51 +00:00
|
|
|
// Update sets last to the hash of v and reports whether its value changed.
|
2022-03-16 23:27:57 +00:00
|
|
|
func Update(last *Sum, v ...any) (changed bool) {
|
2021-07-05 04:25:15 +00:00
|
|
|
sum := Hash(v)
|
2021-07-20 05:49:51 +00:00
|
|
|
if sum == *last {
|
2021-07-05 04:25:15 +00:00
|
|
|
// unchanged.
|
|
|
|
return false
|
|
|
|
}
|
2021-07-20 05:49:51 +00:00
|
|
|
*last = sum
|
2021-07-05 04:25:15 +00:00
|
|
|
return true
|
2020-07-29 01:47:23 +00:00
|
|
|
}
|
|
|
|
|
2021-05-24 22:13:18 +00:00
|
|
|
var appenderToType = reflect.TypeOf((*appenderTo)(nil)).Elem()
|
2021-05-10 21:15:31 +00:00
|
|
|
|
2021-05-24 21:31:24 +00:00
|
|
|
type appenderTo interface {
|
|
|
|
AppendTo([]byte) []byte
|
|
|
|
}
|
|
|
|
|
2021-08-03 04:44:13 +00:00
|
|
|
func (h *hasher) hashUint8(i uint8) {
|
|
|
|
h.bw.WriteByte(i)
|
2021-07-06 05:13:33 +00:00
|
|
|
}
|
2021-08-03 04:44:13 +00:00
|
|
|
func (h *hasher) hashUint16(i uint16) {
|
|
|
|
binary.LittleEndian.PutUint16(h.scratch[:2], i)
|
|
|
|
h.bw.Write(h.scratch[:2])
|
|
|
|
}
|
|
|
|
func (h *hasher) hashUint32(i uint32) {
|
|
|
|
binary.LittleEndian.PutUint32(h.scratch[:4], i)
|
|
|
|
h.bw.Write(h.scratch[:4])
|
|
|
|
}
|
2022-06-15 05:49:11 +00:00
|
|
|
func (h *hasher) hashLen(n int) {
|
|
|
|
binary.LittleEndian.PutUint64(h.scratch[:8], uint64(n))
|
|
|
|
h.bw.Write(h.scratch[:8])
|
|
|
|
}
|
2021-08-03 04:44:13 +00:00
|
|
|
func (h *hasher) hashUint64(i uint64) {
|
|
|
|
binary.LittleEndian.PutUint64(h.scratch[:8], i)
|
2021-07-06 05:13:33 +00:00
|
|
|
h.bw.Write(h.scratch[:8])
|
|
|
|
}
|
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
var (
|
|
|
|
uint8Type = reflect.TypeOf(byte(0))
|
|
|
|
timeTimeType = reflect.TypeOf(time.Time{})
|
|
|
|
)
|
2021-07-07 18:58:02 +00:00
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
// typeInfo describes properties of a type.
|
2022-06-15 05:49:11 +00:00
|
|
|
//
|
|
|
|
// A non-nil typeInfo is populated into the typeHasher map
|
|
|
|
// when its type is first requested, before its func is created.
|
|
|
|
// Its func field fn is only populated once the type has been created.
|
|
|
|
// This is used for recursive types.
|
2022-06-15 05:49:11 +00:00
|
|
|
type typeInfo struct {
|
|
|
|
rtype reflect.Type
|
2022-06-25 19:46:01 +00:00
|
|
|
canMemHash bool
|
2022-06-15 05:49:11 +00:00
|
|
|
isRecursive bool
|
|
|
|
|
|
|
|
// elemTypeInfo is the element type's typeInfo.
|
|
|
|
// It's set when rtype is of Kind Ptr, Slice, Array, Map.
|
|
|
|
elemTypeInfo *typeInfo
|
|
|
|
|
|
|
|
// keyTypeInfo is the map key type's typeInfo.
|
|
|
|
// It's set when rtype is of Kind Map.
|
|
|
|
keyTypeInfo *typeInfo
|
2022-06-15 05:49:11 +00:00
|
|
|
|
|
|
|
hashFuncOnce sync.Once
|
|
|
|
hashFuncLazy typeHasherFunc // nil until created
|
2022-06-15 05:49:11 +00:00
|
|
|
}
|
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
// returns ok if it was handled; else slow path runs
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
type typeHasherFunc func(h *hasher, v addressableValue) (ok bool)
|
2022-06-15 05:49:11 +00:00
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
var typeInfoMap sync.Map // map[reflect.Type]*typeInfo
|
|
|
|
var typeInfoMapPopulate sync.Mutex // just for adding to typeInfoMap
|
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
func (ti *typeInfo) hasher() typeHasherFunc {
|
|
|
|
ti.hashFuncOnce.Do(ti.buildHashFuncOnce)
|
|
|
|
return ti.hashFuncLazy
|
|
|
|
}
|
|
|
|
|
|
|
|
func (ti *typeInfo) buildHashFuncOnce() {
|
|
|
|
ti.hashFuncLazy = genTypeHasher(ti.rtype)
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashBoolv(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
var b byte
|
|
|
|
if v.Bool() {
|
|
|
|
b = 1
|
|
|
|
}
|
|
|
|
h.hashUint8(b)
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashUint8v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint8(uint8(v.Uint()))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashInt8v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint8(uint8(v.Int()))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashUint16v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint16(uint16(v.Uint()))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashInt16v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint16(uint16(v.Int()))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashUint32v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint32(uint32(v.Uint()))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashInt32v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint32(uint32(v.Int()))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashUint64v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint64(v.Uint())
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashInt64v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint64(uint64(v.Int()))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func hashStructAppenderTo(h *hasher, v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
if !v.CanInterface() {
|
|
|
|
return false // slow path
|
|
|
|
}
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
a := v.Addr().Interface().(appenderTo)
|
2022-06-15 05:49:11 +00:00
|
|
|
size := h.scratch[:8]
|
|
|
|
record := a.AppendTo(size)
|
|
|
|
binary.LittleEndian.PutUint64(record, uint64(len(record)-len(size)))
|
|
|
|
h.bw.Write(record)
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
|
|
|
// hashPointerAppenderTo hashes v, a reflect.Ptr, that implements appenderTo.
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func hashPointerAppenderTo(h *hasher, v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
if !v.CanInterface() {
|
|
|
|
return false // slow path
|
|
|
|
}
|
|
|
|
if v.IsNil() {
|
|
|
|
h.hashUint8(0) // indicates nil
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
h.hashUint8(1) // indicates visiting a pointer
|
|
|
|
a := v.Interface().(appenderTo)
|
|
|
|
size := h.scratch[:8]
|
|
|
|
record := a.AppendTo(size)
|
|
|
|
binary.LittleEndian.PutUint64(record, uint64(len(record)-len(size)))
|
|
|
|
h.bw.Write(record)
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
|
|
|
// fieldInfo describes a struct field.
|
|
|
|
type fieldInfo struct {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
index int // index of field for reflect.Value.Field(n); -1 if invalid
|
2022-06-15 05:49:11 +00:00
|
|
|
typeInfo *typeInfo
|
|
|
|
canMemHash bool
|
|
|
|
offset uintptr // when we can memhash the field
|
|
|
|
size uintptr // when we can memhash the field
|
|
|
|
}
|
|
|
|
|
|
|
|
// mergeContiguousFieldsCopy returns a copy of f with contiguous memhashable fields
|
|
|
|
// merged together. Such fields get a bogus index and fu value.
|
|
|
|
func mergeContiguousFieldsCopy(in []fieldInfo) []fieldInfo {
|
|
|
|
ret := make([]fieldInfo, 0, len(in))
|
|
|
|
var last *fieldInfo
|
|
|
|
for _, f := range in {
|
|
|
|
// Combine two fields if they're both contiguous & memhash-able.
|
|
|
|
if f.canMemHash && last != nil && last.canMemHash && last.offset+last.size == f.offset {
|
|
|
|
last.size += f.size
|
|
|
|
last.index = -1
|
|
|
|
last.typeInfo = nil
|
|
|
|
} else {
|
|
|
|
ret = append(ret, f)
|
|
|
|
last = &ret[len(ret)-1]
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return ret
|
|
|
|
}
|
|
|
|
|
|
|
|
// genHashStructFields generates a typeHasherFunc for t, which must be of kind Struct.
|
|
|
|
func genHashStructFields(t reflect.Type) typeHasherFunc {
|
|
|
|
fields := make([]fieldInfo, 0, t.NumField())
|
|
|
|
for i, n := 0, t.NumField(); i < n; i++ {
|
|
|
|
sf := t.Field(i)
|
|
|
|
if sf.Type.Size() == 0 {
|
|
|
|
continue
|
|
|
|
}
|
|
|
|
fields = append(fields, fieldInfo{
|
|
|
|
index: i,
|
|
|
|
typeInfo: getTypeInfo(sf.Type),
|
|
|
|
canMemHash: canMemHash(sf.Type),
|
|
|
|
offset: sf.Offset,
|
|
|
|
size: sf.Type.Size(),
|
|
|
|
})
|
|
|
|
}
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
fields = mergeContiguousFieldsCopy(fields)
|
|
|
|
return structHasher{fields}.hash
|
2022-06-15 05:49:11 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
type structHasher struct {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
fields []fieldInfo
|
2022-06-15 05:49:11 +00:00
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (sh structHasher) hash(h *hasher, v addressableValue) bool {
|
|
|
|
base := v.Addr().UnsafePointer()
|
|
|
|
for _, f := range sh.fields {
|
|
|
|
if f.canMemHash {
|
|
|
|
h.bw.Write(unsafe.Slice((*byte)(unsafe.Pointer(uintptr(base)+f.offset)), f.size))
|
|
|
|
continue
|
2022-06-15 05:49:11 +00:00
|
|
|
}
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
va := addressableValue{v.Field(f.index)} // field is addressable if parent struct is addressable
|
|
|
|
if !f.typeInfo.hasher()(h, va) {
|
|
|
|
return false
|
2022-06-15 05:49:11 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
|
|
|
// genHashPtrToMemoryRange returns a hasher where the reflect.Value is a Ptr to
|
|
|
|
// the provided eleType.
|
|
|
|
func genHashPtrToMemoryRange(eleType reflect.Type) typeHasherFunc {
|
|
|
|
size := eleType.Size()
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
return func(h *hasher, v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
if v.IsNil() {
|
|
|
|
h.hashUint8(0) // indicates nil
|
|
|
|
} else {
|
|
|
|
h.hashUint8(1) // indicates visiting a pointer
|
|
|
|
h.bw.Write(unsafe.Slice((*byte)(v.UnsafePointer()), size))
|
|
|
|
}
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
const debug = false
|
|
|
|
|
|
|
|
func genTypeHasher(t reflect.Type) typeHasherFunc {
|
|
|
|
if debug {
|
|
|
|
log.Printf("generating func for %v", t)
|
|
|
|
}
|
|
|
|
|
|
|
|
switch t.Kind() {
|
|
|
|
case reflect.Bool:
|
|
|
|
return (*hasher).hashBoolv
|
|
|
|
case reflect.Int8:
|
|
|
|
return (*hasher).hashInt8v
|
|
|
|
case reflect.Int16:
|
|
|
|
return (*hasher).hashInt16v
|
|
|
|
case reflect.Int32:
|
|
|
|
return (*hasher).hashInt32v
|
|
|
|
case reflect.Int, reflect.Int64:
|
|
|
|
return (*hasher).hashInt64v
|
|
|
|
case reflect.Uint8:
|
|
|
|
return (*hasher).hashUint8v
|
|
|
|
case reflect.Uint16:
|
|
|
|
return (*hasher).hashUint16v
|
|
|
|
case reflect.Uint32:
|
|
|
|
return (*hasher).hashUint32v
|
|
|
|
case reflect.Uint, reflect.Uintptr, reflect.Uint64:
|
|
|
|
return (*hasher).hashUint64v
|
|
|
|
case reflect.Float32:
|
|
|
|
return (*hasher).hashFloat32v
|
|
|
|
case reflect.Float64:
|
|
|
|
return (*hasher).hashFloat64v
|
|
|
|
case reflect.Complex64:
|
|
|
|
return (*hasher).hashComplex64v
|
|
|
|
case reflect.Complex128:
|
|
|
|
return (*hasher).hashComplex128v
|
|
|
|
case reflect.String:
|
|
|
|
return (*hasher).hashString
|
|
|
|
case reflect.Slice:
|
|
|
|
et := t.Elem()
|
|
|
|
if canMemHash(et) {
|
|
|
|
return (*hasher).hashSliceMem
|
|
|
|
}
|
|
|
|
eti := getTypeInfo(et)
|
|
|
|
return genHashSliceElements(eti)
|
|
|
|
case reflect.Array:
|
|
|
|
et := t.Elem()
|
|
|
|
eti := getTypeInfo(et)
|
|
|
|
return genHashArray(t, eti)
|
|
|
|
case reflect.Struct:
|
|
|
|
if t == timeTimeType {
|
|
|
|
return (*hasher).hashTimev
|
|
|
|
}
|
|
|
|
if t.Implements(appenderToType) {
|
|
|
|
return hashStructAppenderTo
|
|
|
|
}
|
|
|
|
return genHashStructFields(t)
|
|
|
|
case reflect.Pointer:
|
|
|
|
et := t.Elem()
|
|
|
|
if canMemHash(et) {
|
|
|
|
return genHashPtrToMemoryRange(et)
|
|
|
|
}
|
|
|
|
if t.Implements(appenderToType) {
|
|
|
|
return hashPointerAppenderTo
|
|
|
|
}
|
|
|
|
if !typeIsRecursive(t) {
|
|
|
|
eti := getTypeInfo(et)
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
return func(h *hasher, v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
if v.IsNil() {
|
|
|
|
h.hashUint8(0) // indicates nil
|
|
|
|
return true
|
|
|
|
}
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
h.hashUint8(1) // indicates visiting a pointer
|
|
|
|
va := addressableValue{v.Elem()} // dereferenced pointer is always addressable
|
|
|
|
return eti.hasher()(h, va)
|
2022-06-15 05:49:11 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
return func(h *hasher, v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
if debug {
|
|
|
|
log.Printf("unhandled type %v", v.Type())
|
|
|
|
}
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// hashString hashes v, of kind String.
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashString(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
s := v.String()
|
|
|
|
h.hashLen(len(s))
|
|
|
|
h.bw.WriteString(s)
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashFloat32v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint32(math.Float32bits(float32(v.Float())))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashFloat64v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashUint64(math.Float64bits(v.Float()))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashComplex64v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
c := complex64(v.Complex())
|
|
|
|
h.hashUint32(math.Float32bits(real(c)))
|
|
|
|
h.hashUint32(math.Float32bits(imag(c)))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashComplex128v(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
c := v.Complex()
|
|
|
|
h.hashUint64(math.Float64bits(real(c)))
|
|
|
|
h.hashUint64(math.Float64bits(imag(c)))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
|
|
|
// hashString hashes v, of kind time.Time.
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashTimev(v addressableValue) bool {
|
|
|
|
t := *(*time.Time)(v.Addr().UnsafePointer())
|
2022-06-15 05:49:11 +00:00
|
|
|
b := t.AppendFormat(h.scratch[:1], time.RFC3339Nano)
|
|
|
|
b[0] = byte(len(b) - 1) // more than sufficient width; if not, good enough.
|
|
|
|
h.bw.Write(b)
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
|
|
|
// hashSliceMem hashes v, of kind Slice, with a memhash-able element type.
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashSliceMem(v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
vLen := v.Len()
|
|
|
|
h.hashUint64(uint64(vLen))
|
|
|
|
if vLen == 0 {
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
h.bw.Write(unsafe.Slice((*byte)(v.UnsafePointer()), v.Type().Elem().Size()*uintptr(vLen)))
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
|
|
|
func genHashArrayMem(n int, arraySize uintptr, efu *typeInfo) typeHasherFunc {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
return func(h *hasher, v addressableValue) bool {
|
|
|
|
h.bw.Write(unsafe.Slice((*byte)(v.Addr().UnsafePointer()), arraySize))
|
|
|
|
return true
|
2022-06-15 05:49:11 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
func genHashArrayElements(n int, eti *typeInfo) typeHasherFunc {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
return func(h *hasher, v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
for i := 0; i < n; i++ {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
va := addressableValue{v.Index(i)} // element is addressable if parent array is addressable
|
|
|
|
if !eti.hasher()(h, va) {
|
2022-06-15 05:49:11 +00:00
|
|
|
return false
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func noopHasherFunc(h *hasher, v addressableValue) bool { return true }
|
2022-06-15 05:49:11 +00:00
|
|
|
|
|
|
|
func genHashArray(t reflect.Type, eti *typeInfo) typeHasherFunc {
|
|
|
|
if t.Size() == 0 {
|
|
|
|
return noopHasherFunc
|
|
|
|
}
|
|
|
|
et := t.Elem()
|
|
|
|
if canMemHash(et) {
|
|
|
|
return genHashArrayMem(t.Len(), t.Size(), eti)
|
|
|
|
}
|
|
|
|
n := t.Len()
|
|
|
|
return genHashArrayElements(n, eti)
|
|
|
|
}
|
|
|
|
|
|
|
|
func genHashSliceElements(eti *typeInfo) typeHasherFunc {
|
|
|
|
return sliceElementHasher{eti}.hash
|
|
|
|
}
|
|
|
|
|
|
|
|
type sliceElementHasher struct {
|
|
|
|
eti *typeInfo
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (seh sliceElementHasher) hash(h *hasher, v addressableValue) bool {
|
2022-06-15 05:49:11 +00:00
|
|
|
vLen := v.Len()
|
|
|
|
h.hashUint64(uint64(vLen))
|
|
|
|
for i := 0; i < vLen; i++ {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
va := addressableValue{v.Index(i)} // slice elements are always addressable
|
|
|
|
if !seh.eti.hasher()(h, va) {
|
2022-06-15 05:49:11 +00:00
|
|
|
return false
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
func getTypeInfo(t reflect.Type) *typeInfo {
|
|
|
|
if f, ok := typeInfoMap.Load(t); ok {
|
|
|
|
return f.(*typeInfo)
|
|
|
|
}
|
|
|
|
typeInfoMapPopulate.Lock()
|
|
|
|
defer typeInfoMapPopulate.Unlock()
|
|
|
|
newTypes := map[reflect.Type]*typeInfo{}
|
|
|
|
ti := getTypeInfoLocked(t, newTypes)
|
|
|
|
for t, ti := range newTypes {
|
|
|
|
typeInfoMap.Store(t, ti)
|
2020-06-28 17:58:21 +00:00
|
|
|
}
|
2022-06-15 05:49:11 +00:00
|
|
|
return ti
|
|
|
|
}
|
2021-05-10 21:15:31 +00:00
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
func getTypeInfoLocked(t reflect.Type, incomplete map[reflect.Type]*typeInfo) *typeInfo {
|
|
|
|
if v, ok := typeInfoMap.Load(t); ok {
|
|
|
|
return v.(*typeInfo)
|
|
|
|
}
|
|
|
|
if ti, ok := incomplete[t]; ok {
|
|
|
|
return ti
|
|
|
|
}
|
|
|
|
ti := &typeInfo{
|
|
|
|
rtype: t,
|
|
|
|
isRecursive: typeIsRecursive(t),
|
2022-06-25 19:46:01 +00:00
|
|
|
canMemHash: canMemHash(t),
|
2022-06-15 05:49:11 +00:00
|
|
|
}
|
|
|
|
incomplete[t] = ti
|
2021-07-07 05:37:32 +00:00
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
switch t.Kind() {
|
|
|
|
case reflect.Map:
|
|
|
|
ti.keyTypeInfo = getTypeInfoLocked(t.Key(), incomplete)
|
|
|
|
fallthrough
|
|
|
|
case reflect.Ptr, reflect.Slice, reflect.Array:
|
|
|
|
ti.elemTypeInfo = getTypeInfoLocked(t.Elem(), incomplete)
|
|
|
|
}
|
|
|
|
|
|
|
|
return ti
|
|
|
|
}
|
|
|
|
|
|
|
|
// typeIsRecursive reports whether t has a path back to itself.
|
|
|
|
//
|
|
|
|
// For interfaces, it currently always reports true.
|
|
|
|
func typeIsRecursive(t reflect.Type) bool {
|
|
|
|
inStack := map[reflect.Type]bool{}
|
|
|
|
|
|
|
|
var stack []reflect.Type
|
|
|
|
|
|
|
|
var visitType func(t reflect.Type) (isRecursiveSoFar bool)
|
|
|
|
visitType = func(t reflect.Type) (isRecursiveSoFar bool) {
|
|
|
|
switch t.Kind() {
|
|
|
|
case reflect.Bool,
|
|
|
|
reflect.Int,
|
|
|
|
reflect.Int8,
|
|
|
|
reflect.Int16,
|
|
|
|
reflect.Int32,
|
|
|
|
reflect.Int64,
|
|
|
|
reflect.Uint,
|
|
|
|
reflect.Uint8,
|
|
|
|
reflect.Uint16,
|
|
|
|
reflect.Uint32,
|
|
|
|
reflect.Uint64,
|
|
|
|
reflect.Uintptr,
|
|
|
|
reflect.Float32,
|
|
|
|
reflect.Float64,
|
|
|
|
reflect.Complex64,
|
|
|
|
reflect.Complex128,
|
|
|
|
reflect.String,
|
|
|
|
reflect.UnsafePointer,
|
|
|
|
reflect.Func:
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
if t.Size() == 0 {
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
if inStack[t] {
|
|
|
|
return true
|
2021-05-24 21:31:24 +00:00
|
|
|
}
|
2022-06-15 05:49:11 +00:00
|
|
|
stack = append(stack, t)
|
|
|
|
inStack[t] = true
|
|
|
|
defer func() {
|
|
|
|
delete(inStack, t)
|
|
|
|
stack = stack[:len(stack)-1]
|
|
|
|
}()
|
|
|
|
|
|
|
|
switch t.Kind() {
|
|
|
|
default:
|
|
|
|
panic("unhandled kind " + t.Kind().String())
|
|
|
|
case reflect.Interface:
|
|
|
|
// Assume the worst for now. TODO(bradfitz): in some cases
|
|
|
|
// we should be able to prove that it's not recursive. Not worth
|
|
|
|
// it for now.
|
|
|
|
return true
|
|
|
|
case reflect.Array, reflect.Chan, reflect.Pointer, reflect.Slice:
|
|
|
|
return visitType(t.Elem())
|
|
|
|
case reflect.Map:
|
|
|
|
if visitType(t.Key()) {
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
if visitType(t.Elem()) {
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
case reflect.Struct:
|
|
|
|
if t.String() == "intern.Value" {
|
|
|
|
// Otherwise its interface{} makes this return true.
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
for i, numField := 0, t.NumField(); i < numField; i++ {
|
|
|
|
if visitType(t.Field(i).Type) {
|
|
|
|
return true
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
return false
|
2021-05-10 21:15:31 +00:00
|
|
|
}
|
2022-06-15 05:49:11 +00:00
|
|
|
return visitType(t)
|
|
|
|
}
|
2021-05-10 21:15:31 +00:00
|
|
|
|
2022-06-25 19:46:01 +00:00
|
|
|
// canMemHash reports whether a slice of t can be hashed by looking at its
|
|
|
|
// contiguous bytes in memory alone. (e.g. structs with gaps aren't memhashable)
|
|
|
|
func canMemHash(t reflect.Type) bool {
|
|
|
|
switch t.Kind() {
|
|
|
|
case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64,
|
|
|
|
reflect.Uint, reflect.Uintptr, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64,
|
|
|
|
reflect.Float64, reflect.Float32, reflect.Complex128, reflect.Complex64:
|
|
|
|
return true
|
|
|
|
case reflect.Array:
|
|
|
|
return canMemHash(t.Elem())
|
|
|
|
case reflect.Struct:
|
|
|
|
var sumFieldSize uintptr
|
|
|
|
for i, numField := 0, t.NumField(); i < numField; i++ {
|
|
|
|
sf := t.Field(i)
|
|
|
|
if !canMemHash(sf.Type) {
|
|
|
|
// Special case for 0-width fields that aren't at the end.
|
|
|
|
if sf.Type.Size() == 0 && i < numField-1 {
|
|
|
|
continue
|
|
|
|
}
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
sumFieldSize += sf.Type.Size()
|
|
|
|
}
|
|
|
|
return sumFieldSize == t.Size() // else there are gaps
|
|
|
|
}
|
|
|
|
return false
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashValue(v addressableValue, forceCycleChecking bool) {
|
2022-06-15 05:49:11 +00:00
|
|
|
if !v.IsValid() {
|
|
|
|
return
|
|
|
|
}
|
|
|
|
ti := getTypeInfo(v.Type())
|
|
|
|
h.hashValueWithType(v, ti, forceCycleChecking)
|
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashValueWithType(v addressableValue, ti *typeInfo, forceCycleChecking bool) {
|
2022-06-15 05:49:11 +00:00
|
|
|
w := h.bw
|
|
|
|
doCheckCycles := forceCycleChecking || ti.isRecursive
|
2021-07-22 22:22:48 +00:00
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
if !doCheckCycles {
|
|
|
|
hf := ti.hasher()
|
|
|
|
if hf(h, v) {
|
|
|
|
return
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2021-05-10 21:15:31 +00:00
|
|
|
// Generic handling.
|
2020-06-28 17:58:21 +00:00
|
|
|
switch v.Kind() {
|
|
|
|
default:
|
|
|
|
panic(fmt.Sprintf("unhandled kind %v for type %v", v.Kind(), v.Type()))
|
|
|
|
case reflect.Ptr:
|
2021-07-22 22:22:48 +00:00
|
|
|
if v.IsNil() {
|
2021-08-03 04:44:13 +00:00
|
|
|
h.hashUint8(0) // indicates nil
|
2021-07-22 22:22:48 +00:00
|
|
|
return
|
|
|
|
}
|
|
|
|
|
2022-06-15 05:49:11 +00:00
|
|
|
if doCheckCycles {
|
|
|
|
ptr := pointerOf(v)
|
|
|
|
if idx, ok := h.visitStack.seen(ptr); ok {
|
|
|
|
h.hashUint8(2) // indicates cycle
|
|
|
|
h.hashUint64(uint64(idx))
|
|
|
|
return
|
|
|
|
}
|
|
|
|
h.visitStack.push(ptr)
|
|
|
|
defer h.visitStack.pop(ptr)
|
2020-06-28 17:58:21 +00:00
|
|
|
}
|
2021-07-22 22:22:48 +00:00
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
h.hashUint8(1) // indicates visiting a pointer
|
|
|
|
va := addressableValue{v.Elem()} // dereferenced pointer is always addressable
|
|
|
|
h.hashValueWithType(va, ti.elemTypeInfo, doCheckCycles)
|
2020-06-28 17:58:21 +00:00
|
|
|
case reflect.Struct:
|
|
|
|
for i, n := 0, v.NumField(); i < n; i++ {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
va := addressableValue{v.Field(i)} // field is addressable if parent struct is addressable
|
|
|
|
h.hashValue(va, doCheckCycles)
|
2020-06-28 17:58:21 +00:00
|
|
|
}
|
|
|
|
case reflect.Slice, reflect.Array:
|
2021-07-06 05:13:33 +00:00
|
|
|
vLen := v.Len()
|
|
|
|
if v.Kind() == reflect.Slice {
|
2021-08-03 04:44:13 +00:00
|
|
|
h.hashUint64(uint64(vLen))
|
2021-07-06 05:13:33 +00:00
|
|
|
}
|
2021-07-07 18:58:02 +00:00
|
|
|
if v.Type().Elem() == uint8Type && v.CanInterface() {
|
2021-07-06 05:13:33 +00:00
|
|
|
if vLen > 0 && vLen <= scratchSize {
|
|
|
|
// If it fits in scratch, avoid the Interface allocation.
|
|
|
|
// It seems tempting to do this for all sizes, doing
|
|
|
|
// scratchSize bytes at a time, but reflect.Slice seems
|
|
|
|
// to allocate, so it's not a win.
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
n := reflect.Copy(reflect.ValueOf(&h.scratch).Elem(), v.Value)
|
2021-07-06 05:13:33 +00:00
|
|
|
w.Write(h.scratch[:n])
|
2021-07-22 22:22:48 +00:00
|
|
|
return
|
2021-07-06 05:13:33 +00:00
|
|
|
}
|
|
|
|
fmt.Fprintf(w, "%s", v.Interface())
|
2021-07-22 22:22:48 +00:00
|
|
|
return
|
2020-06-28 17:58:21 +00:00
|
|
|
}
|
2021-07-06 05:13:33 +00:00
|
|
|
for i := 0; i < vLen; i++ {
|
2021-07-22 22:22:48 +00:00
|
|
|
// TODO(dsnet): Perform cycle detection for slices,
|
|
|
|
// which is functionally a list of pointers.
|
|
|
|
// See https://github.com/google/go-cmp/blob/402949e8139bb890c71a707b6faf6dd05c92f4e5/cmp/compare.go#L438-L450
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
va := addressableValue{v.Index(i)} // slice elements are always addressable
|
|
|
|
h.hashValueWithType(va, ti.elemTypeInfo, doCheckCycles)
|
2020-06-28 17:58:21 +00:00
|
|
|
}
|
|
|
|
case reflect.Interface:
|
2021-07-21 17:26:04 +00:00
|
|
|
if v.IsNil() {
|
2021-08-03 04:44:13 +00:00
|
|
|
h.hashUint8(0) // indicates nil
|
2021-07-22 22:22:48 +00:00
|
|
|
return
|
2021-07-21 17:26:04 +00:00
|
|
|
}
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
// TODO: Use a valueCache here?
|
|
|
|
va := newAddressableValue(v.Elem().Type())
|
|
|
|
va.Set(v.Elem())
|
2021-07-21 17:26:04 +00:00
|
|
|
|
2021-08-03 04:44:13 +00:00
|
|
|
h.hashUint8(1) // indicates visiting interface value
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
h.hashType(va.Type())
|
|
|
|
h.hashValue(va, doCheckCycles)
|
2020-06-28 17:58:21 +00:00
|
|
|
case reflect.Map:
|
2021-07-22 22:22:48 +00:00
|
|
|
// Check for cycle.
|
2022-06-15 05:49:11 +00:00
|
|
|
if doCheckCycles {
|
|
|
|
ptr := pointerOf(v)
|
|
|
|
if idx, ok := h.visitStack.seen(ptr); ok {
|
|
|
|
h.hashUint8(2) // indicates cycle
|
|
|
|
h.hashUint64(uint64(idx))
|
|
|
|
return
|
|
|
|
}
|
|
|
|
h.visitStack.push(ptr)
|
|
|
|
defer h.visitStack.pop(ptr)
|
2021-07-07 04:41:18 +00:00
|
|
|
}
|
2021-08-03 04:44:13 +00:00
|
|
|
h.hashUint8(1) // indicates visiting a map
|
2022-06-15 05:49:11 +00:00
|
|
|
h.hashMap(v, ti, doCheckCycles)
|
2020-06-28 17:58:21 +00:00
|
|
|
case reflect.String:
|
2021-08-03 04:44:13 +00:00
|
|
|
s := v.String()
|
|
|
|
h.hashUint64(uint64(len(s)))
|
|
|
|
w.WriteString(s)
|
2020-06-28 17:58:21 +00:00
|
|
|
case reflect.Bool:
|
2021-08-03 04:44:13 +00:00
|
|
|
if v.Bool() {
|
|
|
|
h.hashUint8(1)
|
|
|
|
} else {
|
|
|
|
h.hashUint8(0)
|
|
|
|
}
|
|
|
|
case reflect.Int8:
|
|
|
|
h.hashUint8(uint8(v.Int()))
|
|
|
|
case reflect.Int16:
|
|
|
|
h.hashUint16(uint16(v.Int()))
|
|
|
|
case reflect.Int32:
|
|
|
|
h.hashUint32(uint32(v.Int()))
|
|
|
|
case reflect.Int64, reflect.Int:
|
|
|
|
h.hashUint64(uint64(v.Int()))
|
|
|
|
case reflect.Uint8:
|
|
|
|
h.hashUint8(uint8(v.Uint()))
|
|
|
|
case reflect.Uint16:
|
|
|
|
h.hashUint16(uint16(v.Uint()))
|
|
|
|
case reflect.Uint32:
|
|
|
|
h.hashUint32(uint32(v.Uint()))
|
|
|
|
case reflect.Uint64, reflect.Uint, reflect.Uintptr:
|
|
|
|
h.hashUint64(uint64(v.Uint()))
|
|
|
|
case reflect.Float32:
|
|
|
|
h.hashUint32(math.Float32bits(float32(v.Float())))
|
|
|
|
case reflect.Float64:
|
|
|
|
h.hashUint64(math.Float64bits(float64(v.Float())))
|
|
|
|
case reflect.Complex64:
|
|
|
|
h.hashUint32(math.Float32bits(real(complex64(v.Complex()))))
|
|
|
|
h.hashUint32(math.Float32bits(imag(complex64(v.Complex()))))
|
|
|
|
case reflect.Complex128:
|
|
|
|
h.hashUint64(math.Float64bits(real(complex128(v.Complex()))))
|
|
|
|
h.hashUint64(math.Float64bits(imag(complex128(v.Complex()))))
|
2020-06-28 17:58:21 +00:00
|
|
|
}
|
2021-05-11 20:17:12 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
type mapHasher struct {
|
2022-06-16 20:21:32 +00:00
|
|
|
h hasher
|
2022-08-11 07:33:40 +00:00
|
|
|
valKey, valElem valueCache // re-usable values for map iteration
|
2021-05-11 20:17:12 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
var mapHasherPool = &sync.Pool{
|
2022-03-16 23:27:57 +00:00
|
|
|
New: func() any { return new(mapHasher) },
|
2021-05-11 20:17:12 +00:00
|
|
|
}
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
type valueCache map[reflect.Type]addressableValue
|
2021-05-18 16:20:52 +00:00
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (c *valueCache) get(t reflect.Type) addressableValue {
|
2021-08-03 04:29:14 +00:00
|
|
|
v, ok := (*c)[t]
|
2021-05-18 16:20:52 +00:00
|
|
|
if !ok {
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
v = newAddressableValue(t)
|
2021-08-03 04:29:14 +00:00
|
|
|
if *c == nil {
|
|
|
|
*c = make(valueCache)
|
|
|
|
}
|
|
|
|
(*c)[t] = v
|
2021-05-18 16:20:52 +00:00
|
|
|
}
|
|
|
|
return v
|
|
|
|
}
|
|
|
|
|
2021-07-22 22:22:48 +00:00
|
|
|
// hashMap hashes a map in a sort-free manner.
|
|
|
|
// It relies on a map being a functionally an unordered set of KV entries.
|
|
|
|
// So long as we hash each KV entry together, we can XOR all
|
|
|
|
// of the individual hashes to produce a unique hash for the entire map.
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func (h *hasher) hashMap(v addressableValue, ti *typeInfo, checkCycles bool) {
|
2021-05-11 20:17:12 +00:00
|
|
|
mh := mapHasherPool.Get().(*mapHasher)
|
|
|
|
defer mapHasherPool.Put(mh)
|
2022-03-15 22:44:28 +00:00
|
|
|
|
2021-08-03 04:29:14 +00:00
|
|
|
var sum Sum
|
2022-06-16 20:21:32 +00:00
|
|
|
if v.IsNil() {
|
|
|
|
sum.sum[0] = 1 // something non-zero
|
|
|
|
}
|
|
|
|
|
|
|
|
k := mh.valKey.get(v.Type().Key())
|
|
|
|
e := mh.valElem.get(v.Type().Elem())
|
2021-08-03 04:29:14 +00:00
|
|
|
mh.h.visitStack = h.visitStack // always use the parent's visit stack to avoid cycles
|
2022-08-11 07:33:40 +00:00
|
|
|
for iter := v.MapRange(); iter.Next(); {
|
2022-03-15 22:44:28 +00:00
|
|
|
k.SetIterKey(iter)
|
|
|
|
e.SetIterValue(iter)
|
2021-08-03 04:29:14 +00:00
|
|
|
mh.h.reset()
|
2022-06-15 05:49:11 +00:00
|
|
|
mh.h.hashValueWithType(k, ti.keyTypeInfo, checkCycles)
|
|
|
|
mh.h.hashValueWithType(e, ti.elemTypeInfo, checkCycles)
|
2021-08-03 04:29:14 +00:00
|
|
|
sum.xor(mh.h.sum())
|
2021-05-11 20:17:12 +00:00
|
|
|
}
|
2021-08-03 04:29:14 +00:00
|
|
|
h.bw.Write(append(h.scratch[:0], sum.sum[:]...)) // append into scratch to avoid heap allocation
|
2021-05-11 20:17:12 +00:00
|
|
|
}
|
|
|
|
|
2021-07-22 22:22:48 +00:00
|
|
|
// visitStack is a stack of pointers visited.
|
|
|
|
// Pointers are pushed onto the stack when visited, and popped when leaving.
|
|
|
|
// The integer value is the depth at which the pointer was visited.
|
|
|
|
// The length of this stack should be zero after every hashing operation.
|
|
|
|
type visitStack map[pointer]int
|
|
|
|
|
|
|
|
func (v visitStack) seen(p pointer) (int, bool) {
|
|
|
|
idx, ok := v[p]
|
|
|
|
return idx, ok
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v *visitStack) push(p pointer) {
|
|
|
|
if *v == nil {
|
|
|
|
*v = make(map[pointer]int)
|
2021-05-11 20:17:12 +00:00
|
|
|
}
|
2021-07-22 22:22:48 +00:00
|
|
|
(*v)[p] = len(*v)
|
|
|
|
}
|
|
|
|
|
|
|
|
func (v visitStack) pop(p pointer) {
|
|
|
|
delete(v, p)
|
|
|
|
}
|
|
|
|
|
|
|
|
// pointer is a thin wrapper over unsafe.Pointer.
|
|
|
|
// We only rely on comparability of pointers; we cannot rely on uintptr since
|
|
|
|
// that would break if Go ever switched to a moving GC.
|
|
|
|
type pointer struct{ p unsafe.Pointer }
|
|
|
|
|
util/deephash: always keep values addressable (#5328)
The logic of deephash is both simpler and easier to reason about
if values are always addressable.
In Go, the composite kinds are slices, arrays, maps, structs,
interfaces, pointers, channels, and functions,
where we define "composite" as a Go value that encapsulates
some other Go value (e.g., a map is a collection of key-value entries).
In the cases of pointers and slices, the sub-values are always addressable.
In the cases of arrays and structs, the sub-values are always addressable
if and only if the parent value is addressable.
In the case of maps and interfaces, the sub-values are never addressable.
To make them addressable, we need to copy them onto the heap.
For the purposes of deephash, we do not care about channels and functions.
For all non-composite kinds (e.g., strings and ints), they are only addressable
if obtained from one of the composite kinds that produce addressable values
(i.e., pointers, slices, addressable arrays, and addressable structs).
A non-addressible, non-composite kind can be made addressable by
allocating it on the heap, obtaining a pointer to it, and dereferencing it.
Thus, if we can ensure that values are addressable at the entry points,
and shallow copy sub-values whenever we encounter an interface or map,
then we can ensure that all values are always addressable and
assume such property throughout all the logic.
Performance:
name old time/op new time/op delta
Hash-24 21.5µs ± 1% 19.7µs ± 1% -8.29% (p=0.000 n=9+9)
HashPacketFilter-24 2.61µs ± 1% 2.62µs ± 0% +0.29% (p=0.037 n=10+9)
HashMapAcyclic-24 30.8µs ± 1% 30.9µs ± 1% ~ (p=0.400 n=9+10)
TailcfgNode-24 1.84µs ± 1% 1.84µs ± 2% ~ (p=0.928 n=10+10)
HashArray-24 324ns ± 2% 332ns ± 2% +2.45% (p=0.000 n=10+10)
Signed-off-by: Joe Tsai <joetsai@digital-static.net>
2022-08-10 05:00:02 +00:00
|
|
|
func pointerOf(v addressableValue) pointer {
|
|
|
|
return pointer{unsafe.Pointer(v.Value.Pointer())}
|
2020-06-28 17:58:21 +00:00
|
|
|
}
|
2021-07-21 17:26:04 +00:00
|
|
|
|
|
|
|
// hashType hashes a reflect.Type.
|
|
|
|
// The hash is only consistent within the lifetime of a program.
|
|
|
|
func (h *hasher) hashType(t reflect.Type) {
|
|
|
|
// This approach relies on reflect.Type always being backed by a unique
|
|
|
|
// *reflect.rtype pointer. A safer approach is to use a global sync.Map
|
|
|
|
// that maps reflect.Type to some arbitrary and unique index.
|
|
|
|
// While safer, it requires global state with memory that can never be GC'd.
|
|
|
|
rtypeAddr := reflect.ValueOf(t).Pointer() // address of *reflect.rtype
|
2021-08-03 04:44:13 +00:00
|
|
|
h.hashUint64(uint64(rtypeAddr))
|
2021-07-21 17:26:04 +00:00
|
|
|
}
|