cmd/{k8s-nameserver,k8s-operator},k8s-operator: add a kube nameserver, make operator deploy it (#11919)

* cmd/k8s-nameserver,k8s-operator: add a nameserver that can resolve ts.net DNS names in cluster.

Adds a simple nameserver that can respond to A record queries for ts.net DNS names.
It can respond to queries from in-memory records, populated from a ConfigMap
mounted at /config. It dynamically updates its records as the ConfigMap
contents changes.
It will respond with NXDOMAIN to queries for any other record types
(AAAA to be implemented in the future).
It can respond to queries over UDP or TCP. It runs a miekg/dns
DNS server with a single registered handler for ts.net domain names.
Queries for other domain names will be refused.

The intended use of this is:
1) to allow non-tailnet cluster workloads to talk to HTTPS tailnet
services exposed via Tailscale operator egress over HTTPS
2) to allow non-tailnet cluster workloads to talk to workloads in
the same cluster that have been exposed to tailnet over their
MagicDNS names but on their cluster IPs.

DNSConfig CRD can be used to configure
the operator to deploy kube nameserver (./cmd/k8s-nameserver) to cluster.

Updates tailscale/tailscale#10499

Signed-off-by: Irbe Krumina <irbe@tailscale.com>

k8s-operator/api.md

@@ -10,6 +10,8 @@ Resource Types:
 
 - [Connector](#connector)
 
+- [DNSConfig](#dnsconfig)
+
 - [ProxyClass](#proxyclass)
 
 
@@ -259,6 +261,274 @@ ConnectorCondition contains condition information for a Connector.
       </tr></tbody>
 </table>
 
+## DNSConfig
+<sup><sup>[↩ Parent](#tailscalecomv1alpha1 )</sup></sup>
+
+
+
+
+
+
+
+
+<table>
+    <thead>
+        <tr>
+            <th>Name</th>
+            <th>Type</th>
+            <th>Description</th>
+            <th>Required</th>
+        </tr>
+    </thead>
+    <tbody><tr>
+      <td><b>apiVersion</b></td>
+      <td>string</td>
+      <td>tailscale.com/v1alpha1</td>
+      <td>true</td>
+      </tr>
+      <tr>
+      <td><b>kind</b></td>
+      <td>string</td>
+      <td>DNSConfig</td>
+      <td>true</td>
+      </tr>
+      <tr>
+      <td><b><a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#objectmeta-v1-meta">metadata</a></b></td>
+      <td>object</td>
+      <td>Refer to the Kubernetes API documentation for the fields of the `metadata` field.</td>
+      <td>true</td>
+      </tr><tr>
+        <td><b><a href="#dnsconfigspec">spec</a></b></td>
+        <td>object</td>
+        <td>
+          <br/>
+        </td>
+        <td>true</td>
+      </tr><tr>
+        <td><b><a href="#dnsconfigstatus">status</a></b></td>
+        <td>object</td>
+        <td>
+          <br/>
+        </td>
+        <td>false</td>
+      </tr></tbody>
+</table>
+
+
+### DNSConfig.spec
+<sup><sup>[↩ Parent](#dnsconfig)</sup></sup>
+
+
+
+
+
+<table>
+    <thead>
+        <tr>
+            <th>Name</th>
+            <th>Type</th>
+            <th>Description</th>
+            <th>Required</th>
+        </tr>
+    </thead>
+    <tbody><tr>
+        <td><b><a href="#dnsconfigspecnameserver">nameserver</a></b></td>
+        <td>object</td>
+        <td>
+          <br/>
+        </td>
+        <td>true</td>
+      </tr></tbody>
+</table>
+
+
+### DNSConfig.spec.nameserver
+<sup><sup>[↩ Parent](#dnsconfigspec)</sup></sup>
+
+
+
+
+
+<table>
+    <thead>
+        <tr>
+            <th>Name</th>
+            <th>Type</th>
+            <th>Description</th>
+            <th>Required</th>
+        </tr>
+    </thead>
+    <tbody><tr>
+        <td><b><a href="#dnsconfigspecnameserverimage">image</a></b></td>
+        <td>object</td>
+        <td>
+          <br/>
+        </td>
+        <td>false</td>
+      </tr></tbody>
+</table>
+
+
+### DNSConfig.spec.nameserver.image
+<sup><sup>[↩ Parent](#dnsconfigspecnameserver)</sup></sup>
+
+
+
+
+
+<table>
+    <thead>
+        <tr>
+            <th>Name</th>
+            <th>Type</th>
+            <th>Description</th>
+            <th>Required</th>
+        </tr>
+    </thead>
+    <tbody><tr>
+        <td><b>repo</b></td>
+        <td>string</td>
+        <td>
+          <br/>
+        </td>
+        <td>false</td>
+      </tr><tr>
+        <td><b>tag</b></td>
+        <td>string</td>
+        <td>
+          <br/>
+        </td>
+        <td>false</td>
+      </tr></tbody>
+</table>
+
+
+### DNSConfig.status
+<sup><sup>[↩ Parent](#dnsconfig)</sup></sup>
+
+
+
+
+
+<table>
+    <thead>
+        <tr>
+            <th>Name</th>
+            <th>Type</th>
+            <th>Description</th>
+            <th>Required</th>
+        </tr>
+    </thead>
+    <tbody><tr>
+        <td><b><a href="#dnsconfigstatusconditionsindex">conditions</a></b></td>
+        <td>[]object</td>
+        <td>
+          <br/>
+        </td>
+        <td>false</td>
+      </tr><tr>
+        <td><b><a href="#dnsconfigstatusnameserverstatus">nameserverStatus</a></b></td>
+        <td>object</td>
+        <td>
+          <br/>
+        </td>
+        <td>false</td>
+      </tr></tbody>
+</table>
+
+
+### DNSConfig.status.conditions[index]
+<sup><sup>[↩ Parent](#dnsconfigstatus)</sup></sup>
+
+
+
+ConnectorCondition contains condition information for a Connector.
+
+<table>
+    <thead>
+        <tr>
+            <th>Name</th>
+            <th>Type</th>
+            <th>Description</th>
+            <th>Required</th>
+        </tr>
+    </thead>
+    <tbody><tr>
+        <td><b>status</b></td>
+        <td>string</td>
+        <td>
+          Status of the condition, one of ('True', 'False', 'Unknown').<br/>
+        </td>
+        <td>true</td>
+      </tr><tr>
+        <td><b>type</b></td>
+        <td>string</td>
+        <td>
+          Type of the condition, known values are (`SubnetRouterReady`).<br/>
+        </td>
+        <td>true</td>
+      </tr><tr>
+        <td><b>lastTransitionTime</b></td>
+        <td>string</td>
+        <td>
+          LastTransitionTime is the timestamp corresponding to the last status change of this condition.<br/>
+          <br/>
+            <i>Format</i>: date-time<br/>
+        </td>
+        <td>false</td>
+      </tr><tr>
+        <td><b>message</b></td>
+        <td>string</td>
+        <td>
+          Message is a human readable description of the details of the last transition, complementing reason.<br/>
+        </td>
+        <td>false</td>
+      </tr><tr>
+        <td><b>observedGeneration</b></td>
+        <td>integer</td>
+        <td>
+          If set, this represents the .metadata.generation that the condition was set based upon. For instance, if .metadata.generation is currently 12, but the .status.condition[x].observedGeneration is 9, the condition is out of date with respect to the current state of the Connector.<br/>
+          <br/>
+            <i>Format</i>: int64<br/>
+        </td>
+        <td>false</td>
+      </tr><tr>
+        <td><b>reason</b></td>
+        <td>string</td>
+        <td>
+          Reason is a brief machine readable explanation for the condition's last transition.<br/>
+        </td>
+        <td>false</td>
+      </tr></tbody>
+</table>
+
+
+### DNSConfig.status.nameserverStatus
+<sup><sup>[↩ Parent](#dnsconfigstatus)</sup></sup>
+
+
+
+
+
+<table>
+    <thead>
+        <tr>
+            <th>Name</th>
+            <th>Type</th>
+            <th>Description</th>
+            <th>Required</th>
+        </tr>
+    </thead>
+    <tbody><tr>
+        <td><b>ip</b></td>
+        <td>string</td>
+        <td>
+          <br/>
+        </td>
+        <td>false</td>
+      </tr></tbody>
+</table>
+
 ## ProxyClass
 <sup><sup>[↩ Parent](#tailscalecomv1alpha1 )</sup></sup>
 

k8s-operator/apis/v1alpha1/register.go

@@ -49,7 +49,7 @@ func init() {
 
 // Adds the list of known types to api.Scheme.
 func addKnownTypes(scheme *runtime.Scheme) error {
-	scheme.AddKnownTypes(SchemeGroupVersion, &Connector{}, &ConnectorList{}, &ProxyClass{}, &ProxyClassList{})
+	scheme.AddKnownTypes(SchemeGroupVersion, &Connector{}, &ConnectorList{}, &ProxyClass{}, &ProxyClassList{}, &DNSConfig{}, &DNSConfigList{})
 
 	metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
 	return nil

k8s-operator/apis/v1alpha1/types_tsdnsconfig.go

@@ -0,0 +1,71 @@
+// Copyright (c) Tailscale Inc & AUTHORS
+// SPDX-License-Identifier: BSD-3-Clause
+
+//go:build !plan9
+
+package v1alpha1
+
+import (
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+// Code comments on these types should be treated as user facing documentation-
+// they will appear on the DNSConfig CRD i.e if someone runs kubectl explain dnsconfig.
+
+var DNSConfigKind = "DNSConfig"
+
+// +kubebuilder:object:root=true
+// +kubebuilder:subresource:status
+// +kubebuilder:resource:scope=Cluster,shortName=dc
+// +kubebuilder:printcolumn:name="NameserverIP",type="string",JSONPath=`.status.nameserverStatus.ip`,description="Service IP address of the nameserver"
+
+type DNSConfig struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	Spec DNSConfigSpec `json:"spec"`
+
+	// +optional
+	Status DNSConfigStatus `json:"status"`
+}
+
+// +kubebuilder:object:root=true
+
+type DNSConfigList struct {
+	metav1.TypeMeta `json:",inline"`
+	metav1.ListMeta `json:"metadata"`
+
+	Items []DNSConfig `json:"items"`
+}
+
+type DNSConfigSpec struct {
+	Nameserver *Nameserver `json:"nameserver"`
+}
+
+type Nameserver struct {
+	// +optional
+	Image *Image `json:"image,omitempty"`
+}
+
+type Image struct {
+	// +optional
+	Repo string `json:"repo,omitempty"`
+	// +optional
+	Tag string `json:"tag,omitempty"`
+}
+
+type DNSConfigStatus struct {
+	// +listType=map
+	// +listMapKey=type
+	// +optional
+	Conditions []ConnectorCondition `json:"conditions"`
+	// +optional
+	NameserverStatus *NameserverStatus `json:"nameserverStatus"`
+}
+
+type NameserverStatus struct {
+	// +optional
+	IP string `json:"ip"`
+}
+
+const NameserverReady ConnectorConditionType = `NameserverReady`

k8s-operator/apis/v1alpha1/zz_generated.deepcopy.go

@@ -163,6 +163,112 @@ func (in *Container) DeepCopy() *Container {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *DNSConfig) DeepCopyInto(out *DNSConfig) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
+	in.Spec.DeepCopyInto(&out.Spec)
+	in.Status.DeepCopyInto(&out.Status)
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DNSConfig.
+func (in *DNSConfig) DeepCopy() *DNSConfig {
+	if in == nil {
+		return nil
+	}
+	out := new(DNSConfig)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *DNSConfig) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *DNSConfigList) DeepCopyInto(out *DNSConfigList) {
+	*out = *in
+	out.TypeMeta = in.TypeMeta
+	in.ListMeta.DeepCopyInto(&out.ListMeta)
+	if in.Items != nil {
+		in, out := &in.Items, &out.Items
+		*out = make([]DNSConfig, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DNSConfigList.
+func (in *DNSConfigList) DeepCopy() *DNSConfigList {
+	if in == nil {
+		return nil
+	}
+	out := new(DNSConfigList)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
+func (in *DNSConfigList) DeepCopyObject() runtime.Object {
+	if c := in.DeepCopy(); c != nil {
+		return c
+	}
+	return nil
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *DNSConfigSpec) DeepCopyInto(out *DNSConfigSpec) {
+	*out = *in
+	if in.Nameserver != nil {
+		in, out := &in.Nameserver, &out.Nameserver
+		*out = new(Nameserver)
+		(*in).DeepCopyInto(*out)
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DNSConfigSpec.
+func (in *DNSConfigSpec) DeepCopy() *DNSConfigSpec {
+	if in == nil {
+		return nil
+	}
+	out := new(DNSConfigSpec)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *DNSConfigStatus) DeepCopyInto(out *DNSConfigStatus) {
+	*out = *in
+	if in.Conditions != nil {
+		in, out := &in.Conditions, &out.Conditions
+		*out = make([]ConnectorCondition, len(*in))
+		for i := range *in {
+			(*in)[i].DeepCopyInto(&(*out)[i])
+		}
+	}
+	if in.NameserverStatus != nil {
+		in, out := &in.NameserverStatus, &out.NameserverStatus
+		*out = new(NameserverStatus)
+		**out = **in
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new DNSConfigStatus.
+func (in *DNSConfigStatus) DeepCopy() *DNSConfigStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(DNSConfigStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *Env) DeepCopyInto(out *Env) {
 	*out = *in
@@ -178,6 +284,21 @@ func (in *Env) DeepCopy() *Env {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *Image) DeepCopyInto(out *Image) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Image.
+func (in *Image) DeepCopy() *Image {
+	if in == nil {
+		return nil
+	}
+	out := new(Image)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *Metrics) DeepCopyInto(out *Metrics) {
 	*out = *in
@@ -193,6 +314,41 @@ func (in *Metrics) DeepCopy() *Metrics {
 	return out
 }
 
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *Nameserver) DeepCopyInto(out *Nameserver) {
+	*out = *in
+	if in.Image != nil {
+		in, out := &in.Image, &out.Image
+		*out = new(Image)
+		**out = **in
+	}
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Nameserver.
+func (in *Nameserver) DeepCopy() *Nameserver {
+	if in == nil {
+		return nil
+	}
+	out := new(Nameserver)
+	in.DeepCopyInto(out)
+	return out
+}
+
+// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
+func (in *NameserverStatus) DeepCopyInto(out *NameserverStatus) {
+	*out = *in
+}
+
+// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new NameserverStatus.
+func (in *NameserverStatus) DeepCopy() *NameserverStatus {
+	if in == nil {
+		return nil
+	}
+	out := new(NameserverStatus)
+	in.DeepCopyInto(out)
+	return out
+}
+
 // DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
 func (in *Pod) DeepCopyInto(out *Pod) {
 	*out = *in

k8s-operator/conditions.go

@@ -24,7 +24,7 @@ func SetConnectorCondition(cn *tsapi.Connector, conditionType tsapi.ConnectorCon
 	cn.Status.Conditions = conds
 }
 
-// RemoveConnectorCondition will remove condition of the given type.
+// RemoveConnectorCondition will remove condition of the given type if it exists.
 func RemoveConnectorCondition(conn *tsapi.Connector, conditionType tsapi.ConnectorConditionType) {
 	conn.Status.Conditions = slices.DeleteFunc(conn.Status.Conditions, func(cond tsapi.ConnectorCondition) bool {
 		return cond.Type == conditionType
@@ -39,6 +39,14 @@ func SetProxyClassCondition(pc *tsapi.ProxyClass, conditionType tsapi.ConnectorC
 	pc.Status.Conditions = conds
 }
 
+// SetDNSConfigCondition ensures that DNSConfig status has a condition with the
+// given attributes. LastTransitionTime gets set every time condition's status
+// changes
+func SetDNSConfigCondition(dnsCfg *tsapi.DNSConfig, conditionType tsapi.ConnectorConditionType, status metav1.ConditionStatus, reason, message string, gen int64, clock tstime.Clock, logger *zap.SugaredLogger) {
+	conds := updateCondition(dnsCfg.Status.Conditions, conditionType, status, reason, message, gen, clock, logger)
+	dnsCfg.Status.Conditions = conds
+}
+
 func updateCondition(conds []tsapi.ConnectorCondition, conditionType tsapi.ConnectorConditionType, status metav1.ConditionStatus, reason, message string, gen int64, clock tstime.Clock, logger *zap.SugaredLogger) []tsapi.ConnectorCondition {
 	newCondition := tsapi.ConnectorCondition{
 		Type:               conditionType,
@@ -61,8 +69,9 @@ func updateCondition(conds []tsapi.ConnectorCondition, conditionType tsapi.Conne
 	}
 
 	cond := conds[idx] // update the existing condition
-	// If this update doesn't contain a state transition, we don't update
-	// the conditions LastTransitionTime to Now().
+
+	// If this update doesn't contain a state transition, don't update last
+	// transition time.
 	if cond.Status == status {
 		newCondition.LastTransitionTime = cond.LastTransitionTime
 	} else {
@@ -82,3 +91,14 @@ func ProxyClassIsReady(pc *tsapi.ProxyClass) bool {
 	cond := pc.Status.Conditions[idx]
 	return cond.Status == metav1.ConditionTrue && cond.ObservedGeneration == pc.Generation
 }
+
+func DNSCfgIsReady(cfg *tsapi.DNSConfig) bool {
+	idx := xslices.IndexFunc(cfg.Status.Conditions, func(cond tsapi.ConnectorCondition) bool {
+		return cond.Type == tsapi.NameserverReady
+	})
+	if idx == -1 {
+		return false
+	}
+	cond := cfg.Status.Conditions[idx]
+	return cond.Status == metav1.ConditionTrue && cond.ObservedGeneration == cfg.Generation
+}

k8s-operator/conditions_test.go

@@ -98,5 +98,4 @@ func TestSetConnectorCondition(t *testing.T) {
 			},
 		},
 	})
-
 }

k8s-operator/tsdns.go

@@ -0,0 +1,17 @@
+// Copyright (c) Tailscale Inc & AUTHORS
+// SPDX-License-Identifier: BSD-3-Clause
+
+//go:build !plan9
+
+package kube
+
+const Alpha1Version = "v1alpha1"
+
+type Records struct {
+	// Version is the version of this Records configuration. Version is
+	// intended to be used by ./cmd/k8s-nameserver to determine whether it
+	// can read this records configuration.
+	Version string `json:"version"`
+	// IP4 contains a mapping of DNS names to IPv4 address(es).
+	IP4 map[string][]string `json:"ip4"`
+}