TheArchive/tailscale
1
0
Fork 0
mirror of https://github.com/tailscale/tailscale.git synced 2025-07-15 18:18:49 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs/k8s: add some high-level operator architecture diagrams

Browse Source 
This is an experiment to see how useful we will find it to have some
text-based diagrams to document how various components of the operator
work. There are no plans to link to this from elsewhere yet, but
hopefully it will be a useful reference internally.

Updates #cleanup

Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>
This commit is contained in:
Tom Proctor 2024-10-25 21:41:15 +01:00
parent e815ae0ec4
commit 82720d455b
1 changed files with 254 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

254
docs/k8s/operator-architecture.md Normal file
View File

@ -0,0 +1,254 @@
# Operator architecture diagrams
The Tailscale Kubernetes operator has a collection of use-cases that can be
mixed and matched as required. The following diagrams illustrate how the
operator implements each use-case.
In each diagram, the "tailscale" namespace is entirely managed by the operator
once the operator itself has been deployed.
Tailscale devices are highlighted as black nodes. The salient devices for each
use-case are marked as "src" or "dst" to denote which node is a source or a
destination in the context of ACL rules that will apply to network traffic.
## API server proxy
[Documentation][kb-operator-proxy]
The operator runs the API server proxy in-process. If the proxy is running in
"noauth" mode, it forwards HTTP requests unmodified. If the proxy is running in
"auth" mode, it deletes any existing auth headers and adds impersonation
headers to the request before forwarding to the API server.
```mermaid
%%{ init: { 'theme':'neutral' } }%%
flowchart LR
classDef tsnode color:#fff,fill:#000;
classDef pod fill:#fff;
subgraph Key
ts[Tailscale device]:::tsnode
end
subgraph k8s[Kubernetes cluster]
subgraph tailscale-ns[tailscale]
operator["operator (dst)"]:::tsnode
end
subgraph controlplane["Control plane"]
api[kube-apiserver]
end
end
client["client (src)"]:::tsnode --> operator
operator-->|proxies requests| api
```
## L3 ingress
[Documentation][kb-operator-l3-ingress]
The user deploys an app to the default namespace, and creates a normal Service
that selects the app's pods. Add the annotation `tailscale.com/expose: "true"`
to the Service, and the operator will create an ingress proxy that allows
devices anywhere on the tailnet to access the Service.
```mermaid
%%{ init: { 'theme':'neutral' } }%%
flowchart TD
classDef tsnode color:#fff,fill:#000;
classDef pod fill:#fff;
subgraph Key
ts[Tailscale device]:::tsnode
pod(Pod):::pod
end
subgraph k8s[Kubernetes cluster]
subgraph tailscale-ns[tailscale]
operator(operator):::tsnode
ingress("ingress proxy (dst)"):::tsnode
secret
end
subgraph defaultns[default]
svc[annotated svc]
svc --> app1(app)
svc --> app2(app)
end
end
client["client (src)"]:::tsnode --> ingress
ingress -->|forwards traffic| svc
operator -.->|deploys| ingress
operator -.->|reads| svc
operator -.->|creates| secret
secret -.->|mounted| ingress
```
## L7 ingress
[Documentation][kb-operator-l7-ingress]
## L3 egress
[Documentation][kb-operator-l3-egress]
1. The user deploys a Service named `db` with `type: ExternalName` and an annotation
`tailscale.com/tailnet-fqdn: db.tails-scales.ts.net`.
1. The operator creates a proxy Pod managed by a single replica StatefulSet, and a headless Service pointing at the proxy Pod.
1. The operator updates the `db` Service's `spec.externalName` field to point
at the headless Service it created in the previous step.
(Optional) If the user also adds the `tailscale.com/proxy-group: egress-proxies`
annotation to their `db` Service, the operator will skip creating a proxy Pod and
instead point the headless Service at the existing ProxyGroup's pods. In this
case, ports are also required in the `db` Service spec.
Note, in some cases, the config and the state Secret may be the same Kubernetes Secret.
```mermaid
%%{ init: { 'theme':'neutral' } }%%
flowchart TD
classDef tsnode color:#fff,fill:#000;
classDef pod fill:#fff;
subgraph Key
ts[Tailscale device]:::tsnode
pod(Pod):::pod
end
subgraph k8s[Kubernetes cluster]
subgraph tailscale-ns[tailscale]
operator(operator):::tsnode
egress("egress proxy (src)"):::tsnode
headless-svc[headless svc]
cfg-secret["config Secret"]
state-secret["state Secret"]
end
subgraph defaultns[default]
svc[db ExternalName svc]
app1(app) --> svc
app2(app) --> svc
end
end
node["db.tails-scales.ts.net (dst)"]:::tsnode
svc -->|DNS points to| headless-svc
headless-svc -->|forwards traffic| egress
egress -->|forwards traffic| node
operator -.->|deploys| egress
operator -.->|deploys| headless-svc
operator -.->|creates| cfg-secret
operator -.->|watches & updates| svc
cfg-secret -.->|mounted| egress
egress -.->|stores state| state-secret
```
## `ProxyGroup`
[Documentation][kb-operator-l3-egress-proxygroup]
The `ProxyGroup` custom resource manages a collection of proxy Pods that can be
configured to egress traffic out of the cluster via ExternalName Services defined
elsewhere. They will also support ingress in the future. In this diagram, the
`ProxyGroup` is named `pg`, and the operator creates proxy pods, via a StatefulSet
but they don't yet serve any traffic.
`ProxyGroups` currently only support egress (see above).
```mermaid
%%{ init: { 'theme':'neutral' } }%%
flowchart TD
classDef tsnode color:#fff,fill:#000;
classDef pod fill:#fff;
subgraph Key
ts[Tailscale device]:::tsnode
pod(Pod):::pod
end
subgraph k8s[Kubernetes cluster]
subgraph tailscale-ns[tailscale]
operator(operator):::tsnode
pg-sts[pg StatefulSet]
pg-0("pg-0 (src)"):::tsnode
pg-1("pg-1 (src)"):::tsnode
cfg-secret-0["pg-0-config Secret"]
cfg-secret-1["pg-1-config Secret"]
state-secret-0["pg-0 Secret"]
state-secret-1["pg-1 Secret"]
end
subgraph cluster-scope["Cluster scoped resources"]
pg["pg ProxyGroup"]
end
end
operator-.->|watches| pg
operator -.->|deploys| pg-sts
pg-sts -.->|manages| pg-0
pg-sts -.->|manages| pg-1
operator -.->|creates| cfg-secret-0
operator -.->|creates| cfg-secret-1
cfg-secret-0 -.->|mounted| pg-0
cfg-secret-1 -.->|mounted| pg-1
pg-0 -.->|stores state| state-secret-0
pg-1 -.->|stores state| state-secret-1
```
## Subnet routers and exit nodes
[Documentation][kb-operator-connector]
## Recorder nodes
[Documentation][kb-operator-recorder]
The `Recorder` custom resource makes it easier to deploy `tsrecorder` to a cluster.
It currently only supports a single replica.
```mermaid
%%{ init: { 'theme':'neutral' } }%%
flowchart TD
classDef tsnode color:#fff,fill:#000;
classDef pod fill:#fff;
subgraph Key
ts[Tailscale device]:::tsnode
pod(Pod):::pod
end
subgraph k8s[Kubernetes cluster]
subgraph tailscale-ns[tailscale]
operator(operator):::tsnode
rec-sts[rec StatefulSet]
rec-0("rec-0 Pod (tsrecorder)"):::tsnode
cfg-secret-0["rec-0-config Secret"]
state-secret-0["rec-0 Secret"]
end
subgraph cluster-scope["Cluster scoped resources"]
rec["rec Recorder"]
end
end
operator-.->|watches| rec
operator -.->|deploys| rec-sts
rec-sts -.->|manages| rec-0
operator -.->|creates| cfg-secret-0
cfg-secret-0 -.->|mounted| rec-0
rec-0 -.->|stores state| state-secret-0
```
[kb-operator-proxy]: https://tailscale.com/kb/1437/kubernetes-operator-api-server-proxy
[kb-operator-l3-ingress]: https://tailscale.com/kb/1439/kubernetes-operator-cluster-ingress#exposing-a-cluster-workload-using-a-kubernetes-service
[kb-operator-l7-ingress]: https://tailscale.com/kb/1439/kubernetes-operator-cluster-ingress#exposing-cluster-workloads-using-a-kubernetes-ingress
[kb-operator-l3-egress]: https://tailscale.com/kb/1438/kubernetes-operator-cluster-egress
[kb-operator-l3-egress-proxygroup]: TODO
[kb-operator-connector]: https://tailscale.com/kb/1441/kubernetes-operator-connector
[kb-operator-recorder]: TODO