all-kube: create Tailscale Service for HA kube-apiserver ProxyGroup (#16572)

Adds a new reconciler for ProxyGroups of type kube-apiserver that will
provision a Tailscale Service for each replica to advertise. Adds two
new condition types to the ProxyGroup, TailscaleServiceValid and
TailscaleServiceConfigured, to post updates on the state of that
reconciler in a way that's consistent with the service-pg reconciler.
The created Tailscale Service name is configurable via a new ProxyGroup
field spec.kubeAPISserver.ServiceName, which expects a string of the
form "svc:<dns-label>".

Lots of supporting changes were needed to implement this in a way that's
consistent with other operator workflows, including:

* Pulled containerboot's ensureServicesUnadvertised and certManager into
  kube/ libraries to be shared with k8s-proxy. Use those in k8s-proxy to
  aid Service cert sharing between replicas and graceful Service shutdown.
* For certManager, add an initial wait to the cert loop to wait until
  the domain appears in the devices's netmap to avoid a guaranteed error
  on the first issue attempt when it's quick to start.
* Made several methods in ingress-for-pg.go and svc-for-pg.go into
  functions to share with the new reconciler
* Added a Resource struct to the owner refs stored in Tailscale Service
  annotations to be able to distinguish between Ingress- and ProxyGroup-
  based Services that need cleaning up in the Tailscale API.
* Added a ListVIPServices method to the internal tailscale client to aid
  cleaning up orphaned Services
* Support for reading config from a kube Secret, and partial support for
  config reloading, to prevent us having to force Pod restarts when
  config changes.
* Fixed up the zap logger so it's possible to set debug log level.

Updates #13358

Change-Id: Ia9607441157dd91fb9b6ecbc318eecbef446e116
Signed-off-by: Tom Proctor <tomhjp@users.noreply.github.com>

internal/client/tailscale/vip_service.go

@@ -36,6 +36,11 @@ type VIPService struct {
 	Tags []string `json:"tags,omitempty"`
 }
 
+// VIPServiceList represents the JSON response to the list VIP Services API.
+type VIPServiceList struct {
+	VIPServices []VIPService `json:"vipServices"`
+}
+
 // GetVIPService retrieves a VIPService by its name. It returns 404 if the VIPService is not found.
 func (client *Client) GetVIPService(ctx context.Context, name tailcfg.ServiceName) (*VIPService, error) {
 	path := client.BuildTailnetURL("vip-services", name.String())
@@ -59,6 +64,29 @@ func (client *Client) GetVIPService(ctx context.Context, name tailcfg.ServiceNam
 	return svc, nil
 }
 
+// ListVIPServices retrieves all existing Services and returns them as a list.
+func (client *Client) ListVIPServices(ctx context.Context) (*VIPServiceList, error) {
+	path := client.BuildTailnetURL("vip-services")
+	req, err := http.NewRequestWithContext(ctx, httpm.GET, path, nil)
+	if err != nil {
+		return nil, fmt.Errorf("error creating new HTTP request: %w", err)
+	}
+	b, resp, err := SendRequest(client, req)
+	if err != nil {
+		return nil, fmt.Errorf("error making Tailsale API request: %w", err)
+	}
+	// If status code was not successful, return the error.
+	// TODO: Change the check for the StatusCode to include other 2XX success codes.
+	if resp.StatusCode != http.StatusOK {
+		return nil, HandleErrorResponse(b, resp)
+	}
+	result := &VIPServiceList{}
+	if err := json.Unmarshal(b, result); err != nil {
+		return nil, err
+	}
+	return result, nil
+}
+
 // CreateOrUpdateVIPService creates or updates a VIPService by its name. Caller must ensure that, if the
 // VIPService already exists, the VIPService is fetched first to ensure that any auto-allocated IP addresses are not
 // lost during the update. If the VIPService was created without any IP addresses explicitly set (so that they were