update changelog for 0.23 release (#2138)

Signed-off-by: Kristoffer Dalby <kristoffer@tailscale.com>

.github/workflows/test-integration.yaml

@@ -52,6 +52,7 @@ jobs:
           - TestExpireNode
           - TestNodeOnlineStatus
           - TestPingAllByIPManyUpDown
+          - Test2118DeletingOnlineNodePanics
           - TestEnablingRoutes
           - TestHASubnetRouterFailover
           - TestEnableDisableAutoApprovedRoute

CHANGELOG.md

@@ -1,8 +1,9 @@
 # CHANGELOG
 
-## 0.23.0 (2023-XX-XX)
+## 0.23.0 (2023-09-18)
 
-This release is mainly a code reorganisation and refactoring, significantly improving the maintainability of the codebase. This should allow us to improve further and make it easier for the maintainers to keep on top of the project.
+This release was intended to be mainly a code reorganisation and refactoring, significantly improving the maintainability of the codebase. This should allow us to improve further and make it easier for the maintainers to keep on top of the project.
+However, as you all have noticed, it turned out to become a much larger, much longer release cycle than anticipated. It has ended up to be a release with a lot of rewrites and changes to the code base and functionality of Headscale, cleaning up a lot of technical debt and introducing a lot of improvements. This does come with some breaking changes,
 
 **Please remember to always back up your database between versions**
 
@@ -16,7 +17,7 @@ The [“poller”, or streaming logic](https://github.com/juanfont/headscale/blo
 
 Headscale now supports sending “delta” updates, thanks to the new mapper and poller logic, allowing us to only inform nodes about new nodes, changed nodes and removed nodes. Previously we sent the entire state of the network every time an update was due.
 
-While we have a pretty good [test harness](https://github.com/search?q=repo%3Ajuanfont%2Fheadscale+path%3A_test.go&type=code) for validating our changes, we have rewritten over [10000 lines of code](https://github.com/juanfont/headscale/compare/b01f1f1867136d9b2d7b1392776eb363b482c525...main) and bugs are expected. We need help testing this release. In addition, while we think the performance should in general be better, there might be regressions in parts of the platform, particularly where we prioritised correctness over speed.
+While we have a pretty good [test harness](https://github.com/search?q=repo%3Ajuanfont%2Fheadscale+path%3A_test.go&type=code) for validating our changes, the changes came down to [284 changed files with 32,316 additions and 24,245 deletions](https://github.com/juanfont/headscale/compare/b01f1f1867136d9b2d7b1392776eb363b482c525...ed78ecd) and bugs are expected. We need help testing this release. In addition, while we think the performance should in general be better, there might be regressions in parts of the platform, particularly where we prioritised correctness over speed.
 
 There are also several bugfixes that has been encountered and fixed as part of implementing these changes, particularly
 after improving the test harness as part of adopting [#1460](https://github.com/juanfont/headscale/pull/1460).

README.md

@@ -62,15 +62,15 @@ buttons available in the repo.
 
 ## Client OS support
 
-| OS      | Supports headscale                                        |
-| ------- | --------------------------------------------------------- |
-| Linux   | Yes                                                       |
-| OpenBSD | Yes                                                       |
-| FreeBSD | Yes                                                       |
-| macOS   | Yes (see `/apple` on your headscale for more information) |
-| Windows | Yes [docs](./docs/windows-client.md)                      |
-| Android | Yes [docs](./docs/android-client.md)                      |
-| iOS     | Yes [docs](./docs/iOS-client.md)                          |
+| OS      | Supports headscale                                                                                 |
+| ------- | -------------------------------------------------------------------------------------------------- |
+| Linux   | Yes                                                                                                |
+| OpenBSD | Yes                                                                                                |
+| FreeBSD | Yes                                                                                                |
+| Windows | Yes (see [docs](./docs/windows-client.md) and `/windows` on your headscale for more information)   |
+| Android | Yes (see [docs](./docs/android-client.md))                                                         |
+| macOS   | Yes (see [docs](./docs/apple-client.md#macos) and `/apple` on your headscale for more information) |
+| iOS     | Yes (see [docs](./docs/apple-client.md#ios) and `/apple` on your headscale for more information)   |
 
 ## Running headscale
 

docs/apple-client.md

@@ -0,0 +1,51 @@
+# Connecting an Apple client
+
+## Goal
+
+This documentation has the goal of showing how a user can use the official iOS and macOS [Tailscale](https://tailscale.com) clients with `headscale`.
+
+!!! info "Instructions on your headscale instance"
+
+    An endpoint with information on how to connect your Apple device
+    is also available at `/apple` on your running instance.
+
+## iOS
+
+### Installation
+
+Install the official Tailscale iOS client from the [App Store](https://apps.apple.com/app/tailscale/id1470499037).
+
+### Configuring the headscale URL
+
+- Open Tailscale and make sure you are _not_ logged in to any account
+- Open Settings on the iOS device
+- Scroll down to the `third party apps` section, under `Game Center` or `TV Provider`
+- Find Tailscale and select it
+  - If the iOS device was previously logged into Tailscale, switch the `Reset Keychain` toggle to `on`
+- Enter the URL of your headscale instance (e.g `https://headscale.example.com`) under `Alternate Coordination Server URL`
+- Restart the app by closing it from the iOS app switcher, open the app and select the regular sign in option
+  _(non-SSO)_. It should open up to the headscale authentication page.
+- Enter your credentials and log in. Headscale should now be working on your iOS device.
+
+## macOS
+
+### Installation
+
+Choose one of the available [Tailscale clients for macOS](https://tailscale.com/kb/1065/macos-variants) and install it.
+
+### Configuring the headscale URL
+
+#### Command line
+
+Use Tailscale's login command to connect with your headscale instance (e.g `https://headscale.example.com`):
+
+```
+tailscale login --login-server <YOUR_HEADSCALE_URL>
+```
+
+#### GUI
+
+- ALT + Click the Tailscale icon in the menu and hover over the Debug menu
+- Under `Custom Login Server`, select `Add Account...`
+- Enter the URL of your headscale instance (e.g `https://headscale.example.com`) and press `Add Account`
+- Follow the login procedure in the browser

docs/exit-node.md

@@ -5,7 +5,7 @@
 Register the node and make it advertise itself as an exit node:
 
 ```console
-$ sudo tailscale up --login-server https://my-server.com --advertise-exit-node
+$ sudo tailscale up --login-server https://headscale.example.com --advertise-exit-node
 ```
 
 If the node is already registered, it can advertise exit capabilities like this:

docs/iOS-client.md

@@ -1,30 +0,0 @@
-# Connecting an iOS client
-
-## Goal
-
-This documentation has the goal of showing how a user can use the official iOS [Tailscale](https://tailscale.com) client with `headscale`.
-
-## Installation
-
-Install the official Tailscale iOS client from the [App Store](https://apps.apple.com/app/tailscale/id1470499037).
-
-Ensure that the installed version is at least 1.38.1, as that is the first release to support alternate control servers.
-
-## Configuring the headscale URL
-
-!!! info "Apple devices"
-
-    An endpoint with information on how to connect your Apple devices
-    (currently macOS only) is available at `/apple` on your running instance.
-
-Ensure that the tailscale app is logged out before proceeding.
-
-Go to iOS settings, scroll down past game center and tv provider to the tailscale app and select it. The headscale URL can be entered into the _"ALTERNATE COORDINATION SERVER URL"_ box.
-
-> **Note**
->
-> If the app was previously logged into tailscale, toggle on the _Reset Keychain_ switch.
-
-Restart the app by closing it from the iOS app switcher, open the app and select the regular _Sign in_ option (non-SSO), and it should open up to the headscale authentication page.
-
-Enter your credentials and log in. Headscale should now be working on your iOS device.

docs/windows-client.md

@@ -4,17 +4,17 @@
 
 This documentation has the goal of showing how a user can use the official Windows [Tailscale](https://tailscale.com) client with `headscale`.
 
+!!! info "Instructions on your headscale instance"
+
+    An endpoint with information on how to connect your Windows device
+    is also available at `/windows` on your running instance.
+
 ## Installation
 
 Download the [Official Windows Client](https://tailscale.com/download/windows) and install it.
 
 ## Configuring the headscale URL
 
-!!! info "Instructions on your headscale instance"
-
-    An endpoint with information on how to connect your Windows device
-    is also available at `/windows` on your running instance.
-
 Open a Command Prompt or Powershell and use Tailscale's login command to connect with your headscale instance (e.g
 `https://headscale.example.com`):
 

flake.lock

@@ -20,11 +20,11 @@
     },
     "nixpkgs": {
       "locked": {
-        "lastModified": 1725534445,
-        "narHash": "sha256-Yd0FK9SkWy+ZPuNqUgmVPXokxDgMJoGuNpMEtkfcf84=",
+        "lastModified": 1726238386,
+        "narHash": "sha256-3//V84fYaGVncFImitM6lSAliRdrGayZLdxWlpcuGk0=",
         "owner": "NixOS",
         "repo": "nixpkgs",
-        "rev": "9bb1e7571aadf31ddb4af77fc64b2d59580f9a39",
+        "rev": "01f064c99c792715054dc7a70e4c1626dbbec0c3",
         "type": "github"
       },
       "original": {

hscontrol/auth.go

@@ -66,7 +66,7 @@ func (h *Headscale) handleRegister(
 	regReq tailcfg.RegisterRequest,
 	machineKey key.MachinePublic,
 ) {
-	logInfo, logTrace, logErr := logAuthFunc(regReq, machineKey)
+	logInfo, logTrace, _ := logAuthFunc(regReq, machineKey)
 	now := time.Now().UTC()
 	logTrace("handleRegister called, looking up machine in DB")
 	node, err := h.db.GetNodeByAnyKey(machineKey, regReq.NodeKey, regReq.OldNodeKey)
@@ -105,16 +105,6 @@ func (h *Headscale) handleRegister(
 
 		logInfo("Node not found in database, creating new")
 
-		givenName, err := h.db.GenerateGivenName(
-			machineKey,
-			regReq.Hostinfo.Hostname,
-		)
-		if err != nil {
-			logErr(err, "Failed to generate given name for node")
-
-			return
-		}
-
 		// The node did not have a key to authenticate, which means
 		// that we rely on a method that calls back some how (OpenID or CLI)
 		// We create the node and then keep it around until a callback
@@ -122,7 +112,6 @@ func (h *Headscale) handleRegister(
 		newNode := types.Node{
 			MachineKey: machineKey,
 			Hostname:   regReq.Hostinfo.Hostname,
-			GivenName:  givenName,
 			NodeKey:    regReq.NodeKey,
 			LastSeen:   &now,
 			Expiry:     &time.Time{},
@@ -354,21 +343,8 @@ func (h *Headscale) handleAuthKey(
 	} else {
 		now := time.Now().UTC()
 
-		givenName, err := h.db.GenerateGivenName(machineKey, registerRequest.Hostinfo.Hostname)
-		if err != nil {
-			log.Error().
-				Caller().
-				Str("func", "RegistrationHandler").
-				Str("hostinfo.name", registerRequest.Hostinfo.Hostname).
-				Err(err).
-				Msg("Failed to generate given name for node")
-
-			return
-		}
-
 		nodeToRegister := types.Node{
 			Hostname:       registerRequest.Hostinfo.Hostname,
-			GivenName:      givenName,
 			UserID:         pak.User.ID,
 			User:           pak.User,
 			MachineKey:     machineKey,

hscontrol/db/node.go

@@ -90,20 +90,6 @@ func (hsdb *HSDatabase) ListEphemeralNodes() (types.Nodes, error) {
 	})
 }
 
-func listNodesByGivenName(tx *gorm.DB, givenName string) (types.Nodes, error) {
-	nodes := types.Nodes{}
-	if err := tx.
-		Preload("AuthKey").
-		Preload("AuthKey.User").
-		Preload("User").
-		Preload("Routes").
-		Where("given_name = ?", givenName).Find(&nodes).Error; err != nil {
-		return nil, err
-	}
-
-	return nodes, nil
-}
-
 func (hsdb *HSDatabase) getNode(user string, name string) (*types.Node, error) {
 	return Read(hsdb.DB, func(rx *gorm.DB) (*types.Node, error) {
 		return getNode(rx, user, name)
@@ -242,9 +228,9 @@ func SetTags(
 }
 
 // RenameNode takes a Node struct and a new GivenName for the nodes
-// and renames it.
+// and renames it. If the name is not unique, it will return an error.
 func RenameNode(tx *gorm.DB,
-	nodeID uint64, newName string,
+	nodeID types.NodeID, newName string,
 ) error {
 	err := util.CheckForFQDNRules(
 		newName,
@@ -253,6 +239,15 @@ func RenameNode(tx *gorm.DB,
 		return fmt.Errorf("renaming node: %w", err)
 	}
 
+	uniq, err := isUnqiueName(tx, newName)
+	if err != nil {
+		return fmt.Errorf("checking if name is unique: %w", err)
+	}
+
+	if !uniq {
+		return fmt.Errorf("name is not unique: %s", newName)
+	}
+
 	if err := tx.Model(&types.Node{}).Where("id = ?", nodeID).Update("given_name", newName).Error; err != nil {
 		return fmt.Errorf("failed to rename node in the database: %w", err)
 	}
@@ -415,6 +410,15 @@ func RegisterNode(tx *gorm.DB, node types.Node, ipv4 *netip.Addr, ipv6 *netip.Ad
 	node.IPv4 = ipv4
 	node.IPv6 = ipv6
 
+	if node.GivenName == "" {
+		givenName, err := ensureUniqueGivenName(tx, node.Hostname)
+		if err != nil {
+			return nil, fmt.Errorf("failed to ensure unique given name: %w", err)
+		}
+
+		node.GivenName = givenName
+	}
+
 	if err := tx.Save(&node).Error; err != nil {
 		return nil, fmt.Errorf("failed register(save) node in the database: %w", err)
 	}
@@ -642,40 +646,32 @@ func generateGivenName(suppliedName string, randomSuffix bool) (string, error) {
 	return normalizedHostname, nil
 }
 
-func (hsdb *HSDatabase) GenerateGivenName(
-	mkey key.MachinePublic,
-	suppliedName string,
-) (string, error) {
-	return Read(hsdb.DB, func(rx *gorm.DB) (string, error) {
-		return GenerateGivenName(rx, mkey, suppliedName)
-	})
+func isUnqiueName(tx *gorm.DB, name string) (bool, error) {
+	nodes := types.Nodes{}
+	if err := tx.
+		Where("given_name = ?", name).Find(&nodes).Error; err != nil {
+		return false, err
+	}
+
+	return len(nodes) == 0, nil
 }
 
-func GenerateGivenName(
+func ensureUniqueGivenName(
 	tx *gorm.DB,
-	mkey key.MachinePublic,
-	suppliedName string,
+	name string,
 ) (string, error) {
-	givenName, err := generateGivenName(suppliedName, false)
+	givenName, err := generateGivenName(name, false)
 	if err != nil {
 		return "", err
 	}
 
-	// Tailscale rules (may differ) https://tailscale.com/kb/1098/machine-names/
-	nodes, err := listNodesByGivenName(tx, givenName)
+	unique, err := isUnqiueName(tx, givenName)
 	if err != nil {
 		return "", err
 	}
 
-	var nodeFound *types.Node
-	for idx, node := range nodes {
-		if node.GivenName == givenName {
-			nodeFound = nodes[idx]
-		}
-	}
-
-	if nodeFound != nil && nodeFound.MachineKey.String() != mkey.String() {
-		postfixedName, err := generateGivenName(suppliedName, true)
+	if !unique {
+		postfixedName, err := generateGivenName(name, true)
 		if err != nil {
 			return "", err
 		}

hscontrol/db/node_test.go

@@ -19,6 +19,7 @@ import (
 	"github.com/puzpuzpuz/xsync/v3"
 	"github.com/stretchr/testify/assert"
 	"gopkg.in/check.v1"
+	"gorm.io/gorm"
 	"tailscale.com/tailcfg"
 	"tailscale.com/types/key"
 	"tailscale.com/types/ptr"
@@ -313,51 +314,6 @@ func (s *Suite) TestExpireNode(c *check.C) {
 	c.Assert(nodeFromDB.IsExpired(), check.Equals, true)
 }
 
-func (s *Suite) TestGenerateGivenName(c *check.C) {
-	user1, err := db.CreateUser("user-1")
-	c.Assert(err, check.IsNil)
-
-	pak, err := db.CreatePreAuthKey(user1.Name, false, false, nil, nil)
-	c.Assert(err, check.IsNil)
-
-	_, err = db.getNode("user-1", "testnode")
-	c.Assert(err, check.NotNil)
-
-	nodeKey := key.NewNode()
-	machineKey := key.NewMachine()
-
-	machineKey2 := key.NewMachine()
-
-	node := &types.Node{
-		ID:             0,
-		MachineKey:     machineKey.Public(),
-		NodeKey:        nodeKey.Public(),
-		Hostname:       "hostname-1",
-		GivenName:      "hostname-1",
-		UserID:         user1.ID,
-		RegisterMethod: util.RegisterMethodAuthKey,
-		AuthKeyID:      ptr.To(pak.ID),
-	}
-
-	trx := db.DB.Save(node)
-	c.Assert(trx.Error, check.IsNil)
-
-	givenName, err := db.GenerateGivenName(machineKey2.Public(), "hostname-2")
-	comment := check.Commentf("Same user, unique nodes, unique hostnames, no conflict")
-	c.Assert(err, check.IsNil, comment)
-	c.Assert(givenName, check.Equals, "hostname-2", comment)
-
-	givenName, err = db.GenerateGivenName(machineKey.Public(), "hostname-1")
-	comment = check.Commentf("Same user, same node, same hostname, no conflict")
-	c.Assert(err, check.IsNil, comment)
-	c.Assert(givenName, check.Equals, "hostname-1", comment)
-
-	givenName, err = db.GenerateGivenName(machineKey2.Public(), "hostname-1")
-	comment = check.Commentf("Same user, unique nodes, same hostname, conflict")
-	c.Assert(err, check.IsNil, comment)
-	c.Assert(givenName, check.Matches, fmt.Sprintf("^hostname-1-[a-z0-9]{%d}$", NodeGivenNameHashLength), comment)
-}
-
 func (s *Suite) TestSetTags(c *check.C) {
 	user, err := db.CreateUser("test")
 	c.Assert(err, check.IsNil)
@@ -778,3 +734,100 @@ func TestListEphemeralNodes(t *testing.T) {
 	assert.Equal(t, nodeEph.UserID, ephemeralNodes[0].UserID)
 	assert.Equal(t, nodeEph.Hostname, ephemeralNodes[0].Hostname)
 }
+
+func TestRenameNode(t *testing.T) {
+	db, err := newTestDB()
+	if err != nil {
+		t.Fatalf("creating db: %s", err)
+	}
+
+	user, err := db.CreateUser("test")
+	assert.NoError(t, err)
+
+	user2, err := db.CreateUser("test2")
+	assert.NoError(t, err)
+
+	node := types.Node{
+		ID:             0,
+		MachineKey:     key.NewMachine().Public(),
+		NodeKey:        key.NewNode().Public(),
+		Hostname:       "test",
+		UserID:         user.ID,
+		RegisterMethod: util.RegisterMethodAuthKey,
+	}
+
+	node2 := types.Node{
+		ID:             0,
+		MachineKey:     key.NewMachine().Public(),
+		NodeKey:        key.NewNode().Public(),
+		Hostname:       "test",
+		UserID:         user2.ID,
+		RegisterMethod: util.RegisterMethodAuthKey,
+	}
+
+	err = db.DB.Save(&node).Error
+	assert.NoError(t, err)
+
+	err = db.DB.Save(&node2).Error
+	assert.NoError(t, err)
+
+	err = db.DB.Transaction(func(tx *gorm.DB) error {
+		_, err := RegisterNode(tx, node, nil, nil)
+		if err != nil {
+			return err
+		}
+		_, err = RegisterNode(tx, node2, nil, nil)
+		return err
+	})
+	assert.NoError(t, err)
+
+	nodes, err := db.ListNodes()
+	assert.NoError(t, err)
+
+	assert.Len(t, nodes, 2)
+
+	t.Logf("node1 %s %s", nodes[0].Hostname, nodes[0].GivenName)
+	t.Logf("node2 %s %s", nodes[1].Hostname, nodes[1].GivenName)
+
+	assert.Equal(t, nodes[0].Hostname, nodes[0].GivenName)
+	assert.NotEqual(t, nodes[1].Hostname, nodes[1].GivenName)
+	assert.Equal(t, nodes[0].Hostname, nodes[1].Hostname)
+	assert.NotEqual(t, nodes[0].Hostname, nodes[1].GivenName)
+	assert.Contains(t, nodes[1].GivenName, nodes[0].Hostname)
+	assert.Equal(t, nodes[0].GivenName, nodes[1].Hostname)
+	assert.Len(t, nodes[0].Hostname, 4)
+	assert.Len(t, nodes[1].Hostname, 4)
+	assert.Len(t, nodes[0].GivenName, 4)
+	assert.Len(t, nodes[1].GivenName, 13)
+
+	// Nodes can be renamed to a unique name
+	err = db.Write(func(tx *gorm.DB) error {
+		return RenameNode(tx, nodes[0].ID, "newname")
+	})
+	assert.NoError(t, err)
+
+	nodes, err = db.ListNodes()
+	assert.NoError(t, err)
+	assert.Len(t, nodes, 2)
+	assert.Equal(t, nodes[0].Hostname, "test")
+	assert.Equal(t, nodes[0].GivenName, "newname")
+
+	// Nodes can reuse name that is no longer used
+	err = db.Write(func(tx *gorm.DB) error {
+		return RenameNode(tx, nodes[1].ID, "test")
+	})
+	assert.NoError(t, err)
+
+	nodes, err = db.ListNodes()
+	assert.NoError(t, err)
+	assert.Len(t, nodes, 2)
+	assert.Equal(t, nodes[0].Hostname, "test")
+	assert.Equal(t, nodes[0].GivenName, "newname")
+	assert.Equal(t, nodes[1].GivenName, "test")
+
+	// Nodes cannot be renamed to used names
+	err = db.Write(func(tx *gorm.DB) error {
+		return RenameNode(tx, nodes[0].ID, "test")
+	})
+	assert.ErrorContains(t, err, "name is not unique")
+}

hscontrol/grpcv1.go

@@ -373,7 +373,7 @@ func (api headscaleV1APIServer) RenameNode(
 	node, err := db.Write(api.h.db.DB, func(tx *gorm.DB) (*types.Node, error) {
 		err := db.RenameNode(
 			tx,
-			request.GetNodeId(),
+			types.NodeID(request.GetNodeId()),
 			request.GetNewName(),
 		)
 		if err != nil {
@@ -802,18 +802,12 @@ func (api headscaleV1APIServer) DebugCreateNode(
 		return nil, err
 	}
 
-	givenName, err := api.h.db.GenerateGivenName(mkey, request.GetName())
-	if err != nil {
-		return nil, err
-	}
-
 	nodeKey := key.NewNode()
 
 	newNode := types.Node{
 		MachineKey: mkey,
 		NodeKey:    nodeKey.Public(),
 		Hostname:   request.GetName(),
-		GivenName:  givenName,
 		User:       *user,
 
 		Expiry:   &time.Time{},

hscontrol/poll.go

@@ -5,6 +5,7 @@ import (
 	"fmt"
 	"math/rand/v2"
 	"net/http"
+	"slices"
 	"sort"
 	"strings"
 	"time"
@@ -273,6 +274,12 @@ func (m *mapSession) serveLongPoll() {
 				return
 			}
 
+			// If the node has been removed from headscale, close the stream
+			if slices.Contains(update.Removed, m.node.ID) {
+				m.tracef("node removed, closing stream")
+				return
+			}
+
 			m.tracef("received stream update: %s %s", update.Type.String(), update.Message)
 			mapResponseUpdateReceived.WithLabelValues(update.Type.String()).Inc()
 

hscontrol/templates/apple.html

@@ -25,17 +25,48 @@
   </head>
 
   <body>
+    <h1>headscale: iOS configuration</h1>
+    <h2>GUI</h2>
+    <ol>
+      <li>
+        Install the official Tailscale iOS client from the
+        <a href="https://apps.apple.com/app/tailscale/id1470499037"
+          >App store</a
+        >
+      </li>
+      <li>
+        Open Tailscale and make sure you are <i>not</i> logged in to any account
+      </li>
+      <li>Open Settings on the iOS device</li>
+      <li>
+        Scroll down to the "third party apps" section, under "Game Center" or
+        "TV Provider"
+      </li>
+      <li>
+        Find Tailscale and select it
+        <ul>
+          <li>
+            If the iOS device was previously logged into Tailscale, switch the
+            "Reset Keychain" toggle to "on"
+          </li>
+        </ul>
+      </li>
+      <li>Enter "{{.URL}}" under "Alternate Coordination Server URL"</li>
+      <li>
+        Restart the app by closing it from the iOS app switcher, open the app
+        and select the regular sign in option <i>(non-SSO)</i>. It should open
+        up to the headscale authentication page.
+      </li>
+      <li>
+        Enter your credentials and log in. Headscale should now be working on
+        your iOS device
+      </li>
+    </ol>
     <h1>headscale: macOS configuration</h1>
-    <h2>Recent Tailscale versions (1.34.0 and higher)</h2>
-    <p>
-      Tailscale added Fast User Switching in version 1.34 and you can now use
-      the new login command to connect to one or more headscale (and Tailscale)
-      servers. The previously used profiles does not have an effect anymore.
-    </p>
-    <h3>Command line</h3>
+    <h2>Command line</h2>
     <p>Use Tailscale's login command to add your profile:</p>
     <pre><code>tailscale login --login-server {{.URL}}</code></pre>
-    <h3>GUI</h3>
+    <h2>GUI</h2>
     <ol>
       <li>
         ALT + Click the Tailscale icon in the menu and hover over the Debug menu
@@ -46,44 +77,7 @@
       </li>
       <li>Follow the login procedure in the browser</li>
     </ol>
-    <h2>Apple configuration profiles (1.32.0 and lower)</h2>
-    <p>
-      This page provides
-      <a href="https://support.apple.com/guide/mdm/mdm-overview-mdmbf9e668/web"
-        >configuration profiles</a
-      >
-      for the official Tailscale clients for
-    </p>
-    <ul>
-      <li>
-        <a href="https://apps.apple.com/app/tailscale/id1475387142"
-          >macOS - AppStore Client</a
-        >.
-      </li>
-      <li>
-        <a href="https://pkgs.tailscale.com/stable/#macos"
-          >macOS - Standalone Client</a
-        >.
-      </li>
-    </ul>
-    <p>
-      The profiles will configure Tailscale.app to use <code>{{.URL}}</code> as
-      its control server.
-    </p>
-    <h3>Caution</h3>
-    <p>
-      You should always download and inspect the profile before installing it:
-    </p>
-    <ul>
-      <li>
-        for app store client: <code>curl {{.URL}}/apple/macos-app-store</code>
-      </li>
-      <li>
-        for standalone client: <code>curl {{.URL}}/apple/macos-standalone</code>
-      </li>
-    </ul>
     <h2>Profiles</h2>
-    <h3>macOS</h3>
     <p>
       Headscale can be set to the default server by installing a Headscale
       configuration profile:
@@ -121,50 +115,17 @@
       </li>
     </ul>
     <p>Restart Tailscale.app and log in.</p>
-    <h1>headscale: iOS configuration</h1>
-    <h2>Recent Tailscale versions (1.38.1 and higher)</h2>
+    <h3>Caution</h3>
     <p>
-      Tailscale 1.38.1 on
-      <a href="https://apps.apple.com/app/tailscale/id1470499037">iOS</a>
-      added a configuration option to allow user to set an "Alternate
-      Coordination server". This can be used to connect to your headscale
-      server.
+      You should always download and inspect the profile before installing it:
     </p>
-    <h3>GUI</h3>
-    <ol>
+    <ul>
       <li>
-        Install the official Tailscale iOS client from the
-        <a href="https://apps.apple.com/app/tailscale/id1470499037"
-          >App store</a
-        >
+        for app store client: <code>curl {{.URL}}/apple/macos-app-store</code>
       </li>
       <li>
-        Open Tailscale and make sure you are <i>not</i> logged in to any account
+        for standalone client: <code>curl {{.URL}}/apple/macos-standalone</code>
       </li>
-      <li>Open Settings on the iOS device</li>
-      <li>
-        Scroll down to the "third party apps" section, under "Game Center" or
-        "TV Provider"
-      </li>
-      <li>
-        Find Tailscale and select it
-        <ul>
-          <li>
-            If the iOS device was previously logged into Tailscale, switch the
-            "Reset Keychain" toggle to "on"
-          </li>
-        </ul>
-      </li>
-      <li>Enter "{{.URL}}" under "Alternate Coordination Server URL"</li>
-      <li>
-        Restart the app by closing it from the iOS app switcher, open the app
-        and select the regular sign in option <i>(non-SSO)</i>. It should open
-        up to the headscale authentication page.
-      </li>
-      <li>
-        Enter your credentials and log in. Headscale should now be working on
-        your iOS device
-      </li>
-    </ol>
+    </ul>
   </body>
 </html>

hscontrol/types/config.go

@@ -732,6 +732,9 @@ func prefixV6() (*netip.Prefix, error) {
 // LoadCLIConfig returns the needed configuration for the CLI client
 // of Headscale to connect to a Headscale server.
 func LoadCLIConfig() (*Config, error) {
+	logConfig := logConfig()
+	zerolog.SetGlobalLevel(logConfig.Level)
+
 	return &Config{
 		DisableUpdateCheck: viper.GetBool("disable_check_updates"),
 		UnixSocket:         viper.GetString("unix_socket"),
@@ -741,6 +744,7 @@ func LoadCLIConfig() (*Config, error) {
 			Timeout:  viper.GetDuration("cli.timeout"),
 			Insecure: viper.GetBool("cli.insecure"),
 		},
+		Log: logConfig,
 	}, nil
 }
 

integration/acl_test.go

@@ -276,7 +276,7 @@ func TestACLHostsInNetMapTable(t *testing.T) {
 				hsic.WithACLPolicy(&testCase.policy),
 			)
 			assertNoErr(t, err)
-			defer scenario.Shutdown()
+			defer scenario.ShutdownAssertNoPanics(t)
 
 			allClients, err := scenario.ListTailscaleClients()
 			assertNoErr(t, err)
@@ -316,7 +316,7 @@ func TestACLAllowUser80Dst(t *testing.T) {
 		},
 		1,
 	)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	user1Clients, err := scenario.ListTailscaleClients("user1")
 	assertNoErr(t, err)
@@ -373,7 +373,7 @@ func TestACLDenyAllPort80(t *testing.T) {
 		},
 		4,
 	)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	allClients, err := scenario.ListTailscaleClients()
 	assertNoErr(t, err)
@@ -417,7 +417,7 @@ func TestACLAllowUserDst(t *testing.T) {
 		},
 		2,
 	)
-	// defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	user1Clients, err := scenario.ListTailscaleClients("user1")
 	assertNoErr(t, err)
@@ -473,7 +473,7 @@ func TestACLAllowStarDst(t *testing.T) {
 		},
 		2,
 	)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	user1Clients, err := scenario.ListTailscaleClients("user1")
 	assertNoErr(t, err)
@@ -534,7 +534,7 @@ func TestACLNamedHostsCanReachBySubnet(t *testing.T) {
 		},
 		3,
 	)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	user1Clients, err := scenario.ListTailscaleClients("user1")
 	assertNoErr(t, err)
@@ -672,7 +672,7 @@ func TestACLNamedHostsCanReach(t *testing.T) {
 				&testCase.policy,
 				2,
 			)
-			defer scenario.Shutdown()
+			defer scenario.ShutdownAssertNoPanics(t)
 
 			// Since user/users dont matter here, we basically expect that some clients
 			// will be assigned these ips and that we can pick them up for our own use.
@@ -1021,7 +1021,7 @@ func TestPolicyUpdateWhileRunningWithCLIInDatabase(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": 1,

integration/auth_oidc_test.go

@@ -48,7 +48,7 @@ func TestOIDCAuthenticationPingAll(t *testing.T) {
 	scenario := AuthOIDCScenario{
 		Scenario: baseScenario,
 	}
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),
@@ -108,7 +108,7 @@ func TestOIDCExpireNodesBasedOnTokenExpiry(t *testing.T) {
 	scenario := AuthOIDCScenario{
 		Scenario: baseScenario,
 	}
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": 3,

integration/auth_web_flow_test.go

@@ -34,7 +34,7 @@ func TestAuthWebFlowAuthenticationPingAll(t *testing.T) {
 	scenario := AuthWebFlowScenario{
 		Scenario: baseScenario,
 	}
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),
@@ -73,7 +73,7 @@ func TestAuthWebFlowLogoutAndRelogin(t *testing.T) {
 	scenario := AuthWebFlowScenario{
 		Scenario: baseScenario,
 	}
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),

integration/cli_test.go

@@ -35,7 +35,7 @@ func TestUserCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": 0,
@@ -115,7 +115,7 @@ func TestPreAuthKeyCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		user: 0,
@@ -257,7 +257,7 @@ func TestPreAuthKeyCommandWithoutExpiry(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		user: 0,
@@ -320,7 +320,7 @@ func TestPreAuthKeyCommandReusableEphemeral(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		user: 0,
@@ -398,7 +398,7 @@ func TestPreAuthKeyCorrectUserLoggedInCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		user1: 1,
@@ -492,7 +492,7 @@ func TestApiKeyCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": 0,
@@ -660,7 +660,7 @@ func TestNodeTagCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": 0,
@@ -785,7 +785,7 @@ func TestNodeAdvertiseTagNoACLCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": 1,
@@ -835,7 +835,7 @@ func TestNodeAdvertiseTagWithACLCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": 1,
@@ -898,7 +898,7 @@ func TestNodeCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"node-user":  0,
@@ -1139,7 +1139,7 @@ func TestNodeExpireCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"node-expire-user": 0,
@@ -1266,7 +1266,7 @@ func TestNodeRenameCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"node-rename-command": 0,
@@ -1432,7 +1432,7 @@ func TestNodeMoveCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"old-user": 0,
@@ -1593,7 +1593,7 @@ func TestPolicyCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"policy-user": 0,
@@ -1673,7 +1673,7 @@ func TestPolicyBrokenConfigCommand(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"policy-user": 1,

integration/control.go

@@ -6,8 +6,8 @@ import (
 )
 
 type ControlServer interface {
-	Shutdown() error
-	SaveLog(string) error
+	Shutdown() (string, string, error)
+	SaveLog(string) (string, string, error)
 	SaveProfile(string) error
 	Execute(command []string) (string, error)
 	WriteFile(path string, content []byte) error

integration/dns_test.go

@@ -17,7 +17,7 @@ func TestResolveMagicDNS(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"magicdns1": len(MustTestVersions),
@@ -208,7 +208,7 @@ func TestValidateResolvConf(t *testing.T) {
 		t.Run(tt.name, func(t *testing.T) {
 			scenario, err := NewScenario(dockertestMaxWait())
 			assertNoErr(t, err)
-			defer scenario.Shutdown()
+			defer scenario.ShutdownAssertNoPanics(t)
 
 			spec := map[string]int{
 				"resolvconf1": 3,

integration/dockertestutil/logs.go

@@ -17,10 +17,10 @@ func SaveLog(
 	pool *dockertest.Pool,
 	resource *dockertest.Resource,
 	basePath string,
-) error {
+) (string, string, error) {
 	err := os.MkdirAll(basePath, os.ModePerm)
 	if err != nil {
-		return err
+		return "", "", err
 	}
 
 	var stdout bytes.Buffer
@@ -41,28 +41,30 @@ func SaveLog(
 		},
 	)
 	if err != nil {
-		return err
+		return "", "", err
 	}
 
 	log.Printf("Saving logs for %s to %s\n", resource.Container.Name, basePath)
 
+	stdoutPath := path.Join(basePath, resource.Container.Name+".stdout.log")
 	err = os.WriteFile(
-		path.Join(basePath, resource.Container.Name+".stdout.log"),
+		stdoutPath,
 		stdout.Bytes(),
 		filePerm,
 	)
 	if err != nil {
-		return err
+		return "", "", err
 	}
 
+	stderrPath := path.Join(basePath, resource.Container.Name+".stderr.log")
 	err = os.WriteFile(
-		path.Join(basePath, resource.Container.Name+".stderr.log"),
+		stderrPath,
 		stderr.Bytes(),
 		filePerm,
 	)
 	if err != nil {
-		return err
+		return "", "", err
 	}
 
-	return nil
+	return stdoutPath, stderrPath, nil
 }

integration/embedded_derp_test.go

@@ -32,7 +32,7 @@ func TestDERPServerScenario(t *testing.T) {
 		Scenario:     baseScenario,
 		tsicNetworks: map[string]*dockertest.Network{},
 	}
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),

integration/general_test.go

@@ -27,7 +27,7 @@ func TestPingAllByIP(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	// TODO(kradalby): it does not look like the user thing works, only second
 	// get created? maybe only when many?
@@ -71,7 +71,7 @@ func TestPingAllByIPPublicDERP(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),
@@ -109,7 +109,7 @@ func TestAuthKeyLogoutAndRelogin(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),
@@ -228,7 +228,7 @@ func testEphemeralWithOptions(t *testing.T, opts ...hsic.Option) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),
@@ -313,7 +313,7 @@ func TestEphemeral2006DeletedTooQuickly(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),
@@ -427,7 +427,7 @@ func TestPingAllByHostname(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user3": len(MustTestVersions),
@@ -476,7 +476,7 @@ func TestTaildrop(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"taildrop": len(MustTestVersions),
@@ -637,7 +637,7 @@ func TestExpireNode(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),
@@ -763,7 +763,7 @@ func TestNodeOnlineStatus(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		"user1": len(MustTestVersions),
@@ -878,7 +878,7 @@ func TestPingAllByIPManyUpDown(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	// TODO(kradalby): it does not look like the user thing works, only second
 	// get created? maybe only when many?
@@ -954,3 +954,102 @@ func TestPingAllByIPManyUpDown(t *testing.T) {
 		t.Logf("%d successful pings out of %d", success, len(allClients)*len(allIps))
 	}
 }
+
+func Test2118DeletingOnlineNodePanics(t *testing.T) {
+	IntegrationSkip(t)
+	t.Parallel()
+
+	scenario, err := NewScenario(dockertestMaxWait())
+	assertNoErr(t, err)
+	defer scenario.ShutdownAssertNoPanics(t)
+
+	// TODO(kradalby): it does not look like the user thing works, only second
+	// get created? maybe only when many?
+	spec := map[string]int{
+		"user1": 1,
+		"user2": 1,
+	}
+
+	err = scenario.CreateHeadscaleEnv(spec,
+		[]tsic.Option{},
+		hsic.WithTestName("deletenocrash"),
+		hsic.WithEmbeddedDERPServerOnly(),
+		hsic.WithTLS(),
+		hsic.WithHostnameAsServerURL(),
+	)
+	assertNoErrHeadscaleEnv(t, err)
+
+	allClients, err := scenario.ListTailscaleClients()
+	assertNoErrListClients(t, err)
+
+	allIps, err := scenario.ListTailscaleClientsIPs()
+	assertNoErrListClientIPs(t, err)
+
+	err = scenario.WaitForTailscaleSync()
+	assertNoErrSync(t, err)
+
+	allAddrs := lo.Map(allIps, func(x netip.Addr, index int) string {
+		return x.String()
+	})
+
+	success := pingAllHelper(t, allClients, allAddrs)
+	t.Logf("%d successful pings out of %d", success, len(allClients)*len(allIps))
+
+	headscale, err := scenario.Headscale()
+	assertNoErr(t, err)
+
+	// Test list all nodes after added otherUser
+	var nodeList []v1.Node
+	err = executeAndUnmarshal(
+		headscale,
+		[]string{
+			"headscale",
+			"nodes",
+			"list",
+			"--output",
+			"json",
+		},
+		&nodeList,
+	)
+	assert.Nil(t, err)
+	assert.Len(t, nodeList, 2)
+	assert.True(t, nodeList[0].Online)
+	assert.True(t, nodeList[1].Online)
+
+	// Delete the first node, which is online
+	_, err = headscale.Execute(
+		[]string{
+			"headscale",
+			"nodes",
+			"delete",
+			"--identifier",
+			// Delete the last added machine
+			fmt.Sprintf("%d", nodeList[0].Id),
+			"--output",
+			"json",
+			"--force",
+		},
+	)
+	assert.Nil(t, err)
+
+	time.Sleep(2 * time.Second)
+
+	// Ensure that the node has been deleted, this did not occur due to a panic.
+	var nodeListAfter []v1.Node
+	err = executeAndUnmarshal(
+		headscale,
+		[]string{
+			"headscale",
+			"nodes",
+			"list",
+			"--output",
+			"json",
+		},
+		&nodeListAfter,
+	)
+	assert.Nil(t, err)
+	assert.Len(t, nodeListAfter, 1)
+	assert.True(t, nodeListAfter[0].Online)
+	assert.Equal(t, nodeList[1].Id, nodeListAfter[0].Id)
+
+}

integration/hsic/hsic.go

@@ -398,8 +398,8 @@ func (t *HeadscaleInContainer) hasTLS() bool {
 }
 
 // Shutdown stops and cleans up the Headscale container.
-func (t *HeadscaleInContainer) Shutdown() error {
-	err := t.SaveLog("/tmp/control")
+func (t *HeadscaleInContainer) Shutdown() (string, string, error) {
+	stdoutPath, stderrPath, err := t.SaveLog("/tmp/control")
 	if err != nil {
 		log.Printf(
 			"Failed to save log from control: %s",
@@ -458,12 +458,12 @@ func (t *HeadscaleInContainer) Shutdown() error {
 		t.pool.Purge(t.pgContainer)
 	}
 
-	return t.pool.Purge(t.container)
+	return stdoutPath, stderrPath, t.pool.Purge(t.container)
 }
 
 // SaveLog saves the current stdout log of the container to a path
 // on the host system.
-func (t *HeadscaleInContainer) SaveLog(path string) error {
+func (t *HeadscaleInContainer) SaveLog(path string) (string, string, error) {
 	return dockertestutil.SaveLog(t.pool, t.container, path)
 }
 

integration/route_test.go

@@ -32,7 +32,7 @@ func TestEnablingRoutes(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErrf(t, "failed to create scenario: %s", err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		user: 3,
@@ -254,7 +254,7 @@ func TestHASubnetRouterFailover(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErrf(t, "failed to create scenario: %s", err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		user: 3,
@@ -826,7 +826,7 @@ func TestEnableDisableAutoApprovedRoute(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErrf(t, "failed to create scenario: %s", err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		user: 1,
@@ -968,7 +968,7 @@ func TestAutoApprovedSubRoute2068(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErrf(t, "failed to create scenario: %s", err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		user: 1,
@@ -1059,7 +1059,7 @@ func TestSubnetRouteACL(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErrf(t, "failed to create scenario: %s", err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	spec := map[string]int{
 		user: 2,

integration/scenario.go

@@ -8,6 +8,7 @@ import (
 	"os"
 	"sort"
 	"sync"
+	"testing"
 	"time"
 
 	v1 "github.com/juanfont/headscale/gen/go/headscale/v1"
@@ -18,6 +19,7 @@ import (
 	"github.com/ory/dockertest/v3"
 	"github.com/puzpuzpuz/xsync/v3"
 	"github.com/samber/lo"
+	"github.com/stretchr/testify/assert"
 	"golang.org/x/sync/errgroup"
 	"tailscale.com/envknob"
 )
@@ -187,13 +189,9 @@ func NewScenario(maxWait time.Duration) (*Scenario, error) {
 	}, nil
 }
 
-// Shutdown shuts down and cleans up all the containers (ControlServer, TailscaleClient)
-// and networks associated with it.
-// In addition, it will save the logs of the ControlServer to `/tmp/control` in the
-// environment running the tests.
-func (s *Scenario) Shutdown() {
+func (s *Scenario) ShutdownAssertNoPanics(t *testing.T) {
 	s.controlServers.Range(func(_ string, control ControlServer) bool {
-		err := control.Shutdown()
+		stdoutPath, stderrPath, err := control.Shutdown()
 		if err != nil {
 			log.Printf(
 				"Failed to shut down control: %s",
@@ -201,6 +199,16 @@ func (s *Scenario) Shutdown() {
 			)
 		}
 
+		if t != nil {
+			stdout, err := os.ReadFile(stdoutPath)
+			assert.NoError(t, err)
+			assert.NotContains(t, string(stdout), "panic")
+
+			stderr, err := os.ReadFile(stderrPath)
+			assert.NoError(t, err)
+			assert.NotContains(t, string(stderr), "panic")
+		}
+
 		return true
 	})
 
@@ -224,6 +232,14 @@ func (s *Scenario) Shutdown() {
 	// }
 }
 
+// Shutdown shuts down and cleans up all the containers (ControlServer, TailscaleClient)
+// and networks associated with it.
+// In addition, it will save the logs of the ControlServer to `/tmp/control` in the
+// environment running the tests.
+func (s *Scenario) Shutdown() {
+	s.ShutdownAssertNoPanics(nil)
+}
+
 // Users returns the name of all users associated with the Scenario.
 func (s *Scenario) Users() []string {
 	users := make([]string, 0)

integration/scenario_test.go

@@ -7,7 +7,7 @@ import (
 )
 
 // This file is intended to "test the test framework", by proxy it will also test
-// some Headcsale/Tailscale stuff, but mostly in very simple ways.
+// some Headscale/Tailscale stuff, but mostly in very simple ways.
 
 func IntegrationSkip(t *testing.T) {
 	t.Helper()
@@ -35,7 +35,7 @@ func TestHeadscale(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	t.Run("start-headscale", func(t *testing.T) {
 		headscale, err := scenario.Headscale()
@@ -80,7 +80,7 @@ func TestCreateTailscale(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	scenario.users[user] = &User{
 		Clients: make(map[string]TailscaleClient),
@@ -116,7 +116,7 @@ func TestTailscaleNodesJoiningHeadcale(t *testing.T) {
 
 	scenario, err := NewScenario(dockertestMaxWait())
 	assertNoErr(t, err)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	t.Run("start-headscale", func(t *testing.T) {
 		headscale, err := scenario.Headscale()

integration/ssh_test.go

@@ -111,7 +111,7 @@ func TestSSHOneUserToAll(t *testing.T) {
 		},
 		len(MustTestVersions),
 	)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	allClients, err := scenario.ListTailscaleClients()
 	assertNoErrListClients(t, err)
@@ -176,7 +176,7 @@ func TestSSHMultipleUsersAllToAll(t *testing.T) {
 		},
 		len(MustTestVersions),
 	)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	nsOneClients, err := scenario.ListTailscaleClients("user1")
 	assertNoErrListClients(t, err)
@@ -222,7 +222,7 @@ func TestSSHNoSSHConfigured(t *testing.T) {
 		},
 		len(MustTestVersions),
 	)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	allClients, err := scenario.ListTailscaleClients()
 	assertNoErrListClients(t, err)
@@ -271,7 +271,7 @@ func TestSSHIsBlockedInACL(t *testing.T) {
 		},
 		len(MustTestVersions),
 	)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	allClients, err := scenario.ListTailscaleClients()
 	assertNoErrListClients(t, err)
@@ -327,7 +327,7 @@ func TestSSHUserOnlyIsolation(t *testing.T) {
 		},
 		len(MustTestVersions),
 	)
-	defer scenario.Shutdown()
+	defer scenario.ShutdownAssertNoPanics(t)
 
 	ssh1Clients, err := scenario.ListTailscaleClients("user1")
 	assertNoErrListClients(t, err)

integration/tsic/tsic.go

@@ -998,7 +998,9 @@ func (t *TailscaleInContainer) WriteFile(path string, data []byte) error {
 // SaveLog saves the current stdout log of the container to a path
 // on the host system.
 func (t *TailscaleInContainer) SaveLog(path string) error {
-	return dockertestutil.SaveLog(t.pool, t.container, path)
+	// TODO(kradalby): Assert if tailscale logs contains panics.
+	_, _, err := dockertestutil.SaveLog(t.pool, t.container, path)
+	return err
 }
 
 // ReadFile reads a file from the Tailscale container.

mkdocs.yml

@@ -55,6 +55,13 @@ theme:
   favicon: assets/favicon.png
   logo: ./logo/headscale3-dots.svg
 
+# Excludes
+exclude_docs: |
+  /packaging/README.md
+  /packaging/postinstall.sh
+  /packaging/postremove.sh
+  /requirements.txt
+
 # Plugins
 plugins:
   - search:
@@ -139,5 +146,5 @@ nav:
           - Remote CLI: remote-cli.md
       - Usage:
           - Android: android-client.md
+          - Apple: apple-client.md
           - Windows: windows-client.md
-          - iOS: iOS-client.md