mirror of
https://github.com/tailscale/tailscale.git
synced 2025-10-20 23:19:27 +00:00
tstest/tailmac: add customized macOS virtualization tooling (#13146)
updates tailcale/corp#22371 Adds custom macOS vm tooling. See the README for the general gist, but this will spin up VMs with unixgram capable network interfaces listening to a named socket, and with a virtio socket device for host-guest communication. We can add other devices like consoles, serial, etc as needed. The whole things is buildable with a single make command, and everything is controllable via the command line using the TailMac utility. This should all be generally functional but takes a few shortcuts with error handling and the like. The virtio socket device support has not been tested and may require some refinement. Signed-off-by: Jonathan Nobels <jonathan@tailscale.com>
This commit is contained in:
161
tstest/tailmac/README.md
Normal file
161
tstest/tailmac/README.md
Normal file
@@ -0,0 +1,161 @@
|
||||
# Lightweight macOS VM's for tstest and natlab
|
||||
|
||||
This utility is designed to provide custom virtual machine tooling support for macOS. The intent
|
||||
is to quickly create and spin up small, preconfigured virtual machines, for executing integration
|
||||
and unit tests.
|
||||
|
||||
The primary driver is to provide support for VZVirtioNetworkDeviceConfiguration which is not
|
||||
supported by other popular macOS VM hosts. This also gives us the freedom to fully customize and script
|
||||
all virtual machine setup and interaction. VZVirtioNetworkDeviceConfiguration lets us
|
||||
directly inject and sink network traffic for simulating various network conditions,
|
||||
protocols, and topologies and ensure that the TailScale clients handle all of these situations correctly.
|
||||
|
||||
This may also be used as a drop-in replacement for UTM or Tart on ARM Macs for quickly spinning up
|
||||
test VMs. It has the added benefit that, unlike UTM which uses AppleScript, it can be run
|
||||
via SSH.
|
||||
|
||||
This uses Virtualization.framework which only supports arm64. The binaries only build for arm64.
|
||||
|
||||
|
||||
## Components
|
||||
|
||||
The application is built in two components:
|
||||
|
||||
The tailmac command line utility is used to set up and configure VM instances. The Host.app does the heavy lifting.
|
||||
|
||||
You will typically initiate all interactions via the tailmac command-line util.
|
||||
|
||||
For a full list of options:
|
||||
```
|
||||
tailmac -h
|
||||
```
|
||||
|
||||
|
||||
## Building
|
||||
|
||||
```
|
||||
% make all
|
||||
```
|
||||
|
||||
Will build both the tailmac command line util and Host.app. You will need a developer account. The default bundle identifiers
|
||||
default to TailScale owned ids, so if you don't have (or aren't using) a TailScale dev account, you will need to change this.
|
||||
This should build automatically as long as you have a valid developer cert. Signing is automatic. The binaries both
|
||||
require the virtualization entitlement, so they do need to be signed.
|
||||
|
||||
There are separate recipes in the makefile to rebuild the individual components if needed.
|
||||
|
||||
All binaries are copied to the bin directory.
|
||||
|
||||
|
||||
## Locations
|
||||
|
||||
All vm images, restore images, block device files, save states, and other supporting files are persisted at ~/VM.bundle
|
||||
|
||||
Each vm gets its own directory. These can be archived for posterity to preserve a particular image and/or state.
|
||||
The mere existence of a directory containing all of the required files in ~/VM.bundle is sufficient for tailmac to
|
||||
be able to see and run it. ~/VM.bundle and it's contents *is* tailmac's state. No other state is maintained elsewhere.
|
||||
|
||||
Each vm has its own custom configuration which can be modified while the vm is idle. It's simple JSON - you may
|
||||
modify this directly, or using 'tailmac configure'.
|
||||
|
||||
|
||||
## Installing
|
||||
|
||||
### Default a parameters
|
||||
|
||||
* The default virtio socket device port is 51009
|
||||
* The default server socket for the virtual network device is /tmp/qemu-dgram.sock
|
||||
* The default memory size is 4Gb
|
||||
* The default mac address for the socket based networking is 52:cc:cc:cc:cc:01
|
||||
* The default mac address for the standard ethernet interface is 52:cc:cc:cc:ce:01
|
||||
|
||||
### Creating and managing VMs
|
||||
|
||||
You generally perform all interactions via the tailmac command line util. A NAT ethernet device is provided so
|
||||
you can ssh into your instance. The ethernet IP will be dhcp assigned by the host and can be determined by parsing
|
||||
the contents of /var/db/dhcpd_leases
|
||||
|
||||
#### Creation
|
||||
|
||||
To create a new VM (this will grab a restore image for what apples deems a 'latest; if needed). Restore images are large
|
||||
(on the order of 10 Gb) and installation after downloading takes a few minutes. If you wish to use a custom restore image,
|
||||
specify it with the --image option. If RestoreImage.ipsw exists in ~/VM.bundle, it will be used. macOS versions from
|
||||
12 to 15 have been tested and appear to work correctly.
|
||||
```
|
||||
tailmac create --id my_vm_id
|
||||
```
|
||||
|
||||
With a custom restore image and parameters:
|
||||
```
|
||||
tailmac create --id my_custom_vm_id --image "/images/macos_ventura.ipsw" --mac 52:cc:cc:cc:cc:07 --mem 8000000000 --sock "/temp/custom.sock" --port 52345
|
||||
```
|
||||
|
||||
A typical workflow would be to create single VM, manually set it up the way you wish including the installation of any required client side software
|
||||
(tailscaled or the client-side test harness for example) then clone that images as required and back up your
|
||||
images for future use.
|
||||
|
||||
Fetching and persisting pre-configured images is left as an exercise for the reader (for now). A previously used image can simply be copied to the
|
||||
~/VM.bundle directory under a unique path and tailmac will automatically pick it up. No versioning is supported so old images may stop working in
|
||||
the future.
|
||||
|
||||
To delete a VM image, you may simply remove it's directory under ~/VM.bundle or
|
||||
```
|
||||
tailmac delete --id my_stale_vm
|
||||
```
|
||||
|
||||
Note that the disk size is fixed, but should be sufficient (perhaps even excessive) for most lightweight workflows.
|
||||
|
||||
#### Restore Images
|
||||
|
||||
To refresh an existing restore image:
|
||||
```
|
||||
tailmac refresh
|
||||
```
|
||||
|
||||
Restore images can also be obtained directly from Apple for all macOS releases. Note Apple restore images are raw installs, and the OS will require
|
||||
configuration, user setup, etc before being useful. Cloning a vm after clicking through the setup, creating a user and disabling things like the
|
||||
lock screen and enabling auto-login will save you time in the future.
|
||||
|
||||
|
||||
#### Cloning
|
||||
|
||||
To clone an existing vm (this will clone the mac and port as well)
|
||||
```
|
||||
tailmac clone --id old_vm_id --target-id new_vm_id
|
||||
```
|
||||
|
||||
#### Configuration
|
||||
|
||||
To reconfigure a existing vm:
|
||||
```
|
||||
tailmac configure --id vm_id --mac 11:22:33:44:55:66 --port 12345 --ethermac 22:33:44:55:66:77 -sock "/tmp/my.sock"
|
||||
```
|
||||
|
||||
## Running a VM
|
||||
|
||||
To list the available VM images
|
||||
```
|
||||
tailmac ls
|
||||
```
|
||||
|
||||
To launch an VM
|
||||
```
|
||||
tailmac run --id machine_1
|
||||
```
|
||||
|
||||
You may invoke multiple vms, but the limit on the number of concurrent instances is on the order of 2. Use the --tail option to watch the stdout of the
|
||||
Host.app process. There is currently no way to list the running VM instances, but invoking stop or halt for a vm instance
|
||||
that is not running is perfectly safe.
|
||||
|
||||
To gracefully stop a running VM and save its state (this is a fire and forget thing):
|
||||
|
||||
```
|
||||
tailmac stop --id machine_1
|
||||
```
|
||||
|
||||
Manually closing a VM's window will save the VM's state (if possible) and is the equivalent of running 'tailmac stop --id vm_id'
|
||||
|
||||
To halt a running vm without saving its state:
|
||||
```
|
||||
tailmac halt --id machine_1
|
||||
```
|
Reference in New Issue
Block a user