mirror of
https://github.com/tailscale/tailscale.git
synced 2024-11-25 19:15:34 +00:00
8fad8c4b9b
updates tailcale/corp#22371 Adds custom macOS vm tooling. See the README for the general gist, but this will spin up VMs with unixgram capable network interfaces listening to a named socket, and with a virtio socket device for host-guest communication. We can add other devices like consoles, serial, etc as needed. The whole things is buildable with a single make command, and everything is controllable via the command line using the TailMac utility. This should all be generally functional but takes a few shortcuts with error handling and the like. The virtio socket device support has not been tested and may require some refinement. Signed-off-by: Jonathan Nobels <jonathan@tailscale.com>
162 lines
6.2 KiB
Markdown
162 lines
6.2 KiB
Markdown
# Lightweight macOS VM's for tstest and natlab
|
|
|
|
This utility is designed to provide custom virtual machine tooling support for macOS. The intent
|
|
is to quickly create and spin up small, preconfigured virtual machines, for executing integration
|
|
and unit tests.
|
|
|
|
The primary driver is to provide support for VZVirtioNetworkDeviceConfiguration which is not
|
|
supported by other popular macOS VM hosts. This also gives us the freedom to fully customize and script
|
|
all virtual machine setup and interaction. VZVirtioNetworkDeviceConfiguration lets us
|
|
directly inject and sink network traffic for simulating various network conditions,
|
|
protocols, and topologies and ensure that the TailScale clients handle all of these situations correctly.
|
|
|
|
This may also be used as a drop-in replacement for UTM or Tart on ARM Macs for quickly spinning up
|
|
test VMs. It has the added benefit that, unlike UTM which uses AppleScript, it can be run
|
|
via SSH.
|
|
|
|
This uses Virtualization.framework which only supports arm64. The binaries only build for arm64.
|
|
|
|
|
|
## Components
|
|
|
|
The application is built in two components:
|
|
|
|
The tailmac command line utility is used to set up and configure VM instances. The Host.app does the heavy lifting.
|
|
|
|
You will typically initiate all interactions via the tailmac command-line util.
|
|
|
|
For a full list of options:
|
|
```
|
|
tailmac -h
|
|
```
|
|
|
|
|
|
## Building
|
|
|
|
```
|
|
% make all
|
|
```
|
|
|
|
Will build both the tailmac command line util and Host.app. You will need a developer account. The default bundle identifiers
|
|
default to TailScale owned ids, so if you don't have (or aren't using) a TailScale dev account, you will need to change this.
|
|
This should build automatically as long as you have a valid developer cert. Signing is automatic. The binaries both
|
|
require the virtualization entitlement, so they do need to be signed.
|
|
|
|
There are separate recipes in the makefile to rebuild the individual components if needed.
|
|
|
|
All binaries are copied to the bin directory.
|
|
|
|
|
|
## Locations
|
|
|
|
All vm images, restore images, block device files, save states, and other supporting files are persisted at ~/VM.bundle
|
|
|
|
Each vm gets its own directory. These can be archived for posterity to preserve a particular image and/or state.
|
|
The mere existence of a directory containing all of the required files in ~/VM.bundle is sufficient for tailmac to
|
|
be able to see and run it. ~/VM.bundle and it's contents *is* tailmac's state. No other state is maintained elsewhere.
|
|
|
|
Each vm has its own custom configuration which can be modified while the vm is idle. It's simple JSON - you may
|
|
modify this directly, or using 'tailmac configure'.
|
|
|
|
|
|
## Installing
|
|
|
|
### Default a parameters
|
|
|
|
* The default virtio socket device port is 51009
|
|
* The default server socket for the virtual network device is /tmp/qemu-dgram.sock
|
|
* The default memory size is 4Gb
|
|
* The default mac address for the socket based networking is 52:cc:cc:cc:cc:01
|
|
* The default mac address for the standard ethernet interface is 52:cc:cc:cc:ce:01
|
|
|
|
### Creating and managing VMs
|
|
|
|
You generally perform all interactions via the tailmac command line util. A NAT ethernet device is provided so
|
|
you can ssh into your instance. The ethernet IP will be dhcp assigned by the host and can be determined by parsing
|
|
the contents of /var/db/dhcpd_leases
|
|
|
|
#### Creation
|
|
|
|
To create a new VM (this will grab a restore image for what apples deems a 'latest; if needed). Restore images are large
|
|
(on the order of 10 Gb) and installation after downloading takes a few minutes. If you wish to use a custom restore image,
|
|
specify it with the --image option. If RestoreImage.ipsw exists in ~/VM.bundle, it will be used. macOS versions from
|
|
12 to 15 have been tested and appear to work correctly.
|
|
```
|
|
tailmac create --id my_vm_id
|
|
```
|
|
|
|
With a custom restore image and parameters:
|
|
```
|
|
tailmac create --id my_custom_vm_id --image "/images/macos_ventura.ipsw" --mac 52:cc:cc:cc:cc:07 --mem 8000000000 --sock "/temp/custom.sock" --port 52345
|
|
```
|
|
|
|
A typical workflow would be to create single VM, manually set it up the way you wish including the installation of any required client side software
|
|
(tailscaled or the client-side test harness for example) then clone that images as required and back up your
|
|
images for future use.
|
|
|
|
Fetching and persisting pre-configured images is left as an exercise for the reader (for now). A previously used image can simply be copied to the
|
|
~/VM.bundle directory under a unique path and tailmac will automatically pick it up. No versioning is supported so old images may stop working in
|
|
the future.
|
|
|
|
To delete a VM image, you may simply remove it's directory under ~/VM.bundle or
|
|
```
|
|
tailmac delete --id my_stale_vm
|
|
```
|
|
|
|
Note that the disk size is fixed, but should be sufficient (perhaps even excessive) for most lightweight workflows.
|
|
|
|
#### Restore Images
|
|
|
|
To refresh an existing restore image:
|
|
```
|
|
tailmac refresh
|
|
```
|
|
|
|
Restore images can also be obtained directly from Apple for all macOS releases. Note Apple restore images are raw installs, and the OS will require
|
|
configuration, user setup, etc before being useful. Cloning a vm after clicking through the setup, creating a user and disabling things like the
|
|
lock screen and enabling auto-login will save you time in the future.
|
|
|
|
|
|
#### Cloning
|
|
|
|
To clone an existing vm (this will clone the mac and port as well)
|
|
```
|
|
tailmac clone --id old_vm_id --target-id new_vm_id
|
|
```
|
|
|
|
#### Configuration
|
|
|
|
To reconfigure a existing vm:
|
|
```
|
|
tailmac configure --id vm_id --mac 11:22:33:44:55:66 --port 12345 --ethermac 22:33:44:55:66:77 -sock "/tmp/my.sock"
|
|
```
|
|
|
|
## Running a VM
|
|
|
|
To list the available VM images
|
|
```
|
|
tailmac ls
|
|
```
|
|
|
|
To launch an VM
|
|
```
|
|
tailmac run --id machine_1
|
|
```
|
|
|
|
You may invoke multiple vms, but the limit on the number of concurrent instances is on the order of 2. Use the --tail option to watch the stdout of the
|
|
Host.app process. There is currently no way to list the running VM instances, but invoking stop or halt for a vm instance
|
|
that is not running is perfectly safe.
|
|
|
|
To gracefully stop a running VM and save its state (this is a fire and forget thing):
|
|
|
|
```
|
|
tailmac stop --id machine_1
|
|
```
|
|
|
|
Manually closing a VM's window will save the VM's state (if possible) and is the equivalent of running 'tailmac stop --id vm_id'
|
|
|
|
To halt a running vm without saving its state:
|
|
```
|
|
tailmac halt --id machine_1
|
|
```
|