mirror of
https://github.com/tailscale/tailscale.git
synced 2024-12-12 03:04:40 +00:00
0d19f5d421
The log ID types were moved to a separate package so that code that only depend on log ID types do not need to link in the logic for the logtail client itself. Not all code need the logtail client. Signed-off-by: Joe Tsai <joetsai@digital-static.net>
195 lines
6.7 KiB
Markdown
195 lines
6.7 KiB
Markdown
# Tailscale Logs Service
|
|
|
|
The Tailscale Logs Service defines a REST interface for configuring, storing,
|
|
retrieving, and processing log entries.
|
|
|
|
# Overview
|
|
|
|
HTTP requests are received at the service **base URL**
|
|
[https://log.tailscale.io](https://log.tailscale.io), and return JSON-encoded
|
|
responses using standard HTTP response codes.
|
|
|
|
Authorization for the configuration and retrieval APIs is done with a secret
|
|
API key passed as the HTTP basic auth username. Secret keys are generated via
|
|
the web UI at base URL. An example of using basic auth with curl:
|
|
|
|
curl -u <log_api_key>: https://log.tailscale.io/collections
|
|
|
|
In the future, an HTTP header will allow using MessagePack instead of JSON.
|
|
|
|
## Collections
|
|
|
|
Logs are organized into collections. Inside each collection is any number of
|
|
instances.
|
|
|
|
A collection is a domain name. It is a grouping of related logs. As a
|
|
guideline, create one collection per product using subdomains of your
|
|
company's domain name. Collections must be registered with the logs service
|
|
before any attempt is made to store logs.
|
|
|
|
## Instances
|
|
|
|
Each collection is a set of instances. There is one instance per machine
|
|
writing logs.
|
|
|
|
An instance has a name and a number. An instance has a **private** and
|
|
**public** ID. The private ID is a 32-byte random number encoded as hex.
|
|
The public ID is the SHA-256 hash of the private ID, encoded as hex.
|
|
|
|
The private ID is used to write logs. The only copy of the private ID
|
|
should be on the machine sending logs. Ideally it is generated on the
|
|
machine. Logs can be written as soon as a private ID is generated.
|
|
|
|
The public ID is used to read and adopt logs. It is designed to be sent
|
|
to a service that also holds a logs service API key.
|
|
|
|
The tailscale logs service will store any logs for a short period of time.
|
|
To enable logs retention, the log can be **adopted** using the public ID
|
|
and a logs service API key.
|
|
Once this is done, logs will be retained long-term (for the configured
|
|
retention period).
|
|
|
|
Unadopted instance logs are stored temporarily to help with debugging:
|
|
a misconfigured machine writing logs with a bad ID can be spotted by
|
|
reading the logs.
|
|
If a public ID is not adopted, storage is tightly capped and logs are
|
|
deleted after 12 hours.
|
|
|
|
# APIs
|
|
|
|
## Storage
|
|
|
|
### `POST /c/<collection-name>/<private-ID>` — send a log
|
|
|
|
The body of the request is JSON.
|
|
|
|
A **single message** is an object with properties:
|
|
|
|
`{ }`
|
|
|
|
The client may send any properties it wants in the JSON message, except
|
|
for the `logtail` property which has special meaning. Inside the logtail
|
|
object the client may only set the following properties:
|
|
|
|
- `client_time` in the format of RFC3339: "2006-01-02T15:04:05.999999999Z07:00"
|
|
|
|
A future version of the logs service API will also support:
|
|
|
|
- `client_time_offset` a integer of nanoseconds since the client was reset
|
|
- `client_time_reset` a boolean if set to true resets the time offset counter
|
|
|
|
On receipt by the server the `client_time_offset` is transformed into a
|
|
`client_time` based on the `server_time` when the first (or
|
|
client_time_reset) event was received.
|
|
|
|
If any other properties are set in the logtail object they are moved into
|
|
the "error" field, the message is saved and a 4xx status code is returned.
|
|
|
|
A **batch of messages** is a JSON array filled with single message objects:
|
|
|
|
`[ { }, { }, ... ]`
|
|
|
|
If any of the array entries are not objects, the content is converted
|
|
into a message with a `"logtail": { "error": ...}` property, saved, and
|
|
a 4xx status code is returned.
|
|
|
|
Similarly any other request content not matching one of these formats is
|
|
saved in a logtail error field, and a 4xx status code is returned.
|
|
|
|
An invalid collection name returns `{"error": "invalid collection name"}`
|
|
along with a 403 status code.
|
|
|
|
Clients are encouraged to:
|
|
|
|
- POST as rapidly as possible (if not battery constrained). This minimizes
|
|
both the time necessary to see logs in a log viewer and the chance of
|
|
losing logs.
|
|
- Use HTTP/2 when streaming logs, as it does a much better job of
|
|
maintaining a TLS connection to minimize overhead for subsequent posts.
|
|
|
|
A future version of logs service API will support sending requests with
|
|
`Content-Encoding: zstd`.
|
|
|
|
## Retrieval
|
|
|
|
### `GET /collections` — query the set of collections and instances
|
|
|
|
Returns a JSON object listing all of the named collections.
|
|
|
|
The caller can query-encode the following fields:
|
|
|
|
- `collection-name` — limit the results to one collection
|
|
|
|
```
|
|
{
|
|
"collections": {
|
|
"collection1.yourcompany.com": {
|
|
"instances": {
|
|
"<logid.PublicID>" :{
|
|
"first-seen": "timestamp",
|
|
"size": 4096
|
|
},
|
|
"<logid.PublicID>" :{
|
|
"first-seen": "timestamp",
|
|
"size": 512000,
|
|
"orphan": true,
|
|
}
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
### `GET /c/<collection_name>` — query stored logs
|
|
|
|
The caller can query-encode the following fields:
|
|
|
|
- `instances` — zero or more log collection instances to limit results to
|
|
- `time-start` — the earliest log to include
|
|
- One of:
|
|
- `time-end` — the latest log to include
|
|
- `max-count` — maximum number of logs to return, allows paging
|
|
- `stream` — boolean that keeps the response dangling, streaming in
|
|
logs like `tail -f`. Incompatible with logtail-time-end.
|
|
|
|
In **stream=false** mode, the response is a single JSON object:
|
|
|
|
{
|
|
// TODO: header fields
|
|
"logs": [ {}, {}, ... ]
|
|
}
|
|
|
|
In **stream=true** mode, the response begins with a JSON header object
|
|
similar to the storage format, and then is a sequence of JSON log
|
|
objects, `{...}`, one per line. The server continues to send these until
|
|
the client closes the connection.
|
|
|
|
## Configuration
|
|
|
|
For organizations with a small number of instances writing logs, the
|
|
Configuration API are best used by a trusted human operator, usually
|
|
through a GUI. Organizations with many instances will need to automate
|
|
the creation of tokens.
|
|
|
|
### `POST /collections` — create or delete a collection
|
|
|
|
The caller must set the `collection` property and `action=create` or
|
|
`action=delete`, either form encoded or JSON encoded. Its character set
|
|
is restricted to the mundane: [a-zA-Z0-9-_.]+
|
|
|
|
Collection names are a global space. Typically they are a domain name.
|
|
|
|
### `POST /instances` — adopt an instance into a collection
|
|
|
|
The caller must send the following properties, form encoded or JSON encoded:
|
|
|
|
- `collection` — a valid FQDN ([a-zA-Z0-9-_.]+)
|
|
- `instances` an instance public ID encoded as hex
|
|
|
|
The collection name must be claimed by a group the caller belongs to.
|
|
The pair (collection-name, instance-public-ID) may or may not already have
|
|
logs associated with it.
|
|
|
|
On failure, an error message is returned with a 4xx or 5xx status code:
|
|
|
|
`{"error": "what went wrong"}` |