mirror of
https://github.com/zitadel/zitadel.git
synced 2025-02-28 19:57:22 +00:00
docs: Eventstore architecture (#3904)
* docs: add zitadel eventstore architecture * docs: add zitadel eventstore architecture * docs: add zitadel eventstore architecture * docs: add zitadel eventstore architecture * docs: add zitadel eventstore architecture * docs: fix broken link * Update docs/docs/concepts/eventstore/overview.md Co-authored-by: Elio Bischof <eliobischof@gmail.com> * Update docs/docs/concepts/eventstore/overview.md Co-authored-by: Elio Bischof <eliobischof@gmail.com> * Update docs/docs/concepts/eventstore/overview.md Co-authored-by: Elio Bischof <eliobischof@gmail.com> * Update docs/docs/concepts/eventstore/zitadel.md Co-authored-by: Elio Bischof <eliobischof@gmail.com> * Update docs/docs/concepts/eventstore/zitadel.md Co-authored-by: Elio Bischof <eliobischof@gmail.com> Co-authored-by: Elio Bischof <eliobischof@gmail.com>
This commit is contained in:
Fabi
GitHub
parent
8434eaa9c0
commit
d15a15c809
@ -1,20 +0,0 @@
|
||||
---
|
||||
title: Eventstore
|
||||
---
|
||||
|
||||
ZITADEL is built on the [event sourcing pattern](./architecture), where changes are stored as events in an eventstore.
|
||||
|
||||
## What is an eventstore?
|
||||
|
||||
Traditionally, data is stored in relations as a state
|
||||
|
||||
- A request needs to know the relations to select valid data
|
||||
- If a relation changes, the requests need to change as well
|
||||
- That is valid for actual, as well as for historical data
|
||||
|
||||
An Eventstore on the other hand stores events, meaning every change that happens to any piece of data relates to an event
|
||||
|
||||
- Think of it as a ledger that gets new entries over time, accumulative
|
||||
- To request data, all you have to do is to sum the events as the summary reflects the actual state
|
||||
- To investigate past changes to your system, you just select the events from your time range of interest
|
||||
- That makes audit/analytics very powerful, due to the historical data available to build queries
|
||||
50
docs/docs/concepts/eventstore/overview.md
Normal file
50
docs/docs/concepts/eventstore/overview.md
Normal file
@ -0,0 +1,50 @@
|
||||
---
|
||||
title: Overview
|
||||
---
|
||||
|
||||
ZITADEL is built on the [event sourcing pattern](../architecture), where changes are stored as events in an eventstore.
|
||||
|
||||
## What is an eventstore?
|
||||
|
||||
Traditionally, data is stored in relations as a state
|
||||
|
||||
- A request needs to know the relations to select valid data
|
||||
- If a relation changes, the requests need to change as well
|
||||
- That is valid for actual, as well as for historical data
|
||||
|
||||
An Eventstore on the other hand stores events, meaning every change that happens to any piece of data relates to an event.
|
||||
The data is stored as events in an append-only log.
|
||||
|
||||
- Think of it as a ledger that gets new entries over time, accumulative
|
||||
- To request data, all you have to do is to sum the events as the summary reflects the actual state
|
||||
- To investigate past changes to your system, you just select the events from your time range of interest
|
||||
- That makes audit/analytics very powerful, due to the historical data available to build queries
|
||||
|
||||
## Benefits
|
||||
|
||||
Audit: You have a built-in audit trail that tracks all changes over an unlimited period of time.
|
||||
Travel back in time: With our way of storing data we can show you all of your resources at a given point in time.
|
||||
|
||||
## Definitions
|
||||
|
||||
Eventsourcing has some specific terms that are often used in our documentation. To understand how ZITADEL works it is important to understand this key definitions.
|
||||
|
||||
### Events
|
||||
|
||||
An event is something that happens in the system and gets written to the database. This is the single source of truth.
|
||||
Events are immutable and the current state of your system is derived from the events.
|
||||
|
||||
Possible Events:
|
||||
- user.added
|
||||
- user.changed
|
||||
- product.added
|
||||
- user.password.checked
|
||||
|
||||
### Aggregate
|
||||
|
||||
An aggregate consist of multiple events. All events together in will lead to the current state of the aggregate.
|
||||
The aggregate can be compared with an object or a resources.
|
||||
|
||||
### Projections
|
||||
|
||||
Projections contain the computed objects, that will be used on the query side for all the requests.
|
||||
97
docs/docs/concepts/eventstore/zitadel.md
Normal file
97
docs/docs/concepts/eventstore/zitadel.md
Normal file
@ -0,0 +1,97 @@
|
||||
---
|
||||
title: Implementation in ZITADEL
|
||||
---
|
||||
|
||||
This documentation gives you an insight into the structure of the ZITADEL database.
|
||||
The goal is to give you a rough overview, so you know where which data is stored and which database schemas and tables are used.
|
||||
|
||||
## Event
|
||||
|
||||
The single source of truth of ZITADEL are the events that are stored in the eventstore.
|
||||
From these events all different kind of resources e.g Users, Projects, Applications, etc. can be computed.
|
||||
|
||||
An event has the following data:
|
||||
|
||||
| Attribute | Description | Example |
|
||||
| --- | --- | --- |
|
||||
| id | unique identifier of the event, this is generated by the database | b6402a60-e4655-4cc0-904b-f5c6760c4406 |
|
||||
| aggregate_type | the type of the aggregate, an aggregate can be compared to a resource or an object. One aggregate contains multiple event types | user |
|
||||
| aggregate_id | The unique identifier of an aggregate, this is generated by ZITADEL and is a sonyflake id | 168096909691353697 |
|
||||
| aggregate_version | The aggregate version shows in which version of the aggregate the event was created. This is needed to be able to compute correct objects | v1 |
|
||||
| event_type | The type of the event | user.human.added |
|
||||
| event_sequence | The event sequence is a sequence number that is incremented by one for each event on the instance. For technical reasons, a number can be omitted in some cases. This is needed so that the sequence of the events can be ensured. | 1234 |
|
||||
| previous_aggregate_sequence| This number is the sequence of the event last created on this specific aggregate. E.g. Last user with specific aggregate_id | 1233|
|
||||
| previous_aggregate_type_sequence | This number is the sequence of the event last created on this aggregate. E.g Last User | 1230 |
|
||||
| creation_date| timestamp when the event was created | 2022-07-05 13:57:56.358774+00 |
|
||||
| editor_user| The editor user contains mostly an unique identifier of a user. And tells who did the request that led to this event. Sometimes this can also be a name of a system within ZITADEL. | 165460784409638965, NOTIFICATION, LOGIN |
|
||||
| editor_service | The service defines which API was called when the event got created. If the event was created from the system itself this is empty. | Admin-API |
|
||||
| resource_owner | The resource owner defines to which organization/resource_owner the event belongs. This is an id generated by ZITADEL as sonyflake id | 168051083313153168 |
|
||||
| instance_id | ZITADEL is capable of containing multiple ZITADEL instances withing the system. This id is the unique identifier of the Instance and is generated by ZITADEL as sonyflake id. | 165460784409737865 |
|
||||
|
||||
|
||||
## Schemas
|
||||
|
||||
| Schema | Description | Examples |
|
||||
| --- | --- | --- |
|
||||
| System | The system contains everything that is needed outside the ZITADEL instances. | assets, encryption_key |
|
||||
| Eventstore | Eventstore is the base of ZITADEL and is the single source of truth. All the events stored in the eventstore can be used to generate different projections. | events, instance sequences, system wide unique constraints
|
||||
| Projections| The projections contain all the computed objects which are used for reading requests. | users, projects, etc|
|
||||
| Auth | This contains projections which are used for the auth api. All projections in this schema should be moved to Projections soon | users, auth_request, etc. |
|
||||
| Adminapi | This contains projections which are used for the admin api. All projections in this schema should be moved to Projections soon | styling|
|
||||
| Notification | This contains projections which are used for sending notification. All projections in this schema should be moved to Projections soon | styling|
|
||||
|
||||
## Projections
|
||||
|
||||
The projections in ZITADEL contain all the computed objects, that are used for the reading requests.
|
||||
It is possible that the projections are slightly behind the actual event and not all objects are up-to-date.
|
||||
|
||||
### Pub-Sub
|
||||
To keep the projections as up-to-date as possible, an internal pub-sub system is used.
|
||||
As soon as an event is written to the event store, it is sent to the projections that have subscribed to this aggregate.
|
||||
|
||||
### Spooler
|
||||
It is sometimes possible for technical reasons that not all events were sent to the projections.
|
||||
For this reason, a spooler runs in parallel, which checks every n minutes whether there are new events that have not yet been processed.
|
||||
|
||||
### Current Sequence
|
||||
|
||||
To ensure that no events get missed when creating the Projections, ZITADEL stores the current sequence, that was processed.
|
||||
You can find the current sequence in the following tables:
|
||||
- projections.current_sequences
|
||||
- notification.current_sequences
|
||||
- auth.current_Sequences
|
||||
- adminapi.current_sequences
|
||||
|
||||
The current sequence is stored for each ZITADEL instance and table.
|
||||
|
||||
| Attribute | Description | Examples |
|
||||
| --- | --- | --- |
|
||||
| projection_name | The name of the projection for which the sequence is valid. | projection.users |
|
||||
| aggregate_type | The aggregate type where the sequence was from | user |
|
||||
| current_sequence | The sequence that was last processed | 1234 |
|
||||
| instance_id | The instance to which the event belongs | 165460784409737834 |
|
||||
| timestamp | Timestamp when the table was updated | 2022-07-05 13:57:59.454798+00 |
|
||||
|
||||
### Failed Events
|
||||
|
||||
Sometimes an event cannot be processed correctly for some reason and an error occurs.
|
||||
The event is then tried to be processed n times.
|
||||
When a defined number of attempts have failed, the event is stored in the Failed Events Table and the next event is processed.
|
||||
This must be done so that the projection is not blocked and no further events are processed.
|
||||
|
||||
You can find the failed_events in the following tables:
|
||||
- projections.failed_events
|
||||
- notification.failed_events
|
||||
- auth.failed_events
|
||||
- adminapi.failed_events
|
||||
|
||||
| Attribute | Description | Examples |
|
||||
| --- | --- | --- |
|
||||
| projection_name | The name of the projection for which the failed event should have been processed. | projection.users |
|
||||
| failed_sequence | The sequence of the event that failed | 1234 |
|
||||
| failure_count | The number of times the event was attempted to be processed. If the number is lower than the max. attempts the event could be processed, but not on the first attempt | 5 |
|
||||
| error | The error message that occurred when the event could not be processed | User not found |
|
||||
| instance_id | The instance to which the event belongs | 165460784409737834 |
|
||||
|
||||
|
||||
|
||||
@ -193,7 +193,15 @@ module.exports = {
|
||||
concepts: [
|
||||
"concepts/introduction",
|
||||
"concepts/principles",
|
||||
"concepts/eventstore",
|
||||
{
|
||||
type: "category",
|
||||
label: "Eventstore",
|
||||
collapsed: false,
|
||||
items: [
|
||||
"concepts/eventstore/overview",
|
||||
"concepts/eventstore/zitadel",
|
||||
],
|
||||
},
|
||||
"concepts/architecture",
|
||||
{
|
||||
type: "category",
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user