zitadel/cmd/setup
Silvan dab5d9e756
refactor(eventstore): move push logic to sql (#8816)
# Which Problems Are Solved

If many events are written to the same aggregate id it can happen that
zitadel [starts to retry the push
transaction](48ffc902cc/internal/eventstore/eventstore.go (L101))
because [the locking
behaviour](48ffc902cc/internal/eventstore/v3/sequence.go (L25))
during push does compute the wrong sequence because newly committed
events are not visible to the transaction. These events impact the
current sequence.

In cases with high command traffic on a single aggregate id this can
have severe impact on general performance of zitadel. Because many
connections of the `eventstore pusher` database pool are blocked by each
other.

# How the Problems Are Solved

To improve the performance this locking mechanism was removed and the
business logic of push is moved to sql functions which reduce network
traffic and can be analyzed by the database before the actual push. For
clients of the eventstore framework nothing changed.

# Additional Changes

- after a connection is established prefetches the newly added database
types
- `eventstore.BaseEvent` now returns the correct revision of the event

# Additional Context

- part of https://github.com/zitadel/zitadel/issues/8931

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Elio Bischof <elio@zitadel.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
Co-authored-by: Miguel Cabrerizo <30386061+doncicuto@users.noreply.github.com>
Co-authored-by: Joakim Lodén <Loddan@users.noreply.github.com>
Co-authored-by: Yxnt <Yxnt@users.noreply.github.com>
Co-authored-by: Stefan Benz <stefan@caos.ch>
Co-authored-by: Harsha Reddy <harsha.reddy@klaviyo.com>
Co-authored-by: Zach H <zhirschtritt@gmail.com>
2024-12-04 13:51:40 +00:00
..
01_sql feat(eventstore): increase parallel write capabilities (#5940) 2023-10-19 12:19:10 +02:00
06 fix(setup): create indexes only if not exists (#4801) 2022-12-01 14:31:01 +01:00
07 fix(init): correct quoting of database and user (#6928) 2023-12-03 08:30:08 +00:00
08 fix(setup): optimise events indexes (#5316) 2023-03-01 00:11:23 +00:00
10 feat(eventstore): increase parallel write capabilities (#5940) 2023-10-19 12:19:10 +02:00
12 feat(login): add OTP (email and sms) (#6353) 2023-08-15 12:47:05 +00:00
13 fix: set quotas (#6597) 2023-09-22 13:06:59 +02:00
14 feat(eventstore): increase parallel write capabilities (#5940) 2023-10-19 12:19:10 +02:00
15 feat(eventstore): increase parallel write capabilities (#5940) 2023-10-19 12:19:10 +02:00
34 perf(cache): pgx pool connector (#8703) 2024-10-04 13:15:41 +00:00
35 perf(oidc): nest position clause for session terminated query (#8738) 2024-10-07 12:49:55 +00:00
40 refactor(eventstore): move push logic to sql (#8816) 2024-12-04 13:51:40 +00:00
01.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
02.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
03.go feat: action v2 signing (#8779) 2024-11-28 10:06:52 +00:00
05.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
05.sql fix(database): create column only if not exists (#4812) 2022-12-02 10:48:39 +00:00
06.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
07.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
08.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
10.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
12.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
13.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
14.go perf(oidc): nest position clause for session terminated query (#8738) 2024-10-07 12:49:55 +00:00
15.go perf(oidc): nest position clause for session terminated query (#8738) 2024-10-07 12:49:55 +00:00
16.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
16.sql fix: ensure uniqueness (#6956) 2023-11-22 12:05:14 +00:00
17.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
17.sql fix(eventstore): improve pagination of handler filter (#6968) 2023-12-01 12:25:41 +00:00
18.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
18.sql fix(query): user performance (#6537) 2023-12-08 13:14:22 +01:00
19.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
19.sql fix(eventstore): increase performance on push (#7125) 2023-12-31 15:30:25 +01:00
20.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
20.sql fix(auth): efficient user session projection (#7187) 2024-01-09 18:36:46 +00:00
21.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
21.sql feat: block instances (#7129) 2024-01-17 10:16:48 +00:00
22.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
22.sql fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
23.go fix(eventstore): consider IsGlobal-flag of constraints (#7518) 2024-03-08 13:33:53 +00:00
23.sql fix(eventstore): consider IsGlobal-flag of constraints (#7518) 2024-03-08 13:33:53 +00:00
24.go feat(oidc): token exchange impersonation (#7516) 2024-03-20 10:18:46 +00:00
24.sql feat(oidc): token exchange impersonation (#7516) 2024-03-20 10:18:46 +00:00
25.go feat: password age policy (#8132) 2024-06-18 11:27:44 +00:00
25.sql feat: password age policy (#8132) 2024-06-18 11:27:44 +00:00
26.go fix(login): improve auth handlers (#7969) 2024-05-22 15:26:02 +00:00
26.sql fix(login): improve auth handlers (#7969) 2024-05-22 15:26:02 +00:00
27.go feat: Add Twilio Verification Service (#8678) 2024-09-26 09:14:33 +02:00
27.sql feat(saml): allow setting nameid-format and alternative mapping for transient format (#7979) 2024-05-23 05:04:07 +00:00
28.go feat(eventstore): add search table (#8191) 2024-07-03 15:00:56 +00:00
28.sql feat(eventstore): add search table (#8191) 2024-07-03 15:00:56 +00:00
29.go feat(eventstore): add search table (#8191) 2024-07-03 15:00:56 +00:00
30.go perf(import): optimize search for domains claimed by other organizations (#8200) 2024-07-05 09:36:00 +02:00
31.go fix(fields): add index to improve search by aggregate (#8267) 2024-07-08 15:54:19 +00:00
31.sql fix(fields): add index to improve search by aggregate (#8267) 2024-07-08 15:54:19 +00:00
32.go feat(oidc): sid claim for id_tokens issued through login V1 (#8525) 2024-09-03 13:19:00 +00:00
32.sql feat(oidc): sid claim for id_tokens issued through login V1 (#8525) 2024-09-03 13:19:00 +00:00
33.go feat: Add Twilio Verification Service (#8678) 2024-09-26 09:14:33 +02:00
33.sql feat: Add Twilio Verification Service (#8678) 2024-09-26 09:14:33 +02:00
34.go perf(cache): pgx pool connector (#8703) 2024-10-04 13:15:41 +00:00
35.go perf(oidc): nest position clause for session terminated query (#8738) 2024-10-07 12:49:55 +00:00
36.go fix(milestones): use previous spelling for milestone types (#8886) 2024-11-11 11:28:27 +00:00
36.sql perf(milestones): refactor (#8788) 2024-10-28 08:29:34 +00:00
37.go feat(OIDC): add back channel logout (#8837) 2024-10-31 15:57:17 +01:00
37.sql feat(OIDC): add back channel logout (#8837) 2024-10-31 15:57:17 +01:00
38.go feat(OIDC): add back channel logout (#8837) 2024-10-31 15:57:17 +01:00
38.sql feat(OIDC): add back channel logout (#8837) 2024-10-31 15:57:17 +01:00
39.go fix(eventstore): cleanup org fields on remove (#8946) 2024-11-26 15:26:41 +00:00
39.sql fix(eventstore): cleanup org fields on remove (#8946) 2024-11-26 15:26:41 +00:00
40.go refactor(eventstore): move push logic to sql (#8816) 2024-12-04 13:51:40 +00:00
cleanup.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
config_change.go feat: action v2 signing (#8779) 2024-11-28 10:06:52 +00:00
config_test.go fix(setup): decode complex config strings (#7854) 2024-05-01 12:17:27 +02:00
config.go refactor(eventstore): move push logic to sql (#8816) 2024-12-04 13:51:40 +00:00
projections.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
setup.go refactor(eventstore): move push logic to sql (#8816) 2024-12-04 13:51:40 +00:00
steps.yaml docs(cmd): document that human and service user can be used in the default instance (#8229) 2024-07-04 11:39:28 +00:00