Commit Graph

16 Commits

Author SHA1 Message Date
Silvan
dab5d9e756
refactor(eventstore): move push logic to sql (#8816)
# Which Problems Are Solved

If many events are written to the same aggregate id it can happen that
zitadel [starts to retry the push
transaction](48ffc902cc/internal/eventstore/eventstore.go (L101))
because [the locking
behaviour](48ffc902cc/internal/eventstore/v3/sequence.go (L25))
during push does compute the wrong sequence because newly committed
events are not visible to the transaction. These events impact the
current sequence.

In cases with high command traffic on a single aggregate id this can
have severe impact on general performance of zitadel. Because many
connections of the `eventstore pusher` database pool are blocked by each
other.

# How the Problems Are Solved

To improve the performance this locking mechanism was removed and the
business logic of push is moved to sql functions which reduce network
traffic and can be analyzed by the database before the actual push. For
clients of the eventstore framework nothing changed.

# Additional Changes

- after a connection is established prefetches the newly added database
types
- `eventstore.BaseEvent` now returns the correct revision of the event

# Additional Context

- part of https://github.com/zitadel/zitadel/issues/8931

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Elio Bischof <elio@zitadel.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
Co-authored-by: Miguel Cabrerizo <30386061+doncicuto@users.noreply.github.com>
Co-authored-by: Joakim Lodén <Loddan@users.noreply.github.com>
Co-authored-by: Yxnt <Yxnt@users.noreply.github.com>
Co-authored-by: Stefan Benz <stefan@caos.ch>
Co-authored-by: Harsha Reddy <harsha.reddy@klaviyo.com>
Co-authored-by: Zach H <zhirschtritt@gmail.com>
2024-12-04 13:51:40 +00:00
Tim Möhlmann
aeb379e7de
fix(eventstore): revert precise decimal (#8527) (#8679) 2024-09-24 18:43:29 +02:00
Silvan
b522588d98
fix(eventstore): precise decimal (#8527)
# Which Problems Are Solved

Float64 which was used for the event.Position field is [not precise in
go and gets rounded](https://github.com/golang/go/issues/47300). This
can lead to unprecies position tracking of events and therefore
projections especially on cockcoachdb as the position used there is a
big number.

example of a unprecies position:
exact: 1725257931223002628
float64: 1725257931223002624.000000

# How the Problems Are Solved

The float64 was replaced by
[github.com/jackc/pgx-shopspring-decimal](https://github.com/jackc/pgx-shopspring-decimal).

# Additional Changes

Correct behaviour of makefile for load tests.
Rename `latestSequence`-queries to `latestPosition`
2024-09-06 12:19:19 +03:00
Silvan
cff0f73e24
fix(init): add setting to enable durable locks on crdb (#7982)
feat(init): add setting to enable durable locks on crdb
2024-05-27 09:03:34 +00:00
Silvan
4224c7ad3a
chore(core): update dependencies (#7517)
* chore(core): update dependencies

* chore(core): update dependencies
2024-03-11 10:43:56 +00:00
Livio Spring
8470649ecb
chore: pin crdb version for unit tests (#7260)
* chore: pin crdb version for unit tests

* use latest 23.1 version

* use latest available 23.1 version
2024-01-18 08:16:54 +00:00
Silvan
b7d027e2fd
fix(db): always use begin tx (#7142)
* fix(db): always use begin tx

* fix(handler): timeout for begin
2024-01-04 16:12:20 +00:00
Silvan
b5564572bc
feat(eventstore): increase parallel write capabilities (#5940)
This implementation increases parallel write capabilities of the eventstore.
Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and  [06](https://zitadel.com/docs/support/advisory/a10006).
The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`.
If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.
2023-10-19 12:19:10 +02:00
Silvan
e38abdcdf3
perf: query data AS OF SYSTEM TIME (#5231)
Queries the data in the storage layser at the timestamp when the call hit the API layer
2023-02-27 22:36:43 +01:00
Silvan
77b4fc5487
feat(database): support for postgres (#3998)
* beginning with postgres statements

* try pgx

* use pgx

* database

* init works for postgres

* arrays working

* init for cockroach

* init

* start tests

* tests

* TESTS

* ch

* ch

* chore: use go 1.18

* read stmts

* fix typo

* tests

* connection string

* add missing error handler

* cleanup

* start all apis

* go mod tidy

* old update

* switch back to minute

* on conflict

* replace string slice with `database.StringArray` in db models

* fix tests and start

* update go version in dockerfile

* setup go

* clean up

* remove notification migration

* update

* docs: add deploy guide for postgres

* fix: revert sonyflake

* use `database.StringArray` for daos

* use `database.StringArray` every where

* new tables

* index naming,
metadata primary key,
project grant role key type

* docs(postgres): change to beta

* chore: correct compose

* fix(defaults): add empty postgres config

* refactor: remove unused code

* docs: add postgres to self hosted

* fix broken link

* so?

* change title

* add mdx to link

* fix stmt

* update goreleaser in test-code

* docs: improve postgres example

* update more projections

* fix: add beta log for postgres

* revert index name change

* prerelease

* fix: add sequence to v1 "reduce paniced"

* log if nil

* add logging

* fix: log output

* fix(import): check if org exists and user

* refactor: imports

* fix(user): ignore malformed events

* refactor: method naming

* fix: test

* refactor: correct errors.Is call

* ci: don't build dev binaries on main

* fix(go releaser): update version to 1.11.0

* fix(user): projection should not break

* fix(user): handle error properly

* docs: correct config example

* Update .releaserc.js

* Update .releaserc.js

Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Elio Bischof <eliobischof@gmail.com>
2022-08-31 07:52:43 +00:00
Livio Spring
12d4d3ea0b
fix: enable env vars in setup steps (and deprecate admin subcommand) (#3871)
* fix: enable env vars in setup steps (and deprecate admin subcommand)

* fix tests and error text
2022-06-27 10:32:34 +00:00
Florian Forster
fa9f581d56
chore(v2): move to new org (#3499)
* chore: move to new org

* logging

* fix: org rename caos -> zitadel

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2022-04-26 23:01:45 +00:00
Fabi
5132ebe07c
feat: add tenant column to eventstore (#3314)
* feat: add tenant column to eventstore

* feat: read tenant from context on push and filter

* Update 07_events_table.sql

* pass tenant to queryFactory

* fix some query tests

* init in tests

* add missing sql files

Co-authored-by: Livio Amstutz <livio.a@gmail.com>
2022-03-15 07:19:02 +01:00
Elio Bischof
51a76150ca
fix: revert migration V1.13 (#1466)
* fix: revert migration V1.13

* fix: set project changes correctly in application view model (#1469)

* fix: import user, hide login name suffix (#1474)

* fix: import user, and label policy command side

* feat: Import user and hide loginname suffix (#1464)

* fix: import user

* fix: label policy

* fix: label policy

* fix: label policy

* fix: migrations

* fix: migrations

* fix: migrations

* fix: label policy

* loginSuffix in login ui

* suffix

* fix cursor on disabled user selection

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

(cherry picked from commit 03ddb8fc38)

* feat: Import user and hide loginname suffix (#1464)

* fix: import user

* fix: label policy

* fix: label policy

* fix: label policy

* fix: migrations

* fix: migrations

* fix: migrations

* fix: label policy

* loginSuffix in login ui

* suffix

* fix cursor on disabled user selection

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

(cherry picked from commit 03ddb8fc38)

* feat: Import user and hide loginname suffix (#1464)

* fix: import user

* fix: label policy

* fix: label policy

* fix: label policy

* fix: migrations

* fix: migrations

* fix: migrations

* fix: label policy

* loginSuffix in login ui

* suffix

* fix cursor on disabled user selection

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

(cherry picked from commit 03ddb8fc38)

* fix: label policy events

* loginname placeholder

* fix: tests

* fix: tests

* Update internal/command/iam_policy_label_model.go

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* fix: console eventstore (#1476)

* fix org switch

* user grants

* fix project grants, user grant update

* fix idp removal, prettier buttons, attribute names

* fix: operator reconciling (#1478)

* fix(operator): align backup and restore commands (#1465)

* fix: crd mode broke backup and restore commands

* fix: remove obscure gitops-per-operator flags

(cherry picked from commit 041cacc4af)

* fix: gitops backup and restore need a kubernetes client too (#1475)

(cherry picked from commit 50bc317d27)

Co-authored-by: Elio Bischof <eliobischof@gmail.com>

* feat: features (#1427)

* features

* features

* features

* fix json tags

* add features handler to auth

* mocks for tests

* add setup step

* fixes

* add featurelist to auth api

* grandfather state and typos

* typo

* merge new-eventstore

* fix login policy tests

* label policy in features

* audit log retention

* fix: new es bug fixes (#1477)

* fix: displayname on members

* fix: user grant update

* fix: user grant id

* console grantid

Co-authored-by: Max Peintner <max@caos.ch>

* fix: backend (#1481)

* fix: check ids in proto

* fix sign out

* improve displayed login name after user selection

* fix init user in login

* fix init password in login

Co-authored-by: fabi <fabienne.gerschwiler@gmail.com>

* feat: console feature api (#1480)

* fix: changes (#1482)

* fix: changes

* change listquery to changequery

* fix: changes in auth

Co-authored-by: Max Peintner <max@caos.ch>

* test without transaction

* test without transaction

Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>
Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: fabi <fabienne.gerschwiler@gmail.com>
2021-03-29 13:38:04 +02:00
Livio Amstutz
8dcbbc87ca
fix: update config to commands (and queries) (#1342)
* fix: adaot config to commands (and queries)

* remove dependency on vv2 in v1

* add queries user to operator

* set password for queries on tests

* set password for queries on tests

* fix config
2021-02-24 11:17:39 +01:00
Fabi
d8e42744b4
fix: move v2 pkgs (#1331)
* fix: move eventstore pkgs

* fix: move eventstore pkgs

* fix: remove v2 view

* fix: remove v2 view
2021-02-23 15:13:04 +01:00