Commit Graph

245 Commits

Author SHA1 Message Date
Silvan
4dc86c2415
fix(eventstore): order by sequence if filter by aggregate id (#8019)
# Which Problems Are Solved

Queriying events by an aggregate id can produce high loads on the
database if the aggregate id contains many events (count > 1000000).

# How the Problems Are Solved

Instead of using the postion and in_tx_order columns we use the sequence
column which guarantees correct ordering in a single aggregate and uses
more optimised indexes.

# Additional Context

Closes https://github.com/zitadel/DevOps/issues/50

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-05-28 06:49:30 +00:00
Silvan
cff0f73e24
fix(init): add setting to enable durable locks on crdb (#7982)
feat(init): add setting to enable durable locks on crdb
2024-05-27 09:03:34 +00:00
Livio Spring
fb162a7d75
fix(login): improve auth handlers (#7969)
# Which Problems Are Solved

During the implementation of #7486 it was noticed, that projections in
the `auth` database schema could be blocked.
Investigations suggested, that this is due to the use of
[GORM](https://gorm.io/index.html) and it's inability to use an existing
(sql) transaction.
With the improved / simplified handling (see below) there should also be
a minimal improvement in performance, resp. reduced database update
statements.

# How the Problems Are Solved

The handlers in `auth` are exchanged to proper (sql) statements and gorm
usage is removed for any writing part.
To further improve / simplify the handling of the users, a new
`auth.users3` table is created, where only attributes are handled, which
are not yet available from the `projections.users`,
`projections.login_name` and `projections.user_auth_methods` do not
provide. This reduces the events handled in that specific handler by a
lot.

# Additional Changes

None

# Additional Context

relates to #7486
2024-05-22 15:26:02 +00:00
Tim Möhlmann
029a6d393a
fix(crdb): obtain latest sequences when the tx is retried (#7795) 2024-04-18 13:07:05 +00:00
Stefan Benz
6dcdef0268
fix: add action v2 execution to features (#7597)
* fix: add action v2 execution to features

* fix: add action v2 execution to features

* fix: add action v2 execution to features

* fix: update internal/command/instance_features_model.go

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>

* fix: merge back main

* fix: merge back main

* fix: rename feature and service

* fix: rename feature and service

* fix: review changes

* fix: review changes

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2024-04-09 20:21:21 +03:00
wanshuangcheng
a096b7e5d4
chore: fix function names in comment (#7723)
Signed-off-by: wanshuangcheng <wanshuangcheng@outlook.com>
2024-04-08 08:15:44 +00:00
Tim Möhlmann
25ef3da9d5
refactor(fmt): run gci on complete project (#7557)
chore(fmt): run gci on complete project

Fix global import formatting in go code by running the `gci` command. This allows us to just use the command directly, instead of fixing the import order manually for the linter, on each PR.

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-04-03 10:43:43 +00:00
Tim Möhlmann
093dd57a78
fix(db): wrap BeginTx in spans to get acquire metrics (#7689)
feat(db): wrap BeginTx in spans to get acquire metrics

This changes adds a span around most db.BeginTx calls so we can get tracings about the connection pool acquire process.
This might help us pinpoint why sometimes some query package traces show longer execution times, while this was not reflected on database side execution times.

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2024-04-03 11:48:24 +03:00
Silvan
56df515e5f
chore: use pgx v5 (#7577)
* chore: use pgx v5

* chore: update go version

* remove direct pq dependency

* remove unnecessary type

* scan test

* map scanner

* converter

* uint8 number array

* duration

* most unit tests work

* unit tests work

* chore: coverage

* go 1.21

* linting

* int64 gopfertammi

* retry go 1.22

* retry go 1.22

* revert to go v1.21.5

* update go toolchain to 1.21.8

* go 1.21.8

* remove test flag

* go 1.21.5

* linting

* update toolchain

* use correct array

* use correct array

* add byte array

* correct value

* correct error message

* go 1.21 compatible
2024-03-27 15:48:22 +02:00
Livio Spring
7494a7b6d9
feat(api): add possibility to retrieve user schemas (#7614)
This PR extends the user schema service (V3 API) with the possibility to ListUserSchemas and GetUserSchemaByID.
The previously started guide is extended to demonstrate how to retrieve the schema(s) and notes the generated revision property.
2024-03-22 13:26:13 +00:00
Silvan
4224c7ad3a
chore(core): update dependencies (#7517)
* chore(core): update dependencies

* chore(core): update dependencies
2024-03-11 10:43:56 +00:00
Silvan
60ee2610f2
fix(eventstore): consider IsGlobal-flag of constraints (#7518)
* fix(eventstore): consider `IsGlobal`-flag of constraints

* fix(setup): set `instance_domain`-constraint global
2024-03-08 13:33:53 +00:00
Silvan
ec3076c3c8
fix(ListEvents): add aggregate types to filter if not set (#7490) 2024-03-05 15:44:51 +00:00
Elio Bischof
38777b478e
perf: filter events by instance ids (#7489)
fix: filter events by instance ids
2024-03-04 07:56:48 +01:00
Tim Möhlmann
26d1563643
feat(api): feature flags (#7356)
* feat(api): feature API proto definitions

* update proto based on discussion with @livio-a

* cleanup old feature flag stuff

* authz instance queries

* align defaults

* projection definitions

* define commands and event reducers

* implement system and instance setter APIs

* api getter implementation

* unit test repository package

* command unit tests

* unit test Get queries

* grpc converter unit tests

* migrate the V1 features

* migrate oidc to dynamic features

* projection unit test

* fix instance by host

* fix instance by id data type in sql

* fix linting errors

* add system projection test

* fix behavior inversion

* resolve proto file comments

* rename SystemDefaultLoginInstanceEventType to SystemLoginDefaultOrgEventType so it's consistent with the instance level event

* use write models and conditional set events

* system features integration tests

* instance features integration tests

* error on empty request

* documentation entry

* typo in feature.proto

* fix start unit tests

* solve linting error on key case switch

* remove system defaults after discussion with @eliobischof

* fix system feature projection

* resolve comments in defaults.yaml

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-02-28 10:55:54 +02:00
Tim Möhlmann
1890e28f79
fix(eventstore): retry push on primary key sequence collision (#7420)
* fix(eventstore): retry push on primary key sequence collision

* MaxRetries config option and unit test
2024-02-23 08:29:10 +00:00
Livio Spring
e000fdd792
fix: handle context correctly in processEvents (#7320) 2024-01-31 11:25:28 +01:00
Silvan
17953e9040
fix(setup): init projections (#7194)
Even though this is a feature it's released as fix so that we can back port to earlier revisions.

As reported by multiple users startup of ZITADEL after leaded to downtime and worst case rollbacks to the previously deployed version.

The problem starts rising when there are too many events to process after the start of ZITADEL. The root cause are changes on projections (database tables) which must be recomputed. This PR solves this problem by adding a new step to the setup phase which prefills the projections. The step can be enabled by adding the `--init-projections`-flag to `setup`, `start-from-init` and `start-from-setup`. Setting this flag results in potentially longer duration of the setup phase but reduces the risk of the problems mentioned in the paragraph above.
2024-01-25 17:28:20 +01:00
Livio Spring
8470649ecb
chore: pin crdb version for unit tests (#7260)
* chore: pin crdb version for unit tests

* use latest 23.1 version

* use latest available 23.1 version
2024-01-18 08:16:54 +00:00
Elio Bischof
ed0bc39ea4
feat: block instances (#7129)
* docs: fix init description typos

* feat: block instances using limits

* translate

* unit tests

* fix translations

* redirect /ui/login

* fix http interceptor

* cleanup

* fix http interceptor

* fix: delete cookies on gateway 200

* add integration tests

* add command test

* docs

* fix integration tests

* add bulk api and integration test

* optimize bulk set limits

* unit test bulk limits

* fix broken link

* fix assets middleware

* fix broken link

* validate instance id format

* Update internal/eventstore/search_query.go

Co-authored-by: Livio Spring <livio.a@gmail.com>

* remove support for owner bulk limit commands

* project limits to instances

* migrate instances projection

* Revert "migrate instances projection"

This reverts commit 214218732a.

* join limits, remove owner

* remove todo

* use optional bool

* normally validate instance ids

* use 302

* cleanup

* cleanup

* Update internal/api/grpc/system/limits_converter.go

Co-authored-by: Livio Spring <livio.a@gmail.com>

* remove owner

* remove owner from reset

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-01-17 10:16:48 +00:00
Silvan
3c5fc31372
fix(handler): handle trigger err correctly (#7205) 2024-01-11 17:55:50 +00:00
Silvan
aa2d642e97
fix(handler): updated failed events (#7146) 2024-01-04 21:36:08 +00:00
Silvan
b7d027e2fd
fix(db): always use begin tx (#7142)
* fix(db): always use begin tx

* fix(handler): timeout for begin
2024-01-04 16:12:20 +00:00
Livio Spring
c0cef4983a
fix: correctly respect maxFailureCount (#7143) 2024-01-04 15:46:25 +00:00
Silvan
cc2dd8b20b
fix(eventstore): increase performance on push (#7125) 2023-12-31 15:30:25 +01:00
Silvan
5ce542b959
fix(handler): allow uint32 offset for migration scenarios (#7103) 2023-12-21 10:40:51 +00:00
Tim Möhlmann
fe1337536f
fix(db): add additional connection pool for projection spooling (#7094)
* fix(db): add additional connection pool for projection spooling

* use correct connection pool for projections

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-12-20 16:13:04 +00:00
Livio Spring
edaa41903e
fix(projections): handle every instance by default and randomize start (#7093) 2023-12-19 13:32:08 +02:00
Tim Möhlmann
f680dd934d
refactor: rename package errors to zerrors (#7039)
* chore: rename package errors to zerrors

* rename package errors to gerrors

* fix error related linting issues

* fix zitadel error assertion

* fix gosimple linting issues

* fix deprecated linting issues

* resolve gci linting issues

* fix import structure

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2023-12-08 15:30:55 +01:00
Tim Möhlmann
ec03340b67
perf(oidc): optimize client verification (#6999)
* fix some spelling errors

* client credential auth

* implementation of client auth

* improve error handling

* unit test command package

* unit test database package

* unit test query package

* cleanup unused tracing func

* fix integration tests

* errz to zerrors

* fix linting and import issues

* fix another linting error

* integration test with client secret

* Revert "integration test with client secret"

This reverts commit 0814ba522f.

* add integration tests

* client credentials integration test

* resolve comments

* pin oidc v3.5.0
2023-12-05 17:01:03 +00:00
Elio Bischof
dd33538c0a
feat: restrict languages (#6931)
* feat: return 404 or 409 if org reg disallowed

* fix: system limit permissions

* feat: add iam limits api

* feat: disallow public org registrations on default instance

* add integration test

* test: integration

* fix test

* docs: describe public org registrations

* avoid updating docs deps

* fix system limits integration test

* silence integration tests

* fix linting

* ignore strange linter complaints

* review

* improve reset properties naming

* redefine the api

* use restrictions aggregate

* test query

* simplify and test projection

* test commands

* fix unit tests

* move integration test

* support restrictions on default instance

* also test GetRestrictions

* self review

* lint

* abstract away resource owner

* fix tests

* configure supported languages

* fix allowed languages

* fix tests

* default lang must not be restricted

* preferred language must be allowed

* change preferred languages

* check languages everywhere

* lint

* test command side

* lint

* add integration test

* add integration test

* restrict supported ui locales

* lint

* lint

* cleanup

* lint

* allow undefined preferred language

* fix integration tests

* update main

* fix env var

* ignore linter

* ignore linter

* improve integration test config

* reduce cognitive complexity

* compile

* check for duplicates

* remove useless restriction checks

* review

* revert restriction renaming

* fix language restrictions

* lint

* generate

* allow custom texts for supported langs for now

* fix tests

* cleanup

* cleanup

* cleanup

* lint

* unsupported preferred lang is allowed

* fix integration test

* finish reverting to old property name

* finish reverting to old property name

* load languages

* refactor(i18n): centralize translators and fs

* lint

* amplify no validations on preferred languages

* fix integration test

* lint

* fix resetting allowed languages

* test unchanged restrictions
2023-12-05 11:12:01 +00:00
Silvan
e3d1ca4d58
fix(eventstore): improve pagination of handler filter (#6968)
* fix(setup): add filter_offset to `projections.current_states`

* fix(eventstore): allow offset in query

* fix(handler): offset for already processed events
2023-12-01 12:25:41 +00:00
Livio Spring
e57076430b
fix: handle context when locking for trigger (#7006) 2023-12-01 11:13:57 +01:00
Elio Bischof
8982e1aae3
fix(postgres <=15): delete unique constraints (#6971)
fix(postgres): delete unique constraints
2023-11-24 07:23:23 +01:00
Silvan
9ed956383f
fix(eventstore): correct handling of wrong unique fields (#6961) 2023-11-23 06:15:40 +01:00
Livio Spring
b563041103
fix: ensure uniqueness (#6956)
* fix: ensure uniqueness

* only update wrong ones

* Update cmd/setup/16.go

Co-authored-by: Silvan <silvan.reusser@gmail.com>

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-11-22 12:05:14 +00:00
Tim Möhlmann
2f91679623
chore(Makefile): add go generate target (#6944)
This change adds a core_generate_all make target.
It installs the required tools and runs generate on the complete project.

`golang/mock` is no longer maintained and a fork is available
from the Uber folks. So the latter is used as tool.
All the mock files have been regenerated and are part of the PR.

The obsolete `tools` directory has been removed,
as all the tools are now part of specific make targets.

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-11-22 10:56:43 +00:00
Elio Bischof
76fe032b5f
feat: option to disallow public org registration (#6917)
* feat: return 404 or 409 if org reg disallowed

* fix: system limit permissions

* feat: add iam limits api

* feat: disallow public org registrations on default instance

* add integration test

* test: integration

* fix test

* docs: describe public org registrations

* avoid updating docs deps

* fix system limits integration test

* silence integration tests

* fix linting

* ignore strange linter complaints

* review

* improve reset properties naming

* redefine the api

* use restrictions aggregate

* test query

* simplify and test projection

* test commands

* fix unit tests

* move integration test

* support restrictions on default instance

* also test GetRestrictions

* self review

* lint

* abstract away resource owner

* fix tests

* lint
2023-11-22 09:29:38 +00:00
Tim Möhlmann
ba9b807854
perf(oidc): optimize the introspection endpoint (#6909)
* get key by id and cache them

* userinfo from events for v2 tokens

* improve keyset caching

* concurrent token and client checks

* client and project in single query

* logging and otel

* drop owner_removed column on apps and authN tables

* userinfo and project roles in go routines

* get  oidc user info from projections and add actions

* add avatar URL

* some cleanup

* pull oidc work branch

* remove storage from server

* add config flag for experimental introspection

* legacy introspection flag

* drop owner_removed column on user projections

* drop owner_removed column on useer_metadata

* query userinfo unit test

* query introspection client test

* add user_grants to the userinfo query

* handle PAT scopes

* bring triggers back

* test instance keys query

* add userinfo unit tests

* unit test keys

* go mod tidy

* solve some bugs

* fix missing preferred login name

* do not run triggers in go routines, they seem to deadlock

* initialize the trigger handlers late with a sync.OnceValue

* Revert "do not run triggers in go routines, they seem to deadlock"

This reverts commit 2a03da2127.

* add missing translations

* chore: update go version for linting

* pin oidc version

* parse a global time location for query test

* fix linter complains

* upgrade go lint

* fix more linting issues

---------

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2023-11-21 13:11:38 +01:00
Elio Bischof
1b6e3dcf27
fix: creation date argument in search events filters (#6855)
* fix: creation date filter in event queries

* fix: creation date with ordering filter

* simplify code

* simplify review

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-11-03 14:52:48 +00:00
Silvan
f8bf8ea256
fix(eventstore): differentiate unique constraint error (#6832)
* fix(eventstore): differentiate unique constraint error format

* docs: add comment to eventstore vars

* fix(eventstore): return correct error type if unique constraint already exists
2023-10-27 14:10:01 +02:00
Silvan
0187487f26
chore: correct tracing of trigger function (#6825)
* chore: correct tracing of trigger function

* refactor: remove import
2023-10-26 17:07:56 +02:00
Tim Möhlmann
ab79855cf0
fix(eventstore): prevent allocation of filtered events (#6749)
* fix(eventstore): prevent allocation of filtered events

Directly reduce each event obtained from a sql.Rows scan,
so that we do not have to allocate all events in a slice.

* reinstate the mutex as RWMutex

* scan data directly

* add todos

* fix(writemodels): add reduce of parent

* test: remove comment

* update comments

---------

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2023-10-19 15:21:31 +00:00
Silvan
b5564572bc
feat(eventstore): increase parallel write capabilities (#5940)
This implementation increases parallel write capabilities of the eventstore.
Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and  [06](https://zitadel.com/docs/support/advisory/a10006).
The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`.
If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.
2023-10-19 12:19:10 +02:00
Elio Bischof
1d4ec6cdba
fix: set quotas (#6597)
* feat: set quotas

* fix: start new period on younger anchor

* cleanup e2e config

* fix set notifications

* lint

* test: fix quota projection tests

* fix add quota tests

* make quota fields nullable

* enable amount 0

* fix initial setup

* create a prerelease

* avoid success comments

* fix quota projection primary key

* Revert "fix quota projection primary key"

This reverts commit e72f4d7fa1.

* simplify write model

* fix aggregate id

* avoid push without changes

* test set quota lifecycle

* test set quota mutations

* fix quota unit test

* fix: quotas

* test quota.set event projection

* use SetQuota in integration tests

* fix: release quotas 3

* reset releaserc

* fix comment

* test notification order doesn't matter

* test notification order doesn't matter

* test with unmarshalled events

* test with unmarshalled events

(cherry picked from commit ae1af6bc8c)
2023-09-22 13:06:59 +02:00
Silvan
e64428b8d4
fix: TestCRDB_CreateInstance (#6522)
(cherry picked from commit 856d0d12e8)
2023-09-12 17:06:01 +02:00
Livio Spring
f5060d70d7 fix: handle password data correct on user creation (#6515) 2023-09-07 16:21:10 +02:00
Silvan
2d75576864 fix(scheduler): add span to trigger method (#6513) 2023-09-07 11:01:18 +02:00
Silvan
f72560291e
fix(eventstore): cache instances (#6501)
* fix(eventstore): cache instances

* fix: consider succeeded once during instance ids query

* fix(eventstore): return correct instances

(cherry picked from commit 0f06e84f40)
2023-09-07 06:23:28 +02:00
Silvan
22af4dcd97
feat(storage): read only transactions (#6417)
feat(storage): read only transactions for queries (#6415)

* fix: tests

* bastle wie en grosse

* fix(database): scan as callback

* fix tests

* fix merge failures

* remove as of system time

* refactor: remove unused test

* refacotr: remove unused lines
2023-08-22 14:49:02 +02:00