Commit Graph

258 Commits

Author SHA1 Message Date
Tim Möhlmann
aeb379e7de
fix(eventstore): revert precise decimal (#8527) (#8679) 2024-09-24 18:43:29 +02:00
Tim Möhlmann
3aba942162
feat: add debug events API (#8533)
# Which Problems Are Solved

Add a debug API which allows pushing a set of events to be reduced in a
dedicated projection.
The events can carry a sleep duration which simulates a slow query
during projection handling.

# How the Problems Are Solved

- `CreateDebugEvents` allows pushing multiple events which simulate the
lifecycle of a resource. Each event has a `projectionSleep` field, which
issues a `pg_sleep()` statement query in the projection handler :
  - Add
  - Change
  - Remove
- `ListDebugEventsStates` list the current state of the projection,
optionally with a Trigger
- `GetDebugEventsStateByID` get the current state of the aggregate ID in
the projection, optionally with a Trigger


# Additional Changes

- none

# Additional Context

-  Allows reproduction of https://github.com/zitadel/zitadel/issues/8517
2024-09-11 08:24:00 +00:00
Silvan
b522588d98
fix(eventstore): precise decimal (#8527)
# Which Problems Are Solved

Float64 which was used for the event.Position field is [not precise in
go and gets rounded](https://github.com/golang/go/issues/47300). This
can lead to unprecies position tracking of events and therefore
projections especially on cockcoachdb as the position used there is a
big number.

example of a unprecies position:
exact: 1725257931223002628
float64: 1725257931223002624.000000

# How the Problems Are Solved

The float64 was replaced by
[github.com/jackc/pgx-shopspring-decimal](https://github.com/jackc/pgx-shopspring-decimal).

# Additional Changes

Correct behaviour of makefile for load tests.
Rename `latestSequence`-queries to `latestPosition`
2024-09-06 12:19:19 +03:00
Livio Spring
9ec9ad4314
feat(oidc): sid claim for id_tokens issued through login V1 (#8525)
# Which Problems Are Solved

id_tokens issued for auth requests created through the login UI
currently do not provide a sid claim.
This is due to the fact that (SSO) sessions for the login UI do not have
one and are only computed by the userAgent(ID), the user(ID) and the
authentication checks of the latter.

This prevents client to track sessions and terminate specific session on
the end_session_endpoint.

# How the Problems Are Solved

- An `id` column is added to the `auth.user_sessions` table.
- The `id` (prefixed with `V1_`) is set whenever a session is added or
updated to active (from terminated)
- The id is passed to the `oidc session` (as v2 sessionIDs), to expose
it as `sid` claim

# Additional Changes

- refactored `getUpdateCols` to handle different column value types and
add arguments for query

# Additional Context

- closes #8499 
- relates to #8501
2024-09-03 13:19:00 +00:00
Tim Möhlmann
64a3bb3149
feat(v3alpha): web key resource (#8262)
# Which Problems Are Solved

Implement a new API service that allows management of OIDC signing web
keys.
This allows users to manage rotation of the instance level keys. which
are currently managed based on expiry.

The API accepts the generation of the following key types and
parameters:

- RSA keys with 2048, 3072 or 4096 bit in size and:
  - Signing with SHA-256 (RS256)
  - Signing with SHA-384 (RS384)
  - Signing with SHA-512 (RS512)
- ECDSA keys with
  - P256 curve
  - P384 curve
  - P512 curve
- ED25519 keys

# How the Problems Are Solved

Keys are serialized for storage using the JSON web key format from the
`jose` library. This is the format that will be used by OIDC for
signing, verification and publication.

Each instance can have a number of key pairs. All existing public keys
are meant to be used for token verification and publication the keys
endpoint. Keys can be activated and the active private key is meant to
sign new tokens. There is always exactly 1 active signing key:

1. When the first key for an instance is generated, it is automatically
activated.
2. Activation of the next key automatically deactivates the previously
active key.
3. Keys cannot be manually deactivated from the API
4. Active keys cannot be deleted

# Additional Changes

- Query methods that later will be used by the OIDC package are already
implemented. Preparation for #8031
- Fix indentation in french translation for instance event
- Move user_schema translations to consistent positions in all
translation files

# Additional Context

- Closes #8030
- Part of #7809

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-08-14 14:18:14 +00:00
Silvan
523d73f674
fix(fields): use read commit isolation level in trigger (#8410)
# Which Problems Are Solved

If the processing time of serializable transactions in the fields
handler take too long, the next iteration can fail.

# How the Problems Are Solved

Changed the isolation level of the current states query to Read Commited
2024-08-09 09:24:28 +00:00
Silvan
646ffe7a26
fix(fields): await running queries during trigger (#8391)
# Which Problems Are Solved

During triggering of the fields table WriteTooOld errors can occure when
using cockroachdb.

# How the Problems Are Solved

The statements exclusively lock the projection before they start to
insert data by using `FOR UPDATE`.
2024-08-06 11:27:28 +00:00
Silvan
99c645cc60
refactor(database): exchange connection pool (#8325)
# Which Problems Are Solved

The connection pool of go uses a high amount of database connections.

# How the Problems Are Solved

The standard lib connection pool was replaced by `pgxpool.Pool`

# Additional Changes

The `db.BeginTx`-spans are removed because they cause to much noise in
the traces.

# Additional Context

- part of https://github.com/zitadel/zitadel/issues/7639
2024-07-17 15:16:02 +00:00
Silvan
5ca8ad2075
fix(eventstore): early return if no events in field handler (#8254)
# Which Problems Are Solved

Fixes a panic which can occur if there are no events to reduce in the fields handler

# How the Problems Are Solved

Check if there are any events to reduce

# Additional Context

- Panic was added in https://github.com/zitadel/zitadel/pull/8191
2024-07-05 10:16:12 +02:00
Silvan
1d84635836
feat(eventstore): add search table (#8191)
# Which Problems Are Solved

To improve performance a new table and method is implemented on
eventstore. The goal of this table is to index searchable fields on
command side to use it on command and query side.

The table allows to store one primitive value (numeric, text) per row.

The eventstore framework is extended by the `Search`-method which allows
to search for objects.
The `Command`-interface is extended by the `SearchOperations()`-method
which does manipulate the the `search`-table.

# How the Problems Are Solved

This PR adds the capability of improving performance for command and
query side by using the `Search`-method of the eventstore instead of
using one of the `Filter`-methods.

# Open Tasks

- [x] Add feature flag
- [x] Unit tests
- [ ] ~~Benchmarks if needed~~
- [x] Ensure no behavior change
- [x] Add setup step to fill table with current data
- [x] Add projection which ensures data added between setup and start of
the new version are also added to the table

# Additional Changes

The `Search`-method is currently used by `ProjectGrant`-command side.

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/8094
2024-07-03 15:00:56 +00:00
Tim Möhlmann
f9742a58f4
fix(import): add tracing spans to all import related functions (#8160)
# Which Problems Are Solved

This fix adds tracing spans to all V1 API import related functions. This
is to troubleshoot import related performance issues reported to us.

# How the Problems Are Solved

Add a tracing span to `api/grpc/admin/import.go` and all related
functions that are called in the `command` package.

# Additional Changes

- none

# Additional Context

- Reported by internal communication
2024-06-19 12:56:33 +02:00
Livio Spring
931a6c7cce
perf(authZ): improve oidc session check (#8091)
# Which Problems Are Solved

Access token checks make sure that there have not been any termination
events (user locked, deactivated, signed out, ...) in the meantime. This
events were filtered based on the creation date of the last session
event, which might cause latency issues in the database.

# How the Problems Are Solved

- Changed the query to use `position` instead of `created_at`.
- removed `AwaitOpenTransactions`

# Additional Changes

Added the `position` field to the `ReadModel`.

# Additional Context

- relates to #8088
- part of #7639
- backport to 2.53.x
2024-06-12 09:11:36 +00:00
Silvan
2243306ef6
feat(cmd): mirror (#7004)
# Which Problems Are Solved

Adds the possibility to mirror an existing database to a new one. 

For that a new command was added `zitadel mirror`. Including it's
subcommands for a more fine grained mirror of the data.

Sub commands:

* `zitadel mirror eventstore`: copies only events and their unique
constraints
* `zitadel mirror system`: mirrors the data of the `system`-schema
*  `zitadel mirror projections`: runs all projections
*  `zitadel mirror auth`: copies auth requests
* `zitadel mirror verify`: counts the amount of rows in the source and
destination database and prints the diff.

The command requires one of the following flags:
* `--system`: copies all instances of the system
* `--instance <instance-id>`, `--instance <comma separated list of
instance ids>`: copies only the defined instances

The command is save to execute multiple times by adding the
`--replace`-flag. This replaces currently existing data except of the
`events`-table

# Additional Changes

A `--for-mirror`-flag was added to `zitadel setup` to prepare the new
database. The flag skips the creation of the first instances and initial
run of projections.

It is now possible to skip the creation of the first instance during
setup by setting `FirstInstance.Skip` to true in the steps
configuration.

# Additional info

It is currently not possible to merge multiple databases. See
https://github.com/zitadel/zitadel/issues/7964 for more details.

It is currently not possible to use files. See
https://github.com/zitadel/zitadel/issues/7966 for more information.

closes https://github.com/zitadel/zitadel/issues/7586
closes https://github.com/zitadel/zitadel/issues/7486

### Definition of Ready

- [x] I am happy with the code
- [x] Short description of the feature/issue is added in the pr
description
- [x] PR is linked to the corresponding user story
- [x] Acceptance criteria are met
- [x] All open todos and follow ups are defined in a new ticket and
justified
- [x] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [x] No debug or dead code
- [x] My code has no repetitions
- [x] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [x] Documentation/examples are up-to-date
- [x] All non-functional requirements are met
- [x] Functionality of the acceptance criteria is checked manually on
the dev system.

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-05-30 09:35:30 +00:00
Silvan
4dc86c2415
fix(eventstore): order by sequence if filter by aggregate id (#8019)
# Which Problems Are Solved

Queriying events by an aggregate id can produce high loads on the
database if the aggregate id contains many events (count > 1000000).

# How the Problems Are Solved

Instead of using the postion and in_tx_order columns we use the sequence
column which guarantees correct ordering in a single aggregate and uses
more optimised indexes.

# Additional Context

Closes https://github.com/zitadel/DevOps/issues/50

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-05-28 06:49:30 +00:00
Silvan
cff0f73e24
fix(init): add setting to enable durable locks on crdb (#7982)
feat(init): add setting to enable durable locks on crdb
2024-05-27 09:03:34 +00:00
Livio Spring
fb162a7d75
fix(login): improve auth handlers (#7969)
# Which Problems Are Solved

During the implementation of #7486 it was noticed, that projections in
the `auth` database schema could be blocked.
Investigations suggested, that this is due to the use of
[GORM](https://gorm.io/index.html) and it's inability to use an existing
(sql) transaction.
With the improved / simplified handling (see below) there should also be
a minimal improvement in performance, resp. reduced database update
statements.

# How the Problems Are Solved

The handlers in `auth` are exchanged to proper (sql) statements and gorm
usage is removed for any writing part.
To further improve / simplify the handling of the users, a new
`auth.users3` table is created, where only attributes are handled, which
are not yet available from the `projections.users`,
`projections.login_name` and `projections.user_auth_methods` do not
provide. This reduces the events handled in that specific handler by a
lot.

# Additional Changes

None

# Additional Context

relates to #7486
2024-05-22 15:26:02 +00:00
Tim Möhlmann
029a6d393a
fix(crdb): obtain latest sequences when the tx is retried (#7795) 2024-04-18 13:07:05 +00:00
Stefan Benz
6dcdef0268
fix: add action v2 execution to features (#7597)
* fix: add action v2 execution to features

* fix: add action v2 execution to features

* fix: add action v2 execution to features

* fix: update internal/command/instance_features_model.go

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>

* fix: merge back main

* fix: merge back main

* fix: rename feature and service

* fix: rename feature and service

* fix: review changes

* fix: review changes

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2024-04-09 20:21:21 +03:00
wanshuangcheng
a096b7e5d4
chore: fix function names in comment (#7723)
Signed-off-by: wanshuangcheng <wanshuangcheng@outlook.com>
2024-04-08 08:15:44 +00:00
Tim Möhlmann
25ef3da9d5
refactor(fmt): run gci on complete project (#7557)
chore(fmt): run gci on complete project

Fix global import formatting in go code by running the `gci` command. This allows us to just use the command directly, instead of fixing the import order manually for the linter, on each PR.

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-04-03 10:43:43 +00:00
Tim Möhlmann
093dd57a78
fix(db): wrap BeginTx in spans to get acquire metrics (#7689)
feat(db): wrap BeginTx in spans to get acquire metrics

This changes adds a span around most db.BeginTx calls so we can get tracings about the connection pool acquire process.
This might help us pinpoint why sometimes some query package traces show longer execution times, while this was not reflected on database side execution times.

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2024-04-03 11:48:24 +03:00
Silvan
56df515e5f
chore: use pgx v5 (#7577)
* chore: use pgx v5

* chore: update go version

* remove direct pq dependency

* remove unnecessary type

* scan test

* map scanner

* converter

* uint8 number array

* duration

* most unit tests work

* unit tests work

* chore: coverage

* go 1.21

* linting

* int64 gopfertammi

* retry go 1.22

* retry go 1.22

* revert to go v1.21.5

* update go toolchain to 1.21.8

* go 1.21.8

* remove test flag

* go 1.21.5

* linting

* update toolchain

* use correct array

* use correct array

* add byte array

* correct value

* correct error message

* go 1.21 compatible
2024-03-27 15:48:22 +02:00
Livio Spring
7494a7b6d9
feat(api): add possibility to retrieve user schemas (#7614)
This PR extends the user schema service (V3 API) with the possibility to ListUserSchemas and GetUserSchemaByID.
The previously started guide is extended to demonstrate how to retrieve the schema(s) and notes the generated revision property.
2024-03-22 13:26:13 +00:00
Silvan
4224c7ad3a
chore(core): update dependencies (#7517)
* chore(core): update dependencies

* chore(core): update dependencies
2024-03-11 10:43:56 +00:00
Silvan
60ee2610f2
fix(eventstore): consider IsGlobal-flag of constraints (#7518)
* fix(eventstore): consider `IsGlobal`-flag of constraints

* fix(setup): set `instance_domain`-constraint global
2024-03-08 13:33:53 +00:00
Silvan
ec3076c3c8
fix(ListEvents): add aggregate types to filter if not set (#7490) 2024-03-05 15:44:51 +00:00
Elio Bischof
38777b478e
perf: filter events by instance ids (#7489)
fix: filter events by instance ids
2024-03-04 07:56:48 +01:00
Tim Möhlmann
26d1563643
feat(api): feature flags (#7356)
* feat(api): feature API proto definitions

* update proto based on discussion with @livio-a

* cleanup old feature flag stuff

* authz instance queries

* align defaults

* projection definitions

* define commands and event reducers

* implement system and instance setter APIs

* api getter implementation

* unit test repository package

* command unit tests

* unit test Get queries

* grpc converter unit tests

* migrate the V1 features

* migrate oidc to dynamic features

* projection unit test

* fix instance by host

* fix instance by id data type in sql

* fix linting errors

* add system projection test

* fix behavior inversion

* resolve proto file comments

* rename SystemDefaultLoginInstanceEventType to SystemLoginDefaultOrgEventType so it's consistent with the instance level event

* use write models and conditional set events

* system features integration tests

* instance features integration tests

* error on empty request

* documentation entry

* typo in feature.proto

* fix start unit tests

* solve linting error on key case switch

* remove system defaults after discussion with @eliobischof

* fix system feature projection

* resolve comments in defaults.yaml

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-02-28 10:55:54 +02:00
Tim Möhlmann
1890e28f79
fix(eventstore): retry push on primary key sequence collision (#7420)
* fix(eventstore): retry push on primary key sequence collision

* MaxRetries config option and unit test
2024-02-23 08:29:10 +00:00
Livio Spring
e000fdd792
fix: handle context correctly in processEvents (#7320) 2024-01-31 11:25:28 +01:00
Silvan
17953e9040
fix(setup): init projections (#7194)
Even though this is a feature it's released as fix so that we can back port to earlier revisions.

As reported by multiple users startup of ZITADEL after leaded to downtime and worst case rollbacks to the previously deployed version.

The problem starts rising when there are too many events to process after the start of ZITADEL. The root cause are changes on projections (database tables) which must be recomputed. This PR solves this problem by adding a new step to the setup phase which prefills the projections. The step can be enabled by adding the `--init-projections`-flag to `setup`, `start-from-init` and `start-from-setup`. Setting this flag results in potentially longer duration of the setup phase but reduces the risk of the problems mentioned in the paragraph above.
2024-01-25 17:28:20 +01:00
Livio Spring
8470649ecb
chore: pin crdb version for unit tests (#7260)
* chore: pin crdb version for unit tests

* use latest 23.1 version

* use latest available 23.1 version
2024-01-18 08:16:54 +00:00
Elio Bischof
ed0bc39ea4
feat: block instances (#7129)
* docs: fix init description typos

* feat: block instances using limits

* translate

* unit tests

* fix translations

* redirect /ui/login

* fix http interceptor

* cleanup

* fix http interceptor

* fix: delete cookies on gateway 200

* add integration tests

* add command test

* docs

* fix integration tests

* add bulk api and integration test

* optimize bulk set limits

* unit test bulk limits

* fix broken link

* fix assets middleware

* fix broken link

* validate instance id format

* Update internal/eventstore/search_query.go

Co-authored-by: Livio Spring <livio.a@gmail.com>

* remove support for owner bulk limit commands

* project limits to instances

* migrate instances projection

* Revert "migrate instances projection"

This reverts commit 214218732a.

* join limits, remove owner

* remove todo

* use optional bool

* normally validate instance ids

* use 302

* cleanup

* cleanup

* Update internal/api/grpc/system/limits_converter.go

Co-authored-by: Livio Spring <livio.a@gmail.com>

* remove owner

* remove owner from reset

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-01-17 10:16:48 +00:00
Silvan
3c5fc31372
fix(handler): handle trigger err correctly (#7205) 2024-01-11 17:55:50 +00:00
Silvan
aa2d642e97
fix(handler): updated failed events (#7146) 2024-01-04 21:36:08 +00:00
Silvan
b7d027e2fd
fix(db): always use begin tx (#7142)
* fix(db): always use begin tx

* fix(handler): timeout for begin
2024-01-04 16:12:20 +00:00
Livio Spring
c0cef4983a
fix: correctly respect maxFailureCount (#7143) 2024-01-04 15:46:25 +00:00
Silvan
cc2dd8b20b
fix(eventstore): increase performance on push (#7125) 2023-12-31 15:30:25 +01:00
Silvan
5ce542b959
fix(handler): allow uint32 offset for migration scenarios (#7103) 2023-12-21 10:40:51 +00:00
Tim Möhlmann
fe1337536f
fix(db): add additional connection pool for projection spooling (#7094)
* fix(db): add additional connection pool for projection spooling

* use correct connection pool for projections

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-12-20 16:13:04 +00:00
Livio Spring
edaa41903e
fix(projections): handle every instance by default and randomize start (#7093) 2023-12-19 13:32:08 +02:00
Tim Möhlmann
f680dd934d
refactor: rename package errors to zerrors (#7039)
* chore: rename package errors to zerrors

* rename package errors to gerrors

* fix error related linting issues

* fix zitadel error assertion

* fix gosimple linting issues

* fix deprecated linting issues

* resolve gci linting issues

* fix import structure

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2023-12-08 15:30:55 +01:00
Tim Möhlmann
ec03340b67
perf(oidc): optimize client verification (#6999)
* fix some spelling errors

* client credential auth

* implementation of client auth

* improve error handling

* unit test command package

* unit test database package

* unit test query package

* cleanup unused tracing func

* fix integration tests

* errz to zerrors

* fix linting and import issues

* fix another linting error

* integration test with client secret

* Revert "integration test with client secret"

This reverts commit 0814ba522f.

* add integration tests

* client credentials integration test

* resolve comments

* pin oidc v3.5.0
2023-12-05 17:01:03 +00:00
Elio Bischof
dd33538c0a
feat: restrict languages (#6931)
* feat: return 404 or 409 if org reg disallowed

* fix: system limit permissions

* feat: add iam limits api

* feat: disallow public org registrations on default instance

* add integration test

* test: integration

* fix test

* docs: describe public org registrations

* avoid updating docs deps

* fix system limits integration test

* silence integration tests

* fix linting

* ignore strange linter complaints

* review

* improve reset properties naming

* redefine the api

* use restrictions aggregate

* test query

* simplify and test projection

* test commands

* fix unit tests

* move integration test

* support restrictions on default instance

* also test GetRestrictions

* self review

* lint

* abstract away resource owner

* fix tests

* configure supported languages

* fix allowed languages

* fix tests

* default lang must not be restricted

* preferred language must be allowed

* change preferred languages

* check languages everywhere

* lint

* test command side

* lint

* add integration test

* add integration test

* restrict supported ui locales

* lint

* lint

* cleanup

* lint

* allow undefined preferred language

* fix integration tests

* update main

* fix env var

* ignore linter

* ignore linter

* improve integration test config

* reduce cognitive complexity

* compile

* check for duplicates

* remove useless restriction checks

* review

* revert restriction renaming

* fix language restrictions

* lint

* generate

* allow custom texts for supported langs for now

* fix tests

* cleanup

* cleanup

* cleanup

* lint

* unsupported preferred lang is allowed

* fix integration test

* finish reverting to old property name

* finish reverting to old property name

* load languages

* refactor(i18n): centralize translators and fs

* lint

* amplify no validations on preferred languages

* fix integration test

* lint

* fix resetting allowed languages

* test unchanged restrictions
2023-12-05 11:12:01 +00:00
Silvan
e3d1ca4d58
fix(eventstore): improve pagination of handler filter (#6968)
* fix(setup): add filter_offset to `projections.current_states`

* fix(eventstore): allow offset in query

* fix(handler): offset for already processed events
2023-12-01 12:25:41 +00:00
Livio Spring
e57076430b
fix: handle context when locking for trigger (#7006) 2023-12-01 11:13:57 +01:00
Elio Bischof
8982e1aae3
fix(postgres <=15): delete unique constraints (#6971)
fix(postgres): delete unique constraints
2023-11-24 07:23:23 +01:00
Silvan
9ed956383f
fix(eventstore): correct handling of wrong unique fields (#6961) 2023-11-23 06:15:40 +01:00
Livio Spring
b563041103
fix: ensure uniqueness (#6956)
* fix: ensure uniqueness

* only update wrong ones

* Update cmd/setup/16.go

Co-authored-by: Silvan <silvan.reusser@gmail.com>

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-11-22 12:05:14 +00:00
Tim Möhlmann
2f91679623
chore(Makefile): add go generate target (#6944)
This change adds a core_generate_all make target.
It installs the required tools and runs generate on the complete project.

`golang/mock` is no longer maintained and a fork is available
from the Uber folks. So the latter is used as tool.
All the mock files have been regenerated and are part of the PR.

The obsolete `tools` directory has been removed,
as all the tools are now part of specific make targets.

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-11-22 10:56:43 +00:00