14 Commits

Author	SHA1	Message	Date
Lars	1915d35605	feat: list users scim v2 endpoint (#9187) ... # Which Problems Are Solved - Adds support for the list users SCIM v2 endpoint # How the Problems Are Solved - Adds support for the list users SCIM v2 endpoints under `GET /scim/v2/{orgID}/Users` and `POST /scim/v2/{orgID}/Users/.search` # Additional Changes - adds a new function `SearchUserMetadataForUsers` to the query layer to query a metadata keyset for given user ids - adds a new function `NewUserMetadataExistsQuery` to the query layer to query a given metadata key value pair exists - adds a new function `CountUsers` to the query layer to count users without reading any rows - handle `ErrorAlreadyExists` as scim errors `uniqueness` - adds `NumberLessOrEqual` and `NumberGreaterOrEqual` query comparison methods - adds `BytesQuery` with `BytesEquals` and `BytesNotEquals` query comparison methods # Additional Context Part of #8140 Supported fields for scim filters: * `meta.created` * `meta.lastModified` * `id` * `username` * `name.familyName` * `name.givenName` * `emails` and `emails.value` * `active` only eq and ne * `externalId` only eq and ne	2025-01-21 13:31:54 +01:00
Silvan	9c3e5e467b	perf(query): remove transactions for queries (#8614) ... # Which Problems Are Solved Queries currently execute 3 statements, begin, query, commit # How the Problems Are Solved remove transaction handling from query methods in database package # Additional Changes - Bump versions of `core_grpc_dependencies`-receipt in Makefile # Additional info During load tests we saw a lot of idle transactions of `zitadel_queries` application name which is the connection pool used to query data in zitadel. Executed query: `select query_start - xact_start, pid, application_name, backend_start, xact_start, query_start, state_change, wait_event_type, wait_event,substring(query, 1, 200) query from pg_stat_activity where datname = 'zitadel' and state <> 'idle';` Mostly the last query executed was `begin isolation level read committed read only`. example: ``` ?column? | pid | application_name | backend_start | xact_start | query_start | state_change | wait_event_type | wait_event | query -----------------+-------+----------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 00:00:00 | 33030 | zitadel_queries | 2024-10-16 16:25:53.906036+00 | 2024-10-16 16:30:19.191661+00 | 2024-10-16 16:30:19.191661+00 | 2024-10-16 16:30:19.19169+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00 | 33035 | zitadel_queries | 2024-10-16 16:25:53.909629+00 | 2024-10-16 16:30:19.19179+00 | 2024-10-16 16:30:19.19179+00 | 2024-10-16 16:30:19.191805+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00.00412 | 33028 | zitadel_queries | 2024-10-16 16:25:53.904247+00 | 2024-10-16 16:30:19.187734+00 | 2024-10-16 16:30:19.191854+00 | 2024-10-16 16:30:19.191964+00 | Client | ClientRead | SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type 00:00:00.084662 | 33134 | zitadel_es_pusher | 2024-10-16 16:29:54.979692+00 | 2024-10-16 16:30:19.178578+00 | 2024-10-16 16:30:19.26324+00 | 2024-10-16 16:30:19.263267+00 | Client | ClientRead | RELEASE SAVEPOINT cockroach_restart 00:00:00.084768 | 33139 | zitadel_es_pusher | 2024-10-16 16:29:54.979585+00 | 2024-10-16 16:30:19.180762+00 | 2024-10-16 16:30:19.26553+00 | 2024-10-16 16:30:19.265531+00 | LWLock | WALWriteLock | commit 00:00:00.077377 | 33136 | zitadel_es_pusher | 2024-10-16 16:29:54.978582+00 | 2024-10-16 16:30:19.187883+00 | 2024-10-16 16:30:19.26526+00 | 2024-10-16 16:30:19.265431+00 | Client | ClientRead | WITH existing AS ( + | | | | | | | | | (SELECT instance_id, aggregate_type, aggregate_id, "sequence" FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND aggregate_id = $3 ORDER BY "sequence" DE 00:00:00.012309 | 33123 | zitadel_es_pusher | 2024-10-16 16:29:54.963484+00 | 2024-10-16 16:30:19.175066+00 | 2024-10-16 16:30:19.187375+00 | 2024-10-16 16:30:19.187376+00 | IO | WalSync | commit 00:00:00 | 33034 | zitadel_queries | 2024-10-16 16:25:53.90791+00 | 2024-10-16 16:30:19.262921+00 | 2024-10-16 16:30:19.262921+00 | 2024-10-16 16:30:19.263133+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00 | 33039 | zitadel_queries | 2024-10-16 16:25:53.914106+00 | 2024-10-16 16:30:19.191676+00 | 2024-10-16 16:30:19.191676+00 | 2024-10-16 16:30:19.191687+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00.24539 | 33083 | zitadel_projection_spooler | 2024-10-16 16:27:49.895548+00 | 2024-10-16 16:30:19.020058+00 | 2024-10-16 16:30:19.265448+00 | 2024-10-16 16:30:19.26546+00 | Client | ClientRead | SAVEPOINT exec_stmt 00:00:00 | 33125 | zitadel_es_pusher | 2024-10-16 16:29:54.963859+00 | 2024-10-16 16:30:19.191715+00 | 2024-10-16 16:30:19.191715+00 | 2024-10-16 16:30:19.191729+00 | Client | ClientRead | begin 00:00:00.004292 | 33032 | zitadel_queries | 2024-10-16 16:25:53.906624+00 | 2024-10-16 16:30:19.187713+00 | 2024-10-16 16:30:19.192005+00 | 2024-10-16 16:30:19.192062+00 | Client | ClientRead | SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type 00:00:00 | 33031 | zitadel_queries | 2024-10-16 16:25:53.906422+00 | 2024-10-16 16:30:19.191625+00 | 2024-10-16 16:30:19.191625+00 | 2024-10-16 16:30:19.191645+00 | Client | ClientRead | begin isolation level read committed read only ``` The amount of idle transactions is significantly less if the query transactions are removed: example: ``` ?column? | pid | application_name | backend_start | xact_start | query_start | state_change | wait_event_type | wait_event | query -----------------+-------+----------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 00:00:00.000094 | 32741 | zitadel_queries | 2024-10-16 16:23:49.73935+00 | 2024-10-16 16:24:59.785589+00 | 2024-10-16 16:24:59.785683+00 | 2024-10-16 16:24:59.785684+00 | | | SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type 00:00:00 | 32762 | zitadel_es_pusher | 2024-10-16 16:24:02.275136+00 | 2024-10-16 16:24:59.784586+00 | 2024-10-16 16:24:59.784586+00 | 2024-10-16 16:24:59.784607+00 | Client | ClientRead | begin 00:00:00.000167 | 32742 | zitadel_queries | 2024-10-16 16:23:49.740489+00 | 2024-10-16 16:24:59.784274+00 | 2024-10-16 16:24:59.784441+00 | 2024-10-16 16:24:59.784442+00 | | | with usr as ( + | | | | | | | | | select u.id, u.creation_date, u.change_date, u.sequence, u.state, u.resource_owner, u.username, n.login_name as preferred_login_name + | | | | | | | | | from projections.users13 u + | | | | | | | | | left join projections.l 00:00:00.256014 | 32759 | zitadel_projection_spooler | 2024-10-16 16:24:01.418429+00 | 2024-10-16 16:24:59.52959+00 | 2024-10-16 16:24:59.785604+00 | 2024-10-16 16:24:59.785649+00 | Client | ClientRead | UPDATE projections.milestones SET reached_date = $1 WHERE (instance_id = $2) AND (type = $3) AND (reached_date IS NULL) 00:00:00.014199 | 32773 | zitadel_es_pusher | 2024-10-16 16:24:02.320404+00 | 2024-10-16 16:24:59.769509+00 | 2024-10-16 16:24:59.783708+00 | 2024-10-16 16:24:59.783709+00 | IO | WalSync | commit 00:00:00 | 32765 | zitadel_es_pusher | 2024-10-16 16:24:02.28173+00 | 2024-10-16 16:24:59.780413+00 | 2024-10-16 16:24:59.780413+00 | 2024-10-16 16:24:59.780426+00 | Client | ClientRead | begin 00:00:00.012729 | 32777 | zitadel_es_pusher | 2024-10-16 16:24:02.339737+00 | 2024-10-16 16:24:59.767432+00 | 2024-10-16 16:24:59.780161+00 | 2024-10-16 16:24:59.780195+00 | Client | ClientRead | RELEASE SAVEPOINT cockroach_restart ``` --------- Co-authored-by: Tim Möhlmann <tim+github@zitadel.com> Co-authored-by: Livio Spring <livio.a@gmail.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Elio Bischof <elio@zitadel.com> Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com> Co-authored-by: Miguel Cabrerizo <30386061+doncicuto@users.noreply.github.com> Co-authored-by: Joakim Lodén <Loddan@users.noreply.github.com> Co-authored-by: Yxnt <Yxnt@users.noreply.github.com> Co-authored-by: Stefan Benz <stefan@caos.ch> Co-authored-by: Harsha Reddy <harsha.reddy@klaviyo.com> Co-authored-by: Zach H <zhirschtritt@gmail.com>	2024-11-04 10:06:14 +01:00
Silvan	56df515e5f	chore: use pgx v5 (#7577) ... * chore: use pgx v5 * chore: update go version * remove direct pq dependency * remove unnecessary type * scan test * map scanner * converter * uint8 number array * duration * most unit tests work * unit tests work * chore: coverage * go 1.21 * linting * int64 gopfertammi * retry go 1.22 * retry go 1.22 * revert to go v1.21.5 * update go toolchain to 1.21.8 * go 1.21.8 * remove test flag * go 1.21.5 * linting * update toolchain * use correct array * use correct array * add byte array * correct value * correct error message * go 1.21 compatible	2024-03-27 15:48:22 +02:00
Livio Spring	7494a7b6d9	feat(api): add possibility to retrieve user schemas (#7614) ... This PR extends the user schema service (V3 API) with the possibility to ListUserSchemas and GetUserSchemaByID. The previously started guide is extended to demonstrate how to retrieve the schema(s) and notes the generated revision property.	2024-03-22 13:26:13 +00:00
Elio Bischof	385a55bd21	feat: limit audit trail (#6744) ... * feat: enable limiting audit trail * support AddExclusiveQuery * fix invalid condition * register event mappers * fix NullDuration validity * test query side for limits * lint * acceptance test audit trail limit * fix acceptance test * translate limits not found * update tests * fix linting * add audit log retention to default instance * fix tests * update docs * remove todo * improve test name	2023-10-25 11:42:00 +00:00
Tim Möhlmann	c71bf85b7a	feat(api/v2): store user agent details in the session (#6711) ... This change adds the ability to set and get user agent data, such as fingerprint, IP, request headers and a description to the session. All fields are optional. Closes #6028	2023-10-12 12:16:59 +00:00
Silvan	99e1c654a3	feat(storage): read only transactions for queries (#6415) ... * fix: tests * bastle wie en grosse * fix(database): scan as callback * fix tests * fix merge failures * remove as of system time * refactor: remove unused test * refacotr: remove unused lines	2023-08-22 10:49:22 +00:00
Livio Spring	14b8cf4894	feat(api): add OIDC session service (#6157) ... This PR starts the OIDC implementation for the API V2 including the Implicit and Code Flow. Co-authored-by: Livio Spring <livio.a@gmail.com> Co-authored-by: Tim Möhlmann <tim+github@zitadel.com> Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>	2023-07-10 13:27:00 +00:00
Silvan	e38abdcdf3	perf: query data AS OF SYSTEM TIME (#5231) ... Queries the data in the storage layser at the timestamp when the call hit the API layer	2023-02-27 22:36:43 +01:00
Livio Amstutz	9ab566fdeb	fix(query): keys (#2755) ... * fix: add keys to projections * change to multiple tables * query keys * query keys * fix race condition * fix timer reset * begin tests * tests * remove migration * only send to keyChannel if not nil	2022-01-12 13:22:04 +01:00
Silvan	6d94975a85	fix(queries): actions prepare funcs (#2689) ... * chore(queries): test suite for prepare stmt funcs * test(queries): prepare project funcs * refactor: add comments * test: simlify expected sql, added possibility to add args to expected queries * test(queries): prepare funcs in org * chore(backend): correct modules * test(queries): org domain prepare funcs * test: correct name * refactor: file name * refactor: add table to login policy columns * chore(prepare_test): only add row to result if columns * test(queries): login policy prepare funcs * chore: add comments for configs * test(queries): prepare idp funcs * fix(queries): add table to password complexity policy cols * test(queries): password complexity policy prepare funcs * fix(queries): add table to password age policy cols * test(queries): password age policy prepare func * fix(queries): set cols on lockout policy * test(queries): lockout policy prepare funs * fix(queries): set table on privacy policy cols * test(queries): privacy policy prepare funcs * fix(queries): set table on org iam policy cols * fix(queries): correct table in org iam policy cols * test(queries): org iam policy prepare funcs * test(queries): prepare project grant funcs * refactor(queries): prepareProjectRoleQuery as func * test(queries): prepare project role funcs * test(queries): project grant check for nulls in joins * fix(queries): allow null values in project grant * refactor(queries): make toQuery private * test(queries): action prepare funcs * refactor: rename prepareFlowQuery to prepareFlowsQuery * test: generic count only if count in cols * refactor: remove param in prepareFlowQuery * fix(queries): correct left joins in action flows * test(queries): action flow prepare funcs	2021-11-16 14:04:22 +01:00
Silvan	54fe34d502	test(queries): login policy prepare funcs (#2649) ... * chore(queries): test suite for prepare stmt funcs * test(queries): prepare project funcs * refactor: add comments * test: simlify expected sql, added possibility to add args to expected queries * test(queries): prepare funcs in org * chore(backend): correct modules * test(queries): org domain prepare funcs * test: correct name * refactor: file name * refactor: add table to login policy columns * chore(prepare_test): only add row to result if columns * test(queries): login policy prepare funcs	2021-11-15 16:04:08 +01:00
Silvan	8d4b2dcc75	test(queries): org prepare funcs (#2647) ... * chore(queries): test suite for prepare stmt funcs * test(queries): prepare project funcs * refactor: add comments * test: simlify expected sql, added possibility to add args to expected queries * test(queries): prepare funcs in org * chore(backend): correct modules	2021-11-15 12:52:49 +00:00
Silvan	f927f4af02	test(queries): suite for prepare stmt funcs (#2644) ... * chore(queries): test suite for prepare stmt funcs * test(queries): prepare project funcs * refactor: add comments * test: simlify expected sql, added possibility to add args to expected queries	2021-11-15 10:04:46 +01:00