53 Commits

Author	SHA1	Message	Date
Silvan	9c3e5e467b	perf(query): remove transactions for queries (#8614) ... # Which Problems Are Solved Queries currently execute 3 statements, begin, query, commit # How the Problems Are Solved remove transaction handling from query methods in database package # Additional Changes - Bump versions of `core_grpc_dependencies`-receipt in Makefile # Additional info During load tests we saw a lot of idle transactions of `zitadel_queries` application name which is the connection pool used to query data in zitadel. Executed query: `select query_start - xact_start, pid, application_name, backend_start, xact_start, query_start, state_change, wait_event_type, wait_event,substring(query, 1, 200) query from pg_stat_activity where datname = 'zitadel' and state <> 'idle';` Mostly the last query executed was `begin isolation level read committed read only`. example: ``` ?column? | pid | application_name | backend_start | xact_start | query_start | state_change | wait_event_type | wait_event | query -----------------+-------+----------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 00:00:00 | 33030 | zitadel_queries | 2024-10-16 16:25:53.906036+00 | 2024-10-16 16:30:19.191661+00 | 2024-10-16 16:30:19.191661+00 | 2024-10-16 16:30:19.19169+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00 | 33035 | zitadel_queries | 2024-10-16 16:25:53.909629+00 | 2024-10-16 16:30:19.19179+00 | 2024-10-16 16:30:19.19179+00 | 2024-10-16 16:30:19.191805+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00.00412 | 33028 | zitadel_queries | 2024-10-16 16:25:53.904247+00 | 2024-10-16 16:30:19.187734+00 | 2024-10-16 16:30:19.191854+00 | 2024-10-16 16:30:19.191964+00 | Client | ClientRead | SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type 00:00:00.084662 | 33134 | zitadel_es_pusher | 2024-10-16 16:29:54.979692+00 | 2024-10-16 16:30:19.178578+00 | 2024-10-16 16:30:19.26324+00 | 2024-10-16 16:30:19.263267+00 | Client | ClientRead | RELEASE SAVEPOINT cockroach_restart 00:00:00.084768 | 33139 | zitadel_es_pusher | 2024-10-16 16:29:54.979585+00 | 2024-10-16 16:30:19.180762+00 | 2024-10-16 16:30:19.26553+00 | 2024-10-16 16:30:19.265531+00 | LWLock | WALWriteLock | commit 00:00:00.077377 | 33136 | zitadel_es_pusher | 2024-10-16 16:29:54.978582+00 | 2024-10-16 16:30:19.187883+00 | 2024-10-16 16:30:19.26526+00 | 2024-10-16 16:30:19.265431+00 | Client | ClientRead | WITH existing AS ( + | | | | | | | | | (SELECT instance_id, aggregate_type, aggregate_id, "sequence" FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND aggregate_id = $3 ORDER BY "sequence" DE 00:00:00.012309 | 33123 | zitadel_es_pusher | 2024-10-16 16:29:54.963484+00 | 2024-10-16 16:30:19.175066+00 | 2024-10-16 16:30:19.187375+00 | 2024-10-16 16:30:19.187376+00 | IO | WalSync | commit 00:00:00 | 33034 | zitadel_queries | 2024-10-16 16:25:53.90791+00 | 2024-10-16 16:30:19.262921+00 | 2024-10-16 16:30:19.262921+00 | 2024-10-16 16:30:19.263133+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00 | 33039 | zitadel_queries | 2024-10-16 16:25:53.914106+00 | 2024-10-16 16:30:19.191676+00 | 2024-10-16 16:30:19.191676+00 | 2024-10-16 16:30:19.191687+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00.24539 | 33083 | zitadel_projection_spooler | 2024-10-16 16:27:49.895548+00 | 2024-10-16 16:30:19.020058+00 | 2024-10-16 16:30:19.265448+00 | 2024-10-16 16:30:19.26546+00 | Client | ClientRead | SAVEPOINT exec_stmt 00:00:00 | 33125 | zitadel_es_pusher | 2024-10-16 16:29:54.963859+00 | 2024-10-16 16:30:19.191715+00 | 2024-10-16 16:30:19.191715+00 | 2024-10-16 16:30:19.191729+00 | Client | ClientRead | begin 00:00:00.004292 | 33032 | zitadel_queries | 2024-10-16 16:25:53.906624+00 | 2024-10-16 16:30:19.187713+00 | 2024-10-16 16:30:19.192005+00 | 2024-10-16 16:30:19.192062+00 | Client | ClientRead | SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type 00:00:00 | 33031 | zitadel_queries | 2024-10-16 16:25:53.906422+00 | 2024-10-16 16:30:19.191625+00 | 2024-10-16 16:30:19.191625+00 | 2024-10-16 16:30:19.191645+00 | Client | ClientRead | begin isolation level read committed read only ``` The amount of idle transactions is significantly less if the query transactions are removed: example: ``` ?column? | pid | application_name | backend_start | xact_start | query_start | state_change | wait_event_type | wait_event | query -----------------+-------+----------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 00:00:00.000094 | 32741 | zitadel_queries | 2024-10-16 16:23:49.73935+00 | 2024-10-16 16:24:59.785589+00 | 2024-10-16 16:24:59.785683+00 | 2024-10-16 16:24:59.785684+00 | | | SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type 00:00:00 | 32762 | zitadel_es_pusher | 2024-10-16 16:24:02.275136+00 | 2024-10-16 16:24:59.784586+00 | 2024-10-16 16:24:59.784586+00 | 2024-10-16 16:24:59.784607+00 | Client | ClientRead | begin 00:00:00.000167 | 32742 | zitadel_queries | 2024-10-16 16:23:49.740489+00 | 2024-10-16 16:24:59.784274+00 | 2024-10-16 16:24:59.784441+00 | 2024-10-16 16:24:59.784442+00 | | | with usr as ( + | | | | | | | | | select u.id, u.creation_date, u.change_date, u.sequence, u.state, u.resource_owner, u.username, n.login_name as preferred_login_name + | | | | | | | | | from projections.users13 u + | | | | | | | | | left join projections.l 00:00:00.256014 | 32759 | zitadel_projection_spooler | 2024-10-16 16:24:01.418429+00 | 2024-10-16 16:24:59.52959+00 | 2024-10-16 16:24:59.785604+00 | 2024-10-16 16:24:59.785649+00 | Client | ClientRead | UPDATE projections.milestones SET reached_date = $1 WHERE (instance_id = $2) AND (type = $3) AND (reached_date IS NULL) 00:00:00.014199 | 32773 | zitadel_es_pusher | 2024-10-16 16:24:02.320404+00 | 2024-10-16 16:24:59.769509+00 | 2024-10-16 16:24:59.783708+00 | 2024-10-16 16:24:59.783709+00 | IO | WalSync | commit 00:00:00 | 32765 | zitadel_es_pusher | 2024-10-16 16:24:02.28173+00 | 2024-10-16 16:24:59.780413+00 | 2024-10-16 16:24:59.780413+00 | 2024-10-16 16:24:59.780426+00 | Client | ClientRead | begin 00:00:00.012729 | 32777 | zitadel_es_pusher | 2024-10-16 16:24:02.339737+00 | 2024-10-16 16:24:59.767432+00 | 2024-10-16 16:24:59.780161+00 | 2024-10-16 16:24:59.780195+00 | Client | ClientRead | RELEASE SAVEPOINT cockroach_restart ``` --------- Co-authored-by: Tim Möhlmann <tim+github@zitadel.com> Co-authored-by: Livio Spring <livio.a@gmail.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Elio Bischof <elio@zitadel.com> Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com> Co-authored-by: Miguel Cabrerizo <30386061+doncicuto@users.noreply.github.com> Co-authored-by: Joakim Lodén <Loddan@users.noreply.github.com> Co-authored-by: Yxnt <Yxnt@users.noreply.github.com> Co-authored-by: Stefan Benz <stefan@caos.ch> Co-authored-by: Harsha Reddy <harsha.reddy@klaviyo.com> Co-authored-by: Zach H <zhirschtritt@gmail.com>	2024-11-04 10:06:14 +01:00
Tim Möhlmann	32bad3feb3	perf(milestones): refactor (#8788) ... # Which Problems Are Solved Milestones used existing events from a number of aggregates. OIDC session is one of them. We noticed in load-tests that the reduction of the oidc_session.added event into the milestone projection is a costly business with payload based conditionals. A milestone is reached once, but even then we remain subscribed to the OIDC events. This requires the projections.current_states to be updated continuously. # How the Problems Are Solved The milestone creation is refactored to use dedicated events instead. The command side decides when a milestone is reached and creates the reached event once for each milestone when required. # Additional Changes In order to prevent reached milestones being created twice, a migration script is provided. When the old `projections.milestones` table exist, the state is read from there and `v2` milestone aggregate events are created, with the original reached and pushed dates. # Additional Context - Closes https://github.com/zitadel/zitadel/issues/8800	2024-10-28 08:29:34 +00:00
Tim Möhlmann	aeb379e7de	fix(eventstore): revert precise decimal (#8527) (#8679)	2024-09-24 18:43:29 +02:00
Silvan	b522588d98	fix(eventstore): precise decimal (#8527) ... # Which Problems Are Solved Float64 which was used for the event.Position field is [not precise in go and gets rounded](https://github.com/golang/go/issues/47300). This can lead to unprecies position tracking of events and therefore projections especially on cockcoachdb as the position used there is a big number. example of a unprecies position: exact: 1725257931223002628 float64: 1725257931223002624.000000 # How the Problems Are Solved The float64 was replaced by [github.com/jackc/pgx-shopspring-decimal](https://github.com/jackc/pgx-shopspring-decimal). # Additional Changes Correct behaviour of makefile for load tests. Rename `latestSequence`-queries to `latestPosition`	2024-09-06 12:19:19 +03:00
Tim Möhlmann	f9742a58f4	fix(import): add tracing spans to all import related functions (#8160) ... # Which Problems Are Solved This fix adds tracing spans to all V1 API import related functions. This is to troubleshoot import related performance issues reported to us. # How the Problems Are Solved Add a tracing span to `api/grpc/admin/import.go` and all related functions that are called in the `command` package. # Additional Changes - none # Additional Context - Reported by internal communication	2024-06-19 12:56:33 +02:00
Silvan	4dc86c2415	fix(eventstore): order by sequence if filter by aggregate id (#8019) ... # Which Problems Are Solved Queriying events by an aggregate id can produce high loads on the database if the aggregate id contains many events (count > 1000000). # How the Problems Are Solved Instead of using the postion and in_tx_order columns we use the sequence column which guarantees correct ordering in a single aggregate and uses more optimised indexes. # Additional Context Closes https://github.com/zitadel/DevOps/issues/50 Co-authored-by: Livio Spring <livio.a@gmail.com>	2024-05-28 06:49:30 +00:00
Silvan	cff0f73e24	fix(init): add setting to enable durable locks on crdb (#7982) ... feat(init): add setting to enable durable locks on crdb	2024-05-27 09:03:34 +00:00
Silvan	56df515e5f	chore: use pgx v5 (#7577) ... * chore: use pgx v5 * chore: update go version * remove direct pq dependency * remove unnecessary type * scan test * map scanner * converter * uint8 number array * duration * most unit tests work * unit tests work * chore: coverage * go 1.21 * linting * int64 gopfertammi * retry go 1.22 * retry go 1.22 * revert to go v1.21.5 * update go toolchain to 1.21.8 * go 1.21.8 * remove test flag * go 1.21.5 * linting * update toolchain * use correct array * use correct array * add byte array * correct value * correct error message * go 1.21 compatible	2024-03-27 15:48:22 +02:00
Silvan	4224c7ad3a	chore(core): update dependencies (#7517) ... * chore(core): update dependencies * chore(core): update dependencies	2024-03-11 10:43:56 +00:00
Elio Bischof	38777b478e	perf: filter events by instance ids (#7489) ... fix: filter events by instance ids	2024-03-04 07:56:48 +01:00
Livio Spring	8470649ecb	chore: pin crdb version for unit tests (#7260) ... * chore: pin crdb version for unit tests * use latest 23.1 version * use latest available 23.1 version	2024-01-18 08:16:54 +00:00
Silvan	b7d027e2fd	fix(db): always use begin tx (#7142) ... * fix(db): always use begin tx * fix(handler): timeout for begin	2024-01-04 16:12:20 +00:00
Tim Möhlmann	fe1337536f	fix(db): add additional connection pool for projection spooling (#7094) ... * fix(db): add additional connection pool for projection spooling * use correct connection pool for projections --------- Co-authored-by: Livio Spring <livio.a@gmail.com>	2023-12-20 16:13:04 +00:00
Tim Möhlmann	f680dd934d	refactor: rename package errors to zerrors (#7039) ... * chore: rename package errors to zerrors * rename package errors to gerrors * fix error related linting issues * fix zitadel error assertion * fix gosimple linting issues * fix deprecated linting issues * resolve gci linting issues * fix import structure --------- Co-authored-by: Elio Bischof <elio@zitadel.com>	2023-12-08 15:30:55 +01:00
Silvan	e3d1ca4d58	fix(eventstore): improve pagination of handler filter (#6968) ... * fix(setup): add filter_offset to `projections.current_states` * fix(eventstore): allow offset in query * fix(handler): offset for already processed events	2023-12-01 12:25:41 +00:00
Elio Bischof	1b6e3dcf27	fix: creation date argument in search events filters (#6855) ... * fix: creation date filter in event queries * fix: creation date with ordering filter * simplify code * simplify review --------- Co-authored-by: Livio Spring <livio.a@gmail.com>	2023-11-03 14:52:48 +00:00
Tim Möhlmann	ab79855cf0	fix(eventstore): prevent allocation of filtered events (#6749) ... * fix(eventstore): prevent allocation of filtered events Directly reduce each event obtained from a sql.Rows scan, so that we do not have to allocate all events in a slice. * reinstate the mutex as RWMutex * scan data directly * add todos * fix(writemodels): add reduce of parent * test: remove comment * update comments --------- Co-authored-by: adlerhurst <silvan.reusser@gmail.com>	2023-10-19 15:21:31 +00:00
Silvan	b5564572bc	feat(eventstore): increase parallel write capabilities (#5940) ... This implementation increases parallel write capabilities of the eventstore. Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and [06](https://zitadel.com/docs/support/advisory/a10006). The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`. If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.	2023-10-19 12:19:10 +02:00
Silvan	e64428b8d4	fix: TestCRDB_CreateInstance (#6522) ... (cherry picked from commit 856d0d12e8)	2023-09-12 17:06:01 +02:00
Silvan	22af4dcd97	feat(storage): read only transactions (#6417) ... feat(storage): read only transactions for queries (#6415) * fix: tests * bastle wie en grosse * fix(database): scan as callback * fix tests * fix merge failures * remove as of system time * refactor: remove unused test * refacotr: remove unused lines	2023-08-22 14:49:02 +02:00
adlerhurst	b43f32eba1	chore: merge main	2023-06-08 10:13:25 +02:00
Silvan	39bdef35e7	chore: merge (#5773) ... * feat: allow skip of success page for native apps (#5627) add possibility to return to callback directly after login without rendering the successful login page * build next * fix(console): disallow inline fonts, critical styles (#5714) fix: disallow inline * fix(setup): step 10 for postgres (#5717) * fix(setup): smaller transactions (#5743) * fix: order by sequence by default * test: add allowCreationDateFilter * fix(step10): separate executions (#5754) * feat: allow skip of success page for native apps (#5627) add possibility to return to callback directly after login without rendering the successful login page * build next * fix(console): disallow inline fonts, critical styles (#5714) fix: disallow inline * fix(setup): step 10 for postgres (#5717) * fix(setup): smaller transactions (#5743) * fix(step10): split statements * fix(step10): split into separate execs * chore: prerelease * add truncate before insert * fix: add truncate * Merge branch 'main' into optimise-step-10 * chore: reset release definition --------- Co-authored-by: Livio Spring <livio.a@gmail.com> Co-authored-by: Max Peintner <max@caos.ch> --------- Co-authored-by: Livio Spring <livio.a@gmail.com> Co-authored-by: Max Peintner <max@caos.ch>	2023-04-28 14:56:51 +00:00
James Schinner	2a79e77c7b	docs: Fix typo (#5727) ... Fix typo	2023-04-24 13:23:38 +02:00
Silvan	8da8fbe6ce	fix(eventstore): correct creation date of events (#5683) ... * fix: add setup step to correct creation dates * fix(eventstore): replace now with statement ts * fix(step10): correct number * fix: handle wrong instance domain removed events	2023-04-18 19:29:04 +02:00
Silvan	4c1169b562	feat(eventstore): order by creation_date and sequence (#5568) ... * feat(eventstore): order by `creation_date` and `sequence` * fix(logstore): use correct event type --------- Co-authored-by: Livio Spring <livio.a@gmail.com>	2023-04-05 09:34:24 +00:00
Silvan	e38abdcdf3	perf: query data AS OF SYSTEM TIME (#5231) ... Queries the data in the storage layser at the timestamp when the call hit the API layer	2023-02-27 22:36:43 +01:00
Livio Spring	d721f725fd	fix: instance remove (#4602)	2022-10-26 13:06:48 +00:00
Livio Spring	5aa91ad105	fix: improve performance (#4300) ... ## Note This release requires a setup step to fully improve performance. Be sure to start ZITADEL with an appropriate command (zitadel start-from-init / start-from-setup) ## Changes - fix: only run projection scheduler on active instances - fix: set default for concurrent instances of projections to 1 (for scheduling) - fix: create more indexes on eventstore.events table - fix: get current sequence for token check (improve reread performance)	2022-09-02 14:05:13 +00:00
Silvan	77b4fc5487	feat(database): support for postgres (#3998) ... * beginning with postgres statements * try pgx * use pgx * database * init works for postgres * arrays working * init for cockroach * init * start tests * tests * TESTS * ch * ch * chore: use go 1.18 * read stmts * fix typo * tests * connection string * add missing error handler * cleanup * start all apis * go mod tidy * old update * switch back to minute * on conflict * replace string slice with `database.StringArray` in db models * fix tests and start * update go version in dockerfile * setup go * clean up * remove notification migration * update * docs: add deploy guide for postgres * fix: revert sonyflake * use `database.StringArray` for daos * use `database.StringArray` every where * new tables * index naming, metadata primary key, project grant role key type * docs(postgres): change to beta * chore: correct compose * fix(defaults): add empty postgres config * refactor: remove unused code * docs: add postgres to self hosted * fix broken link * so? * change title * add mdx to link * fix stmt * update goreleaser in test-code * docs: improve postgres example * update more projections * fix: add beta log for postgres * revert index name change * prerelease * fix: add sequence to v1 "reduce paniced" * log if nil * add logging * fix: log output * fix(import): check if org exists and user * refactor: imports * fix(user): ignore malformed events * refactor: method naming * fix: test * refactor: correct errors.Is call * ci: don't build dev binaries on main * fix(go releaser): update version to 1.11.0 * fix(user): projection should not break * fix(user): handle error properly * docs: correct config example * Update .releaserc.js * Update .releaserc.js Co-authored-by: Livio Amstutz <livio.a@gmail.com> Co-authored-by: Elio Bischof <eliobischof@gmail.com>	2022-08-31 07:52:43 +00:00
Livio Spring	aed7010508	fix: scheduling (#3978) ... * fix: improve scheduling * build pre-release * fix: locker * fix: user handler and print stack in case of panic in reducer * chore: remove sentry * fix: improve handler projection and implement tests * more tests * fix: race condition in tests * Update internal/eventstore/repository/sql/query.go Co-authored-by: Silvan <silvan.reusser@gmail.com> * fix: implemented suggested changes * fix: lock statement Co-authored-by: Silvan <silvan.reusser@gmail.com>	2022-07-22 10:08:39 +00:00
Livio Spring	12d4d3ea0b	fix: enable env vars in setup steps (and deprecate admin subcommand) (#3871) ... * fix: enable env vars in setup steps (and deprecate admin subcommand) * fix tests and error text	2022-06-27 10:32:34 +00:00
Silvan	a95b1ab3d0	fix(storage): resolve deadlock occuring durring projection (#3671)	2022-05-19 11:44:16 +00:00
Florian Forster	fa9f581d56	chore(v2): move to new org (#3499) ... * chore: move to new org * logging * fix: org rename caos -> zitadel Co-authored-by: adlerhurst <silvan.reusser@gmail.com>	2022-04-26 23:01:45 +00:00
Livio Amstutz	1305c14e49	feat: handle instanceID in projections (#3442) ... * feat: handle instanceID in projections * rename functions * fix key lock * fix import	2022-04-19 08:26:12 +02:00
Silvan	db554536a1	fix: v2 setup sequence (#3437) ... * add/register human command done * validations * crypto * move clientid * keys * fix: clientID * remove v2 package * tests * tests running * fix: add init instance to eventstore * fix: mig * test(eventstore): create instance * revert old code * instance domain from ctx * chore: rename zitadel app ids * comments * fix: test * fix: mock * fix: test	2022-04-13 05:42:48 +00:00
Silvan	cea2567e22	fix: v2 human command (#3435) ... * add/register human command done * validations * crypto * move clientid * keys * fix: clientID * remove v2 package * tests * tests running * revert old code * instance domain from ctx * chore: rename zitadel app ids * comments * fix: test	2022-04-12 16:20:17 +02:00
Silvan	c5b99274d7	feat(cli): setup (#3267) ... * commander * commander * selber! * move to packages * fix(errors): implement Is interface * test: command * test: commands * add init steps * setup tenant * add default step yaml * possibility to set password * merge v2 into v2-commander * fix: rename iam command side to instance * fix: rename iam command side to instance * fix: rename iam command side to instance * fix: rename iam command side to instance * fix: search query builder can filter events in memory * fix: filters for add member * fix(setup): add `ExternalSecure` to config * chore: name iam to instance * fix: matching * remove unsued func * base url * base url * test(command): filter funcs * test: commands * fix: rename orgiampolicy to domain policy * start from init * commands * config * fix indexes and add constraints * fixes * fix: merge conflicts * fix: protos * fix: md files * setup * add deprecated org iam policy again * typo * fix search query * fix filter * Apply suggestions from code review * remove custom org from org setup * add todos for verification * change apps creation * simplify package structure * fix error * move preparation helper for tests * fix unique constraints * fix config mapping in setup * fix error handling in encryption_keys.go * fix projection config * fix query from old views to projection * fix setup of mgmt api * set iam project and fix instance projection * imports Co-authored-by: Livio Amstutz <livio.a@gmail.com> Co-authored-by: fabi <fabienne.gerschwiler@gmail.com>	2022-03-28 10:05:09 +02:00
Livio Amstutz	56b916a2b0	feat: projections auto create their tables (#3324) ... * begin init checks for projections * first projection checks * debug notification providers with query fixes * more projections and first index * more projections * more projections * finish projections * fix tests (remove db name) * create tables in setup * fix logging / error handling * add tenant to views * rename tenant to instance_id * add instance_id to all projections * add instance_id to all queries * correct instance_id on projections * add instance_id to failed_events * use separate context for instance * implement features projection * implement features projection * remove unique constraint from setup when migration failed * add error to failed setup event * add instance_id to primary keys * fix IAM projection * remove old migrations folder * fix keysFromYAML test	2022-03-23 09:02:39 +01:00
Fabi	5132ebe07c	feat: add tenant column to eventstore (#3314) ... * feat: add tenant column to eventstore * feat: read tenant from context on push and filter * Update 07_events_table.sql * pass tenant to queryFactory * fix some query tests * init in tests * add missing sql files Co-authored-by: Livio Amstutz <livio.a@gmail.com>	2022-03-15 07:19:02 +01:00
Livio Amstutz	19b095e6c2	fix: check resourceowner not empty string (#2922) ... * fix: check resourceowner not empty string * fix test * fix tests * fix tests of command pkg * enable RoleSelfManagementGlobal as org member role * fix tests of query pkg * Update eventstore_test.go * update docusaurus	2022-01-06 08:29:58 +01:00
Silvan	296f1c3c71	fix(eventstore): fill new column with data (#2288) ... * fix: smaller outage on events migration first part * fix: fill old events with sequence * fix: migration add transactions * fix: mig * rename mig * replace migration with setup step * regenerate mock * add step 20 to config * log * simplify step * limit 1000 * limit 500	2021-09-01 09:25:52 +00:00
Silvan	37ee5b4bab	feat: jobs for projection tables (#1730) ... * job queue * wg improvements * start handler * statement * statements * imporve handler * improve statement * statement in seperate file * move handlers * move query/old to query * handler * read models * bulk works * cleanup * contrib * rename readmodel to projection * rename read_models schema to projections * rename read_models schema to projections * search query as func, bulk iterates as long as new events * add event sequence less query * update checks for events between current sequence and sequence of first statement if it has previous sequence 0 * cleanup crdb projection * refactor projection handler * start with testing * tests for handler * remove todo * refactor statement: remove table name, add tests * improve projection handler shutdown, no savepoint if noop stmt, tests for stmt handler * tests * start failed events * seperate branch for contrib * move statement constructors to crdb pkg * correct import * Subscribe for eventtypes (#1800) * fix: is default (#1737) * fix: use email as username on global org (#1738) * fix: use email as username on global org * Update user_human.go * Update register_handler.go * chore(deps): update docusaurus (#1739) * chore: remove PAT and use GH Token (#1716) * chore: remove PAT and use GH Token * fix env * fix env * fix env * md lint * trigger ci * change user * fix GH bug * replace login part * chore: add GH Token to sem rel (#1746) * chore: add GH Token to sem rel * try branch * add GH Token * remove test branch again * docs: changes acme to acme-caos (#1744) * changes acme to acme-caos * Apply suggestions from code review Co-authored-by: Florian Forster <florian@caos.ch> Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com> Co-authored-by: Florian Forster <florian@caos.ch> * feat: add additional origins on applications (#1691) * feat: add additional origins on applications * app additional redirects * chore(deps-dev): bump @angular/cli from 11.2.8 to 11.2.11 in /console (#1706) * fix: show org with regex (#1688) * fix: flag mapping (#1699) * chore(deps-dev): bump @angular/cli from 11.2.8 to 11.2.11 in /console Bumps [@angular/cli](https://github.com/angular/angular-cli) from 11.2.8 to 11.2.11. - [Release notes](https://github.com/angular/angular-cli/releases) - [Commits](https://github.com/angular/angular-cli/compare/v11.2.8...v11.2.11) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Silvan <silvan.reusser@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump stylelint from 13.10.0 to 13.13.1 in /console (#1703) * fix: show org with regex (#1688) * fix: flag mapping (#1699) * chore(deps-dev): bump stylelint from 13.10.0 to 13.13.1 in /console Bumps [stylelint](https://github.com/stylelint/stylelint) from 13.10.0 to 13.13.1. - [Release notes](https://github.com/stylelint/stylelint/releases) - [Changelog](https://github.com/stylelint/stylelint/blob/master/CHANGELOG.md) - [Commits](https://github.com/stylelint/stylelint/compare/13.10.0...13.13.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Silvan <silvan.reusser@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump @types/node from 14.14.37 to 15.0.1 in /console (#1702) * fix: show org with regex (#1688) * fix: flag mapping (#1699) * chore(deps-dev): bump @types/node from 14.14.37 to 15.0.1 in /console Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 14.14.37 to 15.0.1. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Silvan <silvan.reusser@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump ts-protoc-gen from 0.14.0 to 0.15.0 in /console (#1701) * fix: show org with regex (#1688) * fix: flag mapping (#1699) * chore(deps): bump ts-protoc-gen from 0.14.0 to 0.15.0 in /console Bumps [ts-protoc-gen](https://github.com/improbable-eng/ts-protoc-gen) from 0.14.0 to 0.15.0. - [Release notes](https://github.com/improbable-eng/ts-protoc-gen/releases) - [Changelog](https://github.com/improbable-eng/ts-protoc-gen/blob/master/CHANGELOG.md) - [Commits](https://github.com/improbable-eng/ts-protoc-gen/compare/0.14.0...0.15.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Silvan <silvan.reusser@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps-dev): bump @types/jasmine from 3.6.9 to 3.6.10 in /console (#1682) Bumps [@types/jasmine](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jasmine) from 3.6.9 to 3.6.10. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jasmine) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump @types/google-protobuf in /console (#1681) Bumps [@types/google-protobuf](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/google-protobuf) from 3.7.4 to 3.15.2. - [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases) - [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/google-protobuf) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore(deps): bump grpc from 1.24.5 to 1.24.7 in /console (#1666) Bumps [grpc](https://github.com/grpc/grpc-node) from 1.24.5 to 1.24.7. - [Release notes](https://github.com/grpc/grpc-node/releases) - [Commits](https://github.com/grpc/grpc-node/compare/grpc@1.24.5...grpc@1.24.7) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * lock * chore(deps-dev): bump @angular/language-service from 11.2.9 to 11.2.12 in /console (#1704) * fix: show org with regex (#1688) * fix: flag mapping (#1699) * chore(deps-dev): bump @angular/language-service in /console Bumps [@angular/language-service](https://github.com/angular/angular/tree/HEAD/packages/language-service) from 11.2.9 to 11.2.12. - [Release notes](https://github.com/angular/angular/releases) - [Changelog](https://github.com/angular/angular/blob/master/CHANGELOG.md) - [Commits](https://github.com/angular/angular/commits/11.2.12/packages/language-service) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Silvan <silvan.reusser@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * package lock * downgrade grpc * downgrade protobuf types * revert npm packs 🥸 Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Silvan <silvan.reusser@gmail.com> * docs: update run and start section texts (#1745) * update run and start section texts * adds showcase Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com> * fix: additional origin list (#1753) * fix: handle api configs in authz handler (#1755) * fix(console): add model for api keys, fix toast, binding (#1757) * fix: add model for api keys, fix toast, binding * show api clientid * fix: missing patchvalue (#1758) * feat: refresh token (#1728) * begin refresh tokens * refresh tokens * list and revoke refresh tokens * handle remove * tests for refresh tokens * uniqueness and default expiration * rename oidc token methods * cleanup * migration version * Update internal/static/i18n/en.yaml Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com> * fixes * feat: update oidc pkg for refresh tokens Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com> * fix: correct json name of clientId in key.json (#1760) * fix: migration version (#1767) * start subscription * eventtypes * fix(login): links (#1778) * fix(login): href for help * fix(login): correct link to tos * fix: access tokens for service users and refresh token infos (#1779) * fix: access token for service user * handle info from refresh request * uniqueness * postpone access token uniqueness change * chore(coc): recommend code of conduct (#1782) * subscribe for events * feat(console): refresh toggle out of granttype context (#1785) * refresh toggle * disable if not code flow, lint * lint * fix: change oidc config order * accept refresh option within flow Co-authored-by: Livio Amstutz <livio.a@gmail.com> * fix: refresh token activation (#1795) * fix: oidc grant type check * docs: add offline_access scope * docs: update refresh token status in supported grant types * fix: update oidc pkg * fix: check refresh token grant type (#1796) * configuration structs * org admins * failed events * fixes Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Livio Amstutz <livio.a@gmail.com> Co-authored-by: Florian Forster <florian@caos.ch> Co-authored-by: mffap <mpa@caos.ch> Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com> * remove comment * aggregate reducer * remove eventtypes * add protoc-get-validate to mod * fix transaltion * upsert * add gender on org admins, allow to retry failed stmts after configurable time * remove if * sub queries * fix: tests * add builder to tests * new search query * rename searchquerybuilder to builder * remove comment from code * test with multiple queries * add filters test * current sequences * make org and org_admins work again * add aggregate type to current sequence * fix(contibute): listing * add validate module * fix: search queries * feat(eventstore): previous aggregate root sequence (#1810) * feat(eventstore): previous aggregate root sequence * fix tests * fix: eventstore v1 test * add col to all mocked rows * next try * fix mig * rename aggregate root to aggregate type * update comment Co-authored-by: Livio Amstutz <livio.a@gmail.com> Co-authored-by: Livio Amstutz <livio.a@gmail.com> * small refactorings * allow update multiple current sequences * unique log id * fix migrations * rename org admin to org owner * improve error handling and logging * fix(migration): optimize prev agg root seq * fix: projection handler test * fix: sub queries * small fixes * additional event types * correct org owner projection * fix primary key * feat(eventstore): jobs for projections (#2026) * fix: template names in login (#1974) * fix: template names in login * fix: error.html * fix: check for features on mgmt only (#1976) * fix: add sentry in ui, http and projection handlers (#1977) * fix: add sentry in ui, http and projection handlers * fix test * fix(eventstore): sub queries (#1805) * sub queries * fix: tests * add builder to tests * new search query * rename searchquerybuilder to builder * remove comment from code * test with multiple queries * add filters test * fix(contibute): listing * add validate module * fix: search queries * remove unused event type in query * ignore query if error in marshal * go mod tidy * update privacy policy query * update queries Co-authored-by: Livio Amstutz <livio.a@gmail.com> * feat: Extend oidc idp with oauth endpoints (#1980) * feat: add oauth attributes to oidc idp configuration * feat: return idpconfig id on create idp * feat: tests * feat: descriptions * feat: docs * feat: tests * docs: update to beta 3 (#1984) * fix: role assertion (#1986) * fix: enum to display access token role assertion * improve assertion descriptions * fix nil pointer * docs: eventstore (#1982) * docs: eventstore * Apply suggestions from code review Co-authored-by: Florian Forster <florian@caos.ch> Co-authored-by: Florian Forster <florian@caos.ch> * fix(sentry): trigger sentry release (#1989) * feat(send sentry release): send sentry release * fix(moved step and added releasetag): moved step and added releasetag * fix: set version for sentry release (#1990) * feat(send sentry release): send sentry release * fix(moved step and added releasetag): moved step and added releasetag * fix(corrected var name): corrected var name Co-authored-by: Livio Amstutz <livio.a@gmail.com> * fix: log error reason on terminate session (#1973) * fix: return default language file, if requested lang does not exist for default login texts (#1988) * fix: return default language file, if requested lang doesnt exists * feat: read default translation file * feat: docs * fix: race condition in auth request unmarshalling (#1993) * feat: handle ui_locales in login (#1994) * fix: handle ui_locales in login * move supportedlanguage func into i18n package * update oidc pkg * fix: handle closed channels on unsubscribe (#1995) * fix: give restore more time (#1997) * fix: translation file read (#2009) * feat: translation file read * feat: readme * fix: enable idp add button for iam users (#2010) * fix: filter event_data (#2011) * feat: Custom message files (#1992) * feat: add get custom message text to admin api * feat: read custom message texts from files * feat: get languages in apis * feat: get languages in apis * feat: get languages in apis * feat: pr feedback * feat: docs * feat: merge main * fix: sms notification (#2013) * fix: phone verifications * feat: fix password reset as sms * fix: phone verification * fix: grpc status in sentry and validation interceptors (#2012) * fix: remove oauth endpoints from oidc config proto (#2014) * try with view * fix(console): disable sw (#2021) * fix: disable sw * angular.json disable sw * project projections * fix typos * customize projections * customizable projections, add change date to projects Co-authored-by: Livio Amstutz <livio.a@gmail.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com> Co-authored-by: Florian Forster <florian@caos.ch> Co-authored-by: mffap <mpa@caos.ch> Co-authored-by: Christian Jakob <47860090+thesephirot@users.noreply.github.com> Co-authored-by: Elio Bischof <eliobischof@gmail.com> * env file * typo * correct users * correct migration * fix: merge fail * fix test * fix(tests): unordered matcher * improve currentSequenceMatcher * correct certs * correct certs * add zitadel database on database list * refctor switch in match * enable all handlers * Delete io.env * cleanup * add handlers * rename view to projection * rename view to projection * fix type typo * remove unnecessary logs * refactor stmts * simplify interval calculation * fix tests * fix unlock test * fix migration * migs * fix(operator): update cockroach and flyway versions (#2138) * chore(deps): bump k8s.io/apiextensions-apiserver from 0.19.2 to 0.21.3 Bumps [k8s.io/apiextensions-apiserver](https://github.com/kubernetes/apiextensions-apiserver) from 0.19.2 to 0.21.3. - [Release notes](https://github.com/kubernetes/apiextensions-apiserver/releases) - [Commits](https://github.com/kubernetes/apiextensions-apiserver/compare/v0.19.2...v0.21.3) --- updated-dependencies: - dependency-name: k8s.io/apiextensions-apiserver dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * chore(deps): bump google.golang.org/api from 0.34.0 to 0.52.0 Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.34.0 to 0.52.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.34.0...v0.52.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> * start update dependencies * update mods and otlp * fix(build): update to go 1.16 * old version for k8s mods * update k8s versions * update orbos * fix(operator): update cockroach and flyway version * Update images.go Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Stefan Benz <stefan@caos.ch> * fix import * fix typo * fix(migration): add org projection * fix(projection): correct table for org events in org owners * better insert stmt * fix typo * fix typo * set max connection lifetime * set max conns and conn lifetime in eventstore v1 * configure sql connection settings * add mig for agg type index * fix replace tab in yaml * check requeue at least 500ms * split column in column and condition * remove useless comment * mig versions * fix migs Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Livio Amstutz <livio.a@gmail.com> Co-authored-by: Florian Forster <florian@caos.ch> Co-authored-by: mffap <mpa@caos.ch> Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com> Co-authored-by: Christian Jakob <47860090+thesephirot@users.noreply.github.com> Co-authored-by: Elio Bischof <eliobischof@gmail.com> Co-authored-by: Stefan Benz <stefan@caos.ch>	2021-08-19 08:31:56 +02:00
Livio Amstutz	7ebf0333c3	fix: filter event_data (#2011)	2021-07-12 14:26:54 +00:00
Silvan	5349d96ce4	fix(eventstore): sub queries (#1805) ... * sub queries * fix: tests * add builder to tests * new search query * rename searchquerybuilder to builder * remove comment from code * test with multiple queries * add filters test * fix(contibute): listing * add validate module * fix: search queries * remove unused event type in query * ignore query if error in marshal * go mod tidy * update privacy policy query * update queries Co-authored-by: Livio Amstutz <livio.a@gmail.com>	2021-07-06 13:55:57 +02:00
Livio Amstutz	87c1dfa3aa	fix: uniqueness (#1710) ... * fix: uniqueconstraint to lower * feat: change org * feat: org change test * feat: change org * fix: tests * fix: handle domain claims correctly * feat: update org Co-authored-by: fabi <fabienne.gerschwiler@gmail.com>	2021-05-04 09:09:24 +00:00
Fabi	667cc30291	feat: asset storage (#1696) ... * feat: remove assets * feat: minio implementation * fix: remove assets from tests * feat: minio implementation * feat: Env vars * fix: sprintf * fix: sprintf * Update internal/eventstore/repository/repository.go Co-authored-by: Livio Amstutz <livio.a@gmail.com> * fix: error handling Co-authored-by: Livio Amstutz <livio.a@gmail.com>	2021-05-03 10:15:50 +02:00
Fabi	f51f0ede5c	feat: add assets to eventstore and event (#1674) ... * fix: add assets to eventstore and event * fix: project member, grant member, app changed tests * fix: asset migrations * feat: add asset tests * feat: add asset tests * Update internal/eventstore/repository/repository.go Co-authored-by: Livio Amstutz <livio.a@gmail.com> * feat: add asset tests Co-authored-by: Livio Amstutz <livio.a@gmail.com>	2021-04-27 12:58:18 +02:00
Livio Amstutz	06281b5ccb	fix: improve performance on db insert (#1649)	2021-04-22 15:06:23 +00:00
Silvan	c919fd6f13	fix: correct imports (#1630) ... * fix: correct import for errors * fix: add missing translations * fix(eventstore): index * fix(eventstore): use table instead of index * fix(proto): correct info * fix(migrations): use eventstore	2021-04-20 22:17:37 +02:00
Livio Amstutz	fa6fb92c7e	fix: add order by for max previous sequence (#1501)	2021-03-30 19:28:00 +02:00