1
0
mirror of https://github.com/zitadel/zitadel.git synced 2024-12-23 00:07:31 +00:00

35 Commits

Author SHA1 Message Date
Silvan
d3bb9c9b3b
fix(setup): init projections ()
Even though this is a feature it's released as fix so that we can back port to earlier revisions.

As reported by multiple users startup of ZITADEL after leaded to downtime and worst case rollbacks to the previously deployed version.

The problem starts rising when there are too many events to process after the start of ZITADEL. The root cause are changes on projections (database tables) which must be recomputed. This PR solves this problem by adding a new step to the setup phase which prefills the projections. The step can be enabled by adding the `--init-projections`-flag to `setup`, `start-from-init` and `start-from-setup`. Setting this flag results in potentially longer duration of the setup phase but reduces the risk of the problems mentioned in the paragraph above.

(cherry picked from commit 17953e904034e6a9d91379cab723d8dbd0f36765)
2024-01-26 07:41:11 +01:00
Elio Bischof
ed0bc39ea4
feat: block instances ()
* docs: fix init description typos

* feat: block instances using limits

* translate

* unit tests

* fix translations

* redirect /ui/login

* fix http interceptor

* cleanup

* fix http interceptor

* fix: delete cookies on gateway 200

* add integration tests

* add command test

* docs

* fix integration tests

* add bulk api and integration test

* optimize bulk set limits

* unit test bulk limits

* fix broken link

* fix assets middleware

* fix broken link

* validate instance id format

* Update internal/eventstore/search_query.go

Co-authored-by: Livio Spring <livio.a@gmail.com>

* remove support for owner bulk limit commands

* project limits to instances

* migrate instances projection

* Revert "migrate instances projection"

This reverts commit 214218732a56e6df823beac1972adfcf8beeded5.

* join limits, remove owner

* remove todo

* use optional bool

* normally validate instance ids

* use 302

* cleanup

* cleanup

* Update internal/api/grpc/system/limits_converter.go

Co-authored-by: Livio Spring <livio.a@gmail.com>

* remove owner

* remove owner from reset

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-01-17 10:16:48 +00:00
Silvan
43f1d59649
fix(auth): efficient user session projection ()
* fix(auth): cache users during session projection

* fix(auth.user_sessions): add index for more efficient by user search
2024-01-09 18:36:46 +00:00
Silvan
cc2dd8b20b
fix(eventstore): increase performance on push () 2023-12-31 15:30:25 +01:00
Tim Möhlmann
fe1337536f
fix(db): add additional connection pool for projection spooling ()
* fix(db): add additional connection pool for projection spooling

* use correct connection pool for projections

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-12-20 16:13:04 +00:00
Silvan
ddbea119f1
fix(query): user performance ()
* start user by id

* ignore debug bin

* use new user by id

* new sql

* fix(sql): replace STRING with text for psql compatabilit

* some changes

* fix: correct user queries

* fix tests

* unify sql statements

* use specific get user methods

* search login name case insensitive

* refactor: optimise user statements

* add index

* fix queries

* fix: correct domain segregation

* return all login names

* fix queries

* improve readability

* query should be correct now

* cleanup statements

* fix username / loginname handling

* fix: psql doesn't support create view if not exists

* fix: create pre-release

* ignore release comments

* add lower fields

* fix: always to lower

* update to latest projection

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-12-08 13:14:22 +01:00
Elio Bischof
dd33538c0a
feat: restrict languages ()
* feat: return 404 or 409 if org reg disallowed

* fix: system limit permissions

* feat: add iam limits api

* feat: disallow public org registrations on default instance

* add integration test

* test: integration

* fix test

* docs: describe public org registrations

* avoid updating docs deps

* fix system limits integration test

* silence integration tests

* fix linting

* ignore strange linter complaints

* review

* improve reset properties naming

* redefine the api

* use restrictions aggregate

* test query

* simplify and test projection

* test commands

* fix unit tests

* move integration test

* support restrictions on default instance

* also test GetRestrictions

* self review

* lint

* abstract away resource owner

* fix tests

* configure supported languages

* fix allowed languages

* fix tests

* default lang must not be restricted

* preferred language must be allowed

* change preferred languages

* check languages everywhere

* lint

* test command side

* lint

* add integration test

* add integration test

* restrict supported ui locales

* lint

* lint

* cleanup

* lint

* allow undefined preferred language

* fix integration tests

* update main

* fix env var

* ignore linter

* ignore linter

* improve integration test config

* reduce cognitive complexity

* compile

* check for duplicates

* remove useless restriction checks

* review

* revert restriction renaming

* fix language restrictions

* lint

* generate

* allow custom texts for supported langs for now

* fix tests

* cleanup

* cleanup

* cleanup

* lint

* unsupported preferred lang is allowed

* fix integration test

* finish reverting to old property name

* finish reverting to old property name

* load languages

* refactor(i18n): centralize translators and fs

* lint

* amplify no validations on preferred languages

* fix integration test

* lint

* fix resetting allowed languages

* test unchanged restrictions
2023-12-05 11:12:01 +00:00
Silvan
e3d1ca4d58
fix(eventstore): improve pagination of handler filter ()
* fix(setup): add filter_offset to `projections.current_states`

* fix(eventstore): allow offset in query

* fix(handler): offset for already processed events
2023-12-01 12:25:41 +00:00
Livio Spring
b563041103
fix: ensure uniqueness ()
* fix: ensure uniqueness

* only update wrong ones

* Update cmd/setup/16.go

Co-authored-by: Silvan <silvan.reusser@gmail.com>

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-11-22 12:05:14 +00:00
Silvan
b5564572bc
feat(eventstore): increase parallel write capabilities ()
This implementation increases parallel write capabilities of the eventstore.
Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and  [06](https://zitadel.com/docs/support/advisory/a10006).
The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`.
If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.
2023-10-19 12:19:10 +02:00
Elio Bischof
1d4ec6cdba
fix: set quotas ()
* feat: set quotas

* fix: start new period on younger anchor

* cleanup e2e config

* fix set notifications

* lint

* test: fix quota projection tests

* fix add quota tests

* make quota fields nullable

* enable amount 0

* fix initial setup

* create a prerelease

* avoid success comments

* fix quota projection primary key

* Revert "fix quota projection primary key"

This reverts commit e72f4d7fa17d03d36493912168490350a320e04f.

* simplify write model

* fix aggregate id

* avoid push without changes

* test set quota lifecycle

* test set quota mutations

* fix quota unit test

* fix: quotas

* test quota.set event projection

* use SetQuota in integration tests

* fix: release quotas 3

* reset releaserc

* fix comment

* test notification order doesn't matter

* test notification order doesn't matter

* test with unmarshalled events

* test with unmarshalled events

(cherry picked from commit ae1af6bc8cd2294f47f6d6412c4b46192105ade5)
2023-09-22 13:06:59 +02:00
Silvan
22af4dcd97
feat(storage): read only transactions ()
feat(storage): read only transactions for queries ()

* fix: tests

* bastle wie en grosse

* fix(database): scan as callback

* fix tests

* fix merge failures

* remove as of system time

* refactor: remove unused test

* refacotr: remove unused lines
2023-08-22 14:49:02 +02:00
Livio Spring
7c494fd219
feat(login): add OTP (email and sms) ()
* feat: login with otp

* fix(i18n): japanese translation

* add missing files

* fix provider change

* add event types translations to en

* add tests

* resourceOwner

* remove unused handler

* fix: secret generators and add comments

* add setup step

* rename

* linting

* fix setup

* improve otp handling

* fix autocomplete

* translations for login and notifications

* translations for event types

* changes from review

* check selected mfa type
2023-08-15 12:47:05 +00:00
Tim Möhlmann
4589ddad4a
feat: integrate passwap for human user password hashing ()
* feat: use passwap for human user passwords

* fix tests

* passwap config

* add the event mapper

* cleanup query side and api

* solve linting errors

* regression test

* try to fix linter errors again

* pass systemdefaults into externalConfigChange migration

* fix: user password set in auth view

* pin passwap v0.2.0

* v2: validate hashed password hash based on prefix

* resolve remaining comments

* add error tag and translation for unsupported hash encoding

* fix unit test

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-07-14 09:49:57 +03:00
Stefan Benz
855d6b1bd5
fix: nil pointer on create instance add machine ()
* fix: nil pointer on create instance add machine

* fix: instance setup with machine user pat

* fix: correct logic to write pat and key from setup without configurable scope

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-06-15 06:16:39 +00:00
Silvan
6be41ea257
fix(setup): steps 10 and 11 ()
* fix(step10): count amount of wrong_events manually

* fix(step11): create index for better performance
2023-06-07 16:30:19 +00:00
adlerhurst
4d7a733e11 fix(step11): execute step 10 to make sure events are in correct order 2023-05-11 11:53:52 +02:00
adlerhurst
c6d29fc201 fix(eventstore): new column to test clock_timestamp() 2023-05-11 10:04:35 +02:00
Silvan
c8c5cf3c5f
feat(cli): add setup cleanup sub command ()
* feat(cli): add `setup cleanup` sub command

* chore: logging

* chore: logging
2023-04-28 11:55:35 +00:00
Silvan
13f6b4686a
fix(setup): smaller transactions () 2023-04-25 08:12:53 +00:00
Silvan
8da8fbe6ce
fix(eventstore): correct creation date of events ()
* fix: add setup step to correct creation dates

* fix(eventstore): replace now with statement ts

* fix(step10): correct number

* fix: handle wrong instance domain removed events
2023-04-18 19:29:04 +02:00
Silvan
3dbb6f7c67
fix(setup): optimise events indexes () 2023-03-01 00:11:23 +00:00
Silvan
e38abdcdf3
perf: query data AS OF SYSTEM TIME ()
Queries the data in the storage layser at the timestamp when the call hit the API layer
2023-02-27 22:36:43 +01:00
Silvan
94116fa04b
fix(storage): add indexes for auth.tokens ()
* fix(storage): add indexes for `auth.tokens`

* refactor: naming
2023-02-21 14:46:47 +00:00
Elio Bischof
058192c22b
test: fix log headers ()
* test: fix log headers

* ensure just public types are tested

* fix(postgres): proper statements for setup step 7

---------

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2023-02-15 03:21:58 +00:00
Elio Bischof
681541f41b
feat: add quotas ()
adds possibilities to cap authenticated requests and execution seconds of actions on a defined intervall
2023-02-15 02:52:11 +01:00
Livio Spring
d21bb902f1
fix: push timeout () ()
* push with timeout

* test: config for eventstore

(cherry picked from commit b9156da76d0f03075589b50eafbf9f48160a0301)

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2022-12-15 09:40:13 +00:00
Silvan
f3e6f3b23b
feat: remove org ()
* feat(command): remove org

* refactor: imports, unused code, error handling

* reduce org removed in action

* add org deletion to projections

* add org removal to projections

* add org removal to projections

* org removed projection

* lint import

* projections

* fix: table names in tests

* fix: table names in tests

* logging

* add org state

* fix(domain): add Owner removed to object details

* feat(ListQuery): add with owner removed

* fix(org-delete): add bool to functions to select with owner removed

* fix(org-delete): add bools to user grants with events to determine if dependencies lost owner

* fix(org-delete): add unit tests for owner removed and org removed events

* fix(org-delete): add handling of org remove for grants and members

* fix(org-delete): correction of unit tests for owner removed

* fix(org-delete): update projections, unit tests and get functions

* fix(org-delete): add change date to authnkeys and owner removed to org metadata

* fix(org-delete): include owner removed for login names

* fix(org-delete): some column fixes in projections and build for queries with owner removed

* indexes

* fix(org-delete): include review changes

* fix(org-delete): change user projection name after merge

* fix(org-delete): include review changes for project grant where no project owner is necessary

* fix(org-delete): include auth and adminapi tables with owner removed information

* fix(org-delete): cleanup username and orgdomain uniqueconstraints when org is removed

* fix(org-delete): add permissions for org.remove

* remove unnecessary unique constraints

* fix column order in primary keys

* fix(org-delete): include review changes

* fix(org-delete): add owner removed indexes and chang setup step to create tables

* fix(org-delete): move PK order of instance_id and change added user_grant from review

* fix(org-delete): no params for prepareUserQuery

* change to step 6

* merge main

* fix(org-delete): OldUserName rename to private

* fix linting

* cleanup

* fix: remove org test

* create prerelease

* chore: delete org-delete as prerelease

Co-authored-by: Stefan Benz <stefan@caos.ch>
Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Fabi <38692350+hifabienne@users.noreply.github.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2022-11-30 17:01:17 +01:00
Livio Spring
29441ce4b6
feat: save last occurrence of failed events and fix instance filtering ()
* fix: filter failed events and current sequence correctly

* fix failed events sorting column

* feat: save last occurrence of failed event

* fix failedEvents query and update sql statements

* change sql statement to only create index

* fix linting

* fix linting

* Update internal/query/failed_events.go

Co-authored-by: Silvan <silvan.reusser@gmail.com>

* update job name on test-docs to match the one from test-code

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2022-11-18 13:49:38 +01:00
Livio Spring
c791f6de58
fix: improve startup times by initializing projection tables during setup ()
* fix: improve startup times by initializing projections table during setup

* add missing file
2022-11-04 09:21:58 +00:00
Livio Spring
5aa91ad105
fix: improve performance ()
## Note

This release requires a setup step to fully improve performance.
Be sure to start ZITADEL with an appropriate command (zitadel start-from-init / start-from-setup)

## Changes

- fix: only run projection scheduler on active instances
- fix: set default for concurrent instances of projections to 1 (for scheduling)
- fix: create more indexes on eventstore.events table
- fix: get current sequence for token check (improve reread performance)
2022-09-02 14:05:13 +00:00
Livio Spring
f610d48569
feat: prepare for multiple database types ()
BREAKING CHANGE: the database and admin user config has changed.
2022-07-28 16:25:42 +02:00
Silvan
0b742233f9
fix: clarify instances in steps and config ()
* fix: clarify instances in steps and config

* docs: update setup step

Co-authored-by: Fabi <38692350+hifabienne@users.noreply.github.com>
Co-authored-by: Livio Amstutz <livio.a@gmail.com>
2022-07-27 10:22:20 +02:00
Livio Spring
95481c2e0b
feat: allow system config changes ()
* feat: run repeatable setup steps

* feat: react to system config changes

* renaming
2022-07-20 11:20:49 +02:00
Livio Spring
12d4d3ea0b
fix: enable env vars in setup steps (and deprecate admin subcommand) ()
* fix: enable env vars in setup steps (and deprecate admin subcommand)

* fix tests and error text
2022-06-27 10:32:34 +00:00