TheArchive/zitadel
1
0
Fork 0
mirror of https://github.com/zitadel/zitadel.git synced 2024-12-14 03:54:21 +00:00
Code Issues Packages Projects Releases Wiki Activity
4552fe7d99
zitadel/internal/query
History
Silvan 2243306ef6
feat(cmd): mirror (#7004) 
# Which Problems Are Solved

Adds the possibility to mirror an existing database to a new one. 

For that a new command was added `zitadel mirror`. Including it's
subcommands for a more fine grained mirror of the data.

Sub commands:

* `zitadel mirror eventstore`: copies only events and their unique
constraints
* `zitadel mirror system`: mirrors the data of the `system`-schema
*  `zitadel mirror projections`: runs all projections
*  `zitadel mirror auth`: copies auth requests
* `zitadel mirror verify`: counts the amount of rows in the source and
destination database and prints the diff.

The command requires one of the following flags:
* `--system`: copies all instances of the system
* `--instance <instance-id>`, `--instance <comma separated list of
instance ids>`: copies only the defined instances

The command is save to execute multiple times by adding the
`--replace`-flag. This replaces currently existing data except of the
`events`-table

# Additional Changes

A `--for-mirror`-flag was added to `zitadel setup` to prepare the new
database. The flag skips the creation of the first instances and initial
run of projections.

It is now possible to skip the creation of the first instance during
setup by setting `FirstInstance.Skip` to true in the steps
configuration.

# Additional info

It is currently not possible to merge multiple databases. See
https://github.com/zitadel/zitadel/issues/7964 for more details.

It is currently not possible to use files. See
https://github.com/zitadel/zitadel/issues/7966 for more information.

closes https://github.com/zitadel/zitadel/issues/7586
closes https://github.com/zitadel/zitadel/issues/7486

### Definition of Ready

- [x] I am happy with the code
- [x] Short description of the feature/issue is added in the pr
description
- [x] PR is linked to the corresponding user story
- [x] Acceptance criteria are met
- [x] All open todos and follow ups are defined in a new ticket and
justified
- [x] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [x] No debug or dead code
- [x] My code has no repetitions
- [x] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [x] Documentation/examples are up-to-date
- [x] All non-functional requirements are met
- [x] Functionality of the acceptance criteria is checked manually on
the dev system.

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-05-30 09:35:30 +00:00
..
projection feat(cmd): mirror (#7004) 2024-05-30 09:35:30 +00:00
testdata feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
access_token.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
action_flow_test.go perf: remove owner removed columns from projections for oidc (#6925) 2023-11-20 17:21:08 +02:00
action_flow.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
action_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
action.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
app_test.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
app.go fix(query): reduce app query overhead (#7817) 2024-04-22 11:30:56 +02:00
auth_request_by_id.sql feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
auth_request_test.go chore: use pgx v5 (#7577) 2024-03-27 15:48:22 +02:00
auth_request.go feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
authn_key_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
authn_key.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
certificate_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
certificate.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
converter.go feat(api): feature flags (#7356) 2024-02-28 10:55:54 +02:00
current_state_test.go feat(eventstore): increase parallel write capabilities (#5940) 2023-10-19 12:19:10 +02:00
current_state.go fix(db): wrap BeginTx in spans to get acquire metrics (#7689) 2024-04-03 11:48:24 +03:00
custom_text_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
custom_text.go feat(idp): provide option to auto link user (#7734) 2024-04-10 15:46:30 +00:00
device_auth_test.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
device_auth.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
domain_policy_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
domain_policy.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
event.go chore: remove bloating span (#7780) 2024-04-16 11:19:17 +00:00
execution_targets.sql feat: add action v2 execution on requests and responses (#7637) 2024-05-04 11:55:57 +02:00
execution_test.go feat: add action v2 execution on requests and responses (#7637) 2024-05-04 11:55:57 +02:00
execution.go feat: add action v2 execution on requests and responses (#7637) 2024-05-04 11:55:57 +02:00
failed_events_test.go feat(eventstore): increase parallel write capabilities (#5940) 2023-10-19 12:19:10 +02:00
failed_events.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
generic.go feat: query side for executions and targets for actions v2 (#7524) 2024-03-14 09:56:23 +00:00
iam_member_test.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
iam_member.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
idp_login_policy_link_test.go feat(idp): provide option to auto link user (#7734) 2024-04-10 15:46:30 +00:00
idp_login_policy_link.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
idp_template_test.go feat(saml): allow setting nameid-format and alternative mapping for transient format (#7979) 2024-05-23 05:04:07 +00:00
idp_template.go feat(saml): allow setting nameid-format and alternative mapping for transient format (#7979) 2024-05-23 05:04:07 +00:00
idp_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
idp_user_link_test.go feat(idp): provide option to auto link user (#7734) 2024-04-10 15:46:30 +00:00
idp_user_link.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
idp.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
instance_by_domain.sql fix(query): optimize instance by domain query (#7513) 2024-03-06 18:02:16 +00:00
instance_by_id.sql fix: assign instance ID to aggregate ID when converting from v1 to v2 feature (#7505) 2024-03-05 16:12:49 +01:00
instance_domain_test.go feat(storage): read only transactions for queries (#6415) 2023-08-22 10:49:22 +00:00
instance_domain.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
instance_features_model.go refactor(query): use new packages for org by id query (#7826) 2024-05-24 13:32:57 +02:00
instance_features_test.go fix: add action v2 execution to features (#7597) 2024-04-09 20:21:21 +03:00
instance_features.go refactor(query): use new packages for org by id query (#7826) 2024-05-24 13:32:57 +02:00
instance_test.go feat(api): feature flags (#7356) 2024-02-28 10:55:54 +02:00
instance.go feat: improve instance not found error (#7413) 2024-02-28 10:49:57 +00:00
introspection_client_by_id.sql feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
introspection_test.go feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
introspection.go feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
key_test.go fix(oidc): ignore public key expiry for ID Token hints (#7293) 2024-01-29 15:11:52 +00:00
key.go fix(oidc): ignore public key expiry for ID Token hints (#7293) 2024-01-29 15:11:52 +00:00
label_policy.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
lockout_policy_test.go feat: provide option to limit (T)OTP checks (#7693) 2024-04-10 09:14:55 +00:00
lockout_policy.go feat: provide option to limit (T)OTP checks (#7693) 2024-04-10 09:14:55 +00:00
login_name.go perf: remove owner removed columns from projections for oidc (#6925) 2023-11-20 17:21:08 +02:00
login_policy_test.go chore: use pgx v5 (#7577) 2024-03-27 15:48:22 +02:00
login_policy.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
mail_template.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
member_roles.go fix(query): realtime data on defined requests (#3726) 2022-06-14 07:51:00 +02:00
member.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
message_text_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
message_text.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
milestone_test.go feat(storage): read only transactions for queries (#6415) 2023-08-22 10:49:22 +00:00
milestone.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
notification_policy_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
notification_policy.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
notification_provider_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
notification_provider.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
oidc_client_by_id.sql feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
oidc_client_test.go feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
oidc_client.go feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
oidc_settings_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
oidc_settings.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
org_domain_test.go feat(storage): read only transactions for queries (#6415) 2023-08-22 10:49:22 +00:00
org_domain.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
org_member_test.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
org_member.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
org_metadata_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
org_metadata.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
org_test.go chore: use pgx v5 (#7577) 2024-03-27 15:48:22 +02:00
org.go refactor(query): use new packages for org by id query (#7826) 2024-05-24 13:32:57 +02:00
password_age_policy_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
password_age_policy.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
password_complexity_policy_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
password_complexity_policy.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
prepare_test.go chore: use pgx v5 (#7577) 2024-03-27 15:48:22 +02:00
privacy_policy_test.go feat(cnsl): docs link can be customized and custom button is available (#7840) 2024-05-13 16:01:50 +02:00
privacy_policy.go feat(cnsl): docs link can be customized and custom button is available (#7840) 2024-05-13 16:01:50 +02:00
project_grant_member_test.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
project_grant_member.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
project_grant_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
project_grant.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
project_member_test.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
project_member.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
project_role_test.go perf: remove owner removed columns from projections for oidc (#6925) 2023-11-20 17:21:08 +02:00
project_role.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
project_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
project.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
query_test.go fix(setup): init projections (#7194) 2024-01-25 17:28:20 +01:00
query.go feat(cmd): mirror (#7004) 2024-05-30 09:35:30 +00:00
quota_notifications_test.go perf: project quotas and usages (#6441) 2023-09-15 16:58:45 +02:00
quota_notifications.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
quota_periods_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
quota_periods.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
quota_test.go chore: use pgx v5 (#7577) 2024-03-27 15:48:22 +02:00
quota.go chore: use pgx v5 (#7577) 2024-03-27 15:48:22 +02:00
restrictions_test.go fix: projection version of restrictions (#7028) 2023-12-06 10:30:56 +00:00
restrictions.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
search_query_test.go feat: add action v2 execution on requests and responses (#7637) 2024-05-04 11:55:57 +02:00
search_query.go feat: add action v2 execution on requests and responses (#7637) 2024-05-04 11:55:57 +02:00
secret_generator_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
secret_generators.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
security_policy.go feat: impersonation roles (#7442) 2024-02-28 10:21:11 +00:00
session.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
sessions_test.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
sms_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
sms.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
smtp_test.go feat: SMTP Templates (#6932) 2024-04-11 09:16:10 +02:00
smtp.go feat: SMTP Templates (#6932) 2024-04-11 09:16:10 +02:00
system_features_model.go refactor(query): use new packages for org by id query (#7826) 2024-05-24 13:32:57 +02:00
system_features_test.go fix: add action v2 execution to features (#7597) 2024-04-09 20:21:21 +03:00
system_features.go refactor(query): use new packages for org by id query (#7826) 2024-05-24 13:32:57 +02:00
target_test.go feat: add action v2 execution on requests and responses (#7637) 2024-05-04 11:55:57 +02:00
target.go feat: add action v2 execution on requests and responses (#7637) 2024-05-04 11:55:57 +02:00
targets_by_execution_id.sql fix(actions): correct statements to query targets (#8006) 2024-05-24 14:13:36 +00:00
targets_by_execution_ids.sql fix(actions): correct statements to query targets (#8006) 2024-05-24 14:13:36 +00:00
user_auth_method_test.go fix(oidc): IDP and passwordless user auth methods (#7998) 2024-05-28 08:59:49 +00:00
user_auth_method.go fix(oidc): IDP and passwordless user auth methods (#7998) 2024-05-28 08:59:49 +00:00
user_by_id.sql feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
user_by_login_name.sql fix: correctly check user by loginname (#7740) 2024-04-10 07:18:57 +00:00
user_grant_test.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
user_grant.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
user_membership_test.go perf: remove owner removed columns from projections for oidc (#6925) 2023-11-20 17:21:08 +02:00
user_membership.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
user_metadata_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
user_metadata.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
user_notify_by_id.sql feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
user_notify_by_login_name.sql fix: correctly check user by loginname (#7740) 2024-04-10 07:18:57 +00:00
user_otp.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
user_password.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
user_personal_access_token_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
user_personal_access_token.go refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
user_schema_test.go chore: use pgx v5 (#7577) 2024-03-27 15:48:22 +02:00
user_schema.go chore: use pgx v5 (#7577) 2024-03-27 15:48:22 +02:00
user_test.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
user.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
userinfo_by_id.sql feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
userinfo_client_by_id.sql feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
userinfo_test.go feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
userinfo.go feat(oidc): optimize the userinfo endpoint (#7706) 2024-04-09 15:15:35 +02:00
zitadel_permission.go perf: remove owner removed columns from projections for oidc (#6925) 2023-11-20 17:21:08 +02:00