zitadel/internal/domain
Livio Spring 8537805ea5
feat(notification): use event worker pool (#8962)
# Which Problems Are Solved

The current handling of notification follows the same pattern as all
other projections:
Created events are handled sequentially (based on "position") by a
handler. During the process, a lot of information is aggregated (user,
texts, templates, ...).
This leads to back pressure on the projection since the handling of
events might take longer than the time before a new event (to be
handled) is created.

# How the Problems Are Solved

- The current user notification handler creates separate notification
events based on the user / session events.
- These events contain all the present and required information
including the userID.
- These notification events get processed by notification workers, which
gather the necessary information (recipient address, texts, templates)
to send out these notifications.
- If a notification fails, a retry event is created based on the current
notification request including the current state of the user (this
prevents race conditions, where a user is changed in the meantime and
the notification already gets the new state).
- The retry event will be handled after a backoff delay. This delay
increases with every attempt.
- If the configured amount of attempts is reached or the message expired
(based on config), a cancel event is created, letting the workers know,
the notification must no longer be handled.
- In case of successful send, a sent event is created for the
notification aggregate and the existing "sent" events for the user /
session object is stored.
- The following is added to the defaults.yaml to allow configuration of
the notification workers:
```yaml

Notifications:
  # The amount of workers processing the notification request events.
  # If set to 0, no notification request events will be handled. This can be useful when running in
  # multi binary / pod setup and allowing only certain executables to process the events.
  Workers: 1 # ZITADEL_NOTIFIACATIONS_WORKERS
  # The amount of events a single worker will process in a run.
  BulkLimit: 10 # ZITADEL_NOTIFIACATIONS_BULKLIMIT
  # Time interval between scheduled notifications for request events
  RequeueEvery: 2s # ZITADEL_NOTIFIACATIONS_REQUEUEEVERY
  # The amount of workers processing the notification retry events.
  # If set to 0, no notification retry events will be handled. This can be useful when running in
  # multi binary / pod setup and allowing only certain executables to process the events.
  RetryWorkers: 1 # ZITADEL_NOTIFIACATIONS_RETRYWORKERS
  # Time interval between scheduled notifications for retry events
  RetryRequeueEvery: 2s # ZITADEL_NOTIFIACATIONS_RETRYREQUEUEEVERY
  # Only instances are projected, for which at least a projection-relevant event exists within the timeframe
  # from HandleActiveInstances duration in the past until the projection's current time
  # If set to 0 (default), every instance is always considered active
  HandleActiveInstances: 0s # ZITADEL_NOTIFIACATIONS_HANDLEACTIVEINSTANCES
  # The maximum duration a transaction remains open
  # before it spots left folding additional events
  # and updates the table.
  TransactionDuration: 1m # ZITADEL_NOTIFIACATIONS_TRANSACTIONDURATION
  # Automatically cancel the notification after the amount of failed attempts
  MaxAttempts: 3 # ZITADEL_NOTIFIACATIONS_MAXATTEMPTS
  # Automatically cancel the notification if it cannot be handled within a specific time
  MaxTtl: 5m  # ZITADEL_NOTIFIACATIONS_MAXTTL
  # Failed attempts are retried after a confogired delay (with exponential backoff).
  # Set a minimum and maximum delay and a factor for the backoff
  MinRetryDelay: 1s  # ZITADEL_NOTIFIACATIONS_MINRETRYDELAY
  MaxRetryDelay: 20s # ZITADEL_NOTIFIACATIONS_MAXRETRYDELAY
  # Any factor below 1 will be set to 1
  RetryDelayFactor: 1.5 # ZITADEL_NOTIFIACATIONS_RETRYDELAYFACTOR
```


# Additional Changes

None

# Additional Context

- closes #8931
2024-11-27 15:01:17 +00:00
..
schema feat: add schema user create and remove (#8494) 2024-08-28 19:46:45 +00:00
testdata/fuzz/FuzzFromRefreshToken fix(crypto): reject decrypted strings with non-UTF8 characters. (#8374) 2024-08-02 08:38:37 +00:00
action.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
application_api.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
application_key.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
application_oauth.go feat(6222): remove @ and project from OIDC client ID (#8178) 2024-07-04 08:31:40 +00:00
application_oidc_test.go fix: remove hard requirement of grant type auth code for device code apps + warnings for missing urls (#7429) 2024-02-29 15:28:06 +00:00
application_oidc.go fix(oidc): do not return access token for response type id_token (#8777) 2024-11-12 15:20:48 +00:00
application_saml.go feat(saml): implementation of saml for ZITADEL v2 (#3618) 2022-09-12 18:18:08 +02:00
application.go feat: protos refactoring 2021-03-09 10:30:11 +01:00
asset.go fix: return absolute asset urls (#3676) 2022-05-20 10:30:12 +02:00
auth_request_test.go feat(oidc): id token for device authorization (#7088) 2023-12-20 13:21:08 +01:00
auth_request.go feat(oidc): sid claim for id_tokens issued through login V1 (#8525) 2024-09-03 13:19:00 +00:00
authn_key.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
browser_info.go fix: correctly set user agent / fingerprint id on user sessions (#8231) 2024-07-03 09:43:34 +02:00
bucket.go feat: asset storage (#1696) 2021-05-03 10:15:50 +02:00
custom_login_text.go fix: automatically link user without prompt (#8487) 2024-08-28 05:33:20 +00:00
custom_message_text.go feat: invite user link (#8578) 2024-09-11 10:53:55 +00:00
custom_text.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
debug_events.go feat: add debug events API (#8533) 2024-09-11 08:24:00 +00:00
device_auth_test.go feat(oidc): id token for device authorization (#7088) 2023-12-20 13:21:08 +01:00
device_auth.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
deviceauthstate_string.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
execution.go feat: add action v2 execution on requests and responses (#7637) 2024-05-04 11:55:57 +02:00
expiration.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
factors.go feat: enable otp email and sms (#6260) 2023-07-28 07:39:30 +02:00
feature.go fix: add action v2 execution to features (#7597) 2024-04-09 20:21:21 +03:00
flow.go feat: add executions for actions v2 (#7433) 2024-02-26 12:49:43 +02:00
human_address.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
human_email_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
human_email.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
human_otp.go fix: import totp in add human user with secret (#7936) 2024-05-14 09:20:31 +02:00
human_password.go feat(crypto): use passwap for machine and app secrets (#7657) 2024-04-05 09:35:49 +00:00
human_phone_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
human_phone.go feat: Add Twilio Verification Service (#8678) 2024-09-26 09:14:33 +02:00
human_profile.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
human_test.go fix: set displayname correctly in EnsureDisplayName (#5702) 2023-04-17 06:26:40 +00:00
human_web_auth_n.go feat(notification): use event worker pool (#8962) 2024-11-27 15:01:17 +00:00
human.go fix(import): add tracing spans to all import related functions (#8160) 2024-06-19 12:56:33 +02:00
idp_config.go feat(login): use new IDP templates (#5315) 2023-02-28 21:20:58 +01:00
idp.go feat(saml): allow setting nameid-format and alternative mapping for transient format (#7979) 2024-05-23 05:04:07 +00:00
instance_domain.go feat: add random string to generated domain (#3634) 2022-05-16 11:26:24 +02:00
instance.go fix: instance remove (#4602) 2022-10-26 13:06:48 +00:00
key_pair.go feat(v3alpha): web key resource (#8262) 2024-08-14 14:18:14 +00:00
language.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
machine_key.go fix: add expiration date information to service users keys (#7497) 2024-03-13 18:21:19 +00:00
machine.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
member.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
metadata.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
mfa.go refactor: cleanup unused code (#7130) 2024-01-02 14:26:31 +00:00
next_step.go feat: invite user link (#8578) 2024-09-11 10:53:55 +00:00
notification.go feat(notification): use event worker pool (#8962) 2024-11-27 15:01:17 +00:00
object.go feat(v3alpha): read actions (#8357) 2024-08-12 22:32:01 +02:00
oidc_code_challenge.go fix: move v2 pkgs (#1331) 2021-02-23 15:13:04 +01:00
oidc_error_reason_test.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
oidc_error_reason.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
oidc_mapping_field.go fix: move v2 pkgs (#1331) 2021-02-23 15:13:04 +01:00
oidc_session.go feat(api): add OIDC session service (#6157) 2023-07-10 13:27:00 +00:00
oidc_settings.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
oidcresponsemode_enumer.go fix(oidc): store requested response_mode (#8145) 2024-06-17 09:50:12 +00:00
org_domain_test.go fix: allow unicode characters in org domains (#6675) 2023-10-11 09:55:01 +02:00
org_domain.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
org.go feat(eventstore): add search table (#8191) 2024-07-03 15:00:56 +00:00
permission.go feat: org v2 ListOrganizations (#8411) 2024-08-15 06:37:06 +02:00
policy_domain.go feat: restrict smtp sender address (#3637) 2022-05-16 14:08:47 +00:00
policy_label_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
policy_label.go refactor: cleanup unused code (#7130) 2024-01-02 14:26:31 +00:00
policy_login_test.go feat: add default redirect uri and handling of unknown usernames (#3616) 2022-05-16 13:39:09 +00:00
policy_login.go fix(login): correct rendering of idps (#7151) 2024-01-05 14:35:51 +00:00
policy_mail_template.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
policy_password_age.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
policy_password_complexity.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
policy_password_lockout.go feat: provide option to limit (T)OTP checks (#7693) 2024-04-10 09:14:55 +00:00
policy_privacy.go feat(cnsl): docs link can be customized and custom button is available (#7840) 2024-05-13 16:01:50 +02:00
policy.go refactor: cleanup unused code (#7130) 2024-01-02 14:26:31 +00:00
project_grant_member.go refactor: cleanup unused code (#7130) 2024-01-02 14:26:31 +00:00
project_grant.go perf(command): user grant pre-condition check using the search table (#8230) 2024-07-04 16:18:43 +00:00
project_role.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
project.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
provider.go refactor: cleanup unused code (#7130) 2024-01-02 14:26:31 +00:00
refresh_token_test.go fix(crypto): reject decrypted strings with non-UTF8 characters. (#8374) 2024-08-02 08:38:37 +00:00
refresh_token.go fix(crypto): reject decrypted strings with non-UTF8 characters. (#8374) 2024-08-02 08:38:37 +00:00
request.go fix: ignore projectID and origin check for service accounts (#8704) 2024-10-01 16:38:28 +02:00
roles.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
search_method.go fix: todos (#1346) 2021-03-01 08:48:50 +01:00
secret_generator.go feat: invite user link (#8578) 2024-09-11 10:53:55 +00:00
secretgeneratortype_enumer.go feat: invite user link (#8578) 2024-09-11 10:53:55 +00:00
session.go fix: pass sessionID to OTP email link (#8745) 2024-10-10 13:53:32 +00:00
sms.go feat: Default configs sms provider (#3187) 2022-02-21 12:22:20 +00:00
smtp.go feat: SMTP Templates (#6932) 2024-04-11 09:16:10 +02:00
target.go feat: add action v2 execution on requests and responses (#7637) 2024-05-04 11:55:57 +02:00
token_test.go feat(oidc): organization roles scope (#8120) 2024-06-14 10:00:43 +02:00
token.go feat(oidc): organization roles scope (#8120) 2024-06-14 10:00:43 +02:00
tokenreason_enumer.go feat(oidc): token exchange impersonation (#7516) 2024-03-20 10:18:46 +00:00
url_template_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
url_template.go feat(notification): use event worker pool (#8962) 2024-11-27 15:01:17 +00:00
user_agent_test.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
user_agent.go perf(oidc): optimize token creation (#7822) 2024-05-16 07:07:56 +02:00
user_grant.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
user_idp_link.go chore(v2): move to new org (#3499) 2022-04-26 23:01:45 +00:00
user_schema.go feat: implement user schema management (#7416) 2024-03-12 13:50:13 +00:00
user_v2_passkey_test.go refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
user_v2_passkey.go feat(v2): register user u2f (#6020) 2023-06-15 05:32:40 +00:00
user.go fix(oidc): IDP and passwordless user auth methods (#7998) 2024-05-28 08:59:49 +00:00
web_key.go feat(v3alpha): web key resource (#8262) 2024-08-14 14:18:14 +00:00