feat: call webhooks at least once (#5454)

* feat: call webhooks at least once

* self review

* feat: improve notification observability

* feat: add notification tracing

* test(e2e): test at-least-once webhook delivery

* fix webhook notifications

* dedicated quota notifications handler

* fix linting

* fix e2e test

* wait less in e2e test

* fix: don't ignore failed events in handlers

* fix: don't ignore failed events in handlers

* faster requeues

* question

* fix retries

* fix retries

* retry

* don't instance ids query

* revert handler_projection

* statements can be nil

* cleanup

* make unit tests pass

* add comments

* add comments

* lint

* spool only active instances

* feat(config): handle inactive instances

* customizable HandleInactiveInstances

* call inactive instances quota webhooks

* test: handling with and w/o inactive instances

* omit retrying noop statements

* docs: describe projection options

* enable global handling of inactive instances

* self review

* requeue quota notifications every 5m

* remove caos_errors reference

* fix comment styles

* make handlers package flat

* fix linting

* fix repeating quota notifications

* test with more usage

* debug log channel init failures

e2e/config/host.docker.internal/zitadel.yaml

@@ -1,3 +1,6 @@
+Log:
+  Level: debug
+
 ExternalDomain: host.docker.internal
 ExternalSecure: false
 
@@ -32,6 +35,11 @@ Quotas:
     ExhaustedCookieKey: "zitadel.quota.limiting"
     ExhaustedCookieMaxAge: "60s"
 
+Projections:
+  Customizations:
+    NotificationsQuotas:
+      RequeueEvery: 1s
+
 DefaultInstance:
   LoginPolicy:
     MfaInitSkipLifetime: "0"

e2e/config/localhost/docker-compose.yaml

@@ -27,4 +27,5 @@ services:
       retries: 5
       start_period: '20s'
     ports:
-      - "26257:26257"
+      - "26257:26257"
+      - "9090:9090"

e2e/config/localhost/zitadel.yaml

@@ -1,3 +1,6 @@
+Log:
+  Level: debug
+
 ExternalDomain: localhost
 ExternalSecure: false
 
@@ -32,6 +35,11 @@ Quotas:
     ExhaustedCookieKey: "zitadel.quota.limiting"
     ExhaustedCookieMaxAge: "60s"
 
+Projections:
+  Customizations:
+    NotificationsQuotas:
+      RequeueEvery: 1s
+
 DefaultInstance:
   LoginPolicy:
     MfaInitSkipLifetime: "0"