zitadel/internal
Tim Möhlmann ff70ede7c7
feat(eventstore): exclude aggregate IDs when event_type occurred (#8940)
# Which Problems Are Solved

For truly event-based notification handler, we need to be able to filter
out events of aggregates which are already handled. For example when an
event like `notify.success` or `notify.failed` was created on an
aggregate, we no longer require events from that aggregate ID.

# How the Problems Are Solved

Extend the query builder to use a `NOT IN` clause which excludes
aggregate IDs when they have certain events for a certain aggregate
type. For optimization and proper index usages, certain filters are
inherited from the parent query, such as:

- Instance ID
- Instance IDs
- Position offset

This is a prettified query as used by the unit tests:

```sql
SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision
FROM eventstore.events2
WHERE instance_id = $1
    AND aggregate_type = $2 
    AND event_type = $3
    AND "position" > $4
    AND aggregate_id NOT IN (
        SELECT aggregate_id
        FROM eventstore.events2
        WHERE aggregate_type = $5
        AND event_type = ANY($6)
        AND instance_id = $7
        AND "position" > $8
    )
ORDER BY "position" DESC, in_tx_order DESC
LIMIT $9
```

I used this query to run it against the `oidc_session` aggregate looking
for added events, excluding aggregates where a token was revoked,
against a recent position. It fully used index scans:

<details>

```json
[
  {
    "Plan": {
      "Node Type": "Index Scan",
      "Parallel Aware": false,
      "Async Capable": false,
      "Scan Direction": "Forward",
      "Index Name": "es_projection",
      "Relation Name": "events2",
      "Alias": "events2",
      "Actual Rows": 2,
      "Actual Loops": 1,
      "Index Cond": "((instance_id = '286399006995644420'::text) AND (aggregate_type = 'oidc_session'::text) AND (event_type = 'oidc_session.added'::text) AND (\"position\" > 1731582100.784168))",
      "Rows Removed by Index Recheck": 0,
      "Filter": "(NOT (hashed SubPlan 1))",
      "Rows Removed by Filter": 1,
      "Plans": [
        {
          "Node Type": "Index Scan",
          "Parent Relationship": "SubPlan",
          "Subplan Name": "SubPlan 1",
          "Parallel Aware": false,
          "Async Capable": false,
          "Scan Direction": "Forward",
          "Index Name": "es_projection",
          "Relation Name": "events2",
          "Alias": "events2_1",
          "Actual Rows": 1,
          "Actual Loops": 1,
          "Index Cond": "((instance_id = '286399006995644420'::text) AND (aggregate_type = 'oidc_session'::text) AND (event_type = 'oidc_session.access_token.revoked'::text) AND (\"position\" > 1731582100.784168))",
          "Rows Removed by Index Recheck": 0
        }
      ]
    },
    "Triggers": [
    ]
  }
]
```

</details>

# Additional Changes

- None

# Additional Context

- Related to https://github.com/zitadel/zitadel/issues/8931

---------

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2024-11-25 15:25:11 +00:00
..
actions fix: correctly check denied domains and ips for actions (#8810) 2024-10-22 16:16:44 +02:00
activity feat: trusted (instance) domains (#8369) 2024-07-31 18:00:38 +03:00
admin/repository/eventsourcing feat(cmd): mirror (#7004) 2024-05-30 09:35:30 +00:00
api fix(saml): provide option to get internal as default ACS (#8888) 2024-11-15 07:19:43 +01:00
auth/repository feat(OIDC): add back channel logout (#8837) 2024-10-31 15:57:17 +01:00
auth_request/repository fix(login): correctly reload policies on auth request (#7839) 2024-04-24 08:44:55 +00:00
authz docs: standardize multi-factor spelling and related string updates (#8752) 2024-10-22 14:59:16 +00:00
cache feat(cache): organization (#8903) 2024-11-21 08:05:03 +02:00
command feat(eventstore): accept transaction in push (#8945) 2024-11-22 17:25:28 +01:00
config feat(v3alpha): read actions (#8357) 2024-08-12 22:32:01 +02:00
crypto perf(query): remove transactions for queries (#8614) 2024-11-04 10:06:14 +01:00
database feat(eventstore): accept transaction in push (#8945) 2024-11-22 17:25:28 +01:00
domain fix(oidc): do not return access token for response type id_token (#8777) 2024-11-12 15:20:48 +00:00
eventstore feat(eventstore): exclude aggregate IDs when event_type occurred (#8940) 2024-11-25 15:25:11 +00:00
execution feat: add schema user create and remove (#8494) 2024-08-28 19:46:45 +00:00
feature feat(OIDC): add back channel logout (#8837) 2024-10-31 15:57:17 +01:00
form refactor: rename package errors to zerrors (#7039) 2023-12-08 15:30:55 +01:00
i18n fix(middleware): init translation messages (#7778) 2024-04-16 12:08:18 +00:00
iam refactor: cleanup unused code (#7130) 2024-01-02 14:26:31 +00:00
id feat(cmd): Added machine ID mode to zitadel start up logs (#8251) 2024-07-16 09:53:57 +00:00
idp fix(SAML): log underlying error if SAML response validation fails (#8721) 2024-10-11 07:04:15 +00:00
integration feat(cache): organization (#8903) 2024-11-21 08:05:03 +02:00
logstore perf: project quotas and usages (#6441) 2023-09-15 16:58:45 +02:00
migration fix(setup): improve search query to use index (#8898) 2024-11-13 07:50:23 +00:00
net perf: project quotas and usages (#6441) 2023-09-15 16:58:45 +02:00
notification feat(OIDC): add back channel logout (#8837) 2024-10-31 15:57:17 +01:00
org refactor: cleanup unused code (#7130) 2024-01-02 14:26:31 +00:00
project refactor: cleanup unused code (#7130) 2024-01-02 14:26:31 +00:00
protoc merge main into next 2023-10-19 12:34:00 +02:00
qrcode docs(legal): Updated agreements and policies v2 (#3823) 2022-06-15 08:30:58 +02:00
query feat(cache): organization (#8903) 2024-11-21 08:05:03 +02:00
renderer fix(login): (re)allow HTML in custom login texts (#7575) 2024-03-15 16:29:10 +01:00
repository fix(milestones): use previous spelling for milestone types (#8886) 2024-11-11 11:28:27 +00:00
static docs: standardize multi-factor spelling and related string updates (#8752) 2024-10-22 14:59:16 +00:00
statik chore: initial version of a devcontainer (#6352) 2023-08-15 10:49:05 +02:00
telemetry feat: enable application performance profiling (#8442) 2024-08-16 13:26:53 +00:00
test refactor(fmt): run gci on complete project (#7557) 2024-04-03 10:43:43 +00:00
user feat(OIDC): add back channel logout (#8837) 2024-10-31 15:57:17 +01:00
v2 fix(eventstore): revert precise decimal (#8527) (#8679) 2024-09-24 18:43:29 +02:00
view/repository perf(query): remove transactions for queries (#8614) 2024-11-04 10:06:14 +01:00
webauthn fix: add domain as attribute to list user auth methods (#8718) 2024-10-10 16:50:53 +00:00
zerrors fix(crypto): reject decrypted strings with non-UTF8 characters. (#8374) 2024-08-02 08:38:37 +00:00