3769 Commits

Author SHA1 Message Date
Livio Spring
87dc3f01fc chore: use crdb 24.3 (#9558)
# Which Problems Are Solved

E2E tests in pipelines started to fail randomly. While debugging it, i
noticed that we use the `latest` tag of cockroach's docker image. They
tagged 25.1 as latest yesterday.

# How the Problems Are Solved

Since we drop support for CRDB with version 3 as there are anyway
multiple issues with various versions, I pinned the docker image tag to
`latest-v24.3`.

# Additional Changes

None

# Additional Context

relates to https://github.com/zitadel/zitadel/actions/runs/13917603587
and https://github.com/zitadel/zitadel/actions/runs/13904928050

(cherry picked from commit f1f500d0e7)
2025-03-18 16:40:19 +01:00
Harsha Reddy
fd9737a3ed fix: reduce cardinality in metrics and tracing for unknown paths (#9523)
# Which Problems Are Solved
Zitadel should not record 404 response counts of unknown paths (check
`/debug/metrics`).
This can lead to high cardinality on metrics endpoint and in traces.

```
GOOD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="200",uri="/.well-known/openid-configuration"} 2
GOOD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="200",uri="/oauth/v2/keys"} 2
BAD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="404",uri="/junk"} 2000
```

After
```
GOOD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="200",uri="/.well-known/openid-configuration"} 2
GOOD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="200",uri="/oauth/v2/keys"} 2
```

# How the Problems Are Solved

This PR makes sure, that any unknown path is recorded as `UNKNOWN_PATH`
instead of the actual path.

# Additional Changes

N/A

# Additional Context

On our production instance, when a penetration test was run, it caused
our metric count to blow up to many thousands due to Zitadel recording
404 response counts.

Next nice to have steps, remove 404 timer recordings which serve no
purpose

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
Co-authored-by: Livio Spring <livio@zitadel.com>
(cherry picked from commit 599850e7e8)
2025-03-18 16:40:16 +01:00
Silvan
11180cfe93 fix(perf): simplify eventstore queries by removing or in projection handlers (#9530)
# Which Problems Are Solved

[A recent performance
enhancement]((https://github.com/zitadel/zitadel/pull/9497)) aimed at
optimizing event store queries, specifically those involving multiple
aggregate type filters, has successfully improved index utilization.
While the query planner now correctly selects relevant indexes, it
employs [bitmap index
scans](https://www.postgresql.org/docs/current/indexes-bitmap-scans.html)
to retrieve data.

This approach, while beneficial in many scenarios, introduces a
potential I/O bottleneck. The bitmap index scan first identifies the
required database blocks and then utilizes a bitmap to access the
corresponding rows from the table's heap. This subsequent "bitmap heap
scan" can result in significant I/O overhead, particularly when queries
return a substantial number of rows across numerous data pages.

## Impact:

Under heavy load or with queries filtering for a wide range of events
across multiple aggregate types, this increased I/O activity may lead
to:

- Increased query latency.
- Elevated disk utilization.
- Potential performance degradation of the event store and dependent
systems.

# How the Problems Are Solved

To address this I/O bottleneck and further optimize query performance,
the projection handler has been modified. Instead of employing multiple
OR clauses for each aggregate type, the aggregate and event type filters
are now combined using IN ARRAY filters.

Technical Details:

This change allows the PostgreSQL query planner to leverage [index-only
scans](https://www.postgresql.org/docs/current/indexes-index-only-scans.html).
By utilizing IN ARRAY filters, the database can efficiently retrieve the
necessary data directly from the index, eliminating the need to access
the table's heap. This results in:

* Reduced I/O: Index-only scans significantly minimize disk I/O
operations, as the database avoids reading data pages from the main
table.
* Improved Query Performance: By reducing I/O, query execution times are
substantially improved, leading to lower latency.

# Additional Changes

- rollback of https://github.com/zitadel/zitadel/pull/9497

# Additional Information

## Query Plan of previous query

```sql
SELECT
    created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision
FROM
    eventstore.events2
WHERE
    instance_id = '<INSTANCE_ID>'
    AND (
        (
            instance_id = '<INSTANCE_ID>'
            AND "position" > <POSITION>
            AND aggregate_type = 'project'
            AND event_type = ANY(ARRAY[
                                'project.application.added'
                                ,'project.application.changed'
                                ,'project.application.deactivated'
                                ,'project.application.reactivated'
                                ,'project.application.removed'
                                ,'project.removed'
                                ,'project.application.config.api.added'
                                ,'project.application.config.api.changed'
                                ,'project.application.config.api.secret.changed'
                ,'project.application.config.api.secret.updated'
                                ,'project.application.config.oidc.added'
                                ,'project.application.config.oidc.changed'
                                ,'project.application.config.oidc.secret.changed'
                ,'project.application.config.oidc.secret.updated'
                                ,'project.application.config.saml.added'
                                ,'project.application.config.saml.changed'
                        ])
        ) OR (
            instance_id = '<INSTANCE_ID>'
            AND "position" > <POSITION>
            AND aggregate_type = 'org'
            AND event_type = 'org.removed'
        ) OR (
            instance_id = '<INSTANCE_ID>'
            AND "position" > <POSITION>
            AND aggregate_type = 'instance'
            AND event_type = 'instance.removed'
        )
    )
    AND "position" > 1741600905.3495
    AND "position" < (
        SELECT
            COALESCE(EXTRACT(EPOCH FROM min(xact_start)), EXTRACT(EPOCH FROM now()))
        FROM
            pg_stat_activity
        WHERE
            datname = current_database()
            AND application_name = ANY(ARRAY['zitadel_es_pusher_', 'zitadel_es_pusher', 'zitadel_es_pusher_<INSTANCE_ID>'])
            AND state <> 'idle'
    )
ORDER BY "position", in_tx_order LIMIT 200 OFFSET 1;
```

```
Limit  (cost=120.08..120.09 rows=7 width=361) (actual time=2.167..2.172 rows=0 loops=1)
   Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
   InitPlan 1
     ->  Aggregate  (cost=2.74..2.76 rows=1 width=32) (actual time=1.813..1.815 rows=1 loops=1)
           Output: COALESCE(EXTRACT(epoch FROM min(s.xact_start)), EXTRACT(epoch FROM now()))
           ->  Nested Loop  (cost=0.00..2.74 rows=1 width=8) (actual time=1.803..1.805 rows=0 loops=1)
                 Output: s.xact_start
                 Join Filter: (d.oid = s.datid)
                 ->  Seq Scan on pg_catalog.pg_database d  (cost=0.00..1.07 rows=1 width=4) (actual time=0.016..0.021 rows=1 loops=1)
                       Output: d.oid, d.datname, d.datdba, d.encoding, d.datlocprovider, d.datistemplate, d.datallowconn, d.dathasloginevt, d.datconnlimit, d.datfrozenxid, d.datminmxid, d.dattablespace, d.datcollate, d.datctype, d.datlocale, d.daticurules, d.datcollversion, d.datacl
                       Filter: (d.datname = current_database())
                       Rows Removed by Filter: 4
                 ->  Function Scan on pg_catalog.pg_stat_get_activity s  (cost=0.00..1.63 rows=3 width=16) (actual time=1.781..1.781 rows=0 loops=1)
                       Output: s.datid, s.pid, s.usesysid, s.application_name, s.state, s.query, s.wait_event_type, s.wait_event, s.xact_start, s.query_start, s.backend_start, s.state_change, s.client_addr, s.client_hostname, s.client_port, s.backend_xid, s.backend_xmin, s.backend_type, s.ssl, s.sslversion, s.sslcipher, s.sslbits, s.ssl_client_dn, s.ssl_client_serial, s.ssl_issuer_dn, s.gss_auth, s.gss_princ, s.gss_enc, s.gss_delegation, s.leader_pid, s.query_id
                       Function Call: pg_stat_get_activity(NULL::integer)
                       Filter: ((s.state <> 'idle'::text) AND (s.application_name = ANY ('{zitadel_es_pusher_,zitadel_es_pusher,zitadel_es_pusher_<INSTANCE_ID>}'::text[])))
                       Rows Removed by Filter: 49
   ->  Sort  (cost=117.31..117.33 rows=8 width=361) (actual time=2.167..2.168 rows=0 loops=1)
         Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
         Sort Key: events2."position", events2.in_tx_order
         Sort Method: quicksort  Memory: 25kB
         ->  Bitmap Heap Scan on eventstore.events2  (cost=84.92..117.19 rows=8 width=361) (actual time=2.088..2.089 rows=0 loops=1)
               Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
               Recheck Cond: (((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'project'::text) AND (events2.event_type = ANY ('{project.application.added,project.application.changed,project.application.deactivated,project.application.reactivated,project.application.removed,project.removed,project.application.config.api.added,project.application.config.api.changed,project.application.config.api.secret.changed,project.application.config.api.secret.updated,project.application.config.oidc.added,project.application.config.oidc.changed,project.application.config.oidc.secret.changed,project.application.config.oidc.secret.updated,project.application.config.saml.added,project.application.config.saml.changed}'::text[])) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1)) OR ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'org'::text) AND (events2.event_type = 'org.removed'::text) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1)) OR ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'instance'::text) AND (events2.event_type = 'instance.removed'::text) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1)))
               ->  BitmapOr  (cost=84.88..84.88 rows=8 width=0) (actual time=2.080..2.081 rows=0 loops=1)
                     ->  Bitmap Index Scan on es_projection  (cost=0.00..75.44 rows=8 width=0) (actual time=2.016..2.017 rows=0 loops=1)
                           Index Cond: ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'project'::text) AND (events2.event_type = ANY ('{project.application.added,project.application.changed,project.application.deactivated,project.application.reactivated,project.application.removed,project.removed,project.application.config.api.added,project.application.config.api.changed,project.application.config.api.secret.changed,project.application.config.api.secret.updated,project.application.config.oidc.added,project.application.config.oidc.changed,project.application.config.oidc.secret.changed,project.application.config.oidc.secret.updated,project.application.config.saml.added,project.application.config.saml.changed}'::text[])) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1))
                     ->  Bitmap Index Scan on es_projection  (cost=0.00..4.71 rows=1 width=0) (actual time=0.016..0.016 rows=0 loops=1)
                           Index Cond: ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'org'::text) AND (events2.event_type = 'org.removed'::text) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1))
                     ->  Bitmap Index Scan on es_projection  (cost=0.00..4.71 rows=1 width=0) (actual time=0.045..0.045 rows=0 loops=1)
                           Index Cond: ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'instance'::text) AND (events2.event_type = 'instance.removed'::text) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1))
 Query Identifier: 3194938266011254479
 Planning Time: 1.295 ms
 Execution Time: 2.832 ms
```

## Query Plan of new query

```sql
SELECT
    created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision
FROM
    eventstore.events2
WHERE
    instance_id = '<INSTANCE_ID>'
    AND "position" > <POSITION>
    AND aggregate_type = ANY(ARRAY['project', 'instance', 'org'])
    AND event_type = ANY(ARRAY[
        'project.application.added'
        ,'project.application.changed'
        ,'project.application.deactivated'
        ,'project.application.reactivated'
        ,'project.application.removed'
        ,'project.removed'
        ,'project.application.config.api.added'
        ,'project.application.config.api.changed'
        ,'project.application.config.api.secret.changed'
        ,'project.application.config.api.secret.updated'
        ,'project.application.config.oidc.added'
        ,'project.application.config.oidc.changed'
        ,'project.application.config.oidc.secret.changed'
        ,'project.application.config.oidc.secret.updated'
        ,'project.application.config.saml.added'
        ,'project.application.config.saml.changed'
        ,'org.removed'
        ,'instance.removed'
    ])
    AND "position" < (
        SELECT
            COALESCE(EXTRACT(EPOCH FROM min(xact_start)), EXTRACT(EPOCH FROM now()))
        FROM
            pg_stat_activity
        WHERE
            datname = current_database()
            AND application_name = ANY(ARRAY['zitadel_es_pusher_', 'zitadel_es_pusher', 'zitadel_es_pusher_<INSTANCE_ID>'])
            AND state <> 'idle'
    )
ORDER BY "position", in_tx_order LIMIT 200 OFFSET 1;
```

```
Limit  (cost=293.34..293.36 rows=8 width=361) (actual time=4.686..4.689 rows=0 loops=1)
   Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
   InitPlan 1
     ->  Aggregate  (cost=2.74..2.76 rows=1 width=32) (actual time=1.717..1.719 rows=1 loops=1)
           Output: COALESCE(EXTRACT(epoch FROM min(s.xact_start)), EXTRACT(epoch FROM now()))
           ->  Nested Loop  (cost=0.00..2.74 rows=1 width=8) (actual time=1.658..1.659 rows=0 loops=1)
                 Output: s.xact_start
                 Join Filter: (d.oid = s.datid)
                 ->  Seq Scan on pg_catalog.pg_database d  (cost=0.00..1.07 rows=1 width=4) (actual time=0.026..0.028 rows=1 loops=1)
                       Output: d.oid, d.datname, d.datdba, d.encoding, d.datlocprovider, d.datistemplate, d.datallowconn, d.dathasloginevt, d.datconnlimit, d.datfrozenxid, d.datminmxid, d.dattablespace, d.datcollate, d.datctype, d.datlocale, d.daticurules, d.datcollversion, d.datacl
                       Filter: (d.datname = current_database())
                       Rows Removed by Filter: 4
                 ->  Function Scan on pg_catalog.pg_stat_get_activity s  (cost=0.00..1.63 rows=3 width=16) (actual time=1.628..1.628 rows=0 loops=1)
                       Output: s.datid, s.pid, s.usesysid, s.application_name, s.state, s.query, s.wait_event_type, s.wait_event, s.xact_start, s.query_start, s.backend_start, s.state_change, s.client_addr, s.client_hostname, s.client_port, s.backend_xid, s.backend_xmin, s.backend_type, s.ssl, s.sslversion, s.sslcipher, s.sslbits, s.ssl_client_dn, s.ssl_client_serial, s.ssl_issuer_dn, s.gss_auth, s.gss_princ, s.gss_enc, s.gss_delegation, s.leader_pid, s.query_id
                       Function Call: pg_stat_get_activity(NULL::integer)
                       Filter: ((s.state <> 'idle'::text) AND (s.application_name = ANY ('{zitadel_es_pusher_,zitadel_es_pusher,zitadel_es_pusher_<INSTANCE_ID>}'::text[])))
                       Rows Removed by Filter: 42
   ->  Sort  (cost=290.58..290.60 rows=9 width=361) (actual time=4.685..4.685 rows=0 loops=1)
         Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
         Sort Key: events2."position", events2.in_tx_order
         Sort Method: quicksort  Memory: 25kB
         ->  Index Scan using es_projection on eventstore.events2  (cost=0.70..290.43 rows=9 width=361) (actual time=4.616..4.617 rows=0 loops=1)
               Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
               Index Cond: ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = ANY ('{project,instance,org}'::text[])) AND (events2.event_type = ANY ('{project.application.added,project.application.changed,project.application.deactivated,project.application.reactivated,project.application.removed,project.removed,project.application.config.api.added,project.application.config.api.changed,project.application.config.api.secret.changed,project.application.config.api.secret.updated,project.application.config.oidc.added,project.application.config.oidc.changed,project.application.config.oidc.secret.changed,project.application.config.oidc.secret.updated,project.application.config.saml.added,project.application.config.saml.changed,org.removed,instance.removed}'::text[])) AND (events2."position" > <POSITION>) AND (events2."position" < (InitPlan 1).col1))
 Query Identifier: -8254550537132386499
 Planning Time: 2.864 ms
 Execution Time: 5.414 ms
 ```

(cherry picked from commit e36f402e09)
2025-03-13 16:52:33 +01:00
Silvan
c88e838616 fix(eventstore): optimise query hints for event filters (#9497)
(cherry picked from commit b578137139)
2025-03-12 12:55:30 +01:00
Livio Spring
5f3a9339e2 fix(OIDC): back channel logout work for custom UI (#9487)
# Which Problems Are Solved

When using a custom / new login UI and an OIDC application with
registered BackChannelLogoutUI, no logout requests were sent to the URI
when the user signed out.
Additionally, as described in #9427, an error was logged:
`level=error msg="event of type *session.TerminateEvent doesn't
implement OriginEvent"
caller="/home/runner/work/zitadel/zitadel/internal/notification/handlers/origin.go:24"`

# How the Problems Are Solved

- Properly pass `TriggerOrigin` information to session.TerminateEvent
creation and implement `OriginEvent` interface.
- Implemented `RegisterLogout` in `CreateOIDCSessionFromAuthRequest` and
`CreateOIDCSessionFromDeviceAuth`, both used when interacting with the
OIDC v2 API.
- Both functions now receive the `BackChannelLogoutURI` of the client
from the OIDC layer.

# Additional Changes

None

# Additional Context

- closes #9427

(cherry picked from commit ed697bbd69)
2025-03-12 12:55:26 +01:00
Livio Spring
6e02957f8e fix(token exchange): properly return an error if membership is missing (#9468)
# Which Problems Are Solved

When requesting a JWT (`urn:ietf:params:oauth:token-type:jwt`) to be
returned in a Token Exchange request, ZITADEL would panic if the `actor`
was not granted the necessary permission.

# How the Problems Are Solved

Properly check the error and return it.

# Additional Changes

None

# Additional Context

- closes #9436

(cherry picked from commit e6ce1af003)
2025-03-12 12:55:03 +01:00
Livio Spring
e133fb2a8a fix: correct required permissions on admin APIs
# Which Problems Are Solved

ZITADEL's Admin API, intended for managing ZITADEL instances, contains 12 HTTP endpoints that are unexpectedly accessible to authenticated ZITADEL users who are not ZITADEL managers. The most critical vulnerable endpoints relate to LDAP configuration:
- /idps/ldap
- /idps/ldap/{id}

By accessing these endpoints, unauthorized users could:
- Modify ZITADEL's instance LDAP settings, redirecting all LDAP login attempts to a malicious server, effectively taking over user accounts.
- Expose the original LDAP server's password, potentially compromising all user accounts.

The following endpoints are also affected by IDOR vulnerabilities, potentially allowing unauthorized modification of instance settings such as languages, labels, and templates:
- /idps/templates/_search
- /idps/templates/{id}
- /policies/label/_activate
- /policies/label/logo
- /policies/label/logo_dark
- /policies/label/icon
- /policies/label/icon_dark
- /policies/label/font
- /text/message/passwordless_registration/{language}
- /text/login/{language}

Please checkout https://github.com/zitadel/zitadel/security/advisories/GHSA-f3gh-529w-v32x for more information.

# How the Problems Are Solved

- Required permission have been fixed (only instance level allowed)

# Additional Changes

None

# Additional Context

- resolves https://github.com/zitadel/zitadel/security/advisories/GHSA-f3gh-529w-v32x

(cherry picked from commit d9d8339813)
2025-03-04 08:52:26 +01:00
Livio Spring
b7d8bdc13b fix: prevent panic when retrieving session by id in internal calls (#9442)
# Which Problems Are Solved

#9110 introduced more possibilities to search for "own" sessions. Due to
this the permission checks for retrieving a session had to be updated
accordingly. Internal calls, such as retrieving them for sending
notifications do not require a permission, but the code was not properly
adjusted and thus could lead to panics.

# How the Problems Are Solved

- Properly handled (do not require) permission check for internal only
calls when retrieving the session by id.

# Additional Changes

None

# Additional Context

- needs backports to 2.68.x, 2.69.x, 2.70.x
- closes zitadel/devops#117

(cherry picked from commit 4e1868e9bb)
2025-03-03 13:09:44 +01:00
Livio Spring
42dfda533a create maintenance branch 2025-03-03 13:09:40 +01:00
Tim Möhlmann
92265dca21 fix(setup): use template for in_tx_order type (#9346)
# Which Problems Are Solved

Systems running with PostgreSQL before Zitadel v2.39 are likely to have
a wrong type for the `in_tx_order` column in the `eventstore.event2`
table. The migration at the time used the `event_sequence` as default
value without typecast, which results in a `bigint` type for that
column. However, when creating the table from scratch, we explicitly
specify the type to be `integer`.

Starting from Zitadel v2.67 we use a Pl/PgSQL function to push events.
The function requires the types from `eventstore.events2` to the same as
the `select` destinations used in the function. In the function
`in_tx_order` is also expected to by of `integer` type.

CochroachDB systems are not affected because `bigint` is an alias to the
`int` type. In other words, CockroachDB uses `int8` when specifying type
`int`. Therefore the types already match.

# How the Problems Are Solved

Retrieve the actual column type currently in use. A template is used to
assign the type to the `ordinality` column returned as `in_tx_order`.

# Additional Changes

- Detailed logging on migration failure

# Additional Context

- Closes #9180

---------

Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
(cherry picked from commit bcc6a689fa)
2025-02-13 17:15:36 +01:00
Livio Spring
464a4718df fix(oidc / login v2): always us login v2 if x-zitadel-login-client header is sent (#9336)
# Which Problems Are Solved

As reported in #9311, even when providing a `x-zitadel-login-client`
header, the auth request would be created as hosted login UI / V1
request.
This is due to a change introduced with #9071, where the login UI
version can be specified using the app configuration.
The configuration set to V1 was not considering if the header was sent.

# How the Problems Are Solved

- Check presence of `x-zitadel-login-client` before the configuration.
Use later only if no header is set.

# Additional Changes

None

# Additional Context

- closes #9311
- needs back ports to 2.67.x, 2.68.x and 2.69.x

(cherry picked from commit e7a73eb6b1)
2025-02-13 17:11:56 +01:00
Livio Spring
c477049d03 fix(login): fix migration to allow login by email again (#9315)
# Which Problems Are Solved

The login by email was not possible anymore. This was due to a newly
generated user projection because of #9255 .
Internal logs showed that the computed lower case column for verified
email was missing.

# How the Problems Are Solved

Update name of setup step 25 to rerun the step, since the underlying sql
changed.

# Additional Changes

None

# Additional Context

- relates to #9255

(cherry picked from commit b63c5fdb17)
2025-02-06 11:21:21 +01:00
Lars
35e8a2bcf9 fix: ensure metadata is projected for scim tests to ensure stable tests (#9305)
# Which Problems Are Solved
- SCIM tests are flaky due to metadata being set by the tests while
shortly after being read by the application, resulting in a race
condition

# How the Problems Are Solved
- whenever metadata is set, the projection is awaited

# Additional Context
Part of #8140

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
(cherry picked from commit 4dc7a58a25)
2025-02-06 11:21:20 +01:00
Lars
703969a5e4 fix: relax parsing of SCIM user 'active' flag to improve compatibility (#9296)
# Which Problems Are Solved
- Microsoft Entra invokes the user patch endpoint with `"active":
"True"` / `"active": "False"` when patching a user. This is a well-known
bug in MS Entra (see
[here](https://learn.microsoft.com/en-us/entra/identity/app-provisioning/application-provisioning-config-problem-scim-compatibility)),
but the bug fix has not landed yet and/or the feature flag does not
work.

# How the Problems Are Solved
- To ensure compatibility with MS Entra, the parsing of the the boolean
active flag of the scim user is relaxed and accepts strings in any
casing that resolve to `true` or `false` as well as raw boolean values.

# Additional Context
Part of https://github.com/zitadel/zitadel/issues/8140

(cherry picked from commit 361f7a2edc)
2025-02-06 11:21:20 +01:00
Emilien GUILMINEAU
b211e09bcd fix(setup): Fix query alias on 46-06 (#9298)
# Which Problems Are Solved

After updating to version 2.69.0, my zitadel instance refuse to start
with this error log :
```
time="2025-02-03T19:46:47Z" level=info msg="starting migration" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:66" name=46_init_permission_functions
time="2025-02-03T19:46:47Z" level=info msg="execute statement" caller="/home/runner/work/zitadel/zitadel/cmd/setup/46.go:29" file=01-role_permissions_view.sql migration=46_init_permission_functions
time="2025-02-03T19:46:47Z" level=info msg="execute statement" caller="/home/runner/work/zitadel/zitadel/cmd/setup/46.go:29" file=02-instance_orgs_view.sql migration=46_init_permission_functions
time="2025-02-03T19:46:47Z" level=info msg="execute statement" caller="/home/runner/work/zitadel/zitadel/cmd/setup/46.go:29" file=03-instance_members_view.sql migration=46_init_permission_functions
time="2025-02-03T19:46:47Z" level=info msg="execute statement" caller="/home/runner/work/zitadel/zitadel/cmd/setup/46.go:29" file=04-org_members_view.sql migration=46_init_permission_functions
time="2025-02-03T19:46:47Z" level=info msg="execute statement" caller="/home/runner/work/zitadel/zitadel/cmd/setup/46.go:29" file=05-project_members_view.sql migration=46_init_permission_functions
time="2025-02-03T19:46:47Z" level=info msg="execute statement" caller="/home/runner/work/zitadel/zitadel/cmd/setup/46.go:29" file=06-permitted_orgs_function.sql migration=46_init_permission_functions
time="2025-02-03T19:46:47Z" level=error msg="migration failed" caller="/home/runner/work/zitadel/zitadel/internal/migration/migration.go:68" error="46_init_permission_functions 06-permitted_orgs_function.sql: ERROR: subquery in FROM must have an alias (SQLSTATE 42601)" name=46_init_permission_functions
time="2025-02-03T19:46:47Z" level=fatal msg="migration failed" caller="/home/runner/work/zitadel/zitadel/cmd/setup/setup.go:274" error="46_init_permission_functions 06-permitted_orgs_function.sql: ERROR: subquery in FROM must have an alias (SQLSTATE 42601)" name=46_init_permission_functions
```

# How the Problems Are Solved

I used the original sql script on my database which gave me the same
error.
So i added an alias for the subquery and the error cas gone

# Additional Context

I was migrating from version 2.58.3

Closes https://github.com/zitadel/zitadel/issues/9300

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
(cherry picked from commit 857812bb9e)
2025-02-04 12:01:45 +01:00
Livio Spring
01bbcc1a48 fix(OTEL): reduce high cardinality in traces and metrics (#9286)
# Which Problems Are Solved

There were multiple issues in the OpenTelemetry (OTEL) implementation
and usage for tracing and metrics, which lead to high cardinality and
potential memory leaks:
- wrongly initiated tracing interceptors
- high cardinality in traces:
  - HTTP/1.1 endpoints containing host names
- HTTP/1.1 endpoints containing object IDs like userID (e.g.
`/management/v1/users/2352839823/`)
- high amount of traces from internal processes (spooler)
- high cardinality in metrics endpoint:
  - GRPC entries containing host names
  - notification metrics containing instanceIDs and error messages

# How the Problems Are Solved

- Properly initialize the interceptors once and update them to use the
grpc stats handler (unary interceptors were deprecated).
- Remove host names from HTTP/1.1 span names and use path as default.
- Set / overwrite the uri for spans on the grpc-gateway with the uri
pattern (`/management/v1/users/{user_id}`). This is used for spans in
traces and metric entries.
- Created a new sampler which will only sample spans in the following
cases:
  - remote was already sampled
- remote was not sampled, root span is of kind `Server` and based on
fraction set in the runtime configuration
- This will prevent having a lot of spans from the spooler back ground
jobs if they were not started by a client call querying an object (e.g.
UserByID).
- Filter out host names and alike from OTEL generated metrics (using a
`view`).
- Removed instance and error messages from notification metrics.

# Additional Changes

Fixed the middleware handling for serving Console. Telemetry and
instance selection are only used for the environment.json, but not on
statically served files.

# Additional Context

- closes #8096
- relates to #9074
- back ports to at least 2.66.x, 2.67.x and 2.68.x

(cherry picked from commit 990e1982c7)
2025-02-04 12:01:45 +01:00
Livio Spring
92e2ba0ea8 Merge branch 'main' into next 2025-02-03 08:38:39 +01:00
Livio Spring
04b9e9b144 fix(console): add posthog to CSP if configured (#9284)
# Which Problems Are Solved

PostHog scripts are currently blocked by content security policy (CSP).

# How the Problems Are Solved

Add `https://*.i.posthog.com` to the CSP according to
https://posthog.com/docs/advanced/content-security-policy#enabling-the-toolbar
(they suggest  `https://*.posthog.com`)

# Additional Changes

None

# Additional Context

relates to https://github.com/zitadel/zitadel/issues/9076
2025-02-03 08:08:01 +01:00
Lars
f65db52247 fix: scim create users dont send init emails (#9283)
# Which Problems Are Solved
- when a scim user is provisioned, a init email could be sent

# How the Problems Are Solved
- no init email should be sent => hard code false for the email init
param

# Additional Context

Related to https://github.com/zitadel/zitadel/issues/8140

Co-authored-by: Fabienne Bühler <fabienne@zitadel.com>
2025-01-31 09:36:18 +00:00
Lars
20cff9c70a fix: scim 2.0 patch ignore op casing (#9282)
# Which Problems Are Solved
- Some SCIM clients send "op" of a patch operation in PascalCase

# How the Problems Are Solved
- Well known "op" values of patch operations are matched
case-insensitive.

# Additional Context
Related to #8140
2025-01-31 09:15:39 +00:00
Lars
563f74640e fix: scim v2 endpoints enforce user resource owner (#9273)
# Which Problems Are Solved
- If a SCIM endpoint is called with an orgID in the URL that is not the
resource owner, no error is returned, and the action is executed.

# How the Problems Are Solved
- The orgID provided in the SCIM URL path must match the resource owner
of the target user. Otherwise, an error will be returned.

# Additional Context

Part of https://github.com/zitadel/zitadel/issues/8140
2025-01-30 16:43:13 +01:00
Lars
60cfa6cb76 docs: scim v2 interface (#9246)
# Which Problems Are Solved
- Lack of documentation for the SCIM v2 interface

# How the Problems Are Solved
- Introduced a new documentation page detailing the SCIM v2 interface

# Additional Context
Part of #8140

---------

Co-authored-by: Fabienne Bühler <fabienne@zitadel.com>
2025-01-30 08:07:28 +00:00
David Skewis
4498f9c8f3 feat: Posthog integration (#9077)
# Which Problems Are Solved

- Adds a service in the console to enable Posthog integration based on
upon user environment variables

# How the Problems Are Solved

- A new service has been created in console for posthog
- This is only initiated based upon provided environment variables

# Additional Changes

N/A

# Additional Context

- Closes #[9076](https://github.com/zitadel/zitadel/issues/9076)
- Cannot be merged until this is completed
#[9070](https://github.com/zitadel/zitadel/issues/9070)
2025-01-30 07:57:51 +01:00
Lars
e15094cdea feat: add scim v2 service provider configuration endpoints (#9258)
# Which Problems Are Solved
* Adds support for the service provider configuration SCIM v2 endpoints

# How the Problems Are Solved
* Adds support for the service provider configuration SCIM v2 endpoints
  * `GET /scim/v2/{orgId}/ServiceProviderConfig`
  * `GET /scim/v2/{orgId}/ResourceTypes`
  * `GET /scim/v2/{orgId}/ResourceTypes/{name}`
  * `GET /scim/v2/{orgId}/Schemas`
  * `GET /scim/v2/{orgId}/Schemas/{id}`

# Additional Context
Part of #8140

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2025-01-29 18:11:12 +00:00
Tim Möhlmann
b6841251b1 feat(users/v2): return prompt information (#9255)
# Which Problems Are Solved

Add the ability to update the timestamp when MFA initialization was last
skipped.
Get User By ID now also returns the timestamps when MFA setup was last
skipped.

# How the Problems Are Solved

- Add a `HumanMFAInitSkipped` method to the `users/v2` API.
- MFA skipped was already projected in the `auth.users3` table. In this
PR the same column is added to the users projection. Event handling is
kept the same as in the `UserView`:

<details>


62804ca45f/internal/user/repository/view/model/user.go (L243-L377)

</details>

# Additional Changes

- none

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/9197
2025-01-29 15:12:31 +00:00
Lars
df8bac8a28 feat: bulk scim v2 endpoint (#9256)
# Which Problems Are Solved
* Adds support for the bulk SCIM v2 endpoint

# How the Problems Are Solved
* Adds support for the bulk SCIM v2 endpoint under `POST
/scim/v2/{orgID}/Bulk`

# Additional Context
Part of #8140

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2025-01-29 14:23:56 +00:00
Lars
accfb7525a fix: scim 2 filter: the username should be treated case-insensitive (#9257)
# Which Problems Are Solved
- when listing users via scim v2.0 filters applied to the username are
applied case-sensitive

# How the Problems Are Solved
- when a query filter is appleid on the username it is applied
case-insensitive

# Additional Context
Part of https://github.com/zitadel/zitadel/issues/8140
2025-01-29 15:22:22 +02:00
Silvan
b10428fb56 test(session): load tests for session api (#9212)
# Which Problems Are Solved

We currently are not able to benchmark the performance of the session
api

# How the Problems Are Solved

Load tests were added to
- use sessions in oidc tokens analog
https://zitadel.com/docs/guides/integrate/login-ui/oidc-standard

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/7847
2025-01-29 12:08:20 +00:00
Stefan Benz
679ab58fa1 docs: support docs for SAML session in Custom Login UI (#9144)
# Which Problems Are Solved

SAML session implemented, but no how-to comparable to the OIDC sessions
for custom login available.

# How the Problems Are Solved

Added documentation, which should be also comparable with the OIDC
session for ease-of-use.

# Additional Changes

Added generated SAML API docs.

# Additional Context

Closes #9088 
Follow-up issue #9267

---------

Co-authored-by: Fabienne Bühler <fabienne@zitadel.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
2025-01-29 11:29:48 +00:00
kkrime
5eeff97ffe feat(session/v2): user password lockout error response (#9233)
# Which Problems Are Solved

Adds `failed attempts` field to the grpc response when a user enters
wrong password when logging in

FYI:

this only covers the senario above; other senarios where this is not
applied are:
SetPasswordWithVerifyCode
setPassword
ChangPassword
setPasswordWithPermission

# How the Problems Are Solved 

Created new grpc message `CredentialsCheckError` -
`proto/zitadel/message.proto` to include `failed_attempts` field.

Had to create a new package -
`github.com/zitadel/zitadel/internal/command/errors` to resolve cycle
dependency between `github.com/zitadel/zitadel/internal/command` and
`github.com/zitadel/zitadel/internal/command`.

# Additional Changes

- none

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/9198

---------

Co-authored-by: Iraq Jaber <IraqJaber@gmail.com>
2025-01-29 10:29:00 +00:00
Lars
21f00c1e6b fix: scim use first email or phone if no primary is set (#9236)
# Which Problems Are Solved
- scim v2 only maps the primary phone/email to the zitadel user, this
does not work if no primary is set

# How the Problems Are Solved
- the first phone / email is mapped if no primary is available

# Additional Context
Part of #8140

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2025-01-29 09:18:00 +00:00
Stefan Benz
a59c6b9f84 fix: change usage from filepath to path (#9260)
# Which Problems Are Solved

Paths for setup steps are joined with "\" when binary is started under
Windows, which results in wrongly joined paths.

# How the Problems Are Solved

Replace the usage of "filepath" with "path" package, which does only
join with "/" and nothing OS specific.

# Additional Changes

None

# Additional Context

Closes #9227
2025-01-29 09:53:27 +01:00
Livio Spring
66a3c814db fix(notifications): cancel on missing channels and Twilio 4xx errors (#9254)
# Which Problems Are Solved

#9185 changed that if a notification channel was not present,
notification workers would no longer retry to send the notification and
would also cancel in case Twilio would return a 4xx error.
However, this would not affect the "legacy" mode.

# How the Problems Are Solved

- Handle `CancelError` in legacy notifier as not failed (event).

# Additional Changes

None

# Additional Context

- relates to #9185
- requires back port to 2.66.x and 2.67.x

(cherry picked from commit 3fc68e5d60)
2025-01-28 07:40:32 +01:00
Livio Spring
3fc68e5d60 fix(notifications): cancel on missing channels and Twilio 4xx errors (#9254)
# Which Problems Are Solved

#9185 changed that if a notification channel was not present,
notification workers would no longer retry to send the notification and
would also cancel in case Twilio would return a 4xx error.
However, this would not affect the "legacy" mode.

# How the Problems Are Solved

- Handle `CancelError` in legacy notifier as not failed (event).

# Additional Changes

None

# Additional Context

- relates to #9185 
- requires back port to 2.66.x and 2.67.x
2025-01-28 06:32:09 +00:00
Lars
30a54fc1eb fix: scim user query endpoint don't allow SortBy custom field (#9235)
# Which Problems Are Solved
- scim list users endpoint (`GET /scim/v2/{orgId}/Users`): handle
unsupported `SortBy` columns correctly

# How the Problems Are Solved
- throw an error if sorting by an unsupported column is requested

# Additional Context
Part of #8140
2025-01-27 17:30:27 +00:00
Lars
b19333726c fix: allow scim content type wildcards (#9245)
# Which Problems Are Solved
- requests to the scim interface with content type `*/*` are rejected

# How the Problems Are Solved
- `*/*` is accepted as content type

# Additional Context
Part of #8140
2025-01-27 16:10:30 +00:00
Lars
741434806a fix: unified scim metadata key casing (#9244)
# Which Problems Are Solved
- SCIM user metadata mapping keys have differing case styles.

# How the Problems Are Solved
- key casing style is unified to strict camelCase

# Additional Context
Part of #8140

Although this is technically a breaking change, it is considered
acceptable because the SCIM feature is still in the preview stage and
not fully implemented yet.

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2025-01-27 13:51:58 +00:00
Lars
189f9770c6 feat: patch user scim v2 endpoint (#9219)
# Which Problems Are Solved
* Adds support for the patch user SCIM v2 endpoint

# How the Problems Are Solved
* Adds support for the patch user SCIM v2 endpoint under `PATCH
/scim/v2/{orgID}/Users/{id}`

# Additional Context
Part of #8140
2025-01-27 13:36:07 +01:00
Tim Möhlmann
934faef717 fix(setup): split membership fields migration (#9230)
# Which Problems Are Solved

The membership fields migration timed out in certain cases. It also
tried to migrate instances which were already removed.

# How the Problems Are Solved

Revert the previous fix that combined the repeatable step for multiple
fill triggers. The membeship migration is now single-run as it might
take a lot of time. It is not worth making it repeatable. Instance IDs
of removed instances are skipped.

# Additional Changes

None

# Additional Context

Introduced in https://github.com/zitadel/zitadel/pull/9199

(cherry picked from commit ec5f18c168)
2025-01-27 06:41:16 +01:00
Livio Spring
1efeb20215 fix(oidc apps): correctly remove last additional origin, redirect uri and post logout redirect uri (#9209)
# Which Problems Are Solved

A customer reached out to support, that the (last) `additional origin`
could not be removed. While testing / implementation it was discovered,
that the same applied to `redirect_uris` and `post_logout_redirect_uris`

# How the Problems Are Solved

- Correctly set the corresponding array to empty in the event so it can
be differentiated to `null` / not set in case of no change.

# Additional Changes

Replaced `reflect.DeepEqual` with `slices.Equal`

# Additional Context

- Reported to support

(cherry picked from commit c9aa5db2a5)
2025-01-27 06:41:16 +01:00
Tim Möhlmann
ec5f18c168 fix(setup): split membership fields migration (#9230)
# Which Problems Are Solved

The membership fields migration timed out in certain cases. It also
tried to migrate instances which were already removed.

# How the Problems Are Solved

Revert the previous fix that combined the repeatable step for multiple
fill triggers. The membeship migration is now single-run as it might
take a lot of time. It is not worth making it repeatable. Instance IDs
of removed instances are skipped.

# Additional Changes

None

# Additional Context

Introduced in https://github.com/zitadel/zitadel/pull/9199
2025-01-24 11:24:35 +01:00
kkrime
73577885bf docs: small update to docs/docs/concepts/architecture/software.md (#9218)
# Which Problems Are Solved
small update to docs/docs/concepts/architecture/software.md
2025-01-23 13:12:49 +00:00
Zach Hirschtritt
e4bbfcccc8 fix: add aggregate type to subquery to utilize indexes (#9226)
# Which Problems Are Solved

The subquery of the notification requested and retry requested is
missing the aggregate_type filter that would allow it to utilize the
`es_projection` or `active_instances_events` on the eventstore.events2
table.

# How the Problems Are Solved

Add additional filter on subquery. Final query: 
```sql
SELECT <all the fields omitted> FROM eventstore.events2
WHERE
    instance_id = $1
    AND aggregate_type = $2
    AND event_type = $3
    AND created_at > $4
    AND aggregate_id NOT IN (
        SELECT aggregate_id
        FROM eventstore.events2
        WHERE
            aggregate_type = $5 <-- NB: previously missing
            AND event_type = ANY ($6)
            AND instance_id = $7
            AND created_at > $8
    )
ORDER BY "position", in_tx_order
LIMIT $9
FOR UPDATE SKIP LOCKED
```

# Additional Changes

# Additional Context

Co-authored-by: Livio Spring <livio.a@gmail.com>
2025-01-22 16:02:37 +00:00
Livio Spring
c9aa5db2a5 fix(oidc apps): correctly remove last additional origin, redirect uri and post logout redirect uri (#9209)
# Which Problems Are Solved

A customer reached out to support, that the (last) `additional origin`
could not be removed. While testing / implementation it was discovered,
that the same applied to `redirect_uris` and `post_logout_redirect_uris`

# How the Problems Are Solved

- Correctly set the corresponding array to empty in the event so it can
be differentiated to `null` / not set in case of no change.

# Additional Changes

Replaced `reflect.DeepEqual` with `slices.Equal`

# Additional Context

- Reported to support
2025-01-22 07:37:37 +00:00
Lars
1915d35605 feat: list users scim v2 endpoint (#9187)
# Which Problems Are Solved
- Adds support for the list users SCIM v2 endpoint

# How the Problems Are Solved
- Adds support for the list users SCIM v2 endpoints under `GET
/scim/v2/{orgID}/Users` and `POST /scim/v2/{orgID}/Users/.search`

# Additional Changes
- adds a new function `SearchUserMetadataForUsers` to the query layer to
query a metadata keyset for given user ids
- adds a new function `NewUserMetadataExistsQuery` to the query layer to
query a given metadata key value pair exists
- adds a new function `CountUsers` to the query layer to count users
without reading any rows
- handle `ErrorAlreadyExists` as scim errors `uniqueness`
- adds `NumberLessOrEqual` and `NumberGreaterOrEqual` query comparison
methods
- adds `BytesQuery` with `BytesEquals` and `BytesNotEquals` query
comparison methods

# Additional Context
Part of #8140
Supported fields for scim filters:
* `meta.created`
* `meta.lastModified`
* `id`
* `username`
* `name.familyName`
* `name.givenName`
* `emails` and `emails.value`
* `active` only eq and ne
* `externalId` only eq and ne
2025-01-21 13:31:54 +01:00
kkrime
926e7169b2 docs: small update to docs/docs/concepts/features/selfservice.md (#9214)
# Which Problems Are Solved
Small update to docs/docs/concepts/features/selfservice.md to fix issue
in grammar

Co-authored-by: Iraq Jaber <IraqJaber@gmail.com>
2025-01-21 09:05:13 +01:00
Livio Spring
a53fc5f5fa Merge branch 'main' into next
# Conflicts:
#	cmd/setup/config.go
#	cmd/setup/setup.go
#	internal/auth/repository/eventsourcing/eventstore/auth_request.go
2025-01-20 14:22:24 +01:00
Tim Möhlmann
94cbf97534 fix(permissions_v2): add membership fields migration (#9199)
# Which Problems Are Solved

Memberships did not have a fields table fill migration.

# How the Problems Are Solved

Add filling of membership fields to the repeatable steps.

# Additional Changes

- Use the same repeatable step for multiple fill fields handlers.
- Fix an error for PostgreSQL 15 where a subquery in a `FROM` clause
needs an alias ing the `permitted_orgs` function.

# Additional Context

- Part of https://github.com/zitadel/zitadel/issues/9188
- Introduced in https://github.com/zitadel/zitadel/pull/9152
2025-01-17 16:16:26 +01:00
Silvan
0719d9d939 fix(eventstore): correct sql push function (#9201)
# Which Problems Are Solved

https://github.com/zitadel/zitadel/pull/9186 introduced the new `push`
sql function for cockroachdb. The function used the wrong database
function to generate the position of the event and would therefore
insert events at a position before events created with an old Zitadel
version.

# How the Problems Are Solved

Instead of `EXTRACT(EPOCH FROM NOW())`, `cluster_logical_timestamp()` is
used to calculate the position of an event.

# Additional Context

- Introduced in https://github.com/zitadel/zitadel/pull/9186
- Affected versions:
https://github.com/zitadel/zitadel/releases/tag/v2.67.3
2025-01-17 15:35:04 +01:00
Silvan
9532c9bea5 fix(eventstore): correct sql push function (#9201)
# Which Problems Are Solved

https://github.com/zitadel/zitadel/pull/9186 introduced the new `push`
sql function for cockroachdb. The function used the wrong database
function to generate the position of the event and would therefore
insert events at a position before events created with an old Zitadel
version.

# How the Problems Are Solved

Instead of `EXTRACT(EPOCH FROM NOW())`, `cluster_logical_timestamp()` is
used to calculate the position of an event.

# Additional Context

- Introduced in https://github.com/zitadel/zitadel/pull/9186
- Affected versions:
https://github.com/zitadel/zitadel/releases/tag/v2.67.3
2025-01-17 15:32:05 +01:00