# Which Problems Are Solved
#10415 added the possibility to filter users based on metadata. To
prevent duplicate results an sql `DISTINCT` was added. This resulted in
issues if the list was sorted on string columns like `username` or
`displayname`, since they are sorted using `lower`. Using `DISTINCT`
requires the `order by` column to be part of the `SELECT` statement.
# How the Problems Are Solved
Added the order by column to the statement.
# Additional Changes
None
# Additional Context
- relates to #10415
- backport to v4.x
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
(cherry picked from commit 2c0ee0008f)
# Which Problems Are Solved
Depending on the metadata values (already existing), the newly created
index (#10415) cannot be created or error in the future.
# How the Problems Are Solved
- Create the index using `sha256` and change the query to use sha256 as
well when comparing bytes values such as user_metadata.
- Added a setup step to cleanup potentially created index on
`projections.user_metadata5`
# Additional Changes
None
# Additional Context
- relates to #10415
- requires backport to v4.x
(cherry picked from commit 57e8033b6e)
# Which Problems Are Solved
Eventual consistency issues.
# How the Problems Are Solved
Correctly handle timeouts and change queries to domains instead of using
the organization name.
# Additional Changes
None
# Additional Context
None
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit b0642a5898)
# Which Problems Are Solved
The current service ping reports can run into body size limit errors and
there's no way of knowing how big the current size is.
# How the Problems Are Solved
Log the current size to have at least some insights and possibly change
bulk size.
# Additional Changes
None
# Additional Context
- noticed internally
- backport to v4.x
(cherry picked from commit bc471b4f78)
# Which Problems Are Solved
I noticed that a failure in the projections handlers `reduce` function
(e.g. creating the statement or checking preconditions for the
statement) would not update the `failed_events2` table.
This was due to a wrong error handling, where as long as the
`maxFailureCount` was not reached, the error was returned after updating
the `failed_events2` table, which causes the transaction to be rolled
back and thus losing the update.
# How the Problems Are Solved
Wrap the error into an `executionError`, so the transaction is not
rolled back.
# Additional Changes
none
# Additional Context
- noticed internally
- requires backport to v3.x and v4.x
(cherry picked from commit ee92560f32)
# Which Problems Are Solved
Cached object may have a different schema between Zitadel versions.
# How the Problems Are Solved
Use the curent build version in DB based cache connectors PostgreSQL and
Redis.
# Additional Changes
- Cleanup the ZitadelVersion field from the authz Instance
- Solve potential race condition on global variables in build package.
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/10648
- Obsoletes https://github.com/zitadel/zitadel/pull/10646
- Needs to be back-ported to v4 over
https://github.com/zitadel/zitadel/pull/10645
(cherry picked from commit f6f37d3a31)
This PR fixes a bug where projections could skip events if they were
written within the same microsecond, which can occur during high load on
different transactions.
## Problem
The event query ordering was not fully deterministic. Events created at
the exact same time (same `position`) and in the same transaction
(`in_tx_order`) were not guaranteed to be returned in the same order on
subsequent queries. This could lead to some events being skipped by the
projection logic.
## Solution
To solve this, the `ORDER BY` clause for event queries has been extended
to include `instance_id`, `aggregate_type`, and `aggregate_id`. This
ensures a stable and deterministic ordering for all events, even if they
share the same timestamp.
## Additionally changes:
* Replaced a manual slice search with the more idiomatic
`slices.Contains` to skip already projected instances.
* Changed the handling of already locked projections to log a debug
message and skip execution instead of returning an error.
* Ensures the database transaction is explicitly committed.
(cherry picked from commit 25ab6b2397)
This PR overhauls our event projection system to make it more robust and
prevent skipped events under high load. The core change replaces our
custom, transaction-based locking with standard PostgreSQL advisory
locks. We also introduce a worker pool to manage concurrency and prevent
database connection exhaustion.
### Key Changes
* **Advisory Locks for Projections:** Replaces exclusive row locks and
inspection of `pg_stat_activity` with PostgreSQL advisory locks for
managing projection state. This is a more reliable and standard approach
to distributed locking.
* **Simplified Await Logic:** Removes the complex logic for awaiting
open transactions, simplifying it to a more straightforward time-based
filtering of events.
* **Projection Worker Pool:** Implements a worker pool to limit
concurrent projection triggers, preventing connection exhaustion and
improving stability under load. A new `MaxParallelTriggers`
configuration option is introduced.
### Problem Solved
Under high throughput, a race condition could cause projections to miss
events from the eventstore. This led to inconsistent data in projection
tables (e.g., a user grant might be missing). This PR fixes the
underlying locking and concurrency issues to ensure all events are
processed reliably.
### How it Works
1. **Event Writing:** When writing events, a *shared* advisory lock is
taken. This signals that a write is in progress.
2. **Event Handling (Projections):**
* A projection worker attempts to acquire an *exclusive* advisory lock
for that specific projection. If the lock is already held, it means
another worker is on the job, so the current one backs off.
* Once the lock is acquired, the worker briefly acquires and releases
the same *shared* lock used by event writers. This acts as a barrier,
ensuring it waits for any in-flight writes to complete.
* Finally, it processes all events that occurred before its transaction
began.
### Additional Information
* ZITADEL no longer modifies the `application_name` PostgreSQL variable
during event writes.
* The lock on the `current_states` table is now `FOR NO KEY UPDATE`.
* Fixes https://github.com/zitadel/zitadel/issues/8509
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
(cherry picked from commit 0575f67e94)
# Which Problems Are Solved
During the implementation of #10687, it was noticed that the import
endpoint might provide unnecessary error details.
# How the Problems Are Solved
Remove the underlying (parent) error from the error message.
# Additional Changes
none
# Additional Context
relates to #10687
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
(cherry picked from commit 25d921b20c)
Closes#10498
The registration form's legal checkboxes had incorrect validation logic
that prevented users from completing registration when only one legal
document (ToS or Privacy Policy) was configured, or when no legal
documents were required.
additionally removes a duplicate description for "or use Identity
Provider"
# Which Problems Are Solved
Having only partial legal documents was blocking users to register. The
logic now conditionally renders checkboxes and checks if all provided
documents are accepted.
# How the Problems Are Solved
- Fixed checkbox validation: Now properly validates based on which legal
documents are actually available
- acceptance logic: Only requires acceptance of checkboxes that are
shown
- No legal docs support: Users can proceed when no legal documents are
configured
- Proper state management: Fixed checkbox state tracking and mixed-up
test IDs
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
(cherry picked from commit b9b9baf67f)
# Which Problems Are Solved
Comparing the v3 and v4 deployments we noticed an increase in memory
usage. A first analysis revealed that it might be related to the
(multiple) initialization of the `i18n.Translator`, partially related
# How the Problems Are Solved
Initialize the tranlator once (apart from the translator interceptor,
which uses context / request specific information) and pass it to all
necessary middleware.
# Additional Changes
Removed unnecessary error return parameter from the translator
initialization.
# Additional Context
- noticed internally
- backport to v4.x
(cherry picked from commit a0c3ccecf7)
# Which Problems Are Solved
Using the service ping, we want to have some additional insights to how
zitadel is configured. The current resource count report contains
already some amount of configured policies, such as the login_policy.
But we do not know if for example MFA is enforced.
# How the Problems Are Solved
- Added the following counts to the report:
- service users per organization
- MFA enforcements (though login policy)
- Notification policies with password change option enabled
- SCIM provisioned users (using user metadata)
- Since all of the above are conditional based on at least a column
inside a projection, a new `migration.CountTriggerConditional` has been
added, where a condition (column values) and an option to track updates
on that column should be considered for the count.
- For this to be possible, the following changes had to be made to the
existing sql resources:
- the `resource_name` has been added to unique constraint on the
`projection.resource_counts` table
- triggers have been added / changed to individually track `INSERT`,
`UPDATE`(s) and `DELETE` and be able to handle conditions
- an optional argument has been added to the
`projections.count_resource()` function to allow providing the
information to `UP` or `DOWN` count the resource on an update.
# Additional Changes
None
# Additional Context
- partially solves #10244 (reporting audit log retention limit will be
handled in #10245 directly)
- backport to v4.x
(cherry picked from commit 2dbe21fb30)
# Which Problems Are Solved
HTTP Request to HTTP providers for Email or SMS are not signed.
# How the Problems Are Solved
Add a Signing Key to the HTTP Provider resources, which is then used to
generate a header to sign the payload.
# Additional Changes
Additional tests for query side of the SMTP provider.
# Additional Context
Closes#10067
---------
Co-authored-by: Marco A. <marco@zitadel.com>
(cherry picked from commit 8909b9a2a6)
# Which Problems Are Solved
This PR adds functionality to propagate request headers in actions v2.
# How the Problems Are Solved
The new functionality is added to the`ExecutionHandler` interceptors,
where the incoming request headers (from a list of allowed headers to be
forwarded) are set in the payload of the request before calling the
target.
# Additional Changes
This PR also contains minor fixes to the Actions V2 example docs.
# Additional Context
- Closes#9941
---------
Co-authored-by: Marco A. <marco@zitadel.com>
(cherry picked from commit 51e12e224d)
# Which Problems Are Solved
Some users have reported the need of retrieving users given a metadata
key, metadata value or both. This change introduces metadata search
filter on the `ListUsers()` endpoint to allow Zitadel users to search
for user records by metadata.
The changes affect only v2 APIs.
# How the Problems Are Solved
- Add new search filter to `ListUserRequest`: `MetaKey` and `MetaValue`
- Add SQL indices on metadata key and metadata value
- Update query to left join `user_metadata` table
# Additional Context
- Closes#9053
- Depends on https://github.com/zitadel/zitadel/pull/10567
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
(cherry picked from commit 8df402fb4f)
# Which Problems Are Solved
When a project is created by a user with only the `PROJECT_CREATOR`
role, they can no longer view/manage the created project. Although the
project is created, the user sees the following error: `No matching
permissions found (AUTH-3jknH)`. This is due to the
[removal](https://github.com/zitadel/zitadel/pull/9317) of
auto-assignment of the `PROJECT_OWNER` role when a project is newly
created.
# How the Problems Are Solved
By introducing optional fields in the CreateProject API to include a
list of users and a list of project member roles to be assigned to the
users. When there are no roles mentioned, the `PROJECT_OWNER` role is
assigned by default to all the users mentioned in the list.
# Additional Changes
N/A
# Additional Context
- Closes#10561
- Closes#10592
- Should be backported as this issue is not specific to v4
---------
Co-authored-by: conblem <mail@conblem.me>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Flakiness and conflicts in value from gofakeit.
# How the Problems Are Solved
Move Gofakeit calls to the integration package, to guarantee proper
usage and values for integration testing.
# Additional Changes
None
# Additional Context
None
(cherry picked from commit 492f1826ee)
# Which Problems Are Solved
Invalid id_tokens used as `id_token_hint` on the authorization endpoints
currently return an error, resp. get display on the endpoint itself.
# How the Problems Are Solved
Ignore invalid id_token_hint errors and just log them.
# Additional Changes
None
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10673
- backport to v4.x
(cherry picked from commit e158f9447e)
# Which Problems Are Solved
Flakiness in integration tests for organization v2beta service.
# How the Problems Are Solved
Fix eventual consistent handling of integration tests.
# Additional Changes
None
# Additional Context
None
---------
Co-authored-by: Marco A. <marco@zitadel.com>
(cherry picked from commit 75774eb64c)
# Which Problems Are Solved
It was noticed that on actions v2 when subscribing to events, the
webhook would always receive an empty `event_payload`:
```
{
"aggregateID": "336494809936035843",
"aggregateType": "user",
"resourceOwner": "336392597046099971",
"instanceID": "336392597046034435",
"version": "v2",
"sequence": 1,
"event_type": "user.human.added",
"created_at": "2025-09-05T08:55:36.156333Z",
"userID": "336392597046755331",
"event_payload":
{}
}
```
The problem was due to using `json.Marshal` on the `Event` interface,
where the underlying `BaseEvent` prevents the data to be marshalled:
131f70db34/internal/eventstore/event_base.go (L38)
# How the Problems Are Solved
The `Event`s `Unmarshal` function is used with a `json.RawMessage`.
# Additional Changes
none
# Additional Context
- backport for v4.x
- relates to https://github.com/zitadel/zitadel/pull/10651
- relates to https://github.com/zitadel/zitadel/pull/10564
# Which Problems Are Solved
We noticed a rapid growth of Redis memory usage. Instance By host did
not set the zitadel version, so instance entries got set on every
request again.
# How the Problems Are Solved
Set the version
# Additional Changes
- none
# Additional Context
- internal incident
# Which Problems Are Solved
Unnecessary default permission check in creating an authorization fails
even if the functionality was called internally.
# How the Problems Are Solved
Move permission check to the proper implementation, so that necessary
permission checks are provided by the responsible API.
# Additional Changes
None
# Additional Context
Closes#10624
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit bdefd9147f)
# Which Problems Are Solved
I noticed some outdated / misleading logs when starting zitadel:
- The `init-projections` were no longer in beta for a long time.
- The LRU auth request cache is disabled by default, which results in
the following message, which has caused confusion by customers:
```level=info msg="auth request cache disabled" error="must provide a positive size"```
# How the Problems Are Solved
- Removed the beta info
- Disable cache initialization if possible
# Additional Changes
None
# Additional Context
- noticed internally
- backport to v4.x
(cherry picked from commit a1ad87387d)
# Which Problems Are Solved
The
[otelriver](https://github.com/riverqueue/rivercontrib/tree/master/otelriver)
package uses default otel histogram buckets that are designed for
millisecond measurements. OTEL docs also suggest standardizing on using
seconds as the measurement unit. However, the default buckets from
opentelemetry-go are more or less useless when used with seconds as the
smallest measurement is 5 seconds and the largest is nearly 3 hours.
Example:
```
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="0"} 0
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="5"} 1144
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="10"} 1144
<...more buckets here...>
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="7500"} 1144
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="10000"} 1144
river_work_duration_histogram_seconds_bucket{attempt="1",kind="notification_request",otel_scope_name="github.com/riverqueue/rivercontrib/otelriver",otel_scope_version="",priority="1",queue="notification",status="ok",tag="[]",le="+Inf"} 1144
```
# How the Problems Are Solved
Change the default unit to "ms" from "s" as supported by the middleware
API:
https://riverqueue.com/docs/open-telemetry#list-of-middleware-options
# Additional Changes
None
# Additional Context
None
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
(cherry picked from commit fcdc598320)
# Which Problems Are Solved
The TestServer_Limits_AuditLogRetention is too reliant on time
constraints when checking that a limit is correctly applied. IN case it
takes to long to do all the preparation, there won't be any events to
read and the test will fail.
# How the Problems Are Solved
Don't require any events to be returned.
# Additional Changes
None
# Additional Context
- Noted a lot of pipeline to fail on this step.
- requires backport to at least v4.x
(cherry picked from commit 8574d6fbab)
# Which Problems Are Solved
The event execution system currently uses a projection handler that
subscribes to and processes all events for all instances. This creates a
high static cost because the system over-fetches event data, handling
many events that are not needed by most instances. This inefficiency is
also reflected in high "rows returned" metrics in the database.
# How the Problems Are Solved
Eliminate the use of a project handler. Instead, events for which
"execution targets" are defined, are directly pushed to the queue by the
eventstore. A Router is populated in the Instance object in the authz
middleware.
- By joining the execution targets to the instance, no additional
queries are needed anymore.
- As part of the instance object, execution targets are now cached as
well.
- Events are queued within the same transaction, giving transactional
guarantees on delivery.
- Uses the "insert many fast` variant of River. Multiple jobs are queued
in a single round-trip to the database.
- Fix compatibility with PostgreSQL 15
# Additional Changes
- The signing key was stored as plain-text in the river job payload in
the DB. This violated our [Secrets
Storage](https://zitadel.com/docs/concepts/architecture/secrets#secrets-storage)
principle. This change removed the field and only uses the encrypted
version of the signing key.
- Fixed the target ordering from descending to ascending.
- Some minor linter warnings on the use of `io.WriteString()`.
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/9249
- Closes https://github.com/zitadel/zitadel/issues/10553
- Closes https://github.com/zitadel/zitadel/issues/9832
- Closes https://github.com/zitadel/zitadel/issues/10372
- Closes https://github.com/zitadel/zitadel/issues/10492
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
(cherry picked from commit a9ebc06c77)
# Which Problems Are Solved
Integration test failed sometimes with `organization already
exists`-errors.
# How the Problems Are Solved
Use a consistent function to generate name used for organization
creation.
# Additional Changes
Correct a eventual consistent test for username around organization
domain changes with eventual consistent loop.
# Additional Context
None
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit 5721b63bcb)
# Which Problems Are Solved
- Most inputs have hardcoded label
# How the Problems Are Solved
- add usage of i18n library for every label
- add labels to i18n translation files
# Additional Changes
- fixed key used in `device-code-form.tsx` by submit button
- `v2-default.json` was update and contains all values from login app
not only newly added key for labels.
# Additional Context
N.A
---------
Co-authored-by: David Skewis <david@zitadel.com>
Co-authored-by: Max Peintner <max@caos.ch>
(cherry picked from commit 832e78f9bc)
# Which Problems Are Solved
- Inconsistencies in the terminology used for "身份认证提供商" (identity
provider) and "身份认证提供者" (identity supplier) in the Chinese translation
files could lead to confusion among users.
- Missing translations for terminology related to identity providers
could hinder user experience and understanding.
# How the Problems Are Solved
- Unified the terms "身份认证提供商" and "身份认证提供者" to consistently use
"身份认证提供者" across all Chinese translation files.
- Added necessary translations to ensure that all relevant terms related
to identity providers are accurately represented in the Chinese
localization.
# Additional Changes
- Improved overall readability and clarity in the Chinese translations
by ensuring consistent terminology for identity-related phrases
throughout the application.
- Complete the missing translations.
# Additional Context
If I have missed any translations, please point them out, and I would be
happy to complete them.
---------
Co-authored-by: Florian Forster <florian@zitadel.com>
(cherry picked from commit 6d0b7ed2aa)
# Which Problems Are Solved
Authorizations (aka user grants) could not be managed correctly if they
were created on a project grant, which itself was based on a project
granted to the own organization. The error persisted if the
corresponding (potentially unintended) project grant was removed again.
# How the Problems Are Solved
Fixed checks for managing user grants: Roles from projects and project
grants get handled individually to ensure cases like project grants on
the own organization.
# Additional Changes
Additional tests for the 3 failing scenarios.
# Additional Context
Closes#10556
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit 8e60cce20d)
# Which Problems Are Solved
When a SAML IDP is created, the signing algorithm defaults to
`RSA-SHA1`.
This PR adds the functionality to configure the signing algorithm while
creating or updating a SAML IDP. When nothing is specified, `RSA-SHA1`
is the default.
Available options:
* RSA_SHA1
* RSA_SHA256
* RSA_SHA512
# How the Problems Are Solved
By introducing a new optional config to specify the Signing Algorithm.
# Additional Changes
N/A
# Additional Context
- Closes#9842
An existing bug in the UpdateSAMLProvider API will be fixed as a
followup in a different
[PR](https://github.com/zitadel/zitadel/pull/10557).
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
(cherry picked from commit 255d42da65)
…urable
# Which Problems Are Solved
BaseURI defined in environment variables or configuration files was
ignored for Login v2 feature flag.
# How the Problems Are Solved
Define BaseURI as string so that the environment variables and
configuration files can be parsed into it.
# Additional Changes
None
# Additional Context
Closes#10405
(cherry picked from commit 2a78fdfe1f)
# Which Problems Are Solved
Emails are still send only with URLs to login v1.
# How the Problems Are Solved
Add configuration for URLs as URL templates, so that links can point at
Login v2.
# Additional Changes
None
# Additional Context
Closes#10236
---------
Co-authored-by: Marco A. <marco@zitadel.com>
(cherry picked from commit 0a14c01412)
# Which Problems Are Solved
This change makes it easier to delete the integration database
# How the Problems Are Solved
Gives the integration database a volume you can address via name
`docker volume rm config_zitadel_integration_db`
(cherry picked from commit 2718d345b8)
# Which Problems Are Solved
We noticed that the startup for v4 was way slower than v3. A query
without an instanceID filter could be traced back to the systemID query
of the service ping.
# How the Problems Are Solved
A an empty instanceID to the query to ensure it used an appropriate
index.
# Additional Changes
None
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/10390
- backport to v4.x
(cherry picked from commit 9621d357c0)
# Which Problems Are Solved
Project Admins would get permission errors when trying to add project
roles
# How the Problems Are Solved
Fixed wrong parameters were being passed into the permission check
- Closes https://github.com/zitadel/zitadel/issues/10505
(cherry picked from commit 24a7d3ceb1)
# Which Problems Are Solved
EntraID userinfo gets incorrectly unmarshalled again in the
`RetrieveIdentityProviderIntent` endpoint.
# How the Problems Are Solved
Correctly use the already available information and not try to marshall
it into a `RawInformation` struct again.
# Additional Changes
None
# Additional Context
Closes https://github.com/zitadel/typescript/issues/578
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit 93ea30ba2e)
# Which Problems Are Solved
User enumeration was possible on the select account page by passing any userID as part of the form POST. Existing users could be selected even if they never authenticated on the same user agent (browser).
# How the Problems Are Solved
A check for an existing session on the same user agent was added to the select user function, resp. only required for the account selection page, since in other cases there doesn't have to be an existing session and the user agent integrity is already checked.
# Additional Changes
None
# Additional Context
None
(cherry picked from commit 7abe759c95)
# Which Problems Are Solved
While investigating a support ticket, it was discovered that some
queries using equals or not equals without case matching were not
correctly escaping the value to compare. If a value contained a
backslash (`\`) the row would not match.
# How the Problems Are Solved
- Fixed the escaping for backslash for `like` operations.
- Changed equals and not equals comparison without case matching to `=`
instead of `like`.
# Additional Changes
None
# Additional Context
- related to a support request
- requires backport to v.3 and v4.x
(cherry picked from commit 6c8d027e72)
The `/introspect` endpoint showed poor performance during v4 load
testing due to an inefficient database query in
`internal/query/introspection_client_by_id.sql`. This PR optimizes the
query structure to significantly improve performance.
## Query Optimizations
**UNION → UNION ALL**: Changed expensive `UNION` to `UNION ALL` since
`client_id` is unique across both API and OIDC config tables,
eliminating unnecessary deduplication overhead (30-50% improvement
expected).
**Simplified Keys CTE**: Optimized the keys lookup logic by using
`$2::text as client_id` instead of `identifier as client_id` with `group
by`, and added explicit `$3 = true` condition to the LEFT JOIN for
better query planning.
**Enhanced Readability**: Added consistent table aliases (c, a, p, o, k)
for better maintainability.
## Benefits
- **Zero-downtime deployment**: Uses existing database indexes, no
schema changes required
- **Secondary performance gains**: Other similar queries
(`oidc_client_by_id.sql`, `userinfo_client_by_id.sql`) will also benefit
from the optimizations
- **Minimal code changes**: Only 13 lines added, 9 lines removed in the
SQL query
- **Backward compatible**: Same result set and API behavior
The optimized query maintains the same functionality while providing
significant performance improvements for the introspection endpoint
under high concurrent load.
Fixes#10389.
<!-- START COPILOT CODING AGENT TIPS -->
---
💬 Share your feedback on Copilot coding agent for the chance to win a
$200 gift card! Click
[here](https://survey.alchemer.com/s3/8343779/Copilot-Coding-agent) to
start the survey.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: muhlemmer <5411563+muhlemmer@users.noreply.github.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
(cherry picked from commit a28950661c)
# Which Problems Are Solved
Some events that are now unused are clogging the event queue from time
to time.
# How the Problems Are Solved
Remove the events described in #10458
# Additional Changes
- Updated `stringer` and `enumer` in Makefile target `core_generate_all`
to resolve generated files compilation issues
# Notes
It looks like there are a lot of changes, but most of it is fixing
translation files. I suggest doing a review per-commit
# Additional Context
- Closes#10458
- Depends on https://github.com/zitadel/zitadel/pull/10513
(cherry picked from commit e8a9cd6964)
# Which Problems Are Solved
`ListAppKeys()` does not work properly, in that it does not return any
app keys.
# How the Problems Are Solved
The issue stems from a mistake SQL query not joining the
`projections.authn_keys2` table to `projections.projects4` instead of
joining to `projections.apps7`
# Additional Changes
`ListAppKeys()` returns the app key IDs in order of their creation
- Closes https://github.com/zitadel/zitadel/issues/10420
- backport to v4.x
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit a637ae5aa5)
## Which problems are solved
The execution of statements of projections did not have the context
present.
## How the problems were solved
Pass the context to the execute function
## Additional info
This change is required to use the repositories of the relational tables
in projections.
(cherry picked from commit 20e7807ee5)
Removes manual schema and application name setup via raw SQL and
switches to using River’s built-in schema configuration.
# Which Problems Are Solved
River provides a configuration flag to set the schema of the queue.
Zitadel sets the schema through database statements which is not needed
anymore.
# How the Problems Are Solved
Set the schema in the river configuration and removed old code
(cherry picked from commit b5f97d64b0)
# Which Problems Are Solved
When a user with an `ORG_PROJECT_CREATOR` role tries to create a
project, the request fails with `No matching permissions found
(AUTH-AWfge)` error. This is because `project.write` was set as the
required permission instead of `project.create` during project creation.
# How the Problems Are Solved
By setting the right required permission (`project.create`) while
creating new projects.
# Additional Changes
N/A
# Additional Context
- Closes#10399
(cherry picked from commit 0929c4d235)
# Which Problems Are Solved
Currently, the prometheus endpoint metrics contain otel specific labels
that increase the overall metric size to the point that the exemplar
implementation in the underlying prom exporter library throws an error,
see https://github.com/zitadel/zitadel/issues/10047. The MaxRuneSize for
metric refs in exemplars is 128 and many of metrics cross this because
of `otel_scope_name`.
# How the Problems Are Solved
This change drops those otel specific labels on the prometheus exporter:
`otel_scope_name` and `otel_scope_version`
Current metrics example:
```
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.53.0",le="0"} 0
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.53.0",le="5"} 100
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",otel_scope_name="go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp",otel_scope_version="0.53.0",le="10"} 100
...
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/ListIAMMemberRoles",otel_scope_name="",otel_scope_version="",return_code="200"} 3
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/ListIAMMembers",otel_scope_name="",otel_scope_version="",return_code="200"} 3
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/ListMilestones",otel_scope_name="",otel_scope_version="",return_code="200"} 1
```
New example:
```
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",le="10"} 8
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",le="25"} 8
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",le="50"} 9
http_server_duration_milliseconds_bucket{http_method="GET",http_status_code="200",le="75"} 9
...
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/GetSupportedLanguages",return_code="200"} 1
grpc_server_grpc_status_code_total{grpc_method="/zitadel.admin.v1.AdminService/ListMilestones",return_code="200"} 1
grpc_server_grpc_status_code_total{grpc_method="/zitadel.auth.v1.AuthService/GetMyLabelPolicy",return_code="200"} 3
```
# Additional Changes
None
# Additional Context
From my understanding, this change is fully spec compliant with
Prometheus and Otel:
*
https://opentelemetry.io/docs/specs/otel/compatibility/prometheus_and_openmetrics/#instrumentation-scope
However, these tags were originally added as optional labels to
disambiguate metrics. But I'm not sure we need to care about that right
now? My gut feeling is that exemplar support (the ability for traces to
reference metrics) would be a preferable tradeoff to this label
standard.
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
(cherry picked from commit 532932ef94)
