Improves compatibility of eventstore and related database components
with the new relational table package.
## Which problems are solved
1. **Incompatible Database Interfaces**: The existing eventstore was
tightly coupled to the database package, which is incompatible with the
new, more abstract relational table package in v3. This prevented the
new command-side logic from pushing events to the legacy eventstore.
2. **Missing Health Checks**: The database interfaces in the new package
lacked a Ping method, making it impossible to perform health checks on
database connections.
3. **Event Publishing Logic**: The command handling logic in domain
needed a way to collect and push events to the legacy eventstore after a
command was successfully executed.
## How the problems are solved
1. **`LegacyEventstore` Interface**:
* A new `LegacyEventstore` interface is introduced in the new
`database/eventstore` . This interface exposes a `PushWithNewClient`
method that accepts the new `database.QueryExecutor` interface,
decoupling the v3 domain from the legacy implementation.
* The `internal/eventstore.Eventstore` now implements this interface. A
wrapper, PushWithClient, is added to convert the old database client
types (`*sql.DB`, `*sql.Tx`) into the new `QueryExecutor` types before
calling `PushWithNewClient`.
2. **Database Interface Updates**:
* The `database.Pool` and `database.Client` interfaces in
`storage/eventstore` have been updated to include a Ping method,
allowing for consistent health checks across different database
dialects.
* The `postgres` and `sql` dialect implementations have been updated to
support this new method.
3. **Command and Invoker Refactoring**:
* The `Commander` interface in domain now includes an `Events()
[]legacy_es.Command` method. This allows commands to declare which
events they will generate.
* The `eventCollector` in the invoker logic has been redesigned. It now
ensures a database transaction is started before executing a command.
After successful execution, it calls the `Events()` method on the
command to collect the generated events and appends them to a list.
* The `eventStoreInvoker` then pushes all collected events to the legacy
eventstore using the new `LegacyEventstore` interface, ensuring that
events are only pushed if the entire command (and any sub-commands)
executes successfully within the transaction.
4. **Testing**:
* New unit tests have been added for the invoker to verify that events
are correctly collected from single commands, batched commands, and
nested commands.
These changes create a clean bridge between the new v3 command-side
logic and the existing v1 eventstore, allowing for incremental adoption
of the new architecture while maintaining full functionality.
## Additional Information
closes https://github.com/zitadel/zitadel/issues/10442
* Abstracting the `OrganizationRepository` in `CommandOpts` to allow for
mock implementations during testing.
* Generating mocks for `OrganizationRepository` and
`OrganizationDomainRepository` using `mockgen`.
* Updating the `UpdateOrgCommand` tests to use the new mock
repositories, which simplifies the test setup and removes the need for
`dbmock`.
* Enhancing the `database.Change`, `database.Column`, and
`database.Condition` interfaces to implement `gomock.Matcher`, enabling
more effective use of gomock for matching database operations in tests.
* Introducing a `noopdb` package that provides a no-operation database
client for testing purposes.
# Which Problems Are Solved
This PR fixes the self-management of users for metadata and own removal
and improves the corresponding permission checks.
While looking into the problems, I also noticed that there's a bug in
the metadata mapping when using `api.metadata.push` in actions v1 and
that re-adding a previously existing key after its removal was not
possible.
# How the Problems Are Solved
- Added a parameter `allowSelfManagement` to checkPermissionOnUser to
not require a permission if a user is changing its own data.
- Updated use of `NewPermissionCheckUserWrite` including prevention of
self-management for metadata.
- Pass permission check to the command side (for metadata functions) to
allow it implicitly for login v1 and actions v1.
- Use of json.Marshal for the metadata mapping (as with
`AppendMetadata`)
- Check the metadata state when comparing the value.
# Additional Changes
- added a variadic `roles` parameter to the `CreateOrgMembership`
integration test helper function to allow defining specific roles.
# Additional Context
- noted internally while testing v4.1.x
- requires backport to v4.x
- closes https://github.com/zitadel/zitadel/issues/10470
- relates to https://github.com/zitadel/zitadel/pull/10426
…of project grant
# Which Problems Are Solved
On Management API the fields for `GrantedOrgId`, `GrantedOrgName` and
`GrantedOrgDomain` were only filled if it was a usergrant for a granted
project.
# How the Problems Are Solved
Correctly query the Organization of the User again to the Organization
the Project is granted to.
Then fill in the information about the Organization of the User in the
fields `GrantedOrgId`, `GrantedOrgName` and `GrantedOrgDomain`.
# Additional Changes
Additionally query the information about the Organization the Project is
granted to, to have it available for the Authorization v2beta API.
# Additional Context
Closes#10723
---------
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
# Which Problems Are Solved
Eventual consistency issues.
# How the Problems Are Solved
Correctly handle timeouts and change queries to domains instead of using
the organization name.
# Additional Changes
None
# Additional Context
None
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
The current service ping reports can run into body size limit errors and
there's no way of knowing how big the current size is.
# How the Problems Are Solved
Log the current size to have at least some insights and possibly change
bulk size.
# Additional Changes
None
# Additional Context
- noticed internally
- backport to v4.x
# Which Problems Are Solved
I noticed that a failure in the projections handlers `reduce` function
(e.g. creating the statement or checking preconditions for the
statement) would not update the `failed_events2` table.
This was due to a wrong error handling, where as long as the
`maxFailureCount` was not reached, the error was returned after updating
the `failed_events2` table, which causes the transaction to be rolled
back and thus losing the update.
# How the Problems Are Solved
Wrap the error into an `executionError`, so the transaction is not
rolled back.
# Additional Changes
none
# Additional Context
- noticed internally
- requires backport to v3.x and v4.x
# Which Problems Are Solved
Cached object may have a different schema between Zitadel versions.
# How the Problems Are Solved
Use the curent build version in DB based cache connectors PostgreSQL and
Redis.
# Additional Changes
- Cleanup the ZitadelVersion field from the authz Instance
- Solve potential race condition on global variables in build package.
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/10648
- Obsoletes https://github.com/zitadel/zitadel/pull/10646
- Needs to be back-ported to v4 over
https://github.com/zitadel/zitadel/pull/10645
<!--
Please inform yourself about the contribution guidelines on submitting a
PR here:
https://github.com/zitadel/zitadel/blob/main/CONTRIBUTING.md#submit-a-pull-request-pr.
Take note of how PR/commit titles should be written and replace the
template texts in the sections below. Don't remove any of the sections.
It is important that the commit history clearly shows what is changed
and why.
Important: By submitting a contribution you agree to the terms from our
Licensing Policy as described here:
https://github.com/zitadel/zitadel/blob/main/LICENSING.md#community-contributions.
-->
# Which Problems Are Solved
Added guide to setup webhook.site
# How the Problems Are Solved
Users can now learn how to setup listener over the internet
# Additional Changes
none
# Additional Context
none
---------
Signed-off-by: Rajat Singh <rajat@zitadel.com>
Co-authored-by: Rajat Singh <rajat@zitadel.com>
This PR fixes a bug where projections could skip events if they were
written within the same microsecond, which can occur during high load on
different transactions.
## Problem
The event query ordering was not fully deterministic. Events created at
the exact same time (same `position`) and in the same transaction
(`in_tx_order`) were not guaranteed to be returned in the same order on
subsequent queries. This could lead to some events being skipped by the
projection logic.
## Solution
To solve this, the `ORDER BY` clause for event queries has been extended
to include `instance_id`, `aggregate_type`, and `aggregate_id`. This
ensures a stable and deterministic ordering for all events, even if they
share the same timestamp.
## Additionally changes:
* Replaced a manual slice search with the more idiomatic
`slices.Contains` to skip already projected instances.
* Changed the handling of already locked projections to log a debug
message and skip execution instead of returning an error.
* Ensures the database transaction is explicitly committed.
# Which Problems Are Solved
During the implementation of #10687, it was noticed that the import
endpoint might provide unnecessary error details.
# How the Problems Are Solved
Remove the underlying (parent) error from the error message.
# Additional Changes
none
# Additional Context
relates to #10687
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
Safari was not creating session cookies during local development,
causing authentication failures. This was due to nextjs default setting
of SameSite cookie property.
We explicitly set "strict" for session cookies now.
Closes#10473
# Which Problems Are Solved
Authentication Issues with Safari in local development
# How the Problems Are Solved
- Cleaner API: Replaced confusing sameSite boolean/string parameters
with iFrameEnabled boolean
- Better logic flow:
iFrameEnabled: true → sameSite: "none" (for iframe embedding)
Production → sameSite: "strict" (maximum security)
This PR removes the Vercel Analytics integration from the login
application to reduce external dependencies and improve privacy.
# Which Problems Are Solved
cleaner csp
# How the Problems Are Solved
- Removed dependency: Uninstalled @vercel/analytics package from
package.json
- Updated layout component: Removed Analytics import and component usage
from layout.tsx
- Updated Content Security Policy: Removed Vercel domains
(https://va.vercel-scripts.com and https://vercel.com) from CSP
configuration in csp.js
Fixed an issue in `isSessionValid()` where users with multiple
configured MFA methods (e.g., TOTP and U2F) would have their sessions
incorrectly invalidated. The function previously used exclusive if-else
logic that only checked the first matching method, causing validation to
fail even when other configured methods were successfully verified.
Closes#10529
# Which Problems Are Solved
[#10529](https://github.com/zitadel/zitadel/issues/10529)
# How the Problems Are Solved
- Replaced exclusive if-else if chain with inclusive validation logic
- Session is now considered valid if ANY configured MFA method has been
verified
- Improved error logging to show all configured methods and their
verification status
Example: A user with both TOTP and U2F configured can now successfully
authenticate using either method, whereas previously the session would
be invalid if they used U2F but TOTP was checked first.
A timing issue (a race condition) was identified in our event processing
system. Under specific circumstances, it was possible for the system to
skip processing certain events, leading to potential data
inconsistencies.
## Which problems are solved
The system tracks its progress through the event log using timestamps.
The issue occurred because we were using the timestamp from the start of
a database transaction. If a query to read new events began after the
transaction started but before the new event was committed, the query
would not see the new event and would fail to process it.
## How the problems are solved
The fix is to change which timestamp is used for tracking. We now use
the precise timestamp of when the event is actually written to the
database. This ensures that the event's timestamp is always correctly
ordered, closing the timing gap and preventing the race condition.
This change enhances the reliability and integrity of our event
processing pipeline. It guarantees that all events are processed in the
correct order and eliminates the risk of skipped events, ensuring data
is always consistent across the system.
## Additional information
original fix: https://github.com/zitadel/zitadel/pull/10560
# Which Problems Are Solved
There was an left-behind index introduced to optimize the old and
removed event execution handler. The index confuses prostgres and it
sometimes picks this index in favor of the projection specific index.
This sometimes leads to bad query performance in the projectio handlers.
# How the Problems Are Solved
Drop the index
# Additional Changes
- none
# Additional Context
- Forgotten in https://github.com/zitadel/zitadel/pull/10564
# Which Problems Are Solved
Actions V2 Method names got cut off in the creation dropdown
<img width="668" height="717" alt="old modal"
src="https://github.com/user-attachments/assets/e3dda16d-5326-464e-abc7-67a8b146037c"
/>
# How the Problems Are Solved
The modal now first requires a Service to be set and only afterwards are
users allowed set Methods. This way we can cut out the Service-Names
from the Method-Name leading to cleaner and shorter names.
<img width="796" height="988" alt="new modal"
src="https://github.com/user-attachments/assets/5002afdf-b639-44ef-954a-5482cca12f96"
/>
# Additional Changes
Changed the Modal dataloading to use Tanstack Query
# Additional Context
- Closes#10596
# Which Problems Are Solved
When using login V2 the Callback URL for an Identity Provider is
different. When following the guideance in the console and using Login
V2 users will use the wrong callback url.
<img width="1234" height="323" alt="grafik"
src="https://github.com/user-attachments/assets/8632ecf2-d9e4-4e3b-8940-2bf80baab8df"
/>
# How the Problems Are Solved
I have added the correct Login V2 url to the identity providers and
updated our docs.
<img width="628" height="388" alt="grafik"
src="https://github.com/user-attachments/assets/2dd4f4f9-d68f-4605-a52e-2e51069da10e"
/>
# Additional Changes
Small refactorings and porting some components over to ChangeDetection
OnPush
# Additional Context
Replace this example with links to related issues, discussions, discord
threads, or other sources with more context.
Use the Closing #issue syntax for issues that are resolved with this PR.
- Closes#10461
---------
Co-authored-by: Max Peintner <max@caos.ch>
# Which Problems Are Solved
Invalid id_tokens used as `id_token_hint` on the authorization endpoints
currently return an error, resp. get display on the endpoint itself.
# How the Problems Are Solved
Ignore invalid id_token_hint errors and just log them.
# Additional Changes
None
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10673
- backport to v4.x
# Which Problems Are Solved
Flakiness and conflicts in value from gofakeit.
# How the Problems Are Solved
Move Gofakeit calls to the integration package, to guarantee proper
usage and values for integration testing.
# Additional Changes
None
# Additional Context
None
Closes#10498
The registration form's legal checkboxes had incorrect validation logic
that prevented users from completing registration when only one legal
document (ToS or Privacy Policy) was configured, or when no legal
documents were required.
additionally removes a duplicate description for "or use Identity
Provider"
# Which Problems Are Solved
Having only partial legal documents was blocking users to register. The
logic now conditionally renders checkboxes and checks if all provided
documents are accepted.
# How the Problems Are Solved
- Fixed checkbox validation: Now properly validates based on which legal
documents are actually available
- acceptance logic: Only requires acceptance of checkboxes that are
shown
- No legal docs support: Users can proceed when no legal documents are
configured
- Proper state management: Fixed checkbox state tracking and mixed-up
test IDs
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
Comparing the v3 and v4 deployments we noticed an increase in memory
usage. A first analysis revealed that it might be related to the
(multiple) initialization of the `i18n.Translator`, partially related
# How the Problems Are Solved
Initialize the tranlator once (apart from the translator interceptor,
which uses context / request specific information) and pass it to all
necessary middleware.
# Additional Changes
Removed unnecessary error return parameter from the translator
initialization.
# Additional Context
- noticed internally
- backport to v4.x
# Which Problems Are Solved
It was noticed that on actions v2 when subscribing to events, the
webhook would always receive an empty `event_payload`:
```
{
"aggregateID": "336494809936035843",
"aggregateType": "user",
"resourceOwner": "336392597046099971",
"instanceID": "336392597046034435",
"version": "v2",
"sequence": 1,
"event_type": "user.human.added",
"created_at": "2025-09-05T08:55:36.156333Z",
"userID": "336392597046755331",
"event_payload":
{}
}
```
The problem was due to using `json.Marshal` on the `Event` interface,
where the underlying `BaseEvent` prevents the data to be marshalled:
131f70db34/internal/eventstore/event_base.go (L38)
# How the Problems Are Solved
The `Event`s `Unmarshal` function is used with a `json.RawMessage`.
# Additional Changes
none
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10650
- requires backport to v4.x
# Which Problems Are Solved
Using the service ping, we want to have some additional insights to how
zitadel is configured. The current resource count report contains
already some amount of configured policies, such as the login_policy.
But we do not know if for example MFA is enforced.
# How the Problems Are Solved
- Added the following counts to the report:
- service users per organization
- MFA enforcements (though login policy)
- Notification policies with password change option enabled
- SCIM provisioned users (using user metadata)
- Since all of the above are conditional based on at least a column
inside a projection, a new `migration.CountTriggerConditional` has been
added, where a condition (column values) and an option to track updates
on that column should be considered for the count.
- For this to be possible, the following changes had to be made to the
existing sql resources:
- the `resource_name` has been added to unique constraint on the
`projection.resource_counts` table
- triggers have been added / changed to individually track `INSERT`,
`UPDATE`(s) and `DELETE` and be able to handle conditions
- an optional argument has been added to the
`projections.count_resource()` function to allow providing the
information to `UP` or `DOWN` count the resource on an update.
# Additional Changes
None
# Additional Context
- partially solves #10244 (reporting audit log retention limit will be
handled in #10245 directly)
- backport to v4.x
# Which Problems Are Solved
For instance settings related to username changes, already removed
organizations were still processed.
# How the Problems Are Solved
Do no process already removed organizations.
# Additional Changes
None
# Additional Context
None
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
Flakiness in integration tests for organization v2beta service.
# How the Problems Are Solved
Fix eventual consistent handling of integration tests.
# Additional Changes
None
# Additional Context
None
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
HTTP Request to HTTP providers for Email or SMS are not signed.
# How the Problems Are Solved
Add a Signing Key to the HTTP Provider resources, which is then used to
generate a header to sign the payload.
# Additional Changes
Additional tests for query side of the SMTP provider.
# Additional Context
Closes#10067
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
Update the Go toolchain to the latest bugfix release, so we include
latest security fixes in the standard library.
# How the Problems Are Solved
Set the toolchain directive to 1.24.7
# Additional Changes
- go mod tidy
# Additional Context
- https://go.dev/doc/devel/release#go1.24.0
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
This PR adds functionality to propagate request headers in actions v2.
# How the Problems Are Solved
The new functionality is added to the`ExecutionHandler` interceptors,
where the incoming request headers (from a list of allowed headers to be
forwarded) are set in the payload of the request before calling the
target.
# Additional Changes
This PR also contains minor fixes to the Actions V2 example docs.
# Additional Context
- Closes#9941
---------
Co-authored-by: Marco A. <marco@zitadel.com>
This pull request fixes an issue where the repository would fail to scan
organization or instance structs if the `domains` column was `NULL`.
## Which problems are solved
If the `domains` column of `orgs` or `instances` was `NULL`, the
repository failed scanning into the structs. This happened because the
scanning mechanism did not correctly handle `NULL` JSONB columns.
## How the problems are solved
A new generic type `JSONArray[T]` is introduced, which implements the
`sql.Scanner` interface. This type can correctly scan JSON arrays from
the database, including handling `NULL` values gracefully.
The repositories for instances and organizations have been updated to
use this new type for the domains field. The SQL queries have also been
improved to use `FILTER` with `jsonb_agg` for better readability and
performance when aggregating domains.
## Additional changes
* An unnecessary cleanup step in the organization domain tests for
already removed domains has been removed.
* The `pgxscan` library has been replaced with `sqlscan` for scanning
`database/sql`.Rows.
* Minor cleanups in integration tests.
This PR initiates the rework of Zitadel's backend to state-persisted
objects. This change is a step towards a more scalable and maintainable
architecture.
## Changes
* **New `/backend/v3` package**: A new package structure has been
introduced to house the reworked backend logic. This includes:
* `domain`: Contains the core business logic, commands, and repository
interfaces.
* `storage`: Implements the repository interfaces for database
interactions with new transactional tables.
* `telemetry`: Provides logging and tracing capabilities.
* **Transactional Tables**: New database tables have been defined for
`instances`, `instance_domains`, `organizations`, and `org_domains`.
* **Projections**: New projections have been created to populate the new
relational tables from the existing event store, ensuring data
consistency during the migration.
* **Repositories**: New repositories provide an abstraction layer for
accessing and manipulating the data in the new tables.
* **Setup**: A new setup step for `TransactionalTables` has been added
to manage the database migrations for the new tables.
This PR lays the foundation for future work to fully transition to
state-persisted objects for these components, which will improve
performance and simplify data access patterns.
This PR initiates the rework of ZITADEL's backend to state-persisted
objects. This is a foundational step towards a new architecture that
will improve performance and maintainability.
The following objects are migrated from event-sourced aggregates to
state-persisted objects:
* Instances
* incl. Domains
* Orgs
* incl. Domains
The structure of the new backend implementation follows the software
architecture defined in this [wiki
page](https://github.com/zitadel/zitadel/wiki/Software-Architecturel).
This PR includes:
* The initial implementation of the new transactional repositories for
the objects listed above.
* Projections to populate the new relational tables from the existing
event store.
* Adjustments to the build and test process to accommodate the new
backend structure.
This is a work in progress and further changes will be made to complete
the migration.
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Iraq Jaber <iraq+github@zitadel.com>
Co-authored-by: Iraq <66622793+kkrime@users.noreply.github.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
