# Which Problems Are Solved
This is the second PR related to the backend implementation of
GroupService to manage user groups.
The first [PR](https://github.com/zitadel/zitadel/pull/10455) implements
the Command-side.
This PR implements the query side.
# How the Problems Are Solved
* Query-side implementation to search/list groups by
* a list of Group IDs
* by the Group name
* by the Organization ID
# Additional Changes
N/A
# Additional Context
- Follow-up for PR #10455
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
As part of our efforts to simplify the structure and versions of our
APIs, were moving all existing v2beta endpoints to v2 and deprecate
them. They will be removed in Zitadel V5.
# How the Problems Are Solved
- This PR deprecates all v2beta service and their endpoints, which have
already a corresponding v2 version and should not be used anymore.
- The comments and have been improved and, where not already done, moved
from swagger annotations to proto.
- All required fields have been marked with `(google.api.field_behavior)
= REQUIRED` and validation rules have been added where missing.
- Removed the "required flag" comments on the Action and WebKey service
endpoints, since they were removed in Zitadel v4.
- The `SetSession` endpoint already documented that the token does not
have to be provided anymore and will be ignored if still sent, but it
was actually still checked if provided. The corresponding check has been
removed and the field is now properly deprecated in the proto as well.
# Additional Changes
None
# Additional Context
- part of #10772
- requires backport to v4.x
# Which Problems Are Solved
In the integration with Github, private emails are not returned with the
userinfo.
# How the Problems Are Solved
If the scope `user:email` is set in the Github IDP and the email is not
included in the userinfo, a request to Github's API is executed to query
the email of the user.
# Additional Changes
Additional tests.
# Additional Context
Closes#10098
---------
Co-authored-by: Marco A. <marco@zitadel.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Currently there is only the option to either filter for all usergrants
of an organization or the usergrants of a singluar user.
# How the Problems Are Solved
Add the option to provide a list of userIDs to query user grants.
# Additional Changes
Fixed internal typo for function.
# Additional Context
Closes#9675
# Which Problems Are Solved
This PR adds API definition and backend implementation for GroupService
to manage user groups.
# How the Problems Are Solved
* API definition to create, update, retrieve, and delete groups is added
* Command-side implementation to create, update, and delete user groups
as part of the GroupV2 API is added
# Additional Changes
N/A
# Additional Context
- Related to #10089, #9702 (parent ticket)
- User contribution: https://github.com/zitadel/zitadel/pull/9428/files
- Additional functionalities to list/search user groups, add
permissions, manage users in groups, group scopes will be added in
subsequent PRs.
- Also needs documentation, which will be added once the entire feature
is available
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Add projects to the relational tables
# How the Problems Are Solved
- Define table migrations
- Define and implement Project and Project Role repositories.
- Provide projection handlers to populate the relational tables.
# Additional Changes
- Statement Builder now has a constructor which allows setting of a base
query with arguments.
- Certain operations, like Get, Update and Delete require the Primary
Key to be set as conditions. However, this requires knowledge of the
implementation and table definition. This PR proposes an additional
condition for repositories: `PrimaryKeyCondition`. This gives clarity on
the required IDs for these operations.
- Added couple of helpers to the repository package, to allow more DRY
code.
- getOne / getMany: generic functions for query execution and scanning.
- checkRestrictingColumns, checkPkCondition: simplify condition
checking, instead of using ladders of conditionals.
- Added a couple of helpers to the repository test package:
- Transaction, savepoint and rollback helpers.
- Create instance and organization helpers for objects that depend on
them (like projects).
# Additional Context
- after https://github.com/zitadel/zitadel/pull/10809
- closes#10765
# Which Problems Are Solved
https://github.com/zitadel/zitadel/pull/10520 added the possibility to
specify the signature algorithm for SAML auth requests. After releasing,
customer noticed that the Console UI would not correctly display the
selected algorithm and that it was not used in the login V1.
# How the Problems Are Solved
- Correctly map the algorithm in the UI
- Provide the option to the idp when creating a SAML request in login V1
# Additional Changes
None
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10780
- closes https://github.com/zitadel/zitadel/issues/10792
- requires backport to v4.x
# Which Problems Are Solved
While reviewing #9954, i noticed eventual consistency issues in the
session integration tests. All creation and change dates as well as
checked_at were tested using a `window` duration, typically one minute
from `time.Now()`. If some precondition took longer, they would all
fail.
# How the Problems Are Solved
Changed the tests to use the information returned by the creation / set
session calls and make sure they're in those timeframes.
Added a clock skew for the factor checks, since there's an inconsistency
in the event payload and event date: #10791
# Additional Changes
None
# Additional Context
- noted in #9954
- requires backport to v4.x
# Which Problems Are Solved
The /userinfo endpoint only returns roles for the current project, even
if the access token includes multiple project aud scopes.
This prevents clients from retrieving all user roles across multiple
projects, making multi-project access control ineffective.
# How the Problems Are Solved
Modified the /userinfo handler logic to resolve roles across all valid
project audience scopes provided in the token, not just the current
project.
Ensured that if **urn:zitadel:iam:org:projects:roles is in the scopes**,
roles from all declared project audiences are collected and included in
the response in **urn:zitadel:iam:org:projects:roles claim**.
# Additional Changes
# Additional Context
This change enables service-to-service authorization workflows and SPA
role resolution across multiple project contexts with a single token.
- Closes#9831
---------
Co-authored-by: Masum Patel <patelmasum98@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
# Which Problems Are Solved
Project Grant ID would have needed to be unique to be handled properly
on the projections, but was defined as the organization ID the project
was granted to, so could be non-unique.
# How the Problems Are Solved
Generate the Project Grant ID even in the v2 APIs, which also includes
fixes in the integration tests.
Additionally to that, the logic for some functionality had to be
extended as the Project Grant ID is not provided anymore in the API, so
had to be queried before creating events for Project Grants.
# Additional Changes
Included fix for authorizations, when an authorization was intended to
be created for a project, without providing any organization
information, which also showed some faulty integration tests.
# Additional Context
Partially closes#10745
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
This PR fixes the self-management of users for metadata and own removal
and improves the corresponding permission checks.
While looking into the problems, I also noticed that there's a bug in
the metadata mapping when using `api.metadata.push` in actions v1 and
that re-adding a previously existing key after its removal was not
possible.
# How the Problems Are Solved
- Added a parameter `allowSelfManagement` to checkPermissionOnUser to
not require a permission if a user is changing its own data.
- Updated use of `NewPermissionCheckUserWrite` including prevention of
self-management for metadata.
- Pass permission check to the command side (for metadata functions) to
allow it implicitly for login v1 and actions v1.
- Use of json.Marshal for the metadata mapping (as with
`AppendMetadata`)
- Check the metadata state when comparing the value.
# Additional Changes
- added a variadic `roles` parameter to the `CreateOrgMembership`
integration test helper function to allow defining specific roles.
# Additional Context
- noted internally while testing v4.1.x
- requires backport to v4.x
- closes https://github.com/zitadel/zitadel/issues/10470
- relates to https://github.com/zitadel/zitadel/pull/10426
…of project grant
# Which Problems Are Solved
On Management API the fields for `GrantedOrgId`, `GrantedOrgName` and
`GrantedOrgDomain` were only filled if it was a usergrant for a granted
project.
# How the Problems Are Solved
Correctly query the Organization of the User again to the Organization
the Project is granted to.
Then fill in the information about the Organization of the User in the
fields `GrantedOrgId`, `GrantedOrgName` and `GrantedOrgDomain`.
# Additional Changes
Additionally query the information about the Organization the Project is
granted to, to have it available for the Authorization v2beta API.
# Additional Context
Closes#10723
---------
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
# Which Problems Are Solved
Eventual consistency issues.
# How the Problems Are Solved
Correctly handle timeouts and change queries to domains instead of using
the organization name.
# Additional Changes
None
# Additional Context
None
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
During the implementation of #10687, it was noticed that the import
endpoint might provide unnecessary error details.
# How the Problems Are Solved
Remove the underlying (parent) error from the error message.
# Additional Changes
none
# Additional Context
relates to #10687
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
Invalid id_tokens used as `id_token_hint` on the authorization endpoints
currently return an error, resp. get display on the endpoint itself.
# How the Problems Are Solved
Ignore invalid id_token_hint errors and just log them.
# Additional Changes
None
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/10673
- backport to v4.x
# Which Problems Are Solved
Flakiness and conflicts in value from gofakeit.
# How the Problems Are Solved
Move Gofakeit calls to the integration package, to guarantee proper
usage and values for integration testing.
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
Comparing the v3 and v4 deployments we noticed an increase in memory
usage. A first analysis revealed that it might be related to the
(multiple) initialization of the `i18n.Translator`, partially related
# How the Problems Are Solved
Initialize the tranlator once (apart from the translator interceptor,
which uses context / request specific information) and pass it to all
necessary middleware.
# Additional Changes
Removed unnecessary error return parameter from the translator
initialization.
# Additional Context
- noticed internally
- backport to v4.x
# Which Problems Are Solved
Flakiness in integration tests for organization v2beta service.
# How the Problems Are Solved
Fix eventual consistent handling of integration tests.
# Additional Changes
None
# Additional Context
None
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
HTTP Request to HTTP providers for Email or SMS are not signed.
# How the Problems Are Solved
Add a Signing Key to the HTTP Provider resources, which is then used to
generate a header to sign the payload.
# Additional Changes
Additional tests for query side of the SMTP provider.
# Additional Context
Closes#10067
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
This PR adds functionality to propagate request headers in actions v2.
# How the Problems Are Solved
The new functionality is added to the`ExecutionHandler` interceptors,
where the incoming request headers (from a list of allowed headers to be
forwarded) are set in the payload of the request before calling the
target.
# Additional Changes
This PR also contains minor fixes to the Actions V2 example docs.
# Additional Context
- Closes#9941
---------
Co-authored-by: Marco A. <marco@zitadel.com>
This pull request fixes an issue where the repository would fail to scan
organization or instance structs if the `domains` column was `NULL`.
## Which problems are solved
If the `domains` column of `orgs` or `instances` was `NULL`, the
repository failed scanning into the structs. This happened because the
scanning mechanism did not correctly handle `NULL` JSONB columns.
## How the problems are solved
A new generic type `JSONArray[T]` is introduced, which implements the
`sql.Scanner` interface. This type can correctly scan JSON arrays from
the database, including handling `NULL` values gracefully.
The repositories for instances and organizations have been updated to
use this new type for the domains field. The SQL queries have also been
improved to use `FILTER` with `jsonb_agg` for better readability and
performance when aggregating domains.
## Additional changes
* An unnecessary cleanup step in the organization domain tests for
already removed domains has been removed.
* The `pgxscan` library has been replaced with `sqlscan` for scanning
`database/sql`.Rows.
* Minor cleanups in integration tests.
# Which Problems Are Solved
Unnecessary default permission check in creating an authorization fails
even if the functionality was called internally.
# How the Problems Are Solved
Move permission check to the proper implementation, so that necessary
permission checks are provided by the responsible API.
# Additional Changes
None
# Additional Context
Closes#10624
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
This PR introduces a new feature flag `EnableRelationalTables` that will
be used in following implementations to decide whether Zitadel should
use the relational model or the event sourcing one.
# TODO
- [x] Implement flag at system level
- [x] Display the flag on console:
https://github.com/zitadel/zitadel/pull/10615
# How the Problems Are Solved
- Implement loading the flag from config
- Add persistence of the flag through gRPC endpoint
(SetInstanceFeatures)
- Implement reading of the flag through gRPC endpoint
(GetInstanceFeatures)
# Additional Changes
Some minor refactoring to remove un-needed generics annotations
# Additional Context
- Closes#10574
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
Some users have reported the need of retrieving users given a metadata
key, metadata value or both. This change introduces metadata search
filter on the `ListUsers()` endpoint to allow Zitadel users to search
for user records by metadata.
The changes affect only v2 APIs.
# How the Problems Are Solved
- Add new search filter to `ListUserRequest`: `MetaKey` and `MetaValue`
- Add SQL indices on metadata key and metadata value
- Update query to left join `user_metadata` table
# Additional Context
- Closes#9053
- Depends on https://github.com/zitadel/zitadel/pull/10567
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
The TestServer_Limits_AuditLogRetention is too reliant on time
constraints when checking that a limit is correctly applied. IN case it
takes to long to do all the preparation, there won't be any events to
read and the test will fail.
# How the Problems Are Solved
Don't require any events to be returned.
# Additional Changes
None
# Additional Context
- Noted a lot of pipeline to fail on this step.
- requires backport to at least v4.x
# Which Problems Are Solved
The event execution system currently uses a projection handler that
subscribes to and processes all events for all instances. This creates a
high static cost because the system over-fetches event data, handling
many events that are not needed by most instances. This inefficiency is
also reflected in high "rows returned" metrics in the database.
# How the Problems Are Solved
Eliminate the use of a project handler. Instead, events for which
"execution targets" are defined, are directly pushed to the queue by the
eventstore. A Router is populated in the Instance object in the authz
middleware.
- By joining the execution targets to the instance, no additional
queries are needed anymore.
- As part of the instance object, execution targets are now cached as
well.
- Events are queued within the same transaction, giving transactional
guarantees on delivery.
- Uses the "insert many fast` variant of River. Multiple jobs are queued
in a single round-trip to the database.
- Fix compatibility with PostgreSQL 15
# Additional Changes
- The signing key was stored as plain-text in the river job payload in
the DB. This violated our [Secrets
Storage](https://zitadel.com/docs/concepts/architecture/secrets#secrets-storage)
principle. This change removed the field and only uses the encrypted
version of the signing key.
- Fixed the target ordering from descending to ascending.
- Some minor linter warnings on the use of `io.WriteString()`.
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/9249
- Closes https://github.com/zitadel/zitadel/issues/10553
- Closes https://github.com/zitadel/zitadel/issues/9832
- Closes https://github.com/zitadel/zitadel/issues/10372
- Closes https://github.com/zitadel/zitadel/issues/10492
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
Integration test failed sometimes with `organization already
exists`-errors.
# How the Problems Are Solved
Use a consistent function to generate name used for organization
creation.
# Additional Changes
Correct a eventual consistent test for username around organization
domain changes with eventual consistent loop.
# Additional Context
None
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
- Inconsistencies in the terminology used for "身份认证提供商" (identity
provider) and "身份认证提供者" (identity supplier) in the Chinese translation
files could lead to confusion among users.
- Missing translations for terminology related to identity providers
could hinder user experience and understanding.
# How the Problems Are Solved
- Unified the terms "身份认证提供商" and "身份认证提供者" to consistently use
"身份认证提供者" across all Chinese translation files.
- Added necessary translations to ensure that all relevant terms related
to identity providers are accurately represented in the Chinese
localization.
# Additional Changes
- Improved overall readability and clarity in the Chinese translations
by ensuring consistent terminology for identity-related phrases
throughout the application.
- Complete the missing translations.
# Additional Context
If I have missed any translations, please point them out, and I would be
happy to complete them.
---------
Co-authored-by: Florian Forster <florian@zitadel.com>
# Which Problems Are Solved
When a SAML IDP is created, the signing algorithm defaults to
`RSA-SHA1`.
This PR adds the functionality to configure the signing algorithm while
creating or updating a SAML IDP. When nothing is specified, `RSA-SHA1`
is the default.
Available options:
* RSA_SHA1
* RSA_SHA256
* RSA_SHA512
# How the Problems Are Solved
By introducing a new optional config to specify the Signing Algorithm.
# Additional Changes
N/A
# Additional Context
- Closes#9842
An existing bug in the UpdateSAMLProvider API will be fixed as a
followup in a different
[PR](https://github.com/zitadel/zitadel/pull/10557).
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
Emails are still send only with URLs to login v1.
# How the Problems Are Solved
Add configuration for URLs as URL templates, so that links can point at
Login v2.
# Additional Changes
None
# Additional Context
Closes#10236
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
Project Admins would get permission errors when trying to add project
roles
# How the Problems Are Solved
Fixed wrong parameters were being passed into the permission check
- Closes https://github.com/zitadel/zitadel/issues/10505
# Which Problems Are Solved
EntraID userinfo gets incorrectly unmarshalled again in the
`RetrieveIdentityProviderIntent` endpoint.
# How the Problems Are Solved
Correctly use the already available information and not try to marshall
it into a `RawInformation` struct again.
# Additional Changes
None
# Additional Context
Closes https://github.com/zitadel/typescript/issues/578
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Some events that are now unused are clogging the event queue from time
to time.
# How the Problems Are Solved
Remove the events described in #10458
# Additional Changes
- Updated `stringer` and `enumer` in Makefile target `core_generate_all`
to resolve generated files compilation issues
# Notes
It looks like there are a lot of changes, but most of it is fixing
translation files. I suggest doing a review per-commit
# Additional Context
- Closes#10458
- Depends on https://github.com/zitadel/zitadel/pull/10513
# Which Problems Are Solved
`ListAppKeys()` does not work properly, in that it does not return any
app keys.
# How the Problems Are Solved
The issue stems from a mistake SQL query not joining the
`projections.authn_keys2` table to `projections.projects4` instead of
joining to `projections.apps7`
# Additional Changes
`ListAppKeys()` returns the app key IDs in order of their creation
- Closes https://github.com/zitadel/zitadel/issues/10420
- backport to v4.x
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
When a user with an `ORG_PROJECT_CREATOR` role tries to create a
project, the request fails with `No matching permissions found
(AUTH-AWfge)` error. This is because `project.write` was set as the
required permission instead of `project.create` during project creation.
# How the Problems Are Solved
By setting the right required permission (`project.create`) while
creating new projects.
# Additional Changes
N/A
# Additional Context
- Closes#10399
Querying an organization by id allowed to trigger the org projection.
This could lead to performance impacts if the projection gets triggered
too often.
Instead of executing the trigger the organization by id query is now
always executed on the eventstore and reduces all event types required
of the organization requested.
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
User information in the context is necessary through the addition of the
resource based API endpoints for user metadata, for the permission
check.
# How the Problems Are Solved
Add user information to the action execution to add metadata to users.
# Additional Changes
None
# Additional Context
Needs to be added to v4 releases, to provide the functionality to add
metadata through actions v1 and actions v2 functions.
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
Flakiness in integration tests regarding gofakeit functions, which
provided the same names on 2 different occasions.
# How the Problems Are Solved
Attach a random string to the provided names, so that they are not
dependent on the gofakeit code.
# Additional Changes
None
# Additional Context
None
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
The deletion of expired sessions does not go through even though a
success response is returned to the user. These expired and supposedly
deleted (to the user) sessions are then returned when the `ListSessions`
API is called.
This PR fixes this issue by:
1. Allowing deletion of expired sessions
2. Providing an `expiration_date` filter in `ListSession` API to filter
sessions by expiration date
# How the Problems Are Solved
1. Remove expired session check during deletion
2. Add an `expiration_date` filter to the `ListSession` API
# Additional Changes
N/A
# Additional Context
- Closes#10045
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
The Actions v2beta API is not yet promoted to GA.
# How the Problems Are Solved
Promote Actions v2Beta API to Actions v2 API.
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
Currently the username uniqueness is on instance level, we want to
achieve a way to set it at organization level.
# How the Problems Are Solved
Addition of endpoints and a resource on organization level, where this
setting can be managed. If nothing it set, the uniqueness is expected to
be at instance level, where only users with instance permissions should
be able to change this setting.
# Additional Changes
None
# Additional Context
Includes #10086Closes#9964
---------
Co-authored-by: Marco A. <marco@zitadel.com>
# Which Problems Are Solved
[Permissions v2](https://github.com/zitadel/zitadel/issues/9972) is not
possible in the current implementation.
# How the Problems Are Solved
We remove Permissions v2 from project grants related API calls, to
alleviate this problems.
Resulting in some removals of testing, implementations and performance
impact
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
There is no information contained in the context info sent to Actions
v2.
# How the Problems Are Solved
Add application information to the context information sent to Actions
v2, to give more information about the execution.
# Additional Changes
None
# Additional Context
Closes#9377
# Which Problems Are Solved
The OIDC session endpoint allows to pass a `id_token_hint` to identify
the session to terminate. In case the application is not able to pass
that, e.g. Console currently allows multiple sessions to be open, but
will only store the id_token of the current session, allowing to pass
the `logout_hint` to identify the user adds some new possibilities.
# How the Problems Are Solved
In case the end_session_endpoint is called with no `id_token_hint`, but
a `logout_hint` and the v2 login UI is configured, the information is
passed to the login UI also as `login_hint` parameter to allow the login
UI to determine the session to be terminated, resp. let the user decide.
# Additional Changes
Also added the `ui_locales` as parameter to handle and pass to the V2
login UI.
# Dependencies ⚠️
~These changes depend on https://github.com/zitadel/oidc/pull/774~
# Additional Context
closes#9847
---------
Co-authored-by: Marco Ardizzone <marco@zitadel.com>
# Which Problems Are Solved
Actions v2beta API does not adhere to the [API
design](https://github.com/zitadel/zitadel/blob/main/API_DESIGN.md)
fully.
# How the Problems Are Solved
- Correct body usage for ListExecutions
- Correct REST path for ListTargets and ListExecutions
- Correct attribute names for ListTargetsResponse and
ListExecutionsResponse
# Additional Changes
- Remove unused object import.
# Additional Context
Closes#10138
---------
Co-authored-by: Marco A. <marco@zitadel.com>
