# Which Problems Are Solved
There is no option to only query auth methods related to specific
domains.
# How the Problems Are Solved
Add domain as attribute to the ListAuthenticationMethodTypes request.
# Additional Changes
OwnerRemoved column removed from the projection.
# Additional Context
Closes#8615
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Since the optiimzation of the token endpoint, we longer push the
`user.token.added` event. However, the milestone projection keeps
quering for it, including a payload query.
This incured a static waste of DB resources.
# How the Problems Are Solved
Remove the `user.token.added` event reducer from the milestone
projection
# Additional Changes
- none
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/8742. Other
changes ommitted so this PR can be backported to stable.
# Which Problems Are Solved
Optimize the query that checks for terminated sessions in the access
token verifier. The verifier is used in auth middleware, userinfo and
introspection.
# How the Problems Are Solved
The previous implementation built a query for certain events and then
appended a single `PositionAfter` clause. This caused the postgreSQL
planner to use indexes only for the instance ID, aggregate IDs,
aggregate types and event types. Followed by an expensive sequential
scan for the position. This resulting in internal over-fetching of rows
before the final filter was applied.
Furthermore, the query was searching for events which are not always
applicable. For example, there was always a session ID search and if
there was a user ID, we would also search for a browser fingerprint in
event payload (expensive). Even if those argument string would be empty.
This PR changes:
1. Nest the position query, so that a full `instance_id, aggregate_id,
aggregate_type, event_type, "position"` index can be matched.
2. Redefine the `es_wm` index to include the `position` column.
3. Only search for events for the IDs that actually have a value. Do not
search (noop) if none of session ID, user ID or fingerpint ID are set.
New query plan:
# Additional Changes
- cleanup how we load multi-statement migrations and make that a bit
more reusable.
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/7639
# Which Problems Are Solved
Cache implementation using a PGX connection pool.
# How the Problems Are Solved
Defines a new schema `cache` in the zitadel database.
A table for string keys and a table for objects is defined.
For postgreSQL, tables are unlogged and partitioned by cache name for
performance.
Cockroach does not have unlogged tables and partitioning is an
enterprise feature that uses alternative syntax combined with sharding.
Regular tables are used here.
# Additional Changes
- `postgres.Config` can return a pxg pool. See following discussion
# Additional Context
- Part of https://github.com/zitadel/zitadel/issues/8648
- Closes https://github.com/zitadel/zitadel/issues/8647
---------
Co-authored-by: Silvan <silvan.reusser@gmail.com>
# Which Problems Are Solved
https://github.com/zitadel/zitadel/pull/8545 incorrectly created the
list of current smtp providers, if an SMTP provider was changed, that
was created before https://github.com/zitadel/zitadel/pull/6932 /
[v2.50.0](https://github.com/zitadel/zitadel/releases/tag/v2.50.0)).
This led to problems when trying to send emails to users (email
verification and OTP email).
# How the Problems Are Solved
Correctly handle events of old SMTP configurations, which do not have an
id set.
# Additional Changes
None
# Additional Context
- relates to #8545
- support requests from cloud customers
# Which Problems Are Solved
Errors messages are mixed up for some policies
# How the Problems Are Solved
Define new error IDs and correct the messages.
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
There are cases where not all statements of multiExec are succeed. This
leads to inconsistent states. One example is [LDAP
IDPs](https://github.com/zitadel/zitadel/issues/7959).
If statements get executed only partially this can lead to inconsistent
states or even break projections for objects which might not were
correctly created in a sub table.
This behaviour is possible because we use
[`SAVEPOINTS`](https://www.postgresql.org/docs/current/sql-savepoint.html)
during each statement of a multiExec.
# How the Problems Are Solved
SAVEPOINTS are only created at the beginning of an exec function not
during every execution like before. Additionally `RELEASE` or `ROLLBACK`
of `SAVEPOINTS` are only used when needed.
# Additional Changes
- refactor some unused parameters
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/7959
# Which Problems Are Solved
When executing many concurrent authentication requests on a single
machine user, there were performance issues. As the same aggregate is
being searched and written to concurrently, we traced it down to a
locking issue on the used index.
We already optimized the token endpoint by creating a separate OIDC
aggregate.
At the time we decided to push a single event to the user aggregate, for
the user audit log. See [technical advisory
10010](https://zitadel.com/docs/support/advisory/a10010) for more
details.
However, a recent security fix introduced an additional search query on
the user aggregate, causing the locking issue we found.
# How the Problems Are Solved
Add a feature flag which disables pushing of the `user.token.v2.added`.
The event has no importance and was only added for informational
purposes on the user objects. The `oidc_session.access_token.added` is
the actual payload event and is pushed on the OIDC session aggregate and
can still be used for audit trail.
# Additional Changes
- Fix an event mapper type for
`SystemOIDCSingleV1SessionTerminationEventType`
# Additional Context
- Reported by support request
- https://github.com/zitadel/zitadel/pull/7822 changed the token
aggregate
- https://github.com/zitadel/zitadel/pull/8631 introduced user state
check
Load test trace graph with `user.token.v2.added` **enabled**. Query
times are steadily increasing:
Load test trace graph with `user.token.v2.added` **disabled**. Query
times constant:
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Twilio supports a robust, multi-channel verification service that
notably supports multi-region SMS sender numbers required for our use
case. Currently, Zitadel does much of the work of the Twilio Verify (eg.
localization, code generation, messaging) but doesn't support the pool
of sender numbers that Twilio Verify does.
# How the Problems Are Solved
To support this API, we need to be able to store the Twilio Service ID
and send that in a verification request where appropriate: phone number
verification and SMS 2FA code paths.
This PR does the following:
- Adds the ability to use Twilio Verify of standard messaging through
Twilio
- Adds support for international numbers and more reliable verification
messages sent from multiple numbers
- Adds a new Twilio configuration option to support Twilio Verify in the
admin console
- Sends verification SMS messages through Twilio Verify
- Implements Twilio Verification Checks for codes generated through the
same
# Additional Changes
# Additional Context
- base was implemented by @zhirschtritt in
https://github.com/zitadel/zitadel/pull/8268❤️
- closes https://github.com/zitadel/zitadel/issues/8581
---------
Co-authored-by: Zachary Hirschtritt <zachary.hirschtritt@klaviyo.com>
Co-authored-by: Joey Biscoglia <joey.biscoglia@klaviyo.com>
# Which Problems Are Solved
We identified the need of caching.
Currently we have a number of places where we use different ways of
caching, like go maps or LRU.
We might also want shared chaches in the future, like Redis-based or in
special SQL tables.
# How the Problems Are Solved
Define a generic Cache interface which allows different implementations.
- A noop implementation is provided and enabled as.
- An implementation using go maps is provided
- disabled in defaults.yaml
- enabled in integration tests
- Authz middleware instance objects are cached using the interface.
# Additional Changes
- Enabled integration test command raceflag
- Fix a race condition in the limits integration test client
- Fix a number of flaky integration tests. (Because zitadel is super
fast now!) 🎸🚀
# Additional Context
Related to https://github.com/zitadel/zitadel/issues/8648
# Which Problems Are Solved
ZITADEL's user account deactivation mechanism did not work correctly
with service accounts. Deactivated service accounts retained the ability
to request tokens, which could lead to unauthorized access to
applications and resources.
# How the Problems Are Solved
Additionally to checking the user state on the session API and login UI,
the state is checked on all oidc session methods resulting in a new
token or when returning the user information (userinfo, introspection,
id_token / access_token and saml attributes)
# Which Problems Are Solved
ZITADEL's user grants deactivation mechanism did not work correctly.
Deactivated user grants were still provided in token, which could lead
to unauthorized access to applications and resources.
Additionally, the management and auth API always returned the state as
active or did not provide any information about the state.
# How the Problems Are Solved
- Correctly check the user grant state on active for tokens and user
information (userinfo, introspection, saml attributes)
- Map state in API and display in Console
# Which Problems Are Solved
In Zitadel, even after an organization is deactivated, associated
projects, respectively their applications remain active. Users across
other organizations can still log in and access through these
applications, leading to unauthorized access.
Additionally, if a project was deactivated access to applications was
also still possible.
# How the Problems Are Solved
- Correctly check the status of the organization and related project.
(Corresponding functions have been renamed to `Active...`)
There were some problems after changing from SMTP providers to email
providers (https://github.com/zitadel/zitadel/pull/8545):
- panic (nil pointer) on SMTP provider retrieval for HTTP configs
- old SMTP configuration created before the multi SMTP configurations
(without id)
- were not projected / listed
- could not be always be activated
- Console treated HTTP configs as SMTP
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Send Email messages as a HTTP call to a relay, for own logic on handling
different Email providers
# How the Problems Are Solved
Create endpoints under Email provider to manage SMTP and HTTP in the
notification handlers.
# Additional Changes
Clean up old logic in command and query side to handle the general Email
providers with deactivate, activate and remove.
# Additional Context
Partially closes#8270
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
As an administrator I want to be able to invite users to my application
with the API V2, some user data I will already prefil, the user should
add the authentication method themself (password, passkey, sso).
# How the Problems Are Solved
- A user can now be created with a email explicitly set to false.
- If a user has no verified email and no authentication method, an
`InviteCode` can be created through the User V2 API.
- the code can be returned or sent through email
- additionally `URLTemplate` and an `ApplicatioName` can provided for
the email
- The code can be resent and verified through the User V2 API
- The V1 login allows users to verify and resend the code and set a
password (analog user initialization)
- The message text for the user invitation can be customized
# Additional Changes
- `verifyUserPasskeyCode` directly uses `crypto.VerifyCode` (instead of
`verifyEncryptedCode`)
- `verifyEncryptedCode` is removed (unnecessarily queried for the code
generator)
# Additional Context
- closes#8310
- TODO: login V2 will have to implement invite flow:
https://github.com/zitadel/typescript/issues/166
# Which Problems Are Solved
Improve performance by removing a GetUserByID call. The call also
executed a Trigger on projections, which significantly impacted
concurrent requests.
# How the Problems Are Solved
Token creation needs information from the user, such as the resource
owner and access token type.
For client credentials this is solved in a single search. By getting the
user by username (`client_id`), the user details and secret were
obtained in a single query. After that verification and token creation
can proceed. For JWT profile it is a bit more complex. We didn't know
anything about the user until after JWT verification.
The verification did a query for the AuthN key and after that we did a
GetUserByID to get remaining details.
This change uses a joined query when the OIDC library calls the
`GetKeyByIDAndClientID` method on the token storage. The found user
details are set to the verifieer object and returned after verification
is completed.
It is safe because the `jwtProfileKeyStorage` is a single-use object as
a wrapper around `query.Queries`.
This way getting the public key and user details are obtained in a
single query.
# Additional Changes
- Correctly set the `client_id` field with machine's username.
# Additional Context
- Related to: https://github.com/zitadel/zitadel/issues/8352
# Which Problems Are Solved
Add a debug API which allows pushing a set of events to be reduced in a
dedicated projection.
The events can carry a sleep duration which simulates a slow query
during projection handling.
# How the Problems Are Solved
- `CreateDebugEvents` allows pushing multiple events which simulate the
lifecycle of a resource. Each event has a `projectionSleep` field, which
issues a `pg_sleep()` statement query in the projection handler :
- Add
- Change
- Remove
- `ListDebugEventsStates` list the current state of the projection,
optionally with a Trigger
- `GetDebugEventsStateByID` get the current state of the aggregate ID in
the projection, optionally with a Trigger
# Additional Changes
- none
# Additional Context
- Allows reproduction of https://github.com/zitadel/zitadel/issues/8517
# Which Problems Are Solved
Send SMS messages as a HTTP call to a relay, for own logic on handling
different SMS providers.
# How the Problems Are Solved
Add HTTP as SMS provider type and handling of webhook messages in the
notification handlers.
# Additional Changes
Clean up old Twilio events, which were supposed to handle the general
SMS providers with deactivate, activate and remove.
# Additional Context
Partially closes#8270
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Float64 which was used for the event.Position field is [not precise in
go and gets rounded](https://github.com/golang/go/issues/47300). This
can lead to unprecies position tracking of events and therefore
projections especially on cockcoachdb as the position used there is a
big number.
example of a unprecies position:
exact: 1725257931223002628
float64: 1725257931223002624.000000
# How the Problems Are Solved
The float64 was replaced by
[github.com/jackc/pgx-shopspring-decimal](https://github.com/jackc/pgx-shopspring-decimal).
# Additional Changes
Correct behaviour of makefile for load tests.
Rename `latestSequence`-queries to `latestPosition`
# Which Problems Are Solved
The end_session_endpoint currently always requires the userAgent cookie
to be able to terminate session created through the hosted login UI.
Only tokens issued through the Login V2 can be used to directly
terminate a specific session and without the need of a cookie.
This PR adds the possibility to terminate a single V1 session or all V1
sessions belonging to the same user agent without the need of the
userAgent cookie by providing an id_token as `id_token_hint` which
contains the id of a V1 session as `sid`.
# How the Problems Are Solved
- #8525 added the `sid` claim for id_tokens issued through the login UI
- The `sid` can now be checked for the `V1_` prefix and queries for
either the userAgentID and depending on the
`OIDCSingleV1SessionTermination` flag all userIDs of active session from
the same user agent id
- The `OIDCSingleV1SessionTermination` flag is added with default value
false to keep the existing behavior of terminating all sessions even in
case of providing an id_token_hint
# Additional Changes
- pass `context.Context` into session view functions for querying the
database with that context
# Additional Context
- relates to #8499
- closes#8501
# Which Problems Are Solved
Added functionality that user with a userschema can be created and
removed.
# How the Problems Are Solved
Added logic and moved APIs so that everything is API v3 conform.
# Additional Changes
- move of user and userschema API to resources folder
- changed testing and parameters
- some renaming
# Additional Context
closes#7308
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
# Which Problems Are Solved
There were UX issue with the autolinking prompt page and users were not
able to link their account or would not understand what to do. Since the
trust to the IdP is already bound by the configuration, the user can
directly be linked without any user input.
# How the Problems Are Solved
- remove the prompt page and directly link the user if possible
- remove corresponding customization texts from the API and Console
# Additional Changes
None
# Additional Context
- relates to https://github.com/zitadel/zitadel/issues/7977
- discussed with customers
- created as a `fix` to be able to backport
---------
Co-authored-by: Max Peintner <max@caos.ch>
# Which Problems Are Solved
The access token verifier returned a permission denied (HTTP 403 / GRPC
7) instead of a unauthenticated (HTTP 401 / GRPC 16) error.
# How the Problems Are Solved
Return the correct error type.
# Additional Changes
None
# Additional Context
close#8392
# Which Problems Are Solved
IDPLinks list and other list endpoints can provide you with empty
results if the used user has no permission for the information.
# How the Problems Are Solved
List endpoints with subelements to users, and provided userIDQuery, will
return a PermissionDenied error if no permission for the user exsists.
# Additional Changes
Function to check for permission is re-used from the GetUserByID.
# Additional Context
Closes#8451
# Which Problems Are Solved
Currently the OIDC API of ZITADEL only prints parent errors to the logs.
Where 4xx status are typically warn level and 5xx error level. This
makes it hard to debug certain errors for client in multi-instance
environments like ZITADEL cloud, where there is no direct access to
logs. In case of support requests we often can't correlate past
log-lines to the error that was reported.
This change adds the possibility to return the parent error in the
response to the OIDC client. For the moment this only applies to JSON
body responses, not error redirects to the RP.
# How the Problems Are Solved
- New instance-level feature flag: `debug_oidc_parent_error`
- Use the new `WithReturnParentToClient()` function from the oidc lib
introduced in https://github.com/zitadel/oidc/pull/629 for all cases
where `WithParent` was already used and the request context is
available.
# Additional Changes
none
# Additional Context
- Depends on: https://github.com/zitadel/oidc/pull/629
- Related to: https://github.com/zitadel/zitadel/issues/8362
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
When Web keys with a large RSA Bitsize were generated, the new web key
did not get projected.
Zitadel logs printed the following errors:
```
time="2024-08-15T09:18:47Z" level=debug msg="execution of statements failed" caller="/home/tim/Repositories/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:518" error="statement failed: ID=CRDB-pKtsr Message=exec failed Parent=(ERROR: index row size 3400 exceeds btree version 4 maximum 2704 for index \"web_keys_web_key_state_idx\" (SQLSTATE 54000))" lastProcessedIndex=-1 projection=projections.web_keys
```
# How the Problems Are Solved
Remove the the `WithInclude` option with the Private key from the
`web_keys` projection index and bump the projection version.
# Additional Changes
- Correct RPC summaries for documentation
- Fix docs slug in sidebar.js
# Additional Context
- Found during release review.
# Which Problems Are Solved
Org v2 service does not have a ListOrganizations endpoint.
# How the Problems Are Solved
Implement ListOrganizations endpoint.
# Additional Changes
- moved descriptions in the protos to comments
- corrected the RemoveNoPermissions for the ListUsers, to get the
correct TotalResults
# Additional Context
For new typescript login
# Which Problems Are Solved
GetIDPByID as endpoint in the API v2 so that it can be available for the
new login.
# How the Problems Are Solved
Create GetIDPByID endpoint with IDP v2 API, throught the GetProviderByID
implementation from admin and management API.
# Additional Changes
- Remove the OwnerType attribute from the response, as the information
is available through the resourceOwner.
- correct refs to messages in proto which are used for doc generation
- renaming of elements for API v3
# Additional Context
Closes#8337
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Implement a new API service that allows management of OIDC signing web
keys.
This allows users to manage rotation of the instance level keys. which
are currently managed based on expiry.
The API accepts the generation of the following key types and
parameters:
- RSA keys with 2048, 3072 or 4096 bit in size and:
- Signing with SHA-256 (RS256)
- Signing with SHA-384 (RS384)
- Signing with SHA-512 (RS512)
- ECDSA keys with
- P256 curve
- P384 curve
- P512 curve
- ED25519 keys
# How the Problems Are Solved
Keys are serialized for storage using the JSON web key format from the
`jose` library. This is the format that will be used by OIDC for
signing, verification and publication.
Each instance can have a number of key pairs. All existing public keys
are meant to be used for token verification and publication the keys
endpoint. Keys can be activated and the active private key is meant to
sign new tokens. There is always exactly 1 active signing key:
1. When the first key for an instance is generated, it is automatically
activated.
2. Activation of the next key automatically deactivates the previously
active key.
3. Keys cannot be manually deactivated from the API
4. Active keys cannot be deleted
# Additional Changes
- Query methods that later will be used by the OIDC package are already
implemented. Preparation for #8031
- Fix indentation in french translation for instance event
- Move user_schema translations to consistent positions in all
translation files
# Additional Context
- Closes#8030
- Part of #7809
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
# Which Problems Are Solved
Log some details when a secret generator is not found.
This should help us debugging such issue.
# How the Problems Are Solved
When a secret generator by type query fails,
we log the generator type and instance id for which
the generator was requested.
# Additional Changes
- none
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/8379
- Also encountered in https://github.com/zitadel/zitadel/pull/8407
# Which Problems Are Solved
The current v3alpha actions APIs don't exactly adhere to the [new
resources API
design](https://zitadel.com/docs/apis/v3#standard-resources).
# How the Problems Are Solved
- **Improved ID access**: The aggregate ID is added to the resource
details object, so accessing resource IDs and constructing proto
messages for resources is easier
- **Explicit Instances**: Optionally, the instance can be explicitly
given in each request
- **Pagination**: A default search limit and a max search limit are
added to the defaults.yaml. They apply to the new v3 APIs (currently
only actions). The search query defaults are changed to ascending by
creation date, because this makes the pagination results the most
deterministic. The creation date is also added to the object details.
The bug with updated creation dates is fixed for executions and targets.
- **Removed Sequences**: Removed Sequence from object details and
ProcessedSequence from search details
# Additional Changes
Object details IDs are checked in unit test only if an empty ID is
expected. Centralizing the details check also makes this internal object
more flexible for future evolutions.
# Additional Context
- Closes#8169
- Depends on https://github.com/zitadel/zitadel/pull/8225
---------
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
ZITADEL currently selects the instance context based on a HTTP header
(see https://github.com/zitadel/zitadel/issues/8279#issue-2399959845 and
checks it against the list of instance domains. Let's call it instance
or API domain.
For any context based URL (e.g. OAuth, OIDC, SAML endpoints, links in
emails, ...) the requested domain (instance domain) will be used. Let's
call it the public domain.
In cases of proxied setups, all exposed domains (public domains) require
the domain to be managed as instance domain.
This can either be done using the "ExternalDomain" in the runtime config
or via system API, which requires a validation through CustomerPortal on
zitadel.cloud.
# How the Problems Are Solved
- Two new headers / header list are added:
- `InstanceHostHeaders`: an ordered list (first sent wins), which will
be used to match the instance.
(For backward compatibility: the `HTTP1HostHeader`, `HTTP2HostHeader`
and `forwarded`, `x-forwarded-for`, `x-forwarded-host` are checked
afterwards as well)
- `PublicHostHeaders`: an ordered list (first sent wins), which will be
used as public host / domain. This will be checked against a list of
trusted domains on the instance.
- The middleware intercepts all requests to the API and passes a
`DomainCtx` object with the hosts and protocol into the context
(previously only a computed `origin` was passed)
- HTTP / GRPC server do not longer try to match the headers to instances
themself, but use the passed `http.DomainContext` in their interceptors.
- The `RequestedHost` and `RequestedDomain` from authz.Instance are
removed in favor of the `http.DomainContext`
- When authenticating to or signing out from Console UI, the current
`http.DomainContext(ctx).Origin` (already checked by instance
interceptor for validity) is used to compute and dynamically add a
`redirect_uri` and `post_logout_redirect_uri`.
- Gateway passes all configured host headers (previously only did
`x-zitadel-*`)
- Admin API allows to manage trusted domain
# Additional Changes
None
# Additional Context
- part of #8279
- open topics:
- "single-instance" mode
- Console UI
# Which Problems Are Solved
The v2beta services are stable but not GA.
# How the Problems Are Solved
The v2beta services are copied to v2. The corresponding v1 and v2beta
services are deprecated.
# Additional Context
Closes#7236
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
# Which Problems Are Solved
#8291 added backwards compatibilty for users who were created through
the user V2 API and want to sign in to the login UI.
There were however to issues, where users might be prompted to set a
password even if they already had one set or they would not be able to
submit the email verification code.
# How the Problems Are Solved
- Replaced `SearchUserAuthMethods `with `ListUserAuthMethodTypes` to
check for set up auth methods.
- Fixed page / javascript to disable submit button.
# Additional Changes
- Changed `ListActiveUserAuthMethodTypes ` to `ListUserAuthMethodTypes`
and a `activeOnly` boolean parameter
# Additional Context
- relates to #8291
- noticed internally on QA
# Which Problems Are Solved
The connection pool of go uses a high amount of database connections.
# How the Problems Are Solved
The standard lib connection pool was replaced by `pgxpool.Pool`
# Additional Changes
The `db.BeginTx`-spans are removed because they cause to much noise in
the traces.
# Additional Context
- part of https://github.com/zitadel/zitadel/issues/7639
# Which Problems Are Solved
ListOrgs has no option to select for organizations specific to Ids.
# How the Problems Are Solved
Add OrgIDQuery to ListOrgs.
# Additional Changes
Clean up double mapping for the OrgQueries.
# Additional Context
- noted internally while checking performance issues (in Console)
# Which Problems Are Solved
Improve the performance of human imports by optimizing the query that
finds domains claimed by other organizations.
# How the Problems Are Solved
Use the fields search table introduced in
https://github.com/zitadel/zitadel/pull/8191 by storing each
organization domain as Object ID and the verified status as field value.
# Additional Changes
- Feature flag for this optimization
# Additional Context
- Performance improvements for import are evaluated and acted upon
internally at the moment
---------
Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
# Which Problems Are Solved
The client ID for OIDC applications has an `@` in it, which is not
allowed in some 3rd-party systems (such as AWS).
# How the Problems Are Solved
Per @fforootd and @hifabienne in #6222, remove the project suffix and
the `@` from the client ID and just use the generated ID.
# Additional Changes
N/A
# Additional Context
- Closes#6222
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
To improve performance a new table and method is implemented on
eventstore. The goal of this table is to index searchable fields on
command side to use it on command and query side.
The table allows to store one primitive value (numeric, text) per row.
The eventstore framework is extended by the `Search`-method which allows
to search for objects.
The `Command`-interface is extended by the `SearchOperations()`-method
which does manipulate the the `search`-table.
# How the Problems Are Solved
This PR adds the capability of improving performance for command and
query side by using the `Search`-method of the eventstore instead of
using one of the `Filter`-methods.
# Open Tasks
- [x] Add feature flag
- [x] Unit tests
- [ ] ~~Benchmarks if needed~~
- [x] Ensure no behavior change
- [x] Add setup step to fill table with current data
- [x] Add projection which ensures data added between setup and start of
the new version are also added to the table
# Additional Changes
The `Search`-method is currently used by `ProjectGrant`-command side.
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/8094
In issue #7841 @mahmoodfathy commented an issue when the API call for
Listing My ZITADEL Manager Roles is called with any kind of query
(orgQuery, projectQuery, projectGrantQuery...). A column XXXXXX does not
exist (SQLSTATE 42703) error is thrown.
The issue was focused in getMembershipFromQuery where filtering queries
functions are called: prepareOrgMember, prepareIAMMember,
prepareProjectMember and prepareProjectGrantMember
Those functions allow queries for columns that are not members of the
table to be queried so I've added a conditional clause to avoid using
the queries that cannot be called.
For example, for prepareOrgMember, member.id, member.project_id and
member.grant_id columns are not added to the filter queries
```
for _, q := range query.Queries {
if q.Col().table.name == membershipAlias.name &&
!slices.Contains([]string{membershipIAMID.name, membershipProjectID.name, membershipGrantID.name}, q.Col().name) {
builder = q.toQuery(builder)
}
}
return builder.MustSql()
```
Here I show one screenshot where the error "column XXXXXX does not exist
(SQLSTATE 42703)" is no longer thrown using an orgQuery.
Should close#7841
### Definition of Ready
- [X] I am happy with the code
- [X] Short description of the feature/issue is added in the pr
description
- [X] PR is linked to the corresponding user story
- [X] Acceptance criteria are met
- [ ] All open todos and follow ups are defined in a new ticket and
justified
- [ ] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [X] No debug or dead code
- [X] My code has no repetitions
- [X] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [ ] Documentation/examples are up-to-date
- [ ] All non-functional requirements are met
- [X] Functionality of the acceptance criteria is checked manually on
the dev system.
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
This fix adds tracing spans to all V1 API import related functions. This
is to troubleshoot import related performance issues reported to us.
# How the Problems Are Solved
Add a tracing span to `api/grpc/admin/import.go` and all related
functions that are called in the `command` package.
# Additional Changes
- none
# Additional Context
- Reported by internal communication
# Which Problems Are Solved
Some organizations / customers have the requirement, that there users
regularly need to change their password.
ZITADEL already had the possibility to manage a `password age policy` (
thought the API) with the maximum amount of days a password should be
valid, resp. days after with the user should be warned of the upcoming
expiration.
The policy could not be managed though the Console UI and was not
checked in the Login UI.
# How the Problems Are Solved
- The policy can be managed in the Console UI's settings sections on an
instance and organization level.
- During an authentication in the Login UI, if a policy is set with an
expiry (>0) and the user's last password change exceeds the amount of
days set, the user will be prompted to change their password.
- The prompt message of the Login UI can be customized in the Custom
Login Texts though the Console and API on the instance and each
organization.
- The information when the user last changed their password is returned
in the Auth, Management and User V2 API.
- The policy can be retrieved in the settings service as `password
expiry settings`.
# Additional Changes
None.
# Additional Context
- closes#8081
---------
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
# Which Problems Are Solved
An admin / application might want to be able to reduce the amount of
roles returned in the token, for example if a user is granted to many
organizations or for specific cases where the application want to narrow
down the access for that token to a specific organization or multiple.
This can now be achieved by providing a scope with the id of the
organization, resp. multiple scopes for every organization, which should
be included.
```
urn:zitadel:iam:org:roles🆔{orgID}
```
**Note:** the new scope does not work when Introspection / Userinfo are
set to legacy mode.
# How the Problems Are Solved
The user info query now has two variants:
1. Variant that returns all organization authorization grants if the new
scope wasn't provided for backward compatibility.
2. Variant that filters the organizations based on the IDs passed in one
or more of the above scopes and returns only those authorization grants.
The query is defined as a `text/template` and both variants are rendered
once in package `init()`.
# Additional Changes
- In the integration tests `assertProjectRoleClaims` now also checks the
org IDs in the roles.
# Additional Context
- Closes#7996
# Which Problems Are Solved
Request to the ZITADEL API currently require multi factor authentication
if the user has set up any second factor.
However, the login UI will only prompt the user to check factors that
are allowed by the login policy.
This can lead to situations, where the user has set up a factor (e.g.
some OTP) which was not allowed by the policy, therefore will not have
to verify the factor, the ZITADEL API however will require the check
since the user has set it up.
# How the Problems Are Solved
The requirement for multi factor authentication based on the user's
authentication methods is removed when accessing the ZITADEL APIs.
Those requests will only require MFA in case the login policy does so
because of `requireMFA` or `requireMFAForLocalUsers`.
# Additional Changes
None.
# Additional Context
- a customer reached out to support
- discussed internally
- relates #7822
- backport to 2.53.x
# Which Problems Are Solved
Access token checks make sure that there have not been any termination
events (user locked, deactivated, signed out, ...) in the meantime. This
events were filtered based on the creation date of the last session
event, which might cause latency issues in the database.
# How the Problems Are Solved
- Changed the query to use `position` instead of `created_at`.
- removed `AwaitOpenTransactions`
# Additional Changes
Added the `position` field to the `ReadModel`.
# Additional Context
- relates to #8088
- part of #7639
- backport to 2.53.x
# Which Problems Are Solved
After migrating the access token events in #7822, milestones based on
authentication, resp. theses events would not be reached.
# How the Problems Are Solved
Additionally use the `oidc_session.Added` event to check for
`milestone.AuthenticationSucceededOnInstance` and
`milestone.AuthenticationSucceededOnApplication`.
# Additional Changes
None.
# Additional Context
- relates to #7822
- noticed internally
# Which Problems Are Solved
Adds the possibility to mirror an existing database to a new one.
For that a new command was added `zitadel mirror`. Including it's
subcommands for a more fine grained mirror of the data.
Sub commands:
* `zitadel mirror eventstore`: copies only events and their unique
constraints
* `zitadel mirror system`: mirrors the data of the `system`-schema
* `zitadel mirror projections`: runs all projections
* `zitadel mirror auth`: copies auth requests
* `zitadel mirror verify`: counts the amount of rows in the source and
destination database and prints the diff.
The command requires one of the following flags:
* `--system`: copies all instances of the system
* `--instance <instance-id>`, `--instance <comma separated list of
instance ids>`: copies only the defined instances
The command is save to execute multiple times by adding the
`--replace`-flag. This replaces currently existing data except of the
`events`-table
# Additional Changes
A `--for-mirror`-flag was added to `zitadel setup` to prepare the new
database. The flag skips the creation of the first instances and initial
run of projections.
It is now possible to skip the creation of the first instance during
setup by setting `FirstInstance.Skip` to true in the steps
configuration.
# Additional info
It is currently not possible to merge multiple databases. See
https://github.com/zitadel/zitadel/issues/7964 for more details.
It is currently not possible to use files. See
https://github.com/zitadel/zitadel/issues/7966 for more information.
closes https://github.com/zitadel/zitadel/issues/7586
closes https://github.com/zitadel/zitadel/issues/7486
### Definition of Ready
- [x] I am happy with the code
- [x] Short description of the feature/issue is added in the pr
description
- [x] PR is linked to the corresponding user story
- [x] Acceptance criteria are met
- [x] All open todos and follow ups are defined in a new ticket and
justified
- [x] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [x] No debug or dead code
- [x] My code has no repetitions
- [x] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [x] Documentation/examples are up-to-date
- [x] All non-functional requirements are met
- [x] Functionality of the acceptance criteria is checked manually on
the dev system.
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
As already mentioned and (partially) fixed in #7992 we discovered,
issues with v2 tokens that where obtained through an IDP, with
passwordless authentication or with password authentication (wihtout any
2FA set up) using the v1 login for zitadel API calls
- (Previous) authentication through an IdP is now correctly treated as
auth method in case of a reauth even when the user is not redirected to
the IdP
- There were some cases where passwordless authentication was
successfully checked but not correctly set as auth method, which denied
access to ZITADEL API
- Users with password and passwordless, but no 2FA set up which
authenticate just wich password can access the ZITADEL API again
Additionally while testing we found out that because of #7969 the login
UI could completely break / block with the following error:
`sql: Scan error on column index 3, name "state": converting NULL to
int32 is unsupported (Internal)`
# How the Problems Are Solved
- IdP checks are treated the same way as other factors and it's ensured
that a succeeded check within the configured timeframe will always
provide the idp auth method
- `MFATypesAllowed` checks for possible passwordless authentication
- As with the v1 login, the token check now only requires MFA if the
policy is set or the user has 2FA set up
- UserAuthMethodsRequirements now always uses the correctly policy to
check for MFA enforcement
- `State` column is handled as nullable and additional events set the
state to active (as before #7969)
# Additional Changes
- Console now also checks for 403 (mfa required) errors (e.g. after
setting up the first 2FA in console) and redirects the user to the login
UI (with the current id_token as id_token_hint)
- Possible duplicates in auth methods / AMRs are removed now as well.
# Additional Context
- Bugs were introduced in #7822 and # and 7969 and only part of a
pre-release.
- partially already fixed with #7992
- Reported internally.
