# Which Problems Are Solved
There were multiple issues in the OpenTelemetry (OTEL) implementation
and usage for tracing and metrics, which lead to high cardinality and
potential memory leaks:
- wrongly initiated tracing interceptors
- high cardinality in traces:
- HTTP/1.1 endpoints containing host names
- HTTP/1.1 endpoints containing object IDs like userID (e.g.
`/management/v1/users/2352839823/`)
- high amount of traces from internal processes (spooler)
- high cardinality in metrics endpoint:
- GRPC entries containing host names
- notification metrics containing instanceIDs and error messages
# How the Problems Are Solved
- Properly initialize the interceptors once and update them to use the
grpc stats handler (unary interceptors were deprecated).
- Remove host names from HTTP/1.1 span names and use path as default.
- Set / overwrite the uri for spans on the grpc-gateway with the uri
pattern (`/management/v1/users/{user_id}`). This is used for spans in
traces and metric entries.
- Created a new sampler which will only sample spans in the following
cases:
- remote was already sampled
- remote was not sampled, root span is of kind `Server` and based on
fraction set in the runtime configuration
- This will prevent having a lot of spans from the spooler back ground
jobs if they were not started by a client call querying an object (e.g.
UserByID).
- Filter out host names and alike from OTEL generated metrics (using a
`view`).
- Removed instance and error messages from notification metrics.
# Additional Changes
Fixed the middleware handling for serving Console. Telemetry and
instance selection are only used for the environment.json, but not on
statically served files.
# Additional Context
- closes#8096
- relates to #9074
- back ports to at least 2.66.x, 2.67.x and 2.68.x
(cherry picked from commit 990e1982c712ba2082f3fc6fc4861f3abf85b0cd)
# Which Problems Are Solved
- when a scim user is provisioned, a init email could be sent
# How the Problems Are Solved
- no init email should be sent => hard code false for the email init
param
# Additional Context
Related to https://github.com/zitadel/zitadel/issues/8140
Co-authored-by: Fabienne Bühler <fabienne@zitadel.com>
# Which Problems Are Solved
- Some SCIM clients send "op" of a patch operation in PascalCase
# How the Problems Are Solved
- Well known "op" values of patch operations are matched
case-insensitive.
# Additional Context
Related to #8140
# Which Problems Are Solved
- If a SCIM endpoint is called with an orgID in the URL that is not the
resource owner, no error is returned, and the action is executed.
# How the Problems Are Solved
- The orgID provided in the SCIM URL path must match the resource owner
of the target user. Otherwise, an error will be returned.
# Additional Context
Part of https://github.com/zitadel/zitadel/issues/8140
# Which Problems Are Solved
- Lack of documentation for the SCIM v2 interface
# How the Problems Are Solved
- Introduced a new documentation page detailing the SCIM v2 interface
# Additional Context
Part of #8140
---------
Co-authored-by: Fabienne Bühler <fabienne@zitadel.com>
# Which Problems Are Solved
- Adds a service in the console to enable Posthog integration based on
upon user environment variables
# How the Problems Are Solved
- A new service has been created in console for posthog
- This is only initiated based upon provided environment variables
# Additional Changes
N/A
# Additional Context
- Closes #[9076](https://github.com/zitadel/zitadel/issues/9076)
- Cannot be merged until this is completed
#[9070](https://github.com/zitadel/zitadel/issues/9070)
# Which Problems Are Solved
* Adds support for the service provider configuration SCIM v2 endpoints
# How the Problems Are Solved
* Adds support for the service provider configuration SCIM v2 endpoints
* `GET /scim/v2/{orgId}/ServiceProviderConfig`
* `GET /scim/v2/{orgId}/ResourceTypes`
* `GET /scim/v2/{orgId}/ResourceTypes/{name}`
* `GET /scim/v2/{orgId}/Schemas`
* `GET /scim/v2/{orgId}/Schemas/{id}`
# Additional Context
Part of #8140
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
Add the ability to update the timestamp when MFA initialization was last
skipped.
Get User By ID now also returns the timestamps when MFA setup was last
skipped.
# How the Problems Are Solved
- Add a `HumanMFAInitSkipped` method to the `users/v2` API.
- MFA skipped was already projected in the `auth.users3` table. In this
PR the same column is added to the users projection. Event handling is
kept the same as in the `UserView`:
<details>
62804ca45f/internal/user/repository/view/model/user.go (L243-L377)
</details>
# Additional Changes
- none
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/9197
# Which Problems Are Solved
* Adds support for the bulk SCIM v2 endpoint
# How the Problems Are Solved
* Adds support for the bulk SCIM v2 endpoint under `POST
/scim/v2/{orgID}/Bulk`
# Additional Context
Part of #8140
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
- when listing users via scim v2.0 filters applied to the username are
applied case-sensitive
# How the Problems Are Solved
- when a query filter is appleid on the username it is applied
case-insensitive
# Additional Context
Part of https://github.com/zitadel/zitadel/issues/8140
# Which Problems Are Solved
SAML session implemented, but no how-to comparable to the OIDC sessions
for custom login available.
# How the Problems Are Solved
Added documentation, which should be also comparable with the OIDC
session for ease-of-use.
# Additional Changes
Added generated SAML API docs.
# Additional Context
Closes#9088
Follow-up issue #9267
---------
Co-authored-by: Fabienne Bühler <fabienne@zitadel.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Adds `failed attempts` field to the grpc response when a user enters
wrong password when logging in
FYI:
this only covers the senario above; other senarios where this is not
applied are:
SetPasswordWithVerifyCode
setPassword
ChangPassword
setPasswordWithPermission
# How the Problems Are Solved
Created new grpc message `CredentialsCheckError` -
`proto/zitadel/message.proto` to include `failed_attempts` field.
Had to create a new package -
`github.com/zitadel/zitadel/internal/command/errors` to resolve cycle
dependency between `github.com/zitadel/zitadel/internal/command` and
`github.com/zitadel/zitadel/internal/command`.
# Additional Changes
- none
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/9198
---------
Co-authored-by: Iraq Jaber <IraqJaber@gmail.com>
# Which Problems Are Solved
- scim v2 only maps the primary phone/email to the zitadel user, this
does not work if no primary is set
# How the Problems Are Solved
- the first phone / email is mapped if no primary is available
# Additional Context
Part of #8140
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
Paths for setup steps are joined with "\" when binary is started under
Windows, which results in wrongly joined paths.
# How the Problems Are Solved
Replace the usage of "filepath" with "path" package, which does only
join with "/" and nothing OS specific.
# Additional Changes
None
# Additional Context
Closes#9227
# Which Problems Are Solved
#9185 changed that if a notification channel was not present,
notification workers would no longer retry to send the notification and
would also cancel in case Twilio would return a 4xx error.
However, this would not affect the "legacy" mode.
# How the Problems Are Solved
- Handle `CancelError` in legacy notifier as not failed (event).
# Additional Changes
None
# Additional Context
- relates to #9185
- requires back port to 2.66.x and 2.67.x
(cherry picked from commit 3fc68e5d60bb105c2ac72ca3d9cac7327f240b9b)
# Which Problems Are Solved
#9185 changed that if a notification channel was not present,
notification workers would no longer retry to send the notification and
would also cancel in case Twilio would return a 4xx error.
However, this would not affect the "legacy" mode.
# How the Problems Are Solved
- Handle `CancelError` in legacy notifier as not failed (event).
# Additional Changes
None
# Additional Context
- relates to #9185
- requires back port to 2.66.x and 2.67.x
# Which Problems Are Solved
- scim list users endpoint (`GET /scim/v2/{orgId}/Users`): handle
unsupported `SortBy` columns correctly
# How the Problems Are Solved
- throw an error if sorting by an unsupported column is requested
# Additional Context
Part of #8140
# Which Problems Are Solved
- requests to the scim interface with content type `*/*` are rejected
# How the Problems Are Solved
- `*/*` is accepted as content type
# Additional Context
Part of #8140
# Which Problems Are Solved
- SCIM user metadata mapping keys have differing case styles.
# How the Problems Are Solved
- key casing style is unified to strict camelCase
# Additional Context
Part of #8140
Although this is technically a breaking change, it is considered
acceptable because the SCIM feature is still in the preview stage and
not fully implemented yet.
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
* Adds support for the patch user SCIM v2 endpoint
# How the Problems Are Solved
* Adds support for the patch user SCIM v2 endpoint under `PATCH
/scim/v2/{orgID}/Users/{id}`
# Additional Context
Part of #8140
# Which Problems Are Solved
The membership fields migration timed out in certain cases. It also
tried to migrate instances which were already removed.
# How the Problems Are Solved
Revert the previous fix that combined the repeatable step for multiple
fill triggers. The membeship migration is now single-run as it might
take a lot of time. It is not worth making it repeatable. Instance IDs
of removed instances are skipped.
# Additional Changes
None
# Additional Context
Introduced in https://github.com/zitadel/zitadel/pull/9199
(cherry picked from commit ec5f18c16899627bdd8ecf90844641a9bdd8ca47)
# Which Problems Are Solved
A customer reached out to support, that the (last) `additional origin`
could not be removed. While testing / implementation it was discovered,
that the same applied to `redirect_uris` and `post_logout_redirect_uris`
# How the Problems Are Solved
- Correctly set the corresponding array to empty in the event so it can
be differentiated to `null` / not set in case of no change.
# Additional Changes
Replaced `reflect.DeepEqual` with `slices.Equal`
# Additional Context
- Reported to support
(cherry picked from commit c9aa5db2a55ef3c4f6fb62292b69b01973b264e7)
# Which Problems Are Solved
The membership fields migration timed out in certain cases. It also
tried to migrate instances which were already removed.
# How the Problems Are Solved
Revert the previous fix that combined the repeatable step for multiple
fill triggers. The membeship migration is now single-run as it might
take a lot of time. It is not worth making it repeatable. Instance IDs
of removed instances are skipped.
# Additional Changes
None
# Additional Context
Introduced in https://github.com/zitadel/zitadel/pull/9199
# Which Problems Are Solved
The subquery of the notification requested and retry requested is
missing the aggregate_type filter that would allow it to utilize the
`es_projection` or `active_instances_events` on the eventstore.events2
table.
# How the Problems Are Solved
Add additional filter on subquery. Final query:
```sql
SELECT <all the fields omitted> FROM eventstore.events2
WHERE
instance_id = $1
AND aggregate_type = $2
AND event_type = $3
AND created_at > $4
AND aggregate_id NOT IN (
SELECT aggregate_id
FROM eventstore.events2
WHERE
aggregate_type = $5 <-- NB: previously missing
AND event_type = ANY ($6)
AND instance_id = $7
AND created_at > $8
)
ORDER BY "position", in_tx_order
LIMIT $9
FOR UPDATE SKIP LOCKED
```
# Additional Changes
# Additional Context
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
A customer reached out to support, that the (last) `additional origin`
could not be removed. While testing / implementation it was discovered,
that the same applied to `redirect_uris` and `post_logout_redirect_uris`
# How the Problems Are Solved
- Correctly set the corresponding array to empty in the event so it can
be differentiated to `null` / not set in case of no change.
# Additional Changes
Replaced `reflect.DeepEqual` with `slices.Equal`
# Additional Context
- Reported to support
# Which Problems Are Solved
- Adds support for the list users SCIM v2 endpoint
# How the Problems Are Solved
- Adds support for the list users SCIM v2 endpoints under `GET
/scim/v2/{orgID}/Users` and `POST /scim/v2/{orgID}/Users/.search`
# Additional Changes
- adds a new function `SearchUserMetadataForUsers` to the query layer to
query a metadata keyset for given user ids
- adds a new function `NewUserMetadataExistsQuery` to the query layer to
query a given metadata key value pair exists
- adds a new function `CountUsers` to the query layer to count users
without reading any rows
- handle `ErrorAlreadyExists` as scim errors `uniqueness`
- adds `NumberLessOrEqual` and `NumberGreaterOrEqual` query comparison
methods
- adds `BytesQuery` with `BytesEquals` and `BytesNotEquals` query
comparison methods
# Additional Context
Part of #8140
Supported fields for scim filters:
* `meta.created`
* `meta.lastModified`
* `id`
* `username`
* `name.familyName`
* `name.givenName`
* `emails` and `emails.value`
* `active` only eq and ne
* `externalId` only eq and ne
# Which Problems Are Solved
Small update to docs/docs/concepts/features/selfservice.md to fix issue
in grammar
Co-authored-by: Iraq Jaber <IraqJaber@gmail.com>
# Which Problems Are Solved
Memberships did not have a fields table fill migration.
# How the Problems Are Solved
Add filling of membership fields to the repeatable steps.
# Additional Changes
- Use the same repeatable step for multiple fill fields handlers.
- Fix an error for PostgreSQL 15 where a subquery in a `FROM` clause
needs an alias ing the `permitted_orgs` function.
# Additional Context
- Part of https://github.com/zitadel/zitadel/issues/9188
- Introduced in https://github.com/zitadel/zitadel/pull/9152
# Which Problems Are Solved
https://github.com/zitadel/zitadel/pull/9186 introduced the new `push`
sql function for cockroachdb. The function used the wrong database
function to generate the position of the event and would therefore
insert events at a position before events created with an old Zitadel
version.
# How the Problems Are Solved
Instead of `EXTRACT(EPOCH FROM NOW())`, `cluster_logical_timestamp()` is
used to calculate the position of an event.
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/9186
- Affected versions:
https://github.com/zitadel/zitadel/releases/tag/v2.67.3
# Which Problems Are Solved
https://github.com/zitadel/zitadel/pull/9186 introduced the new `push`
sql function for cockroachdb. The function used the wrong database
function to generate the position of the event and would therefore
insert events at a position before events created with an old Zitadel
version.
# How the Problems Are Solved
Instead of `EXTRACT(EPOCH FROM NOW())`, `cluster_logical_timestamp()` is
used to calculate the position of an event.
# Additional Context
- Introduced in https://github.com/zitadel/zitadel/pull/9186
- Affected versions:
https://github.com/zitadel/zitadel/releases/tag/v2.67.3
# Which Problems Are Solved
If a notification channel was not present, notification workers would
retry to the max attempts. This leads to unnecessary load.
Additionally, a client noticed bad actors trying to abuse SMS MFA.
# How the Problems Are Solved
- Directly cancel the notification on:
- a missing channel and stop retries.
- any `4xx` errors from Twilio Verify
# Additional Changes
None
# Additional Context
reported by customer
(cherry picked from commit 60857c8d3e0108437a8a8eb5cc4efae5e440af85)
# Which Problems Are Solved
The performance of the initial push function can further be increased
# How the Problems Are Solved
`eventstore.push`- and `eventstore.commands_to_events`-functions were
rewritten
# Additional Changes
none
# Additional Context
same optimizations as for postgres:
https://github.com/zitadel/zitadel/pull/9092
(cherry picked from commit 690147b30e51322ead9fd1c099141aac17b7d6bc)
# Which Problems Are Solved
If a notification channel was not present, notification workers would
retry to the max attempts. This leads to unnecessary load.
Additionally, a client noticed bad actors trying to abuse SMS MFA.
# How the Problems Are Solved
- Directly cancel the notification on:
- a missing channel and stop retries.
- any `4xx` errors from Twilio Verify
# Additional Changes
None
# Additional Context
reported by customer
# Which Problems Are Solved
Organization name change results in domain events even if the domain
itself doesn't change.
# How the Problems Are Solved
Check if the domain itself really changes, and if not, don't create the
events.
# Additional Changes
Unittest for this specific case.
# Additional Context
None
(cherry picked from commit 69372e52091634d6654130fd99fabdd9d8f0b61d)
# Which Problems Are Solved
Events like "password check succeeded" store some information about the
caller including their IP.
The `X-Forwarded-For` was not correctly logged, but instead the
RemoteAddress.
# How the Problems Are Solved
- Correctly get the `X-Forwarded-For` in canonical form.
# Additional Changes
None
# Additional Context
closes [#9106](https://github.com/zitadel/zitadel/issues/9106)
(cherry picked from commit c966446f803aacfc03fbc0c152e11dbe34e9d64e)
# Which Problems Are Solved
It was possible to set a diffent algorithm for the legacy signer. This
is not supported howerver and breaks the token endpoint.
# How the Problems Are Solved
Remove the OIDC.SigningKeyAlgorithm config option and hard-code RS256
for the legacy signer.
# Additional Changes
- none
# Additional Context
Only RS256 is supported by the legacy signer. It was mentioned in the
comment of the config not to use it and use the webkeys resource
instead.
- closes#9121
(cherry picked from commit db8d794794eba191d1b1f3a79ea5b4ec2c90a821)
# Which Problems Are Solved
Typo in RU localization on login page.
# How the Problems Are Solved
Fixed typo by replacing to correct text.
# Additional Changes
n/a
# Additional Context
n/a
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
(cherry picked from commit 42cc6dce79bb0659956d358d142a5ecf2cac59f3)
# Which Problems Are Solved
If a not allowed IDP is selected or now not allowed IDP was selected
before at login, the login will still try to use it as fallback.
The same goes for the linked IDPs which are not necessarily active
anymore, or disallowed through policies.
# How the Problems Are Solved
Check all possible or configured IDPs if they can be used.
# Additional Changes
None
# Additional Context
Addition to #6466
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit 8d8f38fb4ca6e767f266ceffa07244c9c55448c2)
# Which Problems Are Solved
Organization name change results in domain events even if the domain
itself doesn't change.
# How the Problems Are Solved
Check if the domain itself really changes, and if not, don't create the
events.
# Additional Changes
Unittest for this specific case.
# Additional Context
None
# Which Problems Are Solved
Zitadel currently uses 3 database pool, 1 for queries, 1 for pushing
events and 1 for scheduled projection updates. This defeats the purpose
of a connection pool which already handles multiple connections.
During load tests we found that the current structure of connection
pools consumes a lot of database resources. The resource usage dropped
after we reduced the amount of database pools to 1 because existing
connections can be used more efficiently.
# How the Problems Are Solved
Removed logic to handle multiple connection pools and use a single one.
# Additional Changes
none
# Additional Context
part of https://github.com/zitadel/zitadel/issues/8352
# Which Problems Are Solved
- include tzdata in the binary to correctly validate time zones in the
scim layer if the os doesn't have timezone data available.
# How the Problems Are Solved
- by importing the go pkg `"time/tzdata"`
# Additional Context
Part of https://github.com/zitadel/zitadel/issues/8140
# Which Problems Are Solved
Currently ZITADEL defines organization and instance member roles and
permissions in defaults.yaml. The permission check is done on API call
level. For example: "is this user allowed to make this call on this
org". This makes sense on the V1 API where the API is permission-level
shaped. For example, a search for users always happens in the context of
the organization. (Either the organization the calling user belongs to,
or through member ship and the x-zitadel-orgid header.
However, for resource based APIs we must be able to resolve permissions
by object. For example, an IAM_OWNER listing users should be able to get
all users in an instance based on the query filters. Alternatively a
user may have user.read permissions on one or more orgs. They should be
able to read just those users.
# How the Problems Are Solved
## Role permission mapping
The role permission mappings defined from `defaults.yaml` or local
config override are synchronized to the database on every run of
`zitadel setup`:
- A single query per **aggregate** builds a list of `add` and `remove`
actions needed to reach the desired state or role permission mappings
from the config.
- The required events based on the actions are pushed to the event
store.
- Events define search fields so that permission checking can use the
indices and is strongly consistent for both query and command sides.
The migration is split in the following aggregates:
- System aggregate for for roles prefixed with `SYSTEM`
- Each instance for roles not prefixed with `SYSTEM`. This is in
anticipation of instance level management over the API.
## Membership
Current instance / org / project membership events now have field table
definitions. Like the role permissions this ensures strong consistency
while still being able to use the indices of the fields table. A
migration is provided to fill the membership fields.
## Permission check
I aimed keeping the mental overhead to the developer to a minimal. The
provided implementation only provides a permission check for list
queries for org level resources, for example users. In the `query`
package there is a simple helper function `wherePermittedOrgs` which
makes sure the underlying database function is called as part of the
`SELECT` query and the permitted organizations are part of the `WHERE`
clause. This makes sure results from non-permitted organizations are
omitted. Under the hood:
- A Pg/PlSQL function searches for a list of organization IDs the passed
user has the passed permission.
- When the user has the permission on instance level, it returns early
with all organizations.
- The functions uses a number of views. The views help mapping the
fields entries into relational data and simplify the code use for the
function. The views provide some pre-filters which allow proper index
usage once the final `WHERE` clauses are set by the function.
# Additional Changes
# Additional Context
Closes#9032
Closes https://github.com/zitadel/zitadel/issues/9014https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for
the new permission framework based on this concept.
# Which Problems Are Solved
The performance of the initial push function can further be increased
# How the Problems Are Solved
`eventstore.push`- and `eventstore.commands_to_events`-functions were
rewritten
# Additional Changes
none
# Additional Context
same optimizations as for postgres:
https://github.com/zitadel/zitadel/pull/9092
# Which Problems Are Solved
As we are going into the Beta testing phase of our new typescript login
(login V2), we need to have a documentation about the capabilities, how
to test, and what the current limitations are.
# How the Problems Are Solved
Added new section for the Login V2
---------
Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Max Peintner <peintnerm@gmail.com>
# Which Problems Are Solved
The current login will always prefer external authentication (through an
IdP) over local authentication. So as soon as either the user had
connected to an IdP or even when the login policy was just set up to
have an IdP allowed, users would be redirected to that IdP for
(re)authentication.
This could lead to problems, where the IdP was not available or any
other error occurred in the process (such as secret expired for
EntraID).
Even when local authentication (passkeys or password) was allowed for
the corresponding user, they would always be redirected to the IdP
again, preventing any authentication. If admins were affected, they
might not even be able to update the client secret of the IdP.
# How the Problems Are Solved
Errors during the external IdP flow are handled in an
`externalAuthFailed` function, which will check if the organisation
allows local authentication and if the user has set up such.
If either password or passkeys is set up, the corresponding login page
will be presented to the user. As already with local auth passkeys is
preferred over password authentication.
The user is informed that the external login failed and fail back to
local auth as an error on the corresponding page in a focused mode. Any
interaction or after 5 second the focus mode is disabled.
# Additional Changes
None.
# Additional Context
closes#6466
