1778 Commits

Author SHA1 Message Date
Stefan Benz
93466055ee
test: add sink functionality for idp intents (#9116)
# Which Problems Are Solved

New integration tests can't use command side to simulate successful
intents.

# How the Problems Are Solved

Add endpoints to only in integration tests available sink to create
already successful intents.

# Additional Changes

None

# Additional Context

Closes #8557

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2025-02-20 13:27:20 +01:00
Kenta Yamaguchi
9b35b98cae
fix(i18n): add some missing elements in Japanese (#9353)
# Which Problems Are Solved

Some i18n elements are not translated in Japanese yet.

# How the Problems Are Solved

Add some missing i18n elements to `console/src/assets/i18n/ja.json`,
`internal/api/ui/login/static/i18n/ja.yaml`, and
`internal/static/i18n/ja.yaml`.

More details are following:
- `console/src/assets/i18n/ja.json`
  - `POLICY.PRIVATELABELING.BACKGROUNDCOLOR`
  - `POLICY.PRIVATELABELING.PRIMARYCOLOR`
  - `POLICY.PRIVATELABELING.WARNCOLOR`
  - `POLICY.PRIVATELABELING.FONTCOLOR`
  - `POLICY.LOGIN_TEXTS.MESSAGE_TEXTS.TYPES.IU`
  - `IDP.CREATE.APPLE.TITLE`
  - `IDP.CREATE.APPLE.DESCRIPTION`
  - `IDP.CREATE.SAML.TITLE`
  - `IDP.CREATE.SAML.DESCRIPTION`
  - `IDP.APPLE.TEAMID`
  - `IDP.APPLE.KEYID`
  - `IDP.APPLE.PRIVATEKEY`
  - `IDP.APPLE.UPDATEPRIVATEKEY`
  - `IDP.APPLE.UPLOADPRIVATEKEY`
  - `IDP.KEYMAXSIZEEXCEEDED`
  - `IDP.SAML.METADATAXML`
  - `IDP.SAML.METADATAURL`
  - `IDP.SAML.BIDNING`
  - `IDP.SAML.SIGNEDREQUEST`
  - `IDP.SAML.NAMEIDFORMAT`
  - `IDP.SAML.TRANSIENTMAPPINGATTRIBUTENAME`
  - `IDP.SAML.TRANSIENTMAPPINGATTRIBUTENAME_DESC`
  - `SMTP.LIST.DIALOG.TEST_TITLE`
  - `SMTP.LIST.DIALOG.TEST_DESCRIPTION`
  - `SMTP.LIST.DIALOG.TEST_EMAIL`
  - `SMTP.LIST.DIALOG.TEST_RESULT`
- `internal/api/ui/login/static/i18n/ja.yaml`
  - `LDAP.Title`
  - `LDAP.Description`
  - `LDAP.LoginNameLabel`
  - `LDAP.PasswordLabel`
  - `LDAP.NextButtonText`
  - `PasswordChange.Footer`
  - `Footer.SupportEmail`
  - `Errors.User.AlreadyExists`
  - `Errors.User.Profile.NotFound`
  - `Errors.User.Profile.NotChanged`
  - `Errors.User.Profile.Empty`
  - `Errors.User.Profile.FirstNameEmpty`
  - `Errors.User.Profile.LastNameEmpty`
  - `Errors.User.Profile.IDMissing`
  - `Errors.User.Email.NotFound`
  - `Errors.User.Email.Invalid`
  - `Errors.User.Email.AlreadyVerified`
  - `Errors.User.Email.NotChanged`
  - `Errors.User.Email.Empty`
  - `Errors.User.Email.IDMissing`
  - `Errors.User.Phone.NotFound`
  - `Errors.User.Phone.Invalid`
  - `Errors.User.Phone.AlreadyVerified`
  - `Errors.User.Phone.Empty`
  - `Errors.User.Phone.NotChanged`
  - `Errors.User.Address.NotFound`
  - `Errors.User.Address.NotChanged`
  - `Errors.User.Username.AlreadyExists`
  - `Errors.User.Username.Reserved`
  - `Errors.User.Username.Empty`
  - `Errors.Org.LoginPolicy.RegistrationNotAllowed`
- `internal/static/i18n/ja.yaml`
  - `Errors.SMSConfig.NotExternalVerification`
  - `Errors.User.Profile.Empty`
  - `Errors.User.Profile.FirstNameEmpty`
  - `Errors.User.Profile.LastNameEmpty`
  - `Errors.User.Email.Empty`
  - `Errors.User.Email.IDMissing`
  - `Errors.User.Phone.Empty`
  - `Errors.User.Phone.NotChanged`
  - `Errors.User.Username.Empty`
  - `Errors.Org.LabelPolicy.NotFound`
  - `Errors.Org.LabelPolicy.NotChanged`
  - `EventTypes.project.application.oidc.key.added`
  - `EventTypes.project.application.oidc.key.removed`

# Additional Changes

- Change some order of the elements in `internal/static/i18n/ja.yaml`
  - `EventTypes.user.human.password.change.sent`
  - `EventTypes.user.human.password.hash.updated`
- Remove an element which is not used in the `us.yaml` from
`internal/static/i18n/ja.yaml`
  - `EventTypes.user.phone.removed`
- Correct a translation in `internal/static/i18n/ja.yaml`
  - `EventTypes.user.human.password.change.sent`
2025-02-19 12:51:53 +00:00
Iraq
5bbb953ffb
feat(ldap): adding root ca option to ldap config (#9292)
# Which Problems Are Solved

Adding ability to add a root CA to LDAP configs

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/7888

---------

Co-authored-by: Iraq Jaber <IraqJaber@gmail.com>
2025-02-18 10:06:50 +00:00
Ramon
3042bbb993
feat: Use V2 API's in Console (#9312)
# Which Problems Are Solved
Solves #8976

# Additional Changes
I have done some intensive refactorings and we are using the new
@zitadel/client package for GRPC access.

# Additional Context
- Closes #8976

---------

Co-authored-by: Max Peintner <peintnerm@gmail.com>
2025-02-17 19:25:46 +01:00
Iraq
0cb0380826
feat: updating eventstore.permitted_orgs sql function (#9309)
# Which Problems Are Solved

Performance issue for GRPC call `zitadel.user.v2.UserService.ListUsers`
due to lack of org filtering on `ListUsers`

# Additional Context

Replace this example with links to related issues, discussions, discord
threads, or other sources with more context.
Use the Closing #issue syntax for issues that are resolved with this PR.
- Closes https://github.com/zitadel/zitadel/issues/9191

---------

Co-authored-by: Iraq Jaber <IraqJaber@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2025-02-17 11:55:28 +02:00
Stefan Benz
49de5c61b2
feat: saml application configuration for login version (#9351)
# Which Problems Are Solved

OIDC applications can configure the used login version, which is
currently not possible for SAML applications.

# How the Problems Are Solved

Add the same functionality dependent on the feature-flag for SAML
applications.

# Additional Changes

None

# Additional Context

Closes #9267
Follow up issue for frontend changes #9354

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2025-02-13 16:03:05 +00:00
Iraq
66296db971
fix: custom userID not being added when specified in zitadel.org.v2.AddOrganizationRequest.AddOrganization() request (#9334)
# Which Problems Are Solved

When specifying a `user_id` as a human admin in
`zitadel.org.v2.AddOrganizationRequest.AddOrganization()` the `user_id`
specified in the request should have been used, before it was being
ignored, this has been fixed with this PR

# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/9308

---------

Co-authored-by: Iraq Jaber <IraqJaber@gmail.com>
2025-02-13 09:17:05 +00:00
Silvan
415bc32ed6
feat: add task queue (#9321)
# Which Problems Are Solved

To integrate river as a task queue we need to ensure the migrations of
river are executed.

# How the Problems Are Solved

- A new schema was added to the Zitadel database called "queue"
- Added a repeatable setup step to Zitadel which executes the
[migrations of
river](https://riverqueue.com/docs/migrations#go-migration-api).

# Additional Changes

- Added more hooks to the databases to properly set the schema for the
task queue

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/9280
2025-02-12 14:51:55 +00:00
Stefan Benz
0ea42f1ddf
fix: no project owner at project creation and cleanup (#9317)
# Which Problems Are Solved

Project creation always requires a user as project owner, in case of a
system user creating the project, there is no valid user existing at
that moment.

# How the Problems Are Solved

Remove the initially created project owner membership, as this is
something which was necessary in old versions, and all should work
perfectly without.
The call to add a project automatically designates the calling user as
the project owner, which is irrelevant currently, as this user always
already has higher permissions to be able to even create the project.

# Additional Changes

Cleanup of the existing checks for the project, which can be improved
through the usage of the fields table.

# Additional Context

Closes #9182
2025-02-12 11:48:28 +00:00
Stefan Benz
39a7977e34
test: session v2beta corrected like v2 (#9350)
# Which Problems Are Solved

Ordering of sessions in v2beta is still relevant in the integration
tests.

# How the Problems Are Solved

Correct the integration tests on session service v2beta like in v2.

# Additional Changes

None

# Additional Context

Failing integration tests in pipeline.
2025-02-12 10:46:14 +00:00
Stefan Benz
840da5be2d
feat: permission check on OIDC and SAML service session API (#9304)
# Which Problems Are Solved

Through configuration on projects, there can be additional permission
checks enabled through an OIDC or SAML flow, which were not included in
the OIDC and SAML services.

# How the Problems Are Solved

Add permission check through the query-side of Zitadel in a singular SQL
query, when an OIDC or SAML flow should be linked to a SSO session. That
way it is eventual consistent, but will not impact the performance on
the eventstore. The permission check is defined in the API, which
provides the necessary function to the command side.

# Additional Changes

Added integration tests for the permission check on OIDC and SAML
service for every combination.
Corrected session list integration test, to content checks without
ordering.
Corrected get auth and saml request integration tests, to check for
timestamp of creation, not start of test.

# Additional Context

Closes #9265

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2025-02-11 18:45:09 +00:00
Vlad Zagvozdkin
13f9d2d142
Add uid to few events (#9332)
# Which Problems Are Solved

When implementing simple stateless event processor, `the
user.grant.changed` bears too little information: just grant id and list
of role keys. This makes it impossible to change a users permissions
solely based on available role keys and requires to either:

- Store a mapping grant id -> user id, making a service stateful
- Make an extra call to zitadel to resolve user id by grant id (And it
doesn't seem that such an endpoint exists)

Same with `user.grant.removed` events.

# How the Problems Are Solved

Added `userId` field to `user.grant.changed` and `user.grant.removed`
events

# Additional Changes

`user.grant.removed` now has `projectId` and `grantId` as well

# Additional Context

- Closes #9113
2025-02-11 18:09:50 +00:00
Livio Spring
e7a73eb6b1
fix(oidc / login v2): always us login v2 if x-zitadel-login-client header is sent (#9336)
# Which Problems Are Solved

As reported in #9311, even when providing a `x-zitadel-login-client`
header, the auth request would be created as hosted login UI / V1
request.
This is due to a change introduced with #9071, where the login UI
version can be specified using the app configuration.
The configuration set to V1 was not considering if the header was sent.

# How the Problems Are Solved

- Check presence of `x-zitadel-login-client` before the configuration.
Use later only if no header is set.

# Additional Changes

None

# Additional Context

- closes #9311 
- needs back ports to 2.67.x, 2.68.x and 2.69.x
2025-02-10 14:46:28 +01:00
Lars
4dc7a58a25
fix: ensure metadata is projected for scim tests to ensure stable tests (#9305)
# Which Problems Are Solved
- SCIM tests are flaky due to metadata being set by the tests while
shortly after being read by the application, resulting in a race
condition

# How the Problems Are Solved
- whenever metadata is set, the projection is awaited

# Additional Context
Part of #8140

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2025-02-05 15:40:56 +00:00
Lars
361f7a2edc
fix: relax parsing of SCIM user 'active' flag to improve compatibility (#9296)
# Which Problems Are Solved
- Microsoft Entra invokes the user patch endpoint with `"active":
"True"` / `"active": "False"` when patching a user. This is a well-known
bug in MS Entra (see
[here](https://learn.microsoft.com/en-us/entra/identity/app-provisioning/application-provisioning-config-problem-scim-compatibility)),
but the bug fix has not landed yet and/or the feature flag does not
work.

# How the Problems Are Solved
- To ensure compatibility with MS Entra, the parsing of the the boolean
active flag of the scim user is relaxed and accepts strings in any
casing that resolve to `true` or `false` as well as raw boolean values.

# Additional Context
Part of https://github.com/zitadel/zitadel/issues/8140
2025-02-05 16:17:20 +01:00
Livio Spring
990e1982c7
fix(OTEL): reduce high cardinality in traces and metrics (#9286)
# Which Problems Are Solved

There were multiple issues in the OpenTelemetry (OTEL) implementation
and usage for tracing and metrics, which lead to high cardinality and
potential memory leaks:
- wrongly initiated tracing interceptors
- high cardinality in traces:
  - HTTP/1.1 endpoints containing host names
- HTTP/1.1 endpoints containing object IDs like userID (e.g.
`/management/v1/users/2352839823/`)
- high amount of traces from internal processes (spooler)
- high cardinality in metrics endpoint:
  - GRPC entries containing host names
  - notification metrics containing instanceIDs and error messages

# How the Problems Are Solved

- Properly initialize the interceptors once and update them to use the
grpc stats handler (unary interceptors were deprecated).
- Remove host names from HTTP/1.1 span names and use path as default.
- Set / overwrite the uri for spans on the grpc-gateway with the uri
pattern (`/management/v1/users/{user_id}`). This is used for spans in
traces and metric entries.
- Created a new sampler which will only sample spans in the following
cases:
  - remote was already sampled
- remote was not sampled, root span is of kind `Server` and based on
fraction set in the runtime configuration
- This will prevent having a lot of spans from the spooler back ground
jobs if they were not started by a client call querying an object (e.g.
UserByID).
- Filter out host names and alike from OTEL generated metrics (using a
`view`).
- Removed instance and error messages from notification metrics.

# Additional Changes

Fixed the middleware handling for serving Console. Telemetry and
instance selection are only used for the environment.json, but not on
statically served files.

# Additional Context

- closes #8096 
- relates to #9074
- back ports to at least 2.66.x, 2.67.x and 2.68.x
2025-02-04 09:55:26 +01:00
Livio Spring
04b9e9b144
fix(console): add posthog to CSP if configured (#9284)
# Which Problems Are Solved

PostHog scripts are currently blocked by content security policy (CSP).

# How the Problems Are Solved

Add `https://*.i.posthog.com` to the CSP according to
https://posthog.com/docs/advanced/content-security-policy#enabling-the-toolbar
(they suggest  `https://*.posthog.com`)

# Additional Changes

None

# Additional Context

relates to https://github.com/zitadel/zitadel/issues/9076
2025-02-03 08:08:01 +01:00
Lars
f65db52247
fix: scim create users dont send init emails (#9283)
# Which Problems Are Solved
- when a scim user is provisioned, a init email could be sent

# How the Problems Are Solved
- no init email should be sent => hard code false for the email init
param

# Additional Context

Related to https://github.com/zitadel/zitadel/issues/8140

Co-authored-by: Fabienne Bühler <fabienne@zitadel.com>
2025-01-31 09:36:18 +00:00
Lars
20cff9c70a
fix: scim 2.0 patch ignore op casing (#9282)
# Which Problems Are Solved
- Some SCIM clients send "op" of a patch operation in PascalCase

# How the Problems Are Solved
- Well known "op" values of patch operations are matched
case-insensitive.

# Additional Context
Related to #8140
2025-01-31 09:15:39 +00:00
Lars
563f74640e
fix: scim v2 endpoints enforce user resource owner (#9273)
# Which Problems Are Solved
- If a SCIM endpoint is called with an orgID in the URL that is not the
resource owner, no error is returned, and the action is executed.

# How the Problems Are Solved
- The orgID provided in the SCIM URL path must match the resource owner
of the target user. Otherwise, an error will be returned.

# Additional Context

Part of https://github.com/zitadel/zitadel/issues/8140
2025-01-30 16:43:13 +01:00
Lars
e15094cdea
feat: add scim v2 service provider configuration endpoints (#9258)
# Which Problems Are Solved
* Adds support for the service provider configuration SCIM v2 endpoints

# How the Problems Are Solved
* Adds support for the service provider configuration SCIM v2 endpoints
  * `GET /scim/v2/{orgId}/ServiceProviderConfig`
  * `GET /scim/v2/{orgId}/ResourceTypes`
  * `GET /scim/v2/{orgId}/ResourceTypes/{name}`
  * `GET /scim/v2/{orgId}/Schemas`
  * `GET /scim/v2/{orgId}/Schemas/{id}`

# Additional Context
Part of #8140

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2025-01-29 18:11:12 +00:00
Tim Möhlmann
b6841251b1
feat(users/v2): return prompt information (#9255)
# Which Problems Are Solved

Add the ability to update the timestamp when MFA initialization was last
skipped.
Get User By ID now also returns the timestamps when MFA setup was last
skipped.

# How the Problems Are Solved

- Add a `HumanMFAInitSkipped` method to the `users/v2` API.
- MFA skipped was already projected in the `auth.users3` table. In this
PR the same column is added to the users projection. Event handling is
kept the same as in the `UserView`:

<details>


62804ca45f/internal/user/repository/view/model/user.go (L243-L377)

</details>

# Additional Changes

- none

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/9197
2025-01-29 15:12:31 +00:00
Lars
df8bac8a28
feat: bulk scim v2 endpoint (#9256)
# Which Problems Are Solved
* Adds support for the bulk SCIM v2 endpoint

# How the Problems Are Solved
* Adds support for the bulk SCIM v2 endpoint under `POST
/scim/v2/{orgID}/Bulk`

# Additional Context
Part of #8140

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2025-01-29 14:23:56 +00:00
Lars
accfb7525a
fix: scim 2 filter: the username should be treated case-insensitive (#9257)
# Which Problems Are Solved
- when listing users via scim v2.0 filters applied to the username are
applied case-sensitive

# How the Problems Are Solved
- when a query filter is appleid on the username it is applied
case-insensitive

# Additional Context
Part of https://github.com/zitadel/zitadel/issues/8140
2025-01-29 15:22:22 +02:00
Silvan
b10428fb56
test(session): load tests for session api (#9212)
# Which Problems Are Solved

We currently are not able to benchmark the performance of the session
api

# How the Problems Are Solved

Load tests were added to
- use sessions in oidc tokens analog
https://zitadel.com/docs/guides/integrate/login-ui/oidc-standard

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/7847
2025-01-29 12:08:20 +00:00
kkrime
5eeff97ffe
feat(session/v2): user password lockout error response (#9233)
# Which Problems Are Solved

Adds `failed attempts` field to the grpc response when a user enters
wrong password when logging in

FYI:

this only covers the senario above; other senarios where this is not
applied are:
SetPasswordWithVerifyCode
setPassword
ChangPassword
setPasswordWithPermission

# How the Problems Are Solved 

Created new grpc message `CredentialsCheckError` -
`proto/zitadel/message.proto` to include `failed_attempts` field.

Had to create a new package -
`github.com/zitadel/zitadel/internal/command/errors` to resolve cycle
dependency between `github.com/zitadel/zitadel/internal/command` and
`github.com/zitadel/zitadel/internal/command`.

# Additional Changes

- none

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/9198

---------

Co-authored-by: Iraq Jaber <IraqJaber@gmail.com>
2025-01-29 10:29:00 +00:00
Lars
21f00c1e6b
fix: scim use first email or phone if no primary is set (#9236)
# Which Problems Are Solved
- scim v2 only maps the primary phone/email to the zitadel user, this
does not work if no primary is set

# How the Problems Are Solved
- the first phone / email is mapped if no primary is available

# Additional Context
Part of #8140

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2025-01-29 09:18:00 +00:00
Livio Spring
3fc68e5d60
fix(notifications): cancel on missing channels and Twilio 4xx errors (#9254)
# Which Problems Are Solved

#9185 changed that if a notification channel was not present,
notification workers would no longer retry to send the notification and
would also cancel in case Twilio would return a 4xx error.
However, this would not affect the "legacy" mode.

# How the Problems Are Solved

- Handle `CancelError` in legacy notifier as not failed (event).

# Additional Changes

None

# Additional Context

- relates to #9185 
- requires back port to 2.66.x and 2.67.x
2025-01-28 06:32:09 +00:00
Lars
30a54fc1eb
fix: scim user query endpoint don't allow SortBy custom field (#9235)
# Which Problems Are Solved
- scim list users endpoint (`GET /scim/v2/{orgId}/Users`): handle
unsupported `SortBy` columns correctly

# How the Problems Are Solved
- throw an error if sorting by an unsupported column is requested

# Additional Context
Part of #8140
2025-01-27 17:30:27 +00:00
Lars
b19333726c
fix: allow scim content type wildcards (#9245)
# Which Problems Are Solved
- requests to the scim interface with content type `*/*` are rejected

# How the Problems Are Solved
- `*/*` is accepted as content type

# Additional Context
Part of #8140
2025-01-27 16:10:30 +00:00
Lars
741434806a
fix: unified scim metadata key casing (#9244)
# Which Problems Are Solved
- SCIM user metadata mapping keys have differing case styles.

# How the Problems Are Solved
- key casing style is unified to strict camelCase

# Additional Context
Part of #8140

Although this is technically a breaking change, it is considered
acceptable because the SCIM feature is still in the preview stage and
not fully implemented yet.

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2025-01-27 13:51:58 +00:00
Lars
189f9770c6
feat: patch user scim v2 endpoint (#9219)
# Which Problems Are Solved
* Adds support for the patch user SCIM v2 endpoint

# How the Problems Are Solved
* Adds support for the patch user SCIM v2 endpoint under `PATCH
/scim/v2/{orgID}/Users/{id}`

# Additional Context
Part of #8140
2025-01-27 13:36:07 +01:00
Zach Hirschtritt
e4bbfcccc8
fix: add aggregate type to subquery to utilize indexes (#9226)
# Which Problems Are Solved

The subquery of the notification requested and retry requested is
missing the aggregate_type filter that would allow it to utilize the
`es_projection` or `active_instances_events` on the eventstore.events2
table.

# How the Problems Are Solved

Add additional filter on subquery. Final query: 
```sql
SELECT <all the fields omitted> FROM eventstore.events2
WHERE
    instance_id = $1
    AND aggregate_type = $2
    AND event_type = $3
    AND created_at > $4
    AND aggregate_id NOT IN (
        SELECT aggregate_id
        FROM eventstore.events2
        WHERE
            aggregate_type = $5 <-- NB: previously missing
            AND event_type = ANY ($6)
            AND instance_id = $7
            AND created_at > $8
    )
ORDER BY "position", in_tx_order
LIMIT $9
FOR UPDATE SKIP LOCKED
```

# Additional Changes

# Additional Context

Co-authored-by: Livio Spring <livio.a@gmail.com>
2025-01-22 16:02:37 +00:00
Livio Spring
c9aa5db2a5
fix(oidc apps): correctly remove last additional origin, redirect uri and post logout redirect uri (#9209)
# Which Problems Are Solved

A customer reached out to support, that the (last) `additional origin`
could not be removed. While testing / implementation it was discovered,
that the same applied to `redirect_uris` and `post_logout_redirect_uris`

# How the Problems Are Solved

- Correctly set the corresponding array to empty in the event so it can
be differentiated to `null` / not set in case of no change.

# Additional Changes

Replaced `reflect.DeepEqual` with `slices.Equal`

# Additional Context

- Reported to support
2025-01-22 07:37:37 +00:00
Lars
1915d35605
feat: list users scim v2 endpoint (#9187)
# Which Problems Are Solved
- Adds support for the list users SCIM v2 endpoint

# How the Problems Are Solved
- Adds support for the list users SCIM v2 endpoints under `GET
/scim/v2/{orgID}/Users` and `POST /scim/v2/{orgID}/Users/.search`

# Additional Changes
- adds a new function `SearchUserMetadataForUsers` to the query layer to
query a metadata keyset for given user ids
- adds a new function `NewUserMetadataExistsQuery` to the query layer to
query a given metadata key value pair exists
- adds a new function `CountUsers` to the query layer to count users
without reading any rows
- handle `ErrorAlreadyExists` as scim errors `uniqueness`
- adds `NumberLessOrEqual` and `NumberGreaterOrEqual` query comparison
methods
- adds `BytesQuery` with `BytesEquals` and `BytesNotEquals` query
comparison methods

# Additional Context
Part of #8140
Supported fields for scim filters:
* `meta.created`
* `meta.lastModified`
* `id`
* `username`
* `name.familyName`
* `name.givenName`
* `emails` and `emails.value`
* `active` only eq and ne
* `externalId` only eq and ne
2025-01-21 13:31:54 +01:00
Tim Möhlmann
94cbf97534
fix(permissions_v2): add membership fields migration (#9199)
# Which Problems Are Solved

Memberships did not have a fields table fill migration.

# How the Problems Are Solved

Add filling of membership fields to the repeatable steps.

# Additional Changes

- Use the same repeatable step for multiple fill fields handlers.
- Fix an error for PostgreSQL 15 where a subquery in a `FROM` clause
needs an alias ing the `permitted_orgs` function.

# Additional Context

- Part of https://github.com/zitadel/zitadel/issues/9188
- Introduced in https://github.com/zitadel/zitadel/pull/9152
2025-01-17 16:16:26 +01:00
Livio Spring
60857c8d3e
fix: cancel notifications on missing channels and configurable (twilio) error codes (#9185)
# Which Problems Are Solved

If a notification channel was not present, notification workers would
retry to the max attempts. This leads to unnecessary load.
Additionally, a client noticed  bad actors trying to abuse SMS MFA. 

# How the Problems Are Solved

- Directly cancel the notification on:
  - a missing channel and stop retries.
  - any `4xx` errors from Twilio Verify

# Additional Changes

None

# Additional Context

reported by customer
2025-01-17 07:42:14 +00:00
Stefan Benz
69372e5209
fix: case changes on org domain (#9196)
# Which Problems Are Solved

Organization name change results in domain events even if the domain
itself doesn't change.

# How the Problems Are Solved

Check if the domain itself really changes, and if not, don't create the
events.

# Additional Changes

Unittest for this specific case.

# Additional Context

None
2025-01-16 16:05:55 +01:00
Silvan
4645045987
refactor: consolidate database pools (#9105)
# Which Problems Are Solved

Zitadel currently uses 3 database pool, 1 for queries, 1 for pushing
events and 1 for scheduled projection updates. This defeats the purpose
of a connection pool which already handles multiple connections.

During load tests we found that the current structure of connection
pools consumes a lot of database resources. The resource usage dropped
after we reduced the amount of database pools to 1 because existing
connections can be used more efficiently.

# How the Problems Are Solved

Removed logic to handle multiple connection pools and use a single one.

# Additional Changes

none

# Additional Context

part of https://github.com/zitadel/zitadel/issues/8352
2025-01-16 11:07:18 +00:00
Lars
07f74730ac
fix: include tzdata to validate timezones in scim (#9195)
# Which Problems Are Solved
- include tzdata in the binary to correctly validate time zones in the
scim layer if the os doesn't have timezone data available.

# How the Problems Are Solved
- by importing the go pkg `"time/tzdata"`

# Additional Context
Part of https://github.com/zitadel/zitadel/issues/8140
2025-01-16 10:34:52 +00:00
Tim Möhlmann
3f6ea78c87
perf: role permissions in database (#9152)
# Which Problems Are Solved

Currently ZITADEL defines organization and instance member roles and
permissions in defaults.yaml. The permission check is done on API call
level. For example: "is this user allowed to make this call on this
org". This makes sense on the V1 API where the API is permission-level
shaped. For example, a search for users always happens in the context of
the organization. (Either the organization the calling user belongs to,
or through member ship and the x-zitadel-orgid header.

However, for resource based APIs we must be able to resolve permissions
by object. For example, an IAM_OWNER listing users should be able to get
all users in an instance based on the query filters. Alternatively a
user may have user.read permissions on one or more orgs. They should be
able to read just those users.

# How the Problems Are Solved

## Role permission mapping

The role permission mappings defined from `defaults.yaml` or local
config override are synchronized to the database on every run of
`zitadel setup`:

- A single query per **aggregate** builds a list of `add` and `remove`
actions needed to reach the desired state or role permission mappings
from the config.
- The required events based on the actions are pushed to the event
store.
- Events define search fields so that permission checking can use the
indices and is strongly consistent for both query and command sides.

The migration is split in the following aggregates:

- System aggregate for for roles prefixed with `SYSTEM`
- Each instance for roles not prefixed with `SYSTEM`. This is in
anticipation of instance level management over the API.

## Membership

Current instance / org / project membership events now have field table
definitions. Like the role permissions this ensures strong consistency
while still being able to use the indices of the fields table. A
migration is provided to fill the membership fields.

## Permission check

I aimed keeping the mental overhead to the developer to a minimal. The
provided implementation only provides a permission check for list
queries for org level resources, for example users. In the `query`
package there is a simple helper function `wherePermittedOrgs` which
makes sure the underlying database function is called as part of the
`SELECT` query and the permitted organizations are part of the `WHERE`
clause. This makes sure results from non-permitted organizations are
omitted. Under the hood:

- A Pg/PlSQL function searches for a list of organization IDs the passed
user has the passed permission.
- When the user has the permission on instance level, it returns early
with all organizations.
- The functions uses a number of views. The views help mapping the
fields entries into relational data and simplify the code use for the
function. The views provide some pre-filters which allow proper index
usage once the final `WHERE` clauses are set by the function.

# Additional Changes



# Additional Context

Closes #9032
Closes https://github.com/zitadel/zitadel/issues/9014

https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for
the new permission framework based on this concept.
2025-01-16 10:09:15 +00:00
Livio Spring
40082745f4
fix(login): allow fallback to local auth in case of IdP errors (#9178)
# Which Problems Are Solved

The current login will always prefer external authentication (through an
IdP) over local authentication. So as soon as either the user had
connected to an IdP or even when the login policy was just set up to
have an IdP allowed, users would be redirected to that IdP for
(re)authentication.
This could lead to problems, where the IdP was not available or any
other error occurred in the process (such as secret expired for
EntraID).
Even when local authentication (passkeys or password) was allowed for
the corresponding user, they would always be redirected to the IdP
again, preventing any authentication. If admins were affected, they
might not even be able to update the client secret of the IdP.

# How the Problems Are Solved

Errors during the external IdP flow are handled in an
`externalAuthFailed` function, which will check if the organisation
allows local authentication and if the user has set up such.
If either password or passkeys is set up, the corresponding login page
will be presented to the user. As already with local auth passkeys is
preferred over password authentication.
The user is informed that the external login failed and fail back to
local auth as an error on the corresponding page in a focused mode. Any
interaction or after 5 second the focus mode is disabled.

# Additional Changes

None.

# Additional Context

closes #6466
2025-01-15 10:39:28 +00:00
Silvan
1949d1546a
fix: set correct owner on project grants (#9089)
# Which Problems Are Solved

In versions previous to v2.66 it was possible to set a different
resource owner on project grants. This was introduced with the new
resource based API. The resource owner was possible to overwrite using
the x-zitadel-org header.

Because of this issue project grants got the wrong resource owner,
instead of the owner of the project it got the granted org which is
wrong because a resource owner of an aggregate is not allowed to change.

# How the Problems Are Solved

- The wrong owners of the events are set to the original owner of the
project.
- A new event is pushed to these aggregates `project.owner.corrected` 
- The projection updates the owners of the user grants if that event was
written

# Additional Changes

The eventstore push function (replaced in version 2.66) writes the
correct resource owner.

# Additional Context

closes https://github.com/zitadel/zitadel/issues/9072
2025-01-15 11:22:16 +01:00
MAHANTH-wq
b664ffe993
feat(/internal): Add User Resource Owner (#9168)
Update the  ../proto/zitadel/member.proto to
include the UserResourceOwner as part of member.

Update the queries to include UserResourceOwner
for the following :
zitadel/internal/query/iam_member.go
zitadel/internal/query/org_member.go
zitadel/internal/query/project_member.go
zitadel/internal/query/project_grant_member.go

Non Breaking Changes

# Which Problems Are Solved

https://github.com/zitadel/zitadel/issues/5062

# How the Problems Are Solved

- Updated the member.proto file to include user_resource_owner. I have
compiled using` make compile` command .
- Changed the queries to include the userResourceOwner as part of
Member.
- Then, updated the converter to map the userResourceOwner.

# Additional Changes

Replace this example text with a concise list of additional changes that
this PR introduces, that are not directly solving the initial problem
but are related.
For example:
- The docs explicitly describe that the property XY is mandatory
- Adds missing translations for validations.

# Additional Context


- Closes #5062 
-
https://discordapp.com/channels/927474939156643850/1326245856193544232/1326476710752948316
2025-01-15 09:40:30 +01:00
Lars
d01d003a03
feat: replace user scim v2 endpoint (#9163)
# Which Problems Are Solved
- Adds support for the replace user SCIM v2 endpoint

# How the Problems Are Solved
- Adds support for the replace user SCIM v2 endpoint under `PUT
/scim/v2/{orgID}/Users/{id}`

# Additional Changes
- Respect the `Active` field in the SCIM v2 create user endpoint `POST
/scim/v2/{orgID}/Users`
- Eventually consistent read endpoints used in SCIM tests are wrapped in
`assert.EventuallyWithT` to work around race conditions

# Additional Context
Part of #8140
2025-01-14 15:44:41 +01:00
Stefan Benz
84997ffe1a
fix(session v2): allow searching for own sessions or user agent (fingerprintID) (#9110)
# Which Problems Are Solved

ListSessions only works to list the sessions that you are the creator
of.

# How the Problems Are Solved

Add options to search for sessions created by other users, sessions
belonging to the same useragent and sessions belonging to your user.
Possible through additional search parameters which as default use the
information contained in your session token but can also be filled with
specific IDs.

# Additional Changes

Remodel integration tests, to separate the Create and Get of sessions
correctly.

# Additional Context

Closes #8301

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2025-01-14 14:15:59 +01:00
Lars
9c7f2a7d50
feat: get user scim v2 endpoint (#9161)
# Which Problems Are Solved
- Adds support for the get user SCIM v2 endpoint

# How the Problems Are Solved
- Adds support for the get user SCIM v2 endpoint under `GET
/scim/v2/{orgID}/Users/{id}`

# Additional Context
Part of #8140
Replaces https://github.com/zitadel/zitadel/pull/9154 as requested by
the maintainers, discussions see
https://github.com/zitadel/zitadel/pull/9154.
2025-01-10 11:15:06 +00:00
Lars
af09e51b1e
feat: delete user scim v2 endpoint (#9151)
# Which Problems Are Solved
- Adds support for the user delete SCIM v2 endpoint

# How the Problems Are Solved
- Adds support for the user delete SCIM v2 endpoint under `DELETE
/scim/v2/{orgID}/Users/{id}`

# Additional Context
Part of #8140
2025-01-09 15:12:13 +01:00
Lars
e621224ab2
feat: create user scim v2 endpoint (#9132)
# Which Problems Are Solved
- Adds infrastructure code (basic implementation, error handling,
middlewares, ...) to implement the SCIM v2 interface
- Adds support for the user create SCIM v2 endpoint

# How the Problems Are Solved
- Adds support for the user create SCIM v2 endpoint under `POST
/scim/v2/{orgID}/Users`

# Additional Context

Part of #8140
2025-01-09 12:46:36 +01:00
Livio Spring
c966446f80
fix: correctly get x-forwarded-for for browser info in events (#9149)
# Which Problems Are Solved

Events like "password check succeeded" store some information about the
caller including their IP.
The `X-Forwarded-For` was not correctly logged, but instead the
RemoteAddress.

# How the Problems Are Solved

- Correctly get the `X-Forwarded-For` in canonical form.

# Additional Changes

None

# Additional Context

closes [#9106](https://github.com/zitadel/zitadel/issues/9106)
2025-01-08 09:30:12 +00:00