Commit Graph

392 Commits

Author SHA1 Message Date
Silvan
ddeeeed303
fix(handler): optimise snapshot hanlding (#8652)
# Which Problems Are Solved

There are cases where not all statements of multiExec are succeed. This
leads to inconsistent states. One example is [LDAP
IDPs](https://github.com/zitadel/zitadel/issues/7959).

If statements get executed only partially this can lead to inconsistent
states or even break projections for objects which might not were
correctly created in a sub table.

This behaviour is possible because we use
[`SAVEPOINTS`](https://www.postgresql.org/docs/current/sql-savepoint.html)
during each statement of a multiExec.

# How the Problems Are Solved

SAVEPOINTS are only created at the beginning of an exec function not
during every execution like before. Additionally `RELEASE` or `ROLLBACK`
of `SAVEPOINTS` are only used when needed.

# Additional Changes

- refactor some unused parameters

# Additional Context

- closes https://github.com/zitadel/zitadel/issues/7959
2024-10-02 18:34:19 +03:00
Tim Möhlmann
63d733b3a2
perf(oidc): disable push of user token meta-event (#8691)
# Which Problems Are Solved

When executing many concurrent authentication requests on a single
machine user, there were performance issues. As the same aggregate is
being searched and written to concurrently, we traced it down to a
locking issue on the used index.
We already optimized the token endpoint by creating a separate OIDC
aggregate.

At the time we decided to push a single event to the user aggregate, for
the user audit log. See [technical advisory
10010](https://zitadel.com/docs/support/advisory/a10010) for more
details.

However, a recent security fix introduced an additional search query on
the user aggregate, causing the locking issue we found.

# How the Problems Are Solved

Add a feature flag which disables pushing of the `user.token.v2.added`.
The event has no importance and was only added for informational
purposes on the user objects. The `oidc_session.access_token.added` is
the actual payload event and is pushed on the OIDC session aggregate and
can still be used for audit trail.

# Additional Changes

- Fix an event mapper type for
`SystemOIDCSingleV1SessionTerminationEventType`

# Additional Context

- Reported by support request
- https://github.com/zitadel/zitadel/pull/7822 changed the token
aggregate
- https://github.com/zitadel/zitadel/pull/8631 introduced user state
check

Load test trace graph with `user.token.v2.added` **enabled**. Query
times are steadily increasing:


![image](https://github.com/user-attachments/assets/4aa25055-8721-4e93-b695-625560979909)

Load test trace graph with `user.token.v2.added` **disabled**. Query
times constant:


![image](https://github.com/user-attachments/assets/a7657f6c-0c55-401b-8291-453da5d5caf9)

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-09-26 13:55:41 +00:00
Livio Spring
14e2aba1bc
feat: Add Twilio Verification Service (#8678)
# Which Problems Are Solved
Twilio supports a robust, multi-channel verification service that
notably supports multi-region SMS sender numbers required for our use
case. Currently, Zitadel does much of the work of the Twilio Verify (eg.
localization, code generation, messaging) but doesn't support the pool
of sender numbers that Twilio Verify does.

# How the Problems Are Solved
To support this API, we need to be able to store the Twilio Service ID
and send that in a verification request where appropriate: phone number
verification and SMS 2FA code paths.

This PR does the following: 
- Adds the ability to use Twilio Verify of standard messaging through
Twilio
- Adds support for international numbers and more reliable verification
messages sent from multiple numbers
- Adds a new Twilio configuration option to support Twilio Verify in the
admin console
- Sends verification SMS messages through Twilio Verify
- Implements Twilio Verification Checks for codes generated through the
same

# Additional Changes

# Additional Context
- base was implemented by @zhirschtritt in
https://github.com/zitadel/zitadel/pull/8268 ❤️
- closes https://github.com/zitadel/zitadel/issues/8581

---------

Co-authored-by: Zachary Hirschtritt <zachary.hirschtritt@klaviyo.com>
Co-authored-by: Joey Biscoglia <joey.biscoglia@klaviyo.com>
2024-09-26 09:14:33 +02:00
Tim Möhlmann
4eaa3163b6
feat(storage): generic cache interface (#8628)
# Which Problems Are Solved

We identified the need of caching.
Currently we have a number of places where we use different ways of
caching, like go maps or LRU.
We might also want shared chaches in the future, like Redis-based or in
special SQL tables.

# How the Problems Are Solved

Define a generic Cache interface which allows different implementations.

- A noop implementation is provided and enabled as.
- An implementation using go maps is provided
  - disabled in defaults.yaml
  - enabled in integration tests
- Authz middleware instance objects are cached using the interface.

# Additional Changes

- Enabled integration test command raceflag
- Fix a race condition in the limits integration test client
- Fix a number of flaky integration tests. (Because zitadel is super
fast now!) 🎸 🚀

# Additional Context

Related to https://github.com/zitadel/zitadel/issues/8648
2024-09-25 21:40:21 +02:00
Tim Möhlmann
aeb379e7de
fix(eventstore): revert precise decimal (#8527) (#8679) 2024-09-24 18:43:29 +02:00
Livio Spring
5b40af79f0
fix: correctly check user state (#8631)
# Which Problems Are Solved

ZITADEL's user account deactivation mechanism did not work correctly
with service accounts. Deactivated service accounts retained the ability
to request tokens, which could lead to unauthorized access to
applications and resources.

# How the Problems Are Solved

Additionally to checking the user state on the session API and login UI,
the state is checked on all oidc session methods resulting in a new
token or when returning the user information (userinfo, introspection,
id_token / access_token and saml attributes)
2024-09-17 13:21:49 +00:00
Livio Spring
ca1914e235
fix: user grants deactivation (#8634)
# Which Problems Are Solved

ZITADEL's user grants deactivation mechanism did not work correctly.
Deactivated user grants were still provided in token, which could lead
to unauthorized access to applications and resources.
Additionally, the management and auth API always returned the state as
active or did not provide any information about the state.

# How the Problems Are Solved

- Correctly check the user grant state on active for tokens and user
information (userinfo, introspection, saml attributes)
- Map state in API and display in Console
2024-09-17 12:18:29 +00:00
Livio Spring
d01bd1c51a
fix: correctly check app state on authentication (#8630)
# Which Problems Are Solved

In Zitadel, even after an organization is deactivated, associated
projects, respectively their applications remain active. Users across
other organizations can still log in and access through these
applications, leading to unauthorized access.
Additionally, if a project was deactivated access to applications was
also still possible.

# How the Problems Are Solved

- Correctly check the status of the organization and related project. 
(Corresponding functions have been renamed to `Active...`)
2024-09-17 11:34:14 +00:00
Stefan Benz
289378713e
fix: smtp provider (#8610)
There were some problems after changing from SMTP providers to email
providers (https://github.com/zitadel/zitadel/pull/8545):
- panic (nil pointer) on SMTP provider retrieval for HTTP configs
- old SMTP configuration created before the multi SMTP configurations
(without id)
  - were not projected / listed
  - could not be always be activated
- Console treated HTTP configs as SMTP

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-09-13 13:22:25 +00:00
Stefan Benz
21c38b061d
feat: add http as smtp provider (#8545)
# Which Problems Are Solved

Send Email messages as a HTTP call to a relay, for own logic on handling
different Email providers

# How the Problems Are Solved

Create endpoints under Email provider to manage SMTP and HTTP in the
notification handlers.

# Additional Changes

Clean up old logic in command and query side to handle the general Email
providers with deactivate, activate and remove.

# Additional Context

Partially closes #8270

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-09-12 06:27:29 +02:00
Livio Spring
a07b2f4677
feat: invite user link (#8578)
# Which Problems Are Solved

As an administrator I want to be able to invite users to my application
with the API V2, some user data I will already prefil, the user should
add the authentication method themself (password, passkey, sso).

# How the Problems Are Solved

- A user can now be created with a email explicitly set to false.
- If a user has no verified email and no authentication method, an
`InviteCode` can be created through the User V2 API.
  - the code can be returned or sent through email
- additionally `URLTemplate` and an `ApplicatioName` can provided for
the email
- The code can be resent and verified through the User V2 API
- The V1 login allows users to verify and resend the code and set a
password (analog user initialization)
- The message text for the user invitation can be customized

# Additional Changes

- `verifyUserPasskeyCode` directly uses `crypto.VerifyCode` (instead of
`verifyEncryptedCode`)
- `verifyEncryptedCode` is removed (unnecessarily queried for the code
generator)

# Additional Context

- closes #8310
- TODO: login V2 will have to implement invite flow:
https://github.com/zitadel/typescript/issues/166
2024-09-11 10:53:55 +00:00
Tim Möhlmann
58a7eb1f26
perf(oidc): remove get user by ID from jwt profile grant (#8580)
# Which Problems Are Solved

Improve performance by removing a GetUserByID call. The call also
executed a Trigger on projections, which significantly impacted
concurrent requests.

# How the Problems Are Solved

Token creation needs information from the user, such as the resource
owner and access token type.

For client credentials this is solved in a single search. By getting the
user by username (`client_id`), the user details and secret were
obtained in a single query. After that verification and token creation
can proceed. For JWT profile it is a bit more complex. We didn't know
anything about the user until after JWT verification.
The verification did a query for the AuthN key and after that we did a
GetUserByID to get remaining details.

This change uses a joined query when the OIDC library calls the
`GetKeyByIDAndClientID` method on the token storage. The found user
details are set to the verifieer object and returned after verification
is completed.
It is safe because the `jwtProfileKeyStorage` is a single-use object as
a wrapper around `query.Queries`.
This way getting the public key and user details are obtained in a
single query.

# Additional Changes

- Correctly set the `client_id` field with machine's username.

# Additional Context

- Related to: https://github.com/zitadel/zitadel/issues/8352
2024-09-11 12:04:09 +03:00
Tim Möhlmann
3aba942162
feat: add debug events API (#8533)
# Which Problems Are Solved

Add a debug API which allows pushing a set of events to be reduced in a
dedicated projection.
The events can carry a sleep duration which simulates a slow query
during projection handling.

# How the Problems Are Solved

- `CreateDebugEvents` allows pushing multiple events which simulate the
lifecycle of a resource. Each event has a `projectionSleep` field, which
issues a `pg_sleep()` statement query in the projection handler :
  - Add
  - Change
  - Remove
- `ListDebugEventsStates` list the current state of the projection,
optionally with a Trigger
- `GetDebugEventsStateByID` get the current state of the aggregate ID in
the projection, optionally with a Trigger


# Additional Changes

- none

# Additional Context

-  Allows reproduction of https://github.com/zitadel/zitadel/issues/8517
2024-09-11 08:24:00 +00:00
Stefan Benz
5bdf1a4547
feat: add http as sms provider (#8540)
# Which Problems Are Solved

Send SMS messages as a HTTP call to a relay, for own logic on handling
different SMS providers.

# How the Problems Are Solved

Add HTTP as SMS provider type and handling of webhook messages in the
notification handlers.

# Additional Changes

Clean up old Twilio events, which were supposed to handle the general
SMS providers with deactivate, activate and remove.

# Additional Context

Partially closes #8270

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-09-06 13:11:36 +00:00
Silvan
b522588d98
fix(eventstore): precise decimal (#8527)
# Which Problems Are Solved

Float64 which was used for the event.Position field is [not precise in
go and gets rounded](https://github.com/golang/go/issues/47300). This
can lead to unprecies position tracking of events and therefore
projections especially on cockcoachdb as the position used there is a
big number.

example of a unprecies position:
exact: 1725257931223002628
float64: 1725257931223002624.000000

# How the Problems Are Solved

The float64 was replaced by
[github.com/jackc/pgx-shopspring-decimal](https://github.com/jackc/pgx-shopspring-decimal).

# Additional Changes

Correct behaviour of makefile for load tests.
Rename `latestSequence`-queries to `latestPosition`
2024-09-06 12:19:19 +03:00
Livio Spring
382a97c30f
feat(oidc): end session by id_token_hint and without cookie (#8542)
# Which Problems Are Solved

The end_session_endpoint currently always requires the userAgent cookie
to be able to terminate session created through the hosted login UI.
Only tokens issued through the Login V2 can be used to directly
terminate a specific session and without the need of a cookie.
This PR adds the possibility to terminate a single V1 session or all V1
sessions belonging to the same user agent without the need of the
userAgent cookie by providing an id_token as `id_token_hint` which
contains the id of a V1 session as `sid`.

# How the Problems Are Solved

- #8525 added the `sid` claim for id_tokens issued through the login UI
- The `sid` can now be checked for the `V1_` prefix and queries for
either the userAgentID and depending on the
`OIDCSingleV1SessionTermination` flag all userIDs of active session from
the same user agent id
- The `OIDCSingleV1SessionTermination` flag is added with default value
false to keep the existing behavior of terminating all sessions even in
case of providing an id_token_hint

# Additional Changes

- pass `context.Context` into session view functions for querying the
database with that context

# Additional Context

- relates to #8499 
- closes #8501
2024-09-04 10:14:50 +00:00
Stefan Benz
41ae35f2ef
feat: add schema user create and remove (#8494)
# Which Problems Are Solved

Added functionality that user with a userschema can be created and
removed.

# How the Problems Are Solved

Added logic and moved APIs so that everything is API v3 conform.

# Additional Changes

- move of user and userschema API to resources folder
- changed testing and parameters
- some renaming

# Additional Context

closes #7308

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-08-28 19:46:45 +00:00
Livio Spring
ca8f82423a
fix: automatically link user without prompt (#8487)
# Which Problems Are Solved

There were UX issue with the autolinking prompt page and users were not
able to link their account or would not understand what to do. Since the
trust to the IdP is already bound by the configuration, the user can
directly be linked without any user input.

# How the Problems Are Solved

- remove the prompt page and directly link the user if possible
- remove corresponding customization texts from the API and Console

# Additional Changes

None

# Additional Context

- relates to https://github.com/zitadel/zitadel/issues/7977
- discussed with customers
- created as a `fix` to be able to backport

---------

Co-authored-by: Max Peintner <max@caos.ch>
2024-08-28 05:33:20 +00:00
Livio Spring
cbbd44c303
fix: return 401 instead of 403 on expired tokens (#8476)
# Which Problems Are Solved

The access token verifier returned a permission denied (HTTP 403 / GRPC
7) instead of a unauthenticated (HTTP 401 / GRPC 16) error.

# How the Problems Are Solved

Return the correct error type.

# Additional Changes

None

# Additional Context

close #8392
2024-08-26 10:15:40 +00:00
Stefan Benz
2847806531
fix: generalise permission check for query user information (#8458)
# Which Problems Are Solved

IDPLinks list and other list endpoints can provide you with empty
results if the used user has no permission for the information.

# How the Problems Are Solved

List endpoints with subelements to users, and provided userIDQuery, will
return a PermissionDenied error if no permission for the user exsists.

# Additional Changes

Function to check for permission is re-used from the GetUserByID.

# Additional Context

Closes #8451
2024-08-23 06:44:18 +00:00
Tim Möhlmann
5fd2061770
feat(oidc): allow returning of parent errors to client (#8376)
# Which Problems Are Solved

Currently the OIDC API of ZITADEL only prints parent errors to the logs.
Where 4xx status are typically warn level and 5xx error level. This
makes it hard to debug certain errors for client in multi-instance
environments like ZITADEL cloud, where there is no direct access to
logs. In case of support requests we often can't correlate past
log-lines to the error that was reported.

This change adds the possibility to return the parent error in the
response to the OIDC client. For the moment this only applies to JSON
body responses, not error redirects to the RP.

# How the Problems Are Solved

- New instance-level feature flag: `debug_oidc_parent_error`
- Use the new `WithReturnParentToClient()` function from the oidc lib
introduced in https://github.com/zitadel/oidc/pull/629 for all cases
where `WithParent` was already used and the request context is
available.

# Additional Changes

none

# Additional Context

- Depends on: https://github.com/zitadel/oidc/pull/629
- Related to: https://github.com/zitadel/zitadel/issues/8362

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-08-20 06:45:24 +00:00
Tim Möhlmann
ab4daa0ebb
fix(webkeys): remove include private key from projection index (#8436)
# Which Problems Are Solved

When Web keys with a large RSA Bitsize were generated, the new web key
did not get projected.
Zitadel logs printed the following errors:

```
time="2024-08-15T09:18:47Z" level=debug msg="execution of statements failed" caller="/home/tim/Repositories/zitadel/zitadel/internal/eventstore/handler/v2/handler.go:518" error="statement failed: ID=CRDB-pKtsr Message=exec failed Parent=(ERROR: index row size 3400 exceeds btree version 4 maximum 2704 for index \"web_keys_web_key_state_idx\" (SQLSTATE 54000))" lastProcessedIndex=-1 projection=projections.web_keys
```


# How the Problems Are Solved

Remove the the `WithInclude` option with the Private key from the
`web_keys` projection index and bump the projection version.

# Additional Changes

- Correct RPC summaries for documentation
- Fix docs slug in sidebar.js

# Additional Context

- Found during release review.
2024-08-16 11:41:09 +00:00
Stefan Benz
5fab533e37
feat: org v2 ListOrganizations (#8411)
# Which Problems Are Solved

Org v2 service does not have a ListOrganizations endpoint.

# How the Problems Are Solved

Implement ListOrganizations endpoint.

# Additional Changes

- moved descriptions in the protos to comments
- corrected the RemoveNoPermissions for the ListUsers, to get the
correct TotalResults

# Additional Context

For new typescript login
2024-08-15 06:37:06 +02:00
Stefan Benz
3e3d46ac0d
feat: idp v2 api GetIDPByID (#8425)
# Which Problems Are Solved

GetIDPByID as endpoint in the API v2 so that it can be available for the
new login.

# How the Problems Are Solved

Create GetIDPByID endpoint with IDP v2 API, throught the GetProviderByID
implementation from admin and management API.

# Additional Changes

- Remove the OwnerType attribute from the response, as the information
is available through the resourceOwner.
- correct refs to messages in proto which are used for doc generation
- renaming of elements for API v3

# Additional Context

Closes #8337

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-08-14 18:18:29 +00:00
Tim Möhlmann
64a3bb3149
feat(v3alpha): web key resource (#8262)
# Which Problems Are Solved

Implement a new API service that allows management of OIDC signing web
keys.
This allows users to manage rotation of the instance level keys. which
are currently managed based on expiry.

The API accepts the generation of the following key types and
parameters:

- RSA keys with 2048, 3072 or 4096 bit in size and:
  - Signing with SHA-256 (RS256)
  - Signing with SHA-384 (RS384)
  - Signing with SHA-512 (RS512)
- ECDSA keys with
  - P256 curve
  - P384 curve
  - P512 curve
- ED25519 keys

# How the Problems Are Solved

Keys are serialized for storage using the JSON web key format from the
`jose` library. This is the format that will be used by OIDC for
signing, verification and publication.

Each instance can have a number of key pairs. All existing public keys
are meant to be used for token verification and publication the keys
endpoint. Keys can be activated and the active private key is meant to
sign new tokens. There is always exactly 1 active signing key:

1. When the first key for an instance is generated, it is automatically
activated.
2. Activation of the next key automatically deactivates the previously
active key.
3. Keys cannot be manually deactivated from the API
4. Active keys cannot be deleted

# Additional Changes

- Query methods that later will be used by the OIDC package are already
implemented. Preparation for #8031
- Fix indentation in french translation for instance event
- Move user_schema translations to consistent positions in all
translation files

# Additional Context

- Closes #8030
- Part of #7809

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-08-14 14:18:14 +00:00
Tim Möhlmann
fcda6580ff
fix(query): print log line on secret generator error (#8424)
# Which Problems Are Solved

Log some details when a secret generator is not found.
This should help us debugging such issue.

# How the Problems Are Solved

When a secret generator by type query fails,
we log the generator type and instance id for which
the generator was requested.

# Additional Changes

- none

# Additional Context

- Related to https://github.com/zitadel/zitadel/issues/8379
- Also encountered in https://github.com/zitadel/zitadel/pull/8407
2024-08-13 14:52:43 +02:00
Elio Bischof
042c438813
feat(v3alpha): read actions (#8357)
# Which Problems Are Solved

The current v3alpha actions APIs don't exactly adhere to the [new
resources API
design](https://zitadel.com/docs/apis/v3#standard-resources).

# How the Problems Are Solved

- **Improved ID access**: The aggregate ID is added to the resource
details object, so accessing resource IDs and constructing proto
messages for resources is easier
- **Explicit Instances**: Optionally, the instance can be explicitly
given in each request
- **Pagination**: A default search limit and a max search limit are
added to the defaults.yaml. They apply to the new v3 APIs (currently
only actions). The search query defaults are changed to ascending by
creation date, because this makes the pagination results the most
deterministic. The creation date is also added to the object details.
The bug with updated creation dates is fixed for executions and targets.
- **Removed Sequences**: Removed Sequence from object details and
ProcessedSequence from search details

# Additional Changes

Object details IDs are checked in unit test only if an empty ID is
expected. Centralizing the details check also makes this internal object
more flexible for future evolutions.

# Additional Context

- Closes #8169 
- Depends on https://github.com/zitadel/zitadel/pull/8225

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-08-12 22:32:01 +02:00
Livio Spring
3d071fc505
feat: trusted (instance) domains (#8369)
# Which Problems Are Solved

ZITADEL currently selects the instance context based on a HTTP header
(see https://github.com/zitadel/zitadel/issues/8279#issue-2399959845 and
checks it against the list of instance domains. Let's call it instance
or API domain.
For any context based URL (e.g. OAuth, OIDC, SAML endpoints, links in
emails, ...) the requested domain (instance domain) will be used. Let's
call it the public domain.
In cases of proxied setups, all exposed domains (public domains) require
the domain to be managed as instance domain.
This can either be done using the "ExternalDomain" in the runtime config
or via system API, which requires a validation through CustomerPortal on
zitadel.cloud.

# How the Problems Are Solved

- Two new headers / header list are added:
- `InstanceHostHeaders`: an ordered list (first sent wins), which will
be used to match the instance.
(For backward compatibility: the `HTTP1HostHeader`, `HTTP2HostHeader`
and `forwarded`, `x-forwarded-for`, `x-forwarded-host` are checked
afterwards as well)
- `PublicHostHeaders`: an ordered list (first sent wins), which will be
used as public host / domain. This will be checked against a list of
trusted domains on the instance.
- The middleware intercepts all requests to the API and passes a
`DomainCtx` object with the hosts and protocol into the context
(previously only a computed `origin` was passed)
- HTTP / GRPC server do not longer try to match the headers to instances
themself, but use the passed `http.DomainContext` in their interceptors.
- The `RequestedHost` and `RequestedDomain` from authz.Instance are
removed in favor of the `http.DomainContext`
- When authenticating to or signing out from Console UI, the current
`http.DomainContext(ctx).Origin` (already checked by instance
interceptor for validity) is used to compute and dynamically add a
`redirect_uri` and `post_logout_redirect_uri`.
- Gateway passes all configured host headers (previously only did
`x-zitadel-*`)
- Admin API allows to manage trusted domain

# Additional Changes

None

# Additional Context

- part of #8279 
- open topics: 
  - "single-instance" mode
  - Console UI
2024-07-31 18:00:38 +03:00
Stefan Benz
7d2d85f57c
feat: api v2beta to api v2 (#8283)
# Which Problems Are Solved

The v2beta services are stable but not GA.

# How the Problems Are Solved

The v2beta services are copied to v2. The corresponding v1 and v2beta
services are deprecated.

# Additional Context

Closes #7236

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-07-26 22:39:55 +02:00
Livio Spring
0e99c8356a
fix: correctly check user auth methods and enable button (#8342)
# Which Problems Are Solved

#8291 added backwards compatibilty for users who were created through
the user V2 API and want to sign in to the login UI.
There were however to issues, where users might be prompted to set a
password even if they already had one set or they would not be able to
submit the email verification code.

# How the Problems Are Solved

- Replaced `SearchUserAuthMethods `with `ListUserAuthMethodTypes` to
check for set up auth methods.
- Fixed page / javascript to disable submit button.

# Additional Changes

- Changed `ListActiveUserAuthMethodTypes ` to `ListUserAuthMethodTypes`
and a `activeOnly` boolean parameter

# Additional Context

- relates to #8291
- noticed internally on QA
2024-07-22 14:46:27 +02:00
Silvan
99c645cc60
refactor(database): exchange connection pool (#8325)
# Which Problems Are Solved

The connection pool of go uses a high amount of database connections.

# How the Problems Are Solved

The standard lib connection pool was replaced by `pgxpool.Pool`

# Additional Changes

The `db.BeginTx`-spans are removed because they cause to much noise in
the traces.

# Additional Context

- part of https://github.com/zitadel/zitadel/issues/7639
2024-07-17 15:16:02 +00:00
Stefan Benz
19561a092c
fix: add OrgIDQuery to the queries for organizations (#8312)
# Which Problems Are Solved

ListOrgs has no option to select for organizations specific to Ids.

# How the Problems Are Solved

Add OrgIDQuery to ListOrgs.

# Additional Changes

Clean up double mapping for the OrgQueries.

# Additional Context

- noted internally while checking performance issues (in Console)
2024-07-16 13:12:19 +00:00
Tim Möhlmann
7967e6f98b
perf(import): optimize search for domains claimed by other organizations (#8200)
# Which Problems Are Solved

Improve the performance of human imports by optimizing the query that
finds domains claimed by other organizations.

# How the Problems Are Solved

Use the fields search table introduced in
https://github.com/zitadel/zitadel/pull/8191 by storing each
organization domain as Object ID and the verified status as field value.

# Additional Changes

- Feature flag for this optimization

# Additional Context

- Performance improvements for import are evaluated and acted upon
internally at the moment

---------

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2024-07-05 09:36:00 +02:00
Brian Tajuddin
32b707cf46
feat(6222): remove @ and project from OIDC client ID (#8178)
# Which Problems Are Solved

The client ID for OIDC applications has an `@` in it, which is not
allowed in some 3rd-party systems (such as AWS).

# How the Problems Are Solved

Per @fforootd and @hifabienne in #6222, remove the project suffix and
the `@` from the client ID and just use the generated ID.

# Additional Changes

N/A

# Additional Context

- Closes #6222

---------

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-07-04 08:31:40 +00:00
Silvan
1d84635836
feat(eventstore): add search table (#8191)
# Which Problems Are Solved

To improve performance a new table and method is implemented on
eventstore. The goal of this table is to index searchable fields on
command side to use it on command and query side.

The table allows to store one primitive value (numeric, text) per row.

The eventstore framework is extended by the `Search`-method which allows
to search for objects.
The `Command`-interface is extended by the `SearchOperations()`-method
which does manipulate the the `search`-table.

# How the Problems Are Solved

This PR adds the capability of improving performance for command and
query side by using the `Search`-method of the eventstore instead of
using one of the `Filter`-methods.

# Open Tasks

- [x] Add feature flag
- [x] Unit tests
- [ ] ~~Benchmarks if needed~~
- [x] Ensure no behavior change
- [x] Add setup step to fill table with current data
- [x] Add projection which ensures data added between setup and start of
the new version are also added to the table

# Additional Changes

The `Search`-method is currently used by `ProjectGrant`-command side.

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/8094
2024-07-03 15:00:56 +00:00
Miguel Cabrerizo
c8e168ac69
fix: remove non-existent colums in getMembershipFromQuery queries (#7869)
In issue #7841 @mahmoodfathy commented an issue when the API call for
Listing My ZITADEL Manager Roles is called with any kind of query
(orgQuery, projectQuery, projectGrantQuery...). A column XXXXXX does not
exist (SQLSTATE 42703) error is thrown.

The issue was focused in getMembershipFromQuery where filtering queries
functions are called: prepareOrgMember, prepareIAMMember,
prepareProjectMember and prepareProjectGrantMember

Those functions allow queries for columns that are not members of the
table to be queried so I've added a conditional clause to avoid using
the queries that cannot be called.

For example, for prepareOrgMember, member.id, member.project_id and
member.grant_id columns are not added to the filter queries

```
for _, q := range query.Queries {
		if q.Col().table.name == membershipAlias.name &&
			!slices.Contains([]string{membershipIAMID.name, membershipProjectID.name, membershipGrantID.name}, q.Col().name) {
			builder = q.toQuery(builder)
		}
	}
	return builder.MustSql()
```

Here I show one screenshot where the error "column XXXXXX does not exist
(SQLSTATE 42703)" is no longer thrown using an orgQuery.


![image](https://github.com/zitadel/zitadel/assets/30386061/77621e69-71df-42de-b3c5-fa9b4dbf1b89)

Should close #7841 

### Definition of Ready

- [X] I am happy with the code
- [X] Short description of the feature/issue is added in the pr
description
- [X] PR is linked to the corresponding user story
- [X] Acceptance criteria are met
- [ ] All open todos and follow ups are defined in a new ticket and
justified
- [ ] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [X] No debug or dead code
- [X] My code has no repetitions
- [X] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [ ] Documentation/examples are up-to-date
- [ ] All non-functional requirements are met
- [X] Functionality of the acceptance criteria is checked manually on
the dev system.

---------

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-07-02 12:08:28 +00:00
Tim Möhlmann
f9742a58f4
fix(import): add tracing spans to all import related functions (#8160)
# Which Problems Are Solved

This fix adds tracing spans to all V1 API import related functions. This
is to troubleshoot import related performance issues reported to us.

# How the Problems Are Solved

Add a tracing span to `api/grpc/admin/import.go` and all related
functions that are called in the `command` package.

# Additional Changes

- none

# Additional Context

- Reported by internal communication
2024-06-19 12:56:33 +02:00
Livio Spring
fb8cd18f93
feat: password age policy (#8132)
# Which Problems Are Solved

Some organizations / customers have the requirement, that there users
regularly need to change their password.
ZITADEL already had the possibility to manage a `password age policy` (
thought the API) with the maximum amount of days a password should be
valid, resp. days after with the user should be warned of the upcoming
expiration.
The policy could not be managed though the Console UI and was not
checked in the Login UI.

# How the Problems Are Solved

- The policy can be managed in the Console UI's settings sections on an
instance and organization level.
- During an authentication in the Login UI, if a policy is set with an
expiry (>0) and the user's last password change exceeds the amount of
days set, the user will be prompted to change their password.
- The prompt message of the Login UI can be customized in the Custom
Login Texts though the Console and API on the instance and each
organization.
- The information when the user last changed their password is returned
in the Auth, Management and User V2 API.
- The policy can be retrieved in the settings service as `password
expiry settings`.

# Additional Changes

None.

# Additional Context

- closes #8081

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2024-06-18 11:27:44 +00:00
Tim Möhlmann
120ed0af73
feat(oidc): organization roles scope (#8120)
# Which Problems Are Solved

An admin / application might want to be able to reduce the amount of
roles returned in the token, for example if a user is granted to many
organizations or for specific cases where the application want to narrow
down the access for that token to a specific organization or multiple.
This can now be achieved by providing a scope with the id of the
organization, resp. multiple scopes for every organization, which should
be included.

```
urn:zitadel:iam:org:roles🆔{orgID}
```

**Note:** the new scope does not work when Introspection / Userinfo are
set to legacy mode.

# How the Problems Are Solved

The user info query now has two variants:

1. Variant that returns all organization authorization grants if the new
scope wasn't provided for backward compatibility.
2. Variant that filters the organizations based on the IDs passed in one
or more of the above scopes and returns only those authorization grants.

The query is defined as a `text/template` and both variants are rendered
once in package `init()`.

# Additional Changes

- In the integration tests `assertProjectRoleClaims` now also checks the
org IDs in the roles.

# Additional Context

- Closes #7996
2024-06-14 10:00:43 +02:00
Livio Spring
fb2b1610f9
fix(oidc): remove MFA requirement on ZITADEL API based on user auth methods (#8069)
# Which Problems Are Solved

Request to the ZITADEL API currently require multi factor authentication
if the user has set up any second factor.
However, the login UI will only prompt the user to check factors that
are allowed by the login policy.
This can lead to situations, where the user has set up a factor (e.g.
some OTP) which was not allowed by the policy, therefore will not have
to verify the factor, the ZITADEL API however will require the check
since the user has set it up.

# How the Problems Are Solved

The requirement for multi factor authentication based on the user's
authentication methods is removed when accessing the ZITADEL APIs.
Those requests will only require MFA in case the login policy does so
because of `requireMFA` or `requireMFAForLocalUsers`.

# Additional Changes

None.

# Additional Context

- a customer reached out to support
- discussed internally
- relates #7822 
- backport to 2.53.x
2024-06-12 12:24:17 +00:00
Livio Spring
931a6c7cce
perf(authZ): improve oidc session check (#8091)
# Which Problems Are Solved

Access token checks make sure that there have not been any termination
events (user locked, deactivated, signed out, ...) in the meantime. This
events were filtered based on the creation date of the last session
event, which might cause latency issues in the database.

# How the Problems Are Solved

- Changed the query to use `position` instead of `created_at`.
- removed `AwaitOpenTransactions`

# Additional Changes

Added the `position` field to the `ReadModel`.

# Additional Context

- relates to #8088
- part of #7639
- backport to 2.53.x
2024-06-12 09:11:36 +00:00
Livio Spring
b6c10c4c83
fix: consider oidc session events for authN milestones (#8089)
# Which Problems Are Solved

After migrating the access token events in #7822, milestones based on
authentication, resp. theses events would not be reached.

# How the Problems Are Solved

Additionally use the `oidc_session.Added` event to check for
`milestone.AuthenticationSucceededOnInstance` and
`milestone.AuthenticationSucceededOnApplication`.

# Additional Changes

None.

# Additional Context

- relates to #7822 
- noticed internally
2024-06-12 06:49:14 +02:00
Silvan
2243306ef6
feat(cmd): mirror (#7004)
# Which Problems Are Solved

Adds the possibility to mirror an existing database to a new one. 

For that a new command was added `zitadel mirror`. Including it's
subcommands for a more fine grained mirror of the data.

Sub commands:

* `zitadel mirror eventstore`: copies only events and their unique
constraints
* `zitadel mirror system`: mirrors the data of the `system`-schema
*  `zitadel mirror projections`: runs all projections
*  `zitadel mirror auth`: copies auth requests
* `zitadel mirror verify`: counts the amount of rows in the source and
destination database and prints the diff.

The command requires one of the following flags:
* `--system`: copies all instances of the system
* `--instance <instance-id>`, `--instance <comma separated list of
instance ids>`: copies only the defined instances

The command is save to execute multiple times by adding the
`--replace`-flag. This replaces currently existing data except of the
`events`-table

# Additional Changes

A `--for-mirror`-flag was added to `zitadel setup` to prepare the new
database. The flag skips the creation of the first instances and initial
run of projections.

It is now possible to skip the creation of the first instance during
setup by setting `FirstInstance.Skip` to true in the steps
configuration.

# Additional info

It is currently not possible to merge multiple databases. See
https://github.com/zitadel/zitadel/issues/7964 for more details.

It is currently not possible to use files. See
https://github.com/zitadel/zitadel/issues/7966 for more information.

closes https://github.com/zitadel/zitadel/issues/7586
closes https://github.com/zitadel/zitadel/issues/7486

### Definition of Ready

- [x] I am happy with the code
- [x] Short description of the feature/issue is added in the pr
description
- [x] PR is linked to the corresponding user story
- [x] Acceptance criteria are met
- [x] All open todos and follow ups are defined in a new ticket and
justified
- [x] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [x] No debug or dead code
- [x] My code has no repetitions
- [x] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [x] Documentation/examples are up-to-date
- [x] All non-functional requirements are met
- [x] Functionality of the acceptance criteria is checked manually on
the dev system.

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-05-30 09:35:30 +00:00
Livio Spring
ec222a13d7
fix(oidc): IDP and passwordless user auth methods (#7998)
# Which Problems Are Solved

As already mentioned and (partially) fixed in #7992 we discovered,
issues with v2 tokens that where obtained through an IDP, with
passwordless authentication or with password authentication (wihtout any
2FA set up) using the v1 login for zitadel API calls
- (Previous) authentication through an IdP is now correctly treated as
auth method in case of a reauth even when the user is not redirected to
the IdP
- There were some cases where passwordless authentication was
successfully checked but not correctly set as auth method, which denied
access to ZITADEL API
- Users with password and passwordless, but no 2FA set up which
authenticate just wich password can access the ZITADEL API again

Additionally while testing we found out that because of #7969 the login
UI could completely break / block with the following error:
`sql: Scan error on column index 3, name "state": converting NULL to
int32 is unsupported (Internal)`
# How the Problems Are Solved

- IdP checks are treated the same way as other factors and it's ensured
that a succeeded check within the configured timeframe will always
provide the idp auth method
- `MFATypesAllowed` checks for possible passwordless authentication
- As with the v1 login, the token check now only requires MFA if the
policy is set or the user has 2FA set up
- UserAuthMethodsRequirements now always uses the correctly policy to
check for MFA enforcement
- `State` column is handled as nullable and additional events set the
state to active (as before #7969)

# Additional Changes

- Console now also checks for 403 (mfa required) errors (e.g. after
setting up the first 2FA in console) and redirects the user to the login
UI (with the current id_token as id_token_hint)
- Possible duplicates in auth methods / AMRs are removed now as well.

# Additional Context

- Bugs were introduced in #7822 and # and 7969 and only part of a
pre-release.
- partially already fixed with #7992
- Reported internally.
2024-05-28 08:59:49 +00:00
Silvan
5cfccb1442
fix(actions): correct statements to query targets (#8006)
# Which Problems Are Solved

Curretly loading targets fails on cockraochdb because the order of
`with`-statements is wrong.

# How the Problems Are Solved

Changed the order of queries in the statements.
2024-05-24 14:13:36 +00:00
Silvan
0bfcf2c317
refactor(query): use new packages for org by id query (#7826)
If the feature is enabled the new packages are used to query org by id

Part of: https://github.com/zitadel/zitadel/issues/7639

### Definition of Ready

- [x] I am happy with the code
- [x] Short description of the feature/issue is added in the pr
description
- [x] PR is linked to the corresponding user story
- [ ] Acceptance criteria are met
- [ ] All open todos and follow ups are defined in a new ticket and
justified
- [ ] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [x] No debug or dead code
- [x] My code has no repetitions
- [ ] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [ ] Documentation/examples are up-to-date
- [ ] All non-functional requirements are met
- [x] Functionality of the acceptance criteria is checked manually on
the dev system.
2024-05-24 13:32:57 +02:00
Tim Möhlmann
f5e9d4f57f
fix(oidc): IDP and machine user auth methods (#7992)
# Which Problems Are Solved

After https://github.com/zitadel/zitadel/pull/7822 was merged we
discovered that
v2 tokens that where obtained through an IDP using the v1 login, can't
be used for
zitadel API calls.

- Because we used to store the AMR claim on the auth request, but
internally use the domain.UserAuthMethod type. AMR has no notion of an
IDP login, so that "factor" was lost
during conversion. Rendering those v2 tokens invalid on the zitadel API.
- A wrong check on machine user tokens falsly allowed some tokens to be
valid
- The client ID was set to tokens from client credentials and JWT
profile, which made client queries fail in the validation middleware.
The middleware expects client ID unset for machine users.

# How the Problems Are Solved

Store the domain.AuthMethods directly in  the auth requests and session,
instead of using AMR claims with lossy conversion.

- IDPs have seperate auth method, which is not an AMR claim
- Machine users are treated specialy, eg auth methods are not required.
- Do not set the client ID for client credentials and JWT profile

# Additional Changes

Cleaned up mostly unused `oidc.getInfoFromRequest()`.

# Additional Context

- Bugs were introduced in https://github.com/zitadel/zitadel/pull/7822
and not yet part of a release.
- Reported internally.
2024-05-23 05:35:10 +00:00
Livio Spring
e57a9b57c8
feat(saml): allow setting nameid-format and alternative mapping for transient format (#7979)
# Which Problems Are Solved

ZITADEL currently always uses
`urn:oasis:names:tc:SAML:2.0:nameid-format:persistent` in SAML requests,
relying on the IdP to respect that flag and always return a peristent
nameid in order to be able to map the external user with an existing
user (idp link) in ZITADEL.
In case the IdP however returns a
`urn:oasis:names:tc:SAML:2.0:nameid-format:transient` (transient)
nameid, the attribute will differ between each request and it will not
be possible to match existing users.

# How the Problems Are Solved

This PR adds the following two options on SAML IdP:
- **nameIDFormat**: allows to set the nameid-format used in the SAML
Request
- **transientMappingAttributeName**: allows to set an attribute name,
which will be used instead of the nameid itself in case the returned
nameid-format is transient

# Additional Changes

To reduce impact on current installations, the `idp_templates6_saml`
table is altered with the two added columns by a setup job. New
installations will automatically get the table with the two columns
directly.
All idp unit tests are updated to use `expectEventstore` instead of the
deprecated `eventstoreExpect`.

# Additional Context

Closes #7483
Closes #7743

---------

Co-authored-by: peintnermax <max@caos.ch>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-05-23 05:04:07 +00:00
Livio Spring
b924fd59a5
fix: allow sorting of MyProjectOrgs (by name) (#7970) 2024-05-17 09:52:15 +02:00
Tim Möhlmann
8e0c8393e9
perf(oidc): optimize token creation (#7822)
* implement code exchange

* port tokenexchange to v2 tokens

* implement refresh token

* implement client credentials

* implement jwt profile

* implement device token

* cleanup unused code

* fix current unit tests

* add user agent unit test

* unit test domain package

* need refresh token as argument

* test commands create oidc session

* test commands device auth

* fix device auth build error

* implicit for oidc session API

* implement authorize callback handler for legacy implicit mode

* upgrade oidc module to working draft

* add missing auth methods and time

* handle all errors in defer

* do not fail auth request on error

the oauth2 Go client automagically retries on any error. If we fail the auth request on the first error, the next attempt will always fail with the Errors.AuthRequest.NoCode, because the auth request state is already set to failed.
The original error is then already lost and the oauth2 library does not return the original error.

Therefore we should not fail the auth request.

Might be worth discussing and perhaps send a bug report to Oauth2?

* fix code flow tests by explicitly setting code exchanged

* fix unit tests in command package

* return allowed scope from client credential client

* add device auth done reducer

* carry nonce thru session into ID token

* fix token exchange integration tests

* allow project role scope prefix in client credentials client

* gci formatting

* do not return refresh token in client credentials and jwt profile

* check org scope

* solve linting issue on authorize callback error

* end session based on v2 session ID

* use preferred language and user agent ID for v2 access tokens

* pin oidc v3.23.2

* add integration test for jwt profile and client credentials with org scopes

* refresh token v1 to v2

* add user token v2 audit event

* add activity trigger

* cleanup and set panics for unused methods

* use the encrypted code for v1 auth request get by code

* add missing event translation

* fix pipeline errors (hopefully)

* fix another test

* revert pointer usage of preferred language

* solve browser info panic in device auth

* remove duplicate entries in AMRToAuthMethodTypes to prevent future `mfa` claim

* revoke v1 refresh token to prevent reuse

* fix terminate oidc session

* always return a new refresh toke in refresh token grant

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-05-16 07:07:56 +02:00