Commit Graph

482 Commits

Author SHA1 Message Date
Livio Spring
a07b2f4677
feat: invite user link (#8578)
# Which Problems Are Solved

As an administrator I want to be able to invite users to my application
with the API V2, some user data I will already prefil, the user should
add the authentication method themself (password, passkey, sso).

# How the Problems Are Solved

- A user can now be created with a email explicitly set to false.
- If a user has no verified email and no authentication method, an
`InviteCode` can be created through the User V2 API.
  - the code can be returned or sent through email
- additionally `URLTemplate` and an `ApplicatioName` can provided for
the email
- The code can be resent and verified through the User V2 API
- The V1 login allows users to verify and resend the code and set a
password (analog user initialization)
- The message text for the user invitation can be customized

# Additional Changes

- `verifyUserPasskeyCode` directly uses `crypto.VerifyCode` (instead of
`verifyEncryptedCode`)
- `verifyEncryptedCode` is removed (unnecessarily queried for the code
generator)

# Additional Context

- closes #8310
- TODO: login V2 will have to implement invite flow:
https://github.com/zitadel/typescript/issues/166
2024-09-11 10:53:55 +00:00
Tim Möhlmann
3aba942162
feat: add debug events API (#8533)
# Which Problems Are Solved

Add a debug API which allows pushing a set of events to be reduced in a
dedicated projection.
The events can carry a sleep duration which simulates a slow query
during projection handling.

# How the Problems Are Solved

- `CreateDebugEvents` allows pushing multiple events which simulate the
lifecycle of a resource. Each event has a `projectionSleep` field, which
issues a `pg_sleep()` statement query in the projection handler :
  - Add
  - Change
  - Remove
- `ListDebugEventsStates` list the current state of the projection,
optionally with a Trigger
- `GetDebugEventsStateByID` get the current state of the aggregate ID in
the projection, optionally with a Trigger


# Additional Changes

- none

# Additional Context

-  Allows reproduction of https://github.com/zitadel/zitadel/issues/8517
2024-09-11 08:24:00 +00:00
Tim Möhlmann
d2e0ac07f1
chore(tests): use a coverage server binary (#8407)
# Which Problems Are Solved

Use a single server instance for API integration tests. This optimizes
the time taken for the integration test pipeline,
because it allows running tests on multiple packages in parallel. Also,
it saves time by not start and stopping a zitadel server for every
package.

# How the Problems Are Solved

- Build a binary with `go build -race -cover ....`
- Integration tests only construct clients. The server remains running
in the background.
- The integration package and tested packages now fully utilize the API.
No more direct database access trough `query` and `command` packages.
- Use Makefile recipes to setup, start and stop the server in the
background.
- The binary has the race detector enabled
- Init and setup jobs are configured to halt immediately on race
condition
- Because the server runs in the background, races are only logged. When
the server is stopped and race logs exist, the Makefile recipe will
throw an error and print the logs.
- Makefile recipes include logic to print logs and convert coverage
reports after the server is stopped.
- Some tests need a downstream HTTP server to make requests, like quota
and milestones. A new `integration/sink` package creates an HTTP server
and uses websockets to forward HTTP request back to the test packages.
The package API uses Go channels for abstraction and easy usage.

# Additional Changes

- Integration test files already used the `//go:build integration`
directive. In order to properly split integration from unit tests,
integration test files need to be in a `integration_test` subdirectory
of their package.
- `UseIsolatedInstance` used to overwrite the `Tester.Client` for each
instance. Now a `Instance` object is returned with a gRPC client that is
connected to the isolated instance's hostname.
- The `Tester` type is now `Instance`. The object is created for the
first instance, used by default in any test. Isolated instances are also
`Instance` objects and therefore benefit from the same methods and
values. The first instance and any other us capable of creating an
isolated instance over the system API.
- All test packages run in an Isolated instance by calling
`NewInstance()`
- Individual tests that use an isolated instance use `t.Parallel()`

# Additional Context

- Closes #6684
- https://go.dev/doc/articles/race_detector
- https://go.dev/doc/build-cover

---------

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-09-06 14:47:57 +02:00
Silvan
b522588d98
fix(eventstore): precise decimal (#8527)
# Which Problems Are Solved

Float64 which was used for the event.Position field is [not precise in
go and gets rounded](https://github.com/golang/go/issues/47300). This
can lead to unprecies position tracking of events and therefore
projections especially on cockcoachdb as the position used there is a
big number.

example of a unprecies position:
exact: 1725257931223002628
float64: 1725257931223002624.000000

# How the Problems Are Solved

The float64 was replaced by
[github.com/jackc/pgx-shopspring-decimal](https://github.com/jackc/pgx-shopspring-decimal).

# Additional Changes

Correct behaviour of makefile for load tests.
Rename `latestSequence`-queries to `latestPosition`
2024-09-06 12:19:19 +03:00
Livio Spring
9ec9ad4314
feat(oidc): sid claim for id_tokens issued through login V1 (#8525)
# Which Problems Are Solved

id_tokens issued for auth requests created through the login UI
currently do not provide a sid claim.
This is due to the fact that (SSO) sessions for the login UI do not have
one and are only computed by the userAgent(ID), the user(ID) and the
authentication checks of the latter.

This prevents client to track sessions and terminate specific session on
the end_session_endpoint.

# How the Problems Are Solved

- An `id` column is added to the `auth.user_sessions` table.
- The `id` (prefixed with `V1_`) is set whenever a session is added or
updated to active (from terminated)
- The id is passed to the `oidc session` (as v2 sessionIDs), to expose
it as `sid` claim

# Additional Changes

- refactored `getUpdateCols` to handle different column value types and
add arguments for query

# Additional Context

- closes #8499 
- relates to #8501
2024-09-03 13:19:00 +00:00
Stefan Benz
41ae35f2ef
feat: add schema user create and remove (#8494)
# Which Problems Are Solved

Added functionality that user with a userschema can be created and
removed.

# How the Problems Are Solved

Added logic and moved APIs so that everything is API v3 conform.

# Additional Changes

- move of user and userschema API to resources folder
- changed testing and parameters
- some renaming

# Additional Context

closes #7308

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-08-28 19:46:45 +00:00
Tim Möhlmann
fd0c15dd4f
feat(oidc): use web keys for token signing and verification (#8449)
# Which Problems Are Solved

Use web keys, managed by the `resources/v3alpha/web_keys` API, for OIDC
token signing and verification,
as well as serving the public web keys on the jwks / keys endpoint.
Response header on the keys endpoint now allows caching of the response.
This is now "safe" to do since keys can be created ahead of time and
caches have sufficient time to pickup the change before keys get
enabled.

# How the Problems Are Solved

- The web key format is used in the `getSignerOnce` function in the
`api/oidc` package.
- The public key cache is changed to get and store web keys.
- The jwks / keys endpoint returns the combined set of valid "legacy"
public keys and all available web keys.
- Cache-Control max-age default to 5 minutes and is configured in
`defaults.yaml`.

When the web keys feature is enabled, fallback mechanisms are in place
to obtain and convert "legacy" `query.PublicKey` as web keys when
needed. This allows transitioning to the feature without invalidating
existing tokens. A small performance overhead may be noticed on the keys
endpoint, because 2 queries need to be run sequentially. This will
disappear once the feature is stable and the legacy code gets cleaned
up.

# Additional Changes

- Extend legacy key lifetimes so that tests can be run on an existing
database with more than 6 hours apart.
- Discovery endpoint returns all supported algorithms when the Web Key
feature is enabled.

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/8031
- Part of https://github.com/zitadel/zitadel/issues/7809
- After https://github.com/zitadel/oidc/pull/637
- After https://github.com/zitadel/oidc/pull/638
2024-08-23 14:43:46 +02:00
Livio Spring
c8e2a3bd49
feat: enable application performance profiling (#8442)
# Which Problems Are Solved

To have more insight on the performance, CPU and memory usage of
ZITADEL, we want to enable profiling.

# How the Problems Are Solved

- Allow profiling by configuration.
- Provide Google Cloud Profiler as first implementation

# Additional Changes

None.

# Additional Context

There were possible memory leaks reported:
https://discord.com/channels/927474939156643850/1273210227918897152

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2024-08-16 13:26:53 +00:00
Stefan Benz
3e3d46ac0d
feat: idp v2 api GetIDPByID (#8425)
# Which Problems Are Solved

GetIDPByID as endpoint in the API v2 so that it can be available for the
new login.

# How the Problems Are Solved

Create GetIDPByID endpoint with IDP v2 API, throught the GetProviderByID
implementation from admin and management API.

# Additional Changes

- Remove the OwnerType attribute from the response, as the information
is available through the resourceOwner.
- correct refs to messages in proto which are used for doc generation
- renaming of elements for API v3

# Additional Context

Closes #8337

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-08-14 18:18:29 +00:00
Tim Möhlmann
64a3bb3149
feat(v3alpha): web key resource (#8262)
# Which Problems Are Solved

Implement a new API service that allows management of OIDC signing web
keys.
This allows users to manage rotation of the instance level keys. which
are currently managed based on expiry.

The API accepts the generation of the following key types and
parameters:

- RSA keys with 2048, 3072 or 4096 bit in size and:
  - Signing with SHA-256 (RS256)
  - Signing with SHA-384 (RS384)
  - Signing with SHA-512 (RS512)
- ECDSA keys with
  - P256 curve
  - P384 curve
  - P512 curve
- ED25519 keys

# How the Problems Are Solved

Keys are serialized for storage using the JSON web key format from the
`jose` library. This is the format that will be used by OIDC for
signing, verification and publication.

Each instance can have a number of key pairs. All existing public keys
are meant to be used for token verification and publication the keys
endpoint. Keys can be activated and the active private key is meant to
sign new tokens. There is always exactly 1 active signing key:

1. When the first key for an instance is generated, it is automatically
activated.
2. Activation of the next key automatically deactivates the previously
active key.
3. Keys cannot be manually deactivated from the API
4. Active keys cannot be deleted

# Additional Changes

- Query methods that later will be used by the OIDC package are already
implemented. Preparation for #8031
- Fix indentation in french translation for instance event
- Move user_schema translations to consistent positions in all
translation files

# Additional Context

- Closes #8030
- Part of #7809

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-08-14 14:18:14 +00:00
Elio Bischof
042c438813
feat(v3alpha): read actions (#8357)
# Which Problems Are Solved

The current v3alpha actions APIs don't exactly adhere to the [new
resources API
design](https://zitadel.com/docs/apis/v3#standard-resources).

# How the Problems Are Solved

- **Improved ID access**: The aggregate ID is added to the resource
details object, so accessing resource IDs and constructing proto
messages for resources is easier
- **Explicit Instances**: Optionally, the instance can be explicitly
given in each request
- **Pagination**: A default search limit and a max search limit are
added to the defaults.yaml. They apply to the new v3 APIs (currently
only actions). The search query defaults are changed to ascending by
creation date, because this makes the pagination results the most
deterministic. The creation date is also added to the object details.
The bug with updated creation dates is fixed for executions and targets.
- **Removed Sequences**: Removed Sequence from object details and
ProcessedSequence from search details

# Additional Changes

Object details IDs are checked in unit test only if an empty ID is
expected. Centralizing the details check also makes this internal object
more flexible for future evolutions.

# Additional Context

- Closes #8169 
- Depends on https://github.com/zitadel/zitadel/pull/8225

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-08-12 22:32:01 +02:00
Silvan
cd3ffbd3eb
fix(mirror): use correct statements on push (#8414)
# Which Problems Are Solved

The mirror command used the wrong position to filter for events if
different database technologies for source and destination were used.

# How the Problems Are Solved

The statements which diverge are stored on the client so that different
technologies can use different statements.

# Additional Context

- https://discord.com/channels/927474939156643850/1256396896243552347
2024-08-12 10:33:45 +00:00
Livio Spring
3f25e36fbd
fix: provide device auth config (#8419)
# Which Problems Are Solved

There was no default configuration for `DeviceAuth`, which makes it
impossible to override by environment variables.
Additionally, a custom `CharAmount` value would overwrite also the
`DashInterval`.

# How the Problems Are Solved

- added to defaults.yaml
- fixed customization

# Additional Changes

None.

# Additional Context

- noticed during a customer request
2024-08-12 12:55:07 +03:00
Livio Spring
3d071fc505
feat: trusted (instance) domains (#8369)
# Which Problems Are Solved

ZITADEL currently selects the instance context based on a HTTP header
(see https://github.com/zitadel/zitadel/issues/8279#issue-2399959845 and
checks it against the list of instance domains. Let's call it instance
or API domain.
For any context based URL (e.g. OAuth, OIDC, SAML endpoints, links in
emails, ...) the requested domain (instance domain) will be used. Let's
call it the public domain.
In cases of proxied setups, all exposed domains (public domains) require
the domain to be managed as instance domain.
This can either be done using the "ExternalDomain" in the runtime config
or via system API, which requires a validation through CustomerPortal on
zitadel.cloud.

# How the Problems Are Solved

- Two new headers / header list are added:
- `InstanceHostHeaders`: an ordered list (first sent wins), which will
be used to match the instance.
(For backward compatibility: the `HTTP1HostHeader`, `HTTP2HostHeader`
and `forwarded`, `x-forwarded-for`, `x-forwarded-host` are checked
afterwards as well)
- `PublicHostHeaders`: an ordered list (first sent wins), which will be
used as public host / domain. This will be checked against a list of
trusted domains on the instance.
- The middleware intercepts all requests to the API and passes a
`DomainCtx` object with the hosts and protocol into the context
(previously only a computed `origin` was passed)
- HTTP / GRPC server do not longer try to match the headers to instances
themself, but use the passed `http.DomainContext` in their interceptors.
- The `RequestedHost` and `RequestedDomain` from authz.Instance are
removed in favor of the `http.DomainContext`
- When authenticating to or signing out from Console UI, the current
`http.DomainContext(ctx).Origin` (already checked by instance
interceptor for validity) is used to compute and dynamically add a
`redirect_uri` and `post_logout_redirect_uri`.
- Gateway passes all configured host headers (previously only did
`x-zitadel-*`)
- Admin API allows to manage trusted domain

# Additional Changes

None

# Additional Context

- part of #8279 
- open topics: 
  - "single-instance" mode
  - Console UI
2024-07-31 18:00:38 +03:00
Elio Bischof
cc3ec1e2a7
feat(v3alpha): write actions (#8225)
# Which Problems Are Solved

The current v3alpha actions APIs don't exactly adhere to the [new
resources API
design](https://zitadel.com/docs/apis/v3#standard-resources).

# How the Problems Are Solved

- **Breaking**: The current v3alpha actions APIs are removed. This is
breaking.
- **Resource Namespace**: New v3alpha actions APIs for targets and
executions are added under the namespace /resources.
- **Feature Flag**: New v3alpha actions APIs still have to be activated
using the actions feature flag
- **Reduced Executions Overhead**: Executions are managed similar to
settings according to the new API design: an empty list of targets
basically makes an execution a Noop. So a single method, SetExecution is
enough to cover all use cases. Noop executions are not returned in
future search requests.
- **Compatibility**: The executions created with previous v3alpha APIs
are still available to be managed with the new executions API.

# Additional Changes

- Removed integration tests which test executions but rely on readable
targets. They are added again with #8169

# Additional Context

Closes #8168
2024-07-31 14:42:12 +02:00
Stefan Benz
7d2d85f57c
feat: api v2beta to api v2 (#8283)
# Which Problems Are Solved

The v2beta services are stable but not GA.

# How the Problems Are Solved

The v2beta services are copied to v2. The corresponding v1 and v2beta
services are deprecated.

# Additional Context

Closes #7236

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-07-26 22:39:55 +02:00
Elio Bischof
693e27b906
fix: remove default TOS and privacy links (#8122)
# Which Problems Are Solved

The default terms of service and privacy policy links are applied to all
new ZITADEL instances, also for self hosters. However, the links
contents don't apply to self-hosters.

# How the Problems Are Solved

The links are removed from the DefaultInstance section in the
*defaults.yaml* file.
By default, the links are not shown anymore in the hosted login pages.
They can still be configured using the privacy policy.

# Additional Context

- Found because of a support request
2024-07-25 08:39:10 +02:00
Livio Spring
8d13247413
revert: "feat(IDP): use single callback endpoint"
This reverts commit e126ccc9aa.

# Which Problems Are Solved

#8295 introduced the possibility to handle idps on a single callback,
but broke current setups.

# How the Problems Are Solved

- Revert the change until a proper solution is found. Revert is needed
as docs were also changed.

# Additional Changes

None.

# Additional Context

- relates to #8295
2024-07-24 12:29:05 +00:00
mffap
91dcebbfc9
docs(migrate): update password section (#8232)
# Which Problems Are Solved

Password reset in combination of md5 was not clear

# How the Problems Are Solved

Some additional information in the docs
2024-07-23 14:13:35 +00:00
Silvan
0ea3c5691f
fix(mirror): read config correctly (#8330)
# Which Problems Are Solved

The mirror command read the configurations in the wrong order

# How the Problems Are Solved

The Pre execution run of `mirror` reads the default config first and
then applies the custom configs
2024-07-18 14:00:58 +00:00
Silvan
99c645cc60
refactor(database): exchange connection pool (#8325)
# Which Problems Are Solved

The connection pool of go uses a high amount of database connections.

# How the Problems Are Solved

The standard lib connection pool was replaced by `pgxpool.Pool`

# Additional Changes

The `db.BeginTx`-spans are removed because they cause to much noise in
the traces.

# Additional Context

- part of https://github.com/zitadel/zitadel/issues/7639
2024-07-17 15:16:02 +00:00
Silvan
6c1df83071
chore: set defaults for fields projections (#8311)
# Which Problems Are Solved

Bigger systems need to process many events during the initialisation
phase of the `eventstore.fields`-table. During setup these calls can
time out.

# How the Problems Are Solved

Changed the default behaviour of these projections to not time out and
increased the bulk limit.
2024-07-17 05:23:29 +00:00
Livio Spring
e126ccc9aa
feat(IDP): use single callback endpoint (#8295)
# Which Problems Are Solved

Both the login UI and the IdP intent flow have their own IdP callback
endpoints.

This makes configuration hard to impossible (e.g. Github only allows one
endpoint) for customers.

# How the Problems Are Solved

- The login UI prefixes the `state` parameter when creating an auth /
SAML request.
- All requests now use the `/idp/callback` or the corresponding
variation (e.g. SAML)
- On callback, the state, resp. its prefix is checked. In case of the
login UI prefix, the request will be forwarded to the existing login UI
handler without the prefix state.
Existing setups will therefore not be affected and also requests started
before this release can be handled without any impact.
- Console only lists the "new" endpoint(s). Any
`/login/externalidp/callback` is removed.

# Additional Changes

- Cleaned up some images  from the IdP documentation.
- fix the error handling in `handleExternalNotFoundOptionCheck`

# Additional Context

- closes #8236
2024-07-16 13:58:22 +00:00
Norman-Lee
1e3b350042
feat(cmd): Added machine ID mode to zitadel start up logs (#8251)
# Which Problems Are Solved

Logs the type of sonyflake strategy used for generating unique machine
IDs

# How the Problems Are Solved

- Created function to log machine id strategy on the start up logs

# Additional Changes
- Added public function for retrieving current strategy set by
configuration

# Additional Context
- Closes #7750
2024-07-16 09:53:57 +00:00
Silvan
23bebc7e30
fix(fields): add index to improve search by aggregate (#8267)
# Which Problems Are Solved

During performance testing of the `eventstore.fields` table we found
some long running queries which searched for the aggregate id.

# How the Problems Are Solved

A new index was added to the `eventstore.fields`-table called
`f_aggregate_object_type_idx`.

# Additional Changes

None

# Additional Context

- Table was added in https://github.com/zitadel/zitadel/pull/8191
- Part of https://github.com/zitadel/zitadel/issues/7639
2024-07-08 15:54:19 +00:00
Tim Möhlmann
7967e6f98b
perf(import): optimize search for domains claimed by other organizations (#8200)
# Which Problems Are Solved

Improve the performance of human imports by optimizing the query that
finds domains claimed by other organizations.

# How the Problems Are Solved

Use the fields search table introduced in
https://github.com/zitadel/zitadel/pull/8191 by storing each
organization domain as Object ID and the verified status as field value.

# Additional Changes

- Feature flag for this optimization

# Additional Context

- Performance improvements for import are evaluated and acted upon
internally at the moment

---------

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2024-07-05 09:36:00 +02:00
Florian Forster
224dc382d0
docs(cmd): document that human and service user can be used in the default instance (#8229)
# Which Problems Are Solved

This solves a documentation issue raised here
https://github.com/zitadel/zitadel/discussions/8217#discussioncomment-9936216

The behavior was changed with
https://github.com/zitadel/zitadel/issues/6430

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-07-04 11:39:28 +00:00
Silvan
1d84635836
feat(eventstore): add search table (#8191)
# Which Problems Are Solved

To improve performance a new table and method is implemented on
eventstore. The goal of this table is to index searchable fields on
command side to use it on command and query side.

The table allows to store one primitive value (numeric, text) per row.

The eventstore framework is extended by the `Search`-method which allows
to search for objects.
The `Command`-interface is extended by the `SearchOperations()`-method
which does manipulate the the `search`-table.

# How the Problems Are Solved

This PR adds the capability of improving performance for command and
query side by using the `Search`-method of the eventstore instead of
using one of the `Filter`-methods.

# Open Tasks

- [x] Add feature flag
- [x] Unit tests
- [ ] ~~Benchmarks if needed~~
- [x] Ensure no behavior change
- [x] Add setup step to fill table with current data
- [x] Add projection which ensures data added between setup and start of
the new version are also added to the table

# Additional Changes

The `Search`-method is currently used by `ProjectGrant`-command side.

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/8094
2024-07-03 15:00:56 +00:00
Tim Möhlmann
dc170dc46e
feat(crypto): support md5 plain for imported password hashes (#8189)
# Which Problems Are Solved

Allow verification of imported passwords hashed with plain md5, without
salt. These are password digests typically created by one of:

- `printf "password" | md5sum` on most linux systems.
- PHP's `md5("password")`
- Python3's `hashlib.md5(b"password").hexdigest()`

# How the Problems Are Solved

- Upgrade passwap to
[v0.6.0](https://github.com/zitadel/passwap/releases/tag/v0.6.0)
- Add md5plain as a new verfier option in `defaults.yaml`

# Additional Changes

- Updated documentation to explain difference between `md5` (crypt) and
`md5plain` verifiers.

# Additional Context

- Requested by customer for import case
2024-06-25 11:10:49 +03:00
Livio Spring
fb8cd18f93
feat: password age policy (#8132)
# Which Problems Are Solved

Some organizations / customers have the requirement, that there users
regularly need to change their password.
ZITADEL already had the possibility to manage a `password age policy` (
thought the API) with the maximum amount of days a password should be
valid, resp. days after with the user should be warned of the upcoming
expiration.
The policy could not be managed though the Console UI and was not
checked in the Login UI.

# How the Problems Are Solved

- The policy can be managed in the Console UI's settings sections on an
instance and organization level.
- During an authentication in the Login UI, if a policy is set with an
expiry (>0) and the user's last password change exceeds the amount of
days set, the user will be prompted to change their password.
- The prompt message of the Login UI can be customized in the Custom
Login Texts though the Console and API on the instance and each
organization.
- The information when the user last changed their password is returned
in the Auth, Management and User V2 API.
- The policy can be retrieved in the settings service as `password
expiry settings`.

# Additional Changes

None.

# Additional Context

- closes #8081

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2024-06-18 11:27:44 +00:00
Elio Bischof
b055d1d9e6
feat(init): configurable existing postgres db (#8045)
# Which Problems Are Solved

The init job fails if no database called *postgres* or *defaultdb* for
cockroach respectively exists.

# How the Problems Are Solved

The value is now configurable, for example by env variable
*ZITADEL_DATABASE_POSTGRES_ADMIN_EXISTINGDATABASE*

# Additional Context

- Closes #5810
2024-06-10 10:49:30 +00:00
Silvan
7d852ad0df
fix: read correct defaults (#8090)
# Which Problems Are Solved

Corrects reading of default configuration, despite reading all default
configs only required defaults are read.

# How the Problems Are Solved

Reading the defualt config of the `mirror`-command was moved to a
pre-run step of the command instead of doing it during initialization of
the executable.

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/8059
- https://discord.com/channels/927474939156643850/1248594307267559535
2024-06-10 11:46:28 +02:00
Silvan
2243306ef6
feat(cmd): mirror (#7004)
# Which Problems Are Solved

Adds the possibility to mirror an existing database to a new one. 

For that a new command was added `zitadel mirror`. Including it's
subcommands for a more fine grained mirror of the data.

Sub commands:

* `zitadel mirror eventstore`: copies only events and their unique
constraints
* `zitadel mirror system`: mirrors the data of the `system`-schema
*  `zitadel mirror projections`: runs all projections
*  `zitadel mirror auth`: copies auth requests
* `zitadel mirror verify`: counts the amount of rows in the source and
destination database and prints the diff.

The command requires one of the following flags:
* `--system`: copies all instances of the system
* `--instance <instance-id>`, `--instance <comma separated list of
instance ids>`: copies only the defined instances

The command is save to execute multiple times by adding the
`--replace`-flag. This replaces currently existing data except of the
`events`-table

# Additional Changes

A `--for-mirror`-flag was added to `zitadel setup` to prepare the new
database. The flag skips the creation of the first instances and initial
run of projections.

It is now possible to skip the creation of the first instance during
setup by setting `FirstInstance.Skip` to true in the steps
configuration.

# Additional info

It is currently not possible to merge multiple databases. See
https://github.com/zitadel/zitadel/issues/7964 for more details.

It is currently not possible to use files. See
https://github.com/zitadel/zitadel/issues/7966 for more information.

closes https://github.com/zitadel/zitadel/issues/7586
closes https://github.com/zitadel/zitadel/issues/7486

### Definition of Ready

- [x] I am happy with the code
- [x] Short description of the feature/issue is added in the pr
description
- [x] PR is linked to the corresponding user story
- [x] Acceptance criteria are met
- [x] All open todos and follow ups are defined in a new ticket and
justified
- [x] Deviations from the acceptance criteria and design are agreed with
the PO and documented.
- [x] No debug or dead code
- [x] My code has no repetitions
- [x] Critical parts are tested automatically
- [ ] Where possible E2E tests are implemented
- [x] Documentation/examples are up-to-date
- [x] All non-functional requirements are met
- [x] Functionality of the acceptance criteria is checked manually on
the dev system.

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-05-30 09:35:30 +00:00
Silvan
cff0f73e24
fix(init): add setting to enable durable locks on crdb (#7982)
feat(init): add setting to enable durable locks on crdb
2024-05-27 09:03:34 +00:00
Livio Spring
e57a9b57c8
feat(saml): allow setting nameid-format and alternative mapping for transient format (#7979)
# Which Problems Are Solved

ZITADEL currently always uses
`urn:oasis:names:tc:SAML:2.0:nameid-format:persistent` in SAML requests,
relying on the IdP to respect that flag and always return a peristent
nameid in order to be able to map the external user with an existing
user (idp link) in ZITADEL.
In case the IdP however returns a
`urn:oasis:names:tc:SAML:2.0:nameid-format:transient` (transient)
nameid, the attribute will differ between each request and it will not
be possible to match existing users.

# How the Problems Are Solved

This PR adds the following two options on SAML IdP:
- **nameIDFormat**: allows to set the nameid-format used in the SAML
Request
- **transientMappingAttributeName**: allows to set an attribute name,
which will be used instead of the nameid itself in case the returned
nameid-format is transient

# Additional Changes

To reduce impact on current installations, the `idp_templates6_saml`
table is altered with the two added columns by a setup job. New
installations will automatically get the table with the two columns
directly.
All idp unit tests are updated to use `expectEventstore` instead of the
deprecated `eventstoreExpect`.

# Additional Context

Closes #7483
Closes #7743

---------

Co-authored-by: peintnermax <max@caos.ch>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-05-23 05:04:07 +00:00
Livio Spring
fb162a7d75
fix(login): improve auth handlers (#7969)
# Which Problems Are Solved

During the implementation of #7486 it was noticed, that projections in
the `auth` database schema could be blocked.
Investigations suggested, that this is due to the use of
[GORM](https://gorm.io/index.html) and it's inability to use an existing
(sql) transaction.
With the improved / simplified handling (see below) there should also be
a minimal improvement in performance, resp. reduced database update
statements.

# How the Problems Are Solved

The handlers in `auth` are exchanged to proper (sql) statements and gorm
usage is removed for any writing part.
To further improve / simplify the handling of the users, a new
`auth.users3` table is created, where only attributes are handled, which
are not yet available from the `projections.users`,
`projections.login_name` and `projections.user_auth_methods` do not
provide. This reduces the events handled in that specific handler by a
lot.

# Additional Changes

None

# Additional Context

relates to #7486
2024-05-22 15:26:02 +00:00
Miguel Cabrerizo
15d5338b91
feat(cnsl): docs link can be customized and custom button is available (#7840)
* feat: customize doc link and additional custom link

* feat: add e2e tests

* fix: update docs

* fix: add @peintnermax changes about cache

* fix: golangci-lint complains preparation.PrepareCommands

---------

Co-authored-by: Max Peintner <max@caos.ch>
2024-05-13 16:01:50 +02:00
Elio Bischof
24a9d7c730
fix(defaults): configure hashers via env (#7872)
fix(setup): configure hashers via env

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2024-05-08 06:48:26 +00:00
Elio Bischof
6c0e7c402d
fix(setup): decode complex config strings (#7854)
* fix(setup): decode complex config strings

* decode complex types before string lists

* lint

* fix hasher env docs

* hasher defaults

* cleanup

* cleanup

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-05-01 12:17:27 +02:00
Livio Spring
6ab06aa249
fix: improve secret generation for apple idp (#7843)
* fix: improve secret generation for apple idp

* remove accidental commit

* change exp time

* change exp time

* change exp time

* change exp time
2024-04-26 15:46:15 +00:00
Livio Spring
4f3564e4e9
fix: disable auth cache by default (#7845) 2024-04-26 09:30:35 +02:00
Livio Spring
d016379e2a
feat: pass and handle auth request context for email links (#7815)
* pass and handle auth request context

* tests and cleanup

* cleanup
2024-04-24 17:50:58 +02:00
Silvan
25030c69b9
perf: cache auth request in memory (#7824)
* perf: cache auth request in memory
2024-04-23 11:23:50 +00:00
Elio Bischof
5b34aabbdd
fix(setup): decode role mappings (#7760) 2024-04-14 12:55:54 +03:00
Livio Spring
153df2e12f
feat: provide option to limit (T)OTP checks (#7693)
* feat: provide option to limit (T)OTP checks

* fix requests in console

* update errors pkg

* cleanup

* cleanup

* improve naming of existing config
2024-04-10 09:14:55 +00:00
Stefan Benz
6dcdef0268
fix: add action v2 execution to features (#7597)
* fix: add action v2 execution to features

* fix: add action v2 execution to features

* fix: add action v2 execution to features

* fix: update internal/command/instance_features_model.go

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>

* fix: merge back main

* fix: merge back main

* fix: rename feature and service

* fix: rename feature and service

* fix: review changes

* fix: review changes

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2024-04-09 20:21:21 +03:00
Tim Möhlmann
2089992d75
feat(crypto): use passwap for machine and app secrets (#7657)
* feat(crypto): use passwap for machine and app secrets

* fix command package tests

* add hash generator command test

* naming convention, fix query tests

* rename PasswordHasher and cleanup start commands

* add reducer tests

* fix intergration tests, cleanup old config

* add app secret unit tests

* solve setup panics

* fix push of updated events

* add missing event translations

* update documentation

* solve linter errors

* remove nolint:SA1019 as it doesn't seem to help anyway

* add nolint to deprecated filter usage

* update users migration version

* remove unused ClientSecret from APIConfigChangedEvent

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-04-05 09:35:49 +00:00
Tim Möhlmann
25ef3da9d5
refactor(fmt): run gci on complete project (#7557)
chore(fmt): run gci on complete project

Fix global import formatting in go code by running the `gci` command. This allows us to just use the command directly, instead of fixing the import order manually for the linter, on each PR.

Co-authored-by: Elio Bischof <elio@zitadel.com>
2024-04-03 10:43:43 +00:00
Stefan Benz
217703395e
feat: add user v2 pw change required information on query (#7603)
* fix: add resource owner as query for user v2 ListUsers and clean up deprecated attribute

* fix: add resource owner as query for user v2 ListUsers and clean up deprecated attribute

* fix: add resource owner as query for user v2 ListUsers and clean up deprecated attribute

* fix: review changes

* fix: review changes

* fix: review changes

* fix: review changes

* fix: add password change required to user v2 get and list

* fix: update unit tests for query side with new column and projection

* fix: change projection in setup steps

* fix: change projection in setup steps

* fix: remove setup step 25

* fix: add password_change_required into ListUsers response

* fix: correct SetUserPassword parameters

* fix: rollback to change setup instead of projection directly

* fix: rollback to change setup instead of projection directly

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-03-28 06:21:21 +00:00
Silvan
56df515e5f
chore: use pgx v5 (#7577)
* chore: use pgx v5

* chore: update go version

* remove direct pq dependency

* remove unnecessary type

* scan test

* map scanner

* converter

* uint8 number array

* duration

* most unit tests work

* unit tests work

* chore: coverage

* go 1.21

* linting

* int64 gopfertammi

* retry go 1.22

* retry go 1.22

* revert to go v1.21.5

* update go toolchain to 1.21.8

* go 1.21.8

* remove test flag

* go 1.21.5

* linting

* update toolchain

* use correct array

* use correct array

* add byte array

* correct value

* correct error message

* go 1.21 compatible
2024-03-27 15:48:22 +02:00