# Which Problems Are Solved
There are cases where not all statements of multiExec are succeed. This
leads to inconsistent states. One example is [LDAP
IDPs](https://github.com/zitadel/zitadel/issues/7959).
If statements get executed only partially this can lead to inconsistent
states or even break projections for objects which might not were
correctly created in a sub table.
This behaviour is possible because we use
[`SAVEPOINTS`](https://www.postgresql.org/docs/current/sql-savepoint.html)
during each statement of a multiExec.
# How the Problems Are Solved
SAVEPOINTS are only created at the beginning of an exec function not
during every execution like before. Additionally `RELEASE` or `ROLLBACK`
of `SAVEPOINTS` are only used when needed.
# Additional Changes
- refactor some unused parameters
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/7959
(cherry picked from commit ddeeeed303)
# Which Problems Are Solved
V2 and V3 APIs allow setting the organization context by providing the
organization domain in the request. Users currently experience the
following error: "rpc error: code = Unauthenticated desc = context
missing (AUTH-rKLWEH)"
# How the Problems Are Solved
Correctly check the org domain when set.
# Additional Changes
None
# Additional Context
- support request
(cherry picked from commit dc7330f251)
# Which Problems Are Solved
Calls with tokens issued through JWT Profile or Client Credentials
Grants were no longer possible and threw a "could not read projectid by
clientid (AUTH-GHpw2)" error.
ZITADEL checks the allowed origins of an application and load its
projectID into the context on any API call.
Tokens from service accounts did not contain any clientID and therefore
never did that check.
But due to a change in https://github.com/zitadel/zitadel/pull/8580,
were the service user id was set as client_id in the OIDC session to fix
the introspection response
(https://github.com/zitadel/zitadel/issues/8590).
# How the Problems Are Solved
- Check if the project and origin were retrieved and only then check the
origins
# Additional Changes
None.
# Additional Context
- closes https://github.com/zitadel/zitadel/issues/8676
- relates to https://github.com/zitadel/zitadel/pull/8580 (released on
2.62.0)
- relates to https://github.com/zitadel/zitadel/issues/8590
(cherry picked from commit c347e75485)
# Which Problems Are Solved
When executing many concurrent authentication requests on a single
machine user, there were performance issues. As the same aggregate is
being searched and written to concurrently, we traced it down to a
locking issue on the used index.
We already optimized the token endpoint by creating a separate OIDC
aggregate.
At the time we decided to push a single event to the user aggregate, for
the user audit log. See [technical advisory
10010](https://zitadel.com/docs/support/advisory/a10010) for more
details.
However, a recent security fix introduced an additional search query on
the user aggregate, causing the locking issue we found.
# How the Problems Are Solved
Add a feature flag which disables pushing of the `user.token.v2.added`.
The event has no importance and was only added for informational
purposes on the user objects. The `oidc_session.access_token.added` is
the actual payload event and is pushed on the OIDC session aggregate and
can still be used for audit trail.
# Additional Changes
- Fix an event mapper type for
`SystemOIDCSingleV1SessionTerminationEventType`
# Additional Context
- Reported by support request
- https://github.com/zitadel/zitadel/pull/7822 changed the token
aggregate
- https://github.com/zitadel/zitadel/pull/8631 introduced user state
check
Load test trace graph with `user.token.v2.added` **enabled**. Query
times are steadily increasing:
Load test trace graph with `user.token.v2.added` **disabled**. Query
times constant:
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
(cherry picked from commit 63d733b3a2)
# Which Problems Are Solved
When executing many concurrent authentication requests on a single
machine user, there were performance issues. As the same aggregate is
being searched and written to concurrently, we traced it down to a
locking issue on the used index.
We already optimized the token endpoint by creating a separate OIDC
aggregate.
At the time we decided to push a single event to the user aggregate, for
the user audit log. See [technical advisory
10010](https://zitadel.com/docs/support/advisory/a10010) for more
details.
However, a recent security fix introduced an additional search query on
the user aggregate, causing the locking issue we found.
# How the Problems Are Solved
Add a feature flag which disables pushing of the `user.token.v2.added`.
The event has no importance and was only added for informational
purposes on the user objects. The `oidc_session.access_token.added` is
the actual payload event and is pushed on the OIDC session aggregate and
can still be used for audit trail.
# Additional Changes
- Fix an event mapper type for
`SystemOIDCSingleV1SessionTerminationEventType`
# Additional Context
- Reported by support request
- https://github.com/zitadel/zitadel/pull/7822 changed the token
aggregate
- https://github.com/zitadel/zitadel/pull/8631 introduced user state
check
Load test trace graph with `user.token.v2.added` **enabled**. Query
times are steadily increasing:
Load test trace graph with `user.token.v2.added` **disabled**. Query
times constant:
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
- There was an incorrect placement of a hyphen in a sentence.
# How the Problems Are Solved
- Corrected by replacing the hyphen with a comma and adding a verb
(ready to go, offering a)
Co-authored-by: Fabi <fabienne@zitadel.com>
# Which Problems Are Solved
Twilio supports a robust, multi-channel verification service that
notably supports multi-region SMS sender numbers required for our use
case. Currently, Zitadel does much of the work of the Twilio Verify (eg.
localization, code generation, messaging) but doesn't support the pool
of sender numbers that Twilio Verify does.
# How the Problems Are Solved
To support this API, we need to be able to store the Twilio Service ID
and send that in a verification request where appropriate: phone number
verification and SMS 2FA code paths.
This PR does the following:
- Adds the ability to use Twilio Verify of standard messaging through
Twilio
- Adds support for international numbers and more reliable verification
messages sent from multiple numbers
- Adds a new Twilio configuration option to support Twilio Verify in the
admin console
- Sends verification SMS messages through Twilio Verify
- Implements Twilio Verification Checks for codes generated through the
same
# Additional Changes
# Additional Context
- base was implemented by @zhirschtritt in
https://github.com/zitadel/zitadel/pull/8268❤️
- closes https://github.com/zitadel/zitadel/issues/8581
---------
Co-authored-by: Zachary Hirschtritt <zachary.hirschtritt@klaviyo.com>
Co-authored-by: Joey Biscoglia <joey.biscoglia@klaviyo.com>
# Which Problems Are Solved
We identified the need of caching.
Currently we have a number of places where we use different ways of
caching, like go maps or LRU.
We might also want shared chaches in the future, like Redis-based or in
special SQL tables.
# How the Problems Are Solved
Define a generic Cache interface which allows different implementations.
- A noop implementation is provided and enabled as.
- An implementation using go maps is provided
- disabled in defaults.yaml
- enabled in integration tests
- Authz middleware instance objects are cached using the interface.
# Additional Changes
- Enabled integration test command raceflag
- Fix a race condition in the limits integration test client
- Fix a number of flaky integration tests. (Because zitadel is super
fast now!) 🎸🚀
# Additional Context
Related to https://github.com/zitadel/zitadel/issues/8648
Bumps [micromatch](https://github.com/micromatch/micromatch) from 4.0.7
to 4.0.8.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/micromatch/micromatch/releases">micromatch's
releases</a>.</em></p>
<blockquote>
<h2>4.0.8</h2>
<p>Ultimate release that fixes both CVE-2024-4067 and CVE-2024-4068. We
consider the issues low-priority, so even if you see automated scanners
saying otherwise, don't be scared.</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/micromatch/micromatch/blob/master/CHANGELOG.md">micromatch's
changelog</a>.</em></p>
<blockquote>
<h2>[4.0.8] - 2024-08-22</h2>
<ul>
<li>backported CVE-2024-4067 fix (from v4.0.6) over to 4.x branch</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="8bd704ec0d"><code>8bd704e</code></a>
4.0.8</li>
<li><a
href="a0e68416a4"><code>a0e6841</code></a>
run verb to generate README documentation</li>
<li><a
href="4ec288484f"><code>4ec2884</code></a>
Merge branch 'v4' into hauserkristof-feature/v4.0.8</li>
<li><a
href="03aa805217"><code>03aa805</code></a>
Merge pull request <a
href="https://redirect.github.com/micromatch/micromatch/issues/266">#266</a>
from hauserkristof/feature/v4.0.8</li>
<li><a
href="814f5f70ef"><code>814f5f7</code></a>
lint</li>
<li><a
href="67fcce6a10"><code>67fcce6</code></a>
fix: CHANGELOG about braces & CVE-2024-4068, v4.0.5</li>
<li><a
href="113f2e3fa7"><code>113f2e3</code></a>
fix: CVE numbers in CHANGELOG</li>
<li><a
href="d9dbd9a266"><code>d9dbd9a</code></a>
feat: updated CHANGELOG</li>
<li><a
href="2ab13157f4"><code>2ab1315</code></a>
fix: use actions/setup-node@v4</li>
<li><a
href="1406ea38f3"><code>1406ea3</code></a>
feat: rework test to work on macos with node 10,12 and 14</li>
<li>Additional commits viewable in <a
href="https://github.com/micromatch/micromatch/compare/4.0.7...4.0.8">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/zitadel/zitadel/network/alerts).
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
# Which Problems Are Solved
Endpoints to maintain email and phone contact on user v3 are not
implemented.
# How the Problems Are Solved
Add 3 endpoints with SetContactEmail, VerifyContactEmail and
ResendContactEmailCode.
Add 3 endpoints with SetContactPhone, VerifyContactPhone and
ResendContactPhoneCode.
Refactor the logic how contact is managed in the user creation and
update.
# Additional Changes
None
# Additional Context
- part of https://github.com/zitadel/zitadel/issues/6433
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Page title was "introduction" and the headings were missing a h2 level.
This makes it difficult to index for search, both internal and external.
# How the Problems Are Solved
* Change the page title
* Pulled all headings one level up
# Additional Changes
- Show all elements in sdk-example folder automaticalls
# Which Problems Are Solved
The docs contained typos and links that led to 404.
More subejectively the docs did not provide enough guidance for new
users what providers are and how to configure an HTTP provider and then
activate them. Only basic links to the API docs were given without
examples on how to achieve a basic configuration.
# How the Problems Are Solved
References and typos fixed and reworked the guide.
# Additional Changes
Added code highlighting for json and bash.
# Additional Context
We could further improve by adding more help on this page on how to
configure SMS and SMTP providers.
# Which Problems Are Solved
ZITADEL's user account deactivation mechanism did not work correctly
with service accounts. Deactivated service accounts retained the ability
to request tokens, which could lead to unauthorized access to
applications and resources.
# How the Problems Are Solved
Additionally to checking the user state on the session API and login UI,
the state is checked on all oidc session methods resulting in a new
token or when returning the user information (userinfo, introspection,
id_token / access_token and saml attributes)
(cherry picked from commit 5b40af79f0)
# Which Problems Are Solved
ZITADEL's user grants deactivation mechanism did not work correctly.
Deactivated user grants were still provided in token, which could lead
to unauthorized access to applications and resources.
Additionally, the management and auth API always returned the state as
active or did not provide any information about the state.
# How the Problems Are Solved
- Correctly check the user grant state on active for tokens and user
information (userinfo, introspection, saml attributes)
- Map state in API and display in Console
(cherry picked from commit ca1914e235)
# Which Problems Are Solved
In Zitadel, even after an organization is deactivated, associated
projects, respectively their applications remain active. Users across
other organizations can still log in and access through these
applications, leading to unauthorized access.
Additionally, if a project was deactivated access to applications was
also still possible.
# How the Problems Are Solved
- Correctly check the status of the organization and related project.
(Corresponding functions have been renamed to `Active...`)
(cherry picked from commit d01bd1c51a)
# Which Problems Are Solved
ZITADEL's user account deactivation mechanism did not work correctly
with service accounts. Deactivated service accounts retained the ability
to request tokens, which could lead to unauthorized access to
applications and resources.
# How the Problems Are Solved
Additionally to checking the user state on the session API and login UI,
the state is checked on all oidc session methods resulting in a new
token or when returning the user information (userinfo, introspection,
id_token / access_token and saml attributes)
# Which Problems Are Solved
ZITADEL's user grants deactivation mechanism did not work correctly.
Deactivated user grants were still provided in token, which could lead
to unauthorized access to applications and resources.
Additionally, the management and auth API always returned the state as
active or did not provide any information about the state.
# How the Problems Are Solved
- Correctly check the user grant state on active for tokens and user
information (userinfo, introspection, saml attributes)
- Map state in API and display in Console
# Which Problems Are Solved
In Zitadel, even after an organization is deactivated, associated
projects, respectively their applications remain active. Users across
other organizations can still log in and access through these
applications, leading to unauthorized access.
Additionally, if a project was deactivated access to applications was
also still possible.
# How the Problems Are Solved
- Correctly check the status of the organization and related project.
(Corresponding functions have been renamed to `Active...`)
# Which Problems Are Solved
Reduce the chance for projection dead-locks. Increasing or disabling the
projection transaction duration solved dead-locks in all reported cases.
# How the Problems Are Solved
Increase the default transaction duration to 1 minute.
Due to the high value it is functionally similar to disabling,
however it still provides a safety net for transaction that do freeze,
perhaps due to connection issues with the database.
# Additional Changes
- Integration test uses default.
- Technical advisory
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/8517
---------
Co-authored-by: Silvan <silvan.reusser@gmail.com>
# Which Problems Are Solved
Users are not yet able to update their information an status in user API
v3.
# How the Problems Are Solved
Add endpoints and functionality to update users and their status in user
API v3.
# Additional Changes
Aggregate_type and event_types are updated with "userschema" to avoid
conflicts with old events.
# Additional Context
closes#7898
# Which Problems Are Solved
There is no documentation on how to use the HTTP providers for SMS and
Email.
# How the Problems Are Solved
Add documentation and correct the tags in the proto files to correctly
group the calls for the API docs.
# Additional Changes
None
# Additional Context
None
---------
Co-authored-by: Fabi <fabienne@zitadel.com>
# Which Problems Are Solved
If a user is created through the login UI with an IdP and an unverified
email, the user is prompted with the email verification and to set a
password.
This was unintentionally changed with #8291
# How the Problems Are Solved
- Check if the user has any IdP linked when computing the
EmailVerification step.
# Additional Changes
None
# Additional Context
- reported in a support request
There were some problems after changing from SMTP providers to email
providers (https://github.com/zitadel/zitadel/pull/8545):
- panic (nil pointer) on SMTP provider retrieval for HTTP configs
- old SMTP configuration created before the multi SMTP configurations
(without id)
- were not projected / listed
- could not be always be activated
- Console treated HTTP configs as SMTP
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
The typescript repository documentation is updated
---------
Co-authored-by: Fabi <fabienne@zitadel.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
When the `openid` scope was not requested, as is possible in machine
authentication, we didn't set the `sub` (subject) claim to tokens and
possibly also userInfo and introspection.
This fix always sets the `sub` claim for all cases.
# How the Problems Are Solved
Set the `Subject` field to regardless of passed scopes.
# Additional Changes
- none
# Additional Context
According to standards:
- [RFC9068 - JSON Web Token (JWT) Profile for OAuth 2.0 Access
Tokens](https://datatracker.ietf.org/doc/html/rfc9068#name-data-structure)
this claim is **required**.
- [RFC7667 - OAuth 2.0 Token
Introspection](https://datatracker.ietf.org/doc/html/rfc7662#section-2.2)
the claim is optional, however there is no correlation to the `openid`
or OpenID Connect. Therefore it doesn't harm to always return this
claim.
- [OpenID connect, User Info
Response](https://openid.net/specs/openid-connect-core-1_0.html#UserInfoResponse):
"The sub (subject) Claim **MUST** always be returned in the UserInfo
Response."
Closes https://github.com/zitadel/zitadel/issues/8591
# Which Problems Are Solved
- If you operate Zitadel behind a Reverse Proxy or Ingress inside a
Kubernetes cluster,
you may encounter an Error like `upstream sent too big header while
reading response header from upstream`
- The Docs explain how to solve it
# How the Problems Are Solved
- Adding Troubleshooting Docs for too big upstream header, so people
that search for it find a solution.
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
# Which Problems Are Solved
Send Email messages as a HTTP call to a relay, for own logic on handling
different Email providers
# How the Problems Are Solved
Create endpoints under Email provider to manage SMTP and HTTP in the
notification handlers.
# Additional Changes
Clean up old logic in command and query side to handle the general Email
providers with deactivate, activate and remove.
# Additional Context
Partially closes#8270
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Upload the integration test server logs as artifacts, even if the tests
fail.
Before this change logs where printed through the Makefile.
However if a test would fail, the logs wouldn't get printed.
# How the Problems Are Solved
- Add an extra build step that pushes `tmp/zitadel.log` and
`tmp/race.log.$pid` to artificats storage.
- Logs are no longer printed in the `core_integration_reports` Makefile
recipe.
# Additional Changes
Do not remove coverage data when generating the coverage report in
`core_integration_reports`. This is to prevent future "File not found"
erros when running the command repeatedly.
# Additional Context
Reported as internal feedback
# Which Problems Are Solved
Apple is not availabe as IDP type in the settings v2 API.
# How the Problems Are Solved
Add Apple as type.
# Additional Changes
None
# Additional Context
None
Co-authored-by: Max Peintner <max@caos.ch>
# Which Problems Are Solved
The exact timestamp including the seconds and milliseconds in the user
audit log is not shown, either in the direct log or in the tooltip that
appears when hovering over the item.
# How the Problems Are Solved
The exact timestamp is added to the tooltip that appears when hovering
over the item.
# Additional Changes
None
# Additional Context
- Closes#8371
---------
Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Fabi <fabienne@zitadel.com>
# Which Problems Are Solved
As an administrator I want to be able to invite users to my application
with the API V2, some user data I will already prefil, the user should
add the authentication method themself (password, passkey, sso).
# How the Problems Are Solved
- A user can now be created with a email explicitly set to false.
- If a user has no verified email and no authentication method, an
`InviteCode` can be created through the User V2 API.
- the code can be returned or sent through email
- additionally `URLTemplate` and an `ApplicatioName` can provided for
the email
- The code can be resent and verified through the User V2 API
- The V1 login allows users to verify and resend the code and set a
password (analog user initialization)
- The message text for the user invitation can be customized
# Additional Changes
- `verifyUserPasskeyCode` directly uses `crypto.VerifyCode` (instead of
`verifyEncryptedCode`)
- `verifyEncryptedCode` is removed (unnecessarily queried for the code
generator)
# Additional Context
- closes#8310
- TODO: login V2 will have to implement invite flow:
https://github.com/zitadel/typescript/issues/166
# Which Problems Are Solved
In actions/v3 there was no check for the feature flag on any of the:
- ListExecutionFunctions
- ListExecutionMethods
- ListExecutionServices
In the integration tests `ensureFeatureEnabled` relies on
`ListExecutionMethods` to return an error if the feature is not enabled.
This fix makes the test wait untill the feature is fully projected.
# How the Problems Are Solved
Add the feature check to all of the above methods.
# Additional Changes
- none
# Additional Context
Flaky introduced in https://github.com/zitadel/zitadel/pull/8407
# Which Problems Are Solved
Improve performance by removing a GetUserByID call. The call also
executed a Trigger on projections, which significantly impacted
concurrent requests.
# How the Problems Are Solved
Token creation needs information from the user, such as the resource
owner and access token type.
For client credentials this is solved in a single search. By getting the
user by username (`client_id`), the user details and secret were
obtained in a single query. After that verification and token creation
can proceed. For JWT profile it is a bit more complex. We didn't know
anything about the user until after JWT verification.
The verification did a query for the AuthN key and after that we did a
GetUserByID to get remaining details.
This change uses a joined query when the OIDC library calls the
`GetKeyByIDAndClientID` method on the token storage. The found user
details are set to the verifieer object and returned after verification
is completed.
It is safe because the `jwtProfileKeyStorage` is a single-use object as
a wrapper around `query.Queries`.
This way getting the public key and user details are obtained in a
single query.
# Additional Changes
- Correctly set the `client_id` field with machine's username.
# Additional Context
- Related to: https://github.com/zitadel/zitadel/issues/8352
# Which Problems Are Solved
Add a debug API which allows pushing a set of events to be reduced in a
dedicated projection.
The events can carry a sleep duration which simulates a slow query
during projection handling.
# How the Problems Are Solved
- `CreateDebugEvents` allows pushing multiple events which simulate the
lifecycle of a resource. Each event has a `projectionSleep` field, which
issues a `pg_sleep()` statement query in the projection handler :
- Add
- Change
- Remove
- `ListDebugEventsStates` list the current state of the projection,
optionally with a Trigger
- `GetDebugEventsStateByID` get the current state of the aggregate ID in
the projection, optionally with a Trigger
# Additional Changes
- none
# Additional Context
- Allows reproduction of https://github.com/zitadel/zitadel/issues/8517
# Which Problems Are Solved
defaults.yaml only specifies defaults for cockroach. Therefore, options
omitted for postgresql are actually set to `0`.
This means that the connections timeouts are set to `0` and connections
were not reused, resulting in a performance penalty while running the
integration tests.
# How the Problems Are Solved
Set MaxConnLifeTime and MaxConnIdleTime options in postgres
# Additional Changes
- none
# Additional Context
- none
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
