# Which Problems Are Solved
With current provided telemetry it's difficult to predict when a
projection handler is under increased load until it's too late and
causes downstream issues. Importantly, projection updating is in the
critical path for many login flows and increased latency there can
result in system downtime for users.
# How the Problems Are Solved
This PR adds three new prometheus-style metrics:
1. **projection_events_processed** (_labels: projection, success_) -
This metric gives us a counter of the number of events processed per
projection update run and whether they we're processed without error. A
high number of events being processed can let us know how busy a
particular projection handler is.
2. **projection_handle_timer** _(labels: projection)_ - This is the time
it takes to process a projection update given a batch of events - time
to take the current_states lock, query for new events, reduce,
update_the projection, and update current_states.
3. **projection_state_latency** _(labels: projection)_ - This is the
time from the last event processed in the current_states table for a
given projection. It tells us how old was the last event you processed?
Or, how far behind are you running for this projection? Higher latencies
could mean high load or stalled projection handling.
# Additional Changes
I also had to initialize the global otel metrics provider (`metrics.M`)
in the `setup` step additionally to `start` since projection handlers
are initialized at setup. The initialization checks if a metrics
provider is already set (in case of `start-from-setup` or
`start-from-init` to prevent overwriting, which causes the otel metrics
provider to stop working.
# Additional Context
## Example Dashboards
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Zitadel setup with v2.71.0 could result in errors regarding the
idp_templates6_ldap3 subtable.
# How the Problems Are Solved
Rename the subtable idp_templates6_ldap3 to idp_templates6_ldap2 if no
idp_templates6_ldap2 is existing and rename column `rootCA` to
`root_ca`.
# Additional Changes
None
# Additional Context
Related PR #9292
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
When using implicit flow through the session API and a login UI on a
custom domain (proxy), the tokens were signed by the API domain of the
instance, rather than the public (proxy) domain.
The SAML response had the same issue. Additionally, the saml library had
an issue and lost the issuer context. This prevented also a successful
login through the hosted login UI.
# How the Problems Are Solved
- The issuer of the SAML and Auth request is persisted to provide the
information when signing the responses and tokens.
- The SAML library is updated to the latest version.
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
Build and test workflows are currently running on specific GitHub hosted
runners. These is not needed for most worklfows and just costs more.
# How the Problems Are Solved
Moved all the steps apart from integration-tests to public runners.
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
Closes most of the dependabot alerts
# How the Problems Are Solved
Updating Versions as much as possible. Angular upgrade was left out on
purpose.
# Additional Changes
Some refactoring of unused code, which I stumbled upon looking at our
used deps.
# Which Problems Are Solved
With the recent updates of our customer portal design, our onboarding
guide and customer portal docs were outdated.
# How the Problems Are Solved
- Updated screenshots of customer portal
- Updated screenshots of onboarding in management console
- Updated quickstarts with changes that have been made in the onbaording
---------
Co-authored-by: Maximilian <mpa@zitadel.com>
# Which Problems Are Solved
It was not possible to use the `api.v1.appendUserGrant` function in the
`postCreation` trigger action as documented.
# How the Problems Are Solved
- Correctly initialize the javascript / Goja function
- Added `projectID` and `projectGrantID` (as documented), but kept
`projectId` and `projectGrantId` (for backwards compatibility) when
mapping the object in the `appendUserGrant` function
# Additional Changes
None
# Additional Context
- A customer reached out to support regarding this issue.
- requires back port to 2.70.x
# Which Problems Are Solved
When we recently changed some permission for the `SYSTEM_OWNER` role on
QA, we noticed that there we multiple `permission.added` even when we
removed specific permissions.
# How the Problems Are Solved
Fixed the event type when removing permissions.
# Additional Changes
None
# Additional Context
Noticed when rolling out some changes on QA
# Which Problems Are Solved
While testing totally unrelated issues, I noticed that the phone
verification in Console didn't work anymore.
# How the Problems Are Solved
Pass the verification `code` entered by the user to the API.
# Additional Changes
None
# Additional Context
probably relates to https://github.com/zitadel/zitadel/pull/9312
# Which Problems Are Solved
Multiple dependencies used by the Zitadel backend are outdated.
# How the Problems Are Solved
Update all direct dependencies (apart from the following two) to the
latest version:
- github.com/go-webauthn/webauthn: see
https://github.com/zitadel/zitadel/issues/9615
- github.com/nicksnyder/go-i18n/v2 (v2.5.0 / 2.5.1 prevent the use of
reserved keys such as `description`:
https://github.com/nicksnyder/go-i18n/releases/tag/v2.5.0)
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
Outdated dependencies foir packages developed by Zitadel.
Some of them included important security updates from sub-dependencies.
# How the Problems Are Solved
Upgrade all packages under `github.com/zitadel/*` to the latest released
version.
# Additional Changes
- The `github.com/superseriousbusiness/exifremove` was removed from
Github. We copied the cached go mod code to
https://github.com/zitadel/exifremove and use this module now.
# Additional Context
- Related to https://github.com/zitadel/zitadel/issues/9422
- Closes https://github.com/zitadel/zitadel/issues/9443
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
The Kubernetes and Knative deployment docs suggest to depend on
CockroachDB installations even though we sunset the CockroachDB support
with the upcoming major version. This can be annoying for users who
create new environments using CockroachDB.
# How the Problems Are Solved
- The Kubernetes deployment is removed and points to examples in the
chart repo directy. This removes redundancies that are prone to getting
outdated without notice.
- The Knative deployment uses commands to deploy a PostgreSQL
StatefulSet instead of a CockroachDB StatefulSet. The DB command from
the Knative Tutorial is used, as users are likely to be familiar with
the tutorials configuration already. The static Kubernetes YAML files
for the DB as well as for the Zitadel Knative service are removed
altogether, as they are prone to getting outdated without notice and
don't serve any purpose.
# Additional Changes
- The outdated and boring video guide for Knative is removed.
- The Knative configuration is changed, so the first-time login becomes
easier which improves DevX.
- The current docker compose file doesn't work, this PR fixes it and
upgrades the used Postgres to v17.
# Additional Context
- Closes https://github.com/zitadel/zitadel-charts/issues/322
- Replaces https://github.com/zitadel/zitadel/pull/9540
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
- The current validation for native redirect URIs does not allow HTTPS
loopback addresses.
# How the Problems Are Solved
- Enhanced the validation logic to permit HTTPS loopback addresses,
ensuring that developers can use these addresses without encountering
validation errors.
- Updated zitadel/oidc to latest version
# Additional Context
- Closes#4091
- This pr need to be closed first in our OIDC lib:
https://github.com/zitadel/oidc/pull/691
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Allow verification of imported salted passwords hashed with plain md5.
# How the Problems Are Solved
- Upgrade passwap to
[v0.7.0](https://github.com/zitadel/passwap/releases/tag/v0.7.0)
- Add md5salted as a new verifier option in `defaults.yaml`
# Additional Changes
- go version and libraries updated (required by passkey v0.7.0)
- secrets.md verifiers updated
- configuration verifiers updated
- added MD5salted and missing MD5Plain to test cases
# Which Problems Are Solved
The service name is hardcoded in the metrics code. Making the service
name to be configurable helps when running multiple instances of
Zitadel.
The defaults remain unchanged, the service name will be defaulted to
ZITADEL.
# How the Problems Are Solved
Add a config option to override the name in defaults.yaml and pass it
down to the corresponding metrics or tracing module (google or otel)
# Additional Changes
NA
# Additional Context
NA
# Which Problems Are Solved
- Refresh Tokens issued by third party authentication providers are lost
# How the Problems Are Solved
- Allows the existing post authentication action to capture the refresh
token
# Additional Changes
- Docs updated to reflect the new property
# Additional Context
- Partially addresses #7851 by allowing the refresh token to be
captured.
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
The milestones query returns multiple results for every milestone for
every instance domain.
# How the Problems Are Solved
Corrected where condition on milestone query.
# Additional Changes
None
# Additional Context
None
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
SCIM integration test failed sometimes, as ListUsers with usernames-sort
was not reliable if the asserted list is not sorted as well.
# How the Problems Are Solved
Sort the list of results in the sorted integration tests.
# Additional Changes
None
# Additional Context
Relates to
https://github.com/zitadel/zitadel/actions/runs/13922326003/job/38960759621
# Which Problems Are Solved
Quota notification integration test failed sometimes due to eventual
consistency issues, which resulted in calls which should have been
counted to the quota not being added. This resulted in flaky integration
tests as the expected API calls to be limited were executed normally.
# How the Problems Are Solved
As there is no API call to query the currently applied Quota, there was
a sleep added as a last effort, to give some time that the event gets
processed into the projection.
# Additional Changes
None
# Additional Context
Related to
https://github.com/zitadel/zitadel/actions/runs/13922326003/job/38959595055
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
Running local development of the docs didn't work due to errors with
sidebar imports.
# How the Problems Are Solved
The imports held the sidebar items in a "default" object. Changed the
imports
# Additional Changes
removed some comments
This PR implements a component which is used to render the feature
settings in a generic way.
All features besides`loginV2` and `improvedPerformance` are rendered.
Note: The feature flags are not autogenerated as the console needs
descriptions for the respective features and there are multiple types
where additional fields like text input is required
Closes#9090
---------
Co-authored-by: conblem <mail@conblem.me>
# Which Problems Are Solved
Allows users to be created using the V2 User API
# How the Problems Are Solved
I added a seperate V2 create user page with the new code using the new
apis.
# Additional Changes
I did some refactorings arround our interceptors as they used an
obselete syntax.
The password complexity form takes the Buf definitions.
# Additional Context
- Closes#9430
---------
Co-authored-by: Max Peintner <peintnerm@gmail.com>
# Which Problems Are Solved
E2E tests in pipelines started to fail randomly. While debugging it, i
noticed that we use the `latest` tag of cockroach's docker image. They
tagged 25.1 as latest yesterday.
# How the Problems Are Solved
Since we drop support for CRDB with version 3 as there are anyway
multiple issues with various versions, I pinned the docker image tag to
`latest-v24.3`.
# Additional Changes
None
# Additional Context
relates to https://github.com/zitadel/zitadel/actions/runs/13917603587
and https://github.com/zitadel/zitadel/actions/runs/13904928050
# Which Problems Are Solved
Integration tests for OIDC service failed irregularly.
# How the Problems Are Solved
Add eventual consistent checks for querying endpoints to the integration
tests for the OIDC service.
# Additional Changes
None
# Additional Context
None
# Which Problems Are Solved
Zitadel should not record 404 response counts of unknown paths (check
`/debug/metrics`).
This can lead to high cardinality on metrics endpoint and in traces.
```
GOOD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="200",uri="/.well-known/openid-configuration"} 2
GOOD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="200",uri="/oauth/v2/keys"} 2
BAD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="404",uri="/junk"} 2000
```
After
```
GOOD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="200",uri="/.well-known/openid-configuration"} 2
GOOD http_server_return_code_counter_total{method="GET",otel_scope_name="",otel_scope_version="",return_code="200",uri="/oauth/v2/keys"} 2
```
# How the Problems Are Solved
This PR makes sure, that any unknown path is recorded as `UNKNOWN_PATH`
instead of the actual path.
# Additional Changes
N/A
# Additional Context
On our production instance, when a penetration test was run, it caused
our metric count to blow up to many thousands due to Zitadel recording
404 response counts.
Next nice to have steps, remove 404 timer recordings which serve no
purpose
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
Co-authored-by: Livio Spring <livio@zitadel.com>
# Which Problems Are Solved
- The `io/ioutil` package was deprecated in Go 1.16.
- Reference: https://go.dev/doc/go1.16#ioutil
# How the Problems Are Solved
- Replaced deprecated functions with their recommended alternatives:
- `ioutil.ReadFile` → `os.ReadFile`
- `ioutil.ReadAll` → `io.ReadAll`
- `ioutil.NopCloser` → `io.NopCloser`
# Which Problems Are Solved
Scripts and other assets for the hosted login UI are served with a
public cache with `max-age` and `s-maxage`. After changing scripts or
assets, old versions might be still used as they might be cached locally
or in a shared cache (CDN, proxy, ...). This can lead to unwanted
behaviour or even errors.
# How the Problems Are Solved
To ensure the correct file is served a query parameter with the build
time is added to the assets filename dynamically. (`?v=2025-03-17...`)
# Additional Changes
None
# Additional Context
- relates to #9485
- requires backport to at least 2.70.x
# Which Problems Are Solved
- The current contrib.rocks link (`https://contrib.rocks`) does not
directly preview the contributor graph for the zitadel/zitadel
repository.
# How the Problems Are Solved
- Updated the contrib.rocks link to
`https://contrib.rocks/preview?repo=zitadel/zitadel`, which directly
shows the contributor graph for this repository.
# Which Problems Are Solved
The reverse proxy docs have too many links to the third party proxy
provider. This is noisy and might result in unintentional redirects.
# How the Problems Are Solved
The link to the proxy provider is only shown on the first occurence of
the provider name instead of all occurences.
# Which Problems Are Solved
With the recent change in Console to use the User V2 API
(https://github.com/zitadel/zitadel/pull/9312), some functionality still
needs to call the management API, which requires the organization
context. The context was not passed anymore, leading to error in cases
where the calling user (e.g. an IAM_OWNER) was not part of the same
organization.
# How the Problems Are Solved
Added an interceptor to provide the `x-zitadel-orgid` header for the new
management client.
# Additional Changes
None
# Additional Context
- closes#9488
# Which Problems Are Solved
Users were not yet able to specify (and test) the new login UI or
self-hosted login UI for SAML applications through Console.
# How the Problems Are Solved
Added the configuration for SAML apps (as already available for OIDC) in
Console.
# Additional Changes
None
# Additional Context
- closes#9354
# Which Problems Are Solved
When using a custom / new login UI and an OIDC application with
registered BackChannelLogoutUI, no logout requests were sent to the URI
when the user signed out.
Additionally, as described in #9427, an error was logged:
`level=error msg="event of type *session.TerminateEvent doesn't
implement OriginEvent"
caller="/home/runner/work/zitadel/zitadel/internal/notification/handlers/origin.go:24"`
# How the Problems Are Solved
- Properly pass `TriggerOrigin` information to session.TerminateEvent
creation and implement `OriginEvent` interface.
- Implemented `RegisterLogout` in `CreateOIDCSessionFromAuthRequest` and
`CreateOIDCSessionFromDeviceAuth`, both used when interacting with the
OIDC v2 API.
- Both functions now receive the `BackChannelLogoutURI` of the client
from the OIDC layer.
# Additional Changes
None
# Additional Context
- closes#9427
# Which Problems Are Solved
When requesting a JWT (`urn:ietf:params:oauth:token-type:jwt`) to be
returned in a Token Exchange request, ZITADEL would panic if the `actor`
was not granted the necessary permission.
# How the Problems Are Solved
Properly check the error and return it.
# Additional Changes
None
# Additional Context
- closes#9436
# Which Problems Are Solved
We configured the default base URL for the hosted v2 login to
`/ui/v2/login`. However, the docs still instruct readers to configure
the URL explicitly. This is unneccesary mental overhead and a risk of
self-DOS due to typos.
# How the Problems Are Solved
The docs instruct readers to not configure the base URL in order to use
the default.
Co-authored-by: Livio Spring <livio.a@gmail.com>
# Which Problems Are Solved
The API guidelines were not completely accurate on how we want to check
permissions.
# How the Problems Are Solved
Made the description clearer and added examples.
# Additional Changes
Improved the error code example as initially intended in #9340
and added notes about the `limits` for listing resources.
# Additional Context
popped up in PR review of https://github.com/zitadel/zitadel/pull/9445
# Which Problems Are Solved
When registering passkeys or u2f methods as second factor, some users
pressed the "Enter" key, rather than clicking the submit button. This
method has bypassed the execution of the device registration and
encoding scripts, resulting in the form being submitted without the
necessary encoded values.
# How the Problems Are Solved
This PR ensures that device registration is always executed and the
required information are submitted in the form regardless of pressing
"Enter" or clicking the button.
# Additional Changes
None
# Additional Context
- closes#6592
- closes#2910
# Which Problems Are Solved
If configuration `notifications.LegacyEnabled` is set to false when
using cockroachdb as a database Zitadel start does not work and prints
the following error: `level=fatal msg="unable to start zitadel"
caller="github.com/zitadel/zitadel/cmd/start/start_from_init.go:44"
error="can't scan into dest[0]: cannot scan NULL into *string"`
# How the Problems Are Solved
The combination of the setting and cockraochdb are checked and a better
error is provided to the user.
# Additional Context
- introduced with https://github.com/zitadel/zitadel/pull/9321
# Which Problems Are Solved
SQL error in `cmd/setup/49/01-permitted_orgs_function.sql`
# How the Problems Are Solved
Updating `cmd/setup/49/01-permitted_orgs_function.sql`
# Additional Context
- Closes https://github.com/zitadel/zitadel/issues/9461
Co-authored-by: Iraq Jaber <IraqJaber@gmail.com>
# Which Problems Are Solved
- Lack of support for the Romanian language in the application.
# How the Problems Are Solved
- Added translations for the Romanian language
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
Actions v2 are not executed in different functions, as provided by the
actions v1.
# How the Problems Are Solved
Add functionality to call actions v2 through OIDC and SAML logic to
complement tokens and SAMLResponses.
# Additional Changes
- Corrected testing for retrieved intent information
- Added testing for IDP types
- Corrected handling of context for issuer in SAML logic
# Additional Context
- Closes#7247
- Dependent on https://github.com/zitadel/saml/pull/97
- docs for migration are done in separate issue:
https://github.com/zitadel/zitadel/issues/9456
---------
Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
# Which Problems Are Solved
Currently I am not able to run the new login with a service account with
an IAM_OWNER role.
As the role is missing some permissions which the LOGIN_CLIENT role does
have
# How the Problems Are Solved
Added session permissions to the IAM_OWNER
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
