mirror of
https://github.com/zitadel/zitadel.git
synced 2025-12-03 18:52:20 +00:00
205229f0d1
# Which Problems Are Solved
[A recent performance
enhancement]((https://github.com/zitadel/zitadel/pull/9497)) aimed at
optimizing event store queries, specifically those involving multiple
aggregate type filters, has successfully improved index utilization.
While the query planner now correctly selects relevant indexes, it
employs [bitmap index
scans](https://www.postgresql.org/docs/current/indexes-bitmap-scans.html)
to retrieve data.
This approach, while beneficial in many scenarios, introduces a
potential I/O bottleneck. The bitmap index scan first identifies the
required database blocks and then utilizes a bitmap to access the
corresponding rows from the table's heap. This subsequent "bitmap heap
scan" can result in significant I/O overhead, particularly when queries
return a substantial number of rows across numerous data pages.
## Impact:
Under heavy load or with queries filtering for a wide range of events
across multiple aggregate types, this increased I/O activity may lead
to:
- Increased query latency.
- Elevated disk utilization.
- Potential performance degradation of the event store and dependent
systems.
# How the Problems Are Solved
To address this I/O bottleneck and further optimize query performance,
the projection handler has been modified. Instead of employing multiple
OR clauses for each aggregate type, the aggregate and event type filters
are now combined using IN ARRAY filters.
Technical Details:
This change allows the PostgreSQL query planner to leverage [index-only
scans](https://www.postgresql.org/docs/current/indexes-index-only-scans.html).
By utilizing IN ARRAY filters, the database can efficiently retrieve the
necessary data directly from the index, eliminating the need to access
the table's heap. This results in:
* Reduced I/O: Index-only scans significantly minimize disk I/O
operations, as the database avoids reading data pages from the main
table.
* Improved Query Performance: By reducing I/O, query execution times are
substantially improved, leading to lower latency.
# Additional Changes
- rollback of https://github.com/zitadel/zitadel/pull/9497
# Additional Information
## Query Plan of previous query
```sql
SELECT
created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision
FROM
eventstore.events2
WHERE
instance_id = '<INSTANCE_ID>'
AND (
(
instance_id = '<INSTANCE_ID>'
AND "position" > <POSITION>
AND aggregate_type = 'project'
AND event_type = ANY(ARRAY[
'project.application.added'
,'project.application.changed'
,'project.application.deactivated'
,'project.application.reactivated'
,'project.application.removed'
,'project.removed'
,'project.application.config.api.added'
,'project.application.config.api.changed'
,'project.application.config.api.secret.changed'
,'project.application.config.api.secret.updated'
,'project.application.config.oidc.added'
,'project.application.config.oidc.changed'
,'project.application.config.oidc.secret.changed'
,'project.application.config.oidc.secret.updated'
,'project.application.config.saml.added'
,'project.application.config.saml.changed'
])
) OR (
instance_id = '<INSTANCE_ID>'
AND "position" > <POSITION>
AND aggregate_type = 'org'
AND event_type = 'org.removed'
) OR (
instance_id = '<INSTANCE_ID>'
AND "position" > <POSITION>
AND aggregate_type = 'instance'
AND event_type = 'instance.removed'
)
)
AND "position" > 1741600905.3495
AND "position" < (
SELECT
COALESCE(EXTRACT(EPOCH FROM min(xact_start)), EXTRACT(EPOCH FROM now()))
FROM
pg_stat_activity
WHERE
datname = current_database()
AND application_name = ANY(ARRAY['zitadel_es_pusher_', 'zitadel_es_pusher', 'zitadel_es_pusher_<INSTANCE_ID>'])
AND state <> 'idle'
)
ORDER BY "position", in_tx_order LIMIT 200 OFFSET 1;
```
```
Limit (cost=120.08..120.09 rows=7 width=361) (actual time=2.167..2.172 rows=0 loops=1)
Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
InitPlan 1
-> Aggregate (cost=2.74..2.76 rows=1 width=32) (actual time=1.813..1.815 rows=1 loops=1)
Output: COALESCE(EXTRACT(epoch FROM min(s.xact_start)), EXTRACT(epoch FROM now()))
-> Nested Loop (cost=0.00..2.74 rows=1 width=8) (actual time=1.803..1.805 rows=0 loops=1)
Output: s.xact_start
Join Filter: (d.oid = s.datid)
-> Seq Scan on pg_catalog.pg_database d (cost=0.00..1.07 rows=1 width=4) (actual time=0.016..0.021 rows=1 loops=1)
Output: d.oid, d.datname, d.datdba, d.encoding, d.datlocprovider, d.datistemplate, d.datallowconn, d.dathasloginevt, d.datconnlimit, d.datfrozenxid, d.datminmxid, d.dattablespace, d.datcollate, d.datctype, d.datlocale, d.daticurules, d.datcollversion, d.datacl
Filter: (d.datname = current_database())
Rows Removed by Filter: 4
-> Function Scan on pg_catalog.pg_stat_get_activity s (cost=0.00..1.63 rows=3 width=16) (actual time=1.781..1.781 rows=0 loops=1)
Output: s.datid, s.pid, s.usesysid, s.application_name, s.state, s.query, s.wait_event_type, s.wait_event, s.xact_start, s.query_start, s.backend_start, s.state_change, s.client_addr, s.client_hostname, s.client_port, s.backend_xid, s.backend_xmin, s.backend_type, s.ssl, s.sslversion, s.sslcipher, s.sslbits, s.ssl_client_dn, s.ssl_client_serial, s.ssl_issuer_dn, s.gss_auth, s.gss_princ, s.gss_enc, s.gss_delegation, s.leader_pid, s.query_id
Function Call: pg_stat_get_activity(NULL::integer)
Filter: ((s.state <> 'idle'::text) AND (s.application_name = ANY ('{zitadel_es_pusher_,zitadel_es_pusher,zitadel_es_pusher_<INSTANCE_ID>}'::text[])))
Rows Removed by Filter: 49
-> Sort (cost=117.31..117.33 rows=8 width=361) (actual time=2.167..2.168 rows=0 loops=1)
Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
Sort Key: events2."position", events2.in_tx_order
Sort Method: quicksort Memory: 25kB
-> Bitmap Heap Scan on eventstore.events2 (cost=84.92..117.19 rows=8 width=361) (actual time=2.088..2.089 rows=0 loops=1)
Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
Recheck Cond: (((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'project'::text) AND (events2.event_type = ANY ('{project.application.added,project.application.changed,project.application.deactivated,project.application.reactivated,project.application.removed,project.removed,project.application.config.api.added,project.application.config.api.changed,project.application.config.api.secret.changed,project.application.config.api.secret.updated,project.application.config.oidc.added,project.application.config.oidc.changed,project.application.config.oidc.secret.changed,project.application.config.oidc.secret.updated,project.application.config.saml.added,project.application.config.saml.changed}'::text[])) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1)) OR ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'org'::text) AND (events2.event_type = 'org.removed'::text) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1)) OR ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'instance'::text) AND (events2.event_type = 'instance.removed'::text) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1)))
-> BitmapOr (cost=84.88..84.88 rows=8 width=0) (actual time=2.080..2.081 rows=0 loops=1)
-> Bitmap Index Scan on es_projection (cost=0.00..75.44 rows=8 width=0) (actual time=2.016..2.017 rows=0 loops=1)
Index Cond: ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'project'::text) AND (events2.event_type = ANY ('{project.application.added,project.application.changed,project.application.deactivated,project.application.reactivated,project.application.removed,project.removed,project.application.config.api.added,project.application.config.api.changed,project.application.config.api.secret.changed,project.application.config.api.secret.updated,project.application.config.oidc.added,project.application.config.oidc.changed,project.application.config.oidc.secret.changed,project.application.config.oidc.secret.updated,project.application.config.saml.added,project.application.config.saml.changed}'::text[])) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1))
-> Bitmap Index Scan on es_projection (cost=0.00..4.71 rows=1 width=0) (actual time=0.016..0.016 rows=0 loops=1)
Index Cond: ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'org'::text) AND (events2.event_type = 'org.removed'::text) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1))
-> Bitmap Index Scan on es_projection (cost=0.00..4.71 rows=1 width=0) (actual time=0.045..0.045 rows=0 loops=1)
Index Cond: ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = 'instance'::text) AND (events2.event_type = 'instance.removed'::text) AND (events2."position" > <POSITION>) AND (events2."position" > 1741600905.3495) AND (events2."position" < (InitPlan 1).col1))
Query Identifier: 3194938266011254479
Planning Time: 1.295 ms
Execution Time: 2.832 ms
```
## Query Plan of new query
```sql
SELECT
created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision
FROM
eventstore.events2
WHERE
instance_id = '<INSTANCE_ID>'
AND "position" > <POSITION>
AND aggregate_type = ANY(ARRAY['project', 'instance', 'org'])
AND event_type = ANY(ARRAY[
'project.application.added'
,'project.application.changed'
,'project.application.deactivated'
,'project.application.reactivated'
,'project.application.removed'
,'project.removed'
,'project.application.config.api.added'
,'project.application.config.api.changed'
,'project.application.config.api.secret.changed'
,'project.application.config.api.secret.updated'
,'project.application.config.oidc.added'
,'project.application.config.oidc.changed'
,'project.application.config.oidc.secret.changed'
,'project.application.config.oidc.secret.updated'
,'project.application.config.saml.added'
,'project.application.config.saml.changed'
,'org.removed'
,'instance.removed'
])
AND "position" < (
SELECT
COALESCE(EXTRACT(EPOCH FROM min(xact_start)), EXTRACT(EPOCH FROM now()))
FROM
pg_stat_activity
WHERE
datname = current_database()
AND application_name = ANY(ARRAY['zitadel_es_pusher_', 'zitadel_es_pusher', 'zitadel_es_pusher_<INSTANCE_ID>'])
AND state <> 'idle'
)
ORDER BY "position", in_tx_order LIMIT 200 OFFSET 1;
```
```
Limit (cost=293.34..293.36 rows=8 width=361) (actual time=4.686..4.689 rows=0 loops=1)
Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
InitPlan 1
-> Aggregate (cost=2.74..2.76 rows=1 width=32) (actual time=1.717..1.719 rows=1 loops=1)
Output: COALESCE(EXTRACT(epoch FROM min(s.xact_start)), EXTRACT(epoch FROM now()))
-> Nested Loop (cost=0.00..2.74 rows=1 width=8) (actual time=1.658..1.659 rows=0 loops=1)
Output: s.xact_start
Join Filter: (d.oid = s.datid)
-> Seq Scan on pg_catalog.pg_database d (cost=0.00..1.07 rows=1 width=4) (actual time=0.026..0.028 rows=1 loops=1)
Output: d.oid, d.datname, d.datdba, d.encoding, d.datlocprovider, d.datistemplate, d.datallowconn, d.dathasloginevt, d.datconnlimit, d.datfrozenxid, d.datminmxid, d.dattablespace, d.datcollate, d.datctype, d.datlocale, d.daticurules, d.datcollversion, d.datacl
Filter: (d.datname = current_database())
Rows Removed by Filter: 4
-> Function Scan on pg_catalog.pg_stat_get_activity s (cost=0.00..1.63 rows=3 width=16) (actual time=1.628..1.628 rows=0 loops=1)
Output: s.datid, s.pid, s.usesysid, s.application_name, s.state, s.query, s.wait_event_type, s.wait_event, s.xact_start, s.query_start, s.backend_start, s.state_change, s.client_addr, s.client_hostname, s.client_port, s.backend_xid, s.backend_xmin, s.backend_type, s.ssl, s.sslversion, s.sslcipher, s.sslbits, s.ssl_client_dn, s.ssl_client_serial, s.ssl_issuer_dn, s.gss_auth, s.gss_princ, s.gss_enc, s.gss_delegation, s.leader_pid, s.query_id
Function Call: pg_stat_get_activity(NULL::integer)
Filter: ((s.state <> 'idle'::text) AND (s.application_name = ANY ('{zitadel_es_pusher_,zitadel_es_pusher,zitadel_es_pusher_<INSTANCE_ID>}'::text[])))
Rows Removed by Filter: 42
-> Sort (cost=290.58..290.60 rows=9 width=361) (actual time=4.685..4.685 rows=0 loops=1)
Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
Sort Key: events2."position", events2.in_tx_order
Sort Method: quicksort Memory: 25kB
-> Index Scan using es_projection on eventstore.events2 (cost=0.70..290.43 rows=9 width=361) (actual time=4.616..4.617 rows=0 loops=1)
Output: events2.created_at, events2.event_type, events2.sequence, events2."position", events2.payload, events2.creator, events2.owner, events2.instance_id, events2.aggregate_type, events2.aggregate_id, events2.revision, events2.in_tx_order
Index Cond: ((events2.instance_id = '<INSTANCE_ID>'::text) AND (events2.aggregate_type = ANY ('{project,instance,org}'::text[])) AND (events2.event_type = ANY ('{project.application.added,project.application.changed,project.application.deactivated,project.application.reactivated,project.application.removed,project.removed,project.application.config.api.added,project.application.config.api.changed,project.application.config.api.secret.changed,project.application.config.api.secret.updated,project.application.config.oidc.added,project.application.config.oidc.changed,project.application.config.oidc.secret.changed,project.application.config.oidc.secret.updated,project.application.config.saml.added,project.application.config.saml.changed,org.removed,instance.removed}'::text[])) AND (events2."position" > <POSITION>) AND (events2."position" < (InitPlan 1).col1))
Query Identifier: -8254550537132386499
Planning Time: 2.864 ms
Execution Time: 5.414 ms
```
(cherry picked from commit e36f402e09)
Are you searching for a user management tool that is quickly set up like Auth0 and open source like Keycloak?
Do you have a project that requires multi-tenant user management with self-service for your customers?
Look no further — ZITADEL is the identity infrastructure, simplified for you.
We provide you with a wide range of out-of-the-box features to accelerate your project, including:
✅ Multi-tenancy with team management
✅ Secure login
✅ Self-service
✅ OpenID Connect
✅ OAuth2.x
✅ SAML2
✅ LDAP
✅ Passkeys / FIDO2
✅ OTP
✅ SCIM 2.0 Server
and an unlimited audit trail is there for you, ready to use.
With ZITADEL, you are assured of a robust and customizable turnkey solution for all your authentication and authorization needs.
🏡 Website 💬 Chat 📋 Docs 🧑💻 Blog 📞 Contact
Get started
Deploy ZITADEL (Self-Hosted)
Deploying ZITADEL locally takes less than 3 minutes. Go ahead and give it a try!
See all guides here
If you are interested to get professional support for your self-hosted ZITADEL please reach out to us!
Setup ZITADEL Cloud (SaaS)
If you want to experience a hands-free ZITADEL, you should use ZITADEL Cloud. Available data regions are:
- 🇺🇸 United States
- 🇪🇺 European Union
- 🇦🇺 Australia
- 🇨🇭 Switzerland
ZITADEL Cloud comes with a free tier, providing you with all the same features as the open-source version. Learn more about the pay-as-you-go pricing.
Adopters
We are grateful to the organizations and individuals who are using ZITADEL. If you are using ZITADEL, please consider adding your name to our Adopters list by submitting a pull request.
Example applications
Clone one of our example applications or deploy them directly to Vercel.
SDKs
Use our SDKs for your favorite language and framework.
Why choose ZITADEL
We built ZITADEL with a complex multi-tenancy architecture in mind and provide the best solution to handle B2B customers and partners. Yet it offers everything you need for a customer identity (CIAM) use case.
- API-first approach
- Multi-tenancy authentication and access management
- Strong audit trail thanks to event sourcing as storage pattern
- Actions to react on events with custom code and extended ZITADEL for you needs
- Branding for a uniform user experience across multiple organizations
- Self-service for end-users, business customers, and administrators
- CockroachDB or a Postgres database as reliable and widespread storage option
Features
Authentication
- Single Sign On (SSO)
- Passkeys support (FIDO2 / WebAuthN)
- Username / Password
- Multifactor authentication with OTP, U2F, Email OTP, SMS OTP
- LDAP
- External enterprise identity providers and social logins
- Device authorization
- OpenID Connect certified => OIDC Endpoints
- SAML 2.0 => SAML Endpoints
- Custom sessions if you need to go beyond OIDC or SAML
- Machine-to-machine with JWT profile, Personal Access Tokens (PAT), and Client Credentials
- Token exchange and impersonation
- Beta: Hosted Login V2 our new login version 2.0
Multi-Tenancy
- Identity Brokering with templates for popular identity providers
- Customizable onboaring for B2B and their users
- Delegate role management to third-parties
- Domain discovery
Integration
- GRPC and REST APIs for every functionality and resource
- Actions to call any API, send webhooks, adjust workflows, or customize tokens
- Role Based Access Control (RBAC)
- SCIM 2.0 Server
- Examples and SDKs
- Audit Log and SOC/SIEM
- User registration and onboarding
- Hosted and custom login user interface
Self-Service
- Self-registration including verification
- Self-service for end-users, business customers, and administrators
- Administration UI (Console)
Deployment
- Postgres (version >= 14) or CockroachDB (version latest stable)
- Zero Downtime Updates
- High scalability
Track upcoming features on our roadmap and follow our changelog for recent updates.
How To Contribute
Find details about how you can contribute in our Contribution Guide. Join our Discord Chat to get help.
Contributors
Made with contrib.rocks.
Showcase
Quick Start Guide
Secure a React Application using OpenID Connect Authorization Code with PKCE
Login with Passkeys
Use our login widget to allow easy and secure access to your applications and enjoy all the benefits of Passkeys (FIDO 2 / WebAuthN):
Admin Console
Use Console or our APIs to setup organizations, projects and applications.
Login V2
Check out our new Login V2 version in our typescript repository or in our documentation
[]
Security
You can find our security policy here.
Technical Advisories are published regarding major issues with the ZITADEL platform that could potentially impact security or stability in production environments.
License
here are our exact licensing terms.
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See our license for detailed information governing permissions and limitations on use.