Go to file
Zach Hirschtritt 7fceb5eaf8 fix: Auto cleanup failed Setup steps if process is killed (#9736)
# Which Problems Are Solved

When running a long-running Zitadel Setup, Kubernetes might decide to
move a pod to a new node automatically. Currently, this puts any
migrations into a broken state that an operator needs to manually run
the "cleanup" command on - assuming they catch the error.

The only super long running commands are typically projection pre-fill
operations, which depending on the size of the event table for that
projection, can take many hours - plenty of time for Kubernetes to make
unexpected decisions, especially in a busy cluster.

# How the Problems Are Solved

This change listens on `os.Interrupt` and `syscall.SIGTERM`, cancels the
current Setup context, and runs the `Cleanup` command. The logs then
look something like this:
```shell
...
INFO[0000] verify migration                              caller="/Users/zach/src/zitadel/internal/migration/migration.go:43" name=repeatable_delete_stale_org_fields
INFO[0000] starting migration                            caller="/Users/zach/src/zitadel/internal/migration/migration.go:66" name=repeatable_delete_stale_org_fields
INFO[0000] execute delete query                          caller="/Users/zach/src/zitadel/cmd/setup/39.go:37" instance_id=281297936179003398 migration=repeatable_delete_stale_org_fields progress=1/1
INFO[0000] verify migration                              caller="/Users/zach/src/zitadel/internal/migration/migration.go:43" name=repeatable_fill_fields_for_instance_domains
INFO[0000] starting migration                            caller="/Users/zach/src/zitadel/internal/migration/migration.go:66" name=repeatable_fill_fields_for_instance_domains
----- SIGTERM signal issued -----
INFO[0000] received interrupt signal, shutting down: interrupt  caller="/Users/zach/src/zitadel/cmd/setup/setup.go:121"
INFO[0000] query failed                                  caller="/Users/zach/src/zitadel/internal/eventstore/repository/sql/query.go:135" error="timeout: context already done: context canceled"
DEBU[0000] filter eventstore failed                      caller="/Users/zach/src/zitadel/internal/eventstore/handler/v2/field_handler.go:155" error="ID=SQL-KyeAx Message=unable to filter events Parent=(timeout: context already done: context canceled)" projection=instance_domain_fields
DEBU[0000] unable to rollback tx                         caller="/Users/zach/src/zitadel/internal/eventstore/handler/v2/field_handler.go:110" error="sql: transaction has already been committed or rolled back" projection=instance_domain_fields
INFO[0000] process events failed                         caller="/Users/zach/src/zitadel/internal/eventstore/handler/v2/field_handler.go:72" error="ID=SQL-KyeAx Message=unable to filter events Parent=(timeout: context already done: context canceled)" projection=instance_domain_fields
DEBU[0000] trigger iteration                             caller="/Users/zach/src/zitadel/internal/eventstore/handler/v2/field_handler.go:73" iteration=0 projection=instance_domain_fields
ERRO[0000] migration failed                              caller="/Users/zach/src/zitadel/internal/migration/migration.go:68" error="ID=SQL-KyeAx Message=unable to filter events Parent=(timeout: context already done: context canceled)" name=repeatable_fill_fields_for_instance_domains
ERRO[0000] migration finish failed                       caller="/Users/zach/src/zitadel/internal/migration/migration.go:71" error="context canceled" name=repeatable_fill_fields_for_instance_domains
----- Cleanup before exiting -----
INFO[0000] cleanup started                               caller="/Users/zach/src/zitadel/cmd/setup/cleanup.go:30"
INFO[0000] cleanup migration                             caller="/Users/zach/src/zitadel/cmd/setup/cleanup.go:47" name=repeatable_fill_fields_for_instance_domains
```

# Additional Changes

* `mustExecuteMigration` -> `executeMigration`: **must**Execute logged a
Fatal error previously which calls os.Exit so no cleanup was possible.
Instead, this PR returns an error and assigns it to a shared error in
the Setup closure that defer can check.
* `initProjections` now returns an error instead of exiting

# Additional Context

This behavior might be unwelcome or at least unexpected in some cases.
Putting it behind a feature flag or config setting is likely a good
followup.

---------

Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
(cherry picked from commit aa9ef8b49e)
2025-04-29 13:04:53 +02:00
2025-04-02 16:53:06 +02:00
2023-07-17 10:08:20 +02:00
2024-07-26 22:39:55 +02:00
2023-07-17 10:08:20 +02:00
2025-01-06 23:21:09 +00:00
2023-07-17 10:08:20 +02:00
2021-02-08 16:48:41 +01:00
2025-04-02 16:53:06 +02:00
2025-04-02 16:53:06 +02:00
2025-04-02 16:53:06 +02:00
2025-04-02 16:53:06 +02:00
2025-04-02 16:53:06 +02:00

Zitadel Logo Zitadel Logo

GitHub Workflow Status (with event) GitHub contributors

Are you searching for a user management tool that is quickly set up like Auth0 and open source like Keycloak?

Do you have a project that requires multi-tenant user management with self-service for your customers?

Look no further — ZITADEL is the identity infrastructure, simplified for you.

We provide you with a wide range of out-of-the-box features to accelerate your project, including:

Multi-tenancy with team management
Secure login
Self-service
OpenID Connect
OAuth2.x
SAML2
LDAP
Passkeys / FIDO2
OTP
SCIM 2.0 Server and an unlimited audit trail is there for you, ready to use.

With ZITADEL, you are assured of a robust and customizable turnkey solution for all your authentication and authorization needs.


🏡 Website 💬 Chat 📋 Docs 🧑‍💻 Blog 📞 Contact

Get started

👉 Quick Start Guide

Deploy ZITADEL (Self-Hosted)

Deploying ZITADEL locally takes less than 3 minutes. Go ahead and give it a try!

See all guides here

If you are interested to get professional support for your self-hosted ZITADEL please reach out to us!

Setup ZITADEL Cloud (SaaS)

If you want to experience a hands-free ZITADEL, you should use ZITADEL Cloud. Available data regions are:

  • 🇺🇸 United States
  • 🇪🇺 European Union
  • 🇦🇺 Australia
  • 🇨🇭 Switzerland

ZITADEL Cloud comes with a free tier, providing you with all the same features as the open-source version. Learn more about the pay-as-you-go pricing.

Adopters

We are grateful to the organizations and individuals who are using ZITADEL. If you are using ZITADEL, please consider adding your name to our Adopters list by submitting a pull request.

Example applications

Clone one of our example applications or deploy them directly to Vercel.

SDKs

Use our SDKs for your favorite language and framework.

Why choose ZITADEL

We built ZITADEL with a complex multi-tenancy architecture in mind and provide the best solution to handle B2B customers and partners. Yet it offers everything you need for a customer identity (CIAM) use case.

Features

Authentication

Multi-Tenancy

Integration

Self-Service

Deployment

Track upcoming features on our roadmap and follow our changelog for recent updates.

How To Contribute

Find details about how you can contribute in our Contribution Guide. Join our Discord Chat to get help.

Contributors

Made with contrib.rocks.

Showcase

Quick Start Guide

Secure a React Application using OpenID Connect Authorization Code with PKCE

Quick Start Guide

Login with Passkeys

Use our login widget to allow easy and secure access to your applications and enjoy all the benefits of Passkeys (FIDO 2 / WebAuthN):

Passkeys

Admin Console

Use Console or our APIs to setup organizations, projects and applications.

Console Showcase

Login V2

Check out our new Login V2 version in our typescript repository or in our documentation [New Login Showcase]

Security

You can find our security policy here.

Technical Advisories are published regarding major issues with the ZITADEL platform that could potentially impact security or stability in production environments.

License

here are our exact licensing terms.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See our license for detailed information governing permissions and limitations on use.

Languages
Go 82.6%
TypeScript 10.3%
HTML 4.2%
SCSS 2%
CSS 0.6%
Other 0.1%