fix(setup): init projections (#7194)

Even though this is a feature it's released as fix so that we can back port to earlier revisions.

As reported by multiple users startup of ZITADEL after leaded to downtime and worst case rollbacks to the previously deployed version.

The problem starts rising when there are too many events to process after the start of ZITADEL. The root cause are changes on projections (database tables) which must be recomputed. This PR solves this problem by adding a new step to the setup phase which prefills the projections. The step can be enabled by adding the `--init-projections`-flag to `setup`, `start-from-init` and `start-from-setup`. Setting this flag results in potentially longer duration of the setup phase but reduces the risk of the problems mentioned in the paragraph above.

docs/docs/self-hosting/manage/reverseproxy/docker-compose.yaml

@@ -5,7 +5,7 @@ services:
   zitadel-disabled-tls:
     extends:
       service: zitadel-init
-    command: 'start-from-setup --masterkey "MasterkeyNeedsToHave32Characters" --config /zitadel.yaml --steps /zitadel.yaml'
+    command: 'start-from-setup --init-projections --masterkey "MasterkeyNeedsToHave32Characters" --config /zitadel.yaml --steps /zitadel.yaml'
     environment:
       - ZITADEL_EXTERNALPORT=80
       - ZITADEL_EXTERNALSECURE=false
@@ -19,7 +19,7 @@ services:
   zitadel-external-tls:
     extends:
       service: zitadel-init
-    command: 'start-from-setup --masterkey "MasterkeyNeedsToHave32Characters" --config /zitadel.yaml --steps /zitadel.yaml'
+    command: 'start-from-setup --init-projections --masterkey "MasterkeyNeedsToHave32Characters" --config /zitadel.yaml --steps /zitadel.yaml'
     environment:
       - ZITADEL_EXTERNALPORT=443
       - ZITADEL_EXTERNALSECURE=true
@@ -33,7 +33,7 @@ services:
   zitadel-enabled-tls:
     extends:
       service: zitadel-init
-    command: 'start-from-setup --masterkey "MasterkeyNeedsToHave32Characters" --config /zitadel.yaml --steps /zitadel.yaml'
+    command: 'start-from-setup --init-projections --masterkey "MasterkeyNeedsToHave32Characters" --config /zitadel.yaml --steps /zitadel.yaml'
     environment:
       - ZITADEL_EXTERNALPORT=443
       - ZITADEL_EXTERNALSECURE=true

docs/docs/self-hosting/manage/updating_scaling.md

@@ -65,7 +65,7 @@ The init phase is idempotent if executed with the same binary version.
 
 ### The Setup Phase
 
-During `zitadel setup`, ZITADEL creates projection tables and migrates existing data.
+During `zitadel setup`, ZITADEL creates projection tables and migrates existing data, if `--init-projections` is set.
 Depending on the ZITADEL version and the runtime resources,
 this step can take several minutes.
 When deploying a new ZITADEL version,
@@ -85,5 +85,4 @@ Beware, in the background, out-of-date projections
 [recompute their state by replaying all missed events](/docs/concepts/eventstore/implementation#projections).
 If a new ZITADEL version is deployed, this can take quite a long time,
 depending on the amount of events to catch up.
-You probably should consider manually migrating these projections first.
-Refer to the [release notes for v2.14.0](https://github.com/zitadel/zitadel/releases/tag/v2.14.0) as an example.
+You probably should consider providing `--init-projections`-flag to the [Setup Phase](#the-setup-phase) to shift the synchronization time to previous steps and delay the startup phase until events are caught up.

docs/docs/support/advisory/a10008.md

@@ -0,0 +1,40 @@
+---
+title: Technical Advisory 10008
+---
+
+## Date and Version
+
+Version: 2.44.0
+
+Date: 2024-01-25
+
+## Description
+
+Version 2.44.0 introduces a new flag `--init-projections` to `zitadel setup` commands (`setup`, `start-from-setup`, `start-from-init`)
+
+This flag enables prefilling of newly added or changed projections (database tables) during setup phase instead of start phase which are used to query data, for example users.
+
+This new feature adds the following fields to the `setup` configuration, previously only used in `start`-command:
+
+- `AssetStore`: Storage for assets like user avatar, organization logo, icon, font, etc.
+- `OIDC`
+- `Login`
+- `WebAuthNName`
+- `Telemetry`
+- `SystemAPIUsers`
+
+If you use different configurations on `setup` and `start` and have overwritten previously mentioned configurations please make sure to also add them to the configuration provided to ZITADEL `setup` command.
+
+## Statement
+
+Filling of projections can get time consuming as your system grows and this can cause downtime of self hosted installations of ZITADEL because queries first need to ensure data consistency.
+
+Before this release, this step was executed after the start of ZITADEL and therefore lead to inconsistent retrieval of data until the projections were up-to-date. 
+
+## Mitigation
+
+Enable the `--init-projections`-flag in setup phase and make sure the previous deployment of ZITADEL remains active until the new revision started properly.
+
+## Impact
+
+Decreases downtime of starting new revisions with new or changed projections dramatically.

docs/docs/support/technical_advisory.mdx

@@ -132,7 +132,7 @@ We understand that these advisories may include breaking changes, and we aim to
   </tr>
   <tr>
     <td>
-      <a href="./advisory/a10007">A-10006</a>
+      <a href="./advisory/a10007">A-10007</a>
     </td>
     <td>Additional grant to cockroach database user</td>
     <td>Breaking Behaviour Change</td>
@@ -142,6 +142,18 @@ We understand that these advisories may include breaking changes, and we aim to
     <td>Upcoming</td>
     <td>Upcoming</td>
   </tr>
+  <tr>
+    <td>
+      <a href="./advisory/a10008">A-10008</a>
+    </td>
+    <td>New flag to prefill projections during setup instead of after start</td>
+    <td>Feature description</td>
+    <td>
+      new flag `--init-projections` introduced to `zitadel setup` commands (`setup`, `start-from-setup`, `start-from-init`)
+    </td>
+    <td>2.44.0</td>
+    <td>2024-01-25</td>
+  </tr>
 </table>
 
 ## Subscribe to our Mailing List
@@ -162,7 +174,7 @@ Therefore, it is important to be aware of breaking behavior changes and their po
 
 ### Expected downtime during upgrade
 
-Expected downtime during upgrade means that ZITADEL might become unavailable during and upgrade.
+Expected downtime during upgrade means that ZITADEL might become unavailable during an upgrade.
 ZITADEL is built for [zero downtime upgrades](/docs/concepts/architecture/solution#zero-downtime-updates) at upgrades can be executed without downtime by just updating to a more recent version.
 When deploying certain changes a zero downtime upgrade might not be possible, for example to guarantee data integrity.
 In such cases we will issue a technical advisory to make you aware of this unexpected behavior.