Files
zitadel/internal/migration/count_trigger.sql

93 lines
2.9 KiB
MySQL
Raw Normal View History

feat(projections): resource counters (#9979) # Which Problems Are Solved Add the ability to keep track of the current counts of projection resources. We want to prevent calling `SELECT COUNT(*)` on tables, as that forces a full scan and sudden spikes of DB resource uses. # How the Problems Are Solved - A resource_counts table is added - Triggers that increment and decrement the counted values on inserts and deletes - Triggers that delete all counts of a table when the source table is TRUNCATEd. This is not in the business logic, but prevents wrong counts in case someone want to force a re-projection. - Triggers that delete all counts if the parent resource is deleted - Script to pre-populate the resource_counts table when a new source table is added. The triggers are reusable for any type of resource, in case we choose to add more in the future. Counts are aggregated by a given parent. Currently only `instance` and `organization` are defined as possible parent. This can later be extended to other types, such as `project`, should the need arise. I deliberately chose to use `parent_id` to distinguish from the de-factor `resource_owner` which is usually an organization ID. For example: - For users the parent is an organization and the `parent_id` matches `resource_owner`. - For organizations the parent is an instance, but the `resource_owner` is the `org_id`. In this case the `parent_id` is the `instance_id`. - Applications would have a similar problem, where the parent is a project, but the `resource_owner` is the `org_id` # Additional Context Closes https://github.com/zitadel/zitadel/issues/9957
2025-06-03 17:15:30 +03:00
{{ define "count_trigger" -}}
feat(service ping): add additional resource counts (#10621) # Which Problems Are Solved Using the service ping, we want to have some additional insights to how zitadel is configured. The current resource count report contains already some amount of configured policies, such as the login_policy. But we do not know if for example MFA is enforced. # How the Problems Are Solved - Added the following counts to the report: - service users per organization - MFA enforcements (though login policy) - Notification policies with password change option enabled - SCIM provisioned users (using user metadata) - Since all of the above are conditional based on at least a column inside a projection, a new `migration.CountTriggerConditional` has been added, where a condition (column values) and an option to track updates on that column should be considered for the count. - For this to be possible, the following changes had to be made to the existing sql resources: - the `resource_name` has been added to unique constraint on the `projection.resource_counts` table - triggers have been added / changed to individually track `INSERT`, `UPDATE`(s) and `DELETE` and be able to handle conditions - an optional argument has been added to the `projections.count_resource()` function to allow providing the information to `UP` or `DOWN` count the resource on an update. # Additional Changes None # Additional Context - partially solves #10244 (reporting audit log retention limit will be handled in #10245 directly) - backport to v4.x (cherry picked from commit 2dbe21fb300907ac430ad3badea1cda140b792a5)
2025-09-08 18:30:03 +02:00
-- In case the old trigger exists, drop it to prevent duplicated counts.
DROP TRIGGER IF EXISTS count_{{ .Resource }} ON {{ .Table }};
CREATE OR REPLACE TRIGGER count_{{ .Resource }}_insert
AFTER INSERT
feat(projections): resource counters (#9979) # Which Problems Are Solved Add the ability to keep track of the current counts of projection resources. We want to prevent calling `SELECT COUNT(*)` on tables, as that forces a full scan and sudden spikes of DB resource uses. # How the Problems Are Solved - A resource_counts table is added - Triggers that increment and decrement the counted values on inserts and deletes - Triggers that delete all counts of a table when the source table is TRUNCATEd. This is not in the business logic, but prevents wrong counts in case someone want to force a re-projection. - Triggers that delete all counts if the parent resource is deleted - Script to pre-populate the resource_counts table when a new source table is added. The triggers are reusable for any type of resource, in case we choose to add more in the future. Counts are aggregated by a given parent. Currently only `instance` and `organization` are defined as possible parent. This can later be extended to other types, such as `project`, should the need arise. I deliberately chose to use `parent_id` to distinguish from the de-factor `resource_owner` which is usually an organization ID. For example: - For users the parent is an organization and the `parent_id` matches `resource_owner`. - For organizations the parent is an instance, but the `resource_owner` is the `org_id`. In this case the `parent_id` is the `instance_id`. - Applications would have a similar problem, where the parent is a project, but the `resource_owner` is the `org_id` # Additional Context Closes https://github.com/zitadel/zitadel/issues/9957
2025-06-03 17:15:30 +03:00
ON {{ .Table }}
FOR EACH ROW
feat(service ping): add additional resource counts (#10621) # Which Problems Are Solved Using the service ping, we want to have some additional insights to how zitadel is configured. The current resource count report contains already some amount of configured policies, such as the login_policy. But we do not know if for example MFA is enforced. # How the Problems Are Solved - Added the following counts to the report: - service users per organization - MFA enforcements (though login policy) - Notification policies with password change option enabled - SCIM provisioned users (using user metadata) - Since all of the above are conditional based on at least a column inside a projection, a new `migration.CountTriggerConditional` has been added, where a condition (column values) and an option to track updates on that column should be considered for the count. - For this to be possible, the following changes had to be made to the existing sql resources: - the `resource_name` has been added to unique constraint on the `projection.resource_counts` table - triggers have been added / changed to individually track `INSERT`, `UPDATE`(s) and `DELETE` and be able to handle conditions - an optional argument has been added to the `projections.count_resource()` function to allow providing the information to `UP` or `DOWN` count the resource on an update. # Additional Changes None # Additional Context - partially solves #10244 (reporting audit log retention limit will be handled in #10245 directly) - backport to v4.x (cherry picked from commit 2dbe21fb300907ac430ad3badea1cda140b792a5)
2025-09-08 18:30:03 +02:00
-- Only count if the conditions are met in the newly added row.
{{if .Conditions}}WHEN ({{.Conditions.ToSQL "NEW" true}}){{end}}
feat(projections): resource counters (#9979) # Which Problems Are Solved Add the ability to keep track of the current counts of projection resources. We want to prevent calling `SELECT COUNT(*)` on tables, as that forces a full scan and sudden spikes of DB resource uses. # How the Problems Are Solved - A resource_counts table is added - Triggers that increment and decrement the counted values on inserts and deletes - Triggers that delete all counts of a table when the source table is TRUNCATEd. This is not in the business logic, but prevents wrong counts in case someone want to force a re-projection. - Triggers that delete all counts if the parent resource is deleted - Script to pre-populate the resource_counts table when a new source table is added. The triggers are reusable for any type of resource, in case we choose to add more in the future. Counts are aggregated by a given parent. Currently only `instance` and `organization` are defined as possible parent. This can later be extended to other types, such as `project`, should the need arise. I deliberately chose to use `parent_id` to distinguish from the de-factor `resource_owner` which is usually an organization ID. For example: - For users the parent is an organization and the `parent_id` matches `resource_owner`. - For organizations the parent is an instance, but the `resource_owner` is the `org_id`. In this case the `parent_id` is the `instance_id`. - Applications would have a similar problem, where the parent is a project, but the `resource_owner` is the `org_id` # Additional Context Closes https://github.com/zitadel/zitadel/issues/9957
2025-06-03 17:15:30 +03:00
EXECUTE FUNCTION projections.count_resource(
'{{ .ParentType }}',
'{{ .InstanceIDColumn }}',
'{{ .ParentIDColumn }}',
'{{ .Resource }}'
);
feat(service ping): add additional resource counts (#10621) # Which Problems Are Solved Using the service ping, we want to have some additional insights to how zitadel is configured. The current resource count report contains already some amount of configured policies, such as the login_policy. But we do not know if for example MFA is enforced. # How the Problems Are Solved - Added the following counts to the report: - service users per organization - MFA enforcements (though login policy) - Notification policies with password change option enabled - SCIM provisioned users (using user metadata) - Since all of the above are conditional based on at least a column inside a projection, a new `migration.CountTriggerConditional` has been added, where a condition (column values) and an option to track updates on that column should be considered for the count. - For this to be possible, the following changes had to be made to the existing sql resources: - the `resource_name` has been added to unique constraint on the `projection.resource_counts` table - triggers have been added / changed to individually track `INSERT`, `UPDATE`(s) and `DELETE` and be able to handle conditions - an optional argument has been added to the `projections.count_resource()` function to allow providing the information to `UP` or `DOWN` count the resource on an update. # Additional Changes None # Additional Context - partially solves #10244 (reporting audit log retention limit will be handled in #10245 directly) - backport to v4.x (cherry picked from commit 2dbe21fb300907ac430ad3badea1cda140b792a5)
2025-09-08 18:30:03 +02:00
CREATE OR REPLACE TRIGGER count_{{ .Resource }}_delete
AFTER DELETE
ON {{ .Table }}
FOR EACH ROW
-- Only count down if the conditions were met in the old / deleted row.
{{if .Conditions}}WHEN ({{.Conditions.ToSQL "OLD" true}}){{end}}
EXECUTE FUNCTION projections.count_resource(
'{{ .ParentType }}',
'{{ .InstanceIDColumn }}',
'{{ .ParentIDColumn }}',
'{{ .Resource }}'
);
{{if .TrackChange}}
CREATE OR REPLACE TRIGGER count_{{ .Resource }}_update_up
AFTER UPDATE
ON {{ .Table }}
FOR EACH ROW
-- Only count up if the conditions are met in the new state, but were not in the old.
WHEN ({{.Conditions.ToSQL "NEW" true}} AND {{.Conditions.ToSQL "OLD" false}})
EXECUTE FUNCTION projections.count_resource(
'{{ .ParentType }}',
'{{ .InstanceIDColumn }}',
'{{ .ParentIDColumn }}',
'{{ .Resource }}',
'UP'
);
CREATE OR REPLACE TRIGGER count_{{ .Resource }}_update_down
AFTER UPDATE
ON {{ .Table }}
FOR EACH ROW
-- Only count down if the conditions are not met in the new state, but were in the old.
WHEN ({{.Conditions.ToSQL "NEW" false}} AND {{.Conditions.ToSQL "OLD" true}})
EXECUTE FUNCTION projections.count_resource(
'{{ .ParentType }}',
'{{ .InstanceIDColumn }}',
'{{ .ParentIDColumn }}',
'{{ .Resource }}',
'DOWN'
);
{{end}}
feat(projections): resource counters (#9979) # Which Problems Are Solved Add the ability to keep track of the current counts of projection resources. We want to prevent calling `SELECT COUNT(*)` on tables, as that forces a full scan and sudden spikes of DB resource uses. # How the Problems Are Solved - A resource_counts table is added - Triggers that increment and decrement the counted values on inserts and deletes - Triggers that delete all counts of a table when the source table is TRUNCATEd. This is not in the business logic, but prevents wrong counts in case someone want to force a re-projection. - Triggers that delete all counts if the parent resource is deleted - Script to pre-populate the resource_counts table when a new source table is added. The triggers are reusable for any type of resource, in case we choose to add more in the future. Counts are aggregated by a given parent. Currently only `instance` and `organization` are defined as possible parent. This can later be extended to other types, such as `project`, should the need arise. I deliberately chose to use `parent_id` to distinguish from the de-factor `resource_owner` which is usually an organization ID. For example: - For users the parent is an organization and the `parent_id` matches `resource_owner`. - For organizations the parent is an instance, but the `resource_owner` is the `org_id`. In this case the `parent_id` is the `instance_id`. - Applications would have a similar problem, where the parent is a project, but the `resource_owner` is the `org_id` # Additional Context Closes https://github.com/zitadel/zitadel/issues/9957
2025-06-03 17:15:30 +03:00
CREATE OR REPLACE TRIGGER truncate_{{ .Resource }}_counts
AFTER TRUNCATE
ON {{ .Table }}
FOR EACH STATEMENT
EXECUTE FUNCTION projections.delete_table_counts();
-- Prevent inserts and deletes while we populate the counts.
LOCK TABLE {{ .Table }} IN SHARE MODE;
-- Populate the resource counts for the existing data in the table.
INSERT INTO projections.resource_counts(
instance_id,
table_name,
parent_type,
parent_id,
resource_name,
amount
)
SELECT
{{ .InstanceIDColumn }},
'{{ .Table }}',
'{{ .ParentType }}',
{{ .ParentIDColumn }},
'{{ .Resource }}',
COUNT(*) AS amount
FROM {{ .Table }}
feat(service ping): add additional resource counts (#10621) # Which Problems Are Solved Using the service ping, we want to have some additional insights to how zitadel is configured. The current resource count report contains already some amount of configured policies, such as the login_policy. But we do not know if for example MFA is enforced. # How the Problems Are Solved - Added the following counts to the report: - service users per organization - MFA enforcements (though login policy) - Notification policies with password change option enabled - SCIM provisioned users (using user metadata) - Since all of the above are conditional based on at least a column inside a projection, a new `migration.CountTriggerConditional` has been added, where a condition (column values) and an option to track updates on that column should be considered for the count. - For this to be possible, the following changes had to be made to the existing sql resources: - the `resource_name` has been added to unique constraint on the `projection.resource_counts` table - triggers have been added / changed to individually track `INSERT`, `UPDATE`(s) and `DELETE` and be able to handle conditions - an optional argument has been added to the `projections.count_resource()` function to allow providing the information to `UP` or `DOWN` count the resource on an update. # Additional Changes None # Additional Context - partially solves #10244 (reporting audit log retention limit will be handled in #10245 directly) - backport to v4.x (cherry picked from commit 2dbe21fb300907ac430ad3badea1cda140b792a5)
2025-09-08 18:30:03 +02:00
{{if .Conditions}}WHERE {{.Conditions.ToSQL .Table true}}{{end}}
feat(projections): resource counters (#9979) # Which Problems Are Solved Add the ability to keep track of the current counts of projection resources. We want to prevent calling `SELECT COUNT(*)` on tables, as that forces a full scan and sudden spikes of DB resource uses. # How the Problems Are Solved - A resource_counts table is added - Triggers that increment and decrement the counted values on inserts and deletes - Triggers that delete all counts of a table when the source table is TRUNCATEd. This is not in the business logic, but prevents wrong counts in case someone want to force a re-projection. - Triggers that delete all counts if the parent resource is deleted - Script to pre-populate the resource_counts table when a new source table is added. The triggers are reusable for any type of resource, in case we choose to add more in the future. Counts are aggregated by a given parent. Currently only `instance` and `organization` are defined as possible parent. This can later be extended to other types, such as `project`, should the need arise. I deliberately chose to use `parent_id` to distinguish from the de-factor `resource_owner` which is usually an organization ID. For example: - For users the parent is an organization and the `parent_id` matches `resource_owner`. - For organizations the parent is an instance, but the `resource_owner` is the `org_id`. In this case the `parent_id` is the `instance_id`. - Applications would have a similar problem, where the parent is a project, but the `resource_owner` is the `org_id` # Additional Context Closes https://github.com/zitadel/zitadel/issues/9957
2025-06-03 17:15:30 +03:00
GROUP BY ({{ .InstanceIDColumn }}, {{ .ParentIDColumn }})
feat(service ping): add additional resource counts (#10621) # Which Problems Are Solved Using the service ping, we want to have some additional insights to how zitadel is configured. The current resource count report contains already some amount of configured policies, such as the login_policy. But we do not know if for example MFA is enforced. # How the Problems Are Solved - Added the following counts to the report: - service users per organization - MFA enforcements (though login policy) - Notification policies with password change option enabled - SCIM provisioned users (using user metadata) - Since all of the above are conditional based on at least a column inside a projection, a new `migration.CountTriggerConditional` has been added, where a condition (column values) and an option to track updates on that column should be considered for the count. - For this to be possible, the following changes had to be made to the existing sql resources: - the `resource_name` has been added to unique constraint on the `projection.resource_counts` table - triggers have been added / changed to individually track `INSERT`, `UPDATE`(s) and `DELETE` and be able to handle conditions - an optional argument has been added to the `projections.count_resource()` function to allow providing the information to `UP` or `DOWN` count the resource on an update. # Additional Changes None # Additional Context - partially solves #10244 (reporting audit log retention limit will be handled in #10245 directly) - backport to v4.x (cherry picked from commit 2dbe21fb300907ac430ad3badea1cda140b792a5)
2025-09-08 18:30:03 +02:00
ON CONFLICT (instance_id, table_name, parent_type, parent_id, resource_name) DO
feat(projections): resource counters (#9979) # Which Problems Are Solved Add the ability to keep track of the current counts of projection resources. We want to prevent calling `SELECT COUNT(*)` on tables, as that forces a full scan and sudden spikes of DB resource uses. # How the Problems Are Solved - A resource_counts table is added - Triggers that increment and decrement the counted values on inserts and deletes - Triggers that delete all counts of a table when the source table is TRUNCATEd. This is not in the business logic, but prevents wrong counts in case someone want to force a re-projection. - Triggers that delete all counts if the parent resource is deleted - Script to pre-populate the resource_counts table when a new source table is added. The triggers are reusable for any type of resource, in case we choose to add more in the future. Counts are aggregated by a given parent. Currently only `instance` and `organization` are defined as possible parent. This can later be extended to other types, such as `project`, should the need arise. I deliberately chose to use `parent_id` to distinguish from the de-factor `resource_owner` which is usually an organization ID. For example: - For users the parent is an organization and the `parent_id` matches `resource_owner`. - For organizations the parent is an instance, but the `resource_owner` is the `org_id`. In this case the `parent_id` is the `instance_id`. - Applications would have a similar problem, where the parent is a project, but the `resource_owner` is the `org_id` # Additional Context Closes https://github.com/zitadel/zitadel/issues/9957
2025-06-03 17:15:30 +03:00
UPDATE SET updated_at = now(), amount = EXCLUDED.amount;
{{- end -}}