TheArchive/zitadel
1
0
Fork 0
mirror of https://github.com/zitadel/zitadel.git synced 2025-04-25 15:10:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
zitadel/cmd/setup/sync_role_permissions.go

77 lines
2.5 KiB
Go
Raw Normal View History

perf: role permissions in database (#9152) # Which Problems Are Solved Currently ZITADEL defines organization and instance member roles and permissions in defaults.yaml. The permission check is done on API call level. For example: "is this user allowed to make this call on this org". This makes sense on the V1 API where the API is permission-level shaped. For example, a search for users always happens in the context of the organization. (Either the organization the calling user belongs to, or through member ship and the x-zitadel-orgid header. However, for resource based APIs we must be able to resolve permissions by object. For example, an IAM_OWNER listing users should be able to get all users in an instance based on the query filters. Alternatively a user may have user.read permissions on one or more orgs. They should be able to read just those users. # How the Problems Are Solved ## Role permission mapping The role permission mappings defined from `defaults.yaml` or local config override are synchronized to the database on every run of `zitadel setup`: - A single query per **aggregate** builds a list of `add` and `remove` actions needed to reach the desired state or role permission mappings from the config. - The required events based on the actions are pushed to the event store. - Events define search fields so that permission checking can use the indices and is strongly consistent for both query and command sides. The migration is split in the following aggregates: - System aggregate for for roles prefixed with `SYSTEM` - Each instance for roles not prefixed with `SYSTEM`. This is in anticipation of instance level management over the API. ## Membership Current instance / org / project membership events now have field table definitions. Like the role permissions this ensures strong consistency while still being able to use the indices of the fields table. A migration is provided to fill the membership fields. ## Permission check I aimed keeping the mental overhead to the developer to a minimal. The provided implementation only provides a permission check for list queries for org level resources, for example users. In the `query` package there is a simple helper function `wherePermittedOrgs` which makes sure the underlying database function is called as part of the `SELECT` query and the permitted organizations are part of the `WHERE` clause. This makes sure results from non-permitted organizations are omitted. Under the hood: - A Pg/PlSQL function searches for a list of organization IDs the passed user has the passed permission. - When the user has the permission on instance level, it returns early with all organizations. - The functions uses a number of views. The views help mapping the fields entries into relational data and simplify the code use for the function. The views provide some pre-filters which allow proper index usage once the final `WHERE` clauses are set by the function. # Additional Changes # Additional Context Closes #9032 Closes https://github.com/zitadel/zitadel/issues/9014 https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for the new permission framework based on this concept.
2025-01-16 11:09:15 +01:00
package setup
import (
"context"
_ "embed"
"fmt"
"github.com/zitadel/logging"
"github.com/zitadel/zitadel/internal/api/authz"
fix(permissions): chunked synchronization of role permission events (#9403) # Which Problems Are Solved Setup fails to push all role permission events when running Zitadel with CockroachDB. `TransactionRetryError`s were visible in logs which finally times out the setup job with `timeout: context deadline exceeded` # How the Problems Are Solved As suggested in the [Cockroach documentation](timeout: context deadline exceeded), _"break down larger transactions"_. The commands to be pushed for the role permissions are chunked in 50 events per push. This chunking is only done with CockroachDB. # Additional Changes - gci run fixed some unrelated imports - access to `command.Commands` for the setup job, so we can reuse the sync logic. # Additional Context Closes #9293 --------- Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
2025-02-26 18:06:50 +02:00
"github.com/zitadel/zitadel/internal/command"
perf: role permissions in database (#9152) # Which Problems Are Solved Currently ZITADEL defines organization and instance member roles and permissions in defaults.yaml. The permission check is done on API call level. For example: "is this user allowed to make this call on this org". This makes sense on the V1 API where the API is permission-level shaped. For example, a search for users always happens in the context of the organization. (Either the organization the calling user belongs to, or through member ship and the x-zitadel-orgid header. However, for resource based APIs we must be able to resolve permissions by object. For example, an IAM_OWNER listing users should be able to get all users in an instance based on the query filters. Alternatively a user may have user.read permissions on one or more orgs. They should be able to read just those users. # How the Problems Are Solved ## Role permission mapping The role permission mappings defined from `defaults.yaml` or local config override are synchronized to the database on every run of `zitadel setup`: - A single query per **aggregate** builds a list of `add` and `remove` actions needed to reach the desired state or role permission mappings from the config. - The required events based on the actions are pushed to the event store. - Events define search fields so that permission checking can use the indices and is strongly consistent for both query and command sides. The migration is split in the following aggregates: - System aggregate for for roles prefixed with `SYSTEM` - Each instance for roles not prefixed with `SYSTEM`. This is in anticipation of instance level management over the API. ## Membership Current instance / org / project membership events now have field table definitions. Like the role permissions this ensures strong consistency while still being able to use the indices of the fields table. A migration is provided to fill the membership fields. ## Permission check I aimed keeping the mental overhead to the developer to a minimal. The provided implementation only provides a permission check for list queries for org level resources, for example users. In the `query` package there is a simple helper function `wherePermittedOrgs` which makes sure the underlying database function is called as part of the `SELECT` query and the permitted organizations are part of the `WHERE` clause. This makes sure results from non-permitted organizations are omitted. Under the hood: - A Pg/PlSQL function searches for a list of organization IDs the passed user has the passed permission. - When the user has the permission on instance level, it returns early with all organizations. - The functions uses a number of views. The views help mapping the fields entries into relational data and simplify the code use for the function. The views provide some pre-filters which allow proper index usage once the final `WHERE` clauses are set by the function. # Additional Changes # Additional Context Closes #9032 Closes https://github.com/zitadel/zitadel/issues/9014 https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for the new permission framework based on this concept.
2025-01-16 11:09:15 +01:00
"github.com/zitadel/zitadel/internal/eventstore"
"github.com/zitadel/zitadel/internal/repository/instance"
)
// SyncRolePermissions is a repeatable step which synchronizes the InternalAuthZ
// RolePermissionMappings from the configuration to the database.
// This is needed until role permissions are manageable over the API.
type SyncRolePermissions struct {
fix(permissions): chunked synchronization of role permission events (#9403) # Which Problems Are Solved Setup fails to push all role permission events when running Zitadel with CockroachDB. `TransactionRetryError`s were visible in logs which finally times out the setup job with `timeout: context deadline exceeded` # How the Problems Are Solved As suggested in the [Cockroach documentation](timeout: context deadline exceeded), _"break down larger transactions"_. The commands to be pushed for the role permissions are chunked in 50 events per push. This chunking is only done with CockroachDB. # Additional Changes - gci run fixed some unrelated imports - access to `command.Commands` for the setup job, so we can reuse the sync logic. # Additional Context Closes #9293 --------- Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
2025-02-26 18:06:50 +02:00
commands *command.Commands
perf: role permissions in database (#9152) # Which Problems Are Solved Currently ZITADEL defines organization and instance member roles and permissions in defaults.yaml. The permission check is done on API call level. For example: "is this user allowed to make this call on this org". This makes sense on the V1 API where the API is permission-level shaped. For example, a search for users always happens in the context of the organization. (Either the organization the calling user belongs to, or through member ship and the x-zitadel-orgid header. However, for resource based APIs we must be able to resolve permissions by object. For example, an IAM_OWNER listing users should be able to get all users in an instance based on the query filters. Alternatively a user may have user.read permissions on one or more orgs. They should be able to read just those users. # How the Problems Are Solved ## Role permission mapping The role permission mappings defined from `defaults.yaml` or local config override are synchronized to the database on every run of `zitadel setup`: - A single query per **aggregate** builds a list of `add` and `remove` actions needed to reach the desired state or role permission mappings from the config. - The required events based on the actions are pushed to the event store. - Events define search fields so that permission checking can use the indices and is strongly consistent for both query and command sides. The migration is split in the following aggregates: - System aggregate for for roles prefixed with `SYSTEM` - Each instance for roles not prefixed with `SYSTEM`. This is in anticipation of instance level management over the API. ## Membership Current instance / org / project membership events now have field table definitions. Like the role permissions this ensures strong consistency while still being able to use the indices of the fields table. A migration is provided to fill the membership fields. ## Permission check I aimed keeping the mental overhead to the developer to a minimal. The provided implementation only provides a permission check for list queries for org level resources, for example users. In the `query` package there is a simple helper function `wherePermittedOrgs` which makes sure the underlying database function is called as part of the `SELECT` query and the permitted organizations are part of the `WHERE` clause. This makes sure results from non-permitted organizations are omitted. Under the hood: - A Pg/PlSQL function searches for a list of organization IDs the passed user has the passed permission. - When the user has the permission on instance level, it returns early with all organizations. - The functions uses a number of views. The views help mapping the fields entries into relational data and simplify the code use for the function. The views provide some pre-filters which allow proper index usage once the final `WHERE` clauses are set by the function. # Additional Changes # Additional Context Closes #9032 Closes https://github.com/zitadel/zitadel/issues/9014 https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for the new permission framework based on this concept.
2025-01-16 11:09:15 +01:00
eventstore *eventstore.Eventstore
rolePermissionMappings []authz.RoleMapping
}
func (mig *SyncRolePermissions) Execute(ctx context.Context, _ eventstore.Event) error {
if err := mig.executeSystem(ctx); err != nil {
return err
}
return mig.executeInstances(ctx)
}
func (mig *SyncRolePermissions) executeSystem(ctx context.Context) error {
logging.WithFields("migration", mig.String()).Info("prepare system role permission sync events")
fix(permissions): chunked synchronization of role permission events (#9403) # Which Problems Are Solved Setup fails to push all role permission events when running Zitadel with CockroachDB. `TransactionRetryError`s were visible in logs which finally times out the setup job with `timeout: context deadline exceeded` # How the Problems Are Solved As suggested in the [Cockroach documentation](timeout: context deadline exceeded), _"break down larger transactions"_. The commands to be pushed for the role permissions are chunked in 50 events per push. This chunking is only done with CockroachDB. # Additional Changes - gci run fixed some unrelated imports - access to `command.Commands` for the setup job, so we can reuse the sync logic. # Additional Context Closes #9293 --------- Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
2025-02-26 18:06:50 +02:00
details, err := mig.commands.SynchronizeRolePermission(ctx, "SYSTEM", mig.rolePermissionMappings)
perf: role permissions in database (#9152) # Which Problems Are Solved Currently ZITADEL defines organization and instance member roles and permissions in defaults.yaml. The permission check is done on API call level. For example: "is this user allowed to make this call on this org". This makes sense on the V1 API where the API is permission-level shaped. For example, a search for users always happens in the context of the organization. (Either the organization the calling user belongs to, or through member ship and the x-zitadel-orgid header. However, for resource based APIs we must be able to resolve permissions by object. For example, an IAM_OWNER listing users should be able to get all users in an instance based on the query filters. Alternatively a user may have user.read permissions on one or more orgs. They should be able to read just those users. # How the Problems Are Solved ## Role permission mapping The role permission mappings defined from `defaults.yaml` or local config override are synchronized to the database on every run of `zitadel setup`: - A single query per **aggregate** builds a list of `add` and `remove` actions needed to reach the desired state or role permission mappings from the config. - The required events based on the actions are pushed to the event store. - Events define search fields so that permission checking can use the indices and is strongly consistent for both query and command sides. The migration is split in the following aggregates: - System aggregate for for roles prefixed with `SYSTEM` - Each instance for roles not prefixed with `SYSTEM`. This is in anticipation of instance level management over the API. ## Membership Current instance / org / project membership events now have field table definitions. Like the role permissions this ensures strong consistency while still being able to use the indices of the fields table. A migration is provided to fill the membership fields. ## Permission check I aimed keeping the mental overhead to the developer to a minimal. The provided implementation only provides a permission check for list queries for org level resources, for example users. In the `query` package there is a simple helper function `wherePermittedOrgs` which makes sure the underlying database function is called as part of the `SELECT` query and the permitted organizations are part of the `WHERE` clause. This makes sure results from non-permitted organizations are omitted. Under the hood: - A Pg/PlSQL function searches for a list of organization IDs the passed user has the passed permission. - When the user has the permission on instance level, it returns early with all organizations. - The functions uses a number of views. The views help mapping the fields entries into relational data and simplify the code use for the function. The views provide some pre-filters which allow proper index usage once the final `WHERE` clauses are set by the function. # Additional Changes # Additional Context Closes #9032 Closes https://github.com/zitadel/zitadel/issues/9014 https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for the new permission framework based on this concept.
2025-01-16 11:09:15 +01:00
if err != nil {
return err
}
fix(permissions): chunked synchronization of role permission events (#9403) # Which Problems Are Solved Setup fails to push all role permission events when running Zitadel with CockroachDB. `TransactionRetryError`s were visible in logs which finally times out the setup job with `timeout: context deadline exceeded` # How the Problems Are Solved As suggested in the [Cockroach documentation](timeout: context deadline exceeded), _"break down larger transactions"_. The commands to be pushed for the role permissions are chunked in 50 events per push. This chunking is only done with CockroachDB. # Additional Changes - gci run fixed some unrelated imports - access to `command.Commands` for the setup job, so we can reuse the sync logic. # Additional Context Closes #9293 --------- Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
2025-02-26 18:06:50 +02:00
logging.WithFields("migration", mig.String(), "sequence", details.Sequence).Info("pushed system role permission sync events")
perf: role permissions in database (#9152) # Which Problems Are Solved Currently ZITADEL defines organization and instance member roles and permissions in defaults.yaml. The permission check is done on API call level. For example: "is this user allowed to make this call on this org". This makes sense on the V1 API where the API is permission-level shaped. For example, a search for users always happens in the context of the organization. (Either the organization the calling user belongs to, or through member ship and the x-zitadel-orgid header. However, for resource based APIs we must be able to resolve permissions by object. For example, an IAM_OWNER listing users should be able to get all users in an instance based on the query filters. Alternatively a user may have user.read permissions on one or more orgs. They should be able to read just those users. # How the Problems Are Solved ## Role permission mapping The role permission mappings defined from `defaults.yaml` or local config override are synchronized to the database on every run of `zitadel setup`: - A single query per **aggregate** builds a list of `add` and `remove` actions needed to reach the desired state or role permission mappings from the config. - The required events based on the actions are pushed to the event store. - Events define search fields so that permission checking can use the indices and is strongly consistent for both query and command sides. The migration is split in the following aggregates: - System aggregate for for roles prefixed with `SYSTEM` - Each instance for roles not prefixed with `SYSTEM`. This is in anticipation of instance level management over the API. ## Membership Current instance / org / project membership events now have field table definitions. Like the role permissions this ensures strong consistency while still being able to use the indices of the fields table. A migration is provided to fill the membership fields. ## Permission check I aimed keeping the mental overhead to the developer to a minimal. The provided implementation only provides a permission check for list queries for org level resources, for example users. In the `query` package there is a simple helper function `wherePermittedOrgs` which makes sure the underlying database function is called as part of the `SELECT` query and the permitted organizations are part of the `WHERE` clause. This makes sure results from non-permitted organizations are omitted. Under the hood: - A Pg/PlSQL function searches for a list of organization IDs the passed user has the passed permission. - When the user has the permission on instance level, it returns early with all organizations. - The functions uses a number of views. The views help mapping the fields entries into relational data and simplify the code use for the function. The views provide some pre-filters which allow proper index usage once the final `WHERE` clauses are set by the function. # Additional Changes # Additional Context Closes #9032 Closes https://github.com/zitadel/zitadel/issues/9014 https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for the new permission framework based on this concept.
2025-01-16 11:09:15 +01:00
return nil
}
func (mig *SyncRolePermissions) executeInstances(ctx context.Context) error {
instances, err := mig.eventstore.InstanceIDs(
ctx,
eventstore.NewSearchQueryBuilder(eventstore.ColumnsInstanceIDs).
OrderDesc().
AddQuery().
AggregateTypes(instance.AggregateType).
EventTypes(instance.InstanceAddedEventType).
Builder().
ExcludeAggregateIDs().
AggregateTypes(instance.AggregateType).
EventTypes(instance.InstanceRemovedEventType).
Builder(),
)
if err != nil {
return err
}
for i, instanceID := range instances {
logging.WithFields("instance_id", instanceID, "migration", mig.String(), "progress", fmt.Sprintf("%d/%d", i+1, len(instances))).Info("prepare instance role permission sync events")
fix(permissions): chunked synchronization of role permission events (#9403) # Which Problems Are Solved Setup fails to push all role permission events when running Zitadel with CockroachDB. `TransactionRetryError`s were visible in logs which finally times out the setup job with `timeout: context deadline exceeded` # How the Problems Are Solved As suggested in the [Cockroach documentation](timeout: context deadline exceeded), _"break down larger transactions"_. The commands to be pushed for the role permissions are chunked in 50 events per push. This chunking is only done with CockroachDB. # Additional Changes - gci run fixed some unrelated imports - access to `command.Commands` for the setup job, so we can reuse the sync logic. # Additional Context Closes #9293 --------- Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
2025-02-26 18:06:50 +02:00
details, err := mig.commands.SynchronizeRolePermission(ctx, instanceID, mig.rolePermissionMappings)
perf: role permissions in database (#9152) # Which Problems Are Solved Currently ZITADEL defines organization and instance member roles and permissions in defaults.yaml. The permission check is done on API call level. For example: "is this user allowed to make this call on this org". This makes sense on the V1 API where the API is permission-level shaped. For example, a search for users always happens in the context of the organization. (Either the organization the calling user belongs to, or through member ship and the x-zitadel-orgid header. However, for resource based APIs we must be able to resolve permissions by object. For example, an IAM_OWNER listing users should be able to get all users in an instance based on the query filters. Alternatively a user may have user.read permissions on one or more orgs. They should be able to read just those users. # How the Problems Are Solved ## Role permission mapping The role permission mappings defined from `defaults.yaml` or local config override are synchronized to the database on every run of `zitadel setup`: - A single query per **aggregate** builds a list of `add` and `remove` actions needed to reach the desired state or role permission mappings from the config. - The required events based on the actions are pushed to the event store. - Events define search fields so that permission checking can use the indices and is strongly consistent for both query and command sides. The migration is split in the following aggregates: - System aggregate for for roles prefixed with `SYSTEM` - Each instance for roles not prefixed with `SYSTEM`. This is in anticipation of instance level management over the API. ## Membership Current instance / org / project membership events now have field table definitions. Like the role permissions this ensures strong consistency while still being able to use the indices of the fields table. A migration is provided to fill the membership fields. ## Permission check I aimed keeping the mental overhead to the developer to a minimal. The provided implementation only provides a permission check for list queries for org level resources, for example users. In the `query` package there is a simple helper function `wherePermittedOrgs` which makes sure the underlying database function is called as part of the `SELECT` query and the permitted organizations are part of the `WHERE` clause. This makes sure results from non-permitted organizations are omitted. Under the hood: - A Pg/PlSQL function searches for a list of organization IDs the passed user has the passed permission. - When the user has the permission on instance level, it returns early with all organizations. - The functions uses a number of views. The views help mapping the fields entries into relational data and simplify the code use for the function. The views provide some pre-filters which allow proper index usage once the final `WHERE` clauses are set by the function. # Additional Changes # Additional Context Closes #9032 Closes https://github.com/zitadel/zitadel/issues/9014 https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for the new permission framework based on this concept.
2025-01-16 11:09:15 +01:00
if err != nil {
return err
}
fix(permissions): chunked synchronization of role permission events (#9403) # Which Problems Are Solved Setup fails to push all role permission events when running Zitadel with CockroachDB. `TransactionRetryError`s were visible in logs which finally times out the setup job with `timeout: context deadline exceeded` # How the Problems Are Solved As suggested in the [Cockroach documentation](timeout: context deadline exceeded), _"break down larger transactions"_. The commands to be pushed for the role permissions are chunked in 50 events per push. This chunking is only done with CockroachDB. # Additional Changes - gci run fixed some unrelated imports - access to `command.Commands` for the setup job, so we can reuse the sync logic. # Additional Context Closes #9293 --------- Co-authored-by: Silvan <27845747+adlerhurst@users.noreply.github.com>
2025-02-26 18:06:50 +02:00
logging.WithFields("instance_id", instanceID, "migration", mig.String(), "sequence", details.Sequence).Info("pushed instance role permission sync events")
perf: role permissions in database (#9152) # Which Problems Are Solved Currently ZITADEL defines organization and instance member roles and permissions in defaults.yaml. The permission check is done on API call level. For example: "is this user allowed to make this call on this org". This makes sense on the V1 API where the API is permission-level shaped. For example, a search for users always happens in the context of the organization. (Either the organization the calling user belongs to, or through member ship and the x-zitadel-orgid header. However, for resource based APIs we must be able to resolve permissions by object. For example, an IAM_OWNER listing users should be able to get all users in an instance based on the query filters. Alternatively a user may have user.read permissions on one or more orgs. They should be able to read just those users. # How the Problems Are Solved ## Role permission mapping The role permission mappings defined from `defaults.yaml` or local config override are synchronized to the database on every run of `zitadel setup`: - A single query per **aggregate** builds a list of `add` and `remove` actions needed to reach the desired state or role permission mappings from the config. - The required events based on the actions are pushed to the event store. - Events define search fields so that permission checking can use the indices and is strongly consistent for both query and command sides. The migration is split in the following aggregates: - System aggregate for for roles prefixed with `SYSTEM` - Each instance for roles not prefixed with `SYSTEM`. This is in anticipation of instance level management over the API. ## Membership Current instance / org / project membership events now have field table definitions. Like the role permissions this ensures strong consistency while still being able to use the indices of the fields table. A migration is provided to fill the membership fields. ## Permission check I aimed keeping the mental overhead to the developer to a minimal. The provided implementation only provides a permission check for list queries for org level resources, for example users. In the `query` package there is a simple helper function `wherePermittedOrgs` which makes sure the underlying database function is called as part of the `SELECT` query and the permitted organizations are part of the `WHERE` clause. This makes sure results from non-permitted organizations are omitted. Under the hood: - A Pg/PlSQL function searches for a list of organization IDs the passed user has the passed permission. - When the user has the permission on instance level, it returns early with all organizations. - The functions uses a number of views. The views help mapping the fields entries into relational data and simplify the code use for the function. The views provide some pre-filters which allow proper index usage once the final `WHERE` clauses are set by the function. # Additional Changes # Additional Context Closes #9032 Closes https://github.com/zitadel/zitadel/issues/9014 https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for the new permission framework based on this concept.
2025-01-16 11:09:15 +01:00
}
return nil
}
func (*SyncRolePermissions) String() string {
return "repeatable_sync_role_permissions"
}
func (*SyncRolePermissions) Check(lastRun map[string]interface{}) bool {
return true
}
Reference in New Issue Copy Permalink