perf: role permissions in database (#9152)

# Which Problems Are Solved

Currently ZITADEL defines organization and instance member roles and
permissions in defaults.yaml. The permission check is done on API call
level. For example: "is this user allowed to make this call on this
org". This makes sense on the V1 API where the API is permission-level
shaped. For example, a search for users always happens in the context of
the organization. (Either the organization the calling user belongs to,
or through member ship and the x-zitadel-orgid header.

However, for resource based APIs we must be able to resolve permissions
by object. For example, an IAM_OWNER listing users should be able to get
all users in an instance based on the query filters. Alternatively a
user may have user.read permissions on one or more orgs. They should be
able to read just those users.

# How the Problems Are Solved

## Role permission mapping

The role permission mappings defined from `defaults.yaml` or local
config override are synchronized to the database on every run of
`zitadel setup`:

- A single query per **aggregate** builds a list of `add` and `remove`
actions needed to reach the desired state or role permission mappings
from the config.
- The required events based on the actions are pushed to the event
store.
- Events define search fields so that permission checking can use the
indices and is strongly consistent for both query and command sides.

The migration is split in the following aggregates:

- System aggregate for for roles prefixed with `SYSTEM`
- Each instance for roles not prefixed with `SYSTEM`. This is in
anticipation of instance level management over the API.

## Membership

Current instance / org / project membership events now have field table
definitions. Like the role permissions this ensures strong consistency
while still being able to use the indices of the fields table. A
migration is provided to fill the membership fields.

## Permission check

I aimed keeping the mental overhead to the developer to a minimal. The
provided implementation only provides a permission check for list
queries for org level resources, for example users. In the `query`
package there is a simple helper function `wherePermittedOrgs` which
makes sure the underlying database function is called as part of the
`SELECT` query and the permitted organizations are part of the `WHERE`
clause. This makes sure results from non-permitted organizations are
omitted. Under the hood:

- A Pg/PlSQL function searches for a list of organization IDs the passed
user has the passed permission.
- When the user has the permission on instance level, it returns early
with all organizations.
- The functions uses a number of views. The views help mapping the
fields entries into relational data and simplify the code use for the
function. The views provide some pre-filters which allow proper index
usage once the final `WHERE` clauses are set by the function.

# Additional Changes



# Additional Context

Closes #9032
Closes https://github.com/zitadel/zitadel/issues/9014

https://github.com/zitadel/zitadel/issues/9188 defines follow-ups for
the new permission framework based on this concept.

internal/api/grpc/feature/v2/converter.go

@@ -29,6 +29,7 @@ func systemFeaturesToCommand(req *feature_pb.SetSystemFeaturesRequest) (*command
 		DisableUserTokenEvent:           req.DisableUserTokenEvent,
 		EnableBackChannelLogout:         req.EnableBackChannelLogout,
 		LoginV2:                         loginV2,
+		PermissionCheckV2:               req.PermissionCheckV2,
 	}, nil
 }
 
@@ -46,6 +47,7 @@ func systemFeaturesToPb(f *query.SystemFeatures) *feature_pb.GetSystemFeaturesRe
 		DisableUserTokenEvent:               featureSourceToFlagPb(&f.DisableUserTokenEvent),
 		EnableBackChannelLogout:             featureSourceToFlagPb(&f.EnableBackChannelLogout),
 		LoginV2:                             loginV2ToLoginV2FlagPb(f.LoginV2),
+		PermissionCheckV2:                   featureSourceToFlagPb(&f.PermissionCheckV2),
 	}
 }
 
@@ -68,6 +70,7 @@ func instanceFeaturesToCommand(req *feature_pb.SetInstanceFeaturesRequest) (*com
 		DisableUserTokenEvent:           req.DisableUserTokenEvent,
 		EnableBackChannelLogout:         req.EnableBackChannelLogout,
 		LoginV2:                         loginV2,
+		PermissionCheckV2:               req.PermissionCheckV2,
 	}, nil
 }
 
@@ -87,6 +90,7 @@ func instanceFeaturesToPb(f *query.InstanceFeatures) *feature_pb.GetInstanceFeat
 		DisableUserTokenEvent:               featureSourceToFlagPb(&f.DisableUserTokenEvent),
 		EnableBackChannelLogout:             featureSourceToFlagPb(&f.EnableBackChannelLogout),
 		LoginV2:                             loginV2ToLoginV2FlagPb(f.LoginV2),
+		PermissionCheckV2:                   featureSourceToFlagPb(&f.PermissionCheckV2),
 	}
 }
 

internal/api/grpc/feature/v2/converter_test.go

@@ -101,6 +101,10 @@ func Test_systemFeaturesToPb(t *testing.T) {
 				BaseURI:  &url.URL{Scheme: "https", Host: "login.com"},
 			},
 		},
+		PermissionCheckV2: query.FeatureSource[bool]{
+			Level: feature.LevelSystem,
+			Value: true,
+		},
 	}
 	want := &feature_pb.GetSystemFeaturesResponse{
 		Details: &object.Details{
@@ -153,6 +157,10 @@ func Test_systemFeaturesToPb(t *testing.T) {
 			BaseUri:  gu.Ptr("https://login.com"),
 			Source:   feature_pb.Source_SOURCE_SYSTEM,
 		},
+		PermissionCheckV2: &feature_pb.FeatureFlag{
+			Enabled: true,
+			Source:  feature_pb.Source_SOURCE_SYSTEM,
+		},
 	}
 	got := systemFeaturesToPb(arg)
 	assert.Equal(t, want, got)
@@ -252,6 +260,10 @@ func Test_instanceFeaturesToPb(t *testing.T) {
 				BaseURI:  &url.URL{Scheme: "https", Host: "login.com"},
 			},
 		},
+		PermissionCheckV2: query.FeatureSource[bool]{
+			Level: feature.LevelInstance,
+			Value: true,
+		},
 	}
 	want := &feature_pb.GetInstanceFeaturesResponse{
 		Details: &object.Details{
@@ -312,6 +324,10 @@ func Test_instanceFeaturesToPb(t *testing.T) {
 			BaseUri:  gu.Ptr("https://login.com"),
 			Source:   feature_pb.Source_SOURCE_INSTANCE,
 		},
+		PermissionCheckV2: &feature_pb.FeatureFlag{
+			Enabled: true,
+			Source:  feature_pb.Source_SOURCE_INSTANCE,
+		},
 	}
 	got := instanceFeaturesToPb(arg)
 	assert.Equal(t, want, got)