TheArchive/zitadel
1
0
Fork 0
mirror of https://github.com/zitadel/zitadel.git synced 2025-08-11 18:07:31 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: protos refactoring

Browse Source 
* start with user

* user first try done in all services

* user, org, idp for discussion

* remove unused stuff

* bla

* dockerbuild

* rename search, get multiple to list...

* add annotation

* update proto dependencies

* update proto dependencies

* change proto imports

* replace all old imports

* fix go out

* remove unused lines

* correct protoc flags

* grpc and openapi flags

* go out source path relative

* -p

* remove dead code

* sourcepath relative

* ls

* is onenapi the problem?

* hobla

* authoption output

* wrong field name

* gopf

* correct option, add correct flags

* small improvments

* SIMPLYFY

* relative path

* gopf bin ich en tubel

* correct path

* default policies in admin

* grpc generation in one file

* remove non ascii

* metadata on manipulations

* correct auth_option import

* fixes

* larry

* idp provider to idp

* fix generate

* admin and auth nearly done

* admin and auth nearly done

* gen

* healthz

* imports

* deleted too much imports

* fix org

* add import

* imports

* import

* naming

* auth_opt

* gopf

* management

* imports

* _TYPE_UNSPECIFIED

* improts

* auth opts

* management policies

* imports

* passwordlessType to MFAType

* auth_opt

* add user grant calls

* add missing messages

* result

* fix option

* improvements

* ids

* fix http

* imports

* fixes

* fields

* body

* add fields

* remove wrong member query

* fix request response

* fixes

* add copy files

* variable versions

* generate all files

* improvements

* add dependencies

* factors

* user session

* oidc information, iam

* remove unused file

* changes

* enums

* dockerfile

* fix build

* remove unused folder

* update readme for build

* move old server impl

* add event type to change

* some changes

* start admin

* remove wrong field

* admin only list calls missing

* fix proto numbers

* surprisingly it compiles

* service ts changes

* admin mgmt

* mgmt

* auth manipulation and gets done, lists missing

* validations and some field changes

* validations

* enum validations

* remove todo

* move proto files to proto/zitadel

* change proto path in dockerfile

* it compiles!

* add validate import

* remove duplicate import

* fix protos

* fix import

* tests

* cleanup

* remove unimplemented methods

* iam member multiple queries

* all auth and admin calls

* add initial password on crate human

* message names

* management user server

* machine done

* fix: todos (#1346)

* fix: pub sub in new eventstore

* fix: todos

* fix: todos

* fix: todos

* fix: todos

* fix: todos

* fix tests

* fix: search method domain

* admin service, user import type typescript

* admin changes

* admin changes

* fix: search method domain

* more user grpc and begin org, fix configs

* fix: return object details

* org grpc

* remove creation date add details

* app

* fix: return object details

* fix: return object details

* mgmt service, project members

* app

* fix: convert policies

* project, members, granted projects, searches

* fix: convert usergrants

* fix: convert usergrants

* auth user detail, user detail, mfa, second factor, auth

* fix: convert usergrants

* mfa, memberships, password, owned proj detail

* fix: convert usergrants

* project grant

* missing details

* changes, userview

* idp table, keys

* org list and user table filter

* unify rest paths (#1381)

* unify rest paths

* post for all searches,
mfa to multi_factor,
secondfactor to second_factor

* remove v1

* fix tests

* rename api client key to app key

* machine keys, age policy

* user list, machine keys, changes

* fix: org states

* add default flag to policy

* second factor to type

* idp id

* app type

* unify ListQuery, ListDetails, ObjectDetails field names

* user grants, apps, memberships

* fix type params

* metadata to detail, linke idps

* api create, membership, app detail, create

* idp, app, policy

* queries, multi -> auth factors and missing fields

* update converters

* provider to user, remove old mgmt refs

* temp remove authfactor dialog, build finish

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>
Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Fabiennne <fabienne.gerschwiler@gmail.com>
This commit is contained in:
Silvan
2021-03-09 10:30:11 +01:00
committed by GitHub
parent 9f417f3957
commit dabd5920dc
372 changed files with 17881 additions and 22036 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
build/README.md
View File

@@ -12,15 +12,15 @@
This command generates the grpc stub for angular into the folder console/src/app/proto/generated for local development
```Bash
DOCKER_BUILDKIT=1 docker build -f build/dockerfile . -t zitadel:local --target npm-copy -o console/src/app/proto/generated
DOCKER_BUILDKIT=1 docker build -f build/dockerfile . -t zitadel:local --target npm-copy -o .
```
### Go
With this command you can generate the stub for golang into the correct dir pkg/
With this command you can generate the stub for golang into the zitadel dir
```Bash
DOCKER_BUILDKIT=1 docker build -f build/dockerfile . -t zitadel:local --target go-copy -o pkg
DOCKER_BUILDKIT=1 docker build -f build/dockerfile . -t zitadel:local --target go-copy -o .
```
## Run

31
build/console/generate-grpc.sh
View File

@@ -10,33 +10,8 @@ mkdir -p $GEN_PATH
echo "Generate grpc"
protoc \
-I=.tmp/protos/message \
-I=.tmp/protos/admin/proto \
-I=.tmp/protos/management/proto \
-I=.tmp/protos/auth/proto \
-I=/proto/include \
-I=node_modules/google-proto-files \
-I=.tmp/protos \
--js_out=import_style=commonjs,binary:$GEN_PATH \
--grpc-web_out=import_style=commonjs+dts,mode=grpcweb:$GEN_PATH \
.tmp/protos/message/proto/*.proto \
.tmp/protos/admin/proto/*.proto \
.tmp/protos/auth/proto/*.proto \
.tmp/protos/management/proto/*.proto
echo "Generate annotations js file (compatibility)"
mkdir -p $GEN_PATH/google/api/
touch $GEN_PATH/google/api/annotations_pb.js
echo "export {}" > $GEN_PATH/google/api/annotations_pb.d.ts
mkdir -p $GEN_PATH/validate
touch $GEN_PATH/validate/validate_pb.js
echo "export {}" > $GEN_PATH/validate/validate_pb.d.ts
mkdir -p $GEN_PATH/protoc-gen-swagger/options
touch $GEN_PATH/protoc-gen-swagger/options/annotations_pb.js
echo "export {}" > $GEN_PATH/protoc-gen-swagger/options/annotations_pb.d.ts
mkdir -p $GEN_PATH/authoption
touch $GEN_PATH/authoption/options_pb.js
echo "export {}" > $GEN_PATH/authoption/options_pb.d.ts
--grpc-web_out=import_style=typescript,mode=grpcweb:$GEN_PATH \
$(find /proto/include -iname "*.proto")

157
build/dockerfile
View File

@@ -1,105 +1,137 @@
#######################
## By default we build the prod enviroment
ARG GO_VERSION=1.15.8
ARG NODE_VERSION=15.8.0
ARG ENV=prod
#######################
## This step downloads the protofiles, protoc and protoc-gen-grpc-web for later use
## This step sets up the folder structure,
## initalices go mods,
## downloads the protofiles,
## protoc and protoc-gen-grpc-web for later use
#######################
FROM alpine as base
RUN apk add tar curl
WORKDIR /.tmp
RUN wget -O protoc https://github.com/protocolbuffers/protobuf/releases/download/v3.13.0/protoc-3.13.0-linux-x86_64.zip \
&& unzip protoc \
&& wget -O bin/protoc-gen-grpc-web https://github.com/grpc/grpc-web/releases/download/1.2.0/protoc-gen-grpc-web-1.2.0-linux-x86_64 \
&& chmod +x bin/protoc-gen-grpc-web
RUN curl https://raw.githubusercontent.com/envoyproxy/protoc-gen-validate/v0.4.1/validate/validate.proto --create-dirs -o validate/validate.proto \
&& curl https://raw.githubusercontent.com/grpc-ecosystem/grpc-gateway/v1.14.6/protoc-gen-swagger/options/annotations.proto --create-dirs -o protoc-gen-swagger/options/annotations.proto \
&& curl https://raw.githubusercontent.com/grpc-ecosystem/grpc-gateway/v1.14.6/protoc-gen-swagger/options/openapiv2.proto --create-dirs -o protoc-gen-swagger/options/openapiv2.proto \
&& curl https://raw.githubusercontent.com/googleapis/googleapis/master/google/api/annotations.proto --create-dirs -o google/api/annotations.proto \
&& curl https://raw.githubusercontent.com/googleapis/googleapis/master/google/api/http.proto --create-dirs -o google/api/http.proto \
&& curl https://raw.githubusercontent.com/protocolbuffers/protobuf/master/src/google/protobuf/empty.proto --create-dirs -o google/protobuf/empty.proto \
&& curl https://raw.githubusercontent.com/protocolbuffers/protobuf/master/src/google/protobuf/timestamp.proto --create-dirs -o google/protobuf/timestamp.proto \
&& curl https://raw.githubusercontent.com/protocolbuffers/protobuf/master/src/google/protobuf/descriptor.proto --create-dirs -o google/protobuf/descriptor.proto \
&& curl https://raw.githubusercontent.com/protocolbuffers/protobuf/master/src/google/protobuf/duration.proto --create-dirs -o google/protobuf/duration.proto \
&& curl https://raw.githubusercontent.com/protocolbuffers/protobuf/master/src/google/protobuf/any.proto --create-dirs -o google/protobuf/any.proto \
&& curl https://raw.githubusercontent.com/protocolbuffers/protobuf/master/src/google/protobuf/struct.proto --create-dirs -o google/protobuf/struct.proto
FROM alpine AS base
ARG PROTOC_VERSION=3.14.0
ARG PROTOC_ZIP=protoc-${PROTOC_VERSION}-linux-x86_64.zip
ARG GRPC_WEB_VERSION=1.2.1
ARG GRPC_WEB=protoc-gen-grpc-web-${GRPC_WEB_VERSION}-linux-x86_64
RUN apk add tar curl
WORKDIR /proto
#protoc
RUN curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/$PROTOC_ZIP \
&& unzip -o $PROTOC_ZIP -d /usr/local bin/protoc \
&& unzip -o $PROTOC_ZIP -d /proto include/* \
&& rm -f $PROTOC_ZIP
#grpc web
RUN curl -OL https://github.com/grpc/grpc-web/releases/download/${GRPC_WEB_VERSION}/${GRPC_WEB} \
&& mv ${GRPC_WEB} /usr/local/bin/protoc-gen-grpc-web \
&& chmod +x /usr/local/bin/protoc-gen-grpc-web
#proto dependencies
RUN curl https://raw.githubusercontent.com/envoyproxy/protoc-gen-validate/v0.4.1/validate/validate.proto --create-dirs -o include/validate/validate.proto \
&& curl https://raw.githubusercontent.com/grpc-ecosystem/grpc-gateway/v2.2.0/protoc-gen-openapiv2/options/annotations.proto --create-dirs -o include/protoc-gen-openapiv2/options/annotations.proto \
&& curl https://raw.githubusercontent.com/grpc-ecosystem/grpc-gateway/v2.2.0/protoc-gen-openapiv2/options/openapiv2.proto --create-dirs -o include/protoc-gen-openapiv2/options/openapiv2.proto \
&& curl https://raw.githubusercontent.com/googleapis/googleapis/master/google/api/annotations.proto --create-dirs -o include/google/api/annotations.proto \
&& curl https://raw.githubusercontent.com/googleapis/googleapis/master/google/api/http.proto --create-dirs -o include/google/api/http.proto \
&& curl https://raw.githubusercontent.com/googleapis/googleapis/master/google/api/field_behavior.proto --create-dirs -o include/google/api/field_behavior.proto
#zitadel protos
COPY /proto/ include/.
COPY pkg/grpc/admin/proto/admin.proto admin/proto/admin.proto
COPY pkg/grpc/auth/proto/auth.proto auth/proto/auth.proto
COPY pkg/grpc/management/proto/management.proto management/proto/management.proto
COPY pkg/grpc/message/proto/message.proto message/proto/message.proto
COPY internal/protoc/protoc-gen-authoption/authoption/options.proto authoption/options.proto
#######################
## With this step we prepare all node_modules, this helps caching the build
## Speed up this step by mounting your local node_modules directory
#######################
FROM node:15 as npm-base
WORKDIR console
FROM node:${NODE_VERSION} as npm-base
WORKDIR /console
COPY console/package.json console/package-lock.json ./
RUN npm install \
&& mkdir .tmp
RUN npm install
COPY console .
COPY --from=base /.tmp/bin /usr/local/bin/
COPY --from=base /.tmp .tmp/protos/
COPY --from=base /proto /proto
COPY --from=base /usr/local/bin /usr/local/bin/.
COPY build/console build/console/
RUN build/console/generate-grpc.sh
FROM scratch as npm-copy
COPY --from=npm-base /console/src/app/proto/generated .
#######################
## copy for local dev
#######################
FROM scratch as npm-copy
COPY --from=npm-base /console/src/app/proto/generated ./console/src/app/proto/generated
#######################
## anular dev build
#######################
FROM npm-base as dev-angular-build
RUN npm install -g @angular/cli
#######################
## anular prod build
#######################
FROM npm-base as prod-angular-build
RUN npm run prodbuild
#######################
## Go base build
## Go dependencies
## Speed up this step by mounting your local go mod pkg directory
#######################
FROM golang:1.15 as go-base
WORKDIR src/github.com/caos/zitadel/
COPY go.mod go.sum ./
FROM golang:${GO_VERSION} as go-dep
RUN mkdir -p src/github.com/caos/zitadel
COPY . src/github.com/caos/zitadel/
WORKDIR /go/src/github.com/caos/zitadel/
RUN go mod download
COPY --from=base /.tmp .tmp/protos/
COPY --from=base /.tmp/bin /usr/local/bin/
COPY internal/protoc/protoc-base internal/protoc/protoc-base/
COPY internal/protoc/protoc-gen-authoption internal/protoc/protoc-gen-authoption/
RUN ./tools/install.sh
RUN go install \
github.com/grpc-ecosystem/grpc-gateway/protoc-gen-grpc-gateway \
github.com/grpc-ecosystem/grpc-gateway/protoc-gen-swagger \
github.com/golang/protobuf/protoc-gen-go \
github.com/envoyproxy/protoc-gen-validate
RUN go get -u github.com/go-bindata/go-bindata/...
RUN go-bindata ./internal/protoc/protoc-gen-authoption/templates \
&& go install ./internal/protoc/protoc-gen-authoption
COPY build/zitadel build/zitadel/
#######################
## Go base build
#######################
FROM go-dep as go-base
COPY --from=base /proto /proto
COPY --from=base /usr/local/bin /usr/local/bin/.
RUN build/zitadel/generate-grpc.sh
FROM scratch as go-copy
COPY --from=go-base /go/src/github.com/caos/zitadel/pkg/ .
#######################
## copy for local dev
#######################
FROM scratch as go-copy
COPY --from=go-base /go/src/github.com/caos/zitadel/pkg/grpc ./pkg/grpc
COPY --from=go-base /go/src/github.com/caos/zitadel/internal/protoc/protoc-gen-authoption/templates.gen.go ./internal/protoc/protoc-gen-authoption/templates.gen.go
COPY --from=go-base /go/src/github.com/caos/zitadel/internal/protoc/protoc-gen-authoption/authoption/options.pb.go ./internal/protoc/protoc-gen-authoption/authoption/options.pb.go
#######################
## Go test
#######################
FROM go-base as go-test
COPY . .
#Migrations for cockroach-secure
# Migrations for cockroach-secure
RUN go install github.com/rakyll/statik
RUN ./build/operator/prebuild.sh ./migrations
RUN go test -race -v -coverprofile=profile.cov $(go list ./... | grep -v /operator/)
## Go test
#######################
## Go test results
#######################
FROM scratch as go-codecov
COPY --from=go-test /go/src/github.com/caos/zitadel/profile.cov profile.cov
#######################
## Go prod build
#######################
FROM go-test as prod-go-build
COPY --from=prod-angular-build console/dist/console console/dist/console/
RUN go get github.com/rakyll/statik \
@@ -109,10 +141,14 @@ RUN go get github.com/rakyll/statik \
&& ./build/zitadel/generate-static.sh
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -ldflags '-extldflags "-static"' -o zitadel-linux-amd64 cmd/zitadel/main.go
#######################
## Go dev build
#######################
FROM go-base as dev-go-build
RUN go get github.com/go-delve/delve/cmd/dlv
#######################
## Final Production Image
#######################
@@ -123,11 +159,14 @@ COPY --from=prod-go-build /go/src/github.com/caos/zitadel/zitadel-linux-amd64 /a
RUN chmod a+x /app/zitadel
RUN ls -la /
#######################
## Scratch Image
#######################
FROM scratch as final
COPY --from=artifact /etc/passwd /etc/passwd
COPY --from=artifact /etc/ssl/certs /etc/ssl/certs
COPY --from=artifact /app /
USER zitadel
HEALTHCHECK NONE
ENTRYPOINT ["/zitadel"]
ENTRYPOINT ["/zitadel"]

106
build/zitadel/generate-grpc.sh
View File

@@ -4,62 +4,70 @@ set -eux
echo "Generate grpc"
protoc \
-I=.tmp/protos/message \
-I=.tmp/protos/admin/proto \
-I=.tmp/protos/management/proto \
-I=.tmp/protos/auth/proto \
-I=.tmp/protos \
-I=${GOPATH}/src \
--go_out=plugins=grpc:$GOPATH/src \
.tmp/protos/message/proto/message.proto
OPENAPI_PATH=${GOPATH}/src/github.com/caos/zitadel/openapi/v2
ZITADEL_PATH=${GOPATH}/src/github.com/caos/zitadel
GRPC_PATH=${ZITADEL_PATH}/pkg/grpc
PROTO_PATH=/proto/include/zitadel
protoc \
-I=.tmp/protos/message \
-I=.tmp/protos/admin/proto \
-I=.tmp/protos/management/proto \
-I=.tmp/protos/auth/proto \
-I=.tmp/protos \
-I=${GOPATH}/src \
--go_out=plugins=grpc:$GOPATH/src \
--grpc-gateway_out=logtostderr=true:$GOPATH/src \
--swagger_out=logtostderr=true:. \
--authoption_out=. \
-I=/proto/include/ \
--go_out $GOPATH/src \
--go-grpc_out $GOPATH/src \
$(find ${PROTO_PATH} -iname *.proto | grep -v "management|admin|auth")
go-bindata \
-pkg main \
-prefix internal/protoc/protoc-gen-authoption \
-o ${ZITADEL_PATH}/internal/protoc/protoc-gen-authoption/templates.gen.go \
${ZITADEL_PATH}/internal/protoc/protoc-gen-authoption/templates
go install ${ZITADEL_PATH}/internal/protoc/protoc-gen-authoption
# output folder for openapi v2
mkdir -p ${OPENAPI_PATH}
protoc \
-I=/proto/include \
--go_out ${GOPATH}/src \
--go-grpc_out ${GOPATH}/src \
--grpc-gateway_out ${GOPATH}/src \
--grpc-gateway_opt logtostderr=true \
--openapiv2_out ${OPENAPI_PATH} \
--openapiv2_opt logtostderr=true \
--authoption_out ${GRPC_PATH}/admin \
--validate_out=lang=go:${GOPATH}/src \
.tmp/protos/admin/proto/admin.proto
mv admin* $GOPATH/src/github.com/caos/zitadel/pkg/grpc/admin/
${PROTO_PATH}/admin.proto
mv ${ZITADEL_PATH}/pkg/grpc/admin/zitadel/* ${ZITADEL_PATH}/pkg/grpc/admin
rm -r ${ZITADEL_PATH}/pkg/grpc/admin/zitadel
protoc \
-I=.tmp/protos/message \
-I=.tmp/protos/admin/proto \
-I=.tmp/protos/management/proto \
-I=.tmp/protos/auth/proto \
-I=.tmp/protos \
-I=${GOPATH}/src \
--go_out=plugins=grpc:$GOPATH/src \
--grpc-gateway_out=logtostderr=true,allow_delete_body=true:${GOPATH}/src \
--swagger_out=logtostderr=true,allow_delete_body=true:. \
--authoption_out=. \
-I=/proto/include \
--go_out $GOPATH/src \
--go-grpc_out $GOPATH/src \
--grpc-gateway_out ${GOPATH}/src \
--grpc-gateway_opt logtostderr=true \
--grpc-gateway_opt allow_delete_body=true \
--openapiv2_out ${OPENAPI_PATH} \
--openapiv2_opt logtostderr=true \
--openapiv2_opt allow_delete_body=true \
--authoption_out ${GRPC_PATH}/management \
--validate_out=lang=go:${GOPATH}/src \
.tmp/protos/management/proto/management.proto
mv management* $GOPATH/src/github.com/caos/zitadel/pkg/grpc/management/
${PROTO_PATH}/management.proto
mv ${ZITADEL_PATH}/pkg/grpc/management/zitadel/* ${ZITADEL_PATH}/pkg/grpc/management
rm -r ${ZITADEL_PATH}/pkg/grpc/management/zitadel
protoc \
-I=.tmp/protos/message \
-I=.tmp/protos/admin/proto \
-I=.tmp/protos/management/proto \
-I=.tmp/protos/auth/proto \
-I=.tmp/protos \
-I=${GOPATH}/src \
--go_out=plugins=grpc:$GOPATH/src \
--grpc-gateway_out=logtostderr=true:$GOPATH/src \
--swagger_out=logtostderr=true:. \
--authoption_out=. \
-I=/proto/include \
--go_out $GOPATH/src \
--go-grpc_out $GOPATH/src \
--grpc-gateway_out ${GOPATH}/src \
--grpc-gateway_opt logtostderr=true \
--openapiv2_out ${OPENAPI_PATH} \
--openapiv2_opt logtostderr=true \
--authoption_out=${GRPC_PATH}/auth \
--validate_out=lang=go:${GOPATH}/src \
.tmp/protos/auth/proto/auth.proto
${PROTO_PATH}/auth.proto
mv ${ZITADEL_PATH}/pkg/grpc/auth/zitadel/* ${ZITADEL_PATH}/pkg/grpc/auth
rm -r ${ZITADEL_PATH}/pkg/grpc/auth/zitadel
mv auth* $GOPATH/src/github.com/caos/zitadel/pkg/grpc/auth/
echo "done generating"
echo "done generating grpc"