TheArchive/zitadel
1
0
Fork 0
mirror of https://github.com/zitadel/zitadel.git synced 2024-12-04 15:35:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
14db628856
zitadel/Makefile

169 lines
6.4 KiB
Makefile
Raw Normal View History

ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
go_bin := "$$(go env GOPATH)/bin"
gen_authopt_path := "$(go_bin)/protoc-gen-authoption"
gen_zitadel_path := "$(go_bin)/protoc-gen-zitadel"
fix(mac): date command options not found #8757 (#8758) # Which Problems Are Solved Closes: #8757 Problem: ```sh ➜ ~ date --rfc-3339=seconds | sed 's/ /T/' date: illegal option -- - usage: date [-jnRu] [-I[date|hours|minutes|seconds]] [-f input_fmt] [-r filename|seconds] [-v[+|-]val[y|m|w|d|H|M|S]] [[[[mm]dd]HH]MM[[cc]yy][.SS] | new_date] [+output_fmt] ``` # How the Problems Are Solved ```sh ➜ date "+%Y-%m-%dT%T%z" | sed -E 's/.([0-9]{2})([0-9]{2})$/-\1:\2/' 2024-10-10T19:09:53-05:30 ➜ TZ=America/Los_Angeles date "+%Y-%m-%dT%T%z" | sed -E 's/.([0-9]{2})([0-9]{2})$/-\1:\2/' 2024-10-10T06:39:41-07:00 ``` - `Mac` support while compiling the source code locally Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-10-28 12:36:57 +00:00
now := $(shell date '+%Y-%m-%dT%T%z' | sed -E 's/.([0-9]{2})([0-9]{2})$$/-\1:\2/')
chore(make): use a timestamp suffix to the default version (#6559) When doing local builds, the `VERSION` was always set to `development`. When running this local build, database migration would be skipped, as the version did not change. This change suffixes `developement` with the dynamic timestamp from `now`, so that migrations are always executed when developing.
2023-09-14 13:30:59 +00:00
VERSION ?= development-$(now)
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
COMMIT_SHA ?= $(shell git rev-parse HEAD)
chore(make): add docker_image make target (#7440)
2024-02-27 16:29:09 +00:00
ZITADEL_IMAGE ?= zitadel:local
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
chore(tests): use a coverage server binary (#8407) # Which Problems Are Solved Use a single server instance for API integration tests. This optimizes the time taken for the integration test pipeline, because it allows running tests on multiple packages in parallel. Also, it saves time by not start and stopping a zitadel server for every package. # How the Problems Are Solved - Build a binary with `go build -race -cover ....` - Integration tests only construct clients. The server remains running in the background. - The integration package and tested packages now fully utilize the API. No more direct database access trough `query` and `command` packages. - Use Makefile recipes to setup, start and stop the server in the background. - The binary has the race detector enabled - Init and setup jobs are configured to halt immediately on race condition - Because the server runs in the background, races are only logged. When the server is stopped and race logs exist, the Makefile recipe will throw an error and print the logs. - Makefile recipes include logic to print logs and convert coverage reports after the server is stopped. - Some tests need a downstream HTTP server to make requests, like quota and milestones. A new `integration/sink` package creates an HTTP server and uses websockets to forward HTTP request back to the test packages. The package API uses Go channels for abstraction and easy usage. # Additional Changes - Integration test files already used the `//go:build integration` directive. In order to properly split integration from unit tests, integration test files need to be in a `integration_test` subdirectory of their package. - `UseIsolatedInstance` used to overwrite the `Tester.Client` for each instance. Now a `Instance` object is returned with a gRPC client that is connected to the isolated instance's hostname. - The `Tester` type is now `Instance`. The object is created for the first instance, used by default in any test. Isolated instances are also `Instance` objects and therefore benefit from the same methods and values. The first instance and any other us capable of creating an isolated instance over the system API. - All test packages run in an Isolated instance by calling `NewInstance()` - Individual tests that use an isolated instance use `t.Parallel()` # Additional Context - Closes #6684 - https://go.dev/doc/articles/race_detector - https://go.dev/doc/build-cover --------- Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-09-06 12:47:57 +00:00
GOCOVERDIR = tmp/coverage
INTEGRATION_DB_FLAVOR ?= postgres
ZITADEL_MASTERKEY ?= MasterkeyNeedsToHave32Characters
export GOCOVERDIR INTEGRATION_DB_FLAVOR ZITADEL_MASTERKEY
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
.PHONY: compile
compile: core_build console_build compile_pipeline
chore(make): add docker_image make target (#7440)
2024-02-27 16:29:09 +00:00
.PHONY: docker_image
docker_image: compile
DOCKER_BUILDKIT=1 docker build -f build/Dockerfile -t $(ZITADEL_IMAGE) .
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
.PHONY: compile_pipeline
compile_pipeline: console_move
CGO_ENABLED=0 go build -o zitadel -v -ldflags="-s -w -X 'github.com/zitadel/zitadel/cmd/build.commit=$(COMMIT_SHA)' -X 'github.com/zitadel/zitadel/cmd/build.date=$(now)' -X 'github.com/zitadel/zitadel/cmd/build.version=$(VERSION)' "
chmod +x zitadel
.PHONY: core_dependencies
core_dependencies:
go mod download
.PHONY: core_static
core_static:
go install github.com/rakyll/statik@v0.1.7
go generate internal/api/ui/login/static/resources/generate.go
fix: build css for login correctly (#6265)
2023-07-25 05:52:59 +00:00
go generate internal/api/ui/login/statik/generate.go
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
go generate internal/notification/statik/generate.go
go generate internal/statik/generate.go
chore(Makefile): add go generate target (#6944) This change adds a core_generate_all make target. It installs the required tools and runs generate on the complete project. `golang/mock` is no longer maintained and a fork is available from the Uber folks. So the latter is used as tool. All the mock files have been regenerated and are part of the PR. The obsolete `tools` directory has been removed, as all the tools are now part of specific make targets. Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-11-22 10:56:43 +00:00
.PHONY: core_generate_all
core_generate_all:
chore: update dependencies (#8242) # Which Problems Are Solved Updates the dependencies. # Additional Context - relates to https://github.com/zitadel/zitadel/security/dependabot/210
2024-07-04 06:28:56 +00:00
go install github.com/dmarkham/enumer@v1.5.10 # https://pkg.go.dev/github.com/dmarkham/enumer?tab=versions
chore(deps): upgrade all go deps (#7227) * chore(deps): upgrade all go deps Also `go mod tidy`. Added comments with URLs for package version lists to makefile commands. * Update Makefile Co-authored-by: Livio Spring <livio.a@gmail.com> --------- Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-01-16 09:27:09 +00:00
go install github.com/rakyll/statik@v0.1.7 # https://pkg.go.dev/github.com/rakyll/statik?tab=versions
go install go.uber.org/mock/mockgen@v0.4.0 # https://pkg.go.dev/go.uber.org/mock/mockgen?tab=versions
chore: update dependencies (#8242) # Which Problems Are Solved Updates the dependencies. # Additional Context - relates to https://github.com/zitadel/zitadel/security/dependabot/210
2024-07-04 06:28:56 +00:00
go install golang.org/x/tools/cmd/stringer@v0.22.0 # https://pkg.go.dev/golang.org/x/tools/cmd/stringer?tab=versions
chore(Makefile): add go generate target (#6944) This change adds a core_generate_all make target. It installs the required tools and runs generate on the complete project. `golang/mock` is no longer maintained and a fork is available from the Uber folks. So the latter is used as tool. All the mock files have been regenerated and are part of the PR. The obsolete `tools` directory has been removed, as all the tools are now part of specific make targets. Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-11-22 10:56:43 +00:00
go generate ./...
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
.PHONY: core_assets
core_assets:
mkdir -p docs/apis/assets
go run internal/api/assets/generator/asset_generator.go -directory=internal/api/assets/generator/ -assets=docs/apis/assets/assets.md
.PHONY: core_api_generator
core_api_generator:
ifeq (,$(wildcard $(gen_authopt_path)))
go install internal/protoc/protoc-gen-authoption/main.go \
&& mv $$(go env GOPATH)/bin/main $(gen_authopt_path)
endif
ifeq (,$(wildcard $(gen_zitadel_path)))
go install internal/protoc/protoc-gen-zitadel/main.go \
&& mv $$(go env GOPATH)/bin/main $(gen_zitadel_path)
endif
.PHONY: core_grpc_dependencies
core_grpc_dependencies:
perf(query): remove transactions for queries (#8614) # Which Problems Are Solved Queries currently execute 3 statements, begin, query, commit # How the Problems Are Solved remove transaction handling from query methods in database package # Additional Changes - Bump versions of `core_grpc_dependencies`-receipt in Makefile # Additional info During load tests we saw a lot of idle transactions of `zitadel_queries` application name which is the connection pool used to query data in zitadel. Executed query: `select query_start - xact_start, pid, application_name, backend_start, xact_start, query_start, state_change, wait_event_type, wait_event,substring(query, 1, 200) query from pg_stat_activity where datname = 'zitadel' and state <> 'idle';` Mostly the last query executed was `begin isolation level read committed read only`. example: ``` ?column? | pid | application_name | backend_start | xact_start | query_start | state_change | wait_event_type | wait_event | query -----------------+-------+----------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+--------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 00:00:00 | 33030 | zitadel_queries | 2024-10-16 16:25:53.906036+00 | 2024-10-16 16:30:19.191661+00 | 2024-10-16 16:30:19.191661+00 | 2024-10-16 16:30:19.19169+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00 | 33035 | zitadel_queries | 2024-10-16 16:25:53.909629+00 | 2024-10-16 16:30:19.19179+00 | 2024-10-16 16:30:19.19179+00 | 2024-10-16 16:30:19.191805+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00.00412 | 33028 | zitadel_queries | 2024-10-16 16:25:53.904247+00 | 2024-10-16 16:30:19.187734+00 | 2024-10-16 16:30:19.191854+00 | 2024-10-16 16:30:19.191964+00 | Client | ClientRead | SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type 00:00:00.084662 | 33134 | zitadel_es_pusher | 2024-10-16 16:29:54.979692+00 | 2024-10-16 16:30:19.178578+00 | 2024-10-16 16:30:19.26324+00 | 2024-10-16 16:30:19.263267+00 | Client | ClientRead | RELEASE SAVEPOINT cockroach_restart 00:00:00.084768 | 33139 | zitadel_es_pusher | 2024-10-16 16:29:54.979585+00 | 2024-10-16 16:30:19.180762+00 | 2024-10-16 16:30:19.26553+00 | 2024-10-16 16:30:19.265531+00 | LWLock | WALWriteLock | commit 00:00:00.077377 | 33136 | zitadel_es_pusher | 2024-10-16 16:29:54.978582+00 | 2024-10-16 16:30:19.187883+00 | 2024-10-16 16:30:19.26526+00 | 2024-10-16 16:30:19.265431+00 | Client | ClientRead | WITH existing AS ( + | | | | | | | | | (SELECT instance_id, aggregate_type, aggregate_id, "sequence" FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type = $2 AND aggregate_id = $3 ORDER BY "sequence" DE 00:00:00.012309 | 33123 | zitadel_es_pusher | 2024-10-16 16:29:54.963484+00 | 2024-10-16 16:30:19.175066+00 | 2024-10-16 16:30:19.187375+00 | 2024-10-16 16:30:19.187376+00 | IO | WalSync | commit 00:00:00 | 33034 | zitadel_queries | 2024-10-16 16:25:53.90791+00 | 2024-10-16 16:30:19.262921+00 | 2024-10-16 16:30:19.262921+00 | 2024-10-16 16:30:19.263133+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00 | 33039 | zitadel_queries | 2024-10-16 16:25:53.914106+00 | 2024-10-16 16:30:19.191676+00 | 2024-10-16 16:30:19.191676+00 | 2024-10-16 16:30:19.191687+00 | Client | ClientRead | begin isolation level read committed read only 00:00:00.24539 | 33083 | zitadel_projection_spooler | 2024-10-16 16:27:49.895548+00 | 2024-10-16 16:30:19.020058+00 | 2024-10-16 16:30:19.265448+00 | 2024-10-16 16:30:19.26546+00 | Client | ClientRead | SAVEPOINT exec_stmt 00:00:00 | 33125 | zitadel_es_pusher | 2024-10-16 16:29:54.963859+00 | 2024-10-16 16:30:19.191715+00 | 2024-10-16 16:30:19.191715+00 | 2024-10-16 16:30:19.191729+00 | Client | ClientRead | begin 00:00:00.004292 | 33032 | zitadel_queries | 2024-10-16 16:25:53.906624+00 | 2024-10-16 16:30:19.187713+00 | 2024-10-16 16:30:19.192005+00 | 2024-10-16 16:30:19.192062+00 | Client | ClientRead | SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type 00:00:00 | 33031 | zitadel_queries | 2024-10-16 16:25:53.906422+00 | 2024-10-16 16:30:19.191625+00 | 2024-10-16 16:30:19.191625+00 | 2024-10-16 16:30:19.191645+00 | Client | ClientRead | begin isolation level read committed read only ``` The amount of idle transactions is significantly less if the query transactions are removed: example: ``` ?column? | pid | application_name | backend_start | xact_start | query_start | state_change | wait_event_type | wait_event | query -----------------+-------+----------------------------+-------------------------------+-------------------------------+-------------------------------+-------------------------------+-----------------+------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 00:00:00.000094 | 32741 | zitadel_queries | 2024-10-16 16:23:49.73935+00 | 2024-10-16 16:24:59.785589+00 | 2024-10-16 16:24:59.785683+00 | 2024-10-16 16:24:59.785684+00 | | | SELECT created_at, event_type, "sequence", "position", payload, creator, "owner", instance_id, aggregate_type, aggregate_id, revision FROM eventstore.events2 WHERE instance_id = $1 AND aggregate_type 00:00:00 | 32762 | zitadel_es_pusher | 2024-10-16 16:24:02.275136+00 | 2024-10-16 16:24:59.784586+00 | 2024-10-16 16:24:59.784586+00 | 2024-10-16 16:24:59.784607+00 | Client | ClientRead | begin 00:00:00.000167 | 32742 | zitadel_queries | 2024-10-16 16:23:49.740489+00 | 2024-10-16 16:24:59.784274+00 | 2024-10-16 16:24:59.784441+00 | 2024-10-16 16:24:59.784442+00 | | | with usr as ( + | | | | | | | | | select u.id, u.creation_date, u.change_date, u.sequence, u.state, u.resource_owner, u.username, n.login_name as preferred_login_name + | | | | | | | | | from projections.users13 u + | | | | | | | | | left join projections.l 00:00:00.256014 | 32759 | zitadel_projection_spooler | 2024-10-16 16:24:01.418429+00 | 2024-10-16 16:24:59.52959+00 | 2024-10-16 16:24:59.785604+00 | 2024-10-16 16:24:59.785649+00 | Client | ClientRead | UPDATE projections.milestones SET reached_date = $1 WHERE (instance_id = $2) AND (type = $3) AND (reached_date IS NULL) 00:00:00.014199 | 32773 | zitadel_es_pusher | 2024-10-16 16:24:02.320404+00 | 2024-10-16 16:24:59.769509+00 | 2024-10-16 16:24:59.783708+00 | 2024-10-16 16:24:59.783709+00 | IO | WalSync | commit 00:00:00 | 32765 | zitadel_es_pusher | 2024-10-16 16:24:02.28173+00 | 2024-10-16 16:24:59.780413+00 | 2024-10-16 16:24:59.780413+00 | 2024-10-16 16:24:59.780426+00 | Client | ClientRead | begin 00:00:00.012729 | 32777 | zitadel_es_pusher | 2024-10-16 16:24:02.339737+00 | 2024-10-16 16:24:59.767432+00 | 2024-10-16 16:24:59.780161+00 | 2024-10-16 16:24:59.780195+00 | Client | ClientRead | RELEASE SAVEPOINT cockroach_restart ``` --------- Co-authored-by: Tim Möhlmann <tim+github@zitadel.com> Co-authored-by: Livio Spring <livio.a@gmail.com> Co-authored-by: Max Peintner <max@caos.ch> Co-authored-by: Elio Bischof <elio@zitadel.com> Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com> Co-authored-by: Miguel Cabrerizo <30386061+doncicuto@users.noreply.github.com> Co-authored-by: Joakim Lodén <Loddan@users.noreply.github.com> Co-authored-by: Yxnt <Yxnt@users.noreply.github.com> Co-authored-by: Stefan Benz <stefan@caos.ch> Co-authored-by: Harsha Reddy <harsha.reddy@klaviyo.com> Co-authored-by: Zach H <zhirschtritt@gmail.com>
2024-11-04 09:06:14 +00:00
go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.35.1 # https://pkg.go.dev/google.golang.org/protobuf/cmd/protoc-gen-go?tab=versions
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.5.1 # https://pkg.go.dev/google.golang.org/grpc/cmd/protoc-gen-go-grpc?tab=versions
go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway@v2.22.0 # https://pkg.go.dev/github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-grpc-gateway?tab=versions
go install github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2@v2.22.0 # https://pkg.go.dev/github.com/grpc-ecosystem/grpc-gateway/v2/protoc-gen-openapiv2?tab=versions
go install github.com/envoyproxy/protoc-gen-validate@v1.1.0 # https://pkg.go.dev/github.com/envoyproxy/protoc-gen-validate?tab=versions
go install github.com/bufbuild/buf/cmd/buf@v1.45.0 # https://pkg.go.dev/github.com/bufbuild/buf/cmd/buf?tab=versions
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
.PHONY: core_api
core_api: core_api_generator core_grpc_dependencies
buf generate
mkdir -p pkg/grpc
cp -r .artifacts/grpc/github.com/zitadel/zitadel/pkg/grpc/* pkg/grpc/
mkdir -p openapi/v2/zitadel
cp -r .artifacts/grpc/zitadel/ openapi/v2/zitadel
.PHONY: core_build
core_build: core_dependencies core_api core_static core_assets
.PHONY: console_move
console_move:
cp -r console/dist/console/* internal/api/ui/console/static
.PHONY: console_dependencies
console_dependencies:
cd console && \
yarn install --immutable
.PHONY: console_client
console_client:
cd console && \
yarn generate
.PHONY: console_build
console_build: console_dependencies console_client
cd console && \
yarn build
.PHONY: clean
clean:
fix(Makefile): add -r to delete .artifacts/grpc (#6697)
2023-10-18 09:46:45 +00:00
$(RM) -r .artifacts/grpc
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
$(RM) $(gen_authopt_path)
$(RM) $(gen_zitadel_path)
chore(tests): use a coverage server binary (#8407) # Which Problems Are Solved Use a single server instance for API integration tests. This optimizes the time taken for the integration test pipeline, because it allows running tests on multiple packages in parallel. Also, it saves time by not start and stopping a zitadel server for every package. # How the Problems Are Solved - Build a binary with `go build -race -cover ....` - Integration tests only construct clients. The server remains running in the background. - The integration package and tested packages now fully utilize the API. No more direct database access trough `query` and `command` packages. - Use Makefile recipes to setup, start and stop the server in the background. - The binary has the race detector enabled - Init and setup jobs are configured to halt immediately on race condition - Because the server runs in the background, races are only logged. When the server is stopped and race logs exist, the Makefile recipe will throw an error and print the logs. - Makefile recipes include logic to print logs and convert coverage reports after the server is stopped. - Some tests need a downstream HTTP server to make requests, like quota and milestones. A new `integration/sink` package creates an HTTP server and uses websockets to forward HTTP request back to the test packages. The package API uses Go channels for abstraction and easy usage. # Additional Changes - Integration test files already used the `//go:build integration` directive. In order to properly split integration from unit tests, integration test files need to be in a `integration_test` subdirectory of their package. - `UseIsolatedInstance` used to overwrite the `Tester.Client` for each instance. Now a `Instance` object is returned with a gRPC client that is connected to the isolated instance's hostname. - The `Tester` type is now `Instance`. The object is created for the first instance, used by default in any test. Isolated instances are also `Instance` objects and therefore benefit from the same methods and values. The first instance and any other us capable of creating an isolated instance over the system API. - All test packages run in an Isolated instance by calling `NewInstance()` - Individual tests that use an isolated instance use `t.Parallel()` # Additional Context - Closes #6684 - https://go.dev/doc/articles/race_detector - https://go.dev/doc/build-cover --------- Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-09-06 12:47:57 +00:00
$(RM) -r tmp/
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
.PHONY: core_unit_test
core_unit_test:
chore(tests): use a coverage server binary (#8407) # Which Problems Are Solved Use a single server instance for API integration tests. This optimizes the time taken for the integration test pipeline, because it allows running tests on multiple packages in parallel. Also, it saves time by not start and stopping a zitadel server for every package. # How the Problems Are Solved - Build a binary with `go build -race -cover ....` - Integration tests only construct clients. The server remains running in the background. - The integration package and tested packages now fully utilize the API. No more direct database access trough `query` and `command` packages. - Use Makefile recipes to setup, start and stop the server in the background. - The binary has the race detector enabled - Init and setup jobs are configured to halt immediately on race condition - Because the server runs in the background, races are only logged. When the server is stopped and race logs exist, the Makefile recipe will throw an error and print the logs. - Makefile recipes include logic to print logs and convert coverage reports after the server is stopped. - Some tests need a downstream HTTP server to make requests, like quota and milestones. A new `integration/sink` package creates an HTTP server and uses websockets to forward HTTP request back to the test packages. The package API uses Go channels for abstraction and easy usage. # Additional Changes - Integration test files already used the `//go:build integration` directive. In order to properly split integration from unit tests, integration test files need to be in a `integration_test` subdirectory of their package. - `UseIsolatedInstance` used to overwrite the `Tester.Client` for each instance. Now a `Instance` object is returned with a gRPC client that is connected to the isolated instance's hostname. - The `Tester` type is now `Instance`. The object is created for the first instance, used by default in any test. Isolated instances are also `Instance` objects and therefore benefit from the same methods and values. The first instance and any other us capable of creating an isolated instance over the system API. - All test packages run in an Isolated instance by calling `NewInstance()` - Individual tests that use an isolated instance use `t.Parallel()` # Additional Context - Closes #6684 - https://go.dev/doc/articles/race_detector - https://go.dev/doc/build-cover --------- Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-09-06 12:47:57 +00:00
go test -race -coverprofile=profile.cov -coverpkg=./internal/... ./...
.PHONY: core_integration_db_up
core_integration_db_up:
feat(cache): redis cache (#8822) # Which Problems Are Solved Add a cache implementation using Redis single mode. This does not add support for Redis Cluster or sentinel. # How the Problems Are Solved Added the `internal/cache/redis` package. All operations occur atomically, including setting of secondary indexes, using LUA scripts where needed. The [`miniredis`](https://github.com/alicebob/miniredis) package is used to run unit tests. # Additional Changes - Move connector code to `internal/cache/connector/...` and remove duplicate code from `query` and `command` packages. - Fix a missed invalidation on the restrictions projection # Additional Context Closes #8130
2024-11-04 10:44:51 +00:00
docker compose -f internal/integration/config/docker-compose.yaml up --pull always --wait $${INTEGRATION_DB_FLAVOR} cache
chore(tests): use a coverage server binary (#8407) # Which Problems Are Solved Use a single server instance for API integration tests. This optimizes the time taken for the integration test pipeline, because it allows running tests on multiple packages in parallel. Also, it saves time by not start and stopping a zitadel server for every package. # How the Problems Are Solved - Build a binary with `go build -race -cover ....` - Integration tests only construct clients. The server remains running in the background. - The integration package and tested packages now fully utilize the API. No more direct database access trough `query` and `command` packages. - Use Makefile recipes to setup, start and stop the server in the background. - The binary has the race detector enabled - Init and setup jobs are configured to halt immediately on race condition - Because the server runs in the background, races are only logged. When the server is stopped and race logs exist, the Makefile recipe will throw an error and print the logs. - Makefile recipes include logic to print logs and convert coverage reports after the server is stopped. - Some tests need a downstream HTTP server to make requests, like quota and milestones. A new `integration/sink` package creates an HTTP server and uses websockets to forward HTTP request back to the test packages. The package API uses Go channels for abstraction and easy usage. # Additional Changes - Integration test files already used the `//go:build integration` directive. In order to properly split integration from unit tests, integration test files need to be in a `integration_test` subdirectory of their package. - `UseIsolatedInstance` used to overwrite the `Tester.Client` for each instance. Now a `Instance` object is returned with a gRPC client that is connected to the isolated instance's hostname. - The `Tester` type is now `Instance`. The object is created for the first instance, used by default in any test. Isolated instances are also `Instance` objects and therefore benefit from the same methods and values. The first instance and any other us capable of creating an isolated instance over the system API. - All test packages run in an Isolated instance by calling `NewInstance()` - Individual tests that use an isolated instance use `t.Parallel()` # Additional Context - Closes #6684 - https://go.dev/doc/articles/race_detector - https://go.dev/doc/build-cover --------- Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-09-06 12:47:57 +00:00
.PHONY: core_integration_db_down
core_integration_db_down:
docker compose -f internal/integration/config/docker-compose.yaml down
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
feat(api/v2): implement U2F session check (#6339)
2023-08-11 15:36:18 +00:00
.PHONY: core_integration_setup
core_integration_setup:
chore(tests): use a coverage server binary (#8407) # Which Problems Are Solved Use a single server instance for API integration tests. This optimizes the time taken for the integration test pipeline, because it allows running tests on multiple packages in parallel. Also, it saves time by not start and stopping a zitadel server for every package. # How the Problems Are Solved - Build a binary with `go build -race -cover ....` - Integration tests only construct clients. The server remains running in the background. - The integration package and tested packages now fully utilize the API. No more direct database access trough `query` and `command` packages. - Use Makefile recipes to setup, start and stop the server in the background. - The binary has the race detector enabled - Init and setup jobs are configured to halt immediately on race condition - Because the server runs in the background, races are only logged. When the server is stopped and race logs exist, the Makefile recipe will throw an error and print the logs. - Makefile recipes include logic to print logs and convert coverage reports after the server is stopped. - Some tests need a downstream HTTP server to make requests, like quota and milestones. A new `integration/sink` package creates an HTTP server and uses websockets to forward HTTP request back to the test packages. The package API uses Go channels for abstraction and easy usage. # Additional Changes - Integration test files already used the `//go:build integration` directive. In order to properly split integration from unit tests, integration test files need to be in a `integration_test` subdirectory of their package. - `UseIsolatedInstance` used to overwrite the `Tester.Client` for each instance. Now a `Instance` object is returned with a gRPC client that is connected to the isolated instance's hostname. - The `Tester` type is now `Instance`. The object is created for the first instance, used by default in any test. Isolated instances are also `Instance` objects and therefore benefit from the same methods and values. The first instance and any other us capable of creating an isolated instance over the system API. - All test packages run in an Isolated instance by calling `NewInstance()` - Individual tests that use an isolated instance use `t.Parallel()` # Additional Context - Closes #6684 - https://go.dev/doc/articles/race_detector - https://go.dev/doc/build-cover --------- Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-09-06 12:47:57 +00:00
go build -cover -race -tags integration -o zitadel.test main.go
mkdir -p $${GOCOVERDIR}
GORACE="halt_on_error=1" ./zitadel.test init --config internal/integration/config/zitadel.yaml --config internal/integration/config/${INTEGRATION_DB_FLAVOR}.yaml
GORACE="halt_on_error=1" ./zitadel.test setup --masterkeyFromEnv --init-projections --config internal/integration/config/zitadel.yaml --config internal/integration/config/${INTEGRATION_DB_FLAVOR}.yaml --steps internal/integration/config/steps.yaml
.PHONY: core_integration_server_start
core_integration_server_start: core_integration_setup
GORACE="log_path=tmp/race.log" \
./zitadel.test start --masterkeyFromEnv --config internal/integration/config/zitadel.yaml --config internal/integration/config/${INTEGRATION_DB_FLAVOR}.yaml \
> tmp/zitadel.log 2>&1 \
& printf $$! > tmp/zitadel.pid
.PHONY: core_integration_test_packages
core_integration_test_packages:
feat(storage): generic cache interface (#8628) # Which Problems Are Solved We identified the need of caching. Currently we have a number of places where we use different ways of caching, like go maps or LRU. We might also want shared chaches in the future, like Redis-based or in special SQL tables. # How the Problems Are Solved Define a generic Cache interface which allows different implementations. - A noop implementation is provided and enabled as. - An implementation using go maps is provided - disabled in defaults.yaml - enabled in integration tests - Authz middleware instance objects are cached using the interface. # Additional Changes - Enabled integration test command raceflag - Fix a race condition in the limits integration test client - Fix a number of flaky integration tests. (Because zitadel is super fast now!) :guitar: :rocket: # Additional Context Related to https://github.com/zitadel/zitadel/issues/8648
2024-09-25 19:40:21 +00:00
go test -race -count 1 -tags integration -timeout 30m $$(go list -tags integration ./... | grep "integration_test")
chore(tests): use a coverage server binary (#8407) # Which Problems Are Solved Use a single server instance for API integration tests. This optimizes the time taken for the integration test pipeline, because it allows running tests on multiple packages in parallel. Also, it saves time by not start and stopping a zitadel server for every package. # How the Problems Are Solved - Build a binary with `go build -race -cover ....` - Integration tests only construct clients. The server remains running in the background. - The integration package and tested packages now fully utilize the API. No more direct database access trough `query` and `command` packages. - Use Makefile recipes to setup, start and stop the server in the background. - The binary has the race detector enabled - Init and setup jobs are configured to halt immediately on race condition - Because the server runs in the background, races are only logged. When the server is stopped and race logs exist, the Makefile recipe will throw an error and print the logs. - Makefile recipes include logic to print logs and convert coverage reports after the server is stopped. - Some tests need a downstream HTTP server to make requests, like quota and milestones. A new `integration/sink` package creates an HTTP server and uses websockets to forward HTTP request back to the test packages. The package API uses Go channels for abstraction and easy usage. # Additional Changes - Integration test files already used the `//go:build integration` directive. In order to properly split integration from unit tests, integration test files need to be in a `integration_test` subdirectory of their package. - `UseIsolatedInstance` used to overwrite the `Tester.Client` for each instance. Now a `Instance` object is returned with a gRPC client that is connected to the isolated instance's hostname. - The `Tester` type is now `Instance`. The object is created for the first instance, used by default in any test. Isolated instances are also `Instance` objects and therefore benefit from the same methods and values. The first instance and any other us capable of creating an isolated instance over the system API. - All test packages run in an Isolated instance by calling `NewInstance()` - Individual tests that use an isolated instance use `t.Parallel()` # Additional Context - Closes #6684 - https://go.dev/doc/articles/race_detector - https://go.dev/doc/build-cover --------- Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-09-06 12:47:57 +00:00
.PHONY: core_integration_server_stop
core_integration_server_stop:
pid=$$(cat tmp/zitadel.pid); \
$(RM) tmp/zitadel.pid; \
kill $$pid; \
if [ -s tmp/race.log.$$pid ]; then \
cat tmp/race.log.$$pid; \
exit 66; \
fi
.PHONY: core_integration_reports
core_integration_reports:
go tool covdata textfmt -i=tmp/coverage -pkg=github.com/zitadel/zitadel/internal/...,github.com/zitadel/zitadel/cmd/... -o profile.cov
feat(api/v2): implement U2F session check (#6339)
2023-08-11 15:36:18 +00:00
.PHONY: core_integration_test
chore(tests): use a coverage server binary (#8407) # Which Problems Are Solved Use a single server instance for API integration tests. This optimizes the time taken for the integration test pipeline, because it allows running tests on multiple packages in parallel. Also, it saves time by not start and stopping a zitadel server for every package. # How the Problems Are Solved - Build a binary with `go build -race -cover ....` - Integration tests only construct clients. The server remains running in the background. - The integration package and tested packages now fully utilize the API. No more direct database access trough `query` and `command` packages. - Use Makefile recipes to setup, start and stop the server in the background. - The binary has the race detector enabled - Init and setup jobs are configured to halt immediately on race condition - Because the server runs in the background, races are only logged. When the server is stopped and race logs exist, the Makefile recipe will throw an error and print the logs. - Makefile recipes include logic to print logs and convert coverage reports after the server is stopped. - Some tests need a downstream HTTP server to make requests, like quota and milestones. A new `integration/sink` package creates an HTTP server and uses websockets to forward HTTP request back to the test packages. The package API uses Go channels for abstraction and easy usage. # Additional Changes - Integration test files already used the `//go:build integration` directive. In order to properly split integration from unit tests, integration test files need to be in a `integration_test` subdirectory of their package. - `UseIsolatedInstance` used to overwrite the `Tester.Client` for each instance. Now a `Instance` object is returned with a gRPC client that is connected to the isolated instance's hostname. - The `Tester` type is now `Instance`. The object is created for the first instance, used by default in any test. Isolated instances are also `Instance` objects and therefore benefit from the same methods and values. The first instance and any other us capable of creating an isolated instance over the system API. - All test packages run in an Isolated instance by calling `NewInstance()` - Individual tests that use an isolated instance use `t.Parallel()` # Additional Context - Closes #6684 - https://go.dev/doc/articles/race_detector - https://go.dev/doc/build-cover --------- Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2024-09-06 12:47:57 +00:00
core_integration_test: core_integration_server_start core_integration_test_packages core_integration_server_stop core_integration_reports
feat(crypto): use passwap for machine and app secrets (#7657) * feat(crypto): use passwap for machine and app secrets * fix command package tests * add hash generator command test * naming convention, fix query tests * rename PasswordHasher and cleanup start commands * add reducer tests * fix intergration tests, cleanup old config * add app secret unit tests * solve setup panics * fix push of updated events * add missing event translations * update documentation * solve linter errors * remove nolint:SA1019 as it doesn't seem to help anyway * add nolint to deprecated filter usage * update users migration version * remove unused ClientSecret from APIConfigChangedEvent --------- Co-authored-by: Livio Spring <livio.a@gmail.com>
2024-04-05 09:35:49 +00:00
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
.PHONY: console_lint
console_lint:
cd console && \
yarn lint
chore(make): add docker_image make target (#7440)
2024-02-27 16:29:09 +00:00
.PHONY: core_lint
ci: improve performance (#5953) * pipeline runs on ubuntu instead of docker * added Makefile to build zitadel core (backend) and console (frontend) * pipeline runs in parallel where possible * pipeline is split into multiple jobs * removed goreleaser * added command to check if zitadel instance is running
2023-07-17 08:08:20 +00:00
core_lint:
golangci-lint run \
--timeout 10m \
--config ./.golangci.yaml \
--out-format=github-actions \
fix: build css for login correctly (#6265)
2023-07-25 05:52:59 +00:00
--concurrency=$$(getconf _NPROCESSORS_ONLN)
Reference in New Issue Copy Permalink