# Which Problems Are Solved
Use a single server instance for API integration tests. This optimizes
the time taken for the integration test pipeline,
because it allows running tests on multiple packages in parallel. Also,
it saves time by not start and stopping a zitadel server for every
package.
# How the Problems Are Solved
- Build a binary with `go build -race -cover ....`
- Integration tests only construct clients. The server remains running
in the background.
- The integration package and tested packages now fully utilize the API.
No more direct database access trough `query` and `command` packages.
- Use Makefile recipes to setup, start and stop the server in the
background.
- The binary has the race detector enabled
- Init and setup jobs are configured to halt immediately on race
condition
- Because the server runs in the background, races are only logged. When
the server is stopped and race logs exist, the Makefile recipe will
throw an error and print the logs.
- Makefile recipes include logic to print logs and convert coverage
reports after the server is stopped.
- Some tests need a downstream HTTP server to make requests, like quota
and milestones. A new `integration/sink` package creates an HTTP server
and uses websockets to forward HTTP request back to the test packages.
The package API uses Go channels for abstraction and easy usage.
# Additional Changes
- Integration test files already used the `//go:build integration`
directive. In order to properly split integration from unit tests,
integration test files need to be in a `integration_test` subdirectory
of their package.
- `UseIsolatedInstance` used to overwrite the `Tester.Client` for each
instance. Now a `Instance` object is returned with a gRPC client that is
connected to the isolated instance's hostname.
- The `Tester` type is now `Instance`. The object is created for the
first instance, used by default in any test. Isolated instances are also
`Instance` objects and therefore benefit from the same methods and
values. The first instance and any other us capable of creating an
isolated instance over the system API.
- All test packages run in an Isolated instance by calling
`NewInstance()`
- Individual tests that use an isolated instance use `t.Parallel()`
# Additional Context
- Closes#6684
- https://go.dev/doc/articles/race_detector
- https://go.dev/doc/build-cover
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
# Which Problems Are Solved
ZITADEL currently selects the instance context based on a HTTP header
(see https://github.com/zitadel/zitadel/issues/8279#issue-2399959845 and
checks it against the list of instance domains. Let's call it instance
or API domain.
For any context based URL (e.g. OAuth, OIDC, SAML endpoints, links in
emails, ...) the requested domain (instance domain) will be used. Let's
call it the public domain.
In cases of proxied setups, all exposed domains (public domains) require
the domain to be managed as instance domain.
This can either be done using the "ExternalDomain" in the runtime config
or via system API, which requires a validation through CustomerPortal on
zitadel.cloud.
# How the Problems Are Solved
- Two new headers / header list are added:
- `InstanceHostHeaders`: an ordered list (first sent wins), which will
be used to match the instance.
(For backward compatibility: the `HTTP1HostHeader`, `HTTP2HostHeader`
and `forwarded`, `x-forwarded-for`, `x-forwarded-host` are checked
afterwards as well)
- `PublicHostHeaders`: an ordered list (first sent wins), which will be
used as public host / domain. This will be checked against a list of
trusted domains on the instance.
- The middleware intercepts all requests to the API and passes a
`DomainCtx` object with the hosts and protocol into the context
(previously only a computed `origin` was passed)
- HTTP / GRPC server do not longer try to match the headers to instances
themself, but use the passed `http.DomainContext` in their interceptors.
- The `RequestedHost` and `RequestedDomain` from authz.Instance are
removed in favor of the `http.DomainContext`
- When authenticating to or signing out from Console UI, the current
`http.DomainContext(ctx).Origin` (already checked by instance
interceptor for validity) is used to compute and dynamically add a
`redirect_uri` and `post_logout_redirect_uri`.
- Gateway passes all configured host headers (previously only did
`x-zitadel-*`)
- Admin API allows to manage trusted domain
# Additional Changes
None
# Additional Context
- part of #8279
- open topics:
- "single-instance" mode
- Console UI
# Which Problems Are Solved
The v2beta services are stable but not GA.
# How the Problems Are Solved
The v2beta services are copied to v2. The corresponding v1 and v2beta
services are deprecated.
# Additional Context
Closes#7236
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
# Which Problems Are Solved
After migrating the access token events in #7822, milestones based on
authentication, resp. theses events would not be reached.
# How the Problems Are Solved
Additionally use the `oidc_session.Added` event to check for
`milestone.AuthenticationSucceededOnInstance` and
`milestone.AuthenticationSucceededOnApplication`.
# Additional Changes
None.
# Additional Context
- relates to #7822
- noticed internally
* feat: smtp templates poc
* feat: add isActive & ProviderType to SMTP backend
* feat: change providertype to uint32 and fix tests
* feat: minimal smtp provider component
* feat: woking on diiferent providers
* feat: keep working on providers
* feat: initial stepper for new provider
* fix: settings list and working on stepper
* feat: step 1 and 2 form inputs
* feat: starter for smtp test step
* fix: misspelled SMPT
* fix: remove tests for now
* feat: add tls toggle remove old google provider
* feat: working on add smtp and table
* fix: duplicated identifiers
* fix: settings list
* fix: add missing smtp config properties
* fix: add configID to smtp config table
* fix: working on listproviders
* feat: working in listSMTPConfigs
* fix: add count to listsmtpconfigs
* fix: getting empty results from listSMTPConfigs
* feat: table now shows real data
* fix: remaining styles for smtp-table
* fix: remove old notification-smtp-provider-component
* feat: delete smtp configuration
* feat: deactivate smtp config
* feat: replace isActive with state for smtp config
* feat: activate smtp config
* fix: remaining errors after main merge
* fix: list smtp providers panic and material mdc
* feat: refactor to only one provider component
* feat: current provider details view
* fix: refactor AddSMTPConfig and ChangeSMTPConfig
* fix: smtp config reduce issue
* fix: recover domain in NewIAMSMTPConfigWriteModel
* fix: add code needed by SetUpInstance
* fix: go tests and warn about passing context to InstanceAggregateFromWriteModel
* fix: i18n and add missing trans for fr, it, zh
* fix: add e2e tests
* docs: add smtp templates
* fix: remove provider_type, add description
* fix: remaining error from merge main
* fix: add @stebenz change for primary key
* fix: inactive placed after removed to prevent deleted configs to show as inactive
* fix: smtp provider id can be empty (migrated)
* feat: add mailchimp transactional template
* feat: add Brevo (Sendinblue) template
* feat: change brevo logo, add color to tls icon
* fix: queries use resourceowner, id must not be empty
* fix: deal with old smtp settings and tests
* fix: resourceOwner is the instanceID
* fix: remove aggregate_id, rename SMTPConfigByAggregateID with SMTPConfigActive
* fix: add tests for multiple configs with different IDs
* fix: conflict
* fix: remove notification-smtp-provider
* fix: add @peintnermax suggestions, rename module and fix e2e tests
* fix: remove material legacy modules
* fix: remove ctx as parameter for InstanceAggregateFromWriteModel
* fix: add Id to SMTPConfigToPb
* fix: change InstanceAggregateFromWriteModel to avoid linter errors
* fix import
* rm unused package-lock
* update yarn lock
---------
Co-authored-by: Elio Bischof <elio@zitadel.com>
Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
* fix: add action v2 execution to features
* fix: add action v2 execution to features
* fix: add action v2 execution to features
* fix: update internal/command/instance_features_model.go
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
* fix: merge back main
* fix: merge back main
* fix: rename feature and service
* fix: rename feature and service
* fix: review changes
* fix: review changes
---------
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
chore(fmt): run gci on complete project
Fix global import formatting in go code by running the `gci` command. This allows us to just use the command directly, instead of fixing the import order manually for the linter, on each PR.
Co-authored-by: Elio Bischof <elio@zitadel.com>
* start user by id
* ignore debug bin
* use new user by id
* new sql
* fix(sql): replace STRING with text for psql compatabilit
* some changes
* fix: correct user queries
* fix tests
* unify sql statements
* use specific get user methods
* search login name case insensitive
* refactor: optimise user statements
* add index
* fix queries
* fix: correct domain segregation
* return all login names
* fix queries
* improve readability
* query should be correct now
* cleanup statements
* fix username / loginname handling
* fix: psql doesn't support create view if not exists
* fix: create pre-release
* ignore release comments
* add lower fields
* fix: always to lower
* update to latest projection
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
* feat: return 404 or 409 if org reg disallowed
* fix: system limit permissions
* feat: add iam limits api
* feat: disallow public org registrations on default instance
* add integration test
* test: integration
* fix test
* docs: describe public org registrations
* avoid updating docs deps
* fix system limits integration test
* silence integration tests
* fix linting
* ignore strange linter complaints
* review
* improve reset properties naming
* redefine the api
* use restrictions aggregate
* test query
* simplify and test projection
* test commands
* fix unit tests
* move integration test
* support restrictions on default instance
* also test GetRestrictions
* self review
* lint
* abstract away resource owner
* fix tests
* configure supported languages
* fix allowed languages
* fix tests
* default lang must not be restricted
* preferred language must be allowed
* change preferred languages
* check languages everywhere
* lint
* test command side
* lint
* add integration test
* add integration test
* restrict supported ui locales
* lint
* lint
* cleanup
* lint
* allow undefined preferred language
* fix integration tests
* update main
* fix env var
* ignore linter
* ignore linter
* improve integration test config
* reduce cognitive complexity
* compile
* fix(console): switch back to saved language
* feat(API): get allowed languages
* fix(console): only make allowed languages selectable
* warn when editing not allowed languages
* check for duplicates
* remove useless restriction checks
* review
* revert restriction renaming
* fix language restrictions
* lint
* generate
* allow custom texts for supported langs for now
* fix tests
* cleanup
* cleanup
* cleanup
* lint
* unsupported preferred lang is allowed
* fix integration test
* allow unsupported preferred languages
* lint
* load languages for tests
* cleanup
* lint
* cleanup
* get allowed only on admin
* cleanup
* reduce flakiness on very limited postgres
* simplify langSvc
* refactor according to suggestions in pr
* lint
* set first allowed language as default
* selectionchange for language in msg texts
* initialize login texts
* init message texts
* lint
---------
Co-authored-by: peintnermax <max@caos.ch>
* feat: return 404 or 409 if org reg disallowed
* fix: system limit permissions
* feat: add iam limits api
* feat: disallow public org registrations on default instance
* add integration test
* test: integration
* fix test
* docs: describe public org registrations
* avoid updating docs deps
* fix system limits integration test
* silence integration tests
* fix linting
* ignore strange linter complaints
* review
* improve reset properties naming
* redefine the api
* use restrictions aggregate
* test query
* simplify and test projection
* test commands
* fix unit tests
* move integration test
* support restrictions on default instance
* also test GetRestrictions
* self review
* lint
* abstract away resource owner
* fix tests
* configure supported languages
* fix allowed languages
* fix tests
* default lang must not be restricted
* preferred language must be allowed
* change preferred languages
* check languages everywhere
* lint
* test command side
* lint
* add integration test
* add integration test
* restrict supported ui locales
* lint
* lint
* cleanup
* lint
* allow undefined preferred language
* fix integration tests
* update main
* fix env var
* ignore linter
* ignore linter
* improve integration test config
* reduce cognitive complexity
* compile
* check for duplicates
* remove useless restriction checks
* review
* revert restriction renaming
* fix language restrictions
* lint
* generate
* allow custom texts for supported langs for now
* fix tests
* cleanup
* cleanup
* cleanup
* lint
* unsupported preferred lang is allowed
* fix integration test
* finish reverting to old property name
* finish reverting to old property name
* load languages
* refactor(i18n): centralize translators and fs
* lint
* amplify no validations on preferred languages
* fix integration test
* lint
* fix resetting allowed languages
* test unchanged restrictions
This change adds a core_generate_all make target.
It installs the required tools and runs generate on the complete project.
`golang/mock` is no longer maintained and a fork is available
from the Uber folks. So the latter is used as tool.
All the mock files have been regenerated and are part of the PR.
The obsolete `tools` directory has been removed,
as all the tools are now part of specific make targets.
Co-authored-by: Silvan <silvan.reusser@gmail.com>
* get key by id and cache them
* userinfo from events for v2 tokens
* improve keyset caching
* concurrent token and client checks
* client and project in single query
* logging and otel
* drop owner_removed column on apps and authN tables
* userinfo and project roles in go routines
* get oidc user info from projections and add actions
* add avatar URL
* some cleanup
* pull oidc work branch
* remove storage from server
* add config flag for experimental introspection
* legacy introspection flag
* drop owner_removed column on user projections
* drop owner_removed column on useer_metadata
* query userinfo unit test
* query introspection client test
* add user_grants to the userinfo query
* handle PAT scopes
* bring triggers back
* test instance keys query
* add userinfo unit tests
* unit test keys
* go mod tidy
* solve some bugs
* fix missing preferred login name
* do not run triggers in go routines, they seem to deadlock
* initialize the trigger handlers late with a sync.OnceValue
* Revert "do not run triggers in go routines, they seem to deadlock"
This reverts commit 2a03da2127.
* add missing translations
* chore: update go version for linting
* pin oidc version
* parse a global time location for query test
* fix linter complains
* upgrade go lint
* fix more linting issues
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
* feat: add activity logs on user actions with authentication, resourceAPI and sessionAPI
* feat: add activity logs on user actions with authentication, resourceAPI and sessionAPI
* feat: add activity logs on user actions with authentication, resourceAPI and sessionAPI
* feat: add activity logs on user actions with authentication, resourceAPI and sessionAPI
* feat: add activity logs on user actions with authentication, resourceAPI and sessionAPI
* fix: add unit tests to info package for context changes
* fix: add activity_interceptor.go suggestion
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
* fix: refactoring and fixes through PR review
* fix: add auth service to lists of resourceAPIs
---------
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
Co-authored-by: Fabi <fabienne@zitadel.com>
This implementation increases parallel write capabilities of the eventstore.
Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and [06](https://zitadel.com/docs/support/advisory/a10006).
The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`.
If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.
* take baseurl if saved on event
* refactor: make es mocks reusable
* Revert "refactor: make es mocks reusable"
This reverts commit 434ce12a6a.
* make messages testable
* test asset url
* fmt
* fmt
* simplify notification.Start
* test url combinations
* support init code added
* support password changed
* support reset pw
* support user domain claimed
* support add pwless login
* support verify phone
* Revert "support verify phone"
This reverts commit e40503303e.
* save trigger origin from ctx
* add ready for review check
* camel
* test email otp
* fix variable naming
* fix DefaultOTPEmailURLV2
* Revert "fix DefaultOTPEmailURLV2"
This reverts commit fa34d4d2a8.
* fix email otp challenged test
* fix email otp challenged test
* pass origin in login and gateway requests
* take origin from header
* take x-forwarded if present
* Update internal/notification/handlers/queries.go
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
* Update internal/notification/handlers/commands.go
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
* move origin header to ctx if available
* generate
* cleanup
* use forwarded header
* support X-Forwarded-* headers
* standardize context handling
* fix linting
---------
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
* feat: add reply-to header to smtp messages
* fix: grpc reply_to_address min 0 and js var name
* fix: add missing translations
* fix merge and linting
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
* feat: add otp (sms and email) checks in session api
* implement sending
* fix tests
* add tests
* add integration tests
* fix merge main and add tests
* put default OTP Email url into config
---------
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
* feat: add phone change and code verification for user v2 api
* feat: add phone change and code verification for user v2 api
* fix: add ignored phone.proto
* fix: integration tests
* Update proto/zitadel/user/v2alpha/user_service.proto
* Update idp_template.go
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
* feat(api): add password reset and change to user service
* integration tests
* invalidate password check after password change
* handle notification type
* fix proto
* command/crypto: DRY the code
- reuse the the algorithm switch to create a secret generator
- add a verifyCryptoCode function
* command: crypto code tests
* migrate webauthn package
* finish integration tests with webauthn mock client
* chore(proto): update versions
* change protoc plugin
* some cleanups
* define api for setting emails in new api
* implement user.SetEmail
* move SetEmail buisiness logic into command
* resuse newCryptoCode
* command: add ChangeEmail unit tests
Not complete, was not able to mock the generator.
* Revert "resuse newCryptoCode"
This reverts commit c89e90ae35.
* undo change to crypto code generators
* command: use a generator so we can test properly
* command: reorganise ChangeEmail
improve test coverage
* implement VerifyEmail
including unit tests
* add URL template tests
* begin user creation
* change protos
* implement metadata and move context
* merge commands
* proto: change context to object
* remove old auth option
* remove old auth option
* fix linting errors
run gci on modified files
* add permission checks and fix some errors
* comments
* comments
* update email requests
* rename proto requests
* cleanup and docs
* simplify
* simplify
* fix setup
* remove unused proto messages / fields
---------
Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
* chore(proto): update versions
* change protoc plugin
* some cleanups
* define api for setting emails in new api
* implement user.SetEmail
* move SetEmail buisiness logic into command
* resuse newCryptoCode
* command: add ChangeEmail unit tests
Not complete, was not able to mock the generator.
* Revert "resuse newCryptoCode"
This reverts commit c89e90ae35.
* undo change to crypto code generators
* command: use a generator so we can test properly
* command: reorganise ChangeEmail
improve test coverage
* implement VerifyEmail
including unit tests
* add URL template tests
* proto: change context to object
* remove old auth option
* remove old auth option
* fix linting errors
run gci on modified files
* add permission checks and fix some errors
* comments
* comments
---------
Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
