288 Commits

Author SHA1 Message Date
Tim Möhlmann
911071a4b5
fix(db): add additional connection pool for projection spooling (#7094)
* fix(db): add additional connection pool for projection spooling

* use correct connection pool for projections

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>

(cherry picked from commit fe1337536f9d7e055e53850b060e9ee806b6c1cb)
2023-12-20 19:39:10 +01:00
Livio Spring
7015d347e1
fix(query): user performance (#7081)
* fix(query): user performance (#6537)

* start user by id

* ignore debug bin

* use new user by id

* new sql

* fix(sql): replace STRING with text for psql compatabilit

* some changes

* fix: correct user queries

* fix tests

* unify sql statements

* use specific get user methods

* search login name case insensitive

* refactor: optimise user statements

* add index

* fix queries

* fix: correct domain segregation

* return all login names

* fix queries

* improve readability

* query should be correct now

* cleanup statements

* fix username / loginname handling

* fix: psql doesn't support create view if not exists

* fix: create pre-release

* ignore release comments

* add lower fields

* fix: always to lower

* update to latest projection

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>

(cherry picked from commit ddbea119f1c49845a3e364fa6dc565ab3180989a)

* update projection versions

* trigger workflow

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-12-14 11:17:35 +00:00
Tim Möhlmann
72bc3ffe14
fix(oidc): add missing fields to introspection (#6967)
during QA I found some user info and org ID was missing.
This change adds those missing fields.
2023-11-23 16:17:50 +02:00
Tim Möhlmann
2f91679623
chore(Makefile): add go generate target (#6944)
This change adds a core_generate_all make target.
It installs the required tools and runs generate on the complete project.

`golang/mock` is no longer maintained and a fork is available
from the Uber folks. So the latter is used as tool.
All the mock files have been regenerated and are part of the PR.

The obsolete `tools` directory has been removed,
as all the tools are now part of specific make targets.

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-11-22 10:56:43 +00:00
Elio Bischof
76fe032b5f
feat: option to disallow public org registration (#6917)
* feat: return 404 or 409 if org reg disallowed

* fix: system limit permissions

* feat: add iam limits api

* feat: disallow public org registrations on default instance

* add integration test

* test: integration

* fix test

* docs: describe public org registrations

* avoid updating docs deps

* fix system limits integration test

* silence integration tests

* fix linting

* ignore strange linter complaints

* review

* improve reset properties naming

* redefine the api

* use restrictions aggregate

* test query

* simplify and test projection

* test commands

* fix unit tests

* move integration test

* support restrictions on default instance

* also test GetRestrictions

* self review

* lint

* abstract away resource owner

* fix tests

* lint
2023-11-22 09:29:38 +00:00
Tim Möhlmann
ba9b807854
perf(oidc): optimize the introspection endpoint (#6909)
* get key by id and cache them

* userinfo from events for v2 tokens

* improve keyset caching

* concurrent token and client checks

* client and project in single query

* logging and otel

* drop owner_removed column on apps and authN tables

* userinfo and project roles in go routines

* get  oidc user info from projections and add actions

* add avatar URL

* some cleanup

* pull oidc work branch

* remove storage from server

* add config flag for experimental introspection

* legacy introspection flag

* drop owner_removed column on user projections

* drop owner_removed column on useer_metadata

* query userinfo unit test

* query introspection client test

* add user_grants to the userinfo query

* handle PAT scopes

* bring triggers back

* test instance keys query

* add userinfo unit tests

* unit test keys

* go mod tidy

* solve some bugs

* fix missing preferred login name

* do not run triggers in go routines, they seem to deadlock

* initialize the trigger handlers late with a sync.OnceValue

* Revert "do not run triggers in go routines, they seem to deadlock"

This reverts commit 2a03da2127b7dc74552ec25d4772282a82cc1cba.

* add missing translations

* chore: update go version for linting

* pin oidc version

* parse a global time location for query test

* fix linter complains

* upgrade go lint

* fix more linting issues

---------

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2023-11-21 13:11:38 +01:00
Stefan Benz
0ec7a74877
perf: remove owner removed columns from projections for oidc (#6925)
* fix: remove owner removed columns from login names projection

* fix: remove owner removed columns from flow projection

* fix: remove owner removed columns from project, projectgrant and member projections

* fix: correct unit tests for session projection

* fix: correct unit tests for session projection
2023-11-20 17:21:08 +02:00
Livio Spring
2e8c3b5a53
feat: allow session deletion without session token (#6889)
* fix: add resource owner of user and change the one of session to instance

* use user resource owner from session projection

* fix session permission check

* integration tests and fixes

* update api docs
2023-11-16 07:35:50 +01:00
Ahmed Fwela
3f22fb3a5c
feat(user/v1): support composite queries (#6361)
* feat(user/v1): support composite queries

* fix: added proper error handling for NotQuery

* Added error when there are too many levels of nesting

* Add localization keys for english

* Update internal/api/grpc/user/query.go
2023-11-09 11:38:34 +01:00
Livio Spring
22e2d55999
Merge pull request from GHSA-7h8m-vrxx-vr4m
* fix: handle locking policy correctly for multiple simultaneous password checks

* recheck events
2023-11-08 14:19:13 +01:00
sp132
9a708b1b78
feat: extend session search service (#6746)
* feat: extend session search service (#6029)

add two more searching criteria - human user id and session creation date

optional sorting by the session creation date

* fix: use correct column identifier

* fix: implement Col()

* chore: fix unit tests

* chore: fix linter warnings

---------

Co-authored-by: Fabi <fabienne@zitadel.com>
2023-11-08 11:32:13 +01:00
Tim Möhlmann
b3ff359fc1
fix(system-api): use distinct in intances filter query (#6867) 2023-11-07 06:12:16 +00:00
Livio Spring
f3b8a3aece
feat: add possibility to set an expiration to a session (#6851)
* add lifetime to session api

* extend session with lifetime

* check session token expiration

* fix typo

* integration test to check session token expiration

* integration test to check session token expiration

* i18n

* cleanup

* improve tests

* prevent negative lifetime

* fix error message

* fix lifetime check
2023-11-06 10:48:28 +01:00
Elio Bischof
1b6e3dcf27
fix: creation date argument in search events filters (#6855)
* fix: creation date filter in event queries

* fix: creation date with ordering filter

* simplify code

* simplify review

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-11-03 14:52:48 +00:00
Stefan Benz
f84eb19637
fix: change error message from metadata query User.NotFound to Metada… (#6830)
fix: change error message from metadata query User.NotFound to Metadata.NotFound

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-11-02 08:35:15 +00:00
Silvan
0187487f26
chore: correct tracing of trigger function (#6825)
* chore: correct tracing of trigger function

* refactor: remove import
2023-10-26 17:07:56 +02:00
Stefan Benz
cb7b50b513
feat: add attribute to only enable specific themes (#6798)
* feat: enable only specific themes in label policy

* feat: enable only specific themes in label policy

* feat: enable only specific themes in label policy

* feat: enable only specific themes in label policy

* add management in console

* pass enabledTheme

* render login ui based on enabled theme

* add in branding / settings service and name consistently

* update console to latest proto state

* fix console linting

* fix linting

* cleanup

* add translations

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2023-10-26 05:54:09 +00:00
Tim Möhlmann
ad26ca88d7
feat(system api): list instances by domains (#6806)
Allow to list instances by their domains on the system API.

closes #6785
2023-10-25 19:20:12 +00:00
Elio Bischof
385a55bd21
feat: limit audit trail (#6744)
* feat: enable limiting audit trail

* support AddExclusiveQuery

* fix invalid condition

* register event mappers

* fix NullDuration validity

* test query side for limits

* lint

* acceptance test audit trail limit

* fix acceptance test

* translate limits not found

* update tests

* fix linting

* add audit log retention to default instance

* fix tests

* update docs

* remove todo

* improve test name
2023-10-25 11:42:00 +00:00
adlerhurst
bd23a7a56f merge main into next 2023-10-19 12:34:00 +02:00
Silvan
b5564572bc
feat(eventstore): increase parallel write capabilities (#5940)
This implementation increases parallel write capabilities of the eventstore.
Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and  [06](https://zitadel.com/docs/support/advisory/a10006).
The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`.
If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.
2023-10-19 12:19:10 +02:00
Stefan Benz
15fd3045e0
feat: add SAML as identity provider (#6454)
* feat: first implementation for saml sp

* fix: add command side instance and org for saml provider

* fix: add query side instance and org for saml provider

* fix: request handling in event and retrieval of finished intent

* fix: add review changes and integration tests

* fix: add integration tests for saml idp

* fix: correct unit tests with review changes

* fix: add saml session unit test

* fix: add saml session unit test

* fix: add saml session unit test

* fix: changes from review

* fix: changes from review

* fix: proto build error

* fix: proto build error

* fix: proto build error

* fix: proto require metadata oneof

* fix: login with saml provider

* fix: integration test for saml assertion

* lint client.go

* fix json tag

* fix: linting

* fix import

* fix: linting

* fix saml idp query

* fix: linting

* lint: try all issues

* revert linting config

* fix: add regenerate endpoints

* fix: translations

* fix mk.yaml

* ignore acs path for user agent cookie

* fix: add AuthFromProvider test for saml

* fix: integration test for saml retrieve information

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-09-29 11:26:14 +02:00
Elio Bischof
ae1af6bc8c
fix: set quotas (#6597)
* feat: set quotas

* fix: start new period on younger anchor

* cleanup e2e config

* fix set notifications

* lint

* test: fix quota projection tests

* fix add quota tests

* make quota fields nullable

* enable amount 0

* fix initial setup

* create a prerelease

* avoid success comments

* fix quota projection primary key

* Revert "fix quota projection primary key"

This reverts commit e72f4d7fa17d03d36493912168490350a320e04f.

* simplify write model

* fix aggregate id

* avoid push without changes

* test set quota lifecycle

* test set quota mutations

* fix quota unit test

* fix: quotas

* test quota.set event projection

* use SetQuota in integration tests

* fix: release quotas 3

* reset releaserc

* fix comment

* test notification order doesn't matter

* test notification order doesn't matter

* test with unmarshalled events

* test with unmarshalled events
2023-09-22 09:37:16 +00:00
Elio Bischof
1a49b7d298
perf: project quotas and usages (#6441)
* project quota added

* project quota removed

* add periods table

* make log record generic

* accumulate usage

* query usage

* count action run seconds

* fix filter in ReportQuotaUsage

* fix existing tests

* fix logstore tests

* fix typo

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* move notifications into debouncer and improve limit querying

* cleanup

* comment

* fix: add quota unit tests command side

* fix remaining quota usage query

* implement InmemLogStorage

* cleanup and linting

* improve test

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* action notifications and fixes for notifications query

* revert console prefix

* fix: add quota unit tests command side

* fix: add quota integration tests

* improve accountable requests

* improve accountable requests

* fix: add quota integration tests

* fix: add quota integration tests

* fix: add quota integration tests

* comment

* remove ability to store logs in db and other changes requested from review

* changes requested from review

* changes requested from review

* Update internal/api/http/middleware/access_interceptor.go

Co-authored-by: Silvan <silvan.reusser@gmail.com>

* tests: fix quotas integration tests

* improve incrementUsageStatement

* linting

* fix: delete e2e tests as intergation tests cover functionality

* Update internal/api/http/middleware/access_interceptor.go

Co-authored-by: Silvan <silvan.reusser@gmail.com>

* backup

* fix conflict

* create rc

* create prerelease

* remove issue release labeling

* fix tracing

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Stefan Benz <stefan@caos.ch>
Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2023-09-15 16:58:45 +02:00
Livio Spring
62d679e553
fix: handle password data correct on user creation (#6515) 2023-09-07 14:06:11 +00:00
Miguel Cabrerizo
dd80109969
feat: delete organizations (#6083)
* feat: delete organizations

* feat: tests and delete all that depends on org

* fix: grpc delete description

* fix: get back reduce OrgRemovedEvent

* fix: add @muhlemmer review suggestions

* fix: new e2e for add/delete org
2023-09-07 04:54:51 +00:00
Livio Spring
e17b49e4ca
feat: add apple as idp (#6442)
* feat: manage apple idp

* handle apple idp callback

* add tests for provider

* basic console implementation

* implement flow for login UI and add logos / styling

* tests

* cleanup

* add upload button

* begin i18n

* apple logo positioning, file upload component

* fix add apple instance idp

* add missing apple logos for login

* update to go 1.21

* fix slice compare

* revert permission changes

* concrete error messages

* translate login apple logo -y-2px

* change form parsing

* sign in button

* fix tests

* lint console

---------

Co-authored-by: peintnermax <max@caos.ch>
2023-08-31 08:39:16 +02:00
Miguel Cabrerizo
fd00ac533a
feat: add reply-to header in email notification (#6393)
* feat: add reply-to header to smtp messages

* fix: grpc reply_to_address min 0 and js var name

* fix: add missing translations

* fix merge and linting

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-08-29 09:08:24 +02:00
Livio Spring
bb40e173bd
feat(api): add otp (sms and email) checks in session api (#6422)
* feat: add otp (sms and email) checks in session api

* implement sending

* fix tests

* add tests

* add integration tests

* fix merge main and add tests

* put default OTP Email url into config

---------

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2023-08-24 09:41:52 +00:00
Silvan
99e1c654a3
feat(storage): read only transactions for queries (#6415)
* fix: tests

* bastle wie en grosse

* fix(database): scan as callback

* fix tests

* fix merge failures

* remove as of system time

* refactor: remove unused test

* refacotr: remove unused lines
2023-08-22 10:49:22 +00:00
Livio Spring
8b44794c75
fix: delete SMTP correctly (#6391) 2023-08-18 14:22:57 +02:00
Livio Spring
7c494fd219
feat(login): add OTP (email and sms) (#6353)
* feat: login with otp

* fix(i18n): japanese translation

* add missing files

* fix provider change

* add event types translations to en

* add tests

* resourceOwner

* remove unused handler

* fix: secret generators and add comments

* add setup step

* rename

* linting

* fix setup

* improve otp handling

* fix autocomplete

* translations for login and notifications

* translations for event types

* changes from review

* check selected mfa type
2023-08-15 12:47:05 +00:00
Tim Möhlmann
0017542aa2
feat(api/v2): implement TOTP session check (#6362)
* feat(api/v2): implement TOTP session check

* add integration test

* correct typo in projection test

* fix event type typos

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-08-15 09:50:42 +00:00
Ahmed Fwela
133789fee9
feat: get multiple users by id (#6210)
* feat: introduce InTextQuery, and the ability to get multiple users by id

* added in query tests

* remove append call

* fix lints
2023-08-12 15:37:42 +02:00
Tim Möhlmann
86af67d1be
feat(api/v2): implement U2F session check (#6339) 2023-08-11 15:36:18 +00:00
Elio Bischof
343a9428b3
feat: SMS and email OTP texts (#6281)
* manage 2 custom texts proto

* implement methods

* default texts

* console

* improve translations

* lint

* test: fix e2e timeout

* fix translations

* add missing console translations

* remove unused text parts

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-08-09 05:49:12 +00:00
Livio Spring
45262e6829
fix: migrate external id of federated users (#6312)
* feat: migrate external id

* implement tests and some renaming

* fix projection

* cleanup

* i18n

* fix event type

* handle migration for new services as well

* typo
2023-08-04 11:35:36 +02:00
Livio Spring
a1942ecdaa
feat(api): add and remove OTP (SMS and email) (#6295)
* refactor: rename otp to totp

* feat: add otp sms and email

* implement tests
2023-08-02 18:57:53 +02:00
Livio Spring
789dcd8615
fix: password hash update and add missing i18n (#6285) 2023-07-28 09:09:15 +02:00
Elio Bischof
31ec1d83b9
feat: enable otp email and sms (#6260)
* feat: enable otp email and sms

* feat: enable otp factors in login settings

* remove tests without value

* translate second factors

* don't add new factors yet

* add comment

* add factors to docs

* backward compatible settings api

* compile tests

* add available 2fa types

* test: add mapping tests

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-07-28 07:39:30 +02:00
Livio Spring
cd5e176e30
fix: user grant by id (#6242) 2023-07-21 11:04:55 +00:00
Livio Spring
fed15574f6
feat: allow to force MFA local only (#6234)
This PR adds an option to the LoginPolicy to "Force MFA for local users", so that users authenticated through an IDP must not configure (and verify) an MFA.
2023-07-20 04:06:16 +00:00
Livio Spring
59f3c328ec
feat(OIDC): add support for end_session for V2 tokens (#6226)
This PR adds support for the OIDC end_session_endpoint for V2 tokens. Sending an id_token_hint as parameter will directly terminate the underlying (SSO) session and all its tokens. Without this param, the user will be redirected to the Login UI, where he will able to choose if to logout.
2023-07-19 13:17:39 +02:00
Livio Spring
e1b3cda98a
feat(OIDC): support token revocation of V2 tokens (#6203)
This PR adds support for OAuth2 token revocation of V2 tokens.

Unlike with V1 tokens, it's now possible to revoke a token not only from the authorized client / client which the token was issued to, but rather from all trusted clients (audience)
2023-07-17 14:33:37 +02:00
Livio Spring
80961125a7
feat(API): support V2 token and session token usage (#6180)
This PR adds support for userinfo and introspection of V2 tokens. Further V2 access tokens and session tokens can be used for authentication on the ZITADEL API (like the current access tokens).
2023-07-14 11:16:16 +00:00
Tim Möhlmann
4589ddad4a
feat: integrate passwap for human user password hashing (#6196)
* feat: use passwap for human user passwords

* fix tests

* passwap config

* add the event mapper

* cleanup query side and api

* solve linting errors

* regression test

* try to fix linter errors again

* pass systemdefaults into externalConfigChange migration

* fix: user password set in auth view

* pin passwap v0.2.0

* v2: validate hashed password hash based on prefix

* resolve remaining comments

* add error tag and translation for unsupported hash encoding

* fix unit test

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-07-14 09:49:57 +03:00
Livio Spring
14b8cf4894
feat(api): add OIDC session service (#6157)
This PR starts the OIDC implementation for the API V2 including the Implicit and Code Flow.


Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2023-07-10 13:27:00 +00:00
Tim Möhlmann
c0e45b63d8
fix: reset the call timestamp after a bulk trigger (#6080)
* reproduce #5808

Add an integration test that imports and gets N amount of human users.
- With N set to 1-10 the operation seems to succeed always
- With N set to 100 the operation seems to fail between 1 and 7 times.

* fix merge issue

* fix: reset the call timestamp after a bulk trigger

With the use of `AS OF SYSTEM TIME` in queries,
there was a change for the query package not
finding the latest projection verson after
a bulk trigger.
If events where processed in the bulk trigger,
the resulting row timestamp would be after the call
start timestamp.
This sometimes resulted in consistency issues when
Set and Get API methods are called in short succession.
For example a Import and Get user could sometimes result in a Not Found
error.

Although the issue was reported for the Management API user import,
it is likely this bug contributed to the flaky integration and e2e tests.

Fixes #5808

* trigger bulk action in GetSession

* don't use the new context in handler schedule

* disable reproduction test

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-07-07 08:15:05 +00:00
Elio Bischof
bb756482c7
feat: push telemetry (#6027)
* document analytics config

* rework configuration and docs

* describe HandleActiveInstances better

* describe active instances on quotas better

* only projected events are considered

* cleanup

* describe changes at runtime

* push milestones

* stop tracking events

* calculate and push 4 in 6 milestones

* reduce milestone pushed

* remove docs

* fix scheduled pseudo event projection

* push 5 in 6 milestones

* push 6 in 6 milestones

* ignore client ids

* fix text array contains

* push human readable milestone type

* statement unit tests

* improve dev and db performance

* organize imports

* cleanup

* organize imports

* test projection

* check rows.Err()

* test search query

* pass linting

* review

* test 4 milestones

* simplify milestone by instance ids query

* use type NamespacedCondition

* cleanup

* lint

* lint

* dont overwrite original error

* no opt-in in examples

* cleanup

* prerelease

* enable request headers

* make limit configurable

* review fixes

* only requeue special handlers secondly

* include integration tests

* Revert "include integration tests"

This reverts commit 96db9504ecdb4e73451f09554fd749cd7c27341f.

* pass reducers

* test handlers

* fix unit test

* feat: increment version

* lint

* remove prerelease

* fix integration tests
2023-07-06 08:38:13 +02:00
Livio Spring
bd5defa96a
fix: provide domain in session, passkey and u2f (#6097)
This fix provides a possibility to pass a domain on the session, which
will be used (as rpID) to create a passkey / u2f assertion and
attestation. This is useful in cases where the login UI is served under
a different domain / origin than the ZITADEL API.
2023-06-27 14:36:07 +02:00