Commit Graph

52 Commits

Author SHA1 Message Date
Elio Bischof
385a55bd21
feat: limit audit trail (#6744)
* feat: enable limiting audit trail

* support AddExclusiveQuery

* fix invalid condition

* register event mappers

* fix NullDuration validity

* test query side for limits

* lint

* acceptance test audit trail limit

* fix acceptance test

* translate limits not found

* update tests

* fix linting

* add audit log retention to default instance

* fix tests

* update docs

* remove todo

* improve test name
2023-10-25 11:42:00 +00:00
adlerhurst
bd23a7a56f merge main into next 2023-10-19 12:34:00 +02:00
Silvan
b5564572bc
feat(eventstore): increase parallel write capabilities (#5940)
This implementation increases parallel write capabilities of the eventstore.
Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and  [06](https://zitadel.com/docs/support/advisory/a10006).
The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`.
If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.
2023-10-19 12:19:10 +02:00
Elio Bischof
8f6cb47567
fix: use triggering origin for notification links (#6628)
* take baseurl if saved on event

* refactor: make es mocks reusable

* Revert "refactor: make es mocks reusable"

This reverts commit 434ce12a6a.

* make messages testable

* test asset url

* fmt

* fmt

* simplify notification.Start

* test url combinations

* support init code added

* support password changed

* support reset pw

* support user domain claimed

* support add pwless login

* support verify phone

* Revert "support verify phone"

This reverts commit e40503303e.

* save trigger origin from ctx

* add ready for review check

* camel

* test email otp

* fix variable naming

* fix DefaultOTPEmailURLV2

* Revert "fix DefaultOTPEmailURLV2"

This reverts commit fa34d4d2a8.

* fix email otp challenged test

* fix email otp challenged test

* pass origin in login and gateway requests

* take origin from header

* take x-forwarded if present

* Update internal/notification/handlers/queries.go

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>

* Update internal/notification/handlers/commands.go

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>

* move origin header to ctx if available

* generate

* cleanup

* use forwarded header

* support X-Forwarded-* headers

* standardize context handling

* fix linting

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2023-10-10 13:20:53 +00:00
Stefan Benz
15fd3045e0
feat: add SAML as identity provider (#6454)
* feat: first implementation for saml sp

* fix: add command side instance and org for saml provider

* fix: add query side instance and org for saml provider

* fix: request handling in event and retrieval of finished intent

* fix: add review changes and integration tests

* fix: add integration tests for saml idp

* fix: correct unit tests with review changes

* fix: add saml session unit test

* fix: add saml session unit test

* fix: add saml session unit test

* fix: changes from review

* fix: changes from review

* fix: proto build error

* fix: proto build error

* fix: proto build error

* fix: proto require metadata oneof

* fix: login with saml provider

* fix: integration test for saml assertion

* lint client.go

* fix json tag

* fix: linting

* fix import

* fix: linting

* fix saml idp query

* fix: linting

* lint: try all issues

* revert linting config

* fix: add regenerate endpoints

* fix: translations

* fix mk.yaml

* ignore acs path for user agent cookie

* fix: add AuthFromProvider test for saml

* fix: integration test for saml retrieve information

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-09-29 11:26:14 +02:00
Livio Spring
68bfab2fb3
feat(login): use default org for login without provided org context (#6625)
* start feature flags

* base feature events on domain const

* setup default features

* allow setting feature in system api

* allow setting feature in admin api

* set settings in login based on feature

* fix rebasing

* unit tests

* i18n

* update policy after domain discovery

* some changes from review

* check feature and value type

* check feature and value type
2023-09-29 08:21:32 +00:00
Elio Bischof
1a49b7d298
perf: project quotas and usages (#6441)
* project quota added

* project quota removed

* add periods table

* make log record generic

* accumulate usage

* query usage

* count action run seconds

* fix filter in ReportQuotaUsage

* fix existing tests

* fix logstore tests

* fix typo

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* move notifications into debouncer and improve limit querying

* cleanup

* comment

* fix: add quota unit tests command side

* fix remaining quota usage query

* implement InmemLogStorage

* cleanup and linting

* improve test

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* fix: add quota unit tests command side

* action notifications and fixes for notifications query

* revert console prefix

* fix: add quota unit tests command side

* fix: add quota integration tests

* improve accountable requests

* improve accountable requests

* fix: add quota integration tests

* fix: add quota integration tests

* fix: add quota integration tests

* comment

* remove ability to store logs in db and other changes requested from review

* changes requested from review

* changes requested from review

* Update internal/api/http/middleware/access_interceptor.go

Co-authored-by: Silvan <silvan.reusser@gmail.com>

* tests: fix quotas integration tests

* improve incrementUsageStatement

* linting

* fix: delete e2e tests as intergation tests cover functionality

* Update internal/api/http/middleware/access_interceptor.go

Co-authored-by: Silvan <silvan.reusser@gmail.com>

* backup

* fix conflict

* create rc

* create prerelease

* remove issue release labeling

* fix tracing

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Stefan Benz <stefan@caos.ch>
Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2023-09-15 16:58:45 +02:00
Livio Spring
e17b49e4ca
feat: add apple as idp (#6442)
* feat: manage apple idp

* handle apple idp callback

* add tests for provider

* basic console implementation

* implement flow for login UI and add logos / styling

* tests

* cleanup

* add upload button

* begin i18n

* apple logo positioning, file upload component

* fix add apple instance idp

* add missing apple logos for login

* update to go 1.21

* fix slice compare

* revert permission changes

* concrete error messages

* translate login apple logo -y-2px

* change form parsing

* sign in button

* fix tests

* lint console

---------

Co-authored-by: peintnermax <max@caos.ch>
2023-08-31 08:39:16 +02:00
Livio Spring
bb40e173bd
feat(api): add otp (sms and email) checks in session api (#6422)
* feat: add otp (sms and email) checks in session api

* implement sending

* fix tests

* add tests

* add integration tests

* fix merge main and add tests

* put default OTP Email url into config

---------

Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2023-08-24 09:41:52 +00:00
Livio Spring
cbd2ef0612
fix: use system secret config if generator type does not exist on instance (#6420)
* fix: use system secret config if generator type does not exist on instance

* remove unused idGenerator
2023-08-23 08:04:29 +00:00
Silvan
99e1c654a3
feat(storage): read only transactions for queries (#6415)
* fix: tests

* bastle wie en grosse

* fix(database): scan as callback

* fix tests

* fix merge failures

* remove as of system time

* refactor: remove unused test

* refacotr: remove unused lines
2023-08-22 10:49:22 +00:00
Livio Spring
372755bddd
feat(api): add organisation service (#6340)
* setup org with multiple admins

* tests

* add missing proto

* remove machine users (for now)

* update tests with idp case

* fix package

* organisation -> organization

* fix test
2023-08-11 14:19:14 +00:00
Elio Bischof
35f4f74b08
fix: remove logstore beta warning (#6244)
Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-07-25 08:44:47 +02:00
Elio Bischof
57d6113e2a
fix: upper ZITADEL banner (#6246)
Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-07-25 05:12:13 +00:00
Miguel Cabrerizo
0f3c33cb86
feat: show basic info start-from-init (#6183) 2023-07-12 20:46:50 +03:00
Livio Spring
14b8cf4894
feat(api): add OIDC session service (#6157)
This PR starts the OIDC implementation for the API V2 including the Implicit and Code Flow.


Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2023-07-10 13:27:00 +00:00
Elio Bischof
9b768003b7
feat: improve milestone format (#6150)
* feat: milestone format

* feat: push external domain

* cleanup

* Revert "remove prerelease"

This reverts commit 7417fdbeb3.

* fix branch

* remove prerelease
2023-07-06 19:31:08 +02:00
Elio Bischof
bb756482c7
feat: push telemetry (#6027)
* document analytics config

* rework configuration and docs

* describe HandleActiveInstances better

* describe active instances on quotas better

* only projected events are considered

* cleanup

* describe changes at runtime

* push milestones

* stop tracking events

* calculate and push 4 in 6 milestones

* reduce milestone pushed

* remove docs

* fix scheduled pseudo event projection

* push 5 in 6 milestones

* push 6 in 6 milestones

* ignore client ids

* fix text array contains

* push human readable milestone type

* statement unit tests

* improve dev and db performance

* organize imports

* cleanup

* organize imports

* test projection

* check rows.Err()

* test search query

* pass linting

* review

* test 4 milestones

* simplify milestone by instance ids query

* use type NamespacedCondition

* cleanup

* lint

* lint

* dont overwrite original error

* no opt-in in examples

* cleanup

* prerelease

* enable request headers

* make limit configurable

* review fixes

* only requeue special handlers secondly

* include integration tests

* Revert "include integration tests"

This reverts commit 96db9504ec.

* pass reducers

* test handlers

* fix unit test

* feat: increment version

* lint

* remove prerelease

* fix integration tests
2023-07-06 08:38:13 +02:00
Livio Spring
7046194530
feat(api): list authentication method types in user api v2 (#6058) 2023-06-20 16:23:28 +00:00
Stefan Benz
fa8f191812
feat: v2alpha user service idp endpoints (#5879)
* feat: v2alpha user service idp endpoints

* feat: v2alpha user service intent endpoints

* begin idp intents (callback)

* some cleanup

* runnable idp authentication

* cleanup

* proto cleanup

* retrieve idp info

* improve success and failure handling

* some unit tests

* grpc unit tests

* add permission check AddUserIDPLink

* feat: v2alpha intent writemodel refactoring

* feat: v2alpha intent writemodel refactoring

* feat: v2alpha intent writemodel refactoring

* provider from write model

* fix idp type model and add integration tests

* proto cleanup

* fix integration test

* add missing import

* add more integration tests

* auth url test

* feat: v2alpha intent writemodel refactoring

* remove unused functions

* check token on RetrieveIdentityProviderInformation

* feat: v2alpha intent writemodel refactoring

* fix TestServer_RetrieveIdentityProviderInformation

* fix test

* i18n and linting

* feat: v2alpha intent review changes

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2023-05-24 18:29:58 +00:00
Elio Bischof
0e251a29c8
fix: set exhausted cookie with env json (#5868)
* fix: set exhausted cookie with env json

* lint
2023-05-15 08:51:02 +02:00
Stefan Benz
8d13f170e8
feat(api): new settings service (#5775)
* feat: add v2alpha policies service

* feat: add v2alpha policies service

* fix: rename of attributes and messages in v2alpha api

* fix: rename of attributes and messages in v2alpha api

* fix: linter corrections

* fix: review corrections

* fix: review corrections

* fix: review corrections

* fix: review corrections

* fix grpc

* refactor: rename to settings and more

* Apply suggestions from code review

Co-authored-by: Fabi <fabienne.gerschwiler@gmail.com>

* add service to docs and rename legal settings

* unit tests for converters

* go mod tidy

* ensure idp name and return list details

* fix: use correct resource owner for active idps

* change query to join

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Fabi <fabienne.gerschwiler@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2023-05-11 09:23:40 +00:00
Livio Spring
c07411e314
fix: only reuse port for integration tests (#5817)
* fix: only reuse port for integration tests

* exclude default listenConfig from integration build
2023-05-11 08:58:35 +00:00
Elio Bischof
35a0977663
fix: improve exhausted SetCookie header (#5789)
* fix: remove access interceptor for console

* feat: template quota cookie value

* fix: send exhausted cookie from grpc-gateway

* refactor: remove ineffectual err assignments

* Update internal/api/grpc/server/gateway.go

Co-authored-by: Livio Spring <livio.a@gmail.com>

* use dynamic host header to find instance

* add instance mgmt url to environment.json

* support hosts with default ports

* fix linting

* docs: update lb example

* print access logs to stdout

* fix grpc gateway exhausted cookies

* cleanup

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-05-11 09:24:44 +02:00
Livio Spring
c2cb84cd24
feat(api): new session service (#5801)
* backup new protoc plugin

* backup

* session

* backup

* initial implementation

* change to specific events

* implement tests

* cleanup

* refactor: use new protoc plugin for api v2

* change package

* simplify code

* cleanup

* cleanup

* fix merge

* start queries

* fix tests

* improve returned values

* add token to projection

* tests

* test db map

* update query

* permission checks

* fix tests and linting

* rework token creation

* i18n

* refactor token check and fix tests

* session to PB test

* request to query tests

* cleanup proto

* test user check

* add comment

* simplify database map type

* Update docs/docs/guides/integrate/access-zitadel-system-api.md

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>

* fix test

* cleanup

* docs

---------

Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2023-05-05 15:34:53 +00:00
Miguel Cabrerizo
3ca7147808
fix: introduce measures to avoid bots crawling and indexing activities (#5728)
* fix: 404 for robots.txt and meta robots tags

* fix: add unit tests for robots txt and tag

* fix: add meta tag robots none for login pages

* fix: weird format issue in header.go

* fix: add x-robots-tag=none to grpcwebserver

* fix linting

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-05-05 10:25:02 +02:00
Tim Möhlmann
4934d6f4fa Merge branch 'main' into integration-tests 2023-05-02 19:24:37 +03:00
Tim Möhlmann
c839cb3ce0 tie loose ends, documentation 2023-05-02 19:24:24 +03:00
Silvan
39bdef35e7
chore: merge (#5773)
* feat: allow skip of success page for native apps (#5627)

add possibility to return to callback directly after login without rendering the successful login page

* build next

* fix(console): disallow inline fonts, critical styles (#5714)

fix: disallow inline

* fix(setup): step 10 for postgres (#5717)

* fix(setup): smaller transactions (#5743)

* fix: order by sequence by default

* test: add allowCreationDateFilter

* fix(step10): separate executions (#5754)

* feat: allow skip of success page for native apps (#5627)

add possibility to return to callback directly after login without rendering the successful login page

* build next

* fix(console): disallow inline fonts, critical styles (#5714)

fix: disallow inline

* fix(setup): step 10 for postgres (#5717)

* fix(setup): smaller transactions (#5743)

* fix(step10): split statements

* fix(step10): split into separate execs

* chore: prerelease

* add truncate before insert

* fix: add truncate

* Merge branch 'main' into optimise-step-10

* chore: reset release definition

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Max Peintner <max@caos.ch>

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Max Peintner <max@caos.ch>
2023-04-28 14:56:51 +00:00
Tim Möhlmann
11ab645bb7 Merge branch 'main' into integration-tests 2023-04-27 12:47:35 +03:00
Tim Möhlmann
90ba3a8d92 poll on test start 2023-04-26 19:54:47 +03:00
Silvan
095ec21678
feat: user v2alpha email API (#5708)
* chore(proto): update versions

* change protoc plugin

* some cleanups

* define api for setting emails in new api

* implement user.SetEmail

* move SetEmail buisiness logic into command

* resuse newCryptoCode

* command: add ChangeEmail unit tests

Not complete, was not able to mock the generator.

* Revert "resuse newCryptoCode"

This reverts commit c89e90ae35.

* undo change to crypto code generators

* command: use a generator so we can test properly

* command: reorganise ChangeEmail

improve test coverage

* implement VerifyEmail

including unit tests

* add URL template tests

* proto: change context to object

* remove old auth option

* remove old auth option

* fix linting errors

run gci on modified files

* add permission checks and fix some errors

* comments

* comments

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Tim Möhlmann <tim+github@zitadel.com>
2023-04-25 09:02:29 +02:00
Tim Möhlmann
e167365548 chore: test server for direct resource access 2023-04-24 20:40:31 +03:00
Tim Möhlmann
5819924275
feat: device authorization RFC 8628 (#5646)
* device auth: implement the write events

* add grant type device code

* fix(init): check if default value implements stringer

---------

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2023-04-19 08:46:02 +00:00
Livio Spring
b3d8787921
feat: add new api services (#5619)
* feat: add new services

* improve demos and comments

* remove unused field

* add comment to demo proto calls

* Apply suggestions from code review

Co-authored-by: Silvan <silvan.reusser@gmail.com>

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2023-04-11 15:37:42 +02:00
Stefan Benz
440ba9f5ef
fix: update saml to v0.0.11 (#5628)
* fix: update saml to v0.0.11

* chore: remove unused sum

---------

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2023-04-11 07:39:12 +00:00
Elio Bischof
cccccd005c
feat: call webhooks at least once (#5454)
* feat: call webhooks at least once

* self review

* feat: improve notification observability

* feat: add notification tracing

* test(e2e): test at-least-once webhook delivery

* fix webhook notifications

* dedicated quota notifications handler

* fix linting

* fix e2e test

* wait less in e2e test

* fix: don't ignore failed events in handlers

* fix: don't ignore failed events in handlers

* faster requeues

* question

* fix retries

* fix retries

* retry

* don't instance ids query

* revert handler_projection

* statements can be nil

* cleanup

* make unit tests pass

* add comments

* add comments

* lint

* spool only active instances

* feat(config): handle inactive instances

* customizable HandleInactiveInstances

* call inactive instances quota webhooks

* test: handling with and w/o inactive instances

* omit retrying noop statements

* docs: describe projection options

* enable global handling of inactive instances

* self review

* requeue quota notifications every 5m

* remove caos_errors reference

* fix comment styles

* make handlers package flat

* fix linting

* fix repeating quota notifications

* test with more usage

* debug log channel init failures
2023-03-28 22:09:06 +00:00
Silvan
a3b36a0138
refactor(changes): use queries.SearchEvents (#5388)
* refactor(changes): use `queries.SearchEvents`

---------

Co-authored-by: Livio Spring <livio.a@gmail.com>
2023-03-17 09:14:06 +00:00
Livio Spring
8cbde57047
fix(api): allow HTTP/1.1 for grpc-web (#5376)
Handles grpc-web on HTTP/1.1 with H2C for HTTP/2, but does not enforce it.
2023-03-08 18:33:43 +00:00
Silvan
e38abdcdf3
perf: query data AS OF SYSTEM TIME (#5231)
Queries the data in the storage layser at the timestamp when the call hit the API layer
2023-02-27 22:36:43 +01:00
Elio Bischof
681541f41b
feat: add quotas (#4779)
adds possibilities to cap authenticated requests and execution seconds of actions on a defined intervall
2023-02-15 02:52:11 +01:00
Livio Spring
d21bb902f1
fix: push timeout (#4882) (#4885)
* push with timeout

* test: config for eventstore

(cherry picked from commit b9156da76d)

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2022-12-15 09:40:13 +00:00
Livio Spring
e8babf1048
fix: reduce load on view tables (#4716)
* fix: reduce load on view tables

* create prerelease

* linting: pass context to view handlers

* fix error handling of refresh token handler

* fix: improve processing of successful instanceIDs on views

* fix revert intended change in .golangci.yaml

* fix: set timeout for processInstances in spooler

* fix: reduce update to active tokens on profile change

* change token expiration query to db now()

* remove branch from .releaserc.js
2022-11-22 07:36:48 +01:00
Livio Spring
7dfa1925cc
feat: restrict login to specific org by id (scope) (#4294)
* feat: add new org scope

* change default of UserLoginMustBeDomain to false

* return resource owner claims

* fix: use email style for first user

* fix: ensure email style for default users (backwards compatibility)

* change to external domain (as it was before UserLoginMustBeDomain change)

* update e2e tests to use email style usernames

* document new scope

* lint e2e

Co-authored-by: Fabi <38692350+hifabienne@users.noreply.github.com>
2022-09-23 12:08:10 +00:00
Livio Spring
7aef0ccfee
fix(email): set correct logo url (#4426) 2022-09-21 14:18:55 +00:00
Stefan Benz
7a5f7f82cf
feat(saml): implementation of saml for ZITADEL v2 (#3618) 2022-09-12 18:18:08 +02:00
Livio Spring
69534a2f7a
feat: allow JWT for ZITADEL APIs (#4206)
* feat: allow JWT for ZITADEL APIs

* improve getTokenIDAndSubject

* comment

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2022-08-23 08:02:36 +02:00
Livio Spring
dcac08b1d5
fix: caching of assets (correct headers and versioned avatar and variables.css url) (#4118)
* fix: caching of assets (correct headers and versioned avatar url)

* serve variables.css versioned and extend shared max age of assets

* fix TestCommandSide_AddHumanAvatar

* refactor: const types

* refactor: return values

Co-authored-by: Fabi <38692350+hifabienne@users.noreply.github.com>
Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2022-08-16 05:04:36 +00:00
Livio Spring
f610d48569
feat: prepare for multiple database types (#4068)
BREAKING CHANGE: the database and admin user config has changed.
2022-07-28 16:25:42 +02:00
Stefan Benz
bc9a85daf3
feat: V2 alpha import and export of organizations (#3798)
* feat(import): add functionality to import data into an instance

* feat(import): move import to admin api and additional checks for nil pointer

* fix(export): export implementation with filtered members and grants

* fix: export and import implementation

* fix: add possibility to export hashed passwords with the user

* fix(import): import with structure of v1 and v2

* docs: add v1 proto

* fix(import): check im imported user is already existing

* fix(import): add otp import function

* fix(import): add external idps, domains, custom text and messages

* fix(import): correct usage of default values from login policy

* fix(export): fix renaming of add project function

* fix(import): move checks for unit tests

* expect filter

* fix(import): move checks for unit tests

* fix(import): move checks for unit tests

* fix(import): produce prerelease from branch

* fix(import): correctly use provided user id for machine user imports

* fix(import): corrected otp import and added guide for export and import

* fix: import verified and primary domains

* fix(import): add reading from gcs, s3 and localfile with tracing

* fix(import): gcs and s3, file size correction and error logging

* Delete docker-compose.yml

* fix(import): progress logging and count of resources

* fix(import): progress logging and count of resources

* log subscription

* fix(import): incorporate review

* fix(import): incorporate review

* docs: add suggestion for import

Co-authored-by: Fabi <38692350+hifabienne@users.noreply.github.com>

* fix(import): add verification otp event and handling of deleted but existing users

Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Fabienne <fabienne.gerschwiler@gmail.com>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: Fabi <38692350+hifabienne@users.noreply.github.com>
2022-07-28 13:42:35 +00:00