Commit Graph

12 Commits

Author SHA1 Message Date
Silvan
1d84635836
feat(eventstore): add search table (#8191)
# Which Problems Are Solved

To improve performance a new table and method is implemented on
eventstore. The goal of this table is to index searchable fields on
command side to use it on command and query side.

The table allows to store one primitive value (numeric, text) per row.

The eventstore framework is extended by the `Search`-method which allows
to search for objects.
The `Command`-interface is extended by the `SearchOperations()`-method
which does manipulate the the `search`-table.

# How the Problems Are Solved

This PR adds the capability of improving performance for command and
query side by using the `Search`-method of the eventstore instead of
using one of the `Filter`-methods.

# Open Tasks

- [x] Add feature flag
- [x] Unit tests
- [ ] ~~Benchmarks if needed~~
- [x] Ensure no behavior change
- [x] Add setup step to fill table with current data
- [x] Add projection which ensures data added between setup and start of
the new version are also added to the table

# Additional Changes

The `Search`-method is currently used by `ProjectGrant`-command side.

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/8094
2024-07-03 15:00:56 +00:00
Tim Möhlmann
f680dd934d
refactor: rename package errors to zerrors (#7039)
* chore: rename package errors to zerrors

* rename package errors to gerrors

* fix error related linting issues

* fix zitadel error assertion

* fix gosimple linting issues

* fix deprecated linting issues

* resolve gci linting issues

* fix import structure

---------

Co-authored-by: Elio Bischof <elio@zitadel.com>
2023-12-08 15:30:55 +01:00
Silvan
b5564572bc
feat(eventstore): increase parallel write capabilities (#5940)
This implementation increases parallel write capabilities of the eventstore.
Please have a look at the technical advisories: [05](https://zitadel.com/docs/support/advisory/a10005) and  [06](https://zitadel.com/docs/support/advisory/a10006).
The implementation of eventstore.push is rewritten and stored events are migrated to a new table `eventstore.events2`.
If you are using cockroach: make sure that the database user of ZITADEL has `VIEWACTIVITY` grant. This is used to query events.
2023-10-19 12:19:10 +02:00
Silvan
f3e6f3b23b
feat: remove org (#4148)
* feat(command): remove org

* refactor: imports, unused code, error handling

* reduce org removed in action

* add org deletion to projections

* add org removal to projections

* add org removal to projections

* org removed projection

* lint import

* projections

* fix: table names in tests

* fix: table names in tests

* logging

* add org state

* fix(domain): add Owner removed to object details

* feat(ListQuery): add with owner removed

* fix(org-delete): add bool to functions to select with owner removed

* fix(org-delete): add bools to user grants with events to determine if dependencies lost owner

* fix(org-delete): add unit tests for owner removed and org removed events

* fix(org-delete): add handling of org remove for grants and members

* fix(org-delete): correction of unit tests for owner removed

* fix(org-delete): update projections, unit tests and get functions

* fix(org-delete): add change date to authnkeys and owner removed to org metadata

* fix(org-delete): include owner removed for login names

* fix(org-delete): some column fixes in projections and build for queries with owner removed

* indexes

* fix(org-delete): include review changes

* fix(org-delete): change user projection name after merge

* fix(org-delete): include review changes for project grant where no project owner is necessary

* fix(org-delete): include auth and adminapi tables with owner removed information

* fix(org-delete): cleanup username and orgdomain uniqueconstraints when org is removed

* fix(org-delete): add permissions for org.remove

* remove unnecessary unique constraints

* fix column order in primary keys

* fix(org-delete): include review changes

* fix(org-delete): add owner removed indexes and chang setup step to create tables

* fix(org-delete): move PK order of instance_id and change added user_grant from review

* fix(org-delete): no params for prepareUserQuery

* change to step 6

* merge main

* fix(org-delete): OldUserName rename to private

* fix linting

* cleanup

* fix: remove org test

* create prerelease

* chore: delete org-delete as prerelease

Co-authored-by: Stefan Benz <stefan@caos.ch>
Co-authored-by: Livio Spring <livio.a@gmail.com>
Co-authored-by: Fabi <38692350+hifabienne@users.noreply.github.com>
Co-authored-by: Stefan Benz <46600784+stebenz@users.noreply.github.com>
2022-11-30 17:01:17 +01:00
Stefan Benz
2c1f9ac4a8
feat(org): add org metadata functionality (#4234)
* feat(org): add org metadata functionality

* fix(metadata): add unit tests and review for org metadata

* fix(org-metadata): move endpoints to /

Co-authored-by: Fabi <38692350+hifabienne@users.noreply.github.com>
2022-09-20 14:32:09 +00:00
Florian Forster
fa9f581d56
chore(v2): move to new org (#3499)
* chore: move to new org

* logging

* fix: org rename caos -> zitadel

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
2022-04-26 23:01:45 +00:00
Silvan
09be70949f
refactor(eventstore): rename EventPusher to Command, EventReader to Event, PushEvents to Push and FilterEvents to Filter (#2907) 2022-01-03 09:19:07 +01:00
Silvan
39c35c9455
feat(queries): use org projection (#2342)
* job queue

* wg improvements

* start handler

* statement

* statements

* imporve handler

* improve statement

* statement in seperate file

* move handlers

* move query/old to query

* handler

* read models

* bulk works

* cleanup

* contrib

* rename readmodel to projection

* rename read_models schema to projections

* rename read_models schema to projections

* search query as func,
bulk iterates as long as new events

* add event sequence less query

* update checks for events between current sequence and sequence of first statement if it has previous sequence 0

* cleanup crdb projection

* refactor projection handler

* start with testing

* tests for handler

* remove todo

* refactor statement: remove table name,
add tests

* improve projection handler shutdown,
no savepoint if noop stmt,
tests for stmt handler

* tests

* start failed events

* seperate branch for contrib

* move statement constructors to crdb pkg

* correct import

* Subscribe for eventtypes (#1800)

* fix: is default (#1737)

* fix: use email as username on global org (#1738)

* fix: use email as username on global org

* Update user_human.go

* Update register_handler.go

* chore(deps): update docusaurus (#1739)

* chore: remove PAT and use GH Token (#1716)

* chore: remove PAT and use GH Token

* fix env

* fix env

* fix env

* md lint

* trigger ci

* change user

* fix GH bug

* replace login part

* chore: add GH Token to sem rel (#1746)

* chore: add GH Token to sem rel

* try branch

* add GH Token

* remove test branch again

* docs: changes acme to acme-caos (#1744)

* changes acme to acme-caos

* Apply suggestions from code review

Co-authored-by: Florian Forster <florian@caos.ch>

Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com>
Co-authored-by: Florian Forster <florian@caos.ch>

* feat: add additional origins on applications (#1691)

* feat: add additional origins on applications

* app additional redirects

* chore(deps-dev): bump @angular/cli from 11.2.8 to 11.2.11 in /console (#1706)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps-dev): bump @angular/cli from 11.2.8 to 11.2.11 in /console

Bumps [@angular/cli](https://github.com/angular/angular-cli) from 11.2.8 to 11.2.11.
- [Release notes](https://github.com/angular/angular-cli/releases)
- [Commits](https://github.com/angular/angular-cli/compare/v11.2.8...v11.2.11)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump stylelint from 13.10.0 to 13.13.1 in /console (#1703)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps-dev): bump stylelint from 13.10.0 to 13.13.1 in /console

Bumps [stylelint](https://github.com/stylelint/stylelint) from 13.10.0 to 13.13.1.
- [Release notes](https://github.com/stylelint/stylelint/releases)
- [Changelog](https://github.com/stylelint/stylelint/blob/master/CHANGELOG.md)
- [Commits](https://github.com/stylelint/stylelint/compare/13.10.0...13.13.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump @types/node from 14.14.37 to 15.0.1 in /console (#1702)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps-dev): bump @types/node from 14.14.37 to 15.0.1 in /console

Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 14.14.37 to 15.0.1.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump ts-protoc-gen from 0.14.0 to 0.15.0 in /console (#1701)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps): bump ts-protoc-gen from 0.14.0 to 0.15.0 in /console

Bumps [ts-protoc-gen](https://github.com/improbable-eng/ts-protoc-gen) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/improbable-eng/ts-protoc-gen/releases)
- [Changelog](https://github.com/improbable-eng/ts-protoc-gen/blob/master/CHANGELOG.md)
- [Commits](https://github.com/improbable-eng/ts-protoc-gen/compare/0.14.0...0.15.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump @types/jasmine from 3.6.9 to 3.6.10 in /console (#1682)

Bumps [@types/jasmine](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jasmine) from 3.6.9 to 3.6.10.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jasmine)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump @types/google-protobuf in /console (#1681)

Bumps [@types/google-protobuf](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/google-protobuf) from 3.7.4 to 3.15.2.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/google-protobuf)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump grpc from 1.24.5 to 1.24.7 in /console (#1666)

Bumps [grpc](https://github.com/grpc/grpc-node) from 1.24.5 to 1.24.7.
- [Release notes](https://github.com/grpc/grpc-node/releases)
- [Commits](https://github.com/grpc/grpc-node/compare/grpc@1.24.5...grpc@1.24.7)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* lock

* chore(deps-dev): bump @angular/language-service from 11.2.9 to 11.2.12 in /console (#1704)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps-dev): bump @angular/language-service in /console

Bumps [@angular/language-service](https://github.com/angular/angular/tree/HEAD/packages/language-service) from 11.2.9 to 11.2.12.
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/master/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/commits/11.2.12/packages/language-service)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* package lock

* downgrade grpc

* downgrade protobuf types

* revert npm packs 🥸

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Silvan <silvan.reusser@gmail.com>

* docs: update run and start section texts (#1745)

* update run and start section texts

* adds showcase

Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com>

* fix: additional origin list (#1753)

* fix: handle api configs in authz handler (#1755)

* fix(console): add model for api keys, fix toast, binding (#1757)

* fix: add model for api keys, fix toast, binding

* show api clientid

* fix: missing patchvalue (#1758)

* feat: refresh token (#1728)

* begin refresh tokens

* refresh tokens

* list and revoke refresh tokens

* handle remove

* tests for refresh tokens

* uniqueness and default expiration

* rename oidc token methods

* cleanup

* migration version

* Update internal/static/i18n/en.yaml

Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>

* fixes

* feat: update oidc pkg for refresh tokens

Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>

* fix: correct json name of clientId in key.json (#1760)

* fix: migration version (#1767)

* start subscription

* eventtypes

* fix(login): links (#1778)

* fix(login): href for help

* fix(login): correct link to tos

* fix: access tokens for service users and refresh token infos (#1779)

* fix: access token for service user

* handle info from refresh request

* uniqueness

* postpone access token uniqueness change

* chore(coc): recommend code of conduct (#1782)

* subscribe for events

* feat(console): refresh toggle out of granttype context (#1785)

* refresh toggle

* disable if not code flow, lint

* lint

* fix: change oidc config order

* accept refresh option within flow

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* fix: refresh token activation (#1795)

* fix: oidc grant type check

* docs: add offline_access scope

* docs: update refresh token status in supported grant types

* fix: update oidc pkg

* fix: check refresh token grant type (#1796)

* configuration structs

* org admins

* failed events

* fixes

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Florian Forster <florian@caos.ch>
Co-authored-by: mffap <mpa@caos.ch>
Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>

* remove comment

* aggregate reducer

* remove eventtypes

* add protoc-get-validate to mod

* fix transaltion

* upsert

* add gender on org admins,
allow to retry failed stmts after configurable time

* remove if

* sub queries

* fix: tests

* add builder to tests

* new search query

* rename searchquerybuilder to builder

* remove comment from code

* test with multiple queries

* add filters test

* current sequences

* make org and org_admins work again

* add aggregate type to current sequence

* fix(contibute): listing

* add validate module

* fix: search queries

* feat(eventstore): previous aggregate root sequence (#1810)

* feat(eventstore): previous aggregate root sequence

* fix tests

* fix: eventstore v1 test

* add col to all mocked rows

* next try

* fix mig

* rename aggregate root to aggregate type

* update comment

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* small refactorings

* allow update multiple current sequences

* unique log id

* fix migrations

* rename org admin to org owner

* improve error handling and logging

* fix(migration): optimize prev agg root seq

* fix: projection handler test

* fix: sub queries

* small fixes

* additional event types

* correct org owner projection

* fix primary key

* feat(eventstore): jobs for projections (#2026)

* fix: template names in login (#1974)

* fix: template names in login

* fix: error.html

* fix: check for features on mgmt only (#1976)

* fix: add sentry in ui, http and projection handlers (#1977)

* fix: add sentry in ui, http and projection handlers

* fix test

* fix(eventstore): sub queries (#1805)

* sub queries

* fix: tests

* add builder to tests

* new search query

* rename searchquerybuilder to builder

* remove comment from code

* test with multiple queries

* add filters test

* fix(contibute): listing

* add validate module

* fix: search queries

* remove unused event type in query

* ignore query if error in marshal

* go mod tidy

* update privacy policy query

* update queries

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* feat: Extend oidc idp with oauth endpoints (#1980)

* feat: add oauth attributes to oidc idp configuration

* feat: return idpconfig id on create idp

* feat: tests

* feat: descriptions

* feat: docs

* feat: tests

* docs: update to beta 3 (#1984)

* fix: role assertion (#1986)

* fix: enum to display access token role assertion

* improve assertion descriptions

* fix nil pointer

* docs: eventstore (#1982)

* docs: eventstore

* Apply suggestions from code review

Co-authored-by: Florian Forster <florian@caos.ch>

Co-authored-by: Florian Forster <florian@caos.ch>

* fix(sentry): trigger sentry release (#1989)

* feat(send sentry release): send sentry release

* fix(moved step and added releasetag): moved step and added releasetag

* fix: set version for sentry release (#1990)

* feat(send sentry release): send sentry release

* fix(moved step and added releasetag): moved step and added releasetag

* fix(corrected var name): corrected var name

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* fix: log error reason on terminate session (#1973)

* fix: return default language file, if requested lang does not exist for default login texts (#1988)

* fix: return default language file, if requested lang doesnt exists

* feat: read default translation file

* feat: docs

* fix: race condition in auth request unmarshalling (#1993)

* feat: handle ui_locales in login (#1994)

* fix: handle ui_locales in login

* move supportedlanguage func into i18n package

* update oidc pkg

* fix: handle closed channels on unsubscribe (#1995)

* fix: give restore more time (#1997)

* fix: translation file read (#2009)

* feat: translation file read

* feat: readme

* fix: enable idp add button for iam users (#2010)

* fix: filter event_data (#2011)

* feat: Custom message files (#1992)

* feat: add get custom message text to admin api

* feat: read custom message texts from files

* feat: get languages in apis

* feat: get languages in apis

* feat: get languages in apis

* feat: pr feedback

* feat: docs

* feat: merge main

* fix: sms notification (#2013)

* fix: phone verifications

* feat: fix password reset as sms

* fix: phone verification

* fix: grpc status in sentry and validation interceptors (#2012)

* fix: remove oauth endpoints from oidc config proto (#2014)

* try with view

* fix(console): disable sw (#2021)

* fix: disable sw

* angular.json disable sw

* project projections

* fix typos

* customize projections

* customizable projections,
add change date to projects

Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>
Co-authored-by: Florian Forster <florian@caos.ch>
Co-authored-by: mffap <mpa@caos.ch>
Co-authored-by: Christian Jakob <47860090+thesephirot@users.noreply.github.com>
Co-authored-by: Elio Bischof <eliobischof@gmail.com>

* env file

* typo

* correct users

* correct migration

* fix: merge fail

* fix test

* fix(tests): unordered matcher

* improve currentSequenceMatcher

* correct certs

* correct certs

* add zitadel database on database list

* refctor switch in match

* enable all handlers

* Delete io.env

* cleanup

* add handlers

* rename view to projection

* rename view to projection

* fix type typo

* remove unnecessary logs

* refactor stmts

* simplify interval calculation

* fix tests

* fix unlock test

* fix migration

* migs

* fix(operator): update cockroach and flyway versions (#2138)

* chore(deps): bump k8s.io/apiextensions-apiserver from 0.19.2 to 0.21.3

Bumps [k8s.io/apiextensions-apiserver](https://github.com/kubernetes/apiextensions-apiserver) from 0.19.2 to 0.21.3.
- [Release notes](https://github.com/kubernetes/apiextensions-apiserver/releases)
- [Commits](https://github.com/kubernetes/apiextensions-apiserver/compare/v0.19.2...v0.21.3)

---
updated-dependencies:
- dependency-name: k8s.io/apiextensions-apiserver
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore(deps): bump google.golang.org/api from 0.34.0 to 0.52.0

Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.34.0 to 0.52.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.34.0...v0.52.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* start update dependencies

* update mods and otlp

* fix(build): update to go 1.16

* old version for k8s mods

* update k8s versions

* update orbos

* fix(operator): update cockroach and flyway version

* Update images.go

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Stefan Benz <stefan@caos.ch>

* fix import

* fix typo

* fix(migration): add org projection

* fix(projection): correct table for org events in org owners

* better insert stmt

* fix typo

* fix typo

* set max connection lifetime

* set max conns and conn lifetime in eventstore v1

* configure sql connection settings

* add mig for agg type index

* fix replace tab in yaml

* handler interfaces

* subscription

* first try

* handler

* move sql client initialization

* first part implemented

* removed all occurencies of org by id and search orgs

* fix merge issues

* cleanup code

* fix: queries implements orgviewprovider

* cleanup

* refactor text comparison

* remove unused file

* remove unused code

* log

* remove unused code

* remove unused field

* remove unused file

* refactor

* tests for search query

* remove try

* simplify state change mappers

* projection tests

* query functions

* move reusable objects to separate files

* rename domain column to primar_domain

* fix tests

* add current sequence

* remove log prints

* fix tests

* fix: verifier

* fix test

* rename domain col migrations

* simplify search response

* add custom column constructors

* fix: org projection table const

* fix: full column name

* feat: text query extension

* fix: tests for query

* number query

* add deprection message

* column in a single place (#2416)

* column in a single place

* use projection for columns

* query column with aliases

* rename methods

* remove unused code

* column for current sequences

* global counter column

* fix is org unique

* fix: merge main and change actions / flow projections to new query side (#2434)

* feat: actions (#2377)

* feat(actions): begin api

* feat(actions): begin api

* api and projections

* fix: handle multiple statements for a single event in projections

* export func type

* fix test

* update to new reduce interface

* flows in login

* feat: jwt idp

* feat: command side

* feat: add tests

* actions and flows

* fill idp views with jwt idps and return apis

* add jwtEndpoint to jwt idp

* begin jwt request handling

* add feature

* merge

* merge

* handle jwt idp

* cleanup

* bug fixes

* autoregister

* get token from specific header name

* fix: proto

* fixes

* i18n

* begin tests

* fix and log http proxy

* remove docker cache

* fixes

* usergrants in actions api

* tests adn cleanup

* cleanup

* fix add user grant

* set login context

* i18n

Co-authored-by: fabi <fabienne.gerschwiler@gmail.com>

* change actions / flow projections to new query side

* fixes

* enable org projection

Co-authored-by: fabi <fabienne.gerschwiler@gmail.com>

* fixes

* cleanup

* add tests

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Florian Forster <florian@caos.ch>
Co-authored-by: mffap <mpa@caos.ch>
Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>
Co-authored-by: Christian Jakob <47860090+thesephirot@users.noreply.github.com>
Co-authored-by: Elio Bischof <eliobischof@gmail.com>
Co-authored-by: Stefan Benz <stefan@caos.ch>
Co-authored-by: fabi <fabienne.gerschwiler@gmail.com>
2021-09-29 13:20:57 +02:00
Silvan
37ee5b4bab
feat: jobs for projection tables (#1730)
* job queue

* wg improvements

* start handler

* statement

* statements

* imporve handler

* improve statement

* statement in seperate file

* move handlers

* move query/old to query

* handler

* read models

* bulk works

* cleanup

* contrib

* rename readmodel to projection

* rename read_models schema to projections

* rename read_models schema to projections

* search query as func,
bulk iterates as long as new events

* add event sequence less query

* update checks for events between current sequence and sequence of first statement if it has previous sequence 0

* cleanup crdb projection

* refactor projection handler

* start with testing

* tests for handler

* remove todo

* refactor statement: remove table name,
add tests

* improve projection handler shutdown,
no savepoint if noop stmt,
tests for stmt handler

* tests

* start failed events

* seperate branch for contrib

* move statement constructors to crdb pkg

* correct import

* Subscribe for eventtypes (#1800)

* fix: is default (#1737)

* fix: use email as username on global org (#1738)

* fix: use email as username on global org

* Update user_human.go

* Update register_handler.go

* chore(deps): update docusaurus (#1739)

* chore: remove PAT and use GH Token (#1716)

* chore: remove PAT and use GH Token

* fix env

* fix env

* fix env

* md lint

* trigger ci

* change user

* fix GH bug

* replace login part

* chore: add GH Token to sem rel (#1746)

* chore: add GH Token to sem rel

* try branch

* add GH Token

* remove test branch again

* docs: changes acme to acme-caos (#1744)

* changes acme to acme-caos

* Apply suggestions from code review

Co-authored-by: Florian Forster <florian@caos.ch>

Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com>
Co-authored-by: Florian Forster <florian@caos.ch>

* feat: add additional origins on applications (#1691)

* feat: add additional origins on applications

* app additional redirects

* chore(deps-dev): bump @angular/cli from 11.2.8 to 11.2.11 in /console (#1706)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps-dev): bump @angular/cli from 11.2.8 to 11.2.11 in /console

Bumps [@angular/cli](https://github.com/angular/angular-cli) from 11.2.8 to 11.2.11.
- [Release notes](https://github.com/angular/angular-cli/releases)
- [Commits](https://github.com/angular/angular-cli/compare/v11.2.8...v11.2.11)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump stylelint from 13.10.0 to 13.13.1 in /console (#1703)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps-dev): bump stylelint from 13.10.0 to 13.13.1 in /console

Bumps [stylelint](https://github.com/stylelint/stylelint) from 13.10.0 to 13.13.1.
- [Release notes](https://github.com/stylelint/stylelint/releases)
- [Changelog](https://github.com/stylelint/stylelint/blob/master/CHANGELOG.md)
- [Commits](https://github.com/stylelint/stylelint/compare/13.10.0...13.13.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump @types/node from 14.14.37 to 15.0.1 in /console (#1702)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps-dev): bump @types/node from 14.14.37 to 15.0.1 in /console

Bumps [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) from 14.14.37 to 15.0.1.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump ts-protoc-gen from 0.14.0 to 0.15.0 in /console (#1701)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps): bump ts-protoc-gen from 0.14.0 to 0.15.0 in /console

Bumps [ts-protoc-gen](https://github.com/improbable-eng/ts-protoc-gen) from 0.14.0 to 0.15.0.
- [Release notes](https://github.com/improbable-eng/ts-protoc-gen/releases)
- [Changelog](https://github.com/improbable-eng/ts-protoc-gen/blob/master/CHANGELOG.md)
- [Commits](https://github.com/improbable-eng/ts-protoc-gen/compare/0.14.0...0.15.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps-dev): bump @types/jasmine from 3.6.9 to 3.6.10 in /console (#1682)

Bumps [@types/jasmine](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/jasmine) from 3.6.9 to 3.6.10.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/jasmine)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump @types/google-protobuf in /console (#1681)

Bumps [@types/google-protobuf](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/google-protobuf) from 3.7.4 to 3.15.2.
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/google-protobuf)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump grpc from 1.24.5 to 1.24.7 in /console (#1666)

Bumps [grpc](https://github.com/grpc/grpc-node) from 1.24.5 to 1.24.7.
- [Release notes](https://github.com/grpc/grpc-node/releases)
- [Commits](https://github.com/grpc/grpc-node/compare/grpc@1.24.5...grpc@1.24.7)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* lock

* chore(deps-dev): bump @angular/language-service from 11.2.9 to 11.2.12 in /console (#1704)

* fix: show org with regex (#1688)

* fix: flag mapping (#1699)

* chore(deps-dev): bump @angular/language-service in /console

Bumps [@angular/language-service](https://github.com/angular/angular/tree/HEAD/packages/language-service) from 11.2.9 to 11.2.12.
- [Release notes](https://github.com/angular/angular/releases)
- [Changelog](https://github.com/angular/angular/blob/master/CHANGELOG.md)
- [Commits](https://github.com/angular/angular/commits/11.2.12/packages/language-service)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Silvan <silvan.reusser@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* package lock

* downgrade grpc

* downgrade protobuf types

* revert npm packs 🥸

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Silvan <silvan.reusser@gmail.com>

* docs: update run and start section texts (#1745)

* update run and start section texts

* adds showcase

Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com>

* fix: additional origin list (#1753)

* fix: handle api configs in authz handler (#1755)

* fix(console): add model for api keys, fix toast, binding (#1757)

* fix: add model for api keys, fix toast, binding

* show api clientid

* fix: missing patchvalue (#1758)

* feat: refresh token (#1728)

* begin refresh tokens

* refresh tokens

* list and revoke refresh tokens

* handle remove

* tests for refresh tokens

* uniqueness and default expiration

* rename oidc token methods

* cleanup

* migration version

* Update internal/static/i18n/en.yaml

Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>

* fixes

* feat: update oidc pkg for refresh tokens

Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>

* fix: correct json name of clientId in key.json (#1760)

* fix: migration version (#1767)

* start subscription

* eventtypes

* fix(login): links (#1778)

* fix(login): href for help

* fix(login): correct link to tos

* fix: access tokens for service users and refresh token infos (#1779)

* fix: access token for service user

* handle info from refresh request

* uniqueness

* postpone access token uniqueness change

* chore(coc): recommend code of conduct (#1782)

* subscribe for events

* feat(console): refresh toggle out of granttype context (#1785)

* refresh toggle

* disable if not code flow, lint

* lint

* fix: change oidc config order

* accept refresh option within flow

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* fix: refresh token activation (#1795)

* fix: oidc grant type check

* docs: add offline_access scope

* docs: update refresh token status in supported grant types

* fix: update oidc pkg

* fix: check refresh token grant type (#1796)

* configuration structs

* org admins

* failed events

* fixes

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Florian Forster <florian@caos.ch>
Co-authored-by: mffap <mpa@caos.ch>
Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>

* remove comment

* aggregate reducer

* remove eventtypes

* add protoc-get-validate to mod

* fix transaltion

* upsert

* add gender on org admins,
allow to retry failed stmts after configurable time

* remove if

* sub queries

* fix: tests

* add builder to tests

* new search query

* rename searchquerybuilder to builder

* remove comment from code

* test with multiple queries

* add filters test

* current sequences

* make org and org_admins work again

* add aggregate type to current sequence

* fix(contibute): listing

* add validate module

* fix: search queries

* feat(eventstore): previous aggregate root sequence (#1810)

* feat(eventstore): previous aggregate root sequence

* fix tests

* fix: eventstore v1 test

* add col to all mocked rows

* next try

* fix mig

* rename aggregate root to aggregate type

* update comment

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* small refactorings

* allow update multiple current sequences

* unique log id

* fix migrations

* rename org admin to org owner

* improve error handling and logging

* fix(migration): optimize prev agg root seq

* fix: projection handler test

* fix: sub queries

* small fixes

* additional event types

* correct org owner projection

* fix primary key

* feat(eventstore): jobs for projections (#2026)

* fix: template names in login (#1974)

* fix: template names in login

* fix: error.html

* fix: check for features on mgmt only (#1976)

* fix: add sentry in ui, http and projection handlers (#1977)

* fix: add sentry in ui, http and projection handlers

* fix test

* fix(eventstore): sub queries (#1805)

* sub queries

* fix: tests

* add builder to tests

* new search query

* rename searchquerybuilder to builder

* remove comment from code

* test with multiple queries

* add filters test

* fix(contibute): listing

* add validate module

* fix: search queries

* remove unused event type in query

* ignore query if error in marshal

* go mod tidy

* update privacy policy query

* update queries

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* feat: Extend oidc idp with oauth endpoints (#1980)

* feat: add oauth attributes to oidc idp configuration

* feat: return idpconfig id on create idp

* feat: tests

* feat: descriptions

* feat: docs

* feat: tests

* docs: update to beta 3 (#1984)

* fix: role assertion (#1986)

* fix: enum to display access token role assertion

* improve assertion descriptions

* fix nil pointer

* docs: eventstore (#1982)

* docs: eventstore

* Apply suggestions from code review

Co-authored-by: Florian Forster <florian@caos.ch>

Co-authored-by: Florian Forster <florian@caos.ch>

* fix(sentry): trigger sentry release (#1989)

* feat(send sentry release): send sentry release

* fix(moved step and added releasetag): moved step and added releasetag

* fix: set version for sentry release (#1990)

* feat(send sentry release): send sentry release

* fix(moved step and added releasetag): moved step and added releasetag

* fix(corrected var name): corrected var name

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* fix: log error reason on terminate session (#1973)

* fix: return default language file, if requested lang does not exist for default login texts (#1988)

* fix: return default language file, if requested lang doesnt exists

* feat: read default translation file

* feat: docs

* fix: race condition in auth request unmarshalling (#1993)

* feat: handle ui_locales in login (#1994)

* fix: handle ui_locales in login

* move supportedlanguage func into i18n package

* update oidc pkg

* fix: handle closed channels on unsubscribe (#1995)

* fix: give restore more time (#1997)

* fix: translation file read (#2009)

* feat: translation file read

* feat: readme

* fix: enable idp add button for iam users (#2010)

* fix: filter event_data (#2011)

* feat: Custom message files (#1992)

* feat: add get custom message text to admin api

* feat: read custom message texts from files

* feat: get languages in apis

* feat: get languages in apis

* feat: get languages in apis

* feat: pr feedback

* feat: docs

* feat: merge main

* fix: sms notification (#2013)

* fix: phone verifications

* feat: fix password reset as sms

* fix: phone verification

* fix: grpc status in sentry and validation interceptors (#2012)

* fix: remove oauth endpoints from oidc config proto (#2014)

* try with view

* fix(console): disable sw (#2021)

* fix: disable sw

* angular.json disable sw

* project projections

* fix typos

* customize projections

* customizable projections,
add change date to projects

Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>
Co-authored-by: Florian Forster <florian@caos.ch>
Co-authored-by: mffap <mpa@caos.ch>
Co-authored-by: Christian Jakob <47860090+thesephirot@users.noreply.github.com>
Co-authored-by: Elio Bischof <eliobischof@gmail.com>

* env file

* typo

* correct users

* correct migration

* fix: merge fail

* fix test

* fix(tests): unordered matcher

* improve currentSequenceMatcher

* correct certs

* correct certs

* add zitadel database on database list

* refctor switch in match

* enable all handlers

* Delete io.env

* cleanup

* add handlers

* rename view to projection

* rename view to projection

* fix type typo

* remove unnecessary logs

* refactor stmts

* simplify interval calculation

* fix tests

* fix unlock test

* fix migration

* migs

* fix(operator): update cockroach and flyway versions (#2138)

* chore(deps): bump k8s.io/apiextensions-apiserver from 0.19.2 to 0.21.3

Bumps [k8s.io/apiextensions-apiserver](https://github.com/kubernetes/apiextensions-apiserver) from 0.19.2 to 0.21.3.
- [Release notes](https://github.com/kubernetes/apiextensions-apiserver/releases)
- [Commits](https://github.com/kubernetes/apiextensions-apiserver/compare/v0.19.2...v0.21.3)

---
updated-dependencies:
- dependency-name: k8s.io/apiextensions-apiserver
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* chore(deps): bump google.golang.org/api from 0.34.0 to 0.52.0

Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.34.0 to 0.52.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/master/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.34.0...v0.52.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* start update dependencies

* update mods and otlp

* fix(build): update to go 1.16

* old version for k8s mods

* update k8s versions

* update orbos

* fix(operator): update cockroach and flyway version

* Update images.go

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Stefan Benz <stefan@caos.ch>

* fix import

* fix typo

* fix(migration): add org projection

* fix(projection): correct table for org events in org owners

* better insert stmt

* fix typo

* fix typo

* set max connection lifetime

* set max conns and conn lifetime in eventstore v1

* configure sql connection settings

* add mig for agg type index

* fix replace tab in yaml

* check requeue at least 500ms

* split column in column and condition

* remove useless comment

* mig versions

* fix migs

Co-authored-by: Max Peintner <max@caos.ch>
Co-authored-by: Livio Amstutz <livio.a@gmail.com>
Co-authored-by: Florian Forster <florian@caos.ch>
Co-authored-by: mffap <mpa@caos.ch>
Co-authored-by: Maximilian Panne <maximilian.panne@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Fabi <38692350+fgerschwiler@users.noreply.github.com>
Co-authored-by: Christian Jakob <47860090+thesephirot@users.noreply.github.com>
Co-authored-by: Elio Bischof <eliobischof@gmail.com>
Co-authored-by: Stefan Benz <stefan@caos.ch>
2021-08-19 08:31:56 +02:00
Fabi
667cc30291
feat: asset storage (#1696)
* feat: remove assets

* feat: minio implementation

* fix: remove assets from tests

* feat: minio implementation

* feat: Env vars

* fix: sprintf

* fix: sprintf

* Update internal/eventstore/repository/repository.go

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* fix: error handling

Co-authored-by: Livio Amstutz <livio.a@gmail.com>
2021-05-03 10:15:50 +02:00
Fabi
f51f0ede5c
feat: add assets to eventstore and event (#1674)
* fix: add assets to eventstore and event

* fix: project member, grant member, app changed tests

* fix: asset migrations

* feat: add asset tests

* feat: add asset tests

* Update internal/eventstore/repository/repository.go

Co-authored-by: Livio Amstutz <livio.a@gmail.com>

* feat: add asset tests

Co-authored-by: Livio Amstutz <livio.a@gmail.com>
2021-04-27 12:58:18 +02:00
Fabi
d8e42744b4
fix: move v2 pkgs (#1331)
* fix: move eventstore pkgs

* fix: move eventstore pkgs

* fix: remove v2 view

* fix: remove v2 view
2021-02-23 15:13:04 +01:00