Commit Graph

6 Commits

Author SHA1 Message Date
Tim Möhlmann
250f2344c8
feat(cache): redis cache (#8822)
# Which Problems Are Solved

Add a cache implementation using Redis single mode. This does not add
support for Redis Cluster or sentinel.

# How the Problems Are Solved

Added the `internal/cache/redis` package. All operations occur
atomically, including setting of secondary indexes, using LUA scripts
where needed.

The [`miniredis`](https://github.com/alicebob/miniredis) package is used
to run unit tests.

# Additional Changes

- Move connector code to `internal/cache/connector/...` and remove
duplicate code from `query` and `command` packages.
- Fix a missed invalidation on the restrictions projection

# Additional Context

Closes #8130
2024-11-04 10:44:51 +00:00
Tim Möhlmann
32bad3feb3
perf(milestones): refactor (#8788)
Some checks are pending
ZITADEL CI/CD / core (push) Waiting to run
ZITADEL CI/CD / console (push) Waiting to run
ZITADEL CI/CD / version (push) Waiting to run
ZITADEL CI/CD / compile (push) Blocked by required conditions
ZITADEL CI/CD / core-unit-test (push) Blocked by required conditions
ZITADEL CI/CD / core-integration-test (push) Blocked by required conditions
ZITADEL CI/CD / lint (push) Blocked by required conditions
ZITADEL CI/CD / container (push) Blocked by required conditions
ZITADEL CI/CD / e2e (push) Blocked by required conditions
ZITADEL CI/CD / release (push) Blocked by required conditions
Code Scanning / CodeQL-Build (go) (push) Waiting to run
Code Scanning / CodeQL-Build (javascript) (push) Waiting to run
# Which Problems Are Solved

Milestones used existing events from a number of aggregates. OIDC
session is one of them. We noticed in load-tests that the reduction of
the oidc_session.added event into the milestone projection is a costly
business with payload based conditionals. A milestone is reached once,
but even then we remain subscribed to the OIDC events. This requires the
projections.current_states to be updated continuously.


# How the Problems Are Solved

The milestone creation is refactored to use dedicated events instead.
The command side decides when a milestone is reached and creates the
reached event once for each milestone when required.

# Additional Changes

In order to prevent reached milestones being created twice, a migration
script is provided. When the old `projections.milestones` table exist,
the state is read from there and `v2` milestone aggregate events are
created, with the original reached and pushed dates.

# Additional Context

- Closes https://github.com/zitadel/zitadel/issues/8800
2024-10-28 08:29:34 +00:00
Tim Möhlmann
25dc7bfe72
perf(cache): pgx pool connector (#8703)
# Which Problems Are Solved

Cache implementation using a PGX connection pool.

# How the Problems Are Solved

Defines a new schema `cache` in the zitadel database.
A table for string keys and a table for objects is defined.
For postgreSQL, tables are unlogged and partitioned by cache name for
performance.

Cockroach does not have unlogged tables and partitioning is an
enterprise feature that uses alternative syntax combined with sharding.
Regular tables are used here.

# Additional Changes

- `postgres.Config` can return a pxg pool. See following discussion

# Additional Context

- Part of https://github.com/zitadel/zitadel/issues/8648
- Closes https://github.com/zitadel/zitadel/issues/8647

---------

Co-authored-by: Silvan <silvan.reusser@gmail.com>
2024-10-04 13:15:41 +00:00
Tim Möhlmann
4eaa3163b6
feat(storage): generic cache interface (#8628)
# Which Problems Are Solved

We identified the need of caching.
Currently we have a number of places where we use different ways of
caching, like go maps or LRU.
We might also want shared chaches in the future, like Redis-based or in
special SQL tables.

# How the Problems Are Solved

Define a generic Cache interface which allows different implementations.

- A noop implementation is provided and enabled as.
- An implementation using go maps is provided
  - disabled in defaults.yaml
  - enabled in integration tests
- Authz middleware instance objects are cached using the interface.

# Additional Changes

- Enabled integration test command raceflag
- Fix a race condition in the limits integration test client
- Fix a number of flaky integration tests. (Because zitadel is super
fast now!) 🎸 🚀

# Additional Context

Related to https://github.com/zitadel/zitadel/issues/8648
2024-09-25 21:40:21 +02:00
Silvan
9892fd92b6
refactor: cleanup unused code (#7130)
* refactor: drop unused code

* refactor: drop unused code
2024-01-02 14:26:31 +00:00
Fabi
802bb56ea1
feat: project cache (#48)
* feat: eventstore repository

* fix: remove gorm

* version

* feat: pkg

* feat: add some files for project

* feat: eventstore without eventstore-lib

* rename files

* gnueg

* fix: key json

* fix: add object

* fix: change imports

* fix: internal models

* fix: some imports

* fix: global model

* fix: add some functions on repo

* feat(eventstore): sdk

* fix(eventstore): search query

* fix(eventstore): rename app to eventstore

* delete empty test

* remove unused func

* merge master

* fix(eventstore): tests

* fix(models): delete unused struct

* fix: some funcitons

* feat(eventstore): implemented push events

* fix: move project eventstore to project package

* fix: change project eventstore funcs

* feat(eventstore): overwrite context data

* fix: change project eventstore

* fix: add project repo to mgmt server

* feat(types): SQL-config

* fix: commented code

* feat(eventstore): options to overwrite editor

* feat: auth interceptor and cockroach migrations

* fix: migrations

* fix: fix filter

* fix: not found on getbyid

* fix: add sequence

* fix: add some tests

* fix(eventstore): nullable sequence

* fix: add some tests

* merge

* fix: add some tests

* fix(migrations): correct statements for sequence

* fix: add some tests

* fix: add some tests

* fix: changes from mr

* Update internal/eventstore/models/field.go

Co-Authored-By: livio-a <livio.a@gmail.com>

* fix(eventstore): code quality

* fix: add types to aggregate/Event-types

* fix(eventstore): rename modifier* to editor*

* fix(eventstore): delete editor_org

* fix(migrations): remove editor_org field,
rename modifier_* to editor_*

* fix: generate files

* fix(eventstore): tests

* fix(eventstore): rename modifier to editor

* fix(migrations): add cluster migration,
fix(migrations): fix typo of host in clean clsuter

* fix(eventstore): move health

* fix(eventstore): AggregateTypeFilter aggregateType as param

* code quality

* feat: add member funcs

* feat: add member model

* feat: add member events

* feat: add member repo model

* fix: project member funcs

* fix: add tests

* fix: add tests

* feat: implement member requests

* fix: merge master

* fix: read existing in project repo

* fix: fix tests

* feat: add internal cache

* feat: add cache mock

* fix: return values of cache mock

* fix: add cache config

* fix: use eventstore sdk

* Update internal/project/model/project_member.go

Co-Authored-By: Silvan <silvan.reusser@gmail.com>

* fix: use get project func

* fix: return err not nil

* fix: change err types

* Update internal/cache/bigcache/cache.go

Co-Authored-By: livio-a <livio.a@gmail.com>

* Update internal/cache/config/config.go

Co-Authored-By: livio-a <livio.a@gmail.com>

* Update internal/cache/config/config.go

Co-Authored-By: livio-a <livio.a@gmail.com>

* fix: config

* fix: config

* resolve conversations

* fix: mr changes

* fix: fix decode of bigcache

* feat: test caches

* fix: remove unnecessary code

Co-authored-by: adlerhurst <silvan.reusser@gmail.com>
Co-authored-by: livio-a <livio.a@gmail.com>
2020-04-20 15:16:33 +02:00