Explain event sourcing and CQRS: command→event flows, append‑only logs and…
- User
Explain event sourcing and CQRS: command→event flows, append‑only logs and serialization, read models and projections, idempotency/replay, snapshotting, consistency models, evolution/versioning, scaling, and testing; include code/architecture sketches.
- Assistant
1) What CQRS and Event Sourcing are (and how they relate)
CQRS (Command Query Responsibility Segregation): separate the write model (commands that change state) from the read model (queries optimized for reads). This is primarily an architectural separation.
Event Sourcing (ES): instead of storing the latest state of an entity/aggregate, store an append-only sequence of events (“facts”) that happened. Current state is derived by replaying events (optionally accelerated by snapshots).
They’re often paired:
- CQRS handles different shapes for writes vs reads.
- Event sourcing provides a natural “source of truth” for the write side and a stream to build read models (projections).
2) Command → Event flow (typical write path)
High-level flow
Client | Command (intent) v API / Command Endpoint | v Command Handler | v Load Aggregate (rehydrate from events [+ snapshot]) | v Aggregate decides -> produces new Events | v Append events to Event Store (optimistic concurrency) | +--> (optional) publish events to bus / outboxArchitecture sketch with CQRS read side
Commands Events Queries Client ------------> Write API -----> Event Store (append-only log) <---- Read API <---- Client | | | | | subscription | v v v Domain Model Projectors Read Models (SQL/Elastic/Redis) (Aggregates) (Projections) (materialized views)Minimal code sketch (event-sourced aggregate)
Example pseudo-TypeScript-like:
type Command = | { type: "OpenAccount"; accountId: string; ownerId: string } | { type: "DepositMoney"; accountId: string; amount: number; commandId: string }; type Event = | { type: "AccountOpened"; accountId: string; ownerId: string } | { type: "MoneyDeposited"; accountId: string; amount: number; commandId: string }; class AccountAggregate { private opened = false; private balance = 0; private processedCommandIds = new Set<string>(); // for idempotency (optional) static rehydrate(history: Event[]): AccountAggregate { const a = new AccountAggregate(); for (const e of history) a.apply(e); return a; } handle(cmd: Command): Event[] { switch (cmd.type) { case "OpenAccount": if (this.opened) throw new Error("Already opened"); return [{ type: "AccountOpened", accountId: cmd.accountId, ownerId: cmd.ownerId }]; case "DepositMoney": if (!this.opened) throw new Error("Not opened"); if (cmd.amount <= 0) throw new Error("Invalid amount"); // Command idempotency at aggregate level (optional) if (this.processedCommandIds.has(cmd.commandId)) return []; return [{ type: "MoneyDeposited", accountId: cmd.accountId, amount: cmd.amount, commandId: cmd.commandId }]; } } apply(e: Event) { switch (e.type) { case "AccountOpened": this.opened = true; break; case "MoneyDeposited": this.balance += e.amount; this.processedCommandIds.add(e.commandId); break; } } }And the command handler that enforces optimistic concurrency:
async function handleCommand(cmd: Command) { const streamId = `Account-${cmd.accountId}`; const { events: history, version } = await eventStore.readStream(streamId); const agg = AccountAggregate.rehydrate(history); const newEvents = agg.handle(cmd); // Append with expected version to prevent lost updates await eventStore.appendToStream(streamId, newEvents, { expectedVersion: version }); }
3) Append-only logs, streams, and serialization
Event store concepts
Most event stores model data as:
- Stream: ordered events for a single aggregate instance (e.g.,
Account-123) - Event: immutable record with metadata
- Global position (optional but common): total order across all streams for subscriptions
Typical event record:
{ "streamId": "Account-123", "streamVersion": 42, "globalPosition": 10938410, "eventType": "MoneyDeposited", "eventId": "uuid", "occurredAt": "2025-12-13T10:00:00Z", "data": { "amount": 50, "commandId": "..." }, "metadata": { "correlationId": "...", "causationId": "...", "actor": "user-7", "schemaVersion": 2 } }Append-only matters
- No in-place updates or deletes (in the logical model).
- Concurrency is handled by “append with expected version”.
- Audit and time-travel are built-in.
Serialization choices
Common options:
- JSON: easy debugging, slower/larger; needs versioning discipline.
- Avro/Protobuf: compact, schema-first, good for long-lived systems.
- Binary + custom: fastest but highest maintenance risk.
Important: store enough metadata to replay correctly:
- timestamps, correlation/causation IDs, tenant, user, schema version, etc.
4) Read models and projections (query side)
A projection subscribes to events and updates a read model (materialized view) optimized for queries.
Projection flow
Event Store --(subscription: at-least-once)--> Projector | | +-- global position ------------------------------> | v Read DB (view tables)Example projection
Suppose we want a read model:
AccountSummary { accountId, ownerId, balance }.async function projectAccountSummary(event: Event, tx: DbTx) { switch (event.type) { case "AccountOpened": await tx.upsert("AccountSummary", { accountId: event.accountId }, { ownerId: event.ownerId, balance: 0 }); break; case "MoneyDeposited": // Must be idempotent (see below) await tx.execute(` UPDATE AccountSummary SET balance = balance + ? WHERE accountId = ?`, [event.amount, event.accountId]); break; } }Read models can be:
- relational tables
- Elasticsearch indexes
- Redis caches
- graph projections
- “denormalized” documents
They’re disposable and rebuildable from the event log.
5) Idempotency and replay (the “at-least-once” reality)
Most event delivery to projectors is at-least-once, meaning duplicates can happen. Therefore:
Projection idempotency patterns
A) Track last processed position (per projector)
- Store a checkpoint:
projector_name -> lastGlobalPosition - Only advance checkpoint after successfully applying event
- Still can get duplicates if you crash after applying but before checkpointing (unless atomic)
B) Exactly-once effect via transactional dedup Use a table keyed by
eventId(or(streamId, streamVersion)) to ensure you apply each event once:-- within the same DB transaction as the read model update INSERT INTO ProjectorProcessedEvents(projector, eventId) VALUES (:projector, :eventId) ON CONFLICT DO NOTHING;Then only update the view if the insert succeeded.
C) Design projections as commutative/idempotent E.g., recompute aggregates from scratch (more expensive), or store final values with event version checks.
Command idempotency patterns
- Client provides
commandId(idempotency key). - Store processed command IDs per aggregate or in a separate store.
- If the same command arrives again, return the previous result or do nothing.
Replay
Replays are used for:
- rebuilding projections after code changes
- populating new read models
- recovering from data loss on the read side
Key requirements:
- events are immutable and complete enough to rebuild state
- projections are deterministic or can handle old events via versioning/upcasting
6) Snapshotting (speeding up rehydration)
Rehydrating an aggregate by replaying thousands of events can be too slow. A snapshot stores a periodic copy of aggregate state.
Snapshot strategy
- Every N events (e.g., 100) or time-based (e.g., daily)
- Store snapshot alongside stream with version
Rehydration:
Load latest snapshot (state at version V) Read events from V+1 to end Apply remaining eventsSnapshot record sketch:
{ "streamId": "Account-123", "snapshotVersion": 500, "state": { "opened": true, "balance": 1200, "processedCommandIds": ["..."] } }Notes:
- Snapshots are an optimization; they must be rebuildable from events.
- Be careful snapshotting large “processed IDs” sets—often handled differently (e.g., command dedup store).
7) Consistency models (what is consistent, when)
With CQRS + ES, consistency is usually:
A) Strong consistency within an aggregate
- Command handling loads the aggregate stream and appends events with expected version.
- This gives linearizable behavior for that aggregate (no concurrent lost updates).
B) Eventual consistency between write model and read models
- Projections lag behind event store.
- Queries hitting read models may see stale data.
C) Options to tighten consistency
- Read-your-writes: after command completion, query by reading from write side (rehydrate aggregate) or use a “consistent read” endpoint.
- Synchronous projection for critical views: update specific read model in the same request path (often defeats some CQRS benefits, but can be pragmatic).
- Transactional outbox pattern (if event store isn’t your DB): ensure events and DB changes are atomically recorded, then published reliably.
Make the consistency explicit in APIs (e.g., return the new stream version / event position so clients can wait until projections catch up).
8) Evolution and versioning (events live forever)
Because events are long-lived, you need a strategy to evolve them.
Event versioning approaches
1) Additive changes (preferred)
- Add optional fields with defaults.
- Keep event meaning stable.
2) Upcasting (read-time transformation)
- Store old versions, transform them into the latest in your deserializer.
function upcast(raw: any): Event { if (raw.eventType === "MoneyDeposited" && raw.metadata.schemaVersion === 1) { // v1 had { amount }, v2 adds { commandId } return { ...raw.data, type: "MoneyDeposited", commandId: raw.metadata.eventId }; } return raw.data as Event; }3) New event types
- Introduce
MoneyDepositedV2and support both in projectors.
4) Event migration (last resort)
- Rewrite historical events (dangerous, requires careful tooling and audit implications).
Rules of thumb
- Treat event types as public contracts.
- Don’t change semantics of an existing event.
- Maintain compatibility in projections (they should handle old + new).
9) Scaling patterns
Write side scaling
- Partition/shard by streamId (aggregateId): all events for an aggregate must stay ordered.
- Concurrency is naturally bounded to per-aggregate streams.
- Hot aggregates can become bottlenecks; mitigate by redesigning aggregate boundaries.
Read side scaling
- Projections scale horizontally:
- multiple projector instances consuming from partitions
- separate read models per query shape (multiple projections)
Event delivery scaling
- Use subscription checkpoints and consumer groups.
- Many systems use:
- event store subscriptions (EventStoreDB)
- Kafka as the log (events are the log)
- or store events in DB and publish to a broker (outbox)
Storage considerations
- Append-only logs grow; plan retention/archival:
- keep events forever (common for audit) + archive cold streams
- compress older segments
- maintain snapshots to reduce read amplification
10) Testing strategies (ES/CQRS-friendly)
A) Aggregate tests (Given–When–Then)
You test pure domain logic without databases:
function given(history: Event[]) { return AccountAggregate.rehydrate(history); } test("deposit produces MoneyDeposited", () => { const agg = given([{ type: "AccountOpened", accountId: "A1", ownerId: "U1" }]); const events = agg.handle({ type: "DepositMoney", accountId: "A1", amount: 10, commandId: "C1" }); expect(events).toEqual([{ type: "MoneyDeposited", accountId: "A1", amount: 10, commandId: "C1" }]); });B) Projection tests
Feed an event sequence and assert the read model state:
- start from empty DB
- apply events
- verify view rows
Also test idempotency by replaying the same event twice.
C) Integration tests
- command → event store append → projector → read model
- include failure scenarios: projector crash mid-batch, duplicates, out-of-order (if possible)
D) Property-based tests (valuable for aggregates)
Generate random sequences of commands/events and check invariants:
- balance never negative (if required)
- streamVersion increases monotonically
- applying produced events yields valid state
11) Practical “gotchas” checklist
- Aggregate boundaries: keep aggregates small; don’t require cross-aggregate transactions in a single command.
- Idempotency everywhere: commands and projections.
- Observability: correlationId/causationId, event positions, lag metrics per projector.
- Backfills/replays: design projectors to be rebuildable and performant.
- Schema discipline: version events intentionally; avoid semantic changes.
- Consistency UX: be explicit about eventual consistency in client flows.
If you share a concrete domain (e.g., order management, payments) and a preferred stack (EventStoreDB, Kafka, Postgres, .NET/Java/Node), I can sketch a more concrete end-to-end architecture with specific storage schemas, subscription setup, and code structure.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.