Skip to content
TheChatArchiveThe Chat Archive
Step 3.5 Flash· unknownHugging Face Frontier Reasoning Traces

Design an event-driven architecture with exactly-once semantics: idempotency,…

  1. User

    Design an event-driven architecture with exactly-once semantics: idempotency, deduplication, outbox pattern, retries/DLQs, and observability.

  2. Assistant

    We are designing an event-driven architecture with exactly-once semantics. The key challenges include ensuring that each event is processed exactly once, even in the face of failures and retries. We'll incorporate idempotency, deduplication, the outbox pattern, retries with dead-letter queues (DLQs), and observability.

    Components and Flow:

    1. Event Producer: Generates events. To ensure exactly-once, the producer must be idempotent and use deduplication at the source (if supported by the message broker). However, we cannot rely solely on the broker. We'll use the outbox pattern for reliable event publishing.

    2. Outbox Pattern:

      • Instead of directly publishing events to the message broker, the producer (or a service that handles the event) writes the event to an outbox table in the same database transaction that updates the business state. This ensures that the event is only published if the transaction commits.
      • A separate process (Outbox Poller) periodically reads from the outbox table and publishes the events to the message broker. After successful publishing, it marks the event as published (or deletes it).
    3. Message Broker: We assume a broker that supports at-least-once delivery (like Kafka, RabbitMQ, etc.). We cannot rely on exactly-once from the broker alone, so we need to handle duplicates in the consumer.

    4. Event Consumer:

      • The consumer must be idempotent. It should track processed events (by event ID) and ignore duplicates.
      • We can use a deduplication store (e.g., a database table with event IDs) to check if an event has already been processed.
    5. Retries and DLQ:

      • If the consumer fails to process an event (due to transient errors), we retry a few times (with backoff).
      • After a certain number of retries, we move the event to a Dead-Letter Queue (DLQ) for further analysis.
    6. Observability: We need logging, metrics, and tracing to monitor the system.

    Let's break down the components in more detail:

    1. Event Producer with Outbox

    When a business transaction occurs that should produce an event:

    • In the same database transaction that updates the business state, insert an event record into an outbox table. The record includes:
      • event_id (unique, generated by the producer, e.g., UUID)
      • aggregate_id (the ID of the aggregate that produced the event)
      • event_type
      • payload (the event data, serialized as JSON)
      • created_at
      • published_at (nullable, initially NULL)

    Example outbox table:

    CREATE TABLE outbox (
        event_id UUID PRIMARY KEY,
        aggregate_id UUID NOT NULL,
        event_type VARCHAR(255) NOT NULL,
        payload JSONB NOT NULL,
        created_at TIMESTAMP NOT NULL,
        published_at TIMESTAMP
    );
    

    2. Outbox Poller

    A separate service (could be part of the same application or a separate process) that:

    • Periodically queries the outbox table for unpublished events (where published_at is NULL) and publishes them to the message broker.
    • After successfully publishing (receiving an ack from the broker), update the published_at to the current time (or delete the record). We should also consider that the broker might not support exactly-once, so we might have duplicates in the broker. But that's okay because the consumer will deduplicate.

    Note: The outbox poller must be careful to handle failures. If the poller crashes after publishing but before updating the outbox, it might republish the same event. That's acceptable because the consumer will deduplicate. However, to avoid unnecessary duplicates, we can use a transaction when updating the outbox and then commit only after the broker ack. But note that the outbox poller and the broker are two different systems. We can use a two-phase commit? Not really. Alternatively, we can use the broker's transaction if supported (like Kafka transactions). But let's assume we don't have that, so we accept that sometimes we republish.

    To minimize duplicates, we can use a "mark as published" in the same transaction that we read from the outbox, but then if the broker fails, we have marked it as published but the event wasn't delivered. So we have to choose: either risk duplicate events or risk lost events. Since we want exactly-once, we should accept duplicates and rely on consumer deduplication. Therefore, we can do:

    • Read a batch of events from the outbox (with published_at NULL) and lock them (using SELECT FOR UPDATE SKIP LOCKED) to avoid multiple pollers processing the same events.
    • For each event, publish to the broker. If publishing fails, we leave the event as unpublished and retry later.
    • If publishing succeeds, update the published_at to the current time.

    But note: if the poller crashes after publishing some events but before updating the outbox, then on restart it will republish those events. That's acceptable because the consumer will deduplicate.

    Alternatively, we can use a separate "published" flag and update it after each event, but the same issue remains.

    3. Message Broker

    We can use any broker that supports at-least-once delivery. Examples: Kafka, RabbitMQ, AWS SNS/SQS, etc.

    We must consider:

    • The broker might reorder events? We might need ordering per aggregate. If we require ordering per aggregate, we must partition by aggregate ID (in Kafka) or use a single queue per aggregate (not practical). So we design for at-least-once and possibly out-of-order, but the consumer must handle that by using event version (or sequence number) per aggregate.

    4. Event Consumer

    The consumer:

    • Subscribes to the topic/queue.
    • For each received event, it must:
      • Check if the event has already been processed (by event_id). We can maintain a processed_events table with event_id and processed_at.
      • If the event is new, process it (execute business logic) and then record the event_id as processed.
      • If the event is a duplicate, ignore it.

    But note: what if the consumer fails after processing but before recording the event_id? Then on retry (or if the event is redelivered) we might process it again. So we must record the event_id in the same transaction as the business state update. That is, the consumer's processing must be atomic: either both the business state and the processed_events record are committed, or neither.

    So the consumer's processing should be:

    1. Begin transaction.
    2. Check if event_id exists in processed_events. If yes, rollback and ignore.
    3. If not, process the event (update business state).
    4. Insert the event_id into processed_events.
    5. Commit transaction.

    But note: if the event is redelivered while the transaction is in progress (but not committed), we might have a race condition. We can use database locks or unique constraint on event_id in processed_events to prevent duplicates.

    Let's design the processed_events table:

    CREATE TABLE processed_events (
        event_id UUID PRIMARY KEY,
        processed_at TIMESTAMP NOT NULL
    );
    

    We can also add an index on processed_at for cleanup of old records.

    In the consumer, we try to insert the event_id into processed_events first (or at the same time as processing). But we must check for duplicates. We can do:

    • Try to insert the event_id with a unique constraint. If it fails (duplicate key), then we know it's already processed and we skip.

    But we also need to update the business state. So:

    Option 1: BEGIN TRANSACTION; INSERT INTO processed_events (event_id, processed_at) VALUES (?, NOW()); -- If duplicate, the insert fails and we rollback. -- Then update business state (using the event data) and commit. COMMIT;

    But what if the business update fails? Then we have a processed_events record but no business state update. That would be a problem because we marked the event as processed but it didn't have an effect. So we must do the business update and the processed_events insert atomically.

    So:

    Option 2 (better): BEGIN TRANSACTION; -- First, check if the event is already processed (by attempting to insert and catching duplicate). Alternatively, we can do: -- INSERT ... ON CONFLICT DO NOTHING and then check if the insert was successful. But we also need to update business state only if it's new.

    Let's use:

    Step 1: Try to insert into processed_events (with the event_id). If duplicate, rollback and skip. Step 2: If insert succeeded, then update business state. Step 3: Commit.

    But note: if the business update fails (e.g., due to a constraint), we must rollback the entire transaction, including the processed_events insert. That is acceptable because we don't want to mark an event as processed if the business update failed.

    However, there is a race condition: if two consumers get the same event (due to duplicate delivery) and both try to insert the event_id at the same time, one will succeed and the other will fail. So we are safe.

    But what if the consumer processes the event and then crashes before committing? Then on restart, we might get the same event again and try to insert. The insert will fail because the event_id is already in the table from the previous transaction that didn't commit? No, because the previous transaction was rolled back. So the processed_events row would not be there. So we are safe.

    However, note that the broker might redeliver the event after the consumer crashes without committing. So we must use manual acknowledgments: only ack the event to the broker after the transaction commits.

    So the consumer flow:

    1. Receive event (with manual ack).
    2. Begin transaction.
    3. Try to insert event_id into processed_events. If duplicate, rollback and ack the event (since it's already processed). But wait: if we get a duplicate, we should ack it and skip.
      • We can do: INSERT INTO processed_events ... If duplicate error (SQLSTATE 23505), then rollback and ack.
      • But note: if the insert fails due to duplicate, that means the event was already processed (and committed). So we can ack and skip.
    4. If insert succeeds, then process the event (update business state). If business update fails, rollback and do not ack (so the event will be redelivered).
    5. Commit transaction.
    6. Ack the event to the broker.

    But what if the broker does not support manual ack? Then we have to ack after processing? That's dangerous because if we ack before committing, we might lose the event if the consumer crashes after ack but before commit. So we must ack only after commit.

    Therefore, we require the broker to support manual acknowledgments (or transactions) so that we can ack after the database transaction commits.

    Alternatively, we can use a two-phase commit between the broker and the database? That's complex and not typically supported. So we rely on the broker's at-least-once and handle duplicates in the consumer.

    But note: if the consumer crashes after committing the database transaction but before acking the broker, then the broker will redeliver the event. That's okay because we have already recorded the event as processed, so the next time we get the duplicate, we will skip. However, we must be cautious: the broker might redeliver very quickly, and the consumer might not have had time to update the processed_events table? No, because we committed. So the processed_events record is there.

    But what if the broker redelivers the event to a different consumer instance? That instance will check the processed_events table and see it's already processed, so it will skip. So we are safe.

    5. Retries and DLQ

    If the consumer fails to process an event (e.g., business rule violation, or transient error that we want to retry), we might want to retry a few times.

    We can implement retries in the consumer:

    • For transient errors (like database deadlock, network issue), we can retry a few times with backoff.
    • For non-transient errors (like validation failure), we might not want to retry and instead send to a DLQ.

    But note: if we get an event that is invalid (bad data), we don't want to keep retrying. So we need to distinguish between transient and permanent failures.

    Alternatively, we can have a retry mechanism that:

    • When the consumer fails to process (for any reason), it does not ack the event. The broker will redeliver.
    • But if the broker has a limit on redeliveries (like SQS with redrive policy), then after a certain number of redeliveries, it moves to a DLQ.

    So we rely on the broker's redelivery and DLQ mechanism.

    In the consumer, we can also implement our own retry count by storing the retry count in a separate table? But that would require state and might be complex. Alternatively, we can use the broker's redelivery count (if available) and then when it exceeds a threshold, we ack and then write to a separate DLQ (or let the broker move it).

    But note: if we ack the event and then write to a DLQ, we are essentially removing it from the main queue and putting it in a DLQ. That's acceptable.

    Alternatively, we can have the consumer, when it detects a non-transient error, publish the event to a DLQ topic and then ack the original event. But then we are duplicating the event? The DLQ should have the original event and some error information.

    We can design:

    • The consumer has a maximum number of processing attempts (say 3). We can store the attempt number in the processed_events table? But that table is only for successful processing. Alternatively, we can have a processing_attempts table that tracks attempts for events that are being processed. But that's complex.

    Simpler: rely on the broker's redelivery. Many brokers (like SQS) have a redrive policy that moves the message to a DLQ after a certain number of receives. So we configure the broker to do that.

    But what if we are using Kafka? Kafka doesn't have a built-in DLQ. We can have a separate topic for DLQ and the consumer, when it fails (and we don't want to retry), we produce to the DLQ topic and then commit the offset (so we don't see it again). However, note that if we commit the offset and then produce to DLQ, we might lose the event if the consumer crashes after committing but before producing to DLQ? So we must do it atomically? Not really, but we can do:

    • In the same transaction that we commit the offset (or not commit the offset) we produce to DLQ? That's not straightforward.

    Alternatively, we can have a separate process that monitors the consumer's processing and moves events that have been retried too many times to a DLQ. But that's complex.

    Given the complexity, we might choose a broker that supports DLQ natively (like RabbitMQ with dead-letter exchange, or SQS). Or we implement our own DLQ by having the consumer, when it fails after a certain number of attempts, publish the event to a DLQ topic and then ack the original event. But we must be cautious about the ack and the DLQ publish being atomic? We can do:

    • In the consumer, we keep a retry count for each event in memory? But if the consumer restarts, we lose the count. So we need persistent retry count.

    We can store the retry count in the processed_events table? But that table is only for successfully processed events. We need a table for events that are being processed and have failed.

    Alternatively, we can use a separate event_processing_attempts table:

    CREATE TABLE event_processing_attempts (
        event_id UUID PRIMARY KEY,
        attempt_count INT NOT NULL,
        last_error TEXT,
        last_attempted_at TIMESTAMP NOT NULL
    );
    

    Then the consumer:

    1. For each event, check if there's an entry in event_processing_attempts. If not, insert with attempt_count=1. If yes, update attempt_count = attempt_count + 1.
    2. If attempt_count > MAX_RETRIES, then:
      • Publish the event to a DLQ topic (with the error information).
      • Ack the original event.
      • Remove the entry from event_processing_attempts (or leave it for auditing?).
    3. Else, try to process the event (using the same transaction as before for idempotency). If processing fails (transient or non-transient), we do not ack and the broker will redeliver. But note: we are already counting the attempt in the event_processing_attempts table. So we need to update the event_processing_attempts table even if we don't ack? But if we don't ack, the event will be redelivered and we will update the attempt count again. That's okay.

    But note: the event_processing_attempts table must be updated regardless of the outcome? We want to count each delivery attempt. But if we update the table and then the transaction for the business update fails, we rollback the update to event_processing_attempts? That would be wrong because we want to count the attempt even if the business update fails (so that we don't retry indefinitely). So we need to separate the attempt count from the business transaction.

    So:

    • The attempt count is about how many times we have tried to process this event (regardless of success). So we should update the event_processing_attempts table in a separate transaction from the business update.

    But then we have two separate transactions: one for updating the attempt count, and one for business update and marking as processed. This is getting complex.

    Alternatively, we can rely on the broker's redelivery count. Many brokers provide a header or property for the number of times the message has been delivered. We can use that to decide if we should move to DLQ.

    If the broker doesn't provide that, we can store the attempt count in a separate table that is updated independently of the business transaction. For example:

    • When we receive an event (first time or redelivery), we increment the attempt count in the event_processing_attempts table (using an upsert). This is done in a separate, small transaction that we commit immediately. Then we try to process the event. If processing fails, we do not ack, and the broker will redeliver. On redelivery, we again increment the attempt count.

    But note: if the consumer crashes after incrementing the attempt count but before processing, then on restart (or another consumer) we will increment again and process. That's acceptable because we are counting attempts.

    However, if the broker redelivers very quickly, we might have multiple increments for the same delivery attempt? We must be idempotent in incrementing. We can do:

    • Use the event_id and the delivery attempt in the broker? But if the broker doesn't give us a unique delivery attempt ID, we cannot.

    So we might end up with overcounting. But we can set a high MAX_RETRIES (like 10) and overcounting by 1 or 2 is acceptable.

    Alternatively, we can avoid tracking attempts and rely on the broker's DLQ mechanism. Let's assume we use a broker that supports DLQ after a certain number of redeliveries (like RabbitMQ with x-death header, or SQS with redrive policy). Then we don't need to track attempts in the consumer.

    So, to simplify, we assume the broker handles redeliveries and DLQ. The consumer only does:

    • If the event is already processed (by event_id), ack and skip.
    • Else, try to process (with the idempotent transaction). If processing fails (throws an exception), we do not ack. The broker will redeliver (and eventually move to DLQ).

    But note: if the event is invalid (like a validation error), we will keep retrying forever. So we must have a way to stop retrying on permanent failures. We can catch specific exceptions and then, instead of throwing, we can ack and then publish to a DLQ. But then we must be careful to not lose the event.

    So:

    In the consumer:

    1. Check if already processed (using the processed_events table). If yes, ack.
    2. Else, try to process. a. If processing succeeds, record the event as processed and ack. b. If processing fails with a transient error (like network, deadlock), throw the exception (so we don't ack and the broker redelivers). c. If processing fails with a permanent error (like validation), then: - Publish the event to a DLQ topic (with error details) and ack the original event.

    But note: step 2c requires two actions (publish to DLQ and ack) to be atomic? If we publish to DLQ and then crash before ack, then the original event will be redelivered and we will try to publish to DLQ again. That might be acceptable (duplicate in DLQ) but we might want to avoid that. We can use a transaction for the DLQ publish and the ack? But the ack is with the broker, and the publish to DLQ is to another topic. We cannot have a distributed transaction easily.

    Alternatively, we can produce the DLQ event and then ack the original. If we crash after producing to DLQ but before ack, then on restart we will get the original event again and produce to DLQ again (duplicate in DLQ). That's acceptable because DLQ is for analysis and duplicates are okay.

    But what if we ack the original event and then fail to produce to DLQ? Then the event is lost from the main queue but not in DLQ. That's bad. So we must produce to DLQ before acking the original. And if producing to DLQ fails, we should not ack the original (so it will be redelivered and we try again). That's safe.

    So:

    • For permanent errors, we try to produce to DLQ. If that succeeds, then ack the original. If producing to DLQ fails, we don't ack (so the original will be redelivered and we try again).

    But note: if producing to DLQ fails due to a permanent error (like invalid data for DLQ? unlikely because we are just forwarding the event) then we might be stuck. But we can catch that and then maybe log and ack anyway? That would lose the event. So we should retry the DLQ publish as well? But then we are back to the same problem.

    Alternatively, we can have a separate process that monitors the main queue for events that have been retried too many times (using the broker's redelivery count) and moves them to DLQ. That way, the consumer doesn't have to worry about DLQ; it just fails and lets the broker redeliver until the broker's DLQ mechanism kicks in.

    Given the complexity, I think the best is to rely on the broker's DLQ mechanism and set a maximum redelivery count. Then the consumer only does:

    • Idempotent processing and ack on success.
    • On failure (any exception), do not ack. The broker will redeliver (and eventually move to DLQ).

    But then we have the problem of permanent failures causing infinite redeliveries until the broker's DLQ threshold. That's acceptable because the broker will eventually move it to DLQ.

    However, if we have a high volume of events, we might not want to retry a permanent failure 100 times. So we might want to set a low redelivery count (like 3) and then let the broker move to DLQ.

    But note: the broker's redelivery count is per consumer? Or per message? It depends on the broker.

    Given the trade-offs, I propose:

    • Use a broker that supports dead-lettering after a certain number of redeliveries (like RabbitMQ, SQS, or Kafka with a custom solution using a separate process).
    • The consumer does not implement its own retry count. It just fails on any exception and relies on the broker's redelivery and DLQ.

    But then we have to be cautious: if the consumer fails due to a transient error, we want to retry. The broker will do that. And if it fails due to a permanent error, the broker will eventually move to DLQ. So we are covered.

    However, we must make sure that the consumer's idempotent processing is safe even for permanent failures. That is, if we get a permanent failure, we don't want to keep trying because it will always fail. But the broker will eventually stop trying. So it's okay.

    But what if the event is invalid and we get it again and again? The broker will redeliver until the DLQ threshold. That's acceptable because we have a DLQ to investigate.

    So the consumer code:

    def on_event(event):
        if is_already_processed(event.event_id):
            ack(event)
            return
    
        try:
            # Process the event (update business state and mark as processed in the same transaction)
            process_event(event)
            ack(event)
        except Exception as e:
            # Log the error and do not ack. The broker will redeliver.
            log_error(e, event)
            # We don't ack, so the broker will redeliver.
    

    But note: if the event is a duplicate (already processed) we ack and skip. That's safe.

    6. Observability

    We need to monitor:

    • Producer: rate of events produced, outbox table size, outbox poller lag (how many events are waiting to be published).
    • Broker: queue/topic depth, publish/consume rates, errors.
    • Consumer: processing rate, processing latency, number of duplicates, errors, DLQ rate.

    We should also have tracing (distributed tracing) to track an event from producer to consumer.

    Implementation:

    • Use structured logging.

    • Emit metrics (e.g., using Prometheus client libraries) for:

      • Events produced
      • Events consumed
      • Duplicate events detected
      • Events processed successfully
      • Events sent to DLQ
      • Processing latency (time from when the event was produced to when it was processed, and also consumer processing time)
      • Outbox table size (number of unpublished events)
      • Retries (if we track them) or redelivery count (from broker metadata)
    • Tracing: We can generate a trace ID at the producer and pass it through the event headers. Then the consumer can continue the trace.

    Summary of the Architecture

    1. Producer Service:

      • Handles a command or business transaction.
      • In the same transaction, writes the event to the outbox table.
      • Commits the transaction.
    2. Outbox Poller:

      • Runs periodically (or uses the database's notify mechanism to react quickly).
      • Fetches a batch of unpublished events (with published_at NULL) and locks them (to avoid multiple pollers).
      • Publishes each event to the broker (topic/queue). If publishing fails, leave the event as unpublished (and maybe retry later).
      • On successful publish, sets published_at to the current time (or deletes the record).
    3. Message Broker:

      • Holds the events until consumers consume them.
      • Configured with a dead-letter queue (or topic) after a certain number of redeliveries.
    4. Consumer Service:

      • Subscribes to the topic/queue with manual acknowledgments.
      • For each event: a. Check the processed_events table for the event_id. If found, ack and skip. b. Else, in a transaction: - Insert into processed_events (if duplicate, rollback and ack). - Apply the business update (using the event payload). - Commit. c. Ack the event.
      • If any exception occurs during processing, do not ack. The broker will redeliver.
    5. Dead-Letter Queue:

      • Configured in the broker. Events that are redelivered too many times end up here.
    6. Observability:

      • Logs with correlation IDs (trace ID) at each step.
      • Metrics for rates, latencies, errors, DLQ.
      • Traces (using OpenTelemetry, for example) to follow an event from producer to consumer.

    Considerations and Trade-offs

    • Idempotency: The consumer must be idempotent. We use the event_id to deduplicate. The event_id must be unique and generated by the producer.
    • Outbox Table Growth: We need to clean up the outbox table. We can delete entries after a certain period (e.g., 30 days) or after they are published and we are sure the event has been processed? But note: we might need the outbox for replaying events if we rebuild a consumer? Not really, because the consumer has its own processed_events. But the outbox is only for ensuring delivery. We can archive or delete old outbox records.
    • Consumer Deduplication Table Growth: The processed_events table will grow indefinitely. We can prune events that are older than a certain period (if we don't need to replay). But note: if we prune, we lose the ability to detect duplicates for old events. But if we assume that events are only relevant for a certain time (e.g., 1 year), we can prune. Alternatively, we can use a separate service to compact the table (like using a bloom filter for very old events? but that's complex). We can also partition the table by time and drop old partitions.
    • Ordering: If we need ordering per aggregate, we must ensure that the broker preserves order for events with the same aggregate ID (e.g., by partitioning by aggregate ID in Kafka). And the consumer must process events in order? Not necessarily if the events are commutative? But if they are not, we need to process in order. How to ensure in-order processing when using the deduplication table? We can store a sequence number per aggregate in the business state and then only apply an event if its sequence number is exactly next. That requires the producer to assign sequence numbers per aggregate. Alternatively, we can rely on the broker to order by partition and then the consumer to process one partition at a time. But if we have multiple consumers for the same partition, we might have out-of-order processing? Not if we use one consumer per partition. So we can have one consumer per partition (in Kafka) and then we process in order. But then we cannot scale per partition. Alternatively, we can have multiple consumers but use a distributed lock per aggregate? That's complex.

    Given the complexity of ordering, we might design the events to be commutative (so order doesn't matter) or use a single partition per aggregate (which limits scalability). This is a separate problem.

    Conclusion

    We have designed an event-driven architecture with exactly-once semantics by:

    • Using the outbox pattern to ensure events are published reliably.
    • Using idempotent consumers with a processed_events table to deduplicate.
    • Relying on the broker's redelivery and DLQ for retries and dead-lettering.
    • Adding observability with logging, metrics, and tracing.

    This design handles producer failures (via outbox), consumer failures (via redelivery and idempotency), and provides a way to handle permanent failures (DLQ). It also allows for monitoring and debugging.

    Note: The exact implementation details (like database schema, broker configuration, etc.) will depend on the specific technologies chosen. I'll design a comprehensive event-driven architecture with exactly-once semantics. This is a complex requirement that needs multiple patterns working together.

    Architecture Overview

    ┌─────────────┐    ┌──────────────┐    ┌──────────────┐    ┌─────────────┐
    │   Service   │───▶│   Outbox     │───▶│   Message    │───▶│  Consumer   │
    │   (Producer)│    │   Table      │    │   Broker     │    │             │
    └─────────────┘    └──────────────┘    └──────────────┘    └─────────────┘
                             │                       │                   │
                             ▼                       ▼                   ▼
                     ┌──────────────┐        ┌─────────────┐   ┌─────────────┐
                     │  Outbox      │        │  Dead-Letter│   │  Idempotency│
                     │  Poller      │        │  Queue      │   │  Store      │
                     └──────────────┘        └─────────────┘   └─────────────┘
    

    Core Components

    1. Producer Service with Outbox Pattern

    // Transactional write to outbox table
    public class EventProducer {
        @Transactional
        public void processBusinessTransaction(Command command) {
            // 1. Update business state
            businessRepository.updateState(command);
            
            // 2. Write event to outbox in SAME transaction
            OutboxEvent event = OutboxEvent.builder()
                .eventId(UUID.randomUUID().toString())
                .aggregateId(command.getAggregateId())
                .eventType("OrderCreated")
                .payload(toJson(command))
                .createdAt(Instant.now())
                .build();
                
            outboxRepository.save(event);
            // Transaction commits - both business state and event are persisted atomically
        }
    }
    

    Outbox Table Schema:

    CREATE TABLE outbox_events (
        event_id VARCHAR(36) PRIMARY KEY,
        aggregate_id VARCHAR(36) NOT NULL,
        event_type VARCHAR(100) NOT NULL,
        payload JSONB NOT NULL,
        created_at TIMESTAMP NOT NULL,
        published_at TIMESTAMP,
        retry_count INT DEFAULT 0,
        last_attempt_at TIMESTAMP,
        INDEX idx_published_created (published_at, created_at)
    );
    

    2. Outbox Poller (Reliable Publisher)

    @Component
    public class OutboxPoller {
        @Scheduled(fixedDelay = 1000) // Configurable
        public void publishOutboxEvents() {
            List<OutboxEvent> events = outboxRepository
                .findUnpublished(100) // Batch size
                .stream()
                .peek(event -> event.setRetryCount(event.getRetryCount() + 1))
                .collect(Collectors.toList());
                
            for (OutboxEvent event : events) {
                try {
                    // 1. Publish to message broker
                    Message<String> message = MessageBuilder
                        .withPayload(event.getPayload())
                        .setHeader("event-id", event.getEventId())
                        .setHeader("aggregate-id", event.getAggregateId())
                        .setHeader("event-type", event.getEventType())
                        .build();
                        
                    kafkaTemplate.send(message).get(10, TimeUnit.SECONDS);
                    
                    // 2. Mark as published
                    event.setPublishedAt(Instant.now());
                    outboxRepository.save(event);
                    
                } catch (Exception e) {
                    event.setLastAttemptAt(Instant.now());
                    outboxRepository.save(event);
                    
                    if (event.getRetryCount() > MAX_RETRIES) {
                        moveToDlq(event, e);
                    }
                }
            }
        }
    }
    

    3. Message Broker Configuration

    # Kafka configuration example
    spring:
      kafka:
        bootstrap-servers: localhost:9092
        producer:
          acks: all  # Wait for all replicas
          retries: 3
          enable-idempotence: true  # Exactly-once at producer
          max-in-flight-requests-per-connection: 5
        consumer:
          enable-auto-commit: false  # Manual acknowledgment
          isolation-level: read_committed  # Read only committed messages
    

    4. Idempotent Consumer with Deduplication

    @Component
    public class EventConsumer {
        
        @KafkaListener(topics = "orders", 
                       containerFactory = "kafkaListenerContainerFactory")
        public void handleEvent(ConsumerRecord<String, String> record, 
                               Acknowledgment acknowledgment) {
            
            String eventId = record.headers()
                .lastHeader("event-id")
                .value()
                .toString();
            
            // 1. Deduplication check
            if (idempotencyRepository.existsById(eventId)) {
                log.info("Duplicate event detected: {}", eventId);
                acknowledgment.acknowledge();
                return;
            }
            
            try {
                // 2. Process event in transaction
                processEvent(record.value());
                
                // 3. Record idempotency key
                idempotencyRepository.save(
                    IdempotencyKey.builder()
                        .eventId(eventId)
                        .processedAt(Instant.now())
                        .build()
                );
                
                acknowledgment.acknowledge();  // Commit offset only after success
                
            } catch (Exception e) {
                log.error("Processing failed for event: {}", eventId, e);
                // Don't ack - will be redelivered
                throw new RuntimeException(e);
            }
        }
        
        @Transactional
        protected void processEvent(String eventJson) {
            // Business logic here
            OrderEvent event = objectMapper.readValue(eventJson, OrderEvent.class);
            
            // Check if already processed (defense in depth)
            if (orderRepository.existsByEventId(event.getEventId())) {
                return;
            }
            
            // Apply business logic
            orderService.applyEvent(event);
        }
    }
    

    Idempotency Store Schema:

    CREATE TABLE idempotency_keys (
        event_id VARCHAR(36) PRIMARY KEY,
        processed_at TIMESTAMP NOT NULL,
        INDEX idx_processed_at (processed_at)
    );
    
    CREATE TABLE orders (
        order_id VARCHAR(36) PRIMARY KEY,
        event_id VARCHAR(36) UNIQUE NOT NULL,  -- Enforce idempotency at DB level
        -- ... other fields
        FOREIGN KEY (event_id) REFERENCES idempotency_keys(event_id)
    );
    

    5. Dead Letter Queue (DLQ) Handling

    @Component
    public class DlqHandler {
        
        @KafkaListener(topics = "orders-dlq")
        public void handleDlq(ConsumerRecord<String, String> record) {
            String originalEventId = record.headers()
                .lastHeader("original-event-id")
                .value()
                .toString();
                
            DlqEvent dlqEvent = DlqEvent.builder()
                .originalEventId(originalEventId)
                .payload(record.value())
                .reason(extractReason(record))
                .receivedAt(Instant.now())
                .retries(getRetryCount(record))
                .build();
                
            // 1. Alert/notify
            alertService.sendAlert(dlqEvent);
            
            // 2. Store for analysis
            dlqRepository.save(dlqEvent);
            
            // 3. Optional: manual replay capability
            if (shouldAutoReplay(dlqEvent)) {
                replayToOriginalTopic(dlqEvent);
            }
        }
        
        private void moveToDlq(OutboxEvent event, Exception e) {
            kafkaTemplate.send("orders-dlq", 
                MessageBuilder.withPayload(event.getPayload())
                    .setHeader("original-event-id", event.getEventId())
                    .setHeader("error-message", e.getMessage())
                    .setHeader("retry-count", event.getRetryCount())
                    .build()
            );
        }
    }
    

    6. Observability & Monitoring

    a) Metrics (Micrometer/Prometheus):

    @MeterTag("event.type")
    @MeterTag("consumer.group")
    public class EventMetrics {
        
        @EventListener
        public void onEventProcessed(EventProcessedEvent event) {
            Counter.builder("events.processed")
                .tag("status", "success")
                .tag("event.type", event.getEventType())
                .increment();
                
            Timer.builder("event.processing.latency")
                .tag("event.type", event.getEventType())
                .record(event.getDuration(), TimeUnit.MILLISECONDS);
        }
        
        @EventListener
        public void onDuplicateDetected(DuplicateEventEvent event) {
            Counter.builder("events.duplicates.detected")
                .increment();
        }
        
        @EventListener
        public void onDlqEvent(DlqEvent event) {
            Counter.builder("events.dlq")
                .tag("reason", event.getReason())
                .increment();
        }
    }
    

    b) Distributed Tracing (OpenTelemetry):

    @Bean
    public KafkaTracing kafkaTracing() {
        return KafkaTracing.newBuilder()
            .build();
    }
    
    // Producer
    Span span = tracer.newTrace()
        .name("publish-event")
        .start();
        
    try (Scope scope = span.makeCurrent()) {
        span.setAttribute("event.id", eventId);
        kafkaTemplate.send(message);
    } finally {
        span.end();
    }
    
    // Consumer - automatic context propagation via headers
    

    c) Structured Logging:

    @Slf4j
    @Component
    public class LoggingAspect {
        
        @Around("execution(* com.example.consumer..*(..))")
        public Object logEventProcessing(ProceedingJoinPoint joinPoint) throws Throwable {
            String eventId = extractEventId(joinPoint);
            
            MDC.put("eventId", eventId);
            MDC.put("spanId", Span.current().getSpanContext().getSpanId());
            
            log.info("Processing event: type={}", 
                joinPoint.getSignature().getName());
                
            try {
                return joinPoint.proceed();
            } finally {
                MDC.clear();
            }
        }
    }
    

    d) Health Checks:

    @Component
    public class EventSystemHealthIndicator implements HealthIndicator {
        
        @Override
        public Health health() {
            Health.Builder builder = Health.up();
            
            // Check outbox lag
            long outboxSize = outboxRepository.countUnpublished();
            builder.withDetail("outbox.size", outboxSize);
            if (outboxSize > THRESHOLD) {
                builder.down().withException(new Exception("Outbox lag high"));
            }
            
            // Check DLQ size
            long dlqSize = dlqRepository.countUnprocessed();
            builder.withDetail("dlq.size", dlqSize);
            
            // Check consumer lag
            Map<String, Long> consumerLag = kafkaConsumerLagChecker.getLag();
            builder.withDetail("consumer.lag", consumerLag);
            
            return builder.build();
        }
    }
    

    7. Retry Strategies & Circuit Breakers

    @Configuration
    public class RetryConfig {
        
        @Bean
        public RetryTemplate eventProcessingRetryTemplate() {
            return RetryTemplate.builder()
                .maxAttempts(3)
                .exponentialBackoff(1000, 2, 10000)
                .retryOn(Exception.class)
                .withListener(new RetryListener() {
                    @Override
                    public <T, E extends Throwable> void onError(
                        RetryContext context, RetryCallback<T, E> callback, Throwable throwable) {
                        
                        log.warn("Retry attempt {} for event: {}", 
                            context.getRetryCount(), 
                            context.getAttribute("eventId"));
                    }
                })
                .build();
        }
        
        @Bean
        public CircuitBreakerFactory circuitBreakerFactory() {
            return new Resilience4JCircuitBreakerFactory();
        }
    }
    

    Deployment & Operations

    Infrastructure Requirements:

    1. Message Broker: Kafka (preferred) or RabbitMQ with:

      • Replication factor ≥ 3
      • Min.insync.replicas = 2
      • Unclean leader election = false
    2. Database:

      • Outbox table: Index on (published_at, created_at) for poller queries
      • Idempotency table: Primary key on event_id
      • Regular cleanup (TTL) of old records (30-90 days)
    3. Monitoring Stack:

      • Prometheus + Grafana for metrics
      • ELK/Loki for logs
      • Jaeger/Zipkin for traces
      • Alertmanager for alerts

    Key Configuration Parameters:

    event-processing:
      outbox:
        poll-interval-ms: 1000
        batch-size: 100
      consumer:
        max-retries: 3
        dlq-after-retries: true
      idempotency:
        ttl-days: 90  # Keep IDs for 90 days
      broker:
        request-timeout-ms: 30000
        delivery-timeout-ms: 120000
    

    Failure Scenarios & Handling

    ScenarioHandling Mechanism
    Producer crashes after DB commit but before outbox writeTransaction rolls back - no event
    Outbox poller crashes after publish but before updateEvent republished (idempotent consumer handles duplicate)
    Consumer crashes after DB commit but before ackEvent redelivered (idempotency check prevents double processing)
    Broker loses messageOutbox poller will republish (since published_at not set)
    Consumer processes but fails to record idempotencyTransaction rolls back - event redelivered
    Duplicate delivery from brokerIdempotency store prevents reprocessing

    Exactly-Once Guarantees Summary

    1. Producer → Broker: Outbox pattern + idempotent producer (Kafka) = at-least-once with duplicates
    2. Broker → Consumer: Manual acknowledgment + idempotent consumer = exactly-once
    3. End-to-End: Database transactions + idempotency keys = exactly-once semantics

    Scaling Considerations

    1. Outbox Poller: Multiple instances possible with row locking (SELECT ... FOR UPDATE SKIP LOCKED)
    2. Consumers: Increase partitions in Kafka for parallel processing (per aggregate ID partitioning for ordering)
    3. Idempotency Store: Shard by event_id prefix or use Redis for high-throughput scenarios
    4. DLQ Processing: Separate consumer group for DLQ analysis/replay

    Testing Strategy

    1. Unit Tests: Idempotency logic, transaction boundaries
    2. Integration Tests: With embedded Kafka, test failover scenarios
    3. Chaos Testing: Simulate crashes at each stage (producer, poller, consumer)
    4. Load Testing: Verify outbox polling doesn't fall behind
    5. Recovery Testing: Restart services and verify no duplicate processing

    This architecture provides strong exactly-once semantics while maintaining scalability and observability. The key is combining multiple patterns where each handles a specific failure mode, creating a robust system that gracefully handles all failure scenarios.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.