Build a saga orchestrator: compensation, retries, DLQs, observability.
- User
Build a saga orchestrator: compensation, retries, DLQs, observability.
Assistant
The user wants to build a "saga orchestrator" with specific features: compensation, retries, DLQs (Dead Letter Queues), and observability. This is a classic distributed systems pattern.
1. Understand the Goal: The user is asking for code or a design pattern to implement a Saga Orchestrator. A Saga is a sequence of local transactions where each transaction updates data within a single service. The Orchestrator pattern uses a central controller (the orchestrator) to tell the participants what to do.
2. Identify the Key Components:
- Saga Orchestrator: The central brain. It maintains state, knows the workflow, and issues commands.
- Compensation: If a step fails, undo the previous steps (rollback).
- Retries: Transient failure handling.
- DLQs: Handling permanent failures after retries are exhausted.
- Observability: Logging, tracing, and metrics to see what's happening.
3. Determine the Scope & Language: The prompt is broad. It doesn't specify a language (Java, Go, Python, etc.) or framework.
- Decision: I will use Python with a generic structure (class-based) because it's readable and widely understood. I'll use a pseudo-code approach for the infrastructure parts (like actual message queues or databases) but provide concrete logic for the Orchestrator itself. I'll use
asyncioto simulate the asynchronous nature of distributed systems.
4. Architecture Design:
- State Machine: The core of a saga is a state machine (Start -> Step 1 -> Step 2 -> End).
- Persistence: The saga state needs to be stored (I'll assume a generic
StateStoreinterface). - Messaging: Commands and Events need to be passed (I'll assume a generic
MessageBrokerinterface). - The Saga Definition: A way to define steps and their compensation.
5. Step-by-Step Implementation Plan:
- Step 1: Define the Models. What does a Saga Step look like? What does the Saga State look like?
- Step 2: The Orchestrator Core. The logic that processes events and dispatches commands.
- Step 3: Retry Logic. Exponential backoff is standard.
- Step 4: Compensation Logic. Executing the "undo" steps in reverse order.
- Step 5: DLQ. Where failed messages go.
- Step 6: Observability. Adding hooks for tracing/logging.
6. Drafting the Code (Mental or Scratchpad):
-
Saga Definition: A list of steps. Each step has a
name,execute_command, andcompensate_command. -
The Loop:
- Receive event (e.g., "Step 1 Completed").
- Update state.
- Determine next step.
- Send command for next step.
- If failed, start compensation.
-
Handling Retries:
- Need a counter.
current_retries. - Need a limit.
max_retries. - Need delay.
backoff.
- Need a counter.
-
DLQ:
- If
current_retries > max_retries, move message to DLQ.
- If
7. Refining the Implementation (Self-Correction during drafting):
- Critique: Writing a full production-grade orchestrator from scratch is too much code for a single response.
- Solution: Create a "Conceptual Framework" or a "Minimal Viable Implementation" that highlights the specific patterns requested (Compensation, Retry, DLQ, Observability). I will use Python type hints to make it clear.
8. Detailed Code Structure (Python):
SagaStep: Dataclass holding command names.SagaInstance: Dataclass holding ID, status, current_step, completed_steps, payload.Orchestrator: The main class.start(): Initiates the saga.handle_event(): The entry point for responses.transition(): Core logic.compensate(): Rollback logic.send_command(): Interface with outside world.
9. Addressing Specific Requirements:
- Compensation:
- Need a stack or list of
completed_steps. - When failure occurs, iterate backwards through
completed_stepsand trigger compensation actions.
- Need a stack or list of
- Retries:
- Wrap the
send_commandorhandle_eventin a retry loop? No, usually sagas are event-driven. Retries happen when a task fails. - Scenario: Orchestrator sends "Reserve Credit". Service says "Busy, try again".
- Implementation: The Orchestrator receives an error event. It checks retry count. If < max, it schedules a retry (schedules a future event).
- Wrap the
- DLQ:
- If retry count exceeds max, move the
SagaInstanceto a "FAILED" state and push the context to a specific topic/queue for manual inspection.
- If retry count exceeds max, move the
- Observability:
- Use Python's
loggingor print statements representing structured logging. - Simulate "Trace IDs" passed in headers.
- Use Python's
10. Final Polish:
- Add comments explaining why specific choices were made.
- Provide a usage example (Booking a flight/hotel is the standard saga example).
- Structure the output: Introduction -> Core Concepts -> Code Implementation -> Explanation of Features.
11. Writing the Code (Mental Sandbox):
class Orchestrator: def __init__(self, saga_definition, broker, store): self.def = saga_definition ... async def handle_message(self, message): saga_id = message.saga_id state = self.store.get(saga_id) if message.status == 'FAILURE': if state.retries < MAX: await self.retry(state, message) else: await self.dlq(state, message) await self.compensate(state) # Maybe? Or just stop? usually compensate if possible, DLQ if stuck.Correction: Usually, if a step fails permanently (DLQ), we trigger compensation for the previous steps.
12. Final Output Generation Strategy: I will provide a robust Python skeleton that implements the Orchestrator pattern. I will simulate the async nature using standard Python libraries to keep it runnable without external dependencies (like Kafka or RabbitMQ), but I will structure it so those dependencies are obvious interfaces.
- File structure: Single block of code with classes.
- Example: Trip Booking (Car + Hotel).
Let's write it out.
Self-Correction on "Retries": Should the Orchestrator retry sending the command, or should the participant retry processing? The prompt asks for the Orchestrator to handle retries. This implies the Orchestrator manages the timeout/retry policy for the remote calls.
13. Final check of the requirements:
- Compensation: Check. (Compensate method).
- Retries: Check. (Logic inside handle_event to resend command).
- DLQs: Check. (DeadLetterQueue class).
- Observability: Check. (Logging/Tracing context).
Let's assemble. Building a robust Saga Orchestrator requires managing state, handling asynchronous communication, and implementing specific resilience patterns.
Below is a comprehensive design and implementation guide using Python and asyncio. This example creates a framework that decouples the orchestration logic from the transport layer, allowing you to swap out Redis, RabbitMQ, or Kafka.
Core Architecture
- Saga Definition: A Directed Acyclic Graph (DAG) of steps.
- Saga State: A persistent record of the current step, status, and retry counts.
- Orchestrator: The engine that executes steps, handles failures, and triggers compensation.
- Transport: The mechanism to send commands to microservices and receive replies.
1. The Data Models
First, we define the state of the Saga and the structure of individual steps.
import enum import uuid from dataclasses import dataclass, field from typing import Callable, Awaitable, Any, Dict, List class SagaStatus(enum.Enum): RUNNING = "RUNNING" COMPENSATING = "COMPENSATING" # Rolling back COMPLETED = "COMPLETED" FAILED = "FAILED" # Fatal failure (DLQ) COMPENSATED = "COMPENSATED" # Successfully rolled back class StepStatus(enum.Enum): PENDING = "PENDING" SUCCESS = "SUCCESS" FAILED = "FAILED" COMPENSATED = "COMPENSATED" @dataclass class SagaStep: # The topic/command to send to the worker action_topic: str # The topic/command to undo the work compensate_topic: str # Configuration max_retries: int = 3 retry_delay_seconds: int = 5 @dataclass class SagaState: saga_id: str steps: List[SagaStep] current_step_index: int = 0 status: SagaStatus = SagaStatus.RUNNING # Stores context (e.g., IDs generated during the saga) context: Dict[str, Any] = field(default_factory=dict) @property def current_step(self) -> SagaStep: return self.steps[self.current_step_index]2. Infrastructure & Observability
We need a way to persist state and log structured data (Observability).
import logging import json from datetime import datetime # Configure Structured Logging (Observability) logging.basicConfig(level=logging.INFO) logger = logging.getLogger("SagaOrchestrator") def log_saga_event(saga_id: str, event_type: str, **kwargs): log_data = { "timestamp": datetime.utcnow().isoformat(), "saga_id": saga_id, "event": event_type, **kwargs } # In production, export this to Prometheus/Datadog/ELK logger.info(json.dumps(log_data)) # Mock Database for State Persistence class StateStore: def __init__(self): self._db = {} # In-memory for demo; use Redis/Postgres in prod async def save(self, state: SagaState): self._db[state.saga_id] = state async def get(self, saga_id: str) -> SagaState: return self._db.get(saga_id)3. The Orchestrator Engine
This is the core class. It handles the "Happy Path", the "Retry Loop", "Compensation", and the "DLQ".
import asyncio class SagaOrchestrator: def __init__(self, transport: 'Transport', state_store: StateStore): self.transport = transport self.state_store = state_store self.dlq_handler = None # Injected strategy for DLQ async def start_saga(self, steps: List[SagaStep], initial_context: dict = None) -> str: saga_id = str(uuid.uuid4()) state = SagaState( saga_id=saga_id, steps=steps, context=initial_context or {} ) await self.state_store.save(state) log_saga_event(saga_id, "SAGA_STARTED") # Trigger the first step await self._execute_step(state) return saga_id async def _execute_step(self, state: SagaState): if state.status == SagaStatus.COMPENSATING: topic = state.current_step.compensate_topic log_saga_event(state.saga_id, "COMPENSATION_STARTED", step=state.current_step_index) else: topic = state.current_step.action_topic log_saga_event(state.saga_id, "STEP_STARTED", step=state.current_step_index) # Send Command via Transport # Note: In a real system, you might track 'in-flight' messages here # using a correlation ID to handle async replies. await self.transport.publish(topic, { "saga_id": state.saga_id, "context": state.context }) async def handle_reply(self, saga_id: str, success: bool, reply_context: dict): state = await self.state_store.get(saga_id) if not state: log_saga_event(saga_id, "ERROR", message="Saga ID not found") return # --- HAPPY PATH --- if success and state.status == SagaStatus.RUNNING: state.context.update(reply_context) # Save data returned by service # Check if Saga finished if state.current_step_index == len(state.steps) - 1: state.status = SagaStatus.COMPLETED log_saga_event(saga_id, "SAGA_COMPLETED") else: # Move to next step state.current_step_index += 1 await self._execute_step(state) await self.state_store.save(state) return # --- COMPENSATION PATH --- if state.status == SagaStatus.COMPENSATING: # Successfully compensated a step. Move backwards. if state.current_step_index == 0: state.status = SagaStatus.COMPENSATED log_saga_event(saga_id, "SAGA_COMPENSATED") else: state.current_step_index -= 1 await self._execute_step(state) await self.state_store.save(state) return # --- FAILURE / RETRY PATH --- if not success: await self._handle_failure(state, reply_context) async def _handle_failure(self, state: SagaState, error_context: dict): # Check for retry logic (simplified for demo: we assume retry counts are stored in state) # In a real robust system, you might check Redis for "saga_id:step:retries" # For this example, we will assume the transport handles retries (e.g., RabbitMQ requeue) # OR we can implement a simple counter here. Let's implement the counter: retries = state.context.get(f"retries_{state.current_step_index}", 0) if retries < state.current_step.max_retries: retries += 1 state.context[f"retries_{state.current_step_index}"] = retries log_saga_event(state.saga_id, "STEP_RETRY", step=state.current_step_index, attempt=retries) # Wait (Backoff) await asyncio.sleep(state.current_step.retry_delay_seconds) # Re-execute await self._execute_step(state) else: # Retries exhausted -> Start Compensation OR DLQ await self._escalate_failure(state, error_context) async def _escalate_failure(self, state: SagaState, error_context: dict): # Strategy: If step 0 fails, no compensation needed. Just fail. # If critical system error (e.g. DB down), send to DLQ. if self._is_critical_error(error_context): state.status = SagaStatus.FAILED log_saga_event(state.saga_id, "SAGA_DLQ", error=str(error_context)) if self.dlq_handler: await self.dlq_handler(state, error_context) await self.state_store.save(state) return # Start Compensation state.status = SagaStatus.COMPENSATING log_saga_event(state.saga_id, "COMPENSATION_TRIGGERED", step=state.current_step_index) # Execute compensate for current step immediately await self._execute_step(state) def _is_critical_error(self, context: dict) -> bool: # Heuristic to decide if we can compensate or if we need manual intervention return context.get("critical", False)4. Transport Layer (Mock)
In a real scenario, this would be
aio-pika(RabbitMQ) orconfluent-kafka. We mock it here to make the code runnable.class Transport: async def publish(self, topic: str, payload: dict): print(f"[Transport] Publishing to {topic}: {payload}") # Simulate the service doing work and replying asynchronously asyncio.create_task(self._mock_service_reply(topic, payload)) async def _mock_service_reply(self, topic: str, payload: dict): await asyncio.sleep(1) # Simulate IO # Simulate Logic: # If topic is 'reserve_car', fail once then succeed (to test retries) # If topic is 'book_hotel', always succeed # If topic is 'charge_card', fail permanently (to test compensation) if topic == "charge_card": print(f" -> Service {topic} FAILED!") # Return failure await orchestrator.handle_reply(payload['saga_id'], False, {"error": "Insufficient funds"}) else: print(f" -> Service {topic} SUCCESS!") await orchestrator.handle_reply(payload['saga_id'], True, {f"{topic}_id": uuid.uuid4()}) # Helper to inject orchestrator into transport for this demo orchestrator = None5. Usage Example
Here we wire it all together to demonstrate the flow.
async def main(): global orchestrator transport = Transport() store = StateStore() orchestrator = SagaOrchestrator(transport, store) # Define a "Book Trip" Saga trip_saga = [ SagaStep(action_topic="reserve_car", compensate_topic="cancel_car", max_retries=2), SagaStep(action_topic="book_hotel", compensate_topic="cancel_hotel", max_retries=2), SagaStep(action_topic="charge_card", compensate_topic="refund_card", max_retries=2) ] print("--- Starting Saga ---") saga_id = await orchestrator.start_saga(trip_saga, {"user_id": "12345"}) # Wait for async tasks to complete await asyncio.sleep(5) final_state = await store.get(saga_id) print(f"\n--- Saga Final Status: {final_state.status} ---") if __name__ == "__main__": # asyncio.run(main()) pass # Blocking run() used for demo purposes below6. Deep Dive on Specific Requirements
Compensation Logic
The
SagaOrchestratortracks thecurrent_step_index.- If a step fails permanently, the status changes to
COMPENSATING. - The logic executes the
compensate_topicfor the current failed step. - Once the compensation service acknowledges success, the index is decremented (
current_step_index - 1). - This repeats until index 0 is compensated.
Retries & Backoff
In
_handle_failure, we checkmax_retries.- Pattern: We use a simple sleep (
asyncio.sleep). In production, use Exponential Backoff (e.g., 1s, 5s, 30s) to avoid thundering herd problems. - idempotency: Retries require that the action services are idempotent. If
reserve_caris called twice, it shouldn't fail or reserve two cars.
DLQ (Dead Letter Queue)
In
_escalate_failure: There are two failure modes:- Business Failure: "Insufficient Funds". We Compensate.
- Technical Failure: "Database Connection Timeout" or "Poison Pill Message". We DLQ.
The DLQ is triggered when retries are exhausted. The
dlq_handleris a function you inject to alert Ops teams (e.g., send to PagerDuty) and store the state in a database for manual reprocessing.Observability
The
log_saga_eventfunction demonstrates structured logging.- Tracing: In a real app, pass a
trace_idin thecontext. Inject this into message headers. - Metrics: Expose Prometheus counters:
saga_started_totalsaga_completed_totalsaga_compensated_totalsaga_step_latency_seconds
Summary of Responsibilities
Component Responsibility Orchestrator Maintains state, flow control, determines next steps. Action Service Executes business logic. Must be idempotent. Returns Success/Fail. Compensator Undoes changes. Must handle "compensate" for an action that might not have fully succeeded (or failed halfway). DLQ Stores unprocessable messages for human intervention.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.