Create an async job platform with idempotency, ordering keys, and backpressure.
- User
Create an async job platform with idempotency, ordering keys, and backpressure.
Assistant
The user wants me to create an async job platform. Key features requested:
- Idempotency: Submitting the same job (based on an idempotency key) multiple times shouldn't result in duplicate processing.
- Ordering Keys: Jobs with the same ordering key should be processed sequentially, in the order they were received.
- Backpressure: The system shouldn't collapse if producers submit jobs faster than consumers can process them. It needs a mechanism to slow down or buffer inputs.
High-Level Architecture:
- API Layer: Entry point for clients to submit jobs.
- Queue/Broker: Needs to support the specific requirements (ordering, idempotency checks). Standard message brokers (Kafka, RabbitMQ, SQS) handle some of this, but "ordering keys" often require specific partitioning, and "idempotency" is often an application-level concern or requires a specific DB check. "Backpressure" is a flow control mechanism.
- Worker Layer: Consumes jobs and executes them.
- State Store: To track job status and idempotency keys.
Technology Stack Choice:
- Language: Python (very common for this type of task,
asyncioworks well). - Broker: Redis is a good choice here. It's fast, supports lists/streams for queues, has atomic operations for idempotency checks (SETNX), and is easy to set up locally.
- Framework: FastAPI for the API (standard for modern Python async apps).
Implementing the Features:
-
Idempotency:
- How? Use a unique
Idempotency-Keyheader or payload field. - Storage: Redis.
- Logic: Before queuing, check if the key exists in Redis.
- If exists: Return the existing job status/ID (or just "Accepted").
- If not exists: Create job, set key in Redis with a TTL.
- How? Use a unique
-
Ordering Keys:
-
Concept: Jobs with
ordering_key="A"must run one after another. Jobs withordering_key="B"can run in parallel with "A". -
Implementation:
- A single global queue breaks ordering for all jobs (FIFO but slow).
- We need per-key queues or a semaphore-per-key mechanism.
- Redis Streams approach: Use
XADDwith a specific key? No, Redis Streams don't guarantee strict ordering across consumers easily without complex logic. - In-memory/Python approach (for a demo): A dictionary mapping
ordering_key->asyncio.Lock. If a worker picks up a job with key "A", it must acquire lock "A". If another worker picks up another job with "A", it waits on lock "A". - Refined approach for a robust demo: Use Redis Lists as queues per ordering key.
LPUSH queue:<ordering_key> <job_data>- A dispatcher reads from these queues.
- Actually, a simpler way for a code example is:
- Push job to a main stream.
- Workers fetch jobs.
- Workers maintain a shared state (or distributed lock) per ordering key.
- Better: A "Scheduler" component that polls specific ordering-key queues and dispatches them only when the previous job for that key is done.
-
Let's go with a "Coordinator" pattern in Python using
asyncio.- We'll have a
JobManager. - It holds a map of
ordering_key->asyncio.Queue. - It holds a map of
ordering_key->Task(the consumer for that specific queue). - When a job comes in for key X, it goes to Queue X. A dedicated worker pulls from Queue X. This guarantees order.
- Backpressure: If Queue X gets too full, we reject or wait.
- We'll have a
-
-
Backpressure:
- Mechanism: If the internal queues (per ordering key) or the global pending count exceeds a threshold, the API endpoint should return
429 Too Many Requestsor503 Service Unavailable, or block (wait) until space is available. - For this demo, let's implement a "bounded queue" approach.
- Mechanism: If the internal queues (per ordering key) or the global pending count exceeds a threshold, the API endpoint should return
Drafting the Code Structure (Python + FastAPI + Redis):
main.py: FastAPI app.job_system.py: The core logic.submit_job: Checks idempotency, pushes to buffer.dispatcher: Pulls from buffer, routes to ordering-key-specific workers.worker: Processes the actual async task.
Refining the "Ordering Key" logic: Strict ordering is hard with parallel workers. Approach:
- Jobs enter a main
asyncio.Queue(bounded for backpressure). - A "Router" pulls from the main queue.
- The Router looks at the
ordering_key. - It maintains a dictionary:
locks = {key: asyncio.Lock()}. - It spawns a task:
run_job_with_lock(job, lock).- Wait... if we spawn tasks immediately, we rely on the lock.
- Job 1 (Key A) starts.
- Job 2 (Key A) comes in. Spawns task. Tries to acquire Lock A. Waits.
- Job 3 (Key B) comes in. Spawns task. Acquires Lock B. Runs immediately.
- This works for parallelism between keys and ordering execution start for same keys.
- Does it guarantee order finish? Not necessarily, but usually "ordering" implies start order.
- What about Backpressure? If the main queue fills up,
put()blocks or raises exception.
Idempotency implementation details:
- Redis key:
idempotency:{key}. - Value: Job ID.
- TTL: 24 hours.
Let's write the solution.
- Components:
- FastAPI endpoint:
/jobs. - Redis: For persistence/state (optional for a pure in-memory demo, but makes it "real"). Let's stick to pure Python
asyncioto make the code runnable without external dependencies (docker-compose etc.), but simulate the concepts accurately. Self-correction: Using Redis is better to demonstrate "real world" constraints, but a pure Python solution is easier for a user to copy-paste and run immediately to see the mechanics. I will provide a pure Python implementation usingasynciowhich models the concepts perfectly, as it avoids infra setup hell.
- FastAPI endpoint:
Plan:
- Class
AsyncJobPlatform:self.pending_jobs:asyncio.Queue(Bounded for backpressure).self.active_ordering_keys: Dict[str, asyncio.Lock].self.idempotency_cache: Dict[str, JobResult].self.workers: Pool of consumers.
- Idempotency:
submit()checksidempotency_cache. If hit, return cached result ID.- If miss, add to
pending_jobs. - If
pending_jobs.full(): Raise Backpressure error (FastAPI 429).
- Ordering:
- The workers pull from
pending_jobs. - When a worker gets a job, it looks up the
Lockforjob.ordering_key. async with lock: await process(job).- Wait, if we have a global queue and multiple workers, they pull in order.
- Queue: [Job A1, Job B1, Job A2]
- Worker 1 pulls A1. Acquires Lock A.
- Worker 2 pulls B1. Acquires Lock B. (Runs in parallel with A1).
- Worker 3 pulls A2. Acquires Lock A. (Waits for A1).
- This satisfies the requirements.
- The workers pull from
- Backpressure:
- The
pending_jobsqueue hasmaxsize=10. - If full,
submitfails immediately (or waits). We'll choose "Fail fast" (Backpressure) to simulate a loaded system.
- The
Code Outline:
- Data models:
JobRequest,Job. - The
JobProcessorclass. - The API routes.
Refining the
ordering_keyedge case: What ifordering_keyis None? (Fire and forget/Parallel). Use a dummy lock or no lock. If no ordering key, just run.Visualizing the Output: The user needs to see it working. I'll add logging to show "Job A started", "Job A finished", "Job B started". This will visualize the ordering.
Let's construct the Python code.
Dependencies:
fastapi,uvicorn,pydantic. Standard Lib:asyncio,logging,random.Step-by-Step Implementation Plan:
- Setup: Imports, logging config.
- Models: Pydantic models for input/output.
- Core Logic (The Platform):
submit_job: Handle idempotency and backpressure (Queue full).worker: Loop consuming from queue. Handle ordering via locks.
- Mock Task:
async def process_job(job)(simulate IO with sleep). - API: FastAPI app initialization.
- Main: Run loop.
Constraint Checklist & Confidence Score:
- Idempotency? Yes, dictionary check.
- Ordering Keys? Yes,
asyncio.Lockper key. - Backpressure? Yes, Bounded Queue.
- Async? Yes.
Confidence Score: 5/5.
Mental Sandbox Simulation:
- User sends Job 1 (Key: "1").
- User sends Job 2 (Key: "1").
- User sends Job 3 (Key: "2").
- Queue: [J1, J2, J3].
- Worker 1 takes J1. Locks "1". Runs.
- Worker 2 takes J2. Locks "1". BLOCKS.
- Worker 3 takes J3. Locks "2". Runs (parallel).
- User sends Job 1 again (Idempotency Key: "X").
- System checks memory, sees "X" associated with J1. Returns J1 ID. Doesn't queue.
- User sends 20 jobs rapidly. Queue size 10.
- First 10 accepted. Next 10 rejected (429).
This looks solid. I will implement this using pure Python
asyncio+FastAPIto ensure it is lightweight and dependency-minimal (no Redis required to run the demo).I will structure the response to explain the concepts first, then provide the code, then explain how to run and test it.
Detailed Code Logic:
JobStore: Holds the locks and the state.self.semaphores: Dictionary ofasyncio.Lock.self.processed_keys: Dictionary for idempotency.
Queue:asyncio.Queue(maxsize=10).FastAPI:await queue.put(job)inside atry/exceptforasyncio.QueueFull.
One tricky part: Idempotency expiration. In a real system, this eventually expires. I'll set a TTL in the docstring or simulate a simple "high water mark" cleanup, or just keep it simple for the demo (in-memory dict). I'll stick to the in-memory dict for simplicity but mention Redis for production.
Ordering Key strictness: If Job A1 is running, and Job A2 comes in. Worker picks up A2, hits lock. Waits. This is correct.
Backpressure Strategy: I will use
queue.put_nowait()inside a try/except block. If it raisesQueueFull, return HTTP 429 or 503. This allows the client to know the system is overloaded.Let's write it. This solution implements a robust, asynchronous job processing platform in Python using FastAPI for the API layer and AsyncIO for the concurrency engine.
It demonstrates three critical distributed system patterns:
- Idempotency: Ensures retrying a request doesn't create duplicate work.
- Ordering Keys: Ensures jobs tagged with the same key are processed strictly sequentially (FIFO).
- Backpressure: Protects the system from overload by rejecting jobs when the queue is full (Load Shedding).
Architecture
- API: Receives requests, checks idempotency, applies backpressure checks.
- Scheduler: Manages internal queues. It holds a separate FIFO queue for every unique
ordering_key. - Worker Pool: A fixed number of workers pull available "runnable" tasks from the scheduler.
The Implementation
You can save this code to a file named
main.py.import asyncio import random import time import uuid from collections import defaultdict from typing import Dict, Optional from pydantic import BaseModel, Field from fastapi import FastAPI, HTTPException, BackgroundTasks from fastapi.responses import JSONResponse # --- Configuration --- MAX_QUEUE_SIZE = 100 # Backpressure threshold NUM_WORKERS = 4 # Concurrency limit JOB_TTL_SECONDS = 3600 # --- Models --- class JobRequest(BaseModel): # The payload to process payload: str # Idempotency key: prevents duplicates idempotency_key: str = Field(..., example="tx-12345") # Ordering key: jobs with same key run sequentially ordering_key: str = Field(default="default", example="user-abc") class JobResponse(BaseModel): job_id: str status: str message: str class Job: def __init__(self, request: JobRequest): self.id = str(uuid.uuid4()) self.idempotency_key = request.idempotency_key self.ordering_key = request.ordering_key self.payload = request.payload self.status = "queued" self.created_at = time.time() # --- Core Logic --- class AsyncJobPlatform: def __init__(self): # Global state self.jobs: Dict[str, Job] = {} self.idempotency_index: Dict[str, str] = {} # Maps idempotency_key -> job_id # Ordering & Queueing # A dictionary of queues. Key is the ordering_key, Value is a deque of job_ids self.queues: Dict[str, asyncio.Queue] = defaultdict(asyncio.Queue) # A set to track which ordering_keys are currently "active" (being processed by a worker) self.active_keys: set = set() # Backpressure self.current_queue_size = 0 self.lock = asyncio.Lock() # Worker coordination self.main_loop = asyncio.Event() self.workers: list[asyncio.Task] = [] async def submit_job(self, request: JobRequest) -> Job: async with self.lock: # 1. IDEMPOTENCY CHECK if request.idempotency_key in self.idempotency_index: existing_id = self.idempotency_index[request.idempotency_key] # Return existing job state return self.jobs[existing_id] # 2. BACKPRESSURE CHECK if self.current_queue_size >= MAX_QUEUE_SIZE: raise HTTPException( status_code=429, detail="System overloaded: Queue limit reached (Backpressure)" ) # 3. CREATE JOB job = Job(request) self.jobs[job.id] = job self.idempotency_index[request.idempotency_key] = job.id # 4. ENQUEUE BASED ON ORDERING KEY q = self.queues[request.ordering_key] await q.put(job.id) self.current_queue_size += 1 print(f"Accepted Job {job.id[:8]} (Key: {request.ordering_key})") # Wake up the dispatcher logic self.main_loop.set() return job def start_workers(self, n: int): for i in range(n): task = asyncio.create_task(self.worker_loop(f"Worker-{i}")) self.workers.append(task) async def worker_loop(self, name: str): print(f"{name} started.") while True: # --- DISPATCHER LOGIC --- # We need to find a job that is "unlocked" (ordering key not active). # We iterate through queues to find work. job_to_run = None async with self.lock: # Iterate over all known ordering keys for key, q in list(self.queues.items()): # If this key is currently being processed by another worker, skip it (Ordering) if key in self.active_keys: continue # Check if queue has items without blocking if not q.empty(): job_id = q.get_nowait() job = self.jobs.get(job_id) if job: # Mark key as active (lock it for this sequence) self.active_keys.add(key) job_to_run = job break if job_to_run: # --- EXECUTION --- try: await self.process_job(job_to_run) finally: # Release the lock on the ordering key async with self.lock: self.active_keys.remove(job_to_run.ordering_key) self.current_queue_size -= 1 else: # No available work (or all keys locked), wait for signal self.main_loop.clear() await self.main_loop.wait() async def process_job(self, job: Job): print(f"Processing Job {job.id[:8]} | Key: {job.ordering_key} | Payload: {job.payload}") job.status = "running" # Simulate IO work duration = random.uniform(0.5, 2.0) await asyncio.sleep(duration) job.status = "completed" print(f"Completed Job {job.id[:8]} (Took {duration:.2}s)") # --- App Setup --- app = FastAPI(title="Async Job Platform") platform = AsyncJobPlatform() @app.on_event("startup") async def startup_event(): # Start the background workers platform.start_workers(NUM_WORKERS) @app.post("/jobs", response_model=JobResponse) async def create_job(request: JobRequest): try: job = await platform.submit_job(request) return JobResponse( job_id=job.id, status=job.status, message="Job accepted" ) except HTTPException: raise except Exception as e: raise HTTPException(status_code=500, detail=str(e)) @app.get("/stats") async def stats(): return { "queue_size": platform.current_queue_size, "active_keys": list(platform.active_keys), "total_jobs_tracked": len(platform.jobs) }How to Run
- Install dependencies:
pip install fastapi uvicorn pydantic - Run the server:
uvicorn main:app --reload - Access the automatic docs at
http://127.0.0.1:8000/docs.
Detailed Mechanism Explanation
1. Idempotency
Before creating a job, the system checks an index (
idempotency_index).- Scenario A: Client sends a request with
idempotency_key: "tx-1". It is new, so we create a job. - Scenario B: Client retries the request due to a network timeout. The key
"tx-1"exists in the index. The system returns the existing job ID and status, doing no extra work and not queuing a duplicate.
2. Ordering Keys (FIFO per Key)
This is the most complex part. We don't want a single global queue because Job A (for User 1) shouldn't wait for Job B (for User 2). However, Job A1 and Job A2 (for User 1) must run in order.
- Data Structure: We maintain a dictionary of queues:
queues[ordering_key]. - Locking Strategy: The workers maintain a set of
active_keys.- If a worker picks up Job A1 (Key: "User1"), it adds "User1" to
active_keys. - Other workers check
active_keys. If "User1" is present, they skip the queue for "User1" and look for work in other queues. - Once A1 finishes, "User1" is removed from
active_keys, allowing A2 to be picked up.
- If a worker picks up Job A1 (Key: "User1"), it adds "User1" to
3. Backpressure
We protect the system memory from overflowing.
- We track
current_queue_size. - In
submit_job, ifcurrent_queue_size >= MAX_QUEUE_SIZE, we immediately raise anHTTP 429 Too Many Requests. - This fails fast, informing the client (or load balancer) that the service is busy, rather than allowing the server to hang or crash due to memory exhaustion.
How to Test the Features
-
Test Idempotency: Send two requests via the Swagger UI with the same
idempotency_key. You will get the samejob_idback both times. -
Test Ordering: Send 3 requests with
ordering_key: "A"quickly. Then send 1 request withordering_key: "B". Watch the console logs. You will see "A" jobs processing strictly one by one. The "B" job will likely process in parallel to the "A" jobs, but the "A" jobs never overlap each other. -
Test Backpressure: Modify
MAX_QUEUE_SIZEin the code to3. Send 5 requests quickly. The first 3 should return 200 OK, and the next ones should return 429.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.