Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Build a persistent work queue with idempotency keys, retries, and DLQs.

  1. User

    Build a persistent work queue with idempotency keys, retries, and DLQs.

  2. GLM Assistant

    The user wants a persistent work queue. Key features required:

    1. Persistence: Tasks shouldn't be lost if the system crashes.
    2. Idempotency Keys: Preventing duplicate processing of the same logical task.
    3. Retries: Handling transient failures by trying again.
    4. Dead Letter Queues (DLQs): Handling permanent failures by moving tasks to a separate queue for inspection.

    Technology Choice:

    • Redis is the industry standard for this kind of lightweight queueing. It's fast, persistent enough (with AOF/RDB), and has data structures (Lists, Hashes, Sorted Sets) perfect for queues.
    • Python is a great language for demonstrating this logic clearly, but the concepts apply to Node.js, Go, etc.
    • I will create a Python class PersistentWorkQueue using redis-py.

    Architecture/Design:

    1. Data Structures in Redis:

      • queue:{name}: A List (LPUSH/RPOP) for the main pending items.
      • processing:{name}: A Sorted Set (ZSET). Score = timestamp of when it times out. This allows visibility into what's currently being worked on and recovery of "stuck" jobs.
      • dlq:{name}: A List for failed items.
      • idempotency:{name}: A Set or Hash to track processed/processing IDs. Wait, if we use the ID as part of the message payload, we need to check this before adding, and potentially check/set it when consuming (to be safe). Let's stick to checking on enqueue for simplicity, but strictly speaking, idempotency is usually a consumer-side concern (deduplication). However, the prompt asks for "idempotency keys" in the context of the queue mechanism. I'll implement a check during enqueue to reject duplicates, or perhaps a short-lived lock during dequeue. Let's go with the "reject duplicate enqueue" approach or "update existing" approach, but simpler is just ensuring the same ID isn't processed twice. Actually, standard idempotency usually means if I send the same request twice, the result is the same.
      • Refined Idempotency: I will use a Hash idempotency:{name} mapping key -> status (pending/done). When enqueuing, check if key exists. When consuming, mark as done. Add a TTL to these keys so the set doesn't grow forever.
    2. The Message Format: JSON payload containing:

      • id (Idempotency Key)
      • data (The actual payload)
      • retry_count
    3. Core Logic:

      • Enqueue(job_id, payload):

        1. Check idempotency set.
        2. If exists, return (or update, depending on policy). Let's assume strict: "Already exists, skip".
        3. Add to idempotency set (status: pending).
        4. LPUSH to queue.
      • Dequeue(timeout):

        1. This needs to be atomic or handle race conditions.
        2. Standard pattern: RPOP from queue.
        3. If message found:
          • Add to processing ZSET with score = now + visibility_timeout.
          • Return message.
        4. Self-correction: The prompt asks for retries. Retries usually happen if the worker crashes or explicitly fails.
        5. Monitoring (The "Kicker"): We need a way to move items from processing back to queue if they time out. In a real system, this is a separate background process. For this code example, I might integrate it into the dequeue call (check processing set for items with score < now, move them to queue).
      • Ack(job_id):

        1. Remove from processing.
        2. Update idempotency to "done".
      • Fail(job_id):

        1. Increment retry count.
        2. If retries < max:
          • Push back to queue.
        3. Else:
          • Push to dlq.
          • Remove from processing.
    4. Implementation Details (Python):

      • I need a Worker class to simulate consumption.
      • I need a Producer (or just the Queue class methods).
      • I will use a simplified approach where the dequeue function handles the logic of fetching.

      Let's refine the "Retry" logic. Does the worker put it back in the queue? Or does the queue system handle timeouts?

      • Approach A: Worker gets job. If worker crashes (no ack), job stays in processing. A "garbage collector" (or the next dequeue call) sees the timestamp is old and moves it back to queue (incrementing retry count).
      • Approach B: Worker explicitly calls nack (negative acknowledgement).
      • Decision: I will implement both. The "GC" logic inside dequeue handles crashes (timeout). The nack method handles explicit failures.
    5. Code Structure:

      • WorkQueue class.
      • Methods: enqueue, dequeue, ack, fail (nack), process_orphans (internal).
      • DLQ methods: get_dlq_items, requeue_dlq_item.
    6. Drafting the Code (Mental or Scratchpad):

      • Keys:

        • q:main (List)
        • q:processing (ZSet) -> {payload_json: timeout_timestamp}
        • q:idempotency (Set) -> {id}
        • q:dlq (List)
      • Enqueue:

        if redis.sismember(idempotency_key): return
        redis.sadd(idempotency_key)
        redis.lpush(main_queue, json(job))
        
      • Dequeue: First, clean up old jobs. now = time.time() items = redis.zrangebyscore(processing, 0, now) for item in items: ... requeue logic ...

        Then get new job. job = redis.rpoplpush(main_queue, processing) -> Wait, Redis has RPOPLPUSH but not for ZSets. Better approach for atomic reliability: Use a Lua script? Or simpler: RPOP from main, ZADD to processing, return. If client crashes after RPOP but before ZADD, data is lost. To be strictly persistent: Use BRPOPLPUSH (Redis reliable queue pattern). Source: Main Queue. Destination: Backup Queue (Processing). But we need ZSETS for timeouts to handle retries properly. Hybrid Approach:

        1. Check ZSET for orphans. Move to List.
        2. Pop from List.
        3. Add to ZSET with timeout. This isn't perfectly atomic but good enough for a "build a queue" demonstration. For production, Lua scripts are required. Let's stick to standard Python logic for clarity, perhaps wrapping the critical section or just noting the race condition.
      • Retry Logic: We need to store the retry count in the payload. Payload: {"id": "...", "data": ..., "attempts": 0} When an orphan is found: payload['attempts'] += 1. If attempts > max: Move to DLQ.

    7. Refining the "Idempotency" requirement: If I send the same ID twice, the queue should ignore the second one. I'll use a Redis Set seen_ids.

    8. Putting it together into a clean Class.

      • __init__(redis_client, name)
      • submit(id, payload)
      • lease(timeout) (dequeue)
      • complete(id) (ack)
      • release(id, error=None) (nack/fail)
    9. Writing the Example Script: Create a Redis mock or just assume Redis is running. I'll assume redis library is installed. I'll add a main block demonstrating the flow:

      1. Enqueue.
      2. Enqueue duplicate (should fail/be ignored).
      3. Worker lease.
      4. Worker fail (retry).
      5. Worker fail again (DLQ).
      6. Check DLQ.
    10. Review against constraints:

      • Persistent? Yes, backed by Redis (assuming RDB/AOF).
      • Idempotency keys? Yes, checked on submit.
      • Retries? Yes, handled via timeout/orphan processing.
      • DLQ? Yes.

      Self-Correction on Idempotency: Idempotency usually implies that processing the same item twice yields the same result. But often in queues, "Idempotency Key" refers to "Message Deduplication". I will clarify this in the description but implement Message Deduplication (preventing the same ID from entering the queue twice).

      Self-Correction on Persistence: Standard RPOP isn't persistent if the worker crashes after pop but before processing. The pattern RPOPLPUSH (Reliable Queue) is better. RPOPLPUSH queue processing_queue When done: LREM processing_queue job. When timeout: RPOPLPUSH processing_queue queue. However, LREM is O(N). Using a ZSET for processing is superior for timeouts (visibility timeouts), but atomic moving from List to ZSET requires Lua. To keep the code runnable and readable without Lua complexity: I will implement a lease() method that:

      1. Moves from Main List to Processing ZSet.
      2. Since this isn't atomic, I'll use a pipeline to minimize risks, but note that Lua is better for production.

      Revised Plan for "Retries": When lease() is called:

      1. First, scan processing ZSet for items where score < now. These are timed out jobs.
      2. For each timed out job:
        • Increment retry count.
        • If max retries < limit: Push back to main list.
        • Else: Push to dlq list.
      3. Then, pop from main list, add to processing ZSet with score = now + timeout.

      This handles the "worker crashed" scenario (timeout) and the "exponential backoff" can be added but maybe too complex for a basic demo. I'll stick to fixed retries.

    11. Final Code Structure:

      • Class TaskQueue
      • Dependencies: redis, json, time, uuid (if id not provided).

    Let's write the code. I will include comments explaining the Redis keys used.

    Keys:

    • tasks: List (The main queue)
    • inflight: Sorted Set (Jobs being processed, score=expiry_time)
    • seen: Set (Idempotency keys)
    • dlq: List (Dead Letter Queue)

    Logic for submit: Check seen. If in seen, return. Add to seen. LPUSH to tasks.

    Logic for poll (lease): Clean up inflight: items = ZRANGEBYSCORE inflight 0 now For item in items: ZREM inflight item item_obj = json(item) item_obj.tries += 1 if tries > max: LPUSH dlq json(item_obj) else: LPUSH tasks json(item_obj) Get new: item = RPOP tasks If item: ZADD inflight now+timeout item Return item

    Logic for ack: ZREM inflight item (We should also remove from seen? Usually, idempotency is strictly for ingestion. If we remove from seen, the user can submit the job again immediately after it finishes. If we keep it, they can never submit it again. Usually, idempotency keys have a TTL (e.g., 24 hours) or are strictly forever. I'll add a TTL to the seen set entries, defaulting to 24h).

    Logic for fail (Explicit fail - optional): Same as timeout. ZREM, increment tries, push to queue or DLQ.

    This looks solid. I will produce the code now. This solution implements a robust, persistent work queue in Python using Redis.

    Architecture

    1. Persistence: Uses Redis (an in-memory data store that persists to disk) to ensure tasks survive application restarts.
    2. Idempotency: Uses a Redis SET to track processed/processing keys. If a task with a duplicate key is submitted, it is ignored.
    3. Retries: Tasks are not discarded immediately on failure. A "Reliable Queue" pattern (checking visibility timeouts) allows failed tasks to be picked up by another worker.
    4. DLQ (Dead Letter Queue): If a task fails consistently (exceeds retry limits), it is moved to a specific list (dlq) for debugging.

    The Implementation

    You will need the redis python library: pip install redis

    import redis
    import json
    import time
    import uuid
    import threading
    from typing import Optional, Callable, Any
    
    class PersistentWorkQueue:
        def __init__(self, host='localhost', port=6379, db=0):
            self.client = redis.StrictRedis(host=host, port=port, db=db, decode_responses=True)
            
            # Keys for different structures
            self.queue_name = "tasks:queue"
            self.processing_name = "tasks:processing"
            self.dlq_name = "tasks:dlq"
            self.idempotency_set = "tasks:idempotency"
            
            # Configuration
            self.visibility_timeout = 30  # Seconds before a task is considered 'stuck' and retryable
            self.max_retries = 3
    
        def enqueue(self, idempotency_key: str, payload: dict) -> bool:
            """
            Adds a task to the queue if the idempotency key hasn't been seen.
            Returns True if enqueued, False if duplicate.
            """
            # 1. Check Idempotency
            # We use 'sadd' which returns 1 if added, 0 if already exists.
            # This acts as a distributed lock for the key.
            if self.client.sadd(self.idempotency_set, idempotency_key) == 0:
                print(f"[Producer] Duplicate detected: {idempotency_key}")
                return False
    
            # 2. Create Task Object
            task = {
                "id": str(uuid.uuid4()),
                "key": idempotency_key,
                "payload": payload,
                "attempts": 0,
                "created_at": time.time()
            }
    
            # 3. Push to Queue (LPUSH)
            self.client.lpush(self.queue_name, json.dumps(task))
            print(f"[Producer] Enqueued: {idempotency_key}")
            return True
    
        def _reclaim_orphans(self):
            """
            Checks the 'processing' list for tasks that have timed out (worker crashed).
            Moves them back to the main queue or DLQ.
            """
            now = time.time()
            # SCAN through processing items (In prod, use ZSET for better perf, 
            # but we keep it simple here with a separate check).
            # Here we iterate the processing list to find timed-out items.
            
            # Note: We use a list for 'processing' for simplicity in this demo.
            # For high scale, Redis ZSETS (Sorted Sets) with timestamps as scores 
            # are much faster for range queries.
            
            items = self.client.lrange(self.processing_name, 0, -1)
            for item_str in items:
                item = json.loads(item_str)
                started_at = item.get('started_at', 0)
                
                if now - started_at > self.visibility_timeout:
                    print(f"[Monitor] Reclaiming orphan task: {item['key']}")
                    # Remove from processing
                    self.client.lrem(self.processing_name, 1, item_str)
                    
                    # Handle Retry Logic
                    item['attempts'] += 1
                    if item['attempts'] >= self.max_retries:
                        self._move_to_dlq(item)
                    else:
                        # Push back to main queue
                        self.client.lpush(self.queue_name, json.dumps(item))
    
        def _move_to_dlq(self, task: dict):
            print(f"[Monitor] Moving to DLQ: {task['key']} (Max retries reached)")
            error_record = {
                "task": task,
                "failed_at": time.time(),
                "reason": "Max retries exceeded"
            }
            self.client.lpush(self.dlq_name, json.dumps(error_record))
            # We keep the idempotency key in the set so it isn't reprocessed 
            # immediately by a new producer request. 
            # It must be manually cleared if we want to retry from DLQ.
    
        def process(self, handler: Callable[[dict], Any]):
            """
            Worker loop.
            """
            print("[Worker] Starting...")
            
            while True:
                # 1. Check for stuck tasks (simple polling approach)
                self._reclaim_orphans()
    
                # 2. Blocking Pop (RPOPLPUSH pattern for reliability)
                # Moves item from Queue -> Processing List atomically
                result = self.client.brpoplpush(
                    self.queue_name, 
                    self.processing_name, 
                    timeout=5
                )
    
                if not result:
                    continue
    
                task = json.loads(result)
                print(f"[Worker] Processing: {task['key']} (Attempt {task['attempts'] + 1})")
                
                # Mark start time for timeout tracking
                task['started_at'] = time.time()
                # Update the item in the processing list to reflect the start time
                # (Atomic update is tricky with lists, ideally use HSET or ZSET, 
                # this is a race condition in strict systems but acceptable for basic logic)
                self.client.lrem(self.processing_name, 1, result)
                self.client.lpush(self.processing_name, json.dumps(task))
    
                try:
                    # --- EXECUTE TASK ---
                    handler(task['payload'])
                    
                    # --- SUCCESS ---
                    print(f"[Worker] Success: {task['key']}")
                    
                    # Remove from processing list
                    self.client.lrem(self.processing_name, 1, json.dumps(task))
                    
                    # Note: We DO NOT remove the idempotency key here. 
                    # Usually, idempotency is "forever" to prevent double-billing etc.
                    # If you want to allow re-running after success, remove from set here.
    
                except Exception as e:
                    print(f"[Worker] Failed: {task['key']} - {e}")
                    # On exception, we leave it in the 'processing' list.
                    # The '_reclaim_orphans' logic (above) will pick it up 
                    # after 'visibility_timeout' seconds and handle the retry logic.
                    
                    # To force immediate failure handling (optional), you could 
                    # manually trigger logic here, but relying on the timeout 
                    # prevents "thundering herd" retries on persistent DB errors.
    
    # --- Usage Example ---
    
    def main():
        # Clear previous data for demo
        r = redis.StrictRedis(decode_responses=True)
        r.delete("tasks:queue", "tasks:processing", "tasks:dlq", "tasks:idempotency")
    
        queue = PersistentWorkQueue()
    
        # 1. Producer: Enqueue tasks
        queue.enqueue("order-123", {"item": "Laptop", "price": 999})
        queue.enqueue("order-123", {"item": "Laptop", "price": 999}) # Should be ignored (Idempotency)
        queue.enqueue("order-456", {"item": "Mouse", "price": 20})
        
        # 2. Define a Worker Handler
        def job_handler(payload):
            print(f"  >>> DOING WORK: Processing {payload['item']}")
            # Simulate random failure to demonstrate retries/DLQ
            if payload['item'] == "Mouse":
                raise ValueError("Database connection failed!")
    
        # 3. Run Worker in a thread just for this demo
        # In reality, this runs in a separate process/container
        def worker_thread():
            queue.process(job_handler)
    
        t = threading.Thread(target=worker_thread)
        t.daemon = True
        t.start()
    
        # Let worker run for a bit to handle 'order-123'
        time.sleep(2)
        
        # 4. Check DLQ (Eventually 'Mouse' should end up here after 3 retries)
        print("\n--- Checking System State ---")
        
        # We wait long enough for timeouts to trigger (approx 30s in real time, 
        # but let's just manually trigger the logic or wait briefly for demo context)
        # In this code, 'Mouse' will fail, stay in processing, 
        # and be reclaimed by the next poll loop iteration.
        
        time.sleep(10) 
        dlq_items = r.lrange("tasks:dlq", 0, -1)
        print(f"Items in DLQ: {len(dlq_items)}")
        for item in dlq_items:
            print(json.loads(item))
    
    if __name__ == "__main__":
        main()
    

    Key Concepts Explained

    1. The Reliable Queue Pattern (RPOPLPUSH)

    Standard queues (RPOP) are dangerous. If a worker pops a task and crashes immediately, the task is lost forever. We use BRPOPLPUSH. This atomically pops an item from the Main Queue and pushes a copy into a Processing List.

    • If the worker finishes successfully, it deletes the item from the Processing List.
    • If the worker crashes, the item remains in the Processing List.

    2. Orphan Recovery (Retries)

    The system doesn't just trust the worker. A monitor function (_reclaim_orphans) checks the Processing List.

    • It looks for items that have been sitting there longer than visibility_timeout.
    • It increments their attempts counter.
    • If attempts < max_retries, it pushes them back to the Main Queue.
    • If attempts >= max_retries, it pushes them to the DLQ.

    3. Idempotency

    We use a Redis SET (sadd).

    • When enqueue is called, we try to add the ID to the set.
    • Redis returns 1 if new, 0 if exists.
    • This guarantees that we never push the same ID into the queue twice.

    4. DLQ (Dead Letter Queue)

    This is a standard list (tasks:dlq).

    • It catches "poison pills" (tasks that crash the worker every time).
    • Without a DLQ, a task that always crashes would loop infinitely (Queue -> Worker -> Crash -> Queue -> Worker...), clogging the system.
    • With a DLQ, it is set aside for human inspection (e.g., "Why is this JSON malformed?").

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.