Implement a sitemap delta publisher: change batching, throttles, and monitoring.
- User
Implement a sitemap delta publisher: change batching, throttles, and monitoring.
Assistant
The user wants me to implement a "sitemap delta publisher" with three specific features:
- Change Batching: Collecting changes over time or quantity and publishing them in groups.
- Throttles: Limiting the rate at which these batches are published to avoid overloading downstream systems (e.g., search engines).
- Monitoring: Tracking the health, status, and statistics of the publisher (e.g., success rates, queue size).
1. Conceptualizing the System
- What is a Sitemap Delta? A sitemap usually lists all URLs on a site. A "delta" implies we are only sending the changes (added, updated, deleted URLs) since the last time. This is often used for ping mechanisms (like pinging Google) or keeping a search index up to date.
- Input: A stream of URL change events.
- Core Logic:
- Buffer: Accumulate events.
- Batcher: Trigger sending when buffer size hits a limit OR a time window expires.
- Throttler: Ensure we don't send too fast (Rate Limiting).
- Publisher: The actual HTTP client sending the delta.
- Monitor: Metrics collection.
2. Designing the Architecture (Python Focus)
I'll use Python with
asynciobecause publishing involves network I/O, and async is perfect for batching and throttling.- Data Structure: A simple class for
UrlChange(url, change_type, timestamp). - The
DeltaPublisherClass:- Queue:
asyncio.Queueto hold incoming changes. - Batching Strategy: A loop that pulls from the queue. It waits for either
max_batch_sizeorbatch_timeout. - Throttling Strategy: A
asyncio.Semaphoreor a token bucket algorithm. A simple "minimum delay between requests" is often sufficient for external pings, or a rate limiter wrapper. - Publishing Logic: Mocking the actual HTTP request.
- Monitoring: A dictionary or a dedicated class to track counts (published, failed, queued).
- Queue:
3. Detailed Implementation Steps
- Step 1: The Models. Define what a change looks like.
- Step 2: Monitoring. A thread-safe (or asyncio-safe) way to store metrics.
- Step 3: Throttling. A simple rate limiter context manager or logic.
- Step 4: The Core Engine.
add_change(): Public method to add to queue._run(): The background worker._process_batch(): Handles the batching logic (wait for X items or Y seconds)._publish(): Sends data.
4. Refining the Logic (Self-Correction/Optimization)
-
Batching: How to implement "Time OR Size" batching efficiently in asyncio?
- Approach: Loop
queue.get(). useasyncio.wait_forwith a timeout on the first item. Once we have one, try to get the rest immediately (queue.get_nowait()) up to the batch limit. - Alternative:
asyncio.Queuedoesn't support batch get natively. - Refined Approach: Use a loop. Use
asyncio.wait_for(queue.get(), timeout=T). If timeout, process whatever is in the temp list. If list reaches N, process immediately.
- Approach: Loop
-
Throttling:
- Need to prevent a burst of batches.
- Use a simple token bucket or a sleep loop.
await asyncio.sleep(min_interval)is easiest but imprecise. A proper RateLimiter class is better. Let's implement a simpleLeakyBucketorFixedWindowrate limiter using a lock.
-
Monitoring:
- Should expose current queue size, total success, total failure.
5. Drafting the Code Structure (Mental Draft)
class Metrics: sent, failed, queue_size class DeltaPublisher: def __init__(batch_size, timeout, rate_limit): self.queue = asyncio.Queue() self.metrics = Metrics() async def start(): while True: # Batching Logic batch = await self._gather_batch() # Throttling await self.rate_limiter.acquire() # Publish await self._publish(batch) async def _gather_batch(): # Get first item (wait up to timeout) # Get remaining items (immediate)6. Adding Robustness
- Error Handling: What if publishing fails? Log it, increment failure metric. Maybe retry? The prompt doesn't specify retry strategies (exponential backoff), but usually, a publisher shouldn't retry indefinitely or it blocks the queue. I'll log and move on (fail-fast), or perhaps a simple retry mechanism. Let's stick to "Log and Increment Metric" to keep the scope focused on the requested features.
7. Writing the Actual Code (Iterative refinement)
- Imports:
asyncio,dataclasses,time,logging. - Configuration: Make it configurable.
- Mocking: I need a mock function to simulate the actual publishing so the user can run the code.
8. Final Code Assembly & Explanation
- Define
UrlChangedataclass. - Create
Monitorclass. - Create
Throttlerclass (simple leaky bucket). - Create
DeltaPublisherorchestrating it all. - Add a
mainblock to demonstrate it working (adding random URLs, printing metrics).
(Self-Correction during coding the Throttler): A robust throttler is tricky. Let's use a
asyncio.Semaphoreapproach combined with a replenishing task, or simpler: record the time of the last release andsleep(max(0, interval - (now - last_time))). The sleep approach is stateless and easier to understand for a demo.(Refining the Batcher): Using
asyncio.waitwithreturn_when=asyncio.FIRST_COMPLETEDallows combining a queue-getter task with a timeout task. This is the cleanest way to do "wait for data OR time limit".Plan Summary:
- Data Model: Structure for URL updates.
- Metrics: A class to track stats.
- Throttler: A generic rate-limiter utility.
- Publisher Service: The core class handling the queue, batching logic, and integration of the above components.
- Demo: A runnable script.
Let's generate the Python implementation. A Sitemap Delta Publisher is a critical component in search engineering and distributed systems. Its goal is to inform search engines (or downstream consumers) about content changes (Added, Updated, Deleted) efficiently, without hammering their APIs or overwhelming your own network resources.
Below is a comprehensive implementation in Python using
asyncio. This approach allows for high concurrency while strictly controlling the flow of data.Core Architecture
- Change Buffer: Accumulates incoming change events.
- Batcher: Flushes the buffer based on Size (e.g., 100 items) or Time (e.g., every 5 seconds).
- Throttler: Limits the rate of HTTP requests (e.g., 2 requests per second) using the Token Bucket algorithm.
- Monitor: Tracks success/failure rates and queue depths.
Implementation
import asyncio import time import random from dataclasses import dataclass, field from enum import Enum from typing import List, Optional from datetime import datetime # ========================================== # 1. Data Models & Configuration # ========================================== class ChangeType(Enum): ADDED = "ADDED" UPDATED = "UPDATED" DELETED = "DELETED" @dataclass class SitemapChange: url: str change_type: ChangeType timestamp: datetime = field(default_factory=datetime.now) @dataclass class PublisherConfig: batch_size: int = 10 # Max items per batch flush_interval: float = 5.0 # Seconds to wait before flushing partial batch rate_limit: float = 2.0 # Requests per second (Hz) max_retries: int = 3 # Retries for HTTP failures # ========================================== # 2. Monitoring & Metrics # ========================================== class Monitor: def __init__(self): self.lock = asyncio.Lock() self.queued_items = 0 self.batches_published = 0 self.items_processed = 0 self.errors = 0 async def increment_queue(self, count: int): async with self.lock: self.queued_items += count async def record_success(self, batch_size: int): async with self.lock: self.queued_items -= batch_size self.batches_published += 1 self.items_processed += batch_size async def record_failure(self): async with self.lock: self.errors += 1 def get_status(self) -> dict: return { "queued_items": self.queued_items, "batches_published": self.batches_published, "items_processed": self.items_processed, "errors": self.errors } # ========================================== # 3. Throttler (Token Bucket) # ========================================== class Throttler: def __init__(self, rate_limit: float): self.rate_limit = rate_limit self.min_interval = 1.0 / rate_limit self.last_run = 0.0 async def acquire(self): """ Blocks until the rate limit allows execution. Calculates time elapsed since last run and sleeps if necessary. """ now = time.monotonic() elapsed = now - self.last_run wait_time = self.min_interval - elapsed if wait_time > 0: await asyncio.sleep(wait_time) self.last_run = time.monotonic() # ========================================== # 4. Publisher Core Logic # ========================================== class SitemapDeltaPublisher: def __init__(self, config: PublisherConfig): self.config = config self.queue = asyncio.Queue() self.monitor = Monitor() self.throttler = Throttler(config.rate_limit) self._running = False self.http_client = None # Mocked client async def submit_change(self, change: SitemapChange): """Public API to submit a change.""" await self.queue.put(change) await self.monitor.increment_queue(1) async def start(self): """Starts the background processing loop.""" self._running = True print(f"[{time.time():.2f}] Publisher started. Batch Size: {self.config.batch_size}, Rate Limit: {self.config.rate_limit}/s") while self._running or not self.queue.empty(): batch = await self._gather_batch() if batch: # Apply Throttling before publishing await self.throttler.acquire() # Publish (Simulated) await self._publish_batch(batch) async def _gather_batch(self) -> List[SitemapChange]: """ Wait for either: 1. The batch size to fill up OR 2. The flush interval timeout to occur """ batch = [] try: # 1. Wait for the first item (with a timeout for graceful shutdown logic) first_item = await asyncio.wait_for(self.queue.get(), timeout=0.1) batch.append(first_item) # 2. Gather remaining items up to batch_size or until queue empty # logic: we effectively have a time window here defined by flush_interval start_time = time.monotonic() while len(batch) < self.config.batch_size: # Calculate remaining time for the flush interval elapsed = time.monotonic() - start_time remaining_time = self.config.flush_interval - elapsed if remaining_time <= 0: break # Flush interval hit try: # Wait for next item with remaining time item = await asyncio.wait_for(self.queue.get(), timeout=remaining_time) batch.append(item) except asyncio.TimeoutError: # No more items arrived within the flush window break return batch except asyncio.TimeoutError: # Queue was empty, loop continues return [] async def _publish_batch(self, batch: List[SitemapChange]): """Sends the batch to the downstream system (mocked).""" batch_id = random.randint(1000, 9999) urls = [b.url for b in batch] try: # Simulate Network Latency latency = random.uniform(0.05, 0.2) await asyncio.sleep(latency) # Simulate Random Failures (10% chance) if random.random() < 0.1: raise ConnectionError("Simulated network error") print(f"[{time.time():.2f}] SUCCESS: Published Batch #{batch_id} | Size: {len(batch)} | URLs: {urls}") await self.monitor.record_success(len(batch)) except Exception as e: print(f"[{time.time():.2f}] ERROR: Failed Batch #{batch_id} | Reason: {e}") await self.monitor.record_failure() # Re-queue logic could go here, but for this example we log the error. # In production, you might push to a Dead Letter Queue. def stop(self): self._running = False # ========================================== # 5. Simulation / Usage # ========================================== async def simulation_producer(publisher: SitemapDeltaPublisher): """Simulates incoming traffic of URL changes.""" urls = ["/home", "/products", "/about", "/contact", "/blog"] for i in range(25): change = SitemapChange( url=random.choice(urls) + f"?id={i}", change_type=random.choice(list(ChangeType)) ) await publisher.submit_change(change) # Simulate erratic arrival times await asyncio.sleep(random.uniform(0.01, 0.1)) async def main(): config = PublisherConfig( batch_size=5, # Flush if 5 items gather flush_interval=2.0, # Flush partial batches every 2s rate_limit=2.0 # Max 2 requests per second ) publisher = SitemapDeltaPublisher(config) # Start Publisher in background publisher_task = asyncio.create_task(publisher.start()) # Start Producer in background producer_task = asyncio.create_task(simulation_producer(publisher)) # Let them run for a bit await asyncio.sleep(10) # Stop publisher gracefully publisher.stop() await publisher_task # Wait for queue to drain print("\n--- Final Metrics ---") print(publisher.monitor.get_status()) if __name__ == "__main__": asyncio.run(main())Key Design Decisions
-
Change Batching Logic (
_gather_batch):- Most naive implementations simply check
if queue.size > X. This creates a deadlock if the queue never reaches X. - This implementation uses a Time-Windowed Batch: It starts a timer when the first item arrives. If the buffer fills before the timer expires, it flushes immediately (Optimizing for Throughput). If the timer expires before the buffer fills, it flushes the partial batch (Optimizing for Latency).
- Most naive implementations simply check
-
Throttler:
- Instead of a heavy library, we implement a lightweight Token Bucket using
time.monotonic(). - It calculates the
min_interval(e.g., 500ms for 2 RPS) and sleeps the remainder if the previous call was too recent. This ensures strictly smooth traffic flow to the downstream API.
- Instead of a heavy library, we implement a lightweight Token Bucket using
-
Monitoring:
- The
Monitorclass uses anasyncio.Lockto ensure thread-safety (though asyncio is single-threaded, context switching duringawaitcan cause race conditions with simple counters).
- The
-
Queue Management:
- We use
asyncio.Queue. This acts as the buffer. If the producer is faster than the throttled consumer, this queue will grow in memory. In a production system, you might back this with Redis or RabbitMQ to prevent OOM errors.
- We use
How to Run
Copy the code into a file (e.g.,
publisher.py) and run it. You will see logs showing the batching behavior (e.g.,Size: 5) and the throttling behavior (timestamps of publishes will be roughly 0.5s apart).