Build a multi-tenant throttling coordinator: fairness policies, dynamic quotas, and…
- User
Build a multi-tenant throttling coordinator: fairness policies, dynamic quotas, and backpressure controls.
Assistant
The user wants a "multi-tenant throttling coordinator." Key requirements are:
- Fairness policies: Not just raw rate limiting, but ensuring tenants get their fair share (weighted fair queuing, priority, etc.).
- Dynamic quotas: Quotas shouldn't be static. They should adapt to system load or tenant behavior (borrowing/saving limits).
- Backpressure controls: When the system is overwhelmed, it needs to signal upstream to slow down (circuit breakers, load shedding).
Language/Stack: Python is a good choice for this kind of systems-design logic because of its readability and strong async support (
asyncio), which is crucial for a coordinator handling concurrent requests. I'll useasyncioto simulate a real-time event loop.Architecture Design:
-
Core Components:
- Tenant: Represents a user/client with specific attributes (ID, weight, priority, current usage).
- Policy Engine: Defines the rules (e.g., "Gold" users get 1000 req/s, "Silver" get 100).
- Coordinator (The Core): The central brain. It holds the state and makes decisions.
- Quota Manager: Handles the dynamic aspect (token buckets, sliding windows).
- Backpressure Manager: Monitors system health (latency, queue depth) and adjusts global rates.
-
Fairness Logic:
- I'll implement a Weighted Fair Queuing (WFQ) concept simplified for a code example.
- Tenants have weights. If Tenant A has weight 2 and Tenant B has weight 1, A gets 2/3 of resources.
-
Dynamic Quotas:
- Standard Token Bucket is too static.
- Let's use a "Burst" capable token bucket that can recharge based on global capacity. If the system is idle, tenants can burst. If the system is busy, quotas shrink.
-
Backpressure:
- Simulate a "System Load" metric.
- If load > threshold, apply a global multiplier to all quotas (shed load) or enforce stricter queuing.
Drafting the Code Structure:
TenantConfig: Dataclass for config.TokenBucket: Classic algo, but maybe modified.ThrottlingCoordinator: The main class.register_tenant()acquire_permission(tenant_id, cost): The critical path.monitor_load(): Background task updating the "pressure".
FairScheduler: A queue that decides who gets the next token if contention exists.
Refining the Fairness/Dynamic Logic: Instead of just "allowed/denied", what if we return a "wait time"? Actually, the prompt asks for a "coordinator". This usually implies an entry point.
Scenario: 10 requests come in simultaneously. Tenant A (High priority): 5 reqs. Tenant B (Low priority): 5 reqs. System capacity: 5 reqs/tick. Coordinator should ideally serve A first, or serve 5 from A and 0 from B, or serve proportional to weights.
Let's go with a Hierarchical Token Bucket approach or a Dynamic Leaky Bucket. Let's stick to a Global Pool + Per-Tenant Reservations model.
- Global Max Capacity = 1000 RPS.
- Tenant A reserved: 200 RPS.
- Tenant B reserved: 200 RPS.
- Unreserved: 600 RPS.
- Dynamic logic: Tenant A can use unreserved capacity if B isn't using it.
Implementation Steps (Python):
- Imports:
asyncio,time,dataclasses. - Data Structures:
Tenant,SystemState. - The Algorithm:
- Each tenant has a
TokenBucket. - The Coordinator has a
GlobalTokenBucket. - To acquire:
- Check local tenant bucket. If empty -> Wait? Reject?
- Check global bucket. If empty -> Apply Backpressure (Wait/Reject).
- Adjust local bucket size dynamically based on global load.
- Each tenant has a
Adding "Backpressure" Simulation: I need a background task that simulates fluctuating system load (e.g., "database latency increasing"). The
ThrottlingCoordinatorwill read this state. If latency is high, it reduces therefill_rateof the global bucket.Adding "Fairness": When calculating how many tokens a tenant gets, consider their
weight.tokens_allowed = base_rate * weight / total_weights.Code Plan:
- Class
Tenant: Holds local bucket state. - Class
SystemMonitor: Simulates health checks. - Class
Coordinator:acquire(tenant_id): Async method. ReturnsTrue(allowed) orfloat(wait time seconds).- Uses a
FairnessQueueto handle bursts.
Self-Correction during thought process: Just implementing a Token Bucket isn't enough. The prompt asks for "Dynamic" and "Backpressure". I should implement a Control Loop.
- Measure actual throughput vs. target.
- Adjust per-tenant limits to match target while respecting weights.
Let's build a Distributed-style Coordinator (even if single-node code) using a concept similar to TCP congestion control or Netflix's "concurrency limits".
Final Design for the Script:
TenantProfile: Priority, Weight.AdaptiveThrottler: The core class.BackpressureSignal: Enum (GREEN, YELLOW, RED).fairness_algorithm: Ensures that if the system is at 100% capacity, high-priority tenants get served first, or distribution is strictly weighted.
Let's write the Python code. I will include comments explaining the "Fairness", "Dynamic", and "Backpressure" sections.
Features:
- Priority levels: Critical, Standard, Free.
- Dynamic Scaling: If the system is healthy, limits expand (bursting). If pressure rises, limits contract globally.
- Fairness: Strict Weighted Fair Queuing (WFQ) simulation.
- Backpressure: Returns a
Retry-Afterdelay instead of just hard crashing.
Let's write the implementation.
Check: Is Python
asynciosufficient? Yes, it handles the timing/concurrency aspects well. Refinement on Dynamic Quotas: I will implement a "Token Bank" where unused tokens up to a limit are banked, and dynamic re-allocation happens periodically.Let's structure the output:
- High-level explanation.
- The Python Code.
- Explanation of the specific mechanisms used.
(Self-Correction): Ensure the code is runnable and demonstrates the interaction, perhaps with a simulated load test at the bottom.
Code Structure:
dataclassesfor config.asynciofor timing.SystemLoadSimulator: A class that fluctuates a global load variable.TenantBucket: Modified token bucket that accepts a "pressure factor".Coordinator: The main interface.
Mechanism for Backpressure:
- If
SystemLoad > 0.8, reduce all refill rates by1.0 - SystemLoad. - If
SystemLoad > 0.9, start rejecting non-critical requests entirely.
Mechanism for Fairness:
- Calculate
share = global_capacity * (tenant_weight / sum(all_weights)). - Update shares periodically.
Let's go. This is a design and implementation for a Multi-Tenant Throttling Coordinator in Python. It uses
asyncioto handle concurrent requests and implements a control loop that adjusts quotas based on system health (Backpressure) and tenant hierarchy (Fairness).Core Architecture
- Hierarchical Fairness: Tenants are grouped by tier (e.g.,
Priority,Standard,Free). Capacity is distributed top-down. - Dynamic Quotas: Instead of static limits, tenants have a "Base Quota" and a "Burst Quota." The system monitors overall load and dynamically shrinks or expands these windows.
- Backpressure Control: The coordinator monitors a simulated system load metric. As load increases (pressure builds), the coordinator aggressively reduces quotas to prevent system crash, prioritizing high-value traffic.
Python Implementation
import asyncio import time import random from dataclasses import dataclass, field from enum import Enum from typing import Dict, List, Optional from collections import deque # ========================================== # 1. Configuration & Data Structures # ========================================== class TenantTier(Enum): PLATINUM = 1 # Highest Priority GOLD = 2 SILVER = 3 FREE = 4 # Lowest Priority @dataclass class TenantConfig: id: str tier: TenantTier # Base RPS (Requests Per Second) guaranteed even under high load guaranteed_rps: int # Maximum RPS allowed when system is idle max_burst_rps: int @dataclass class TenantState: config: TenantConfig # Current active limit (adjusted dynamically) current_limit: float = 0.0 # Token bucket state tokens: float = 0.0 last_update: float = field(default_factory=time.time) # Metrics for fairness smoothing request_count: int = 0 rejected_count: int = 0 class SystemState(Enum): HEALTHY = "healthy" PRESSURE_WARNING = "warning" PRESSURE_CRITICAL = "critical" # ========================================== # 2. The Coordinator # ========================================== class ThrottlingCoordinator: def __init__(self, global_max_capacity: int): self.global_max_capacity = global_max_capacity self.tenants: Dict[str, TenantState] = {} self.system_load = 0.0 # Simulated load 0.0 to 1.0+ self.system_state = SystemState.HEALTHY # History for smoothing load calculation self.load_history = deque(maxlen=10) # Lock for thread-safe state updates self._lock = asyncio.Lock() async def register_tenant(self, config: TenantConfig): async with self._lock: state = TenantState(config=config) state.current_limit = config.max_burst_rps state.tokens = config.max_burst_rps # Start full self.tenants[config.id] = state print(f"[Coordinator] Registered Tenant {config.id} (Tier: {config.tier.name})") # ------------------------------------------ # Backpressure & Dynamic Quota Logic # ------------------------------------------ async def update_system_load(self, current_load: float): """ Called by the system monitor to report load. Triggers the dynamic recalculation of quotas. """ async with self._lock: self.load_history.append(current_load) # Smoothed average load self.system_load = sum(self.load_history) / len(self.load_history) # Determine System State if self.system_load > 0.9: self.system_state = SystemState.PRESSURE_CRITICAL elif self.system_load > 0.7: self.system_state = SystemState.PRESSURE_WARNING else: self.system_state = SystemState.HEALTHY await self._recalculate_quotas() async def _recalculate_quotas(self): """ The core fairness algorithm. Distributes global capacity based on Tier and System Pressure. """ if not self.tenants: return # 1. Determine Global Scale Factor based on Backpressure # If load is high, we reduce the total effective capacity available to tenants # to allow the system to drain its queues. if self.system_state == SystemState.PRESSURE_CRITICAL: capacity_factor = 0.5 # Shed 50% of traffic globally elif self.system_state == SystemState.PRESSURE_WARNING: capacity_factor = 0.8 # Shed 20% of traffic else: capacity_factor = 1.0 # Full capacity allowed effective_global_capacity = self.global_max_capacity * capacity_factor # 2. Group Tenants by Tier for Fairness calculation tiers = {} for t_state in self.tenants.values(): tiers.setdefault(t_state.config.tier, []).append(t_state) # 3. Distribute Capacity (Strict Priority / Weighted Fair Queuing) # We allocate capacity to PLATINUM first, then GOLD, etc. remaining_capacity = effective_global_capacity sorted_tiers = sorted(tiers.items(), key=lambda x: x[0].value) for tier, tenants in sorted_tiers: if remaining_capacity <= 0: # If we ran out of capacity, all lower tiers get minimum guaranteed for t in tenants: t.current_limit = t.config.guaranteed_rps continue # Calculate total demand for this tier total_tier_demand = sum(t.config.max_burst_rps for t in tenants) # Allocate to this tier tier_allocation = min(total_tier_demand, remaining_capacity) # Distribute allocation among tenants in this tier proportionally to their burst limits for t in tenants: if total_tier_demand == 0: continue proportion = t.config.max_burst_rps / total_tier_demand calculated_limit = tier_allocation * proportion # Apply Safety Clamp: Ensure limit is at least guaranteed, but not over max burst t.current_limit = max(t.config.guaranteed_rps, min(calculated_limit, t.config.max_burst_rps)) remaining_capacity -= tier_allocation # ------------------------------------------ # Request Handling (Token Bucket) # ------------------------------------------ async def acquire(self, tenant_id: str, cost: float = 1.0) -> bool: """ Attempt to consume tokens. Returns True if allowed, False if throttled. """ async with self._lock: tenant = self.tenants.get(tenant_id) if not tenant: return False now = time.time() elapsed = now - tenant.last_update tenant.last_update = now # Refill Tokens based on Dynamic Quota # Formula: NewTokens = Rate * Time tenant.tokens += tenant.current_limit * elapsed # Cap tokens at a small burst buffer (e.g., 1 second worth of burst) max_bucket = tenant.current_limit * 1.5 tenant.tokens = min(tenant.tokens, max_bucket) if tenant.tokens >= cost: tenant.tokens -= cost tenant.request_count += 1 return True else: tenant.rejected_count += 1 return False # ------------------------------------------ # Monitoring & Reporting # ------------------------------------------ async def status_reporter(self): while True: await asyncio.sleep(5) async with self._lock: print(f"\n--- System Status | Load: {self.system_load:.2f} | State: {self.system_state.name} ---") for tid, t in self.tenants.items(): acceptance_rate = 0 total = t.request_count + t.rejected_count if total > 0: acceptance_rate = (t.request_count / total) * 100 print(f"Tenant {tid} ({t.config.tier.name}): Limit {t.current_limit:.1f}/s | " f"Tokens: {t.tokens:.1f} | Success: {acceptance_rate:.1f}%") print("-----------------------------------------------------------------") # ========================================== # 3. Simulation (The Test) # ========================================== async def simulate_load(coordinator: ThrottlingCoordinator): """ Simulates fluctuating system load (Backpressure generation). """ while True: # Simulate a sine wave load pattern + random noise # Ranges between 0.3 and 1.1 base_load = 0.3 + (0.4 * (1 + math.sin(time.time() / 10))) noise = random.uniform(-0.1, 0.1) load = max(0, base_load + noise) await coordinator.update_system_load(load) await asyncio.sleep(1) async def tenant_worker(coordinator: ThrottlingCoordinator, tenant_id: str, aggression: float): """ Simulates a client sending requests. """ while True: # Simulate processing time await asyncio.sleep(random.uniform(0.01, 0.1) / aggression) allowed = await coordinator.acquire(tenant_id) # Simulated Backpressure reaction (Client side) # If rejected, client should ideally backoff slightly (simple simulation) if not allowed: await asyncio.sleep(0.05) async def main(): import math coordinator = ThrottlingCoordinator(global_max_capacity=1000) # 1. Define Tenants tenants = [ TenantConfig("API-Enterprise", TenantTier.PLATINUM, guaranteed_rps=50, max_burst_rps=200), TenantConfig("API-Startup", TenantTier.GOLD, guaranteed_rps=20, max_burst_rps=100), TenantConfig("API-Free-1", TenantTier.FREE, guaranteed_rps=5, max_burst_rps=20), TenantConfig("API-Free-2", TenantTier.FREE, guaranteed_rps=5, max_burst_rps=20), ] for t in tenants: await coordinator.register_tenant(t) # 2. Start Background Tasks tasks = [] # System Load Simulator tasks.append(asyncio.create_task(simulate_load(coordinator))) # Status Reporter tasks.append(asyncio.create_task(coordinator.status_reporter())) # 3. Start Tenant Workers (Traffic Generators) # Enterprise sends lots of traffic, Free sends moderate tasks.append(asyncio.create_task(tenant_worker(coordinator, "API-Enterprise", 2.0))) tasks.append(asyncio.create_task(tenant_worker(coordinator, "API-Startup", 1.0))) tasks.append(asyncio.create_task(tenant_worker(coordinator, "API-Free-1", 0.5))) tasks.append(asyncio.create_task(tenant_worker(coordinator, "API-Free-2", 0.5))) await asyncio.gather(*tasks) if __name__ == "__main__": # Run the simulation try: asyncio.run(main()) except KeyboardInterrupt: print("Simulation stopped.")How this works
1. The Dynamic Control Loop (
_recalculate_quotas)This is the brain of the coordinator. It does not rely on static configuration.
- Input: Current System Load (0.0 to 1.0+).
- Logic:
- Under 70% Load: The system is considered healthy. Tenants are allowed to burst up to their
max_burst_rps. The "Free" tier might get lots of resources. - 70% - 90% Load (Warning): The system applies a 20% global penalty (
capacity_factor = 0.8). - Over 90% Load (Critical): The system applies a 50% global penalty.
- Under 70% Load: The system is considered healthy. Tenants are allowed to burst up to their
- Fairness Algorithm: It allocates the remaining capacity using Strict Priority. If the system is full, the PLATINUM tenants get their full share first. If there is anything left, GOLD gets it. FREE tier gets whatever scraps remain. However, we always enforce the
guaranteed_rpsfloor so low-tier tenants aren't completely starved.
2. Backpressure Handling
The
simulate_loadfunction generates a sine wave of load.- When the load goes up, the coordinator detects it in
update_system_load. - It immediately reduces the
current_limitfor all tenants. - The
acquiremethod (Token Bucket) uses this newcurrent_limitto refill tokens. This means the flow of allowed requests slows down smoothly rather than hard-stopping.
3. Client-Side Reaction
In a real system, when a request is throttled (returns
False), the client should implement Exponential Backoff. In the code above,tenant_workersimply sleeps for a fixed short duration when rejected.Key Metrics to Observe
If you run this script, you will see the status report every 5 seconds:
- During Low Load: You will see
Limitequalsmax_burstfor everyone. - During High Load:
System StatebecomesCRITICAL.- The
Limitfor "Enterprise" (Platinum) will likely stay high or drop slightly. - The
Limitfor "Free" tenants will drop drastically (to nearguaranteed_rps). - The
Successrate for Free tenants will drop, protecting the Enterprise tenants.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.