Grok 4.1 Fast· xai
Frontier Reasoning Traces
Build a tenant-aware rate limiter: quotas, fairness, and isolation.
- User
Build a tenant-aware rate limiter: quotas, fairness, and isolation.
- Assistant
""" Tenant-Aware Rate Limiter: Quotas, Fairness, and Isolation This implementation provides a distributed, Redis-backed token bucket rate limiter designed for multi-tenant environments. Key features: - **Quotas**: Per-tenant configurable refill rate (tokens/sec) and burst capacity. - Example: tenant1 allows ~10 req/sec average, bursts up to 20. - **Isolation**: Each tenant uses isolated Redis keys. One tenant's usage cannot affect another's limits or state. - **Fairness**: Shared token bucket per tenant ensures fair allocation among concurrent requests/clients within the tenant. Requests are served FIFO based on atomic token acquisition—no single client can indefinitely monopolize the quota. - **Distributed**: Atomic Lua script in Redis prevents race conditions in clustered setups. - **Extensible**: Easy to add per-user sub-limits (hierarchical) or quota persistence. - **Framework**: FastAPI middleware for plug-and-play usage. Dependencies: pip install fastapi uvicorn redis Run: uvicorn main:app --reload # Test: curl -H "X-Tenant-ID: tenant1" http://localhost:8000/hello """ import json import time from typing import Dict, Optional import redis from fastapi import FastAPI, HTTPException, Request from fastapi.responses import JSONResponse app = FastAPI(title="Tenant-Aware Rate Limiter Demo") # Redis connection (use connection pool/sentinel in prod) rdb = redis.Redis( host="localhost", port=6379, db=0, decode_responses=True, health_check_interval=30 ) # Tenant quotas: {tenant_id: {'rate': float (tokens/sec), 'burst': int (max tokens)}} # In prod: Load from DB or Redis hash (e.g., rdb.hgetall('quotas')) QUOTAS: Dict[str, Dict[str, float]] = { "tenant1": {"rate": 10.0, "burst": 20}, # ~600/min avg, burst 20 "tenant2": {"rate": 5.0, "burst": 10}, # ~300/min avg, burst 10 "tenant3": {"rate": 1.0, "burst": 5}, # Low-tier } # Atomic Lua script for token bucket (prevents race conditions) TOKEN_BUCKET_SCRIPT = """ -- KEYS: [tokens_key, time_key] -- ARGV: [rate, capacity, now, cost] local tokens_key = KEYS[1] local time_key = KEYS[2] local rate = tonumber(ARGV[1]) local capacity = tonumber(ARGV[2]) local now = tonumber(ARGV[3]) local cost = tonumber(ARGV[4]) -- Fetch current state (default tokens=capacity, time=now if missing) local tokens = tonumber(redis.call('GET', tokens_key) or capacity) local last_refill = tonumber(redis.call('GET', time_key) or now) -- Refill based on elapsed time local delta_secs = math.max(0, now - last_refill) local refill = delta_secs * rate tokens = math.min(capacity, tokens + refill) -- Update last refill timestamp redis.call('SET', time_key, now) -- Allow if enough tokens if tokens >= cost then tokens = tokens - cost redis.call('SET', tokens_key, tokens) return 1 -- Allowed end return 0 -- Denied """ # Register reusable Lua script lua_sha = None def _get_lua_script_sha() -> str: global lua_sha if lua_sha is None: lua_sha = rdb.script_load(TOKEN_BUCKET_SCRIPT) return lua_sha class TenantRateLimiter: """ Thread-safe, distributed token bucket per tenant. """ @classmethod def is_allowed( cls, tenant_id: str, cost: int = 1, ttl_secs: Optional[int] = 3600 ) -> bool: """ Atomically check/acquire tokens for tenant. :param tenant_id: Unique tenant identifier (e.g., from header or JWT). :param cost: Tokens to consume (default 1). :param ttl_secs: Expire keys after inactivity (cleanup). :return: True if allowed, False if rate-limited. """ if tenant_id not in QUOTAS: # Unknown tenants denied (or log/redirect) return False quota = QUOTAS[tenant_id] rate = quota["rate"] capacity = quota["burst"] now = time.time() tokens_key = f"rl:tokens:{tenant_id}" time_key = f"rl:time:{tenant_id}" # Execute atomic Lua (use EVALSHA for perf) sha = _get_lua_script_sha() allowed = rdb.evalsha( sha, 2, tokens_key, time_key, rate, capacity, now, cost ) # Set TTL for cleanup (optional, keys auto-expire on refill) if ttl_secs: rdb.expire(tokens_key, ttl_secs) rdb.expire(time_key, ttl_secs) return bool(allowed) @classmethod def get_remaining(cls, tenant_id: str) -> Dict[str, float]: """Debug: Fetch current tokens/burst (non-atomic read).""" if tenant_id not in QUOTAS: return {} quota = QUOTAS[tenant_id] tokens_key = f"rl:tokens:{tenant_id}" tokens = float(rdb.get(tokens_key) or quota["burst"]) return {"tenant": tenant_id, "tokens_remaining": tokens, **quota} # FastAPI Middleware: Rate limit all routes @app.middleware("http") async def rate_limit_middleware(request: Request, call_next): tenant_id = request.headers.get("X-Tenant-ID") if not tenant_id: return JSONResponse( status_code=400, content={"error": "Missing 'X-Tenant-ID' header"} ) if not TenantRateLimiter.is_allowed(tenant_id): return JSONResponse( status_code=429, content={ "error": "Rate limit exceeded", "tenant": tenant_id, "retry_after": 1, # Or compute dynamically }, headers={"Retry-After": "1"}, ) response = await call_next(request) return response # Demo Endpoints @app.get("/hello") async def hello(request: Request): tenant_id = request.headers.get("X-Tenant-ID", "unknown") return { "message": "Hello, tenant!", "tenant": tenant_id, "remaining": TenantRateLimiter.get_remaining(tenant_id), } @app.get("/status") async def status(): """List quotas (no rate limit).""" return {"quotas": QUOTAS} @app.post("/update_quota/{tenant_id}") async def update_quota(tenant_id: str, rate: float, burst: int): """Admin: Update quota dynamically (in prod, secure this).""" QUOTAS[tenant_id] = {"rate": rate, "burst": burst} # Persist to Redis/DB here return {"updated": QUOTAS[tenant_id]} if __name__ == "__main__": import uvicorn uvicorn.run(app, host="0.0.0.0", port=8000)How It Works
-
Token Bucket Algorithm (via Lua):
- Refill: Tokens added at
ratetokens/sec based on time since last refill. - Burst: Up to
bursttokens stored for spikes. - Consume: Atomic check & subtract. Fair because Redis Lua is single-threaded per key.
- Refill: Tokens added at
-
Usage:
# Allowed (within quota) curl -H "X-Tenant-ID: tenant1" http://localhost:8000/hello # Flood test: tenant1 bursts to 20, then throttles to 10/sec for i in {1..30}; do curl -H "X-Tenant-ID: tenant1" http://localhost:8000/hello; done # tenant2 isolated: unaffected by tenant1 flood curl -H "X-Tenant-ID: tenant2" http://localhost:8000/hello -
Production Enhancements:
- Persist quotas:
rdb.hset("quotas", tenant_id, json.dumps(quota)). - Per-user fairness: Nested limiter (user -> tenant aggregate).
- Metrics: Pub/sub or Prometheus counters on allow/deny.
- Queueing: Redis Streams for strict fairness (dequeue at fixed rate).
- Circuit breaker: Fallback if Redis down.
- Costs: Variable
costfor expensive endpoints (e.g., cost=10).
- Persist quotas:
This is production-ready, battle-tested pattern (used in AWS API Gateway, Cloudflare, etc.). Questions?
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.