Design a read-replica lag-aware routing layer: staleness budgets, consistency hints,…
- User
Design a read-replica lag-aware routing layer: staleness budgets, consistency hints, retry policies, and observability.
- Assistant
Read-Replica Lag-Aware Routing Layer Design
This design outlines a high-performance, configurable routing layer (e.g., implemented as a Rust/Go proxy like
pgcator Envoy filter) that intelligently routes read queries to read replicas while respecting staleness budgets, consistency hints, retry policies, and providing rich observability. It assumes a primary-replica setup (e.g., PostgreSQL with streaming replication, CockroachDB, or Vitess shards).1. High-Level Architecture
Clients ──(gRPC/HTTP/TCP)──> Router Layer ──┬──> Primary (writes + strong reads) ├──> Replica 1 (lag: 200ms) ├──> Replica 2 (lag: 1.2s) └──> Replica N (lag: 50ms)- Router Layer: Stateless, horizontally scalable service (e.g., via Kubernetes). Uses consistent hashing for session affinity (e.g., preserve transaction state).
- Key Assumptions:
- Replication uses logical/physical streaming (e.g., PostgreSQL WAL).
- Queries are classified as READ/WRITE via SQL parsing (e.g.,
mozpgorpg_queryparser). - Multi-tenant support (budgets per tenant/user).
Core Data Structures: - ReplicaPool: { id, endpoint, current_lag_ns: u64, health: bool, load: f64 } - Ring: Consistent hash ring for load balancing across eligible replicas.2. Lag Monitoring (Foundation)
Accurate, low-overhead lag measurement is critical.
-
Metrics per Replica:
Metric Source Computation replication_lag_ns Primary: pg_current_wal_lsn()Replica:pg_last_wal_replay_lsn()+pg_last_wal_replay_lsn_time()(primary_lsn_time - replica_lsn_time) * 1e9(nanoseconds)query_lag_ns Replica heartbeat now() - commit_time_of_last_lsn(viapg_xact_commit_timestamp)load_factor Prometheus scrape CPU/Connections normalized [0-1] -
Monitoring Loop (every 100ms):
- Primary broadcasts LSN/timestamp via gossip (e.g., Redis pub/sub or etcd watch).
- Replicas report via heartbeat endpoint (
/health?lsn=0xABC&ts=1234567890). - Router computes
effective_lag_ns = max(replication_lag_ns, query_lag_ns). - Health check: Mark unhealthy if lag > 10s or heartbeat miss > 3x.
-
Edge Case: Clock skew → Use primary-relative timestamps.
3. Staleness Budgets
Per-request configurable max staleness (default: 5s / 5e9 ns).
-
Configuration:
staleness_budgets: default: 5s tenant_A: { reads: 1s, analytics: 30s } global_max: 60s # Failover to primary beyond this -
Enforcement:
- Resolve
effective_lag_ns <= budget_ns. - Dynamic budgets: Scale by query type (e.g.,
SELECT COUNT(*)allows 2x budget).
- Resolve
4. Consistency Hints
Client-driven overrides via headers/query params (e.g.,
X-Consistency: bounded(2s)).Hint Semantics Routing Behavior strongLinearizable (no staleness) Primary only. bounded(Xs)Staleness ≤ X Filter replicas where lag <= X.eventualAny healthy replica Cheapest eligible (min lag + load). nearestLowest latency Geo-aware + lag filter. noneDisable routing Direct to primary. - Parsing: Extract from
X-Routing-Hints: consistency=bounded(5s);tenant=prod. - Fallback: If hint unmet, use retry policy.
5. Routing Logic
Atomic, sub-ms decision per query.
// Pseudocode fn route(query: ParsedQuery, hints: Hints) -> Target { let budget = resolve_budget(hints.consistency, query.tenant); let candidates = replica_pool .filter(|r| r.health && r.effective_lag_ns <= budget) .sort_by(|a, b| cmp_lag_then_load(a, b)); // Primary sort: lag asc, secondary: load asc if candidates.is_empty() { return retry_policy.select_fallback(hints); } // Load balance via consistent hash ring (key: session_id + query_hash) ring.pick(candidates, session_key(query)) }-
Load Balancing:
- Weighted round-robin on ring.
- Adaptive: Demote high-lag replicas (e.g., score = lag_ns * load_factor).
-
Query Classification:
Type Route WRITE/DML Primary. READ (SELECT) Replica if budget met. Transactions Sticky to primary until COMMIT.
6. Retry Policies
Handle "no eligible replica" gracefully.
-
Policy Config (per-tenant):
retry: max_attempts: 3 backoff: exp(50ms, 500ms, jitter=10%) fallbacks: [ { delay: 100ms, retry_replicas: true }, { target: primary, strict: true }, { error: "STALE_UNAVAILABLE" } ] circuit_breaker: { error_rate: 0.5, window: 10s } -
Behavior:
- Retry Replicas: Wait + re-monitor (most common).
- Primary Fallback: Transparent redirect.
- Client Error: With hint metadata (e.g.,
retry-after: 200ms).
- Circuit breakers per replica/tenant to avoid thundering herd.
-
Idempotency: Use query fingerprint + client nonce for safe retries.
7. Observability
Zero-overhead where possible (async metrics).
Metrics (Prometheus)
router_requests_total{op=read/write, target=primary/replica_id, tenant, consistency_hint, eligible=true/false} # Counter router_latency_us{p99, target} # Histogram replica_lag_ns{replica_id, p50/p95/p99} # Gauge + Histogram fallback_rate{reason=no_candidates/primary} # Ratio routing_efficiency = eligible_replicas / total_replicas # Gauge [0-1]Tracing (OpenTelemetry/ Jaeger)
span: router-route tags: { lag_ns=200e6, budget_ns=5e9, selected=rep-2, eligible_count=3 } child: db-execute { sql_fingerprint }Logging/Alerts
- Logs: Structured JSON, e.g.,
{"level":"warn","msg":"Fallback to primary","tenant":"A","min_lag_seen":8.2,"budget":5.0} - Alerts:
Condition Alert p95_lag > 10s "High replication lag" fallback_rate > 5% "Routing pressure" eligible_replicas < 1 "No read capacity"
Dashboards
- Grafana: Lag heatmap by replica/tenant, routing funnel (eligible → selected → success).
Implementation Notes
- Languages: Rust (Tokio + sqlparser-rs) for speed/safety; Go (pgx) alternative.
- Deployment: Envoy + WASM filter for L7 routing, or standalone proxy.
- Testing:
- Chaos: Inject lag/delays (e.g., Toxiproxy).
- Benchmarks: 1M qps, <1ms routing overhead.
- Extensions: Geo-routing (Cloudflare Workers), sharding (Vitess integration), ML lag prediction.
This design ensures 99.99% read scalability with tunable consistency, minimizing primary load while exposing failures transparently. Questions?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.