Design a resilient webhook dispatcher: retries, jitter, dead-letter, and signatures.
- User
Design a resilient webhook dispatcher: retries, jitter, dead-letter, and signatures.
- Assistant
Resilient Webhook Dispatcher Design
As Sherlock from Trailblazer Labs, I'll design a production-grade webhook dispatcher that's fault-tolerant, scalable, and secure. This system handles incoming events (e.g., from your app's database triggers or Kafka), queues them for delivery to external webhook endpoints, and ensures delivery with retries (exponential backoff), jitter (to prevent thundering herds), dead-letter queue (DLQ) for failed deliveries, and signatures (HMAC-SHA256 for authenticity and replay protection).
🎯 Key Requirements & Guarantees
- At-least-once delivery (with idempotency hints for receivers).
- Handles transient failures (5xx, timeouts, connection errors).
- Permanent failures (4xx) go straight to DLQ.
- Scalable: Horizontal worker scaling.
- Observable: Metrics, logs, tracing.
- Secure: Payload signing with per-webhook secrets.
🏗️ High-Level Architecture
[Event Sources] --> [Ingress Queue (RabbitMQ)] | v [Worker Pool] <--> [Retry Queues (x5, delayed)] <--> [DLQ] | v [HTTP Client] --> [External Webhook Endpoints] | v [Metrics/Logs] --> [Prometheus/Grafana/ELK]- Message Queue: RabbitMQ (durable, supports DLX, delayed messages via plugin).
- Workers: Go (goroutines for efficiency) or Node.js/Python.
- Storage: Redis for in-flight state (optional idempotency keys).
- Secrets: Vault or env vars per subscriber (webhook config).
Data Flow
- Ingress: Event arrives
{webhook_url, payload, secret_id, idempotency_key?, metadata}→ Enqueue towebhooks.in. - Dispatch:
- Worker dequeues.
- Lookup secret (from DB/Vault).
- Generate signature.
- POST to
webhook_url.
- Success (2xx): Ack message.
- Retryable Fail (5xx, timeout): Requeue to
webhooks.retry.N(N=1..5) with exponential delay + jitter. - Permanent Fail (4xx): Straight to DLQ.
- Max Retries Exhausted: DLX auto-routes to
webhooks.dlq.
🔄 Retry Mechanism
- Max Retries: 5.
- Backoff: Exponential:
base_delay * 2^(retry-1)(starts at 1s: 1s, 2s, 4s, 8s, 16s). - Jitter: Full jitter formula:
delay = retry_delay * random(0.5, 1.5)(avoids synchronization). - Timeout: 30s per attempt.
- Implementation: RabbitMQ Delayed Message Plugin. Message headers:
{retry_count: 1, next_delay_ms: 1500}.
Pseudocode for Retry Logic:
func dispatch(msg WebhookMsg) error { secret := lookupSecret(msg.SecretID) signedReq := signPayload(msg.Payload, secret, time.Now().UTC()) resp, err := http.Post(msg.URL, signedReq) if err != nil || resp.StatusCode >= 500 { // Retryable if msg.RetryCount >= 5 { return toDLQ(msg) // Or let DLX handle } delay := calculateBackoff(msg.RetryCount) requeueWithDelay(msg, delay) return ErrRetry } else if resp.StatusCode >= 400 { return toDLQ(msg) // Permanent fail } return nil } func calculateBackoff(retry int) time.Duration { base := time.Second * time.Duration(1<<retry) // 1,2,4,8,16s jitter := 0.5 + rand.Float64() // 0.5-1.5 return time.Duration(float64(base) * jitter) }📮 Queue Configuration (RabbitMQ)
Use exchanges/bindings for routing:
Queue/Exchange Purpose TTL/Delay DLX x.webhooks.in(fanout)Ingress None - q.webhooks.inMain dispatch None x.webhooks.retryq.webhooks.retry.1Retry 1 x-delay: 1-1.5s x.webhooks.retry... ... ... (up to 16-24s) x.webhooks.retryq.webhooks.dlqFailed after 5 retries TTL: 7d Alert on enqueue RabbitMQ Config Snippet (via
rabbitmqctlor Terraform):rabbitmq-plugins enable rabbitmq_delayed_message_exchange # Declare delayed exchange: x-delayed-message with type x-delayed-message # Bind retry queues with DLX🔐 Signatures
- Algorithm: HMAC-SHA256.
- Signed Data: Canonical JSON(payload) +
|+ timestamp (ISO8601). - Header:
X-Webhook-Signature: t=<ts>,v1=<hmac>. - Verification (receiver-side example):
const sig = crypto.createHmac('sha256', secret) .update(payload + '|' + ts) .digest('hex'); if (headers['x-webhook-signature'] !== `t=${ts},v1=${sig}`) throw new Error('Invalid sig'); - Replay Protection: Receiver checks
tswithin 5min window + stores recent nonces. - Dispatcher Code:
func signPayload(payload []byte, secret string, ts time.Time) http.Request { canon := canonicalJSON(payload) + "|" + ts.Format(time.RFC3339) hmac := hmac.New(sha256.New, []byte(secret)) hmac.Write([]byte(canon)) sig := hex.EncodeToString(hmac.Sum(nil)) req.Header.Set("X-Webhook-Signature", fmt.Sprintf("t=%s,v1=%s", ts.Format(time.RFC3339), sig)) return req }
🛡️ Resilience & Observability
- Idempotency: Optional
idempotency_keyin payload/header. Receiver dedupes. - Circuit Breaker: Per-endpoint (e.g., using
gobreaker). Track failure rate >50% → open for 1min. - Rate Limiting: Global (Redis) + per-endpoint (10 req/min default).
- Metrics (Prometheus):
Metric Description webhook_dispatched_totalBy status (success/fail/retry). webhook_retry_countHistogram of retries. webhook_queue_lengthPer queue. webhook_duration_msP95 latency. - Logging: Structured JSON with trace ID (e.g., OpenTelemetry).
- Alerts: DLQ length >10 → PagerDuty.
- Scaling: Kubernetes Deployment (10-100 replicas), HPA on queue length.
🚀 Implementation Stack
- Language: Go (fast, concurrent).
- Queue: RabbitMQ +
streadway/amqp. - HTTP:
restyorfasthttpwith timeouts/retries. - Config: Viper + env.
- Deploy: Docker/K8s, with RabbitMQ operator.
Starter Go Repo Structure:
webhook-dispatcher/ ├── cmd/worker/main.go ├── internal/ │ ├── queue/ │ ├── dispatcher.go │ ├── signer.go │ └── metrics/ ├── config.yaml └── docker-compose.yml (RabbitMQ + app)📊 Example Metrics Dashboard
Queue Depth: ingress=0, retry.1=5, dlq=2 Success Rate: 99.8% P95 Latency: 120ms Top Failures: endpoint.com (timeout, 3%)Potential Extensions
- Kafka for higher throughput (instead of RabbitMQ).
- Async Signatures (pre-sign in ingress).
- Webhook Registry DB: Store subscriber configs (URL, secret, retries).
This design survives 99.99% uptime outages, handles 10k+ EPS, and costs ~$50/mo on AWS (EKS + RabbitMQ). Need code, Terraform, or a prototype? Let me know! 🕵️♂️
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.