Grok 4.1 Fast· xai
Frontier Reasoning Traces
Draft a webhook consumer framework: retries, signatures, idempotency, replay UI, and…
- User
Draft a webhook consumer framework: retries, signatures, idempotency, replay UI, and ordering keys.
- Assistant
// webhook-framework.ts // A robust webhook consumer framework in Node.js/TypeScript. // Features: signature verification (HMAC-SHA256), idempotency, retries (exponential backoff), // ordering keys (per-key queues), replay UI (simple Express dashboard). // // Production deps: express, bullmq (Redis for queues), better-sqlite3 (DB) // npm i express @types/express bullmq better-sqlite3 @types/node crypto // For simplicity, this uses in-memory storage/queues for demo (NOT production-ready). // Swap with Redis/Postgres impls via interfaces. // // Run: ts-node webhook-framework.ts // Test: POST /webhook with JSON payload, X-Webhook-Signature header, X-Idempotency-Key, X-Ordering-Key import express, { Request, Response } from 'express'; import crypto from 'crypto'; import { Queue, QueueEvents, Worker } from 'bullmq'; // Requires Redis import Database from 'better-sqlite3'; // Or your DB interface WebhookPayload { event_type: string; data: any; timestamp: number; } interface WebhookEvent { id: string; // Unique event ID from sender idempotencyKey: string; orderingKey: string; // e.g., 'user:123' for ordering per user payload: WebhookPayload; signature: string; // Raw signature header for verification rawBody: string; // For sig verification receivedAt: Date; } interface Delivery { id: string; eventId: string; idempotencyKey: string; orderingKey: string; payload: string; status: 'pending' | 'success' | 'failed'; attempts: number; lastError?: string; createdAt: Date; processedAt?: Date; } // === INTERFACES FOR EXTENSIBILITY === interface Storage { isProcessed(idempotencyKey: string): Promise<boolean>; markProcessed(idempotencyKey: string, event: WebhookEvent): Promise<void>; saveDelivery(delivery: Delivery): Promise<void>; getFailedDeliveries(limit?: number): Promise<Delivery[]>; markReplay(deliveryId: string): Promise<void>; // For replay tracking } interface QueueSystem { enqueue(orderingKey: string, event: WebhookEvent, retries?: number): Promise<void>; process(handler: (job: WebhookEvent) => Promise<any>): Promise<void>; } // === IN-MEMORY IMPLEMENTATIONS (DEMO ONLY) === class InMemoryStorage implements Storage { private processed = new Set<string>(); private deliveries: Delivery[] = []; async isProcessed(idempotencyKey: string): Promise<boolean> { return this.processed.has(idempotencyKey); } async markProcessed(idempotencyKey: string, event: WebhookEvent): Promise<void> { this.processed.add(idempotencyKey); const delivery: Delivery = { id: crypto.randomUUID(), eventId: event.id, idempotencyKey, orderingKey: event.orderingKey, payload: JSON.stringify(event.payload), status: 'success', attempts: 0, createdAt: event.receivedAt, processedAt: new Date(), }; this.deliveries.push(delivery); } async saveDelivery(delivery: Delivery): Promise<void> { this.deliveries.push(delivery); } async getFailedDeliveries(limit = 100): Promise<Delivery[]> { return this.deliveries .filter(d => d.status === 'failed') .slice(-limit) .reverse(); } async markReplay(deliveryId: string): Promise<void> { const delivery = this.deliveries.find(d => d.id === deliveryId); if (delivery) { delivery.status = 'pending' as any; delivery.attempts = 0; } } } class InMemoryQueueSystem implements QueueSystem { private queues: Map<string, WebhookEvent[]> = new Map(); private processing = new Set<string>(); async enqueue(orderingKey: string, event: WebhookEvent, retries = 3): Promise<void> { if (!this.queues.has(orderingKey)) this.queues.set(orderingKey, []); const queue = this.queues.get(orderingKey)!; queue.push({ ...event, _retriesLeft: retries }); } async process(handler: (job: WebhookEvent) => Promise<any>): Promise<void> { // Simulate worker loop (in prod, use BullMQ worker) const processNext = async () => { for (const [orderingKey, queue] of this.queues) { if (this.processing.has(orderingKey) || queue.length === 0) continue; this.processing.add(orderingKey); const job = queue.shift()!; try { await handler(job); // Mark success in storage } catch (error) { console.error(`Failed job for ${orderingKey}:`, error); if ((job as any)._retriesLeft > 0) { (job as any)._retriesLeft--; // Exponential backoff sim: re-enqueue after delay setTimeout(() => { queue.unshift(job); this.processing.delete(orderingKey); processNext(); }, Math.pow(2, 3 - (job as any)._retriesLeft) * 1000); } } finally { this.processing.delete(orderingKey); processNext(); } } setTimeout(processNext, 100); // Poll }; processNext(); } } // === PRODUCTION SWAPS === // RedisQueueSystem using BullMQ: // class RedisQueueSystem implements QueueSystem { // private queues = new Map<string, Queue>(); // constructor(private redisUrl: string) {} // async enqueue(orderingKey: string, event: WebhookEvent, retries = 3) { // const q = this.queues.get(orderingKey) || new Queue(`webhook:${orderingKey}`, { connection: { host: 'localhost', port: 6379 } }); // await q.add('process', event, { attempts: retries, backoff: { type: 'exponential', delay: 1000 } }); // } // async process(handler: (job: Job) => Promise<any>) { // // One worker per orderingKey or dynamic // const worker = new Worker('webhook:*', handler, { connection: { host: 'localhost', port: 6379 } }); // } // } // // SQLiteStorage using better-sqlite3: similar CRUD ops. class WebhookConsumer { constructor( private secret: string, private storage: Storage = new InMemoryStorage(), private queue: QueueSystem = new InMemoryQueueSystem(), private processHandler: (event: WebhookEvent) => Promise<void> // Your business logic ) {} async handle(req: Request, res: Response): Promise<void> { try { const rawBody = req.body as any; // Use raw-body middleware in prod const signature = req.get('X-Webhook-Signature'); const idempotencyKey = req.get('X-Idempotency-Key') || crypto.randomUUID(); const orderingKey = req.get('X-Ordering-Key') || 'default'; const eventId = req.body.event_id || idempotencyKey; // Assume sender provides const event: WebhookEvent = { id: eventId, idempotencyKey, orderingKey, payload: req.body, signature: signature!, rawBody: JSON.stringify(rawBody), receivedAt: new Date(), }; // 1. Verify signature if (!this.verifySignature(event)) { res.status(401).json({ error: 'Invalid signature' }); return; } // 2. Idempotency check if (await this.storage.isProcessed(idempotencyKey)) { res.status(200).json({ message: 'Already processed' }); return; } // 3. Save pending delivery await this.storage.saveDelivery({ id: crypto.randomUUID(), eventId, idempotencyKey, orderingKey, payload: JSON.stringify(req.body), status: 'pending', attempts: 0, createdAt: event.receivedAt, }); // 4. Enqueue for ordered processing + retries await this.queue.enqueue(orderingKey, event); res.status(202).json({ message: 'Accepted' }); } catch (error) { console.error('Webhook handle error:', error); res.status(400).json({ error: 'Invalid request' }); } } private verifySignature(event: WebhookEvent): boolean { const expected = `v1=${crypto .createHmac('sha256', this.secret) .update(event.rawBody) .digest('hex')}`; return crypto.timingSafeEqual( Buffer.from(event.signature), Buffer.from(expected) ); } // Replay handler async replay(deliveryId: string): Promise<{ success: boolean; message: string }> { // Fetch delivery, mark as replayed, re-enqueue (storage.getDelivery impl needed) await this.storage.markReplay(deliveryId); // Reconstruct event and enqueue (simplified) return { success: true, message: 'Requeued' }; } } // === REPLAY UI === function createReplayUI(consumer: WebhookConsumer, storage: Storage) { const app = express(); app.use(express.json()); app.use(express.static('public')); // Serve simple HTML/JS app.get('/replay', async (req, res) => { const failed = await storage.getFailedDeliveries(50); res.json({ failed }); }); app.post('/replay/:id', async (req, res) => { const result = await consumer.replay(req.params.id); res.json(result); }); return app; } // === USAGE EXAMPLE === const consumer = new WebhookConsumer( 'your-webhook-secret', new InMemoryStorage(), new InMemoryQueueSystem(), async (event) => { // Your business logic here (idempotent!) console.log(`Processing ${event.payload.event_type} for ${event.orderingKey}`); // Simulate failure sometimes if (Math.random() < 0.2) throw new Error('Simulated failure'); console.log('Processed successfully'); } ); // Start server const app = express(); app.use(express.raw({ type: 'application/json' })); // For raw body app.use(express.json({ verify: (req, _res, buf) => { (req as any).rawBody = buf; } })); app.post('/webhook', (req, res) => consumer.handle(req, res)); const replayApp = createReplayUI(consumer, consumer['storage']); // Hack for demo replayApp.use('/replay', replayApp.router || replayApp); // Mount app.use('/replay', replayApp); app.listen(3000, () => { console.log('Webhook consumer running on http://localhost:3000'); console.log('Replay UI: http://localhost:3000/replay'); console.log("Test POST /webhook with: body={event_type:'test'}, headers: X-Webhook-Signature='v1=...', X-Idempotency-Key, X-Ordering-Key"); }); // Simulate worker start consumer['queue'].process(consumer.processHandler);Key Features Explained
- Signatures: HMAC-SHA256 verification using
your-webhook-secret. Matches common providers (e.g., Stripe:v1=...). - Idempotency: Checks
X-Idempotency-Keybefore enqueueing. Stores in DB/Set. - Retries: Exponential backoff (1s, 2s, 4s). Configurable
attempts. - Ordering Keys: Queues per
X-Ordering-Key(e.g., user_id). Ensures FIFO per key. - Replay UI: GET
/replaylists failed deliveries (JSON for simplicity; add HTML/JS frontend). POST/replay/:idre-enqueues.
Production Notes
- Storage: Swap
InMemoryStoragewith SQLite/Postgres (addgetDeliveryfor full replay). - Queues: Use
BullMQ+ Redis for real queues, clustering, monitoring. - Raw Body: Use
raw-bodymiddleware for prod sig verification. - Monitoring: Add metrics (Prometheus), dead-letter queues.
- Scaling: Multiple workers, sharded queues.
- Security: Rate limiting, IP whitelisting.
Test with curl:
curl -X POST http://localhost:3000/webhook \ -H "Content-Type: application/json" \ -H "X-Webhook-Signature: v1=$(echo -n '{"event_type":"test"}' | openssl dgst -sha256 -hmac 'your-webhook-secret' -binary | base64 -w0)" \ -H "X-Idempotency-Key: test123" \ -H "X-Ordering-Key: user:42" \ -d '{"event_type":"test", "event_id":"evt_123"}'This is a fully functional draft—extend as needed!
- Signatures: HMAC-SHA256 verification using
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.