Build a continuous verification pipeline for service-level objectives that replays…
- User
Build a continuous verification pipeline for service-level objectives that replays production traces against shadow environments.
- Assistant
Continuous Verification Pipeline for SLOs Using Production Trace Replay in Shadow Environments
I'll outline a complete, production-ready pipeline to continuously verify Service-Level Objectives (SLOs) by capturing production traces, storing them, replaying them against shadow environments (e.g., staging clusters), and enforcing SLO gates in your CI/CD workflow. This ensures changes don't regress SLOs like latency (p99 < 500ms), error rate (< 0.1%), or availability.
The pipeline is event-driven and GitOps-native, integrating with tools like OpenTelemetry (OTel), Kafka for trace storage, GoReplay for traffic replay, Prometheus for SLO metrics, and GitHub Actions/ArgoCD for orchestration. It runs on every PR merge, deploy, or cron schedule.
High-Level Architecture
Production ──(OTel Traces)──> Kafka/ClickHouse ──(Replay Triggers)──> Shadow Env (K8s Staging) │ │ ▼ ▼ Trace Storage SLO Metrics (Prometheus) │ │ ▼ ▼ CI/CD Pipeline (GitHub Actions/ArgoCD) ───┐── SLO Gate (Alert/Block Deploy) │ Verification Dashboard (Grafana)- Trace Capture: Instrument services with OTel to export traces to Kafka.
- Storage: Durable, queryable store (ClickHouse for analytics).
- Replay: Shadow deploy new code, replay traces, measure SLOs.
- Verification: Compare baseline vs. new SLOs; fail pipeline if regression.
- Continuous: Triggers on git push, deploy events.
Prerequisites
- Kubernetes cluster for shadow env (separate namespace:
shadow-*). - Services instrumented with OpenTelemetry (auto-instrumentation for Java/Node/Python/Go).
- GitHub repo with CI/CD.
- AWS/GCP for storage (S3/ GCS for traces if not Kafka).
Step 1: Instrument Production for Trace Capture
Use OpenTelemetry Collector to capture traces without performance overhead.
otel-collector-config.yaml (DaemonSet in prod):
receivers: otlp: protocols: grpc: {} exporters: kafka: brokers: "kafka.prod:9092" topic: "traces" processors: batch: {} service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [kafka]Deploy via Helm:
helm install otel-collector open-telemetry/opentelemetry-collector -f otel-collector-config.yaml -n monitoring.Traces include: HTTP requests, spans, metadata (headers, body hashes for replay).
Step 2: Store and Query Traces
Use ClickHouse for efficient storage/querying of traces (parses OTel protobuf).
Kafka to ClickHouse Connector (Confluent or custom):
# Docker Compose for dev docker run -d --name clickhouse \ clickhouse/clickhouse-server --ulimit nofile=262144:262144 # Create table curl -X POST 'http://localhost:8123/?query=CREATE%20TABLE%20traces%20(%20trace_id%20String,%20timestamp%20DateTime,%20request_headers%20Map(String,String),%20request_body%20String,%20service%20String%20)%20ENGINE%20MergeTree()%20ORDER%20BY%20(timestamp)'Custom Kafka consumer (Python) to parse/store:
import kafka, clickhouse_connect, json from opentelemetry.proto.trace.v1 import trace_pb2 consumer = kafka.KafkaConsumer('traces', bootstrap_servers=['kafka:9092']) client = clickhouse_connect.get_client(host='localhost') for msg in consumer: trace = trace_pb2.TracesData.FromString(msg.value) # Parse protobuf for batch in trace.resource_spans: # Extract request (first span), store as JSON req_data = json.dumps({ 'trace_id': trace_id, 'method': spans[0].attributes['http.method'], 'url': spans[0].attributes['http.url'], 'headers': dict(spans[0].attributes['http.request.headers']), 'body': extract_body(spans[0]) # Sanitize PII }) client.insert('traces', [{'trace_id': trace_id, 'request': req_data}])Retention: TTL 7 days; sample 10% traffic to avoid storage bloat.
Step 3: Shadow Environment Setup
Mirror prod K8s but discard responses (no feedback to prod).
argo-cd-app-shadow.yaml (GitOps deploy):
apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: my-service-shadow spec: source: repoURL: https://github.com/yourorg/service targetRevision: HEAD # Auto-sync on merge destination: namespace: shadow-v1 # Dynamic namespaces per version syncPolicy: automated: { prune: true, selfHeal: true }Services in shadow: Identical replicas, but outbound traffic routed to prod mocks (Istio VirtualService).
Step 4: Trace Replay Mechanism
Use GoReplay (goreplay.io) for low-latency replay. Deploy as K8s Job per verification run.
replay-job.yaml (templated by CI):
apiVersion: batch/v1 kind: Job metadata: name: slo-replay-{{ .CommitSHA }} spec: template: spec: containers: - name: goreplay image: goproxy.io/goreplay:latest command: ["/bin/goreplay"] args: - "--input-raw" - "--input-file=traces-{{ .StartTime }}.har" # Downloaded traces - "--output-http=shadow-service:8080" # Target shadow - "--shadow" # Discard shadow responses resources: { requests: { cpu: "1", memory: "2Gi" } } restartPolicy: NeverHAR Conversion: Traces → HAR (HTTP Archive) for replay.
# Node.js script (ci-replay.js) const traces = await fetchTracesFromClickHouse(startTime, endTime, service='api'); const har = traces.map(t => ({ log: { version: '1.2', entries: [{ request: { method: t.method, url: t.url, headers: t.headers, postData: { text: t.body } } }] } })); fs.writeFileSync('traces.har', JSON.stringify(har));Scale: Replay 1k RPS; use multiple pods.
Step 5: SLO Computation and Verification
Prometheus in Shadow Namespace:
- Scrape shadow service metrics (latency, errors via OTel metrics exporter).
- SLO Queries:
# Error Rate SLO: < 0.1% sum(rate(http_server_errors_total[5m])) / sum(rate(http_server_requests_total[5m])) < 0.001 # p99 Latency SLO: < 500ms histogram_quantile(0.99, sum(rate(http_server_duration_seconds_bucket[5m])) by (le)) < 0.5
Custom SLO Verifier (Go binary, run post-replay):
package main import ( "context" "github.com/prometheus/client_golang/prometheus/promhttp" "github.com/thanos-io/thanos/pkg/promclient" ) func verifySLOs(ctx context.Context, promURL string, baseline map[string]float64) error { client, _ := promclient.New(promURL) q, _ := client.Query(ctx, "histogram_quantile(0.99, rate(http_server_duration_seconds_bucket[5m]))") current := parseFloat(q) if current > baseline["p99_latency"]*1.1 { // 10% regression tolerance return fmt.Errorf("p99 latency regressed: %f > %f", current, baseline["p99_latency"]) } return nil }Step 6: CI/CD Pipeline (GitHub Actions)
Triggers on
pushtomainor PRs. Fetches recent traces (last 1h), deploys shadow, replays, verifies..github/workflows/slo-verify.yaml:
name: SLO Verification Pipeline on: [push, pull_request] jobs: verify: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Fetch Traces run: | python fetch_traces.py --service=api --hours=1 --output=traces.har - name: Deploy Shadow uses: argo-cd/argo-cd-action@v2 with: { command: 'app sync my-service-shadow' } - name: Replay Traces run: kubectl apply -f replay-job.yaml && kubectl wait --for=condition=complete job/slo-replay-${{ github.sha }} - name: Verify SLOs run: ./slo-verifier --prom-url=http://shadow-prom:9090 --baseline=baseline.json continue-on-error: false # Block on failure - name: Update Baseline if: github.ref == 'refs/heads/main' run: curl -X POST baseline-api/update --data baseline.jsonBaseline Storage: Store prod SLOs in S3/Redis; compare shadow vs. baseline.
Monitoring & Dashboard
- Grafana: Replay success rate, SLO burn rate, trace coverage.
- Alerts: Slack/ PagerDuty on SLO regression.
- Dashboard Panel:
Metric Prod Baseline Shadow Pass? p99 Latency 450ms 480ms ✅ Error Rate 0.05% 0.08% ✅
Challenges & Mitigations
Challenge Mitigation Stateful Services/DB Replay idempotent traces; mock DB with WireMock/Testcontainers. Auth/Secrets Strip/impersonate headers; use shadow auth (e.g., JWT signer). PII/Security Sanitize bodies (hash/anonymize); replay only sampled non-sensitive traffic. Scaling Throttle replay RPS; use k6 for load-testing variant. Cost Spot instances for shadow; trace sampling (1-10%). Flakiness Run 3x replays; use Canary analysis in Argo Rollouts. Deployment Timeline
- Week 1: OTel + Storage.
- Week 2: Replay + Shadow K8s.
- Week 3: CI/CD + SLO gates.
- Ongoing: Monitor, iterate.
This pipeline catches regressions pre-prod. Fork/customize for your stack (e.g., Terraform for infra). Questions on specifics? Provide your tech stack!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.