Create a deployment failure playbook: causes, mitigations, and drills.
- User
Create a deployment failure playbook: causes, mitigations, and drills.
- Assistant
Hmm, the user wants a deployment failure playbook covering causes, mitigations, and drills. This is a practical request likely from someone in DevOps or SRE who needs a structured reference for handling production incidents.
I should structure this clearly: first categorize common failure patterns with concrete examples, then provide actionable mitigations for each, and finally suggest recurring drills to build resilience. The tone must be professional yet accessible—avoiding jargon overload while ensuring technical accuracy.
For causes, I’ll group them by failure domain (infrastructure, application, process). Mitigations should map directly to causes, emphasizing automation and guardrails. Drills need to be realistic, scalable, and tied to specific failure modes. A summary table at the end might help for quick reference, but the detailed sections are the core.
The user didn’t specify their tech stack, so I’ll keep examples cloud-agnostic but mention common tools (K8s, Terraform) where relevant. I’ll also assume they value blameless culture—so the playbook should emphasize learning over finger-pointing.
Deployment Failure Playbook
A structured guide for diagnosing, mitigating, and preventing deployment incidents.
1. Purpose & Principles
Goal: Minimize downtime, accelerate recovery, and institutionalize learning.
Principles:- Blameless post-mortems → Focus on system flaws, not individuals.
- Speed vs. Safety → Balance rapid deployment with guardrails.
- Automate first → Manual steps are failure points.
- Document everything → Institutional memory is critical.
2. Common Failure Causes & Mitigations
A. Infrastructure/Platform Failures
Cause Symptoms Immediate Mitigation Long-Term Fix Resource exhaustion (CPU/memory/disk) Pods restarting, OOM kills, slow responses 1. Scale horizontally/vertically2. Redirect traffic (canary→stable)3. Clear disk space/logs - Set resource limits/requests- Autoscaling policies (HPA/VPA)- Monitoring alerts for thresholds Network partition/firewall rules Inter-service timeouts, 5xx errors 1. Validate network policies/security groups2. Roll back to previous version - Infrastructure-as-Code (IaC) validation- Canary testing with synthetic traffic Cloud provider outage Multi-service degradation, region-wide errors 1. Activate DR region (if available)2. Failover to backup systems3. Communicate status internally - Multi-region/active-active design- Chaos engineering to test DR Configuration drift (IaC/config management) Inconsistent environments, "works on staging" issues 1. Compare configs across envs (git diff)2. Re-apply immutable infrastructure - GitOps workflow (ArgoCD/Flux)- Immutable infrastructure principles
B. Application/Code Failures
Cause Symptoms Immediate Mitigation Long-Term Fix Bug in new code (logic error, missing null check) 500 errors, data corruption, feature breakage 1. Roll back deployment2. Hotfix critical path - Feature flags (launch darkly)- Incremental rollout (5% → 25% → 100%)- Contract testing (Pact) Database schema change (breaking migration) DB connection errors, corruption, failed transactions 1. Restore DB backup2. Roll back app + DB (if coupled) - Blue-green DB migrations- Backward-compatible schema changes- Shadow migrations Dependency outage (third-party API/library) Timeout errors, cascading failures 1. Circuit breaker opens automatically2. Serve cached/degraded response - Dependency resilience patterns (circuit breaker, timeout, retry)- Fallback mechanisms Memory leak / resource hog Gradual performance degradation, eventual crash 1. Restart affected services2. Scale out temporarily - Load testing with profiling- APM integration (New Relic/Datadog)
C. Deployment Process/Orchestration Failures
Cause Symptoms Immediate Mitigation Long-Term Fix Pipeline misconfiguration (e.g., wrong env vars) Deployment succeeds but app misbehaves 1. Audit pipeline logs2. Deploy with corrected config - Pipeline as code with PR reviews- Secret management (Vault/Sealed Secrets) Incorrect artifact (old image, wrong branch) Unexpected behavior, missing features 1. Verify image digest/tag2. Redeploy correct artifact - Immutable tags (SHA-based)- Deployment audit trail Rollback failure (rollback script broken) Unable to revert quickly 1. Manual override: deploy old manifest/image2. Use Git history to restore - Test rollbacks in staging- Keep previous 3 versions ready Coordinated multi-service failure Multiple services fail together 1. Deploy in waves (not all at once)2. Pause dependent deployments - Deployment dependency mapping- Orchestrated rollout (Spinnaker/Argo Rollouts)
3. Immediate Response Checklist (First 15 Minutes)
- Detect & Alert
- Confirm incident: dashboards, logs, synthetic checks, customer tickets.
- Alert on-call engineer (PagerDuty/Opsgenie).
- Assess Impact
- Scope: Single service vs. full outage?
- User impact: % affected, severity ( Sev1-4 ).
- Contain
- Rollback if code-related.
- Disable feature flag if flag caused issue.
- Rate-limit/throttle if overload.
- Failover if infra issue.
- Communicate
- Internal Slack/Teams channel (#incident-).
- Stakeholder update every 30 mins.
- Customer status page if >5% users affected.
4. Post-Incident Process
- Blameless Post-Mortem (within 48 hours)
- Timeline reconstruction (from first detection to resolution).
- Root cause: “Why did the process allow this?” (5 Whys).
- Contributing factors (tooling, alerting, documentation).
- Action Items
- Temporary (e.g., “add alert on X metric”).
- Long-term (e.g., “implement canary analysis with automated rollback”).
- Assign owners and due dates.
- Share Learnings
- Doc in shared wiki (runbook updates).
- 15-minute “failure share” in next sprint retro.
5. Drills & Resilience Testing
Run monthly or after major incidents.
Drill Type Scenario Success Criteria Tools Chaos Engineering Kill random pods, block network between services System auto-recovers within 5 mins, alerts trigger Chaos Mesh, Gremlin, Litmus Rollback Drill Force deployment of bad config (test env) Rollback completes in <2 mins, no data loss Argo Rollouts, Spinnaker Dependency Failure Block third-party API (e.g., payment) Degraded mode works, circuit breaker trips Toxiproxy, WireMock DB Migration Drill Run breaking migration in staging Migration reversible, app compatible Flyway, Liquibase Full Outage Simulation Shut down primary region DR region takes over, RTO <15 mins AWS Fault Injection Simulator, GCP Chaos
6. Prevention Checklist (Before Any Deployment)
- All tests passing (unit, integration, contract).
- Canary/blue-green rollout plan documented.
- Rollback procedure tested in staging.
- Alerts configured for new metrics (error rate, latency).
- Feature flag toggled off by default.
- Database changes backward-compatible.
- On-call engineer aware and scheduled.
7. Key Metrics to Track
- MTTR (Mean Time to Recover) → Target: <30 mins for Sev2.
- Deployment success rate → Target: >99%.
- Change failure rate → % of deployments causing incidents (target: <15%).
- Time to detect → From failure to alert (target: <5 mins).
8. Runbook Template (Per Service)
Copy this for each critical service:
## Service: <name> **Owner team**: @team-slack **On-call rotation**: PagerDuty link **Last update**: YYYY-MM-DD ### Common Failure Modes 1. **Symptom**: 5xx errors from API gateway **Likely cause**: Pod crash loop / DB connection pool exhausted **Check**: `kubectl logs -f <pod>`, `kubectl top pod`, DB connection metrics **Mitigate**: Restart pod, scale DB connection pool, roll back if recent deploy 2. **Symptom**: High latency (>2s) **Likely cause**: Memory leak, N+1 query, downstream dependency **Check**: APM flame graph, GC logs, dependency latency dashboards **Mitigate**: Restart, scale horizontally, disable non-critical feature flag ### Rollback Procedure ```bash # Example for ArgoCD argocd app rollback <app-name> --revision <prev-revision-number>Verify:
/healthendpoint returns 200, error rate <0.1% for 5 mins.Escalation
- L2: @senior-engineer-slack
- L3: @engineering-manager
- Infrastructure: @platform-team
--- ## 9. Culture & Continuous Improvement - **No blame, all systems**: Every failure is a process/tooling gap. - **Share failures widely**: Monthly “Failure Friday” talks. - **Reward resilience**: Recognize quick mitigations and proactive drills. - **Update playbooks**: After every incident, revise this document. --- **Remember**: A deployment failure is not a “fire to be put out” but a **feedback signal** from your system. Treat it as an opportunity to build robustness. *Last reviewed: [Date]* *Next drill scheduled: [Date]*
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.