Create a SRE on-call readiness checklist: runbooks, dashboards, and drills.
- User
Create a SRE on-call readiness checklist: runbooks, dashboards, and drills.
- Assistant
SRE On-Call Readiness Checklist
This checklist ensures Site Reliability Engineers (SREs) are fully prepared for on-call rotations. It focuses on runbooks (incident response procedures), dashboards (monitoring and observability), and drills (practice and simulation). Review and complete this checklist weekly or before starting an on-call shift. Mark items as ✅ when verified.
1. Runbooks (Incident Response Playbooks)
Runbooks provide scripted actions for common incidents. Ensure you're proficient and they are up-to-date.
- Access Verification: Confirm login access to runbook repository (e.g., GitHub, Confluence, or Runbook tool like FireHydrant/PagerDuty).
- Core Runbooks Reviewed:
Runbook Category Verified? Last Updated Notes High-priority alerts (e.g., CPU >90%, latency spikes) ☐ Service outages (e.g., DB failover, API downtime) ☐ Capacity exhaustion (e.g., disk full, queue backlogs) ☐ Security incidents (e.g., DDoS, unauthorized access) ☐ Third-party failures (e.g., AWS outage, CDN issues) ☐ - Personal Familiarity: Walk through 3-5 critical runbooks end-to-end (time yourself <10 min each).
- Updates & Gaps: Flag any outdated steps, missing runbooks, or improvements in a shared doc/ticket.
- Handover Review: Read recent incident post-mortems and runbook usage notes from last shift.
2. Dashboards (Monitoring & Observability)
Dashboards must be bookmarked, customized, and alerting correctly for rapid triage.
- Access & Permissions:
- Grafana/Prometheus/Datadog: ✅ Login tested.
- PagerDuty/Slack/Incident tool: ✅ Notifications enabled.
- Read-only access to prod environments (e.g., Kubernetes, AWS Console).
- Key Dashboards Bookmarked & Functional:
Dashboard URL/Pin Verified? SLO Metrics Visible? Alerts Active? Global Overview (error rates, latency, traffic) ☐ ☐ ☐ Service-Specific (e.g., API, DB, Cache) ☐ ☐ ☐ Infrastructure (CPU/Mem/Disk, Nodes/Pods) ☐ ☐ ☐ Business SLOs (uptime, throughput) ☐ ☐ ☐ Logs & Traces (ELK/Loki/Tempo) ☐ ☐ ☐ - Customization Check: Filters set for your services; mobile view tested.
- Alert Fatigue Audit: Review last 24h alerts; mute/suppress false positives.
- Redundancy Test: Verify fallback dashboards (e.g., if Grafana down, use Kibana).
3. Drills (Practice & Simulation)
Drills build muscle memory. Participate regularly to reduce MTTR (Mean Time to Recovery).
- Recent Participation:
- Last chaos engineering drill (e.g., kill pod, inject latency): Date: ________ ✅
- Tabletop exercise (hypothetical outage): Date: ________ ✅
- On-call handover simulation: Date: ________ ✅
- Scheduled Drills:
Drill Type Frequency Next Scheduled Status Alert Response (PagerDuty fire-drill) Weekly ☐ Failover Practice (e.g., DB switchover) Bi-weekly ☐ Full Incident Simulation (GameDay) Monthly ☐ Escalation Drill (eng → ops → exec) Quarterly ☐ - Personal Drill:
- Simulate 1-2 alerts from runbooks (use staging/prod-safe mode).
- Time to resolution: <15 min? ✅
- Feedback Loop: Review drill post-mortems; update runbooks/dashboards as needed.
- Escalation Contacts: Confirm list of SMEs (Subject Matter Experts) and off-hours numbers.
General On-Call Readiness
- Tools & Environment: Laptop charged; VPN stable; 2FA ready; mobile app notifications on.
- Shift Handover: Reviewed notes from previous engineer.
- Self-Assessment: Rate confidence (1-10): Runbooks __/10 | Dashboards __/10 | Drills __/10.
- Sign-Off: Checklist complete. Ready for on-call! 🟢
Pro Tips:
- Automate where possible (e.g., auto-runbook links in alerts).
- Track checklist completion in a shared sheet for team visibility.
- If gaps found, create a ticket in your incident management system.
Checklist version 1.0 | Last updated: [Insert Date]
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.