Skip to content
TheChatArchiveThe Chat Archive
Grok 4.1 Fast· xaiHugging Face Frontier Reasoning Traces

Create a SRE on-call readiness checklist: runbooks, dashboards, and drills.

  1. User

    Create a SRE on-call readiness checklist: runbooks, dashboards, and drills.

  2. Assistant

    SRE On-Call Readiness Checklist

    This checklist ensures Site Reliability Engineers (SREs) are fully prepared for on-call rotations. It focuses on runbooks (incident response procedures), dashboards (monitoring and observability), and drills (practice and simulation). Review and complete this checklist weekly or before starting an on-call shift. Mark items as ✅ when verified.

    1. Runbooks (Incident Response Playbooks)

    Runbooks provide scripted actions for common incidents. Ensure you're proficient and they are up-to-date.

    • Access Verification: Confirm login access to runbook repository (e.g., GitHub, Confluence, or Runbook tool like FireHydrant/PagerDuty).
    • Core Runbooks Reviewed:
      Runbook CategoryVerified?Last UpdatedNotes
      High-priority alerts (e.g., CPU >90%, latency spikes)☐
      Service outages (e.g., DB failover, API downtime)☐
      Capacity exhaustion (e.g., disk full, queue backlogs)☐
      Security incidents (e.g., DDoS, unauthorized access)☐
      Third-party failures (e.g., AWS outage, CDN issues)☐
    • Personal Familiarity: Walk through 3-5 critical runbooks end-to-end (time yourself <10 min each).
    • Updates & Gaps: Flag any outdated steps, missing runbooks, or improvements in a shared doc/ticket.
    • Handover Review: Read recent incident post-mortems and runbook usage notes from last shift.

    2. Dashboards (Monitoring & Observability)

    Dashboards must be bookmarked, customized, and alerting correctly for rapid triage.

    • Access & Permissions:
      • Grafana/Prometheus/Datadog: ✅ Login tested.
      • PagerDuty/Slack/Incident tool: ✅ Notifications enabled.
      • Read-only access to prod environments (e.g., Kubernetes, AWS Console).
    • Key Dashboards Bookmarked & Functional:
      DashboardURL/Pin Verified?SLO Metrics Visible?Alerts Active?
      Global Overview (error rates, latency, traffic)☐☐☐
      Service-Specific (e.g., API, DB, Cache)☐☐☐
      Infrastructure (CPU/Mem/Disk, Nodes/Pods)☐☐☐
      Business SLOs (uptime, throughput)☐☐☐
      Logs & Traces (ELK/Loki/Tempo)☐☐☐
    • Customization Check: Filters set for your services; mobile view tested.
    • Alert Fatigue Audit: Review last 24h alerts; mute/suppress false positives.
    • Redundancy Test: Verify fallback dashboards (e.g., if Grafana down, use Kibana).

    3. Drills (Practice & Simulation)

    Drills build muscle memory. Participate regularly to reduce MTTR (Mean Time to Recovery).

    • Recent Participation:
      • Last chaos engineering drill (e.g., kill pod, inject latency): Date: ________ ✅
      • Tabletop exercise (hypothetical outage): Date: ________ ✅
      • On-call handover simulation: Date: ________ ✅
    • Scheduled Drills:
      Drill TypeFrequencyNext ScheduledStatus
      Alert Response (PagerDuty fire-drill)Weekly☐
      Failover Practice (e.g., DB switchover)Bi-weekly☐
      Full Incident Simulation (GameDay)Monthly☐
      Escalation Drill (eng → ops → exec)Quarterly☐
    • Personal Drill:
      • Simulate 1-2 alerts from runbooks (use staging/prod-safe mode).
      • Time to resolution: <15 min? ✅
    • Feedback Loop: Review drill post-mortems; update runbooks/dashboards as needed.
    • Escalation Contacts: Confirm list of SMEs (Subject Matter Experts) and off-hours numbers.

    General On-Call Readiness

    • Tools & Environment: Laptop charged; VPN stable; 2FA ready; mobile app notifications on.
    • Shift Handover: Reviewed notes from previous engineer.
    • Self-Assessment: Rate confidence (1-10): Runbooks __/10 | Dashboards __/10 | Drills __/10.
    • Sign-Off: Checklist complete. Ready for on-call! 🟢

    Pro Tips:

    • Automate where possible (e.g., auto-runbook links in alerts).
    • Track checklist completion in a shared sheet for team visibility.
    • If gaps found, create a ticket in your incident management system.

    Checklist version 1.0 | Last updated: [Insert Date]

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.