Skip to content
TheChatArchiveThe Chat Archive
Grok 4.1 Fast· xaiHugging Face Frontier Reasoning Traces

Design a production readiness checklist per service: SLOs, runbooks, dashboards, and…

  1. User

    Design a production readiness checklist per service: SLOs, runbooks, dashboards, and alerts.

  2. Assistant

    Production Readiness Checklist Template (Per Service)

    This checklist is designed to ensure each microservice or component is production-ready before deployment. Apply it per service (e.g., user-service, payment-service). Use tools like Prometheus/Grafana, Datadog, PagerDuty, or Opsgenie for monitoring/alerting. Mark items as complete with evidence (e.g., links to dashboards, docs).

    Track progress in a shared doc (e.g., Google Sheet, Notion, GitHub Issue) with service owners assigned.

    1. SLOs (Service Level Objectives)

    SLOs define measurable targets for reliability. Aim for SLI (Service Level Indicators) like availability, latency, throughput, and freshness.

    • SLIs defined: Core metrics documented (e.g., HTTP 5xx error rate <0.1%, p95 latency <200ms, throughput >95% of peak).
    • SLO targets set: Realistic targets based on user impact (e.g., 99.9% availability over 28 days).
    • Error budget calculated: Budget = (1 - SLO) * time period (e.g., 43 minutes/month for 99.9%). Policy for budget exhaustion defined (e.g., halt features).
    • SLI measurement implemented: Queries/metrics exported (e.g., Prometheus queries like histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))).
    • SLO compliance tracked: Automated burn rate alerts (e.g., >100% burn triggers page).
    • Historical data reviewed: Backtested SLOs against prod-like traffic for at least 2 weeks.
    • SLO doc published: In service README or wiki, with ownership and review cadence (quarterly).

    Evidence required: SLO doc link, metric queries.

    2. Runbooks

    Detailed, actionable procedures for incidents. Store in versioned repo (e.g., GitHub Wiki, Blameless repo) with searchability.

    • On-call rotation defined: Schedule in PagerDuty/ops tool, with 24/7 coverage and handoff procedures.
    • Incident command structure: Roles defined (Incident Commander, Communications Lead) per SRE best practices.
    • Core runbooks written:
      • High error rate / SLO breach.
      • High latency / timeouts.
      • High CPU/memory/disk usage.
      • Deployment rollback.
      • Dependency failure (e.g., DB, cache outage).
      • Traffic spike / scaling failure.
    • Runbooks tested: Tabletop exercises or chaos engineering (e.g., simulated failures via Gremlin).
    • Mitigation & rollback steps: Every runbook includes "blast radius" assessment, quick fixes, and full recovery.
    • Postmortem template: Linked in runbooks for after-action reviews (what happened, root cause, timeline, actions).
    • Accessibility: Publicly readable, with alerts linking directly to runbooks.

    Evidence required: Runbook repo links, test logs.

    3. Dashboards

    Visual, real-time monitoring. Use Grafana, Datadog, or Kibana. Ensure dashboards are self-service and mobile-friendly.

    • Service overview dashboard:
      • Key SLOs/SLIs (burn rates, compliance graphs).
      • Request volume, latency (p50/p95/p99), error rates (4xx/5xx).
      • Resource usage (CPU, mem, disk I/O per pod/node).
      • Traffic sources (e.g., ingress logs).
    • SLO-specific dashboard: Error budget trends, burn-up/down rates over 7/28/90 days.
    • Dependency dashboards: Traces for upstream/downstream services (e.g., via Jaeger/OpenTelemetry).
    • Logs & traces integrated: Top errors, slow queries, with filters by service/version.
    • Annotations for context: Deployment markers, incidents auto-annotated.
    • Shared & access-controlled: Team bookmarks, public for stakeholders.
    • Tested & refreshed: Verified in staging/prod, auto-refreshes every 1-5m.

    Evidence required: Dashboard URLs, screenshots.

    4. Alerts

    Reliable, low-noise notifications. Follow "alert on symptoms, not causes." Integrate with PagerDuty/Slack/Teams.

    • Critical alerts (page on-call):
      • SLO breach (e.g., error rate >1% for 5m).
      • High burn rate (>10x for 5m).
      • Resource saturation (CPU >90% for 5m).
      • Deadman's switch (no traffic for 5m).
    • Warning alerts (Slack/ticket):
      • SLO approaching breach (e.g., 80% error budget used).
      • Anomalies (e.g., ML-based deviation detection).
    • Alert grouping & routing: By service/team, with labels (severity: critical/warning/info).
    • Runbook links embedded: Every alert message includes direct runbook URL.
    • Noisiness mitigated:
      • Min duration/group-by (e.g., 5m window).
      • Silencing/stale alert policies.
      • Alert fatigue review (MTTR <15m, <5 false positives/week).
    • Escalation & ACK policies: Auto-escalate after 5m un-ACKed.
    • Tested end-to-end: Fire test alerts, confirm paging.

    Evidence required: Alert rule configs (YAML/JSON), PagerDuty schedule link.

    Overall Service Readiness Gates

    • All items above 100% complete for the service.
    • Peer review: SRE/eng lead sign-off.
    • Prod simulation: Load test to 2x peak traffic; verify SLOs hold.
    • Ownership: Service owner named, with quarterly review commitment.
    Service NameSLOs CompleteRunbooks CompleteDashboards CompleteAlerts CompleteGo/No-GoDateOwner
    (e.g., user-service)✅✅✅✅GOYYYY-MM-DD@eng-lead

    Usage Tips:

    • Automate where possible (e.g., Terraform for alerts/dashboards, GitHub Actions for checklist validation).
    • Scale for monoliths by grouping logical services.
    • Review post-incident: Update checklist based on lessons learned.

    This template aligns with Google SRE practices. Ping me for customizations! 🕵️‍♂️

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.