Create a service ownership model: SLOs, error budgets, runbooks, and escalation ladders.
- User
Create a service ownership model: SLOs, error budgets, runbooks, and escalation ladders.
- Assistant
Service Ownership Model: User Authentication Service
Service Owner Team: Auth Squad (SRE + Engineering)
Service Description: A microservice handling user login, token issuance, and session management for the Trailblazer platform. Deployed on Kubernetes in AWS us-east-1, with Redis for caching and PostgreSQL for persistence. Critical to all user-facing features.This model follows SRE best practices, ensuring reliability through measurable SLOs, error budgets to balance innovation and stability, automated/manual runbooks for incident response, and clear escalation paths.
1. Service Level Objectives (SLOs)
SLOs are reliability targets based on user-impacting metrics. We target SLA: 99.9% availability (customer-facing promise), with SLOs set conservatively inside that.
Objective Target Measurement Window Good Notable Measurement Tool Availability 99.95% Rolling 30 days ≥99.95% 99.5–99.949% Synthetics (e.g., Pingdom/CloudWatch) + user request success rate Latency (P99) ≤150ms Rolling 28 days ≤150ms 150–300ms APM (Datadog/New Relic) on valid requests Error Rate ≤0.1% Rolling 7 days ≤0.1% 0.1–1% Log-based (ELK stack) 5xx errors + auth failures - SLI Definitions:
- Availability SLI:
(successful requests) / (total valid requests). - Excludes: Throttled requests, maintenance windows (<4h/mo), force majeure.
- Availability SLI:
2. Error Budgets
Error budgets represent allowable "bad" states before halting feature development. Calculated as
100% - SLO target.SLO Monthly Budget (30 days) Weekly Budget (7 days) Action if Exhausted Availability (99.95%) ~21.6 minutes downtime ~5 minutes Freeze non-critical releases; prioritize reliability work Latency P99 ~4.3 hours over budget ~1 hour over budget Alert → Capacity analysis Error Rate (0.1%) ~10.08 minutes of errors ~2.4 minutes of errors Revert deploys; deep-dive postmortem - Budget Policy: If budget <20% remaining in window, no production changes except reliability fixes. Tracked in dashboard (e.g., Google Sheets + Grafana).
- Reset: Rolling windows; review quarterly.
3. Runbooks
Runbooks are step-by-step procedures for common incidents, stored in GitOps repo (e.g., GitHub Wiki + Terraform). Automated where possible (PagerDuty integrations). Tested quarterly.
Runbook 1: Service Outage (High Error Rate / Unavailability)
- Triage (0-2 min): Confirm via dashboard (Datadog). Check PagerDuty for alerts.
- Immediate Mitigation (2-5 min):
- Scale HPA (Horizontal Pod Autoscaler) to max replicas.
- Failover to secondary region (Route53 weighted routing).
- Root Cause (5-15 min):
kubectl logs, Redis evictions? DB connections? - Remediation:
Issue Action Pod crashloop kubectl rollout restart deploy/auth-svcDB overload Scale RDS read replicas; query optimizer Redis OOM Increase memory limit; evict cold keys - Verify & Close: SLO back to green? Postmortem in 24h.
ETA to Recovery: <15 min (MTTR target).
Runbook 2: High Latency
- Triage: Check P95/P99 traces in Datadog.
- Mitigate:
- Warm cache:
/health/cache-refresh. - Offload to CDN for static tokens.
- Warm cache:
- Deep Dive: Flame graphs → Slow SQL? Network hops?
- Long-term: Capacity planning script triggered.
Runbook 3: Capacity Exhaustion
- Alert on 80% CPU/Mem.
- Auto-scale + notify if manual intervention needed.
Full runbooks linked: GitHub Repo
4. Escalation Ladders
Escalation based on severity (PagerDuty levels) and time-to-resolve. 24/7 on-call rotation (Auth Squad).
Severity Description Initial Responder Escalate After Escalate To Contact SEV-1 (Critical) >50% outage, SLA breach Primary On-Call (5 min rotation) 15 min Secondary On-Call + Tech Lead PagerDuty + Slack #auth-incidents SEV-2 (High) Degraded perf, SLO warning Primary On-Call 30 min Engineering Manager Slack DM + phone SEV-3 (Medium) Isolated issues, monitoring alerts Primary On-Call 1 hour Squad Lead Slack channel SEV-4 (Low) Non-urgent (e.g., logs full) Self-service / Ticket N/A N/A Jira board - Incident Command Structure:
- Incident Commander (IC): Primary On-Call.
- Communications Lead: Posts updates to Slack/Statuspage every 15 min.
- After-hours Exec: CTO (post-10pm/ pre-6am).
- Postmortem: Always, blameless, within 48h. Action items owned.
This model is live and iterable—feedback welcome to refine! Questions, Holmes? 🕵️♂️
- SLI Definitions:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.