Implement observability: logging patterns, metrics, tracing, red/USE/SRE golden signals,…
- User
Implement observability: logging patterns, metrics, tracing, red/USE/SRE golden signals, instrumentation, sampling, dashboards, and alerting; include examples.
- Assistant
Implementing Observability in a Modern Microservice
As Sherlock, I'll deduce the optimal path to full observability for your system. Observability isn't just monitoring—it's the ability to understand internal states from external outputs: logs (what happened), metrics (how much/aggregated), traces (why it happened, causally), and events/profiles (changes/context). We'll implement this using OpenTelemetry (OTel) for instrumentation (unified across languages), Prometheus for metrics, Jaeger or Tempo for traces, Loki or ELK for logs, Grafana for dashboards, and Alertmanager for alerting.
Target stack: A Node.js microservice (e.g., Express API) for examples. Deployable on Kubernetes. Use Docker Compose for local setup.
1. Instrumentation (Foundation)
Instrument your code with OTel SDKs. Auto-instrument common libs (e.g., HTTP, DB), manual for business logic.
Install (Node.js):
npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/exporter-trace-otlp-http @opentelemetry/exporter-metrics-otlp-http @opentelemetry/instrumentation-express @opentelemetry/instrumentation-httpBasic Instrumentation (index.js):
const opentelemetry = require('@opentelemetry/sdk-node'); const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node'); const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http'); const { PeriodicRecordingSapling } = require('@opentelemetry/sdk-metrics'); const sdk = new opentelemetry.NodeSDK({ traceExporter: new OTLPTraceExporter({ url: 'http://jaeger:4318/v1/traces' }), // Or Tempo/collector metricReader: new PeriodicRecordingSapling({ // Prometheus exporter via OTel Collector exporter: new OTLPTraceExporter({ url: 'http://otel-collector:4318/v1/metrics' }) }), instrumentations: [getNodeAutoInstrumentations()] // Auto: Express, HTTP, etc. }); sdk.start(); // Manual span example const api = require('express')(); api.get('/users/:id', (req, res) => { const tracer = opentelemetry.trace.getTracer('user-service'); const span = tracer.startSpan('get-user'); span.setAttribute('user.id', req.params.id); // Business logic... span.end(); res.json({ id: req.params.id }); });Export to OTel Collector (YAML config snippet):
receivers: otlp: protocols: grpc: {} http: {} exporters: prometheus: # Metrics jaeger: # Traces loki: # Logs service: pipelines: traces: [otlp, jaeger] metrics: [otlp, prometheus] logs: [otlp, loki]2. Logging Patterns
Structured logging with context (traceID, spanID, requestID). Levels: DEBUG/INFO/WARN/ERROR/FATAL. Patterns:
- Request-scoped: Log per request.
- Error enrichment: Stack traces, metadata.
- Sampling: Log 100% errors, 1% INFO.
Example (winston + OTel context):
npm install winston @opentelemetry/contextconst winston = require('winston'); const { context, trace } = require('@opentelemetry/api'); const logger = winston.createLogger({ format: winston.format.combine( winston.format((info) => { const ctx = context.active(); const span = trace.getSpan(ctx); info.trace_id = span.spanContext().traceId; info.span_id = span.spanContext().spanId; return info; })(), winston.format.json() ), transports: [new winston.transports.Console()] }); // Usage logger.info('User fetched', { userId: 123, method: 'GET' }); logger.error('DB error', { error: err.message, stack: err.stack });Push to Loki via OTel Collector or Promtail.
3. Metrics & Golden Signals
SRE Golden Signals: Latency, Traffic, Errors, Saturation.
- RED (app-facing): Rate (req/s), Errors (%), Duration (p99 latency).
- USE (resource-facing): Utilization (%), Saturation (queue depth), Errors.
Instrument with OTel Metrics:
const { Counter, Histogram, UpDownCounter } = require('@opentelemetry/api-metrics'); const meter = opentelemetry.metrics.getMeter('user-service'); const requestCounter = meter.createCounter('http_requests_total', { description: 'RED: Rate' }); const latencyHistogram = meter.createHistogram('http_request_duration_ms', { description: 'RED: Duration' }); const errorCounter = meter.createCounter('http_errors_total', { description: 'RED: Errors' }); const cpuUtil = meter.createUpDownCounter('system_cpu_utilization_percent', { description: 'USE: Utilization' }); const queueSaturation = meter.createGauge('db_queue_length', { description: 'USE: Saturation' }); // In middleware app.use((req, res, next) => { const end = latencyHistogram.startObservation(); res.on('finish', () => { requestCounter.add(1, { method: req.method, status: res.statusCode }); latencyHistogram.record(end.end(), { method: req.method }); if (res.statusCode >= 500) errorCounter.add(1); }); next(); });Prometheus scrape config (prometheus.yml):
scrape_configs: - job_name: 'node-app' static_configs: [targets: ['otel-collector:8889']] # OTel -> Prometheus4. Tracing
Captures causal chains across services. Sampling: Head (always sample entry), Tail (sample high-error), Rate (1%).
OTel Sampling (collector config):
processors: tail_sampling: policies: - name: errors type: tracestate tracestate: - { key: "error", value: "true" } - name: rate type: probabilistic sampling_percentage: 5 # 5%Jaeger/Tempo queries traces by traceID from logs/metrics.
5. Sampling Strategies
- Traces: Probabilistic (5-10% prod), AlwaysOn dev, Tail (errors 100%).
- Logs: 100% ERROR/WARN, 1% INFO (Loki sampler).
- Metrics: Always aggregate.
Example Loki log query (Grafana):
{job="user-service"} | json | level="ERROR" |~ "trace_id:.*"6. Dashboards (Grafana)
Import dashboards:
- SRE Golden Signals:
Panel 1: Latency - Histogram: rate(http_request_duration_ms_bucket[5m]) p99 Panel 2: Traffic - Counter: rate(http_requests_total[5m]) Panel 3: Errors - Rate: rate(http_errors_total[5m]) / rate(http_requests_total[5m]) * 100 Panel 4: Saturation - Gauge: db_queue_length > 100 - USE: CPU/Mem/Disk panels from node_exporter.
- Trace View: Integrate Jaeger datasource, query by service.
Grafana JSON snippet for RED dashboard:
{ "targets": [{ "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_ms_bucket[5m])) by (le))", "legendFormat": "p99 Latency" }], "title": "RED: Duration" }7. Alerting (Prometheus Alertmanager)
SLO-based: Target 99.9% error budget.
prometheus.rules.yml:
groups: - name: golden_signals rules: - alert: HighErrorRate expr: rate(http_errors_total[5m]) / rate(http_requests_total[5m]) > 0.01 # >1% errors for: 5m labels: { severity: critical } annotations: { summary: 'RED: High error rate {{ $labels.job }}' } - alert: HighLatency expr: histogram_quantile(0.99, rate(http_request_duration_ms_bucket[5m])) > 500 # >500ms p99 for: 2m labels: { severity: warning } - alert: HighSaturation expr: db_queue_length > 50 # USE for: 1m labels: { severity: critical }Alertmanager config (YAML):
route: group_by: ['alertname'] receiver: 'slack' # Or PagerDuty receivers: - name: slack slack_configs: - channel: '#alerts' text: '{{ .CommonAnnotations.summary }}'Deployment & Best Practices
Docker Compose (quickstart):
services: jaeger: image: jaegertracing/all-in-one prometheus: ... # With configs grafana: ... # Datasource: Prometheus/Loki/Jaeger otel-collector: ... # Bridge app: build: . ports: ["3000:3000"]Prod Tips:
- SLOs: Define 99.9% on golden signals.
- Retention: Metrics 15d, Traces 7d, Logs 30d.
- Cost: Sample aggressively, use columnar stores (Loki/Mimir).
- Chaos: Test with Gremlin for resilience.
This setup gives actionable observability. Query a traceID across logs/metrics/traces to debug. Need Kubernetes/Go/Python variants? Deduce further! 🚀
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.