Design a multi-tenant observability platform: log/metric/trace ingestion, tenant…
- User
Design a multi-tenant observability platform: log/metric/trace ingestion, tenant isolation, retention tiers, self-service dashboards, budget controls, and compliance auditing.
Assistant
The user wants a design for a multi-tenant observability platform. The request is comprehensive, covering:
- Core Telemetry: Logs, Metrics, Traces ingestion.
- Multi-tenancy: Tenant isolation.
- Data Lifecycle: Retention tiers.
- User Interface: Self-service dashboards.
- Cost Management: Budget controls.
- Security/Governance: Compliance auditing.
This looks like a system design interview question or a high-level architectural blueprint request. I need to structure the response logically, covering the architecture, data flow, and specific mechanisms for each requirement.
Deconstruction & Brainstorming:
-
High-Level Architecture:
- Ingestion Layer (Gateway/Collectors).
- Routing/Processing Layer (Kafka/SQS).
- Storage Layer (Hot/Warm/Cold - ClickHouse, S3, Prometheus/Thanos, Elastic/Opensearch).
- Query Layer (Query Frontend).
- Control Plane (Metadata, RBAC, Billing).
- UI Layer (Grafana-based or custom).
-
Ingestion:
- Protocol support: OTLP (OpenTelemetry) is the standard. Also PromQL remote_write, Elastic/Logstash formats.
- High scalability: Load balancers -> Ingestion Agents (Stateless).
- Validation: Authentication (API Keys/Tokens), Quota checks (soft/hard limits).
-
Tenant Isolation:
- Logical vs. Physical.
- Physical is too expensive for high density.
- Logical is standard: Tenant ID in headers/metadata.
- Data segregation at storage: separate databases or column-based filtering (e.g.,
tenant_idcolumn in ClickHouse). Correction: For strict compliance, maybe separate buckets in object storage, but usually shared tables with strict query enforcement is efficient. Let's aim for "Logical Isolation with Hard Enforcement."
-
Retention Tiers:
- Hot: SSD/NVMe (Fast queries, high cost). e.g., ClickHouse or VictoriaMetrics. Retention: 7 days.
- Warm: HDD/Object Storage (Slower queries, lower cost). e.g., S3 with Index. Retention: 30 days.
- Cold: Object Storage (Archive, restore to query). Retention: 1 year.
- Downsampling/Aggregation: Rollups for metrics to save space.
-
Self-Service Dashboards:
- Don't reinvent the wheel. Embed or fork Grafana, or build a custom React UI on top of a unified query API.
- Let's assume a custom UI or a multi-tenant Grafana setup.
- Features: Drag-and-drop, Alerts, Annotations.
-
Budget Controls:
- This is crucial. Observability costs explode.
- Mechanism: Pre-ingestion volume checks, Post-ingestion usage tracking (Daily/Weekly).
- "Scrubbing/Filtering": Drop high-volume debug logs automatically if over budget.
- Cardinality management for metrics.
-
Compliance Auditing:
- Audit Logs: Who accessed what data? Who changed the retention policy?
- Data Residency: EU vs. US data centers (Region locking).
- RBAC (Role Based Access Control).
Structuring the Design:
- Introduction: Goals and Scope.
- System Architecture Diagram (Mental Model): The flow of data.
- Core Components:
- Ingestion Pipeline (The Entry).
- Storage Strategy (The Data Lake/Warehouse).
- Query Engine (The Retrieval).
- Multi-Tenancy & Isolation: Deep dive into how to separate customers.
- Data Lifecycle (Retention): Hot/Warm/Cold + Downsampling.
- User Experience (Dashboards): Customization and alerts.
- Cost Management (Budgeting): Quotas and billing.
- Security & Compliance: RBAC, Auditing, PII handling.
Refining Specific Tech Choices (for a concrete example):
- Ingestion: OpenTelemetry Collector + Kafka.
- Metrics: Prometheus + Thanos/Cortex or VictoriaMetrics.
- Logs/Traces: ClickHouse (very fast for logs/traces) or Elasticsearch/OpenSearch (classic but heavy). ClickHouse is trendier for cost-efficiency. Let's suggest a hybrid: ClickHouse for logs/traces, VictoriaMetrics for metrics.
- Storage: S3/GCS for cold storage.
Drafting the Content - Section by Section:
- Ingestion: Needs a "Traffic Cop" (Rate Limiter). Needs to tag data with
tenant_idimmediately if not present. - Isolation: Shared cluster approach is most cost-effective.
- Strictness: Use separate namespaces in K8s? No, usually just logical separation in the DB is enough if the query layer enforces the
tenant_id. Key Risk: Noisy neighbor. Need QoS (Quality of Service) queues.
- Strictness: Use separate namespaces in K8s? No, usually just logical separation in the DB is enough if the query layer enforces the
- Retention: Need a reaper/cleaner process. For object storage, lifecycle policies.
- Budget: Usage-based billing model. "Soft limits" (alert) vs "Hard limits" (drop data).
- Compliance: PII masking at ingestion (scrubbers). Audit trails stored in an immutable ledger (or just a locked down DB table).
Self-Correction/Review:
- Is "Design" asking for code? No, system architecture.
- Is it cloud-agnostic? Yes, assume generic cloud components (LB, Queue, Object Store).
- Focus: The prompt asks for specific features. I must ensure "Budget Controls" and "Compliance" get dedicated sections, not just footnotes.
Let's structure the final output with clear headings and architectural patterns.
- Pattern: Gateway Pattern (for ingestion), Sidecar/Agent Pattern (for collection), Lambda/Kappa (for processing).
- The "Control Plane": This is the brain. It holds Tenant config, Quotas, Billing info.
Final Polish: Ensure the distinction between "Logical Isolation" (Shared DB) and "Physical Isolation" (Dedicated DB) is clear, as this drives cost vs. compliance tradeoffs.
(Self-Correction during drafting the "Storage" section): Don't just say "Database." Explain how the data is organized. e.g., Partitioning by day + tenant_id.
(Self-Correction on "Budget"): How do you control costs in real-time? Token bucket algorithms at the ingestion gateway.
Okay, ready to write the response. Designing a multi-tenant observability platform requires balancing cost efficiency (shared infrastructure) with strict isolation (security and performance). Below is a comprehensive architectural design for a system capable of handling high-scale logs, metrics, and traces (The "Three Pillars").
1. High-Level Architecture
We will adopt a Shared-Everything architecture with logical isolation, using a Control Plane (manages configuration/tenants) and a Data Plane (processes data).
Core Components
- Ingestion Gateway: Stateless edge layer handling auth, routing, and rate limiting.
- Stream Processing: Buffering and ETL (Kafka/Pulsar).
- Storage Engines: Purpose-built databases for Metrics, Logs, and Traces.
- Query Engine: Distributed query layer enforcing tenant context.
- Control Plane: Manages tenant configs, RBAC, and billing metadata.
2. Ingestion & Routing (The Data Plane)
To support heterogenous clients, the ingestion layer must support standard protocols (OTLP, Prometheus remote_write, Syslog, Fluentd).
The "Smart Gateway" Pattern:
- Protocol Translation: Accept data via OTLP (gRPC) or HTTP. Convert legacy formats (e.g., StatsD) to internal standard.
- Tenant Resolution:
- Ingestion Key (API Key) Lookup
Tenant_ID. - Header Injection: The Gateway injects a
X-Tenant-IDinto the payload metadata. This header travels with the data throughout the entire pipeline.
- Ingestion Key (API Key) Lookup
- Cardinality/Volume Protection:
- Metrics: Cardinality estimator. If a tenant sends >100k unique time series in a minute, reject or aggregate immediately to protect the storage engine.
- Logs/Traces: Sampling strategies (Head/Tail based sampling).
3. Storage Strategy & Retention Tiers
We use a Tiered Storage approach to balance cost and query speed. Data moves from Hot (Fast/Expensive) to Cold (Slow/Cheap) based on retention policies.
A. Metrics (Time-Series DB)
- Engine: VictoriaMetrics or Cortex (Supports multi-tenancy natively via
tenant_idlabel). - Storage:
- Hot (0-7 days): NVMe SSDs. High resolution.
- Warm (8-30 days): HDDs or High-IOPS Cloud Block Storage.
- Downsampling: Aggregating raw data into 5m/1h averages for long-term trend analysis.
B. Logs & Traces (Columnar / Index-free)
- Engine: ClickHouse or Grafana Loki.
- Architecture: Hash-Based Sharding (Partition by
Tenant_ID+Timestamp). - Storage Tiers:
- Hot (0-3 days): SSDs. Highly compressed row storage.
- Cold (3-90 days): Object Storage (S3/GCS). The query engine fetches data from S3 transparently if it's not in the hot cache.
- Archive (90+ days): Compressed parquet files in Glacier (for compliance restore only).
4. Tenant Isolation (The "Noisy Neighbor" Defense)
Isolation is implemented at three layers: Ingestion, Storage, and Query.
-
Ingestion Isolation (Rate Limiting):
- Use a Token Bucket algorithm per tenant.
- Hard Limit: If exceeded, return
429 Too Many Requests. - Soft Limit: If exceeded, drop low-priority logs (debug/info) but keep errors.
-
Storage Isolation (Sharding):
- Do not store Tenant A and Tenant B on the same physical shard if they are massive (Enterprise tier). Use Shard Mapping:
Tenant_IDShard_Group_ID. - Small tenants share shards; large tenants get dedicated shards.
- Do not store Tenant A and Tenant B on the same physical shard if they are massive (Enterprise tier). Use Shard Mapping:
-
Query Isolation (Resource Quotas):
- The Query Engine must enforce timeouts.
- If a Tenant A query takes >30s, kill it.
- Cache Isolation: Query results are cached in Redis keyed by
Tenant_ID. Strict ACLs ensure Tenant A cannot access Tenant B's cache keys.
5. Budget Controls & Cost Attribution
Observability bills can explode. The platform must offer self-service cost controls.
- Usage Metering:
- Metrics: Active Series (Cardinality).
- Logs/Traces: Ingested Volume (GB) and Scanned Data (GB) during queries.
- Budgets & Alerts:
- Tenants set a monthly budget (e.g., $5,000).
- The platform exposes a "Current Spend" API.
- Automated Actions: If a tenant hits 90% of their budget, automatically switch
Log LevelfromDebugtoErroror drop trace sampling to 1%.
- Attribution: Tag resources (e.g.,
service:payment,env:prod) to allow tenants to see which team is burning the budget.
6. Self-Service Dashboards & Alerting
Avoid giving users direct database access. Provide a curated UI (e.g., Grafana or a custom React frontend).
- Dashboard-as-Code: Store dashboard JSON in Git (GitOps workflow) or the Control Plane DB.
- Dynamic Provisioning: When a user logs in, the UI fetches their specific dashboard configurations.
- Alerting:
- Multi-tenant Alertmanager.
- Tenants configure their own Slack/PagerDuty webhooks.
- Isolation: Ensure routing rules strictly separate tenant alert streams.
7. Compliance Auditing & Security
This is critical for Enterprise/B2B SaaS.
- RBAC (Role Based Access Control):
- Granular permissions:
reader(view logs),editor(modify dashboards),admin(manage API keys). - Scoped to teams/projects within a tenant.
- Granular permissions:
- Audit Logs:
- An immutable log of who accessed what data and when.
- Stored in a separate, secured storage bucket (WORM - Write Once Read Many).
- Retention: 7 years (financial compliance).
- Data Residency (Geo-fencing):
- For GDPR/Sovereign Cloud: The Ingestion Gateway must route data for
EU_TenantstoEU_Region_Clusterand strictly forbid replication toUS_Region.
- For GDPR/Sovereign Cloud: The Ingestion Gateway must route data for
- PII Masking:
- Offer "Scrubbers" at ingestion. Regex patterns to redact Credit Cards / Emails before data hits the disk.
Summary Workflow
- Agent sends OTLP data to Gateway with API Key.
- Gateway validates key, resolves
Tenant_ID, checks Rate Limits. - Data enters Kafka (partitioned by tenant).
- Writer Service consumes and writes to Hot Storage (ClickHouse/VictoriaMetrics) partitioned by tenant.
- RetentionPolicy Job moves old data to S3 (Cold Storage).
- User logs into UI, runs query.
- Query Service injects
WHERE tenant_id=Xautomatically to ensure isolation, fetches data, and returns to UI. - Billing Service updates usage counters for the tenant.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.