Draft a production config-as-data
- User
Draft a production config-as-data system: schemas, validation, rollout safety, and audit logging.
- Assistant
Okay, the user wants me to outline a production-ready config-as-data system focusing on schemas, validation, rollout safety, and audit logging. This is a complex but common infrastructure requirement, so I need to provide a structured yet actionable framework.
First, I should define what "config-as-data" means in practice - treating configuration as versioned, declarative data rather than code. The core components are schema definition, validation pipeline, rollout mechanics, and audit trail. I'll organize each as a separate section with concrete implementation details.
For schemas, JSON Schema is the industry standard but needs strict conventions for production. I should emphasize mandatory fields, versioning strategies, and schema evolution rules. Validation needs both static checks (type/format) and dynamic business logic - the latter often gets overlooked.
Rollout safety is critical: canary validation, heartbeat monitoring, and automatic rollback triggers must be explicit. Audit logging isn't just about recording changes - it needs to capture intent (tickets/PRs) and system responses. I should stress that logs must be tamper-evident and queryable.
The user might be an SRE or platform engineer building internal tooling. They likely need this for compliance or reliability reasons, so I should highlight operational guardrails (approval flows, RBAC) and disaster recovery (immutable storage). Avoid over-engineering but don't skip hardened details like schema registry backups or log integrity checks.
Finally, I'll summarize with a checklist-style recap since operational systems thrive on clear requirements. The tone should be authoritative but not academic - this is for builders who need to implement, not just theorize. Excellent. Here is a comprehensive draft for a production-grade Config-as-Data (CaD) System. This design treats configuration as a first-class, versioned, and validated data asset, separating intent (the config) from execution (the agent applying it).
1. Core Philosophy & Goals
- Declarative & Versioned: All configuration is stored as data (e.g., JSON/YAML) in a central, version-controlled repository (Git).
- Schema-Driven: Every config type has a strict, machine-readable schema (JSON Schema/OpenAPI).
- Validated & Safe: Changes undergo multi-stage validation before rollout. Rollouts are gated, observable, and automatically reversible.
- Auditable & Traceable: Every change is linked to a human/process, with a complete, immutable audit log of system actions.
- Agent-Based: A lightweight, privileged agent (the "applier") on each target system fetches and applies configs. It does not interpret logic.
2. System Architecture
[Developer/Platform Engineer] | v [Git Repository] (Source of Truth) | (Push/PR/Merge) v [CI/CD Pipeline] (Schema Validation, Policy Checks) | (Approved Config Version) v [Config Store] (Immutable, Versioned Artifact Store - e.g., S3, GCS, dedicated DB) | (Polling/Webhook) v [Agent on Target System] (Pulls config, validates locally, applies idempotently) | v [Target System] (K8s, VM, Service, Firewall Rule, etc.) | v [Telemetry & Audit Log] -> [Central Log Store (e.g., ELK, Splunk, Loki)]
3. Key Components & Detailed Design
A. Schema Definition & Registry
- Schema Format: JSON Schema (Draft 2020-12) for maximum tooling support. Define schemas per config type (e.g.,
nginx-config.schema.json,firewall-rule.schema.json). - Schema Registry: A dedicated, versioned service or Git repo path (
/schemas/) storing the canonical schemas. - Schema Requirements:
- Strict Types: No
additionalProperties: false. All fields must be declared. - Semantic Versioning: Schemas themselves are versioned (
v1.2.0). A config must declare itsschema_version. - Constraints: Use
format(email, uri, date-time),pattern(regex),minimum/maximum,enum. - Example:
{ "$schema": "https://json-schema.org/draft/2020-12/schema", "$id": "https://yourco.com/schemas/v1/microservice-config.json", "title": "Microservice Configuration", "version": "1.3.0", "type": "object", "properties": { "service_name": { "type": "string", "pattern": "^[a-z0-9-]+$" }, "port": { "type": "integer", "minimum": 1024, "maximum": 65535 }, "replicas": { "type": "integer", "default": 2, "minimum": 1, "maximum": 50 }, "log_level": { "type": "string", "enum": ["DEBUG", "INFO", "WARN", "ERROR"] } }, "required": ["service_name", "port"], "additionalProperties": false }
- Strict Types: No
B. Validation Pipeline (The "Gate")
Multi-layered validation occurs before a config is committed to the Config Store.
-
Static Schema Validation (CI/CD):
- Tool:
ajv-cli,python-jsonschema, or custom library. - Process: On PR/merge to
main, CI runs:- Syntax Check: Is valid JSON/YAML?
- Schema Match: Does config match its declared
schema_version? - Schema Compatibility Check (Crucial): If a config uses
schema_version: "1.3.0", check backward compatibility against1.2.xand1.1.x. Failing this blocks merge. Use tools likejson-schema-diff-validator.
- Tool:
-
Policy & Business Logic Validation (CI/CD or Admission Controller):
- Implemented as custom policy-as-code (OPA/Rego, Kyverno, or custom script).
- Examples:
deny[msg] { input.replicas > 100; msg := "Replica count > 100 requires CFO ticket #XYZ" }deny[msg] { not input.service_name in allowed_services[input.env] }deny[msg] { input.port == 80 and input.ssl_enabled == false }
- Must reference external data: ticket systems, CMDB, cost data.
-
Target-Specific/Contextual Validation (Agent-Side Pre-Apply):
- The agent re-validates the config just before application against the current system state.
- Examples:
- "Config requests port 8080, but port 8080 is already in use by process
X." - "Config wants 10 replicas, but cluster quota only allows 5."
- "Config requests port 8080, but port 8080 is already in use by process
- This is the last line of defense.
C. Rollout Safety & Change Management
- Immutable Config Versions: Once a config version is stored, it is never mutated. New changes create a new version (e.g.,
config-v1,config-v2). - Change Propagation:
- Agent Polling: Agent checks Config Store for a new
config_versionfor itsconfig_idandtarget_id(e.g.,service:frontend, cluster:us-east-1a). (Simple, reliable). - Webhook Push: Config Store sends signed event to agent. (Faster, requires mutable endpoint).
- Agent Polling: Agent checks Config Store for a new
- Rollout Steps:
- Staging First: New config version is automatically pushed to staging environments (defined by label
env: staging) for a sanity check period (e.g., 15 mins). Automated health checks run. - Phased Rollout (Canary): For production:
- Phase 1: 5% of targets (based on label
zone: a) get new config. Monitor key metrics (error rate, latency, resource usage) for 5 mins. - Phase 2: If healthy, 25% more.
- Phase 3: 100%.
- Phase 1: 5% of targets (based on label
- Health Gates: Each phase has automated, configurable health gates. Failure triggers immediate automatic rollback to the last known-good version.
- Example Gate:
curl -f http://localhost:8080/healthz || exit 1 - Example Gate:
avg(rate(http_requests_total{status=~"5.."}[5m])) < 0.01
- Example Gate:
- Staging First: New config version is automatically pushed to staging environments (defined by label
- Manual Override & Rollback:
- CLI/UI command to force a specific version to all targets immediately (break-glass).
- One-click rollback from UI/CLI to previous version.
- Rollback is just another config deployment (of the old version), going through the same phased process. Never directly overwrite the agent's local state.
D. Audit Logging & Observability
- What to Log (The "Who, What, When, Why"):
- Intent Log:
git commitmetadata.actor,timestamp,commit_sha,ticket_id(from PR/branch name),diff. - System Decision Log:
config_storelogs:config_id,from_version,to_version,target_selector,action: "schedule_rollout",approval_source(e.g., "automatic_stage_pass"). - Agent Action Log (CRITICAL):
agent_id,timestampevent: "fetch_config",config_id,version_fetched,response_codeevent: "pre_apply_validation",result: "success/fail",error_messageevent: "apply_start",method: "atomic_replace/stop_start"event: "apply_success",rollback_available: trueevent: "apply_fail",error,trigger_rollback: true
- Health Check Log: Results of each health gate pass/fail per target/phase.
- Intent Log:
- Log Requirements:
- Structured JSON: All logs in JSON format.
- Immutable & Append-Only: Write to non-rewritable storage quickly. Use WORM storage if compliant.
- Tamper-Evident: Cryptographically sign log batches (e.g., Merkle hash) or stream to a secure, separate audit bucket.
- Rich Context: Include
trace_idthat follows a config from PR -> Deploy -> Agent -> Apply. - Centralized Indexing: All logs must be queryable in a central system (e.g., Elasticsearch, Datadog Logs).
- Audit Reports: Pre-built queries to show:
- "All config changes for service
paymentin Q4." - "Who rolled back
frontendconfig and why?" - "Failed health gates for rollout
v42."
- "All config changes for service
4. Agent Implementation Details
- Idempotent & Atomic: Application must be safe to retry. Use atomic operations (e.g.,
kubectl apply --server-side, writing to temp file thenmv). - Dry-Run Mode: Always supports
--dry-runfor validation without change. - Local Caching: Cache last known-good config and schema. Validate against this if store is temporarily unreachable.
- Minimal Privilege: Agent service account has only
geton config store and the exact permissions needed to modify its target (e.g.,patch deployment/frontend). - Heartbeat: Regularly sends
heartbeatlog withlast_successful_config_version.
5. Operational Runbook & Guardrails
- Approval Workflow: PRs require
platform-teamapproval. Forhigh_risklabels (defined in policy), requiresecurity-teamandproduct-ownerapproval. - Schema Change Process: Changing a schema is a separate, high-visibility process.
- Propose new schema version in PR.
- Document migration path for existing configs (how to upgrade them?).
- CI checks that all existing configs are forward-compatible with the new schema or are explicitly marked for forced migration.
- Merge only after migration plan is executed.
- Disaster Recovery:
- Config Store Backup: Daily immutable snapshots of the entire store.
- Schema Registry Backup: Part of the Git repo backup.
- Agent Recovery: If agent is broken, use stored
last_good_configfrom its heartbeat log to manually apply.
- Monitoring Dashboards:
- Rollout Status Map (by cluster, zone, state).
- Age of configs (alert on >30 days without update?).
- Agent health (heartbeat lag).
- Validation failure metrics (by schema, by policy).
6. Technology Stack Example
- Config Store: Git (as source of truth) + Immutable Artifact Store (S3/GCS with versioning). CI pipeline publishes validated configs to S3 with key
configs/{config_id}/{version}/config.json. - Schema Registry: Git subdirectory (
/schemas/) served via static web server or GitHub API. - Validation: OPA for policies, ajv for JSON Schema, conftest for policy testing.
- Agent: Lightweight Go/Python binary (e.g.,
config-agent). For K8s, it could be a K8s Operator (usingcontroller-runtime) that watches aConfigCRD. - Orchestration: Argo CD or Flux can act as the "agent" for Kubernetes, but you must enforce the validation pipeline before the Git repo sync. For non-K8s targets, custom agent.
- Audit Log: Loki (for query), S3 (for long-term immutable storage), HashiCorp Vault Audit Devices (if secrets are involved).
- UI/CLI: Backstage (for catalog), custom CLI for rollout control/rollback.
Summary Checklist for Production Readiness
- Schemas are strict, versioned, and stored in a registry.
- CI/CD enforces schema validation and policy checks on every change.
- Config versions are immutable in the store.
- Rollouts are staged (staging -> canary -> prod) with automated health gates.
- Automatic rollback is triggered on health gate failure.
- Agents perform local validation and apply idempotently.
- Every action (intent, system, agent) is logged as structured JSON.
- Audit logs are shipped to a tamper-evident, centralized, queryable store.
- Schema changes follow a separate, controlled process with migration plans.
- Disaster recovery procedures exist for config store and agent failures.
- Monitoring covers rollout status, agent health, and validation failures.
This system puts safety, traceability, and operational control at the forefront, transforming configuration from a fragile set of files into a robust, governable, and observable data product.