Build a prompt registry for AI features with approvals and metrics.
- User
Build a prompt registry for AI features with approvals and metrics.
Assistant
The user wants to build a "prompt registry" for AI features. This implies a system to manage, version, approve, and track the performance of prompts used in AI applications. It's a system design or architecture task, or perhaps a request for a UI/UX design or a technical specification.
Key Components:
- Storage: Where prompts live (DB).
- Versioning: Git-like semantics (v1.0, v1.1, draft, production).
- Approval Workflow: Status changes (Draft -> Pending Review -> Approved -> Production).
- Metrics/Observability: Tracking usage, cost, latency, and success rates of specific prompt versions.
- Interface: A dashboard for developers and product managers to manage this.
Target Audience: Likely developers or product managers building an AI platform.
Structure of the Response: I should provide a comprehensive guide covering:
- Conceptual Architecture: How the pieces fit together.
- Database Schema: A concrete way to store the data.
- API Design: How to interact with the registry.
- Workflow: The lifecycle of a prompt.
- Metrics: What to measure and how.
- Example UI/UX: What it should look like to the user.
Refining the details:
- Schema: Needs tables for
Prompts,PromptVersions,ApprovalRequests,Deployments, andMetrics. - Approvals: Needs roles (Author, Reviewer, Admin).
- Metrics: Needs integration hooks. The registry needs to receive data back from the LLM calls.
- Tech Stack: Agnostic, but mentioning things like PostgreSQL, Redis, and maybe a framework like LangChain or LlamaIndex for context helps.
Drafting the Content:
-
Title: Enterprise Prompt Registry System Design.
-
Core Features:
- Prompt Library: Centralized storage.
- Version Control: Immutable versions.
- Access Control: Who can edit/approve.
- Observability: Feedback loops.
-
The "Approval" aspect:
- Draft -> Ready for Review -> Approved -> Production.
- Comments/Annotations.
-
The "Metrics" aspect:
- Tokens used.
- Latency.
- User feedback (thumbs up/down).
- Error rates.
-
Implementation Steps:
- Define Schema.
- Build API (CRUD + Deploy).
- Build Approval Logic.
- Build Metrics Ingestion.
- Build Dashboard.
Self-Correction during drafting: I shouldn't just give code. I should give a system design and a solution. A SQL schema is good, but a Python client library example is also helpful to show how the application actually calls the prompt.
Let's organize the response into logical sections: Architecture, Data Model, Workflow, and Implementation Example.
Step-by-step plan:
- High-Level Overview: What is it and why do you need it?
- System Architecture Diagram (Text-based): Visualizing components.
- Database Schema (SQL): The backbone.
- The Lifecycle (Workflow): Draft to Production.
- Metrics Integration: How to track performance.
- Client SDK Design: How developers actually use it in code.
- UI/UX Features: What the dashboard looks like.
Let's flesh out the SQL schema.
promptstable: id, name, description, created_at.prompt_versionstable: id, prompt_id, version_string, content, is_active, status (draft, approved, archived), created_by.approval_logstable: id, version_id, approver_id, status, comments, timestamp.metricstable: id, version_id, latency_ms, tokens_in, tokens_out, success, feedback_score.This looks solid. I will present this as a "System Design Blueprint."
Additional thought: Security. Prompt injection is a thing, but here we are managing the storage. The security aspect here is about who can promote a prompt to production to prevent malicious or broken prompts from going live.
Final Polish: Make sure to distinguish between "Soft Delete" (Archiving) and "Hard Delete" (rarely used). Emphasize immutability of versions.
Let's write it out. This is a comprehensive system design and implementation blueprint for a Prompt Registry. This system acts as the "Source of Truth" for your LLM prompts, separating code from content, managing version control, handling approvals, and tracking performance metrics.
System Architecture: The Prompt Registry
The registry sits between your developers, your product, and your LLM application code.
- The Registry Service (API/DB): Stores prompt text, version history, and metadata (model, temperature).
- The Approval Engine: A workflow layer that enforces review states (Draft Pending Approved).
- The SDK / Client: A lightweight library used in your AI application to fetch the specific prompt version.
- The Metrics Collector: A background worker that logs latency, token usage, and user feedback mapped to specific prompt versions.
1. Database Schema (PostgreSQL)
We need a relational structure to handle versioning and approvals rigorously.
-- 1. PROMPTS: The parent container for a feature's prompt CREATE TABLE prompts ( id UUID PRIMARY KEY, slug VARCHAR(255) UNIQUE NOT NULL, -- e.g., "chat-support-v2" name VARCHAR(255) NOT NULL, description TEXT, created_at TIMESTAMP DEFAULT NOW() ); -- 2. PROMPT_VERSIONS: Immutable snapshots of the prompt content CREATE TABLE prompt_versions ( id UUID PRIMARY KEY, prompt_id UUID REFERENCES prompts(id), version_tag VARCHAR(50) NOT NULL, -- e.g., "v1.0.2" or git commit hash content TEXT NOT NULL, -- The actual prompt template config JSONB, -- {"model": "gpt-4", "temperature": 0.7} status VARCHAR(50) DEFAULT 'draft', -- draft, pending_review, approved, archived created_by UUID, -- User ID created_at TIMESTAMP DEFAULT NOW(), UNIQUE(prompt_id, version_tag) ); -- 3. APPROVAL_WORKFLOWS: Audit trail for approvals CREATE TABLE approval_workflows ( id UUID PRIMARY KEY, prompt_version_id UUID REFERENCES prompt_versions(id), reviewer_id UUID, action VARCHAR(50), -- 'approve', 'reject', 'request_changes' comment TEXT, created_at TIMESTAMP DEFAULT NOW() ); -- 4. EXECUTION_METRICS: Performance tracking CREATE TABLE execution_metrics ( id UUID PRIMARY KEY, prompt_version_id UUID REFERENCES prompt_versions(id), session_id UUID, -- To link input/output pairs latency_ms INT, input_tokens INT, output_tokens INT, user_feedback INT, -- e.g., 1 (good), -1 (bad), 0 (neutral) created_at TIMESTAMP DEFAULT NOW() );
2. The API Design
A. Management API (CRUD & Workflow)
Used by the internal dashboard.
POST /prompts: Create a new prompt container.POST /prompts/{slug}/versions: Create a new version (defaults todraft).POST /versions/{id}/submit-for-review: Changes status topending_review.POST /versions/{id}/approve: Changes status toapproved. Locks the version.GET /prompts/{slug}/latest: Retrieves the latestapprovedversion.
B. Serving API (High Traffic)
Used by your AI Application backend.
GET /serve/{slug}?version=v1.0.2:- Returns the prompt content and config.
- Caching Strategy: This endpoint should be heavily cached (Redis) to avoid DB hits during high-volume inference.
3. The Approval Workflow
This ensures no prompt goes into production without human oversight.
- Draft: A Prompt Engineer writes a new version.
- Diff View: The dashboard shows the
git-stylediff between the new version and the currently live version. - Submission: Engineer submits for review.
- Review: A designated "Approver" (Product Manager or Lead Dev) checks for:
- Safety (bias, injection risks).
- Brand tone.
- Logic correctness.
- Promotion: Upon approval, the version is tagged
approved. - Deployment: The application is configured to point to the new
version_tagvia environment variables or a feature flag system.
4. Metrics & Observability Implementation
To measure which prompts perform best, you must pass a Trace ID.
Step 1: Fetch Prompt (Pre-generation) Your app calls the registry.
{ "prompt_id": "chat-support-v2", "version": "v1.0.5", "content": "You are a helpful assistant...", "trace_id": "uuid-1234" }Step 2: Log Result (Post-generation) After the LLM response is streamed to the user, send a background log to the registry.
// Conceptual Logging Function async function logMetric(traceId, promptVersion, latency, tokens, feedback) { await db.insert('execution_metrics', { prompt_version_id: promptVersion, session_id: traceId, latency_ms: latency, input_tokens: tokens.prompt, output_tokens: tokens.completion, user_feedback: feedback // Thumbs up/down }); }Dashboard Visualization: You can now build a view comparing
v1.0.4vsv1.0.5:- v1.0.4: Avg Latency 1200ms, Satisfaction 85%
- v1.0.5: Avg Latency 900ms, Satisfaction 92% (Winner)
5. Python SDK Example (Client Side)
This is the code your backend engineers will use. It abstracts away the API calls and logging.
class PromptRegistry: def __init__(self, api_key, cache_enabled=True): self.api_key = api_key self.cache = {} # Simple in-memory cache for demo def get_prompt(self, slug, version="latest"): """ Fetches the prompt. In production, this hits Redis first, then the DB. """ cache_key = f"{slug}:{version}" if cache_key in self.cache: return self.cache[cache_key] # Simulate API Call response = requests.get( f"https://registry.mycompany.com/serve/{slug}", params={"version": version}, headers={"Authorization": f"Bearer {self.api_key}"} ) data = response.json() prompt_obj = Prompt( content=data['content'], config=data['config'], version_id=data['id'], trace_id=str(uuid.uuid4()) ) self.cache[cache_key] = prompt_obj return prompt_obj def log_outcome(self, prompt_obj, latency_ms, input_tokens, output_tokens, feedback=None): """ Async fire-and-forget logging of metrics. """ payload = { "trace_id": prompt_obj.trace_id, "prompt_version_id": prompt_obj.version_id, "latency_ms": latency_ms, "input_tokens": input_tokens, "output_tokens": output_tokens, "feedback": feedback } # Send to metrics endpoint (Kafka, Segment, or Direct DB insert) requests.post("https://registry.mycompany.com/metrics", json=payload) # --- USAGE IN YOUR AI APP --- # 1. Initialize registry = PromptRegistry(api_key="sk_123") # 2. Get the prompt prompt = registry.get_prompt("chat-support", version="v1.0.5") # 3. Call OpenAI start_time = time.time() response = openai.ChatCompletion.create( model=prompt.config['model'], messages=[ {"role": "system", "content": prompt.content}, {"role": "user", "content": user_input} ] ) latency = (time.time() - start_time) * 1000 # 4. Log metrics (user feedback is optional here) registry.log_outcome( prompt, latency_ms=latency, input_tokens=response.usage.prompt_tokens, output_tokens=response.usage.completion_tokens )
6. UI Requirements (Frontend)
You will need a simple internal admin panel.
- Editor: A markdown/text editor with Jinja2 or Handlebars syntax highlighting (for variables like
{{user_name}}). - Diff Tool: A side-by-side comparison view for approvals.
- Metric Charts: A line graph showing "Token Cost" and "User Satisfaction" over time, filterable by Prompt Version.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.