Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Build a prompt registry for AI features with approvals and metrics.

  1. User

    Build a prompt registry for AI features with approvals and metrics.

  2. GLM Assistant

    The user wants to build a "prompt registry" for AI features. This implies a system to manage, version, approve, and track the performance of prompts used in AI applications. It's a system design or architecture task, or perhaps a request for a UI/UX design or a technical specification.

    Key Components:

    1. Storage: Where prompts live (DB).
    2. Versioning: Git-like semantics (v1.0, v1.1, draft, production).
    3. Approval Workflow: Status changes (Draft -> Pending Review -> Approved -> Production).
    4. Metrics/Observability: Tracking usage, cost, latency, and success rates of specific prompt versions.
    5. Interface: A dashboard for developers and product managers to manage this.

    Target Audience: Likely developers or product managers building an AI platform.

    Structure of the Response: I should provide a comprehensive guide covering:

    1. Conceptual Architecture: How the pieces fit together.
    2. Database Schema: A concrete way to store the data.
    3. API Design: How to interact with the registry.
    4. Workflow: The lifecycle of a prompt.
    5. Metrics: What to measure and how.
    6. Example UI/UX: What it should look like to the user.

    Refining the details:

    • Schema: Needs tables for Prompts, PromptVersions, ApprovalRequests, Deployments, and Metrics.
    • Approvals: Needs roles (Author, Reviewer, Admin).
    • Metrics: Needs integration hooks. The registry needs to receive data back from the LLM calls.
    • Tech Stack: Agnostic, but mentioning things like PostgreSQL, Redis, and maybe a framework like LangChain or LlamaIndex for context helps.

    Drafting the Content:

    • Title: Enterprise Prompt Registry System Design.

    • Core Features:

      • Prompt Library: Centralized storage.
      • Version Control: Immutable versions.
      • Access Control: Who can edit/approve.
      • Observability: Feedback loops.
    • The "Approval" aspect:

      • Draft -> Ready for Review -> Approved -> Production.
      • Comments/Annotations.
    • The "Metrics" aspect:

      • Tokens used.
      • Latency.
      • User feedback (thumbs up/down).
      • Error rates.
    • Implementation Steps:

      1. Define Schema.
      2. Build API (CRUD + Deploy).
      3. Build Approval Logic.
      4. Build Metrics Ingestion.
      5. Build Dashboard.

    Self-Correction during drafting: I shouldn't just give code. I should give a system design and a solution. A SQL schema is good, but a Python client library example is also helpful to show how the application actually calls the prompt.

    Let's organize the response into logical sections: Architecture, Data Model, Workflow, and Implementation Example.

    Step-by-step plan:

    1. High-Level Overview: What is it and why do you need it?
    2. System Architecture Diagram (Text-based): Visualizing components.
    3. Database Schema (SQL): The backbone.
    4. The Lifecycle (Workflow): Draft to Production.
    5. Metrics Integration: How to track performance.
    6. Client SDK Design: How developers actually use it in code.
    7. UI/UX Features: What the dashboard looks like.

    Let's flesh out the SQL schema. prompts table: id, name, description, created_at. prompt_versions table: id, prompt_id, version_string, content, is_active, status (draft, approved, archived), created_by. approval_logs table: id, version_id, approver_id, status, comments, timestamp. metrics table: id, version_id, latency_ms, tokens_in, tokens_out, success, feedback_score.

    This looks solid. I will present this as a "System Design Blueprint."

    Additional thought: Security. Prompt injection is a thing, but here we are managing the storage. The security aspect here is about who can promote a prompt to production to prevent malicious or broken prompts from going live.

    Final Polish: Make sure to distinguish between "Soft Delete" (Archiving) and "Hard Delete" (rarely used). Emphasize immutability of versions.

    Let's write it out. This is a comprehensive system design and implementation blueprint for a Prompt Registry. This system acts as the "Source of Truth" for your LLM prompts, separating code from content, managing version control, handling approvals, and tracking performance metrics.


    System Architecture: The Prompt Registry

    The registry sits between your developers, your product, and your LLM application code.

    1. The Registry Service (API/DB): Stores prompt text, version history, and metadata (model, temperature).
    2. The Approval Engine: A workflow layer that enforces review states (Draft →\to Pending →\to Approved).
    3. The SDK / Client: A lightweight library used in your AI application to fetch the specific prompt version.
    4. The Metrics Collector: A background worker that logs latency, token usage, and user feedback mapped to specific prompt versions.

    1. Database Schema (PostgreSQL)

    We need a relational structure to handle versioning and approvals rigorously.

    -- 1. PROMPTS: The parent container for a feature's prompt
    CREATE TABLE prompts (
        id UUID PRIMARY KEY,
        slug VARCHAR(255) UNIQUE NOT NULL, -- e.g., "chat-support-v2"
        name VARCHAR(255) NOT NULL,
        description TEXT,
        created_at TIMESTAMP DEFAULT NOW()
    );
    
    -- 2. PROMPT_VERSIONS: Immutable snapshots of the prompt content
    CREATE TABLE prompt_versions (
        id UUID PRIMARY KEY,
        prompt_id UUID REFERENCES prompts(id),
        version_tag VARCHAR(50) NOT NULL, -- e.g., "v1.0.2" or git commit hash
        content TEXT NOT NULL,            -- The actual prompt template
        config JSONB,                     -- {"model": "gpt-4", "temperature": 0.7}
        status VARCHAR(50) DEFAULT 'draft', -- draft, pending_review, approved, archived
        created_by UUID,                  -- User ID
        created_at TIMESTAMP DEFAULT NOW(),
        UNIQUE(prompt_id, version_tag)
    );
    
    -- 3. APPROVAL_WORKFLOWS: Audit trail for approvals
    CREATE TABLE approval_workflows (
        id UUID PRIMARY KEY,
        prompt_version_id UUID REFERENCES prompt_versions(id),
        reviewer_id UUID,
        action VARCHAR(50), -- 'approve', 'reject', 'request_changes'
        comment TEXT,
        created_at TIMESTAMP DEFAULT NOW()
    );
    
    -- 4. EXECUTION_METRICS: Performance tracking
    CREATE TABLE execution_metrics (
        id UUID PRIMARY KEY,
        prompt_version_id UUID REFERENCES prompt_versions(id),
        session_id UUID,                  -- To link input/output pairs
        latency_ms INT,
        input_tokens INT,
        output_tokens INT,
        user_feedback INT,                -- e.g., 1 (good), -1 (bad), 0 (neutral)
        created_at TIMESTAMP DEFAULT NOW()
    );
    

    2. The API Design

    A. Management API (CRUD & Workflow)

    Used by the internal dashboard.

    • POST /prompts: Create a new prompt container.
    • POST /prompts/{slug}/versions: Create a new version (defaults to draft).
    • POST /versions/{id}/submit-for-review: Changes status to pending_review.
    • POST /versions/{id}/approve: Changes status to approved. Locks the version.
    • GET /prompts/{slug}/latest: Retrieves the latest approved version.

    B. Serving API (High Traffic)

    Used by your AI Application backend.

    • GET /serve/{slug}?version=v1.0.2:
      • Returns the prompt content and config.
      • Caching Strategy: This endpoint should be heavily cached (Redis) to avoid DB hits during high-volume inference.

    3. The Approval Workflow

    This ensures no prompt goes into production without human oversight.

    1. Draft: A Prompt Engineer writes a new version.
    2. Diff View: The dashboard shows the git-style diff between the new version and the currently live version.
    3. Submission: Engineer submits for review.
    4. Review: A designated "Approver" (Product Manager or Lead Dev) checks for:
      • Safety (bias, injection risks).
      • Brand tone.
      • Logic correctness.
    5. Promotion: Upon approval, the version is tagged approved.
    6. Deployment: The application is configured to point to the new version_tag via environment variables or a feature flag system.

    4. Metrics & Observability Implementation

    To measure which prompts perform best, you must pass a Trace ID.

    Step 1: Fetch Prompt (Pre-generation) Your app calls the registry.

    {
      "prompt_id": "chat-support-v2",
      "version": "v1.0.5",
      "content": "You are a helpful assistant...",
      "trace_id": "uuid-1234" 
    }
    

    Step 2: Log Result (Post-generation) After the LLM response is streamed to the user, send a background log to the registry.

    // Conceptual Logging Function
    async function logMetric(traceId, promptVersion, latency, tokens, feedback) {
      await db.insert('execution_metrics', {
        prompt_version_id: promptVersion,
        session_id: traceId,
        latency_ms: latency,
        input_tokens: tokens.prompt,
        output_tokens: tokens.completion,
        user_feedback: feedback // Thumbs up/down
      });
    }
    

    Dashboard Visualization: You can now build a view comparing v1.0.4 vs v1.0.5:

    • v1.0.4: Avg Latency 1200ms, Satisfaction 85%
    • v1.0.5: Avg Latency 900ms, Satisfaction 92% (Winner)

    5. Python SDK Example (Client Side)

    This is the code your backend engineers will use. It abstracts away the API calls and logging.

    class PromptRegistry:
        def __init__(self, api_key, cache_enabled=True):
            self.api_key = api_key
            self.cache = {} # Simple in-memory cache for demo
    
        def get_prompt(self, slug, version="latest"):
            """
            Fetches the prompt. 
            In production, this hits Redis first, then the DB.
            """
            cache_key = f"{slug}:{version}"
            if cache_key in self.cache:
                return self.cache[cache_key]
                
            # Simulate API Call
            response = requests.get(
                f"https://registry.mycompany.com/serve/{slug}", 
                params={"version": version},
                headers={"Authorization": f"Bearer {self.api_key}"}
            )
            data = response.json()
            
            prompt_obj = Prompt(
                content=data['content'],
                config=data['config'],
                version_id=data['id'],
                trace_id=str(uuid.uuid4())
            )
            
            self.cache[cache_key] = prompt_obj
            return prompt_obj
    
        def log_outcome(self, prompt_obj, latency_ms, input_tokens, output_tokens, feedback=None):
            """
            Async fire-and-forget logging of metrics.
            """
            payload = {
                "trace_id": prompt_obj.trace_id,
                "prompt_version_id": prompt_obj.version_id,
                "latency_ms": latency_ms,
                "input_tokens": input_tokens,
                "output_tokens": output_tokens,
                "feedback": feedback
            }
            # Send to metrics endpoint (Kafka, Segment, or Direct DB insert)
            requests.post("https://registry.mycompany.com/metrics", json=payload)
    
    # --- USAGE IN YOUR AI APP ---
    
    # 1. Initialize
    registry = PromptRegistry(api_key="sk_123")
    
    # 2. Get the prompt
    prompt = registry.get_prompt("chat-support", version="v1.0.5")
    
    # 3. Call OpenAI
    start_time = time.time()
    response = openai.ChatCompletion.create(
        model=prompt.config['model'],
        messages=[
            {"role": "system", "content": prompt.content},
            {"role": "user", "content": user_input}
        ]
    )
    latency = (time.time() - start_time) * 1000
    
    # 4. Log metrics (user feedback is optional here)
    registry.log_outcome(
        prompt, 
        latency_ms=latency, 
        input_tokens=response.usage.prompt_tokens,
        output_tokens=response.usage.completion_tokens
    )
    

    6. UI Requirements (Frontend)

    You will need a simple internal admin panel.

    1. Editor: A markdown/text editor with Jinja2 or Handlebars syntax highlighting (for variables like {{user_name}}).
    2. Diff Tool: A side-by-side comparison view for approvals.
    3. Metric Charts: A line graph showing "Token Cost" and "User Satisfaction" over time, filterable by Prompt Version.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.