Create a prompt engineering service: templates, filters, PII scrubbing, and telemetry.
- User
Create a prompt engineering service: templates, filters, PII scrubbing, and telemetry.
Assistant
The user wants a "prompt engineering service" that includes four specific features:
- Templates: Reusable structures for prompts.
- Filters: Mechanisms to sanitize or modify input/output (often related to safety or content moderation).
- PII Scrubbing: Personally Identifiable Information detection and removal/redaction.
- Telemetry: Logging, monitoring, and analytics of prompt usage.
The user hasn't specified how they want this delivered (e.g., a conceptual architecture, a Python library, a microservices design, or a specific code implementation). Given the complexity, providing a comprehensive design proposal combined with a reference implementation (Python/FastAPI) is usually the best approach. It covers the theoretical "what" and the practical "how".
Plan:
- Architecture Overview: Define the components and how they interact (Middleware pattern is usually best for this).
- Core Data Structures: Define what a Template looks like, what a Filter does, etc.
- Implementation Details (Python):
- Templates: A Jinja2 based system is standard for string interpolation.
- PII Scrubbing: I need a library or a regex approach.
presidiois the industry standard for PII, but it's heavy. For a code example, maybe a regex approach or a mock function is better to keep dependencies low, but mentioning Presidio is crucial. Let's use a simulated scrubber or a simple regex for the demo, but note Presidio for production. - Filters: A pipeline pattern where functions take a string and return a modified string or raise an error.
- Telemetry: A logging decorator or middleware that captures latency, token count (mocked), and prompt content (scrubbed).
- API Interface: A simple REST API (using FastAPI) to demonstrate the service aspect.
- Usage Example: How a client would call this service.
Refining the "Service" aspect: It shouldn't just be a library; it needs to act like a service. So, a FastAPI application is the right wrapper.
Feature Breakdown:
- Templates: Need a
PromptTemplateclass.name: stringcontent: string (with placeholders)variables: list
- Filters:
- Input validation (length, banned words).
- Output moderation (stop sequences).
- PII Scrubbing:
- Regex for emails/phones is easy to show.
- Integration point before sending to the LLM and before logging.
- Telemetry:
- Store logs.
- Metrics: Latency, Model used, Success/Fail.
Drafting the Code Structure:
models.py: Pydantic models for API requests.services/:template_service.py: CRUD for templates.pii_service.py: Logic to redact.filter_service.py: Logic to clean.telemetry_service.py: Logic to log.
main.py: The FastAPI endpoints.
Self-Correction/Constraint Check: Building a full PII scrubber from scratch is hard/insecure. I will implement a mock PII scrubber using Regex for emails/phone numbers to demonstrate the integration point, while explicitly stating that production systems should use Microsoft Presidio or AWS Macie.
Let's write the prompt response.
- Introduction: High-level architecture.
- Tech Stack: Python, FastAPI, Pydantic, Jinja2.
- Code:
- Define the Template Engine.
- Define the Middleware/Pipeline (PII + Filters).
- Define the Telemetry Wrapper.
- Combine in a FastAPI app.
- Summary: How to run and extend.
Let's make the code modular so the user can see how to separate concerns.
-
Step 1: Architecture Diagram (Text based) Request -> API Gateway -> [PII Scrubbing] -> [Filtering] -> Template Rendering -> [Telemetry] -> LLM Provider.
-
Step 2: The Code I'll create a single file structure for ease of copy-pasting, but organize it clearly.
- PII: Use
reto detect emails/phones. - Templates: Use
jinja2(standard). - Telemetry: Use standard
logging+ a structured dictionary.
Actually, let's make it a "Wrapper Class" design. A class
PromptEngineeringServicethat takes a raw request and processes it.Key Method:
process_prompt(template_id, variables)- Load Template.
- Inject Variables -> Raw Prompt.
- Telemetry (Start): Log request start.
- PII Scrub: Scan variables or raw prompt? Usually scrub variables before injection or scrub the final string. Let's scrub the final string for telemetry, but maybe keep raw for the LLM if the user consents? No, standard practice is usually to scrub inputs to prevent the LLM from seeing PII (unless authorized), or at least scrub logs. I will implement Input Scrubbing (redacting PII before sending to LLM) as the default behavior for safety.
Wait, usually you want the LLM to see the email if the prompt is "summarize this email". Okay, nuance: PII scrubbing is usually for Telemetry/Logging (don't log PII) and Compliance (maybe don't send to API). I will implement a configurable PII policy.
ScrubForLLM: Boolean.ScrubForLogs: Boolean (True by default).
Let's stick to a robust default:
- Render Prompt.
- Run PII check (returns redacted text + detected entities).
- Run Filters (banned words, injection attacks).
- Log (Redacted version).
- Send to LLM (Original or Redacted based on config).
Filters: Simple keyword blocking for safety.
Telemetry: Just print JSON logs for now, easy to ingest.
- PII: Use
Final Polish: Ensure the code is clean, type-hinted, and uses Pydantic for data validation.
Let's go. This is a design and implementation for a Prompt Engineering Service. This service acts as a middleware layer between your application and an LLM provider (like OpenAI or Anthropic).
It is built using Python and FastAPI, following a modular architecture to handle the four requirements: Templating, PII Scrubbing, Filtering, and Telemetry.
Architecture Overview
- API Layer: Receives generation requests.
- Template Engine: Fills variables into predefined blueprints.
- PII Guard: Detects and redacts/masks sensitive info using RegEx (and placeholders for NLP).
- Filter Layer: Checks for banned words or prompt injection attacks.
- Telemetry: Logs latency, token usage, and request metadata.
1. Project Structure
prompt_service/ ├── main.py # API Entry point ├── models.py # Pydantic models for requests ├── service.py # Core logic orchestrator ├── template_engine.py # Handles prompt templating ├── security.py # PII scrubbing and Filters └── telemetry.py # Logging and metrics2. Implementation
Step 1: Dependencies
You will need
fastapi,uvicorn,pydantic, andjinja2.pip install fastapi uvicorn pydantic jinja2Step 2: The Code
models.py(Data Structures)from pydantic import BaseModel, Field from typing import List, Optional, Dict, Any class PromptRequest(BaseModel): template_id: str = Field(..., description="Identifier for the prompt template") variables: Dict[str, Any] = Field(default_factory=dict, description="Data to fill the template") # Optional config overrides scrub_pii: bool = True apply_filters: bool = True class PromptResponse(BaseModel): rendered_prompt: str pii_detected: bool filter_triggered: bool telemetry_id: strtemplate_engine.py(Templating) Uses Jinja2 to allow logic (loops, if-statements) inside prompts.from jinja2 import Template, Environment, BaseLoader # In a real app, store these in a DB or file system TEMPLATES = { "summarize_email": """ Summarize the following email content. Do not mention PII directly. Email Content: {{ content }} """, "code_review": """ Review the following {{ language }} code for security vulnerabilities: {{ code_snippet }} """ } class TemplateEngine: def render(self, template_id: str, variables: dict) -> str: template_str = TEMPLATES.get(template_id) if not template_str: raise ValueError(f"Template {template_id} not found") # strict=False allows passing extra vars; handle missing vars logic as needed template = Template(template_str) return template.render(**variables)security.py(PII & Filters) Note: For production PII, use Microsoft Presidio or AWS Macie. Here we use Regex for demonstration.import re from typing import Tuple class SecurityFilter: # Simple Regex for Emails and Phone Numbers PII_PATTERNS = { "email": re.compile(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'), "phone": re.compile(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b') } BANNED_WORDS = ["password", "api_key", "secret"] def scrub_pii(self, text: str) -> Tuple[str, bool]: """ Replaces PII with [REDACTED]. Returns: (cleaned_text, was_pii_found) """ found_pii = False for category, pattern in self.PII_PATTERNS.items(): if pattern.search(text): found_pii = True text = pattern.sub(f"[REDACTED_{category.upper()}]", text) return text, found_pii def apply_content_filters(self, text: str) -> bool: """ Returns True if content is unsafe/blocked. """ text_lower = text.lower() for word in BANNED_WORDS: if word in text_lower: return True # Block return False # Safetelemetry.py(Logging & Metrics)import time import uuid import logging # Configure logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger("prompt_telemetry") class TelemetryService: def start_span(self) -> str: trace_id = str(uuid.uuid4()) logger.info(f"TRACE_START: {trace_id}") return trace_id def log_event(self, trace_id: str, event_name: str, data: dict): logger.info(f"TRACE_ID: {trace_id} | EVENT: {event_name} | DATA: {data}") def end_span(self, trace_id: str, start_time: float): latency = time.time() - start_time logger.info(f"TRACE_END: {trace_id} | LATENCY_MS: {latency * 1000:.2f}")service.py(Orchestrator) This ties everything together.from template_engine import TemplateEngine from security import SecurityFilter from telemetry import TelemetryService import time class PromptService: def __init__(self): self.templates = TemplateEngine() self.security = SecurityFilter() self.telemetry = TelemetryService() def process(self, req: 'PromptRequest') -> dict: start_time = time.time() trace_id = self.telemetry.start_span() # 1. Render Template try: raw_prompt = self.templates.render(req.template_id, req.variables) self.telemetry.log_event(trace_id, "TEMPLATE_RENDERED", {"id": req.template_id}) except ValueError as e: self.telemetry.log_event(trace_id, "ERROR", {"msg": str(e)}) raise # 2. Filters (Check for banned content in input) if req.apply_filters: is_blocked = self.security.apply_content_filters(raw_prompt) if is_blocked: self.telemetry.log_event(trace_id, "FILTER_BLOCK", {}) raise ValueError("Input blocked by content filter") # 3. PII Scrubbing (Clean before sending to LLM) clean_prompt, pii_found = self.security.scrub_pii(raw_prompt) if req.scrub_pii else (raw_prompt, False) if pii_found: self.telemetry.log_event(trace_id, "PII_DETECTED", {}) # 4. Finalize self.telemetry.end_span(trace_id, start_time) return { "rendered_prompt": clean_prompt, "pii_detected": pii_found, "filter_triggered": False, # Would be true if not raising error "telemetry_id": trace_id }main.py(API Entry)from fastapi import FastAPI, HTTPException from models import PromptRequest, PromptResponse from service import PromptService app = FastAPI(title="Prompt Engineering Service") prompt_svc = PromptService() @app.post("/v1/generate", response_model=PromptResponse) async def generate_prompt(request: PromptRequest): try: result = prompt_svc.process(request) return PromptResponse(**result) except ValueError as e: raise HTTPException(status_code=400, detail=str(e)) @app.get("/health") def health(): return {"status": "healthy"}3. How to Run & Test
-
Run the server:
uvicorn main:app --reload -
Test Request (cURL): This request includes PII (an email) inside the content variable.
curl -X 'POST' \ 'http://127.0.0.1:8000/v1/generate' \ -H 'Content-Type: application/json' \ -d '{ "template_id": "summarize_email", "variables": { "content": "My name is John. Contact me at john.doe@example.com or 555-0199." } }' -
Expected Output: Note that the email and phone number are automatically redacted in the
rendered_prompt.{ "rendered_prompt": "\n Summarize the following email content. \n Do not mention PII directly.\n \n Email Content:\n My name is John. Contact me at [REDACTED_EMAIL] or [REDACTED_PHONE].\n ", "pii_detected": true, "filter_triggered": false, "telemetry_id": "a1b2c3d4-..." } -
Check Telemetry (Console): You will see structured logs in your terminal:
INFO:prompt_telemetry:TRACE_START: 1234... INFO:prompt_telemetry:TRACE_ID: 1234... | EVENT: TEMPLATE_RENDERED | DATA: {'id': 'summarize_email'} INFO:prompt_telemetry:TRACE_ID: 1234... | EVENT: PII_DETECTED | DATA: {} INFO:prompt_telemetry:TRACE_END: 1234... | LATENCY_MS: 1.05
4. Future Improvements for Production
- Persistent Templates: Move
TEMPLATESdict to a database (Postgres/Mongo) or S3 so you can update prompts without redeploying code. - Advanced PII: Replace the Regex class with Microsoft Presidio for robust NLP-based entity recognition (names, addresses, credit cards).
- Prompt Injection Defense: Add a filter that detects common injection strings (e.g., "Ignore previous instructions") using a small classifier model or heuristic.
- SDK: Build a Python/TypeScript client SDK so developers can call
prompt_service.generate(...)easily from their backend.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.