Design an LLM application gateway: prompt policies, PII scrubbing, and audit.
- User
Design an LLM application gateway: prompt policies, PII scrubbing, and audit.
- Assistant
LLM Application Gateway Design
Overview
An LLM Application Gateway is a middleware layer that intercepts requests to Large Language Model (LLM) APIs (e.g., OpenAI, Anthropic, or custom models). It enforces security, compliance, and operational best practices by implementing:
- Prompt Policies: Enforce rules on incoming prompts to block harmful, disallowed, or risky content.
- PII Scrubbing: Detect and redact Personally Identifiable Information (PII) from prompts and responses.
- Audit: Comprehensive logging for traceability, compliance (e.g., GDPR, SOC2), and analytics.
Goals:
- Protect the LLM from abuse (e.g., jailbreaks, toxicity).
- Safeguard user privacy.
- Enable monitoring and forensic analysis.
- Scale to high throughput with low latency (<100ms overhead).
Key Benefits:
- Centralized control without modifying client apps.
- Vendor-agnostic (works with any LLM provider).
- Real-time enforcement with fallback options.
High-Level Architecture
[Client/App] --> [Gateway API (HTTP/REST or gRPC)] | v +-------------------+ | 1. Prompt Policies| +-------------------+ | (Block/Allow)| v +-------------------+ | 2. PII Scrubbing | +-------------------+ | (Redacted) | v +-------------------+ | 3. LLM Proxy | +-------------------+ | (Response) | v +-------------------+ | 4. PII Scrubbing | +-------------------+ | (Redacted) | v +-------------------+ | 5. Audit & Logging| +-------------------+ | v [Client/App] <-- [Response/Block]- Deployment: Kubernetes pods with Envoy/Istio for routing, Redis for caching, Kafka/ELK for logging.
- Tech Stack: Node.js/Python (FastAPI/Express), regex/ML for PII, OpenPolicyAgent (OPA) for policies.
Detailed Components
1. Prompt Policies
Filters incoming prompts against configurable rules. Processes in <50ms.
Policy Types:
Type Description Enforcement Examples Content Block disallowed topics Regex + LLM classifier Hate speech, illegal activities (e.g., "how to hack"), NSFW. Jailbreak Detect prompt injections Regex patterns + embedding similarity DAN-style prompts, "ignore rules". Rate Limiting Per-user/IP/API-key Token bucket (Redis) 100 req/min/user. Length/Complexity Prompt size limits Simple checks Max 32k tokens. Custom Business rules YAML/JSON config Block finance queries if not authenticated. Implementation:
- Regex Prescan: Fast blocklist (e.g.,
/bomb|drug|(?:ignore|override)\s+rules/i). - LLM Classifier: Lightweight model (e.g., fine-tuned Llama-3-8B) scores risk (0-1). Threshold >0.8 = block.
- OPA Integration: Rego policies for complex logic (e.g.,
allow if user.role == "admin"). - Actions:
ALLOW,BLOCK(with reason),QUARANTINE(log + human review),MODIFY(e.g., append safety instructions).
Config Example (YAML):
policies: - name: jailbreak type: regex patterns: ["ignore previous instructions", "you are DAN"] action: block - name: toxicity type: llm model: "toxicity-classifier" threshold: 0.8 action: quarantine2. PII Scrubbing
Detects and redacts PII in prompts (pre-LLM) and responses (post-LLM) to prevent leakage.
PII Categories (NER-based):
- Names, emails, phones, SSNs, credit cards, addresses, IP addresses, API keys.
- Custom: Company-specific (e.g., internal IDs).
Implementation:
-
Hybrid Detection:
Method Pros/Cons Use Case Regex Fast, precise for patterns Emails (`\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+.[A-Z NER Model Contextual accuracy spaCy/Presidio (en_core_web_sm) or fine-tuned BERT. LLM Embeddings Semantic detection Flag "John Doe" in context. -
Redaction: Replace with
[REDACTED-PII-<type>](e.g.,[REDACTED-EMAIL]). Configurable (hash, anonymize). -
Bidirectional: Scrub prompt → LLM → scrub response.
-
Caching: Hash scrubbed prompts for dedup (Redis TTL 1h).
-
Accuracy: >95% F1-score; false positives tuned via allowlist.
Flow:
- Input: "Contact John Doe at john.doe@example.com"
- Scrubbed: "Contact [REDACTED-NAME] at [REDACTED-EMAIL]"
- LLM Response: Scrubbed similarly.
3. Audit
Immutable logging for every request/response.
Logged Fields (JSON Schema)**:
{ "request_id": "uuid", "timestamp": "ISO8601", "user_id": "anon-hash", "ip": "hashed", "prompt": "scrubbed", "response": "scrubbed", "policy_decisions": ["allowed", "pii_redacted:2"], "llm_provider": "openai/gpt-4o", "tokens_in/out": 150/200, "latency_ms": 2500, "status": "completed/blocked" }Features:
- Storage: Kafka → S3/ClickHouse (queryable). Retention: 90 days hot, 7y cold.
- Alerts: Slack/ PagerDuty on blocks/quarantines/high PII volume.
- Dashboards: Grafana for metrics (req/sec, block rate, PII types).
- Search: Full-text on scrubbed fields; export for compliance.
- Anonymization: User/IP hashed (SHA256) pre-log.
Data Flow (Step-by-Step)
- Ingress: Receive POST
/chatwith{prompt, user_id, metadata}. - Auth: JWT/API-key validation.
- Policies: Evaluate → BLOCK (400: "Policy violation: jailbreak") or proceed.
- Scrub Prompt: Redact PII → Forward to LLM.
- Proxy LLM: Call backend (with retries, token limits).
- Scrub Response: Redact PII.
- Audit: Log everything → Return response.
- Egress: 200 OK or error.
Latency Budget: Policies (20ms) + Scrub (30ms) + Audit (10ms) = <100ms overhead.
Implementation Considerations
- Scalability: Horizontal pods, async scrubbing (worker queues).
- Security: mTLS to LLM, WAF (Cloudflare), secrets in Vault.
- Config Management: GitOps (ArgoCD), hot-reload policies.
- Testing: Unit (regex), integration (mock LLM), chaos (fault injection).
- Cost: ~$0.01/1k req; optimize with batching.
- Open-Source Base: Use LiteLLM (proxy) + Presidio (PII) + OPA.
Edge Cases & Error Handling
- High-Risk Prompt: Block + notify security.
- PII in Response: Log incident, fallback to safe response ("PII detected").
- LLM Failure: Retry 3x, then 429.
- Zero PII False Positives: User override via
?allow_pii=true(audit-flagged). - Multi-Turn: Session state in Redis for context scrubbing.
This design is production-ready and extensible. For a POC, I can provide Docker Compose + code snippets. Questions?