Build a conversion API governance: endpoint SLOs, retries, de‑dup, and data minimization.
- User
Build a conversion API governance: endpoint SLOs, retries, de‑dup, and data minimization.
Assistant
The user wants to build a "conversion API governance" system. Key requirements:
- Endpoint SLOs (Service Level Objectives): Defining performance targets.
- Retries: Handling failures robustly.
- De-duplication (De-dup): Preventing duplicate processing of events.
- Data Minimization: Sending only necessary data.
This sounds like a system design or architectural guide for a backend service (likely related to marketing/conversion tracking like Facebook/Meta CAPI, Google Ads API, etc.).
Target Audience: Backend engineers, architects, or technical product managers setting up a Conversion API (CAPI) infrastructure.
Scope:
- Conceptual: Explain the principles.
- Technical: Provide code patterns or architectural components.
- Specifics: Address the four specific points requested.
Step-by-Step Breakdown:
-
Introduction: Define what Conversion API Governance is. It's about reliability, privacy, and efficiency when sending server-side events to ad platforms.
-
Component 1: Endpoint SLOs (Service Level Objectives)
- Why: Ad platforms have rate limits and strict latency requirements. If you are too slow, you lose attribution.
- Metrics: Latency (p95/p99), Availability (99.9%), Error Rate (<0.1%).
- Implementation:
- Circuit Breakers (stop hitting a failing API).
- Rate Limiting (token bucket).
- Timeout configuration (don't wait forever).
- Governance aspect: Monitoring and alerting when SLOs are breached.
-
Component 2: Retries & Backoff
- Why: Networks fail. APIs glitch. We need eventual consistency.
- Strategy: Exponential Backoff with Jitter (to avoid thundering herd).
- Idempotency: Crucial here. If a retry actually goes through twice, did we charge the user twice? (Though CAPI is usually just events, deduplication handles the logic, but the retry request needs to look the same).
- Implementation: Queues (Kafka/SQS) are better than inline retries.
-
Component 3: De-duplication (De-dup)
- Why: Browsers send events (pixels) AND servers send events (CAPI). We need to count the conversion once.
- Mechanism: Hashing PII (email/phone) + Event ID (transaction ID).
- The "FBC" vs. "FBP" cookie problem: Handling the linkage between browser and server.
- Storage: Redis (Bloom filters or TTL-based keys) to check if an event was recently seen.
-
Component 4: Data Minimization (Privacy by Design)
- Why: GDPR/CCPA compliance and reducing risk. Don't send what isn't needed.
- Mechanism:
- Hashing (SHA256) before sending.
- Field Dropping (only send required parameters for the specific ad platform).
- PII masking/redaction.
- Schema Validation: Reject events with forbidden fields at the ingress point.
-
Architecture Diagram (Mental Model):
- Client (App/Web) -> Load Balancer -> API Gateway (Validation/Schema) -> Queue (Buffer) -> Worker (Transformation/Hashing) -> Dedup Store (Redis) -> Ad Platform API.
-
Code/Structure Example: I should provide a Python/Node pseudo-code structure to show how these fit together.
-
Review against constraints:
- SLOs covered? Yes (Timeouts/Circuit Breakers).
- Retries covered? Yes (Backoff/Queues).
- De-dup covered? Yes (Event ID hashing).
- Data Minimization covered? Yes (Schema enforcement/Hashing).
Refining the "Governance" angle: It's not just building the API, it's governing it. This implies rules and policy enforcement.
- Policy: "No raw PII leaves the database."
- Policy: "Duplicate events must be dropped within 48 hours."
- Policy: "Latency must be < 200ms."
I will structure the response as a comprehensive framework: "The CAPI Governance Framework."
Drafting the Content:
- SLOs: Focus on Circuit Breakers to protect the SLO. If the downstream API (Meta/Google) is down, don't waste resources retrying immediately.
- Retries: Emphasize asynchronous processing. The client shouldn't wait for the ad platform to respond. Use a message queue.
- De-dup: The standard is
event_name+event_id+user_data. I need to explain the difference between deduping parallel streams (Pixel vs Server) and retries. - Data Minimization: This is about the "Transformation Layer." Raw data in -> Clean/Minimized data out.
Self-Correction/Refinement during drafting:
-
Initial thought: Just show code.
-
Correction: Governance is high-level + implementation. I should structure it by "Governance Domain" -> "Implementation Strategy" -> "Code Pattern".
-
Specific Detail: For De-dup, mention that the Event ID is the key. If the browser pixel sends
event_id=123and server sendsevent_id=123, the ad platform dedupes it. But the server needs to dedupe its own retries too. Actually, the ad platform usually handles the dedup of Pixel vs Server if you send the event_id. The server-side governance needs to prevent the server from sending the same event twice due to internal logic bugs. Wait, standard CAPI deduplication (Pixel vs Server) relies on passing the sameevent_id. I must clarify that. -
Data Minimization: Hashing is key. SHA256 is the standard. Normalization (lowercase, trim) is mandatory before hashing.
Final Structure:
- Architecture Overview: The "Middleware" pattern.
- Domain 1: Reliability (SLOs & Retries): Queues, Circuit Breakers, Exponential Backoff.
- Domain 2: Data Integrity (De-duplication): Bloom filters, Event IDs, Event Matching.
- Domain 3: Privacy & Compliance (Data Minimization): Normalization, Hashing, Schema allow-lists.
- Implementation Example (Python/Pseudo-code): Tying it together.
Let's build this out. Building a Governance framework for a Conversion API (CAPI) requires moving beyond "just sending events" to treating your event pipeline as a mission-critical data product. Ad platforms (like Meta, Google, TikTok) penalize accounts with poor data quality or spammy patterns, and regulations (GDPR/CCPA) penalize data leaks.
Here is a comprehensive architectural guide and implementation pattern for CAPI Governance, covering Endpoint SLOs, Retries, De-duplication, and Data Minimization.
1. System Architecture Overview
Do not send events directly from your web server to the Ad Platform. Use an intermediate Event Gateway (or Message Queue).
- Flow:
Client/App->Ingestion API->Message Queue (Kafka/SQS)->Processing Worker->Ad Platform API - Benefit: This decouples user latency from ad platform latency and allows for robust de-duplication and retry logic.
2. Governance Domain 1: Endpoint SLOs (Service Level Objectives)
Ad platforms are notorious for latency spikes and rate limits. Your governance policy must define thresholds for how you interact with them to prevent your own system from clogging.
The Policies
- Latency Target: P95 latency for the
HTTP POSTto the partner must be < 500ms. - Error Budget: < 0.1% 5xx errors from the partner over a rolling 24-hour window.
- Throughput Limits: Respect partner rate limits (e.g., Meta's 100k calls/hour) strictly.
Implementation: Circuit Breakers & Timeouts
Use the Circuit Breaker Pattern. If the Ad Platform starts failing, stop sending requests immediately to allow them to recover, queueing local data instead.
import pybreaker import requests from datetime import timedelta # Define the SLO enforcement via Circuit Breaker # Open circuit if 5 failures occur within 60 seconds ad_platform_breaker = pybreaker.CircuitBreaker( fail_max=5, reset_timeout=timedelta(seconds=60) ) @ad_platform_breaker def send_to_partner(event_payload): # Enforce strict timeout to prevent thread blocking # Connect: 200ms, Read: 500ms response = requests.post( "https://graph.facebook.com/vXX.X/.../events", json=event_payload, timeout=(0.2, 0.5) ) response.raise_for_status() return response
3. Governance Domain 2: Intelligent Retries
Blindly retrying API calls causes "thundering herd" problems and can get your API access revoked.
The Policies
- Idempotency: All retries must be idempotent (safe to repeat).
- Backoff Strategy: Exponential Jitter Backoff.
- Retry Eligibility: Only retry
5xx(Server Errors) or429(Rate Limit). Never retry400(Bad Request) – fix the data instead.
Implementation: Exponential Backoff with Jitter
"Jitter" (randomness) prevents all your servers from retrying at the exact same second after an outage.
import time import random def dispatch_with_retry(payload, max_retries=3): attempt = 0 while attempt < max_retries: try: return send_to_partner(payload) except pybreaker.CircuitBreakerError: # Queue for later processing (e.g., write to DB) save_to_dead_letter_queue(payload) return except requests.exceptions.RequestException as e: if e.response.status_code == 400: # Governance Alert: Bad data detected, do not retry log_data_quality_error(payload, e) return # Calculate wait time: Exponential + Jitter # e.g., 2^1 + random(0,1) = 2.x seconds wait_time = (2 ** attempt) + random.uniform(0, 1) time.sleep(wait_time) attempt += 1 # If max retries reached save_to_dead_letter_queue(payload)
4. Governance Domain 3: De-duplication (De-dup)
Duplicate events artificially inflate performance metrics, leading to incorrect ad spend optimization. Governance here acts at two levels: Browser vs. Server collisions and Server-Side retries.
The Policies
- Event ID Standardization: Every event MUST have a unique
event_id(e.g., UUID). - Dedup Window: Check for existence of this ID within the last 48 hours.
Implementation: The Dedup Store (Redis)
Before sending an event, check if the hash exists. If it does, discard it.
import redis import hashlib r = redis.Redis(host='localhost', port=6379, db=0) def is_duplicate(event_id, user_data): # Create a unique signature combining EventID + User (to prevent ID collision between users) # SHA256 is fast and sufficient for short-lived dedup keys signature = hashlib.sha256(f"{event_id}:{user_data}".encode()).hexdigest() key = f"dedup:{signature}" # SETNX returns True if the key was NOT set (meaning it's new) # We set a TTL of 48 hours (172800 seconds) if r.setnx(key, 1): r.expire(key, 172800) return False # Not a duplicate else: return True # Duplicate found def process_event(payload): if is_duplicate(payload['event_id'], payload['user_data']): metrics.increment("governance.duplicates_dropped") return dispatch_with_retry(payload)
5. Governance Domain 4: Data Minimization
This is the most critical governance aspect for privacy compliance. Ad platforms request many fields; you should only send what is contractually and legally necessary.
The Policies
- PII Normalization: All PII (Personally Identifiable Information) must be hashed (SHA256) and lowercased before leaving your trust boundary.
- Schema Allow-listing: Drop any fields that are not explicitly required by the marketing analytics use case (e.g., do not send "medical_history" even if the frontend collects it).
- Consent Enforcement: Do not process events if the
consent_signalis missing or false.
Implementation: The Sanitization Layer
Use a transformation function that acts as a firewall.
import hashlib # Governance Schema: Only these fields are allowed ALLOWED_FIELDS = ['em', 'ph', 'fn', 'ln', 'ct', 'st', 'zp', 'country'] REQUIRED_SOURCES = ['email', 'phone'] def normalize_and_hash(value): if not value: return "" # Lowercase, trim, remove spaces clean_val = value.strip().lower() return hashlib.sha256(clean_val.encode('utf-8')).hexdigest() def minimize_and_transform(raw_event): # 1. Consent Check (Hard Stop) if not raw_event.get('gdpr_consent') and not raw_event.get('ccpa_consent'): raise ValueError("Governance Violation: Missing Consent") # 2. Minimize Payload clean_payload = { "event_name": raw_event['event_name'], "event_time": raw_event['event_time'], "event_id": raw_event['event_id'], "user_data": {}, "custom_data": {} } # 3. Hash and Move PII # Map internal fields to Partner keys (e.g., 'email' -> 'em') if raw_event.get('email'): clean_payload['user_data']['em'] = normalize_and_hash(raw_event['email']) if raw_event.get('phone'): clean_payload['user_data']['ph'] = normalize_and_hash(raw_event['phone']) # 4. Strip non-allowed custom data # Only keep custom data relevant to ROAS (Return on Ad Spend), e.g., value, currency, content_ids for k in ['value', 'currency', 'content_ids']: if raw_event.get(k): clean_payload['custom_data'][k] = raw_event[k] return clean_payload
Summary: The Governance Pipeline
To operationalize this, combine the four domains into a single processing pipeline:
- Ingest: Receive raw event.
- Minimize: Strip unnecessary fields and Hash PII (Domain 4).
- Deduplicate: Check Redis for
event_id(Domain 3). - Dispatch:
- Check Circuit Breaker status (Domain 1).
- Send with Timeout enforcement (Domain 1).
- Handle failure with Exponential Backoff (Domain 2).
This architecture ensures your Conversion API is reliable, privacy-compliant, and trusted by both your organization and the ad platforms.