Skip to content
TheChatArchiveThe Chat Archive
GLMGLM-5 / Pony Alpha· unknownHugging Face Frontier Reasoning Traces

Build a conversion API governance: endpoint SLOs, retries, de‑dup, and data minimization.

  1. User

    Build a conversion API governance: endpoint SLOs, retries, de‑dup, and data minimization.

  2. GLM Assistant

    The user wants to build a "conversion API governance" system. Key requirements:

    1. Endpoint SLOs (Service Level Objectives): Defining performance targets.
    2. Retries: Handling failures robustly.
    3. De-duplication (De-dup): Preventing duplicate processing of events.
    4. Data Minimization: Sending only necessary data.

    This sounds like a system design or architectural guide for a backend service (likely related to marketing/conversion tracking like Facebook/Meta CAPI, Google Ads API, etc.).

    Target Audience: Backend engineers, architects, or technical product managers setting up a Conversion API (CAPI) infrastructure.

    Scope:

    • Conceptual: Explain the principles.
    • Technical: Provide code patterns or architectural components.
    • Specifics: Address the four specific points requested.

    Step-by-Step Breakdown:

    1. Introduction: Define what Conversion API Governance is. It's about reliability, privacy, and efficiency when sending server-side events to ad platforms.

    2. Component 1: Endpoint SLOs (Service Level Objectives)

      • Why: Ad platforms have rate limits and strict latency requirements. If you are too slow, you lose attribution.
      • Metrics: Latency (p95/p99), Availability (99.9%), Error Rate (<0.1%).
      • Implementation:
        • Circuit Breakers (stop hitting a failing API).
        • Rate Limiting (token bucket).
        • Timeout configuration (don't wait forever).
      • Governance aspect: Monitoring and alerting when SLOs are breached.
    3. Component 2: Retries & Backoff

      • Why: Networks fail. APIs glitch. We need eventual consistency.
      • Strategy: Exponential Backoff with Jitter (to avoid thundering herd).
      • Idempotency: Crucial here. If a retry actually goes through twice, did we charge the user twice? (Though CAPI is usually just events, deduplication handles the logic, but the retry request needs to look the same).
      • Implementation: Queues (Kafka/SQS) are better than inline retries.
    4. Component 3: De-duplication (De-dup)

      • Why: Browsers send events (pixels) AND servers send events (CAPI). We need to count the conversion once.
      • Mechanism: Hashing PII (email/phone) + Event ID (transaction ID).
      • The "FBC" vs. "FBP" cookie problem: Handling the linkage between browser and server.
      • Storage: Redis (Bloom filters or TTL-based keys) to check if an event was recently seen.
    5. Component 4: Data Minimization (Privacy by Design)

      • Why: GDPR/CCPA compliance and reducing risk. Don't send what isn't needed.
      • Mechanism:
        • Hashing (SHA256) before sending.
        • Field Dropping (only send required parameters for the specific ad platform).
        • PII masking/redaction.
      • Schema Validation: Reject events with forbidden fields at the ingress point.
    6. Architecture Diagram (Mental Model):

      • Client (App/Web) -> Load Balancer -> API Gateway (Validation/Schema) -> Queue (Buffer) -> Worker (Transformation/Hashing) -> Dedup Store (Redis) -> Ad Platform API.
    7. Code/Structure Example: I should provide a Python/Node pseudo-code structure to show how these fit together.

    8. Review against constraints:

      • SLOs covered? Yes (Timeouts/Circuit Breakers).
      • Retries covered? Yes (Backoff/Queues).
      • De-dup covered? Yes (Event ID hashing).
      • Data Minimization covered? Yes (Schema enforcement/Hashing).

    Refining the "Governance" angle: It's not just building the API, it's governing it. This implies rules and policy enforcement.

    • Policy: "No raw PII leaves the database."
    • Policy: "Duplicate events must be dropped within 48 hours."
    • Policy: "Latency must be < 200ms."

    I will structure the response as a comprehensive framework: "The CAPI Governance Framework."

    Drafting the Content:

    • SLOs: Focus on Circuit Breakers to protect the SLO. If the downstream API (Meta/Google) is down, don't waste resources retrying immediately.
    • Retries: Emphasize asynchronous processing. The client shouldn't wait for the ad platform to respond. Use a message queue.
    • De-dup: The standard is event_name + event_id + user_data. I need to explain the difference between deduping parallel streams (Pixel vs Server) and retries.
    • Data Minimization: This is about the "Transformation Layer." Raw data in -> Clean/Minimized data out.

    Self-Correction/Refinement during drafting:

    • Initial thought: Just show code.

    • Correction: Governance is high-level + implementation. I should structure it by "Governance Domain" -> "Implementation Strategy" -> "Code Pattern".

    • Specific Detail: For De-dup, mention that the Event ID is the key. If the browser pixel sends event_id=123 and server sends event_id=123, the ad platform dedupes it. But the server needs to dedupe its own retries too. Actually, the ad platform usually handles the dedup of Pixel vs Server if you send the event_id. The server-side governance needs to prevent the server from sending the same event twice due to internal logic bugs. Wait, standard CAPI deduplication (Pixel vs Server) relies on passing the same event_id. I must clarify that.

    • Data Minimization: Hashing is key. SHA256 is the standard. Normalization (lowercase, trim) is mandatory before hashing.

    Final Structure:

    1. Architecture Overview: The "Middleware" pattern.
    2. Domain 1: Reliability (SLOs & Retries): Queues, Circuit Breakers, Exponential Backoff.
    3. Domain 2: Data Integrity (De-duplication): Bloom filters, Event IDs, Event Matching.
    4. Domain 3: Privacy & Compliance (Data Minimization): Normalization, Hashing, Schema allow-lists.
    5. Implementation Example (Python/Pseudo-code): Tying it together.

    Let's build this out. Building a Governance framework for a Conversion API (CAPI) requires moving beyond "just sending events" to treating your event pipeline as a mission-critical data product. Ad platforms (like Meta, Google, TikTok) penalize accounts with poor data quality or spammy patterns, and regulations (GDPR/CCPA) penalize data leaks.

    Here is a comprehensive architectural guide and implementation pattern for CAPI Governance, covering Endpoint SLOs, Retries, De-duplication, and Data Minimization.


    1. System Architecture Overview

    Do not send events directly from your web server to the Ad Platform. Use an intermediate Event Gateway (or Message Queue).

    • Flow: Client/App -> Ingestion API -> Message Queue (Kafka/SQS) -> Processing Worker -> Ad Platform API
    • Benefit: This decouples user latency from ad platform latency and allows for robust de-duplication and retry logic.

    2. Governance Domain 1: Endpoint SLOs (Service Level Objectives)

    Ad platforms are notorious for latency spikes and rate limits. Your governance policy must define thresholds for how you interact with them to prevent your own system from clogging.

    The Policies

    1. Latency Target: P95 latency for the HTTP POST to the partner must be < 500ms.
    2. Error Budget: < 0.1% 5xx errors from the partner over a rolling 24-hour window.
    3. Throughput Limits: Respect partner rate limits (e.g., Meta's 100k calls/hour) strictly.

    Implementation: Circuit Breakers & Timeouts

    Use the Circuit Breaker Pattern. If the Ad Platform starts failing, stop sending requests immediately to allow them to recover, queueing local data instead.

    import pybreaker
    import requests
    from datetime import timedelta
    
    # Define the SLO enforcement via Circuit Breaker
    # Open circuit if 5 failures occur within 60 seconds
    ad_platform_breaker = pybreaker.CircuitBreaker(
        fail_max=5,
        reset_timeout=timedelta(seconds=60)
    )
    
    @ad_platform_breaker
    def send_to_partner(event_payload):
        # Enforce strict timeout to prevent thread blocking
        # Connect: 200ms, Read: 500ms
        response = requests.post(
            "https://graph.facebook.com/vXX.X/.../events", 
            json=event_payload, 
            timeout=(0.2, 0.5)
        )
        response.raise_for_status()
        return response
    

    3. Governance Domain 2: Intelligent Retries

    Blindly retrying API calls causes "thundering herd" problems and can get your API access revoked.

    The Policies

    1. Idempotency: All retries must be idempotent (safe to repeat).
    2. Backoff Strategy: Exponential Jitter Backoff.
    3. Retry Eligibility: Only retry 5xx (Server Errors) or 429 (Rate Limit). Never retry 400 (Bad Request) – fix the data instead.

    Implementation: Exponential Backoff with Jitter

    "Jitter" (randomness) prevents all your servers from retrying at the exact same second after an outage.

    import time
    import random
    
    def dispatch_with_retry(payload, max_retries=3):
        attempt = 0
        while attempt < max_retries:
            try:
                return send_to_partner(payload)
            except pybreaker.CircuitBreakerError:
                # Queue for later processing (e.g., write to DB)
                save_to_dead_letter_queue(payload)
                return
            except requests.exceptions.RequestException as e:
                if e.response.status_code == 400:
                    # Governance Alert: Bad data detected, do not retry
                    log_data_quality_error(payload, e)
                    return
                
                # Calculate wait time: Exponential + Jitter
                # e.g., 2^1 + random(0,1) = 2.x seconds
                wait_time = (2 ** attempt) + random.uniform(0, 1)
                time.sleep(wait_time)
                attempt += 1
                
        # If max retries reached
        save_to_dead_letter_queue(payload)
    

    4. Governance Domain 3: De-duplication (De-dup)

    Duplicate events artificially inflate performance metrics, leading to incorrect ad spend optimization. Governance here acts at two levels: Browser vs. Server collisions and Server-Side retries.

    The Policies

    1. Event ID Standardization: Every event MUST have a unique event_id (e.g., UUID).
    2. Dedup Window: Check for existence of this ID within the last 48 hours.

    Implementation: The Dedup Store (Redis)

    Before sending an event, check if the hash exists. If it does, discard it.

    import redis
    import hashlib
    
    r = redis.Redis(host='localhost', port=6379, db=0)
    
    def is_duplicate(event_id, user_data):
        # Create a unique signature combining EventID + User (to prevent ID collision between users)
        # SHA256 is fast and sufficient for short-lived dedup keys
        signature = hashlib.sha256(f"{event_id}:{user_data}".encode()).hexdigest()
        
        key = f"dedup:{signature}"
        
        # SETNX returns True if the key was NOT set (meaning it's new)
        # We set a TTL of 48 hours (172800 seconds)
        if r.setnx(key, 1):
            r.expire(key, 172800)
            return False  # Not a duplicate
        else:
            return True   # Duplicate found
    
    def process_event(payload):
        if is_duplicate(payload['event_id'], payload['user_data']):
            metrics.increment("governance.duplicates_dropped")
            return
        
        dispatch_with_retry(payload)
    

    5. Governance Domain 4: Data Minimization

    This is the most critical governance aspect for privacy compliance. Ad platforms request many fields; you should only send what is contractually and legally necessary.

    The Policies

    1. PII Normalization: All PII (Personally Identifiable Information) must be hashed (SHA256) and lowercased before leaving your trust boundary.
    2. Schema Allow-listing: Drop any fields that are not explicitly required by the marketing analytics use case (e.g., do not send "medical_history" even if the frontend collects it).
    3. Consent Enforcement: Do not process events if the consent_signal is missing or false.

    Implementation: The Sanitization Layer

    Use a transformation function that acts as a firewall.

    import hashlib
    
    # Governance Schema: Only these fields are allowed
    ALLOWED_FIELDS = ['em', 'ph', 'fn', 'ln', 'ct', 'st', 'zp', 'country']
    REQUIRED_SOURCES = ['email', 'phone']
    
    def normalize_and_hash(value):
        if not value: return ""
        # Lowercase, trim, remove spaces
        clean_val = value.strip().lower()
        return hashlib.sha256(clean_val.encode('utf-8')).hexdigest()
    
    def minimize_and_transform(raw_event):
        # 1. Consent Check (Hard Stop)
        if not raw_event.get('gdpr_consent') and not raw_event.get('ccpa_consent'):
            raise ValueError("Governance Violation: Missing Consent")
    
        # 2. Minimize Payload
        clean_payload = {
            "event_name": raw_event['event_name'],
            "event_time": raw_event['event_time'],
            "event_id": raw_event['event_id'],
            "user_data": {},
            "custom_data": {}
        }
    
        # 3. Hash and Move PII
        # Map internal fields to Partner keys (e.g., 'email' -> 'em')
        if raw_event.get('email'):
            clean_payload['user_data']['em'] = normalize_and_hash(raw_event['email'])
        
        if raw_event.get('phone'):
            clean_payload['user_data']['ph'] = normalize_and_hash(raw_event['phone'])
    
        # 4. Strip non-allowed custom data
        # Only keep custom data relevant to ROAS (Return on Ad Spend), e.g., value, currency, content_ids
        for k in ['value', 'currency', 'content_ids']:
            if raw_event.get(k):
                clean_payload['custom_data'][k] = raw_event[k]
    
        return clean_payload
    

    Summary: The Governance Pipeline

    To operationalize this, combine the four domains into a single processing pipeline:

    1. Ingest: Receive raw event.
    2. Minimize: Strip unnecessary fields and Hash PII (Domain 4).
    3. Deduplicate: Check Redis for event_id (Domain 3).
    4. Dispatch:
      • Check Circuit Breaker status (Domain 1).
      • Send with Timeout enforcement (Domain 1).
      • Handle failure with Exponential Backoff (Domain 2).

    This architecture ensures your Conversion API is reliable, privacy-compliant, and trusted by both your organization and the ad platforms.