Create an attribution sanity check: triangulate with lift tests and MMM.
- User
Create an attribution sanity check: triangulate with lift tests and MMM.
- Assistant
Attribution Sanity Check: Triangulation Framework with Lift Tests and MMM
As Sherlock, the detective of data from Trailblazer Labs, I'll arm you with a rigorous Attribution Sanity Check framework. This triangulates your attribution model's outputs (e.g., channel ROAS, CPA, or conversion credit) against lift tests (causal incrementality) and Marketing Mix Modeling (MMM) (statistical long-term effects). The goal? Detect biases like over-attribution to lower-funnel channels, underestimation of brand/upper-funnel, or model hallucinations.
Triangulation ensures no single method dominates: Attribution is observational (correlational), lift tests are experimental (causal short-term), and MMM is econometric (causal long-term with controls). If they align within ~20-30% tolerance, your attribution is sane. Wild divergences? Time to investigate (e.g., recency bias, cannibalization).
Core Principles
-
Metrics to Triangulate:
Metric Attribution Lift Tests MMM ROAS Modeled credit Incremental revenue / spend Elasticity-derived ROAS CPA Cost per attributed conversion Incremental cost per lift Adstock-optimized CPA Channel Share % credit to channel Lift % vs. control Coefficient % of total Incrementality Modeled lift Direct measure Halo + base effects -
Sanity Thresholds (adjust for business volatility):
Alignment Level ROAS/CPA Variance Action Green (Sane) <20% Trust attribution Yellow (Probe) 20-50% Segment/deeper diag Red (Broken) >50% Halt decisions; rebuild
Step-by-Step Framework to Build & Run the Sanity Check
Step 1: Data Preparation (1-2 weeks)
- Pull Attribution Data: Export from GA4, CAPI, or platforms like Northbeam/Hyros. Get weekly/monthly: spend, impressions, conversions, ROAS/CPA by channel (e.g., Meta, Google Search, TV).
- Lift Test Data: Run/aggregate geo-holdouts, PSA experiments, or A/B tests (e.g., Google Ads Experiments, Meta Lift). Key: Incremental lift % = (Test - Control) / Control.
- Minimum: 1-2 tests per major channel/quarter.
- MMM Data: Fit a Bayesian MMM (e.g., via Lightweight-MMM in Python/R) on 2-3 years of data.
- Inputs: Spend by channel, organic traffic, macro variables (CPI, seasonality), external factors (competitor spend).
- Outputs: ROAS curves, response curves (saturation/adstock).
- Tools: PyMC, Orbit (Uber), or Robyn (Meta).
Pro Tip: Use a unified data warehouse (Snowflake/BQ) with dbt for joins. Timestamp to week-grain.
Step 2: Build the Triangulation Dashboard (Tableau/Looker/Excel Prototype)
Create a single pane of glass:
┌─────────────────────────────────────────────────────────────┐ │ Attribution Sanity Check: Q3 2024 │ ├─────────────────┬──────────┬──────────┬──────────┬──────────┤ │ Channel │ Attrib. │ Lift │ MMM │ Variance │ │ │ ROAS/CPA │ ROAS/CPA │ ROAS/CPA │ (Max) │ ├─────────────────┼──────────┼──────────┼──────────┼──────────┤ │ Google Search │ 4.2 / 45 │ 3.8 / 52 │ 4.5 / 40 │ 15% 🟢 │ │ Meta Ads │ 2.1 / 90 │ 1.5 / 120│ 2.3 / 85 │ 44% 🟡 │ │ TV/Brand │ 1.2 / 150│ N/A │ 1.8 / 110│ 50% 🔴 │ │ Email │ 8.5 / 20 │ 7.9 / 22 │ 9.1 / 18 │ 14% 🟢 │ └─────────────────┴──────────┴──────────┴──────────┴──────────┘ Overall Sanity Score: 78% 🟢 (Weighted Avg Variance)- Formulas:
- Variance = Max(|Attrib - Lift|, |Attrib - MMM|, |Lift - MMM|) / Avg of three.
- Weight by spend volume.
- Visuals: Spider charts for channel shares; waterfall for divergences.
Step 3: Run the Check (Monthly Cadence)
-
Compute Baselines:
- Attribution: Run data-driven model (Shapley/Markov).
- Lift: Average last 3-6 tests per channel.
- MMM: Refit quarterly with new data.
-
Triangulate:
Test How to Run Expected Insight ROAS Ladder Plot ROAS bars side-by-side Upper-funnel (TV) often underrated in attribution Incrementality Ratio Lift % / Attribution lift % >100%? Attribution conservative; <80%? Over-credits Spend Elasticity MMM curve vs. attribution scaling Saturation mismatch flags over-spend Cross-Validation Holdout 20% data; predict with each method MSE < threshold -
Diagnose Red Flags:
Divergence Pattern Likely Cause Fix Attribution > Lift/MMM Privacy thresholds, recency bias Switch to CAPI; add time-decay Lift > Attribution Short-term burst Blend with MMM adstock MMM > Both Unmodeled halo Add carryover variables All low External shock (economy) Add macros to MMM
Step 4: Automate & Scale
- Code Snippet (Python Prototype):
import pandas as pd import numpy as np # Sample data df = pd.DataFrame({ 'channel': ['Google', 'Meta'], 'attrib_roas': [4.2, 2.1], 'lift_roas': [3.8, 1.5], 'mmm_roas': [4.5, 2.3] }) # Variance calc df['max_var'] = np.maximum.reduce([ abs(df['attrib_roas'] - df['lift_roas']) / ((df['attrib_roas'] + df['lift_roas'])/2), abs(df['attrib_roas'] - df['mmm_roas']) / ((df['attrib_roas'] + df['mmm_roas'])/2), abs(df['lift_roas'] - df['mmm_roas']) / ((df['lift_roas'] + df['mmm_roas'])/2) ]) * 100 df['status'] = np.where(df['max_var'] < 20, '🟢', np.where(df['max_var'] < 50, '🟡', '🔴')) print(df)- Tools Stack: dbt + Airflow for ETL; Streamlit/Plotly for dashboard; GitHub for versioned MMM.
Example Output: Real-World Sanity
- Pre-Check: Attribution says Meta ROAS=3.0. You're scaling budget.
- Post-Triangulate: Lift=1.8, MMM=2.5 → 40% variance 🟡 → Probe: Test shows high cannibalization from organic. Action: Cap Meta at 70% attribution ROAS.
This framework catches 80% of attribution pitfalls. Run it before every budget pivot. Need custom code, MMM fit, or a deeper case? The game is afoot—provide your data schema! 🔍
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.