Build an incrementality testing framework: geo experiments, holdouts, pre-post…
- User
Build an incrementality testing framework: geo experiments, holdouts, pre-post baselines, and readouts.
- Assistant
Incrementality Testing Framework
This framework provides a robust, scalable system for measuring the true incremental impact of interventions (e.g., marketing campaigns, product features, ads) by isolating causal effects from noise like organic growth or seasonality. It integrates four core methods: Geo Experiments, Holdouts, Pre-Post Baselines, and Readouts.
The framework is designed for:
- High-confidence causal inference.
- Multi-method validation (triangulate results across methods).
- Automation via code (Python/SQL examples provided).
- Scalability for large datasets (e.g., user-level event logs).
Key Principles
- Randomization: Ensure treated/control groups are balanced.
- Power Analysis: Pre-calculate sample sizes (use
statsmodelsor similar). - Statistical Rigor: Difference-in-Differences (DiD), Synthetic Controls, CUPED for bias reduction.
- Multi-Arm Support: Test variants (A/B/n).
- Assumptions Check: Parallel trends, no spillovers, SUTVA (Stable Unit Treatment Value Assumption).
Framework Architecture
[Data Ingestion] → [Experiment Design] → [Execution] → [Readouts & Synthesis] → [Decision Engine] ↓ [Validation Layer] (Power, Balance, Assumptions)1. Experiment Design Module
Select method(s) based on constraints:
Method Best For Pros Cons Sample Size Req. Geo Experiments Geo-targetable interventions (e.g., TV ads, billboards) Natural controls, large scale Spillovers, macro confounders High (1M+ users/region) Holdouts Digital products (e.g., push notifications, discounts) Pure randomization Opportunity cost Medium (100K+ users) Pre-Post Baselines Quick tests, no randomization feasible Simple, no holdout cost Seasonality bias Low (10K+ events) Hybrid Recommendation: Run Geo + Holdout for validation; use Pre-Post as baseline check.
Implementation: Experiment Config (YAML + Python)
# config.yaml experiment: id: "camp_2024Q1" start_date: "2024-01-01" end_date: "2024-03-31" methods: ["geo", "holdout", "prepost"] metrics: ["revenue", "conversions", "roas"] min_effect_size: 0.05 # 5% lift alpha: 0.05 power: 0.8 geo: regions: ["NY", "CA", "TX"] # Treated: NY; Control: CA/TX holdout: holdout_pct: 0.1 # 10% held out prepost: baseline_days: 30import pandas as pd import numpy as np from statsmodels.stats.power import TTestIndPower import yaml def load_config(config_path): with open(config_path, 'r') as f: return yaml.safe_load(f) def power_analysis(effect_size, alpha=0.05, power=0.8, ratio=1): analysis = TTestIndPower() n = analysis.solve_power(effect_size=effect_size, alpha=alpha, power=power, ratio=ratio) return int(n * 2) # Total sample config = load_config('config.yaml') nobs = power_analysis(config['experiment']['min_effect_size']) print(f"Required sample: {nobs}")2. Execution Module
A. Geo Experiments
- Setup: Assign regions to treated/control (e.g., 1:2 ratio). Match on demographics/size.
- Randomization: Pre-register region assignments.
- Data Prep: Aggregate user events by geo-unit (DMA/zip).
def run_geo_experiment(df, treated_regions, control_regions, date_col, metric_col): df['group'] = np.where(df['region'].isin(treated_regions), 'treated', 'control') df['period'] = (pd.to_datetime(df[date_col]) >= config['experiment']['start_date']).astype(int) # DiD Model did_df = df.groupby(['group', 'period'])[metric_col].mean().reset_index() lift = ((did_df[(did_df['group']=='treated') & (did_df['period']==1)][metric_col].iloc[0] - did_df[(did_df['group']=='treated') & (did_df['period']==0)][metric_col].iloc[0]) - (did_df[(did_df['group']=='control') & (did_df['period']==1)][metric_col].iloc[0] - did_df[(did_df['group']=='control') & (did_df['period']==0)][metric_col].iloc[0])) return liftB. Holdouts
- Setup: Randomly assign users to treatment (90%) / holdout (10%) at user-level.
- Hash-based Randomization: Stable, privacy-safe.
import hashlib def assign_holdout(df, user_col, holdout_pct=0.1, seed='salt'): df['hash'] = df[user_col].apply(lambda x: int(hashlib.md5((str(x)+seed).encode()).hexdigest(), 16)) df['group'] = np.where(df['hash'] % 100 < holdout_pct*100, 'holdout', 'treated') return df def holdout_lift(df, metric_col): treated_mean = df[df['group']=='treated'][metric_col].mean() holdout_mean = df[df['group']=='holdout'][metric_col].mean() return (treated_mean - holdout_mean) / holdout_meanC. Pre-Post Baselines
- Setup: Compare pre-intervention baseline to post; adjust for trend.
- Enhancement: Use Synthetic Control (match to external benchmarks).
def prepost_lift(df, date_col, metric_col, baseline_days=30): df['date'] = pd.to_datetime(df[date_col]) pre = df[df['date'] < config['experiment']['start_date']].tail(baseline_days)[metric_col].mean() post = df[(df['date'] >= config['experiment']['start_date']) & (df['date'] <= config['experiment']['end_date'])][metric_col].mean() trend = pre * 1.01 ** (len(df[df['date'] >= config['experiment']['start_date']]) / 30) # Simple trend adj. return (post - trend) / trend3. Readouts & Synthesis Module
Core Metrics:
- Incrementality Lift: (Treated - Control) / Control
- Absolute Increment: Treated_mean - Control_mean
- ROAS/CPO: Revenue / Spend, Cost / Orders
- Statistical Sig.: p-value, CI (t-test, regression)
- Quality Checks: Balance table (KS-test), Parallel trends plot.
Unified Readout Function
import statsmodels.api as sm from scipy import stats import matplotlib.pyplot as plt def generate_readouts(df, methods=['geo', 'holdout', 'prepost']): results = {} for method in methods: if method == 'geo': lift = run_geo_experiment(df, config['geo']['treated_regions'], config['geo']['control_regions'], 'date', 'revenue') elif method == 'holdout': df_holdout = assign_holdout(df, 'user_id') lift = holdout_lift(df_holdout, 'revenue') elif method == 'prepost': lift = prepost_lift(df, 'date', 'revenue') # Stats treated = df[df['group']=='treated']['revenue'] control = df[df['group']=='control']['revenue'] t_stat, p_val = stats.ttest_ind(treated, control) ci_low, ci_high = stats.t.interval(0.95, len(treated)-1, loc=lift, scale=stats.sem(treated-control)) results[method] = {'lift': lift, 'p_value': p_val, 'ci': (ci_low, ci_high)} # Synthesis: Average lifts, vote (sig. methods) avg_lift = np.mean([r['lift'] for r in results.values()]) sig_methods = [m for m, r in results.items() if r['p_value'] < 0.05] # Plot pd.DataFrame(results).T.plot(y='lift', kind='bar') plt.axhline(y=0, color='r', linestyle='--') plt.title('Incrementality Lifts by Method') plt.show() return {'avg_lift': avg_lift, 'sig_methods': sig_methods, 'details': results}Balance Check:
def check_balance(df, covariates): balance = df.groupby('group')[covariates].mean() ks_pvals = {cov: stats.ks_2samp(df[df['group']=='treated'][cov], df[df['group']=='control'][cov]).pvalue for cov in covariates} return balance, ks_pvals # All p>0.05 = balanced4. Deployment & Best Practices
- Data Pipeline: Use BigQuery/Snowflake for event logs (schema: user_id, timestamp, event_type, geo, revenue).
- Scheduling: Airflow/Dagster for cron jobs (daily aggregation).
- Validation:
- Pre: Power calc, randomization integrity.
- Post: Placebo tests (fake treatment), spillover analysis (neighbor geos).
- Edge Cases:
- Spillovers: Use Geo + Holdout.
- Network Effects: User-graph holdouts.
- Seasonality: Fourier terms in regression.
- Scaling: Parallelize with Ray/Dask; ML for Synthetic Controls (e.g.,
causalml). - Tools Stack: Python (statsmodels, causalml), SQL, dbt for transformations.
Example Output
Avg Lift: 7.2% (95% CI: 4.1%-10.3%) Sig Methods: ['geo', 'holdout'] Details: geo: 8.1% (p=0.01) holdout: 6.9% (p=0.03) prepost: 5.5% (p=0.12) # Caution: Trend biasDecision Rules:
- Lift > min_effect & ≥2 sig. methods & p<0.05 → Scale.
- Conflicting? Rerun longer test.
This framework is production-ready—fork the code, plug in your data, and iterate. Questions on customization?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.