Plan a contextual bandit offline evaluation harness using doubly robust estimators and…
- User
Plan a contextual bandit offline evaluation harness using doubly robust estimators and guardrails.
- Assistant
Okay, the user wants a practical guide for building an offline evaluation system for contextual bandits using doubly robust estimators with safety guardrails. This is a fairly advanced request in the causal inference/reinforcement learning space.
First, I need to confirm the core components they’re asking for: doubly robust (DR) estimation for off-policy evaluation + guardrails to prevent unreliable extrapolation. The user likely works in applied ML (ads, recommendations, or clinical treatment personalization) where offline evaluation is critical before deploying new bandit policies.
The real challenge here isn’t just implementing DR—it’s making the system robust to practical issues like model misspecification, distribution shift, and silent failures. The user might be silently worried about "garbage in, garbage out" scenarios where the estimator gives high variance or biased results without warning.
I should structure this as an end-to-end pipeline: data prep, propensity modeling, outcome modeling, DR computation, and then the guardrails. The guardrails need to be actionable—not just theoretical checks. Things like overlap diagnostics, smoothness constraints, and sanity bounds.
Also, the user might not realize they need logging schema requirements upfront. I should emphasize the logging format (context, action, reward, propensity) because bad logging breaks everything. Maybe add a "minimum viable logging" table to make it concrete.
For the DR formula, I’ll present both the generic form and the specific implementation for multi-actions (since bandits often have >2 arms). Clarify that DR reduces to
\hat{V}_{DR} = \frac{1}{n} \sum \frac{\pi(a|x)}{\mu(a|x)} r + \sum ...but emphasize the importance of the second term being a model-based correction.Guardrails should tiered: (1) pre-execution checks (overlap, model health), (2) runtime safeguards (bounded importance weights), (3) post-hoc validation (sensitivity, counterfactual consistency). The variance diagnostic via bootstrapping is non-negotiable—no one wants to deploy a policy with ±50% CI.
Finally, I’ll stress that this is a stack: no single estimator fixes everything. They’ll need a suite (DR, SNW, IS) and to monitor disagreements between them. The tone should be "here’s how to not shoot your foot off" rather than pure theory. Excellent. Building a robust offline evaluation harness for contextual bandits is a critical, non-trivial task. It's the safety net that prevents policy degradation when you can't do live A/B tests. Using Doubly Robust (DR) estimators with guardrails is the current best practice for balancing bias and variance.
Here is a comprehensive, actionable plan for such a harness, structured as an engineering blueprint.
1. Core Philosophy & Goal
- Goal: Estimate the expected value (V) of a new candidate policy
π_newusing only logged historical dataD = {(x_i, a_i, r_i, p_i)}, where:x_i: Context (features).a_i: Action taken by the logging policyπ_log.r_i: Observed reward (can be binary, continuous, etc.).p_i = π_log(a_i | x_i): Propensity score (action probability fromπ_log). This log is mandatory.
- Why DR? Doubly Robust estimators combine:
- Importance Sampling (IS): Rewards are re-weighted by
π_new(a|x) / π_log(a|x). Prone to high variance ifπ_lograrely took actions favored byπ_new. - Direct Method (DM): Train a reward model
μ(x, a)to predictE[r | x, a]. Then estimateV(π_new) = E_x[ E_{a~π_new}[ μ(x, a) ] ]. Prone to bias if the model is misspecified. DR = IS + (DM - IS_correction). It remains consistent if either the propensity model (for IS) or the reward model (for DM) is correct. This is a powerful robustness guarantee.
- Importance Sampling (IS): Rewards are re-weighted by
2. System Architecture & Pipeline
graph TD A[Logged Data D] --> B{Preprocessing & Validation}; B --> C[Split: Train/Val/Test]; C --> D[Propensity Model p(x,a)]; C --> E[Reward Model μ(x,a)]; D --> F[Compute Estimators]; E --> F; F --> G[Guardrail Module]; G --> H{Pass?}; H -->|Yes| I[Report: V_DR ± CI, Diagnostics]; H -->|No| J[Flag: Unreliable/High Risk]; I --> K[Decision: Deploy?]; J --> L[Require New Logging/Modeling];
3. Step-by-Step Implementation Plan
Phase 0: Prerequisite - Data Logging Schema
You must have the following logged for every decision point:
{ "context_id": "...", # Unique ID for x (optional but useful) "context_features": {...}, # All features x used by π_log & π_new "action": a, # Action label/index "reward": r, # Observed outcome "propensity_score": p, # π_log(a|x) **at the time of logging** "action_set": [list_of_all_possible_actions], # Critical for multi-armed bandits "timestamp": "..." }Guardrail 0: Fail immediately if
propensity_scoreis missing or cannot be reconstructed for(x, a).Phase 1: Data Preparation & Splitting
- Temporal Split: Split by timestamp (e.g., train on older 80%, val/test on newer 20%). Prevents leakage.
- Action Coverage Filter (CRITICAL): Remove (or separately analyze) contexts
xin the test set where the logging policy did not explore actions the new policyπ_newmight choose.- Define a minimum threshold
ε(e.g.,π_log(a|x) >= 0.001for allawithπ_new(a|x) > 0). - Guardrail 1 (Overlap/Support Check): Compute for each
xin test set:w_max(x) = max_{a in A} [ π_new(a|x) / π_log(a|x) ]. Flag and exclude contexts wherew_max(x) > W_threshold(e.g., 100 or 1000). High weights = extreme extrapolation = high variance, unreliable.
- Define a minimum threshold
- Reward Transformation: If reward is non-negative, consider transformations (e.g., log for heavy-tailed) to stabilize variance. Document this choice.
Phase 2: Model Training (on Train Set)
- Propensity Model
π_log_hat(a|x)(Optional but recommended for variance reduction):- Train a multi-class classifier (e.g., XGBoost, LightGBM, neural net) to predict
afromx. - Output calibrated probabilities. Use temperature scaling if needed.
- Why? The IS term uses
1/π_log(a|x). A badπ_log_hatused only for variance weighting in some DR variants (like Self-Normalized DR) can help. More importantly, it helps diagnose logging policy coverage.
- Train a multi-class classifier (e.g., XGBoost, LightGBM, neural net) to predict
- Reward Model
μ_hat(x, a):- Train a regression model to predict
rfrom(x, a). For contextual bandits, this is typically a multi-task model or one model with action embeddings:- Option A: One model per action (if actions are few).
- Option B: Single model input
[x, a_one_hot]or[x, a_embedding](for many actions).
- Use a proper scoring rule (e.g., cross-entropy for binary rewards, RMSE for continuous).
- Cross-validate rigorously on the validation set. Monitor calibration.
- Train a regression model to predict
Phase 3: Doubly Robust Estimation (on Test Set)
For each logged tuple
(x_i, a_i, r_i)in the test set:- Get Propensity Score:
p_i = π_log(a_i|x_i)(from log) orπ_log_hat(a_i|x_i). - Compute Importance Weight for
π_new:w_i = π_new(a_i|x_i) / p_i. - Get Reward Prediction from Model:
μ_i = μ_hat(x_i, a_i). - Compute DR per-sample estimate:
dr_i = w_i * (r_i - μ_i) + (1/N) * Σ_{a in A} [π_new(a|x_i) * μ_hat(x_i, a)]- The second term is the model-based policy value estimate for context
x_i. - The first term is the IS residual correction.
- The second term is the model-based policy value estimate for context
- Final Estimate:
V_DR = mean(dr_i)over all test samples. - Variance Estimation:
- Bootstrap: Resample test set with replacement (e.g., 1000 iterations), compute
V_DReach time. Use percentile bootstrap for a robust confidence interval (CI). - Formulas: Can also use influence function-based variance, but bootstrap is simpler and captures complex dependencies.
- Bootstrap: Resample test set with replacement (e.g., 1000 iterations), compute
Phase 4: The Guardrail Module (Critical!)
This is what makes the harness safe. Run these checks sequentially. Any critical failure should invalidate the
V_DRestimate.Guardrail Check Method Threshold/Flag Action G1: Overlap w_max(x)distributionHistogram, 95th percentile p95(w_max) > 100FAIL: Catastrophic extrapolation. G2: Smoothness Reward model extrapolation error For random subset of x, compareμ_hat(x, a)forawith high `π_new(ax) but **low/zero**π_log(ax)`. G3: Model Health Reward model calibration & performance on test set. - Calibration plot (for prob/reals). - R² / RMSE on held-out test (x,a,r).R² < 0.1 or poor calibration FAIL/WARN: Poor DM component. G4: Variance Bootstrap CI width relative to point estimate. `(CI_upper - CI_lower) / V_DR ` G5: Consistency Compare V_DRwith other estimators.Also compute: - Direct Method (DM): `mean( Σ_a π_new(a x)μ_hat(x,a) ) <br> - **Self-Normalized IS (SNIS).Large disagreement (e.g., > 2*CI) between DR, DM, SNIS. G6: Counterfactual Sanity "Off-policy" sanity check on data from π_log.Compute V_DRforπ_new = π_log(should be ~mean(r)).Large absolute error FAIL: Fundamental bug in pipeline.
4. Output & Decision Report
A good harness produces a structured report, not just a number:
{ "candidate_policy_id": "policy_v3", "evaluation_timestamp": "...", "test_set_size": 50000, "estimates": { "V_DR": 0.142, "V_DR_CI_lower": 0.125, "V_DR_CI_upper": 0.159, "V_DM": 0.151, "V_SNIS": 0.138 }, "guardrail_status": { "G1_overlap": "PASS (p95(w_max)=12.4)", "G2_smoothness": "WARN (3 high-weight actions have high model uncertainty)", "G3_model_R2": "PASS (0.35)", "G4_variance_ratio": "PASS (0.15)", "G5_consistency": "WARN (DR vs DM diff = 0.009, within 1*CI)", "G6_sanity": "PASS (π_log self-estimate: 0.117 vs true mean: 0.116)" }, "conclusion": "CAUTIOUS_POSITIVE", "recommendation": "Policy shows statistically significant lift with acceptable guardrail flags. Monitor G2 (smoothness) closely in first live deployment phase. Consider targeted logging for actions with high π_new but low π_log coverage." }Decision Matrix:
- ALL PASS: High confidence. DECOMMISSION old policy if
V_DRis significantly higher. - WARNs only (G2, G5): Medium confidence. PROCEED WITH CAUTION (e.g., canary deployment, tighter monitoring).
- Any FAIL: DO NOT DEPLOY. Revisit logging (increase exploration?), model architecture, or policy design. Run new A/A test to validate harness itself.
5. Advanced Guardrails & Best Practices
- Capped Importance Weights: During estimation, clip
w_iat aW_clip(e.g., 10 or 50). This is a bias-variance tradeoff. Report results for multipleW_clipvalues as a sensitivity analysis. - F-Test: For continuous rewards, perform an F-test on the residuals in the DR correction term
(r_i - μ_i). If they are not mean-zero, it indicates systematic model bias. - Slate Evaluation: If actions are presented as a set (a "slate"), the problem is non-separable. You need a slate-level reward model and slate-level propensity. The DR formula generalizes but is more complex. Guardrail: Ensure the action set
Ain the logging data exactly matches the candidate policy's slate generation logic. - Time-Varying Guardrails: If the world is non-stationary (e.g., user trends change), evaluate the policy only on recent data (e.g., last week). Include a temporal guardrail that checks for significant drift in
π_log's action distribution over time. - A/A Test: Periodically run the harness on data where
π_new = π_log. The estimate should be within noise of the true average reward. This validates the entire pipeline's unbiasedness.
6. Technology Stack Suggestions
- Python Ecosystem:
numpy,pandas,scikit-learn(for models),causalml(has DR implementations),evaluation(OpenAI's library for bandit eval), or customPyTorch/TensorFlowfor complex models. - Workflow Orchestration: Use
Airflow,Prefect, orMetaflowto chain the steps (preprocess -> train -> eval -> guardrails -> report). - Storage: Test sets and diagnostics should be versioned (DVC, MLflow) and stored for auditability.
- Visualization: Build a dashboard (Streamlit, Dash) showing key guardrail metrics (weight histograms, calibration plots, bootstrap distributions) over time.
Summary: The Essence of Safety
The Doubly Robust estimator gives you a single number (
V_DR). The guardrails tell you whether you should trust that number. A harness without guardrails is dangerous; a guardrail without a DR estimator is inefficient. This plan provides a layered defense:- Pre-filter contexts with poor overlap (G1).
- Validate the models that form the DR's two pillars (G3, G2).
- Cross-check with alternative estimators (G5).
- Quantify uncertainty rigorously (G4).
- Sanity-check the entire system (G6).
Final Rule: When in doubt, trust the guardrails over the point estimate. A "fail" on any critical guardrail should trigger a deeper investigation, not a threshold tweak. This is how you build a reliable, trustworthy offline evaluation system for high-stakes contextual bandit applications.
- Goal: Estimate the expected value (V) of a new candidate policy
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.