Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

I have an ensemble of binary classification models whose outputs will be combined to…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwen3-30b-a3b’s answer.
  1. User

    I have an ensemble of binary classification models whose outputs will be combined to produce the ensemble prediction. The models' prediction distribution are quite diverse. Assuming the output will be a linear combination of the predictions what are some methods for determining model weights?

  2. Assistant

    Below is a “toolbox” of the most common—and generally most effective—ways to learn the weights of a linear ensemble of binary-class probability predictions
    (ŷ = ∑ᵢ wᵢ pᵢ, or after a link function if you prefer). They range from dead-simple heuristics to fully-fledged optimization procedures. Which one to pick depends on how much you can afford in terms of validation data, compute, interpretability and maintenance.


    1. Calibration-blind heuristics (fast baselines)

    MethodHow the weight is setProsCons
    Uniform averagewᵢ = 1/MNo tuning, hard to beat when models have similar skillIgnores differences in skill/correlation
    Accuracy (or AUC) proportionalwᵢ ∝ metricᵢExploits performance differencesMetrics measured on 0/1 labels ignore calibration & correlation
    Log-loss inversewᵢ ∝ 1 / loglossᵢUses a proper scoring ruleStill ignores correlation between models
    Sharpe-ratio likew = Σ⁻¹ μ (μ = vector of scores, Σ = covariance of errors)Analogy to portfolio theory, rewards low correlationNeeds covariance estimate, may overfit on small validation sets

    2. Direct loss minimisation (constrained regression)

    You have a validation set with true labels y ∈ {0,1} and model probabilities {pᵢ}.
    Learn w by minimising a proper scoring rule, usually with constraints:

    1. Brier / MSE
      w* = argmin_w ∑ (y – Σ wᵢ pᵢ)²
      • Quadratic → closed form (ordinary least squares) if no constraints.
      • Add ∑wᵢ =1 and wᵢ ≥0 constraints → solve a small quadratic program (QP).

    2. Log-loss / cross-entropy
      w* = argmin_w –∑ [ y log(Σ wᵢ pᵢ) + (1–y) log(1–Σ wᵢ pᵢ) ]
      • Convex if you force wᵢ ≥0 and Σwᵢ ≤1.
      • Solve with LBFGS or projected gradient.

    3. AUC-optimised linear Floyd (rare)
      • Non-convex, often tackled with RankNet-style surrogates or black-box search.

    Most teams running Kaggle-style ensembles just pick (1) or (2) with non-negativity and a “weights sum to 1” constraint. It is fast, differentiable, and produces intuitive weights.


    3. Stacking = “meta-model” on top of base predictions

    Treat each pᵢ as a feature and fit a logistic regression (or any learner) on a held-out fold:

    ŷ = σ(β₀ + Σ βᵢ pᵢ)

    • If you drop the intercept and force βᵢ ≥0 you recover the constrained log-loss approach above.
    • Leave β₀ free if you want automatic recalibration.
    • Add L2 or L1 to avoid over-fitting when M is large.
    • K-fold out-of-fold predictions required to avoid leakage.

    Variants
    • Elastic-net → continuous shrinkage and feature selection.
    • Non-linear meta-model (GBDT, NN) → can exploit interactions but loses the “linear combination” purity you asked for.


    4. Bayesian model averaging & Bayesian stacking

    Instead of point estimates, put a prior on w and integrate:

    • Dirichlet prior ⇒ posterior mean ŵ gives weights.
    • “Stacking” of predictive distributions (Yao et al., 2018): choose w to maximise the leave-one-out log predictive density. Works very well when individual models come from Bayesian pipelines already.

    Pros: principled uncertainty; Cons: heavier machinery.


    5. Correlation-aware or diversity-boosting schemes

    If models are numerous and somewhat redundant, you may want to penalise correlation:

    min_w L(w) + λ wᵀ Σ w

    where Σ is the empirical correlation (or covariance) of prediction errors. Equivalent to a ridge term in the Brier-loss setup. Another practical hack: start from accuracy-proportional weights and iteratively down-weight the most correlated pair.


    6. Search / meta-heuristics

    When M is small (<10) and you don’t need to guarantee convexity:

    • Grid / random search on the simplex.
    • Differential evolution or CMA-ES.
    • Bayesian optimisation on w (still low-dimensional).

    Often used in competitions to squeeze the last 0.01 % lift after a good convex solution is found.


    7. Post-ensemble calibration

    Whatever weighting scheme you pick, the resulting ensemble can be re-calibrated with:

    • Platt scaling (sigmoid fit on ensemble output).
    • Isotonic regression.
    • Temperature scaling (for deep nets).

    —

    Practical recipe (works 90 % of the time)

    1. Make K-fold out-of-fold predictions for every base model.
    2. Fit a logistic regression meta-model with L2 penalty, non-negativity, weights constrained to sum to 1 (optional).
    3. Evaluate on a separate validation or via nested CV.
    4. If you see over-fitting or unstable weights, add stronger L2 or fall back to equal weights.
    5. Optionally run a quick simplex Bayesian-opt around those weights for a small extra lift.

    Tips & gotchas

    • Check that each base model is individually calibrated; bad calibration causes unstable weight estimates.
    • Non-negativity often helps interpretability and prevents weird cancelling effects, but occasionally a negative weight can correct a systematically biased model. Use cross-validation to decide.
    • When the validation set is small, prefer simpler or regularised schemes; the variance of the weight estimate can kill any benefit.
    • If class imbalance is extreme, optimise on a metric that reflects business need (e.g., PR-AUC instead of ROC-AUC).

    Happy ensembling! 🎛️

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.