I have an ensemble of binary classification models whose outputs will be combined to…
- User
I have an ensemble of binary classification models whose outputs will be combined to produce the ensemble prediction. The models' prediction distribution are quite diverse. Assuming the output will be a linear combination of the predictions what are some methods for determining model weights?
- Assistant
Below is a “toolbox” of the most common—and generally most effective—ways to learn the weights of a linear ensemble of binary-class probability predictions
(ŷ = ∑ᵢ wᵢ pᵢ, or after a link function if you prefer). They range from dead-simple heuristics to fully-fledged optimization procedures. Which one to pick depends on how much you can afford in terms of validation data, compute, interpretability and maintenance.
1. Calibration-blind heuristics (fast baselines)
Method How the weight is set Pros Cons Uniform average wᵢ = 1/M No tuning, hard to beat when models have similar skill Ignores differences in skill/correlation Accuracy (or AUC) proportional wᵢ ∝ metricᵢ Exploits performance differences Metrics measured on 0/1 labels ignore calibration & correlation Log-loss inverse wᵢ ∝ 1 / loglossᵢ Uses a proper scoring rule Still ignores correlation between models Sharpe-ratio like w = Σ⁻¹ μ (μ = vector of scores, Σ = covariance of errors) Analogy to portfolio theory, rewards low correlation Needs covariance estimate, may overfit on small validation sets
2. Direct loss minimisation (constrained regression)
You have a validation set with true labels y ∈ {0,1} and model probabilities {pᵢ}.
Learn w by minimising a proper scoring rule, usually with constraints:-
Brier / MSE
w* = argmin_w ∑ (y – Σ wᵢ pᵢ)²
• Quadratic → closed form (ordinary least squares) if no constraints.
• Add ∑wᵢ =1 and wᵢ ≥0 constraints → solve a small quadratic program (QP). -
Log-loss / cross-entropy
w* = argmin_w –∑ [ y log(Σ wᵢ pᵢ) + (1–y) log(1–Σ wᵢ pᵢ) ]
• Convex if you force wᵢ ≥0 and Σwᵢ ≤1.
• Solve with LBFGS or projected gradient. -
AUC-optimised linear Floyd (rare)
• Non-convex, often tackled with RankNet-style surrogates or black-box search.
Most teams running Kaggle-style ensembles just pick (1) or (2) with non-negativity and a “weights sum to 1” constraint. It is fast, differentiable, and produces intuitive weights.
3. Stacking = “meta-model” on top of base predictions
Treat each pᵢ as a feature and fit a logistic regression (or any learner) on a held-out fold:
ŷ = σ(β₀ + Σ βᵢ pᵢ)
• If you drop the intercept and force βᵢ ≥0 you recover the constrained log-loss approach above.
• Leave β₀ free if you want automatic recalibration.
• Add L2 or L1 to avoid over-fitting when M is large.
• K-fold out-of-fold predictions required to avoid leakage.Variants
• Elastic-net → continuous shrinkage and feature selection.
• Non-linear meta-model (GBDT, NN) → can exploit interactions but loses the “linear combination” purity you asked for.
4. Bayesian model averaging & Bayesian stacking
Instead of point estimates, put a prior on w and integrate:
• Dirichlet prior ⇒ posterior mean ŵ gives weights.
• “Stacking” of predictive distributions (Yao et al., 2018): choose w to maximise the leave-one-out log predictive density. Works very well when individual models come from Bayesian pipelines already.Pros: principled uncertainty; Cons: heavier machinery.
5. Correlation-aware or diversity-boosting schemes
If models are numerous and somewhat redundant, you may want to penalise correlation:
min_w L(w) + λ wᵀ Σ w
where Σ is the empirical correlation (or covariance) of prediction errors. Equivalent to a ridge term in the Brier-loss setup. Another practical hack: start from accuracy-proportional weights and iteratively down-weight the most correlated pair.
6. Search / meta-heuristics
When M is small (<10) and you don’t need to guarantee convexity:
• Grid / random search on the simplex.
• Differential evolution or CMA-ES.
• Bayesian optimisation on w (still low-dimensional).Often used in competitions to squeeze the last 0.01 % lift after a good convex solution is found.
7. Post-ensemble calibration
Whatever weighting scheme you pick, the resulting ensemble can be re-calibrated with:
• Platt scaling (sigmoid fit on ensemble output).
• Isotonic regression.
• Temperature scaling (for deep nets).—
Practical recipe (works 90 % of the time)
- Make K-fold out-of-fold predictions for every base model.
- Fit a logistic regression meta-model with L2 penalty, non-negativity, weights constrained to sum to 1 (optional).
- Evaluate on a separate validation or via nested CV.
- If you see over-fitting or unstable weights, add stronger L2 or fall back to equal weights.
- Optionally run a quick simplex Bayesian-opt around those weights for a small extra lift.
Tips & gotchas
• Check that each base model is individually calibrated; bad calibration causes unstable weight estimates.
• Non-negativity often helps interpretability and prevents weird cancelling effects, but occasionally a negative weight can correct a systematically biased model. Use cross-validation to decide.
• When the validation set is small, prefer simpler or regularised schemes; the variance of the weight estimate can kill any benefit.
• If class imbalance is extreme, optimise on a metric that reflects business need (e.g., PR-AUC instead of ROC-AUC).Happy ensembling! 🎛️
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.