Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

I have an ensemble based demand forecasting model consisting of XGBoost, Prophet and…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-sonnet-4-20250514’s answer.
  1. User

    I have an ensemble based demand forecasting model consisting of XGBoost, Prophet and ARIMAX. These have temporal variables like trend and seasonality and non-temporal variables such as weather, price, etc. How do I find out contribution of each such independent variable toward the demand forecast?

  2. Assistant

    Below is a practical “menu” of ways you can decompose the final forecast and quantify how much each driver (trend, yearly seasonality, weather, price, …) contributes.
    Pick the level of granularity you need:

    ────────────────────────────────────────

    1. Decide what “contribution” means ──────────────────────────────────────── A. Point-wise contribution
      “For the forecast of week t, how many units are attributed to weather vs. price vs. trend?”

    B. Overall importance
    “If I removed the weather feature, how much would my MAPE worsen?”

    The techniques below give you one or the other (or both).

    ──────────────────────────────────────── 2. Do it model-by-model first ──────────────────────────────────────── 2-A XGBoost
    • Tree SHAP is the de-facto standard for additive explanations of gradient-boosted trees.
    • Code:

    import xgboost as xgb, shap, pandas as pd
    model = xgb.XGBRegressor(...).fit(X_train, y_train)
    explainer = shap.TreeExplainer(model)
    shap_vals = explainer.shap_values(X_test)        # matrix n_obs × n_features
    shap.summary_plot(shap_vals, X_test)             # global importance
    shap.force_plot(explainer.expected_value,
                    shap_vals[0,:], X_test.iloc[0,:])# point forecast
    

    Output: for every timestamp you’ll get φ₀ (bias) + Σφ_j (feature j) = ŷ_t.
    Aggregate |φ_j| or mean φ_j to compare features.

    2-B Prophet
    Prophet’s forecast is additive by design:

     ŷ_t = trend_t + Σ_k seasonality_k,t + Σ_r β_r · regressor_r,t

    Using m.predict(df) you already get separate columns:

    yhat, trend, additive_terms, 
    yearly, weekly, extra_regressor_weather, extra_regressor_price, …
    

    So the per-timestamp contributions are explicitly available.
    For global importance, take mean(|component|) or var(component).

    2-C ARIMAX (or SARIMAX)
    After fitting, you have coefficients β_r.
    Standardise regressors first (zero-mean, unit σ) so the β’s are comparable, then:

     contribution_r,t = β_r · x_r,t

    For “feature importance”: see |β_r| or t-stat.
    For point-wise decomposition: keep the per-timestamp product above.

    ──────────────────────────────────────── 3. Roll the contributions up to the ensemble ──────────────────────────────────────── Suppose the ensemble prediction is a weighted average

     ŷ_ens,t = w₁ ŷ_xgb,t + w₂ ŷ_prophet,t + w₃ ŷ_arimax,t

    (If you use stacking, replace w’s by meta-model outputs.)

    Step 1 – collect per-model contributions C_m,j,t
     e.g. for weather inside XGBoost: C_xgb,weather,t = SHAP_weather,t
      for weather inside Prophet: C_prophet,weather,t = regression column
      for trend in Prophet: C_prophet,trend,t

    Step 2 – weight them

     C_total,j,t = Σ_m w_m · C_m,j,t

    That gives an additive decomposition of the ensemble forecast:

     ŷ_ens,t = bias + Σ_j C_total,j,t

    Implementation snippet:

    import numpy as np
    
    # Assume same index ordering
    C_xgb   = shap_mat           # n_obs × n_features
    C_prop  = prophet_components # DataFrame n_obs × components
    C_arima = arimax_contribs    # DataFrame n_obs × regressors
    
    # Align column names
    C_prop.columns  = ['trend','seasonality_year','seasonality_week',
                       'weather','price']
    C_arima.columns = ['weather','promo']
    
    # Choose weights
    w = {'xgb':0.4, 'prophet':0.35, 'arima':0.25}
    
    # Combine
    C_total = w['xgb']*C_xgb.add_suffix('_xgb')\
            + w['prophet']*C_prop.add_suffix('_prop')\
            + w['arima']*C_arima.add_suffix('_arima')
    
    point_forecast = C_total.sum(axis=1) + bias
    

    Validation: the summed contributions should reconstruct the ensemble’s forecast to machine precision.

    ──────────────────────────────────────── 4. Model-agnostic alternative: Shapley on the whole ensemble ──────────────────────────────────────── If stitching the three models feels fragile, treat the ensemble as a single black-box function f(X).
    Run SHAP KernelExplainer (slow) or Permutation Importance:

    import shap
    f = lambda X: w1*model_xgb.predict(X) + \
                  w2*model_prophet.predict(X)['yhat'] + \
                  w3*model_arima.predict(X)
    explainer = shap.KernelExplainer(f, X_background)
    shap_vals = explainer.shap_values(X_eval, nsamples=200)  # slower
    

    Pros: one explainer covers all interactions, no need to sync column names.
    Cons: Kernel SHAP is expensive and can be noisy; you lose the built-in structure Prophet gave you (trend, seasonality).

    ──────────────────────────────────────── 5. Accuracy-based (“leave-one-out”) importance ──────────────────────────────────────── For each driver (or group of drivers):

    1. Re-fit the full ensemble → error_full
    2. Re-fit dropping that driver everywhere it appears → error_minus_j
    3. ΔMAPE_j = error_minus_j – error_full

    Larger ΔMAPE means the variable is more valuable to the accuracy of the system.
    This is easy to interpret for business users, but expensive because you retrain 3 models × N_drivers.

    ──────────────────────────────────────── 6. Visualisation ideas ──────────────────────────────────────── • Stacked-bar time series: plot C_total,j,t each week to show seasonality vs. weather.
    • Waterfall chart for a single date: start from bias, add positive/negative contributions.
    • SHAP summary beeswarm to show distribution and direction of impact.

    ──────────────────────────────────────── 7. Caveats & tips ────────────────────────────────────────

    1. Correlated drivers (e.g., temperature and promotion timing) will split credit in different ways depending on the method.
    2. Standardise/normalise non-temporal variables before SHAP or coefficient comparison.
    3. Prophet regressors should be centred (add_regressor(..., standardize='auto')) so the intercept is meaningful.
    4. Remember: contribution ≠ causality. SHAP is an attribution, not proof that changing weather will move demand by that amount.

    ──────────────────────────────────────── TL;DR • Use Tree SHAP for XGBoost, Prophet’s built-in component columns, and β·X for ARIMAX to get per-timestamp contributions.
    • Aggregate them with the ensemble weights to obtain the final decomposition, or run Kernel SHAP on the whole ensemble for a model-agnostic route.
    • For global importance, either average absolute contributions |C| or run leave-one-out retrain tests.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.