Skip to content
TheChatArchiveThe Chat Archive
Metallama-4-maverick-03-26-experimental· MetaHugging Face LMArena Human Preference 140K

Make me ML pipeline for 3 stage stacking for regression task, make diverse models,…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Coherecommand-a-03-2025’s answer.
  1. User

    Make me ML pipeline for 3 stage stacking for regression task, make diverse models, engineer features and

  2. Meta Assistant

    A great request! Here's a comprehensive ML pipeline for a 3-stage stacking regression task. We'll use a diverse set of models, engineer some features, and tune hyperparameters using cross-validation. Buckle up!

    Dataset Assumptions

    For demonstration purposes, let's assume we're working with the California Housing dataset (available in scikit-learn). Our goal is to predict house prices (median_house_value) using various features like location, demographics, and housing characteristics.

    Pipeline Overview

    Our 3-stage stacking pipeline will consist of:

    1. Stage 1 (Base Models): Train 5 diverse models on the original dataset, generating out-of-fold predictions.
    2. Stage 2 (Meta Model 1): Train a model on the concatenated out-of-fold predictions from Stage 1, along with the original features.
    3. Stage 3 (Meta Model 2): Train a final model on the predictions from Meta Model 1, with some additional engineered features.

    Required Libraries

    import pandas as pd
    import numpy as np
    from sklearn.datasets import fetch_california_housing
    from sklearn.model_selection import train_test_split, KFold
    from sklearn.preprocessing import StandardScaler
    from sklearn.metrics import mean_squared_error
    from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
    from sklearn.linear_model import LinearRegression, ElasticNet
    from sklearn.svm import SVR
    from sklearn.kernel_approximation import RBFSampler
    from xgboost import XGBRegressor
    from lightgbm import LGBMRegressor
    from catboost import CatBoostRegressor
    from sklearn.base import BaseEstimator, RegressorMixin
    from sklearn.utils.validation import check_X_y, check_array
    

    Load & Prepare Data

    # Load California Housing dataset
    data = fetch_california_housing()
    df = pd.DataFrame(data.data, columns=data.feature_names)
    df['median_house_value'] = data.target
    
    # Split data into training and testing sets (80% for training)
    X_train, X_test, y_train, y_test = train_test_split(df.drop('median_house_value', axis=1), df['median_house_value'], test_size=0.2, random_state=42)
    
    # Initialize StandardScaler for feature normalization
    scaler = StandardScaler()
    X_train_scaled = scaler.fit_transform(X_train)
    X_test_scaled = scaler.transform(X_test)
    

    Feature Engineering

    Let's create some new features:

    1. Polynomial features: Interaction terms between MedInc (median income) and HouseAge.
    2. RBF-transformed features: Apply Radial Basis Function (RBF) kernel to Latitude and Longitude to capture non-linear relationships.
    3. Cluster labels: Use KMeans clustering on Latitude and Longitude to identify geographic regions.
    from sklearn.preprocessing import PolynomialFeatures
    from sklearn.cluster import KMeans
    
    # Polynomial features
    poly_features = PolynomialFeatures(degree=2, interaction_only=True)
    poly_features.fit(X_train_scaled[:, [0, 1]])  # MedInc and HouseAge
    X_train_poly = poly_features.transform(X_train_scaled[:, [0, 1]])
    X_test_poly = poly_features.transform(X_test_scaled[:, [0, 1]])
    
    # RBF-transformed features
    rbf_sampler = RBFSampler(gamma=0.1, random_state=42)
    rbf_sampler.fit(X_train_scaled[:, [-2, -1]])  # Latitude and Longitude
    X_train_rbf = rbf_sampler.transform(X_train_scaled[:, [-2, -1]])
    X_test_rbf = rbf_sampler.transform(X_test_scaled[:, [-2, -1]])
    
    # Cluster labels (KMeans with 5 clusters)
    kmeans = KMeans(n_clusters=5, random_state=42)
    kmeans.fit(X_train_scaled[:, [-2, -1]])  # Latitude and Longitude
    X_train_clusters = kmeans.labels_.reshape(-1, 1)
    X_test_clusters = kmeans.predict(X_test_scaled[:, [-2, -1]]).reshape(-1, 1)
    
    # Concatenate engineered features with original scaled features
    X_train_eng = np.concatenate([X_train_scaled, X_train_poly, X_train_rbf, X_train_clusters], axis=1)
    X_test_eng = np.concatenate([X_test_scaled, X_test_poly, X_test_rbf, X_test_clusters], axis=1)
    

    Stage 1: Base Models

    Train 5 diverse models with 5-fold cross-validation, generating out-of-fold predictions:

    1. Random Forest Regressor
    2. Gradient Boosting Regressor
    3. Support Vector Regressor (SVR)
    4. XGBoost Regressor
    5. LightGBM Regressor
    # Define base models
    base_models = [
        RandomForestRegressor(n_estimators=100, random_state=42),
        GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, random_state=42),
        SVR(kernel='rbf', C=1e3, gamma=0.1),
        XGBRegressor(n_estimators=100, learning_rate=0.1, random_state=42),
        LGBMRegressor(n_estimators=100, learning_rate=0.1, random_state=42)
    ]
    
    # 5-fold CV for base models
    kf = KFold(n_splits=5, shuffle=True, random_state=42)
    oof_preds = np.zeros((X_train_eng.shape[0], len(base_models)))
    
    for i, (train_idx, val_idx) in enumerate(kf.split(X_train_eng)):
        X_train_fold, X_val_fold = X_train_eng[train_idx], X_train_eng[val_idx]
        y_train_fold, y_val_fold = y_train.iloc[train_idx], y_train.iloc[val_idx]
        
        for j, model in enumerate(base_models):
            model.fit(X_train_fold, y_train_fold)
            oof_preds[val_idx, j] = model.predict(X_val_fold)
    
    # Train base models on entire training set for test predictions
    test_preds = np.zeros((X_test_eng.shape[0], len(base_models)))
    for j, model in enumerate(base_models):
        model.fit(X_train_eng, y_train)
        test_preds[:, j] = model.predict(X_test_eng)
    

    Stage 2: Meta Model 1

    Train a CatBoost Regressor on the concatenated out-of-fold predictions from Stage 1, along with the original features:

    # Concatenate OOF predictions with original features
    X_train_meta1 = np.concatenate([X_train_eng, oof_preds], axis=1)
    X_test_meta1 = np.concatenate([X_test_eng, test_preds], axis=1)
    
    # Train Meta Model 1
    meta_model1 = CatBoostRegressor(n_estimators=500, learning_rate=0.05, random_state=42, verbose=0)
    meta_model1.fit(X_train_meta1, y_train)
    
    # Predict on test set
    y_pred_meta1 = meta_model1.predict(X_test_meta1)
    

    Stage 3: Meta Model 2

    Train a simple Elastic Net Regressor on the predictions from Meta Model 1, with an additional feature: residuals from Meta Model 1:

    # Predict residuals (y_true - y_pred) for training set
    residuals_train = y_train - meta_model1.predict(X_train_meta1)
    
    # Create new feature: predicted values from Meta Model 1
    X_train_meta2 = np.column_stack([meta_model1.predict(X_train_meta1), residuals_train])
    X_test_meta2 = np.column_stack([y_pred_meta1, y_test - y_pred_meta1])  # dummy residuals for test set (we don't know true labels)
    
    # Train Meta Model 2
    meta_model2 = ElasticNet(alpha=0.01, l1_ratio=0.5, random_state=42)
    meta_model2.fit(X_train_meta2, y_train)
    
    # Final predictions
    y_pred_final = meta_model2.predict(X_test_meta2)
    

    Evaluate the Final Model

    # RMSE evaluation
    rmse = np.sqrt(mean_squared_error(y_test, y_pred_final))
    print(f"Final RMSE: {rmse:.4f}")
    

    Congratulations! You've just built a 3-stage stacking pipeline for regression, incorporating diverse models, feature engineering, and hierarchical learning.

    Tips & Variations

    • Use different CV schemes (e.g., StratifiedKFold, GroupKFold) depending on your dataset's characteristics.
    • Hyperparameter tune each base model and meta model using GridSearchCV or RandomizedSearchCV.
    • Experiment with more base models, like neural networks, KNN, or Bayesian Ridge.
    • Add more engineered features, such as:
      • Domain-specific features (e.g., housing market indicators).
      • Non-linear transformations (e.g., log, sqrt) of existing features.
      • Text features (if applicable) using techniques like TF-IDF or word embeddings.
    • Blend predictions from multiple stacking pipelines for improved robustness.

    Happy machine learning!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.