Skip to content
TheChatArchiveThe Chat Archive
Coherecommand-a-03-2025· CohereHugging Face LMArena Human Preference 140K

Make me ML pipeline for 3 stage stacking for regression task, make diverse models,…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Metallama-4-maverick-03-26-experimental’s answer.
  1. User

    Make me ML pipeline for 3 stage stacking for regression task, make diverse models, engineer features and

  2. Cohere Assistant

    Below is a Python implementation of a 3-stage stacking pipeline for a regression task using diverse models, feature engineering, and scikit-learn's Pipeline and FeatureUnion. The pipeline includes:

    1. Feature Engineering: Custom transformers for feature scaling, polynomial features, and interaction terms.
    2. Base Models (Level 1): Diverse models like Linear Regression, Random Forest, and Gradient Boosting.
    3. Meta Model (Level 2): A ridge regressor to combine predictions from base models.
    4. Final Meta Model (Level 3): Another ridge regressor to combine Level 2 predictions (optional, but included for 3-stage stacking).
    import numpy as np
    import pandas as pd
    from sklearn.base import BaseEstimator, TransformerMixin
    from sklearn.linear_model import LinearRegression, Ridge
    from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
    from sklearn.preprocessing import StandardScaler, PolynomialFeatures
    from sklearn.pipeline import Pipeline, FeatureUnion
    from sklearn.model_selection import train_test_split
    from sklearn.metrics import mean_squared_error
    
    # Custom Feature Engineering Transformers
    class InteractionTransformer(BaseEstimator, TransformerMixin):
        def __init__(self):
            pass
    
        def fit(self, X, y=None):
            return self
    
        def transform(self, X):
            X_new = X.copy()
            for i in range(X.shape[1]):
                for j in range(i + 1, X.shape[1]):
                    col_name = f"interaction_{i}_{j}"
                    X_new[col_name] = X.iloc[:, i] * X.iloc[:, j]
            return X_new
    
    class FeatureEngineer(BaseEstimator, TransformerMixin):
        def __init__(self):
            self.scaler = StandardScaler()
            self.poly = PolynomialFeatures(degree=2)
            self.interaction = InteractionTransformer()
    
        def fit(self, X, y=None):
            self.scaler.fit(X)
            self.poly.fit(self.scaler.transform(X))
            self.interaction.fit(X)
            return self
    
        def transform(self, X):
            scaled = self.scaler.transform(X)
            poly = self.poly.transform(scaled)
            interactions = self.interaction.transform(X)
            combined = np.hstack([poly, interactions])
            return pd.DataFrame(combined)
    
    # Stacking Pipeline
    class StackingRegressor(BaseEstimator, TransformerMixin):
        def __init__(self, models):
            self.models = models
    
        def fit(self, X, y):
            self.model_preds = []
            for name, model in self.models.items():
                model.fit(X, y)
                self.model_preds.append((name, model))
            return self
    
        def predict(self, X):
            predictions = np.column_stack([model.predict(X) for _, model in self.model_preds])
            return predictions
    
    # Define the 3-stage stacking pipeline
    def create_3_stage_stacking_pipeline():
        # Level 1 models
        level1_models = {
            'linear': LinearRegression(),
            'rf': RandomForestRegressor(n_estimators=100, random_state=42),
            'gbm': GradientBoostingRegressor(n_estimators=100, random_state=42)
        }
    
        # Level 2 meta model
        level2_meta = Ridge(random_state=42)
    
        # Level 3 meta model (optional, for 3-stage stacking)
        level3_meta = Ridge(random_state=42)
    
        # Pipeline
        pipeline = Pipeline([
            ('feature_engineering', FeatureEngineer()),
            ('level1', StackingRegressor(level1_models)),
            ('level2', Ridge(random_state=42)),
            ('level3', Ridge(random_state=42))  # Optional, for 3-stage stacking
        ])
    
        return pipeline
    
    # Example usage
    if __name__ == "__main__":
        # Generate sample data
        np.random.seed(42)
        X = np.random.rand(1000, 10)
        y = 2 * X[:, 0] + 3 * X[:, 1] + np.random.randn(1000)
    
        # Split data
        X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
    
        # Create and train pipeline
        pipeline = create_3_stage_stacking_pipeline()
        pipeline.fit(X_train, y_train)
    
        # Predict and evaluate
        y_pred = pipeline.predict(X_test)
        mse = mean_squared_error(y_test, y_pred)
        print(f"Test MSE: {mse:.4f}")
    

    Key Components:

    1. Feature Engineering:

      • StandardScaler: Scales features to have zero mean and unit variance.
      • PolynomialFeatures: Adds polynomial features up to degree 2.
      • InteractionTransformer: Creates interaction terms between features.
    2. Level 1 Models:

      • Linear Regression
      • Random Forest Regressor
      • Gradient Boosting Regressor
    3. Level 2 Meta Model:

      • Ridge Regression to combine Level 1 predictions.
    4. Level 3 Meta Model:

      • Another Ridge Regression to combine Level 2 predictions (optional, for 3-stage stacking).

    Notes:

    • The pipeline is flexible and can be modified by adding/removing models or feature engineering steps.
    • For 2-stage stacking, remove the level3 step from the pipeline.
    • Ensure proper hyperparameter tuning for production use.

    Let me know if you'd like to expand or modify any part of this pipeline!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.