Make me ML pipeline for 3 stage stacking for regression task, make diverse models,…
- User
Make me ML pipeline for 3 stage stacking for regression task, make diverse models, engineer features and
Assistant
Below is a Python implementation of a 3-stage stacking pipeline for a regression task using diverse models, feature engineering, and scikit-learn's
PipelineandFeatureUnion. The pipeline includes:- Feature Engineering: Custom transformers for feature scaling, polynomial features, and interaction terms.
- Base Models (Level 1): Diverse models like Linear Regression, Random Forest, and Gradient Boosting.
- Meta Model (Level 2): A ridge regressor to combine predictions from base models.
- Final Meta Model (Level 3): Another ridge regressor to combine Level 2 predictions (optional, but included for 3-stage stacking).
import numpy as np import pandas as pd from sklearn.base import BaseEstimator, TransformerMixin from sklearn.linear_model import LinearRegression, Ridge from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor from sklearn.preprocessing import StandardScaler, PolynomialFeatures from sklearn.pipeline import Pipeline, FeatureUnion from sklearn.model_selection import train_test_split from sklearn.metrics import mean_squared_error # Custom Feature Engineering Transformers class InteractionTransformer(BaseEstimator, TransformerMixin): def __init__(self): pass def fit(self, X, y=None): return self def transform(self, X): X_new = X.copy() for i in range(X.shape[1]): for j in range(i + 1, X.shape[1]): col_name = f"interaction_{i}_{j}" X_new[col_name] = X.iloc[:, i] * X.iloc[:, j] return X_new class FeatureEngineer(BaseEstimator, TransformerMixin): def __init__(self): self.scaler = StandardScaler() self.poly = PolynomialFeatures(degree=2) self.interaction = InteractionTransformer() def fit(self, X, y=None): self.scaler.fit(X) self.poly.fit(self.scaler.transform(X)) self.interaction.fit(X) return self def transform(self, X): scaled = self.scaler.transform(X) poly = self.poly.transform(scaled) interactions = self.interaction.transform(X) combined = np.hstack([poly, interactions]) return pd.DataFrame(combined) # Stacking Pipeline class StackingRegressor(BaseEstimator, TransformerMixin): def __init__(self, models): self.models = models def fit(self, X, y): self.model_preds = [] for name, model in self.models.items(): model.fit(X, y) self.model_preds.append((name, model)) return self def predict(self, X): predictions = np.column_stack([model.predict(X) for _, model in self.model_preds]) return predictions # Define the 3-stage stacking pipeline def create_3_stage_stacking_pipeline(): # Level 1 models level1_models = { 'linear': LinearRegression(), 'rf': RandomForestRegressor(n_estimators=100, random_state=42), 'gbm': GradientBoostingRegressor(n_estimators=100, random_state=42) } # Level 2 meta model level2_meta = Ridge(random_state=42) # Level 3 meta model (optional, for 3-stage stacking) level3_meta = Ridge(random_state=42) # Pipeline pipeline = Pipeline([ ('feature_engineering', FeatureEngineer()), ('level1', StackingRegressor(level1_models)), ('level2', Ridge(random_state=42)), ('level3', Ridge(random_state=42)) # Optional, for 3-stage stacking ]) return pipeline # Example usage if __name__ == "__main__": # Generate sample data np.random.seed(42) X = np.random.rand(1000, 10) y = 2 * X[:, 0] + 3 * X[:, 1] + np.random.randn(1000) # Split data X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Create and train pipeline pipeline = create_3_stage_stacking_pipeline() pipeline.fit(X_train, y_train) # Predict and evaluate y_pred = pipeline.predict(X_test) mse = mean_squared_error(y_test, y_pred) print(f"Test MSE: {mse:.4f}")Key Components:
-
Feature Engineering:
StandardScaler: Scales features to have zero mean and unit variance.PolynomialFeatures: Adds polynomial features up to degree 2.InteractionTransformer: Creates interaction terms between features.
-
Level 1 Models:
- Linear Regression
- Random Forest Regressor
- Gradient Boosting Regressor
-
Level 2 Meta Model:
- Ridge Regression to combine Level 1 predictions.
-
Level 3 Meta Model:
- Another Ridge Regression to combine Level 2 predictions (optional, for 3-stage stacking).
Notes:
- The pipeline is flexible and can be modified by adding/removing models or feature engineering steps.
- For 2-stage stacking, remove the
level3step from the pipeline. - Ensure proper hyperparameter tuning for production use.
Let me know if you'd like to expand or modify any part of this pipeline!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.