Make me ML pipeline for 3 stage stacking for regression task, make diverse models,…
- User
Make me ML pipeline for 3 stage stacking for regression task, make diverse models, engineer features and
Assistant
A great request! Here's a comprehensive ML pipeline for a 3-stage stacking regression task. We'll use a diverse set of models, engineer some features, and tune hyperparameters using cross-validation. Buckle up!
Dataset Assumptions
For demonstration purposes, let's assume we're working with the California Housing dataset (available in scikit-learn). Our goal is to predict house prices (
median_house_value) using various features like location, demographics, and housing characteristics.Pipeline Overview
Our 3-stage stacking pipeline will consist of:
- Stage 1 (Base Models): Train 5 diverse models on the original dataset, generating out-of-fold predictions.
- Stage 2 (Meta Model 1): Train a model on the concatenated out-of-fold predictions from Stage 1, along with the original features.
- Stage 3 (Meta Model 2): Train a final model on the predictions from Meta Model 1, with some additional engineered features.
Required Libraries
import pandas as pd import numpy as np from sklearn.datasets import fetch_california_housing from sklearn.model_selection import train_test_split, KFold from sklearn.preprocessing import StandardScaler from sklearn.metrics import mean_squared_error from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor from sklearn.linear_model import LinearRegression, ElasticNet from sklearn.svm import SVR from sklearn.kernel_approximation import RBFSampler from xgboost import XGBRegressor from lightgbm import LGBMRegressor from catboost import CatBoostRegressor from sklearn.base import BaseEstimator, RegressorMixin from sklearn.utils.validation import check_X_y, check_arrayLoad & Prepare Data
# Load California Housing dataset data = fetch_california_housing() df = pd.DataFrame(data.data, columns=data.feature_names) df['median_house_value'] = data.target # Split data into training and testing sets (80% for training) X_train, X_test, y_train, y_test = train_test_split(df.drop('median_house_value', axis=1), df['median_house_value'], test_size=0.2, random_state=42) # Initialize StandardScaler for feature normalization scaler = StandardScaler() X_train_scaled = scaler.fit_transform(X_train) X_test_scaled = scaler.transform(X_test)Feature Engineering
Let's create some new features:
- Polynomial features: Interaction terms between
MedInc(median income) andHouseAge. - RBF-transformed features: Apply Radial Basis Function (RBF) kernel to
LatitudeandLongitudeto capture non-linear relationships. - Cluster labels: Use KMeans clustering on
LatitudeandLongitudeto identify geographic regions.
from sklearn.preprocessing import PolynomialFeatures from sklearn.cluster import KMeans # Polynomial features poly_features = PolynomialFeatures(degree=2, interaction_only=True) poly_features.fit(X_train_scaled[:, [0, 1]]) # MedInc and HouseAge X_train_poly = poly_features.transform(X_train_scaled[:, [0, 1]]) X_test_poly = poly_features.transform(X_test_scaled[:, [0, 1]]) # RBF-transformed features rbf_sampler = RBFSampler(gamma=0.1, random_state=42) rbf_sampler.fit(X_train_scaled[:, [-2, -1]]) # Latitude and Longitude X_train_rbf = rbf_sampler.transform(X_train_scaled[:, [-2, -1]]) X_test_rbf = rbf_sampler.transform(X_test_scaled[:, [-2, -1]]) # Cluster labels (KMeans with 5 clusters) kmeans = KMeans(n_clusters=5, random_state=42) kmeans.fit(X_train_scaled[:, [-2, -1]]) # Latitude and Longitude X_train_clusters = kmeans.labels_.reshape(-1, 1) X_test_clusters = kmeans.predict(X_test_scaled[:, [-2, -1]]).reshape(-1, 1) # Concatenate engineered features with original scaled features X_train_eng = np.concatenate([X_train_scaled, X_train_poly, X_train_rbf, X_train_clusters], axis=1) X_test_eng = np.concatenate([X_test_scaled, X_test_poly, X_test_rbf, X_test_clusters], axis=1)Stage 1: Base Models
Train 5 diverse models with 5-fold cross-validation, generating out-of-fold predictions:
- Random Forest Regressor
- Gradient Boosting Regressor
- Support Vector Regressor (SVR)
- XGBoost Regressor
- LightGBM Regressor
# Define base models base_models = [ RandomForestRegressor(n_estimators=100, random_state=42), GradientBoostingRegressor(n_estimators=100, learning_rate=0.1, random_state=42), SVR(kernel='rbf', C=1e3, gamma=0.1), XGBRegressor(n_estimators=100, learning_rate=0.1, random_state=42), LGBMRegressor(n_estimators=100, learning_rate=0.1, random_state=42) ] # 5-fold CV for base models kf = KFold(n_splits=5, shuffle=True, random_state=42) oof_preds = np.zeros((X_train_eng.shape[0], len(base_models))) for i, (train_idx, val_idx) in enumerate(kf.split(X_train_eng)): X_train_fold, X_val_fold = X_train_eng[train_idx], X_train_eng[val_idx] y_train_fold, y_val_fold = y_train.iloc[train_idx], y_train.iloc[val_idx] for j, model in enumerate(base_models): model.fit(X_train_fold, y_train_fold) oof_preds[val_idx, j] = model.predict(X_val_fold) # Train base models on entire training set for test predictions test_preds = np.zeros((X_test_eng.shape[0], len(base_models))) for j, model in enumerate(base_models): model.fit(X_train_eng, y_train) test_preds[:, j] = model.predict(X_test_eng)Stage 2: Meta Model 1
Train a CatBoost Regressor on the concatenated out-of-fold predictions from Stage 1, along with the original features:
# Concatenate OOF predictions with original features X_train_meta1 = np.concatenate([X_train_eng, oof_preds], axis=1) X_test_meta1 = np.concatenate([X_test_eng, test_preds], axis=1) # Train Meta Model 1 meta_model1 = CatBoostRegressor(n_estimators=500, learning_rate=0.05, random_state=42, verbose=0) meta_model1.fit(X_train_meta1, y_train) # Predict on test set y_pred_meta1 = meta_model1.predict(X_test_meta1)Stage 3: Meta Model 2
Train a simple Elastic Net Regressor on the predictions from Meta Model 1, with an additional feature: residuals from Meta Model 1:
# Predict residuals (y_true - y_pred) for training set residuals_train = y_train - meta_model1.predict(X_train_meta1) # Create new feature: predicted values from Meta Model 1 X_train_meta2 = np.column_stack([meta_model1.predict(X_train_meta1), residuals_train]) X_test_meta2 = np.column_stack([y_pred_meta1, y_test - y_pred_meta1]) # dummy residuals for test set (we don't know true labels) # Train Meta Model 2 meta_model2 = ElasticNet(alpha=0.01, l1_ratio=0.5, random_state=42) meta_model2.fit(X_train_meta2, y_train) # Final predictions y_pred_final = meta_model2.predict(X_test_meta2)Evaluate the Final Model
# RMSE evaluation rmse = np.sqrt(mean_squared_error(y_test, y_pred_final)) print(f"Final RMSE: {rmse:.4f}")Congratulations! You've just built a 3-stage stacking pipeline for regression, incorporating diverse models, feature engineering, and hierarchical learning.
Tips & Variations
- Use different CV schemes (e.g., StratifiedKFold, GroupKFold) depending on your dataset's characteristics.
- Hyperparameter tune each base model and meta model using GridSearchCV or RandomizedSearchCV.
- Experiment with more base models, like neural networks, KNN, or Bayesian Ridge.
- Add more engineered features, such as:
- Domain-specific features (e.g., housing market indicators).
- Non-linear transformations (e.g., log, sqrt) of existing features.
- Text features (if applicable) using techniques like TF-IDF or word embeddings.
- Blend predictions from multiple stacking pipelines for improved robustness.
Happy machine learning!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.