"Act as a lead quantitative researcher. Your task is to design and build a…
- User
"Act as a lead quantitative researcher. Your task is to design and build a state-of-the-art, end-to-end Python script for a sophisticated ensemble AI model. The model's objective is short-term price movement forecasting for a highly liquid US stock. Your solution must be robust, production-ready, and adhere to rigorous financial modeling standards. I am giving you the liberty to choose the specific models and features, but you must justify your architectural decisions. Core Objective & Constraints: Forecasting Task: Predict the directional movement (up or down) and the closing price 5 trading days into the future. Asset: Choose a volatile, high-volume US stock that presents a suitable challenge (e.g., NVDA, TSLA, AMD). Justify your choice briefly. Data Sourcing: Use a reliable, free API like yfinance to fetch the necessary daily data. Critical Methodological Constraint: You must implement a time-series-aware validation strategy. Walk-forward validation or time-series cross-validation is mandatory to prevent lookahead bias and accurately simulate a live trading environment. A simple random train-test split is unacceptable. Data Leakage Prevention: All preprocessing steps (like feature scaling) must be fitted only on the training data for each fold. Areas for Your AI Expertise & Liberty: Feature Engineering: Engineer a comprehensive and powerful feature set. I leave the selection to you, but it should be well-reasoned. Create features that capture different market dynamics (e.g., trend, momentum, volatility, volume). In your comments, explain the categories of indicators you chose and why they are relevant for this forecasting task. Ensemble Architecture: Design a heterogeneous ensemble model. Your goal is to combine models with different strengths to create a more robust system. You have the freedom to choose your base learners and your ensembling method (e.g., stacking, blending, weighted averaging). Justify your choices. For example, explain why you might combine a deep learning model for sequence analysis (like a Transformer, GRU, or TCN) with a gradient boosting model for its performance on tabular data (like LightGBM or CatBoost). Explain why your chosen ensembling technique is suitable. Final Deliverable: A single, fully runnable, and well-documented Python script. Use modern, mainstream libraries (e.g., PyTorch/TensorFlow, scikit-learn, LightGBM/XGBoost, pandas). Extensive Markdown Comments: Your code should read like a research notebook. Explain the 'why' behind your code—your architectural decisions, feature choices, and the implementation of the walk-forward validation. Performance Evaluation: Conclude by evaluating the model on the final test set(s). Report key metrics appropriate for a financial forecast: For Price Prediction: Root Mean Squared Error (RMSE) & Mean Absolute Error (MAE). For Directional Prediction: Classification Accuracy, F1-Score, and a Confusion Matrix.
Assistant
Sophisticated Ensemble AI Model for Short-Term Stock Price Forecasting
""" Production-Ready Ensemble Model for 5-Day Stock Price Forecasting ================================================================ Author: Lead Quantitative Researcher Objective: Predict directional movement and closing price 5 days ahead Target Asset: NVIDIA Corporation (NVDA) Methodology: Heterogeneous ensemble with walk-forward validation """ import numpy as np import pandas as pd import warnings warnings.filterwarnings('ignore') # Data acquisition and manipulation import yfinance as yf from datetime import datetime, timedelta # Feature engineering from ta import add_all_ta_features from ta.utils import dropna from scipy import stats # Machine Learning models from sklearn.preprocessing import StandardScaler, RobustScaler from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier from sklearn.metrics import (mean_squared_error, mean_absolute_error, accuracy_score, f1_score, confusion_matrix, classification_report) import lightgbm as lgb import xgboost as xgb # Deep Learning import torch import torch.nn as nn import torch.optim as optim from torch.utils.data import DataLoader, TensorDataset from torch.nn.utils import clip_grad_norm_ # Visualization import matplotlib.pyplot as plt import seaborn as sns # Set random seeds for reproducibility np.random.seed(42) torch.manual_seed(42) # ================================================================================ # SECTION 1: ASSET SELECTION AND DATA ACQUISITION # ================================================================================ """ Asset Selection Rationale: NVIDIA (NVDA) ----------------------------------------- 1. High Liquidity: One of the most traded stocks in the US market 2. Volatility: Tech sector with exposure to AI/GPU markets creates price swings 3. Data Richness: Strong reaction to market events, earnings, and sector news 4. Forecasting Challenge: Complex price dynamics suitable for ensemble modeling """ class DataAcquisition: """ Handles data fetching and initial preprocessing """ def __init__(self, ticker='NVDA', start_date='2018-01-01', end_date=None): self.ticker = ticker self.start_date = start_date self.end_date = end_date or datetime.now().strftime('%Y-%m-%d') def fetch_data(self): """ Fetch OHLCV data using yfinance """ print(f"📊 Fetching data for {self.ticker} from {self.start_date} to {self.end_date}") # Download data with auto-adjust for splits/dividends data = yf.download(self.ticker, start=self.start_date, end=self.end_date, auto_adjust=True, progress=False) # Ensure we have sufficient data if len(data) < 500: raise ValueError("Insufficient data points. Need at least 500 trading days.") print(f"✅ Successfully fetched {len(data)} trading days of data") return data # ================================================================================ # SECTION 2: FEATURE ENGINEERING # ================================================================================ class FeatureEngineer: """ Comprehensive Feature Engineering Pipeline Feature Categories: 1. Price-Based Features: Capture price dynamics and patterns 2. Technical Indicators: Traditional TA signals (momentum, trend, volatility) 3. Volume Features: Market participation and liquidity measures 4. Market Microstructure: Advanced features capturing market behavior 5. Statistical Features: Distribution and regime characteristics """ def __init__(self, lookback_periods=[5, 10, 20, 50]): self.lookback_periods = lookback_periods def create_price_features(self, df): """ Price-based features capturing various return dynamics """ feature_df = df.copy() # Log returns - more stable for modeling feature_df['log_return'] = np.log(df['Close'] / df['Close'].shift(1)) # Multiple timeframe returns for period in self.lookback_periods: feature_df[f'return_{period}d'] = df['Close'].pct_change(period) feature_df[f'log_return_{period}d'] = np.log(df['Close'] / df['Close'].shift(period)) # Price relative to moving averages (momentum/trend) for period in self.lookback_periods: ma = df['Close'].rolling(window=period).mean() feature_df[f'price_to_ma{period}'] = df['Close'] / ma # High-Low spread (intraday volatility proxy) feature_df['hl_spread'] = (df['High'] - df['Low']) / df['Close'] # Close position within day's range (momentum indicator) feature_df['close_position'] = (df['Close'] - df['Low']) / (df['High'] - df['Low'] + 1e-10) return feature_df def create_volume_features(self, df): """ Volume-based features for liquidity and participation analysis """ feature_df = pd.DataFrame(index=df.index) # Volume rate of change for period in self.lookback_periods: feature_df[f'volume_roc_{period}d'] = df['Volume'].pct_change(period) # Volume moving average ratio (unusual volume detector) for period in self.lookback_periods: vol_ma = df['Volume'].rolling(window=period).mean() feature_df[f'volume_ratio_{period}d'] = df['Volume'] / (vol_ma + 1e-10) # Price-Volume correlation (accumulation/distribution signal) for period in [10, 20]: price_change = df['Close'].pct_change() feature_df[f'pv_corr_{period}d'] = price_change.rolling(window=period).corr(df['Volume']) # On-Balance Volume momentum obv = (np.sign(df['Close'].diff()) * df['Volume']).cumsum() feature_df['obv_momentum'] = obv.pct_change(20) return feature_df def create_volatility_features(self, df): """ Volatility measures for risk assessment """ feature_df = pd.DataFrame(index=df.index) # Historical volatility (different windows) returns = df['Close'].pct_change() for period in self.lookback_periods: feature_df[f'volatility_{period}d'] = returns.rolling(window=period).std() * np.sqrt(252) # Parkinson volatility (using high-low range) for period in [5, 20]: hl_ratio = np.log(df['High'] / df['Low']) feature_df[f'parkinson_vol_{period}d'] = ( hl_ratio.rolling(window=period).apply( lambda x: np.sqrt(np.sum(x**2) / (4 * period * np.log(2))) ) * np.sqrt(252) ) # Average True Range (ATR) - volatility measure high_low = df['High'] - df['Low'] high_close = np.abs(df['High'] - df['Close'].shift()) low_close = np.abs(df['Low'] - df['Close'].shift()) true_range = pd.concat([high_low, high_close, low_close], axis=1).max(axis=1) for period in [10, 20]: feature_df[f'atr_{period}d'] = true_range.rolling(window=period).mean() / df['Close'] return feature_df def create_market_microstructure_features(self, df): """ Advanced market microstructure features """ feature_df = pd.DataFrame(index=df.index) # Amihud illiquidity measure proxy returns = df['Close'].pct_change() dollar_volume = df['Close'] * df['Volume'] feature_df['illiquidity'] = (np.abs(returns) / (dollar_volume + 1e-10)).rolling(window=20).mean() # Kyle's lambda (price impact) proxy for period in [10, 20]: signed_volume = np.sign(returns) * df['Volume'] price_change = df['Close'].diff() # Rolling regression coefficient feature_df[f'kyle_lambda_{period}d'] = ( price_change.rolling(window=period).apply( lambda y: np.nan if period > len(y) else stats.linregress(signed_volume[-period:], y[-period:])[0], raw=False ) ) # Volatility of volatility (vol clustering) volatility = returns.rolling(window=20).std() feature_df['vol_of_vol'] = volatility.rolling(window=20).std() return feature_df def create_technical_indicators(self, df): """ Traditional technical analysis indicators using TA library """ # Add all TA features ta_df = add_all_ta_features( df, open="Open", high="High", low="Low", close="Close", volume="Volume", fillna=True ) # Select most relevant indicators selected_columns = [ 'momentum_rsi', # RSI for overbought/oversold 'momentum_stoch_rsi', # Stochastic RSI 'trend_macd_signal', # MACD signal 'trend_cci', # Commodity Channel Index 'volatility_bbhi', # Bollinger Bands indicators 'volatility_bbli', 'momentum_ao', # Awesome Oscillator 'trend_adx', # ADX for trend strength 'volume_adi', # Accumulation/Distribution 'volume_mfi' # Money Flow Index ] # Filter available columns available_cols = [col for col in selected_columns if col in ta_df.columns] return ta_df[available_cols] def create_target_variables(self, df, forecast_horizon=5): """ Create target variables for both regression and classification """ # Future closing price (regression target) df['target_price'] = df['Close'].shift(-forecast_horizon) # Future return for classification df['future_return'] = df['Close'].shift(-forecast_horizon) / df['Close'] - 1 # Directional movement (classification target) df['target_direction'] = (df['future_return'] > 0).astype(int) return df def engineer_all_features(self, df): """ Combine all feature engineering steps """ print("🔧 Engineering comprehensive feature set...") # Create different feature categories price_features = self.create_price_features(df) volume_features = self.create_volume_features(df) volatility_features = self.create_volatility_features(df) microstructure_features = self.create_market_microstructure_features(df) technical_features = self.create_technical_indicators(df) # Combine all features feature_df = pd.concat([ price_features, volume_features, volatility_features, microstructure_features, technical_features ], axis=1) # Add target variables feature_df = self.create_target_variables(feature_df) # Remove any remaining NaN values feature_df = feature_df.dropna() print(f"✅ Created {len(feature_df.columns) - 3} features") # -3 for targets and future_return return feature_df # ================================================================================ # SECTION 3: DEEP LEARNING MODELS # ================================================================================ class TransformerModel(nn.Module): """ Transformer-based model for sequence modeling Architecture Rationale: - Self-attention mechanism captures long-range dependencies - Position encoding preserves temporal order - Multi-head attention learns different pattern types """ def __init__(self, input_dim, d_model=64, n_heads=4, n_layers=2, dropout=0.1): super(TransformerModel, self).__init__() self.input_projection = nn.Linear(input_dim, d_model) self.positional_encoding = PositionalEncoding(d_model, dropout) encoder_layers = nn.TransformerEncoderLayer( d_model=d_model, nhead=n_heads, dim_feedforward=d_model * 4, dropout=dropout, batch_first=True ) self.transformer_encoder = nn.TransformerEncoder(encoder_layers, num_layers=n_layers) # Dual heads for regression and classification self.regression_head = nn.Sequential( nn.Linear(d_model, d_model // 2), nn.ReLU(), nn.Dropout(dropout), nn.Linear(d_model // 2, 1) ) self.classification_head = nn.Sequential( nn.Linear(d_model, d_model // 2), nn.ReLU(), nn.Dropout(dropout), nn.Linear(d_model // 2, 2) ) def forward(self, x): # Project input to model dimension x = self.input_projection(x) x = self.positional_encoding(x) # Transformer encoding x = self.transformer_encoder(x) # Global pooling (use last timestep) x = x[:, -1, :] # Dual outputs regression_out = self.regression_head(x) classification_out = self.classification_head(x) return regression_out, classification_out class PositionalEncoding(nn.Module): """ Add positional encoding to preserve sequence order information """ def __init__(self, d_model, dropout=0.1, max_len=5000): super(PositionalEncoding, self).__init__() self.dropout = nn.Dropout(p=dropout) pe = torch.zeros(max_len, d_model) position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1) div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-np.log(10000.0) / d_model)) pe[:, 0::2] = torch.sin(position * div_term) pe[:, 1::2] = torch.cos(position * div_term) pe = pe.unsqueeze(0).transpose(0, 1) self.register_buffer('pe', pe) def forward(self, x): x = x + self.pe[:x.size(1), :].transpose(0, 1) return self.dropout(x) class GRUModel(nn.Module): """ GRU-based model for sequence modeling Architecture Rationale: - GRU gates handle vanishing gradient problem - Bidirectional processing captures both past and future context - Fewer parameters than LSTM, reducing overfitting risk """ def __init__(self, input_dim, hidden_dim=64, n_layers=2, dropout=0.2): super(GRUModel, self).__init__() self.gru = nn.GRU( input_dim, hidden_dim, n_layers, batch_first=True, dropout=dropout if n_layers > 1 else 0, bidirectional=True ) # Account for bidirectional final_dim = hidden_dim * 2 # Dual heads self.regression_head = nn.Sequential( nn.Linear(final_dim, hidden_dim), nn.ReLU(), nn.Dropout(dropout), nn.Linear(hidden_dim, 1) ) self.classification_head = nn.Sequential( nn.Linear(final_dim, hidden_dim), nn.ReLU(), nn.Dropout(dropout), nn.Linear(hidden_dim, 2) ) def forward(self, x): # GRU encoding gru_out, _ = self.gru(x) # Use last timestep x = gru_out[:, -1, :] # Dual outputs regression_out = self.regression_head(x) classification_out = self.classification_head(x) return regression_out, classification_out # ================================================================================ # SECTION 4: WALK-FORWARD VALIDATION FRAMEWORK # ================================================================================ class WalkForwardValidator: """ Time-Series Aware Walk-Forward Validation Methodology: - Expanding window approach: Train on all available historical data - Fixed test window: Evaluate on next period - No data leakage: Preprocessing fitted only on training data - Realistic simulation: Mimics real-world deployment scenario """ def __init__(self, n_splits=5, test_size=60, gap=5): """ Parameters: - n_splits: Number of walk-forward windows - test_size: Size of each test window (trading days) - gap: Gap between train and test to account for execution lag """ self.n_splits = n_splits self.test_size = test_size self.gap = gap def split(self, X, y): """ Generate train/test indices for walk-forward validation """ n_samples = len(X) splits = [] # Calculate size of each fold fold_size = (n_samples - self.test_size - self.gap) // self.n_splits for i in range(self.n_splits): # Expanding window for training train_end = fold_size * (i + 1) train_idx = np.arange(0, train_end) # Fixed window for testing (with gap) test_start = train_end + self.gap test_end = min(test_start + self.test_size, n_samples) test_idx = np.arange(test_start, test_end) if len(test_idx) > 0: splits.append((train_idx, test_idx)) return splits # ================================================================================ # SECTION 5: ENSEMBLE MODEL ARCHITECTURE # ================================================================================ class HeterogeneousEnsemble: """ Sophisticated Ensemble Architecture Design Philosophy: 1. Model Diversity: Combine models with different inductive biases - Transformer: Long-range pattern recognition - GRU: Sequential dependency modeling - LightGBM: Non-linear feature interactions - Random Forest: Robust to outliers 2. Stacking Strategy: Meta-learner combines base predictions - Learns optimal weighting dynamically - Captures model complementarity 3. Dual-Task Learning: Simultaneous regression and classification - Shared representations improve both tasks - Classification helps regression by identifying regimes """ def __init__(self, input_dim, sequence_length=20, device='cpu'): self.input_dim = input_dim self.sequence_length = sequence_length self.device = device # Initialize base models self.transformer = TransformerModel(input_dim).to(device) self.gru = GRUModel(input_dim).to(device) # Gradient boosting models self.lgb_regressor = None self.lgb_classifier = None # Random Forest models self.rf_regressor = None self.rf_classifier = None # Meta-learners for stacking self.meta_regressor = None self.meta_classifier = None # Scalers (will be fitted on training data only) self.scaler = None self.meta_scaler_reg = None self.meta_scaler_clf = None def prepare_sequences(self, X, y_reg=None, y_clf=None): """ Prepare sequences for deep learning models """ sequences = [] targets_reg = [] if y_reg is not None else None targets_clf = [] if y_clf is not None else None for i in range(len(X) - self.sequence_length): sequences.append(X[i:i + self.sequence_length]) if y_reg is not None: targets_reg.append(y_reg[i + self.sequence_length]) if y_clf is not None: targets_clf.append(y_clf[i + self.sequence_length]) return np.array(sequences), np.array(targets_reg) if targets_reg else None, \ np.array(targets_clf) if targets_clf else None def train_deep_models(self, X_seq, y_reg, y_clf, epochs=50, batch_size=32): """ Train deep learning models (Transformer and GRU) """ # Convert to tensors X_tensor = torch.FloatTensor(X_seq).to(self.device) y_reg_tensor = torch.FloatTensor(y_reg.reshape(-1, 1)).to(self.device) y_clf_tensor = torch.LongTensor(y_clf).to(self.device) # Create data loader dataset = TensorDataset(X_tensor, y_reg_tensor, y_clf_tensor) loader = DataLoader(dataset, batch_size=batch_size, shuffle=True) # Training settings models = [self.transformer, self.gru] optimizers = [ optim.AdamW(self.transformer.parameters(), lr=0.001, weight_decay=0.01), optim.AdamW(self.gru.parameters(), lr=0.001, weight_decay=0.01) ] criterion_reg = nn.MSELoss() criterion_clf = nn.CrossEntropyLoss() # Train each model for model, optimizer in zip(models, optimizers): model.train() for epoch in range(epochs): epoch_loss = 0 for batch_X, batch_y_reg, batch_y_clf in loader: optimizer.zero_grad() # Forward pass reg_out, clf_out = model(batch_X) # Combined loss loss_reg = criterion_reg(reg_out, batch_y_reg) loss_clf = criterion_clf(clf_out, batch_y_clf) loss = loss_reg + 0.5 * loss_clf # Weight classification less # Backward pass loss.backward() clip_grad_norm_(model.parameters(), max_norm=1.0) optimizer.step() epoch_loss += loss.item() model.eval() def train_gbm_models(self, X, y_reg, y_clf): """ Train gradient boosting models (LightGBM) """ # LightGBM parameters tuned for financial data lgb_params = { 'objective': 'regression', 'metric': 'rmse', 'boosting_type': 'gbdt', 'num_leaves': 31, 'learning_rate': 0.05, 'feature_fraction': 0.8, 'bagging_fraction': 0.7, 'bagging_freq': 5, 'verbose': -1, 'random_state': 42, 'n_estimators': 200, 'early_stopping_rounds': 20 } # Regression model self.lgb_regressor = lgb.LGBMRegressor(**lgb_params) self.lgb_regressor.fit( X, y_reg, eval_set=[(X[-500:], y_reg[-500:])], # Use last 500 samples for validation verbose=False ) # Classification model lgb_params['objective'] = 'binary' lgb_params['metric'] = 'binary_logloss' self.lgb_classifier = lgb.LGBMClassifier(**lgb_params) self.lgb_classifier.fit( X, y_clf, eval_set=[(X[-500:], y_clf[-500:])], verbose=False ) def train_rf_models(self, X, y_reg, y_clf): """ Train Random Forest models for robustness """ # Random Forest parameters rf_params = { 'n_estimators': 100, 'max_depth': 10, 'min_samples_split': 20, 'min_samples_leaf': 10, 'max_features': 'sqrt', 'random_state': 42, 'n_jobs': -1 } self.rf_regressor = RandomForestRegressor(**rf_params) self.rf_regressor.fit(X, y_reg) self.rf_classifier = RandomForestClassifier(**rf_params) self.rf_classifier.fit(X, y_clf) def get_base_predictions(self, X, X_seq): """ Get predictions from all base models """ predictions = {} # Deep learning predictions with torch.no_grad(): X_tensor = torch.FloatTensor(X_seq).to(self.device) # Transformer predictions trans_reg, trans_clf = self.transformer(X_tensor) predictions['transformer_reg'] = trans_reg.cpu().numpy().flatten() predictions['transformer_clf'] = torch.softmax(trans_clf, dim=1)[:, 1].cpu().numpy() # GRU predictions gru_reg, gru_clf = self.gru(X_tensor) predictions['gru_reg'] = gru_reg.cpu().numpy().flatten() predictions['gru_clf'] = torch.softmax(gru_clf, dim=1)[:, 1].cpu().numpy() # Adjust X for non-sequence models (use latest values) X_latest = X[self.sequence_length:] # LightGBM predictions if self.lgb_regressor is not None: predictions['lgb_reg'] = self.lgb_regressor.predict(X_latest) predictions['lgb_clf'] = self.lgb_classifier.predict_proba(X_latest)[:, 1] # Random Forest predictions if self.rf_regressor is not None: predictions['rf_reg'] = self.rf_regressor.predict(X_latest) predictions['rf_clf'] = self.rf_classifier.predict_proba(X_latest)[:, 1] return predictions def train_meta_learners(self, base_predictions, y_reg, y_clf): """ Train meta-learners for stacking ensemble """ # Prepare meta-features meta_features_reg = np.column_stack([ base_predictions['transformer_reg'], base_predictions['gru_reg'], base_predictions['lgb_reg'], base_predictions['rf_reg'] ]) meta_features_clf = np.column_stack([ base_predictions['transformer_clf'], base_predictions['gru_clf'], base_predictions['lgb_clf'], base_predictions['rf_clf'] ]) # Scale meta-features self.meta_scaler_reg = StandardScaler() self.meta_scaler_clf = StandardScaler() meta_features_reg = self.meta_scaler_reg.fit_transform(meta_features_reg) meta_features_clf = self.meta_scaler_clf.fit_transform(meta_features_clf) # Train meta-learners (using XGBoost for final ensemble) self.meta_regressor = xgb.XGBRegressor( n_estimators=50, max_depth=3, learning_rate=0.1, random_state=42 ) self.meta_regressor.fit(meta_features_reg, y_reg) self.meta_classifier = xgb.XGBClassifier( n_estimators=50, max_depth=3, learning_rate=0.1, random_state=42 ) self.meta_classifier.fit(meta_features_clf, y_clf) def fit(self, X, y_reg, y_clf): """ Full training pipeline for ensemble """ print("🚀 Training Heterogeneous Ensemble Model") # Scale features self.scaler = RobustScaler() X_scaled = self.scaler.fit_transform(X) # Prepare sequences X_seq, y_reg_seq, y_clf_seq = self.prepare_sequences(X_scaled, y_reg, y_clf) # Train deep learning models print(" 📊 Training deep learning models...") self.train_deep_models(X_seq, y_reg_seq, y_clf_seq) # Train gradient boosting models print(" 📊 Training gradient boosting models...") X_latest = X_scaled[self.sequence_length:] self.train_gbm_models(X_latest, y_reg_seq, y_clf_seq) # Train random forest models print(" 📊 Training random forest models...") self.train_rf_models(X_latest, y_reg_seq, y_clf_seq) # Get base predictions for meta-learning print(" 📊 Training meta-learners...") base_predictions = self.get_base_predictions(X_scaled, X_seq) self.train_meta_learners(base_predictions, y_reg_seq, y_clf_seq) print("✅ Ensemble training complete") def predict(self, X): """ Make predictions using trained ensemble """ # Scale features X_scaled = self.scaler.transform(X) # Prepare sequences X_seq, _, _ = self.prepare_sequences(X_scaled) # Get base predictions base_predictions = self.get_base_predictions(X_scaled, X_seq) # Prepare meta-features meta_features_reg = np.column_stack([ base_predictions['transformer_reg'], base_predictions['gru_reg'], base_predictions['lgb_reg'], base_predictions['rf_reg'] ]) meta_features_clf = np.column_stack([ base_predictions['transformer_clf'], base_predictions['gru_clf'], base_predictions['lgb_clf'], base_predictions['rf_clf'] ]) # Scale meta-features meta_features_reg = self.meta_scaler_reg.transform(meta_features_reg) meta_features_clf = self.meta_scaler_clf.transform(meta_features_clf) # Get final predictions final_reg = self.meta_regressor.predict(meta_features_reg) final_clf_proba = self.meta_classifier.predict_proba(meta_features_clf)[:, 1] final_clf = (final_clf_proba > 0.5).astype(int) return final_reg, final_clf, final_clf_proba # ================================================================================ # SECTION 6: MODEL EVALUATION AND BACKTESTING # ================================================================================ class ModelEvaluator: """ Comprehensive evaluation framework for financial forecasting """ @staticmethod def evaluate_regression(y_true, y_pred, prices_true, prices_pred): """ Evaluate regression performance (price prediction) """ # Standard metrics rmse = np.sqrt(mean_squared_error(y_true, y_pred)) mae = mean_absolute_error(y_true, y_pred) # Financial metrics # Convert back to actual prices for interpretability price_rmse = np.sqrt(mean_squared_error(prices_true, prices_pred)) price_mae = mean_absolute_error(prices_true, prices_pred) # Directional accuracy from regression direction_true = (y_true > 0).astype(int) direction_pred = (y_pred > 0).astype(int) directional_accuracy = accuracy_score(direction_true, direction_pred) return { 'RMSE': rmse, 'MAE': mae, 'Price_RMSE': price_rmse, 'Price_MAE': price_mae, 'Directional_Accuracy': directional_accuracy } @staticmethod def evaluate_classification(y_true, y_pred, y_proba=None): """ Evaluate classification performance (directional prediction) """ accuracy = accuracy_score(y_true, y_pred) f1 = f1_score(y_true, y_pred, average='weighted') # Confusion matrix cm = confusion_matrix(y_true, y_pred) # Calculate precision for up moves (important for trading) if cm[1, 1] + cm[0, 1] > 0: precision_up = cm[1, 1] / (cm[1, 1] + cm[0, 1]) else: precision_up = 0 return { 'Accuracy': accuracy, 'F1_Score': f1, 'Precision_Up': precision_up, 'Confusion_Matrix': cm } @staticmethod def plot_results(y_true_reg, y_pred_reg, y_true_clf, y_pred_clf, dates=None): """ Visualize prediction results """ fig, axes = plt.subplots(2, 2, figsize=(15, 10)) # Plot 1: Predicted vs Actual Returns axes[0, 0].scatter(y_true_reg, y_pred_reg, alpha=0.5) axes[0, 0].plot([y_true_reg.min(), y_true_reg.max()], [y_true_reg.min(), y_true_reg.max()], 'r--', lw=2) axes[0, 0].set_xlabel('Actual Returns') axes[0, 0].set_ylabel('Predicted Returns') axes[0, 0].set_title('Regression: Predicted vs Actual Returns') axes[0, 0].grid(True, alpha=0.3) # Plot 2: Time series of predictions if dates is not None: axes[0, 1].plot(dates, y_true_reg, label='Actual', alpha=0.7) axes[0, 1].plot(dates, y_pred_reg, label='Predicted', alpha=0.7) axes[0, 1].set_xlabel('Date') axes[0, 1].set_ylabel('Returns') axes[0, 1].set_title('Time Series: Actual vs Predicted Returns') axes[0, 1].legend() axes[0, 1].grid(True, alpha=0.3) # Plot 3: Confusion Matrix cm = confusion_matrix(y_true_clf, y_pred_clf) sns.heatmap(cm, annot=True, fmt='d', cmap='Blues', ax=axes[1, 0]) axes[1, 0].set_xlabel('Predicted') axes[1, 0].set_ylabel('Actual') axes[1, 0].set_title('Classification: Confusion Matrix') # Plot 4: Residual distribution residuals = y_true_reg - y_pred_reg axes[1, 1].hist(residuals, bins=50, edgecolor='black', alpha=0.7) axes[1, 1].axvline(x=0, color='r', linestyle='--') axes[1, 1].set_xlabel('Residuals') axes[1, 1].set_ylabel('Frequency') axes[1, 1].set_title('Residual Distribution') axes[1, 1].grid(True, alpha=0.3) plt.tight_layout() plt.show() # ================================================================================ # SECTION 7: MAIN EXECUTION PIPELINE # ================================================================================ def main(): """ Main execution pipeline with walk-forward validation """ print("=" * 80) print("SOPHISTICATED ENSEMBLE MODEL FOR STOCK PRICE FORECASTING") print("=" * 80) # Set device for deep learning device = torch.device('cuda' if torch.cuda.is_available() else 'cpu') print(f"🖥️ Using device: {device}") # Step 1: Data Acquisition data_fetcher = DataAcquisition(ticker='NVDA', start_date='2018-01-01') raw_data = data_fetcher.fetch_data() # Step 2: Feature Engineering feature_engineer = FeatureEngineer() data = feature_engineer.engineer_all_features(raw_data) # Separate features and targets feature_cols = [col for col in data.columns if col not in ['target_price', 'target_direction', 'future_return']] X = data[feature_cols].values y_price = data['target_price'].values y_direction = data['target_direction'].values # Store original prices for evaluation original_prices = data['Close'].values # Step 3: Walk-Forward Validation print("\n📈 Starting Walk-Forward Validation") print("-" * 40) validator = WalkForwardValidator(n_splits=5, test_size=60, gap=5) splits = validator.split(X, y_price) # Store results for each fold all_results_reg = [] all_results_clf = [] all_predictions_reg = [] all_predictions_clf = [] all_true_reg = [] all_true_clf = [] for fold, (train_idx, test_idx) in enumerate(splits): print(f"\n📊 Fold {fold + 1}/{len(splits)}") print(f" Training samples: {len(train_idx)}") print(f" Testing samples: {len(test_idx)}") # Split data X_train, X_test = X[train_idx], X[test_idx] y_train_price, y_test_price = y_price[train_idx], y_price[test_idx] y_train_dir, y_test_dir = y_direction[train_idx], y_direction[test_idx] # Calculate returns for evaluation train_prices = original_prices[train_idx] test_prices = original_prices[test_idx] # Adjust for sequence length in test indices seq_length = 20 actual_test_idx = test_idx[seq_length:] # Train ensemble ensemble = HeterogeneousEnsemble( input_dim=X_train.shape[1], sequence_length=seq_length, device=device ) ensemble.fit(X_train, y_train_price, y_train_dir) # Make predictions pred_price, pred_dir, pred_dir_proba = ensemble.predict(X_test) # Adjust test targets for sequence length y_test_price_adj = y_test_price[seq_length:] y_test_dir_adj = y_test_dir[seq_length:] test_prices_adj = test_prices[seq_length:] # Convert predictions to returns for evaluation test_returns_true = (y_test_price_adj - test_prices_adj) / test_prices_adj test_returns_pred = (pred_price - test_prices_adj) / test_prices_adj # Evaluate fold evaluator = ModelEvaluator() reg_metrics = evaluator.evaluate_regression( test_returns_true, test_returns_pred, y_test_price_adj, pred_price ) clf_metrics = evaluator.evaluate_classification( y_test_dir_adj, pred_dir, pred_dir_proba ) # Store results all_results_reg.append(reg_metrics) all_results_clf.append(clf_metrics) all_predictions_reg.extend(test_returns_pred) all_predictions_clf.extend(pred_dir) all_true_reg.extend(test_returns_true) all_true_clf.extend(y_test_dir_adj) # Print fold results print(f"\n Regression Metrics:") print(f" - RMSE: {reg_metrics['RMSE']:.4f}") print(f" - MAE: {reg_metrics['MAE']:.4f}") print(f" - Price RMSE: ${reg_metrics['Price_RMSE']:.2f}") print(f" - Directional Accuracy: {reg_metrics['Directional_Accuracy']:.2%}") print(f"\n Classification Metrics:") print(f" - Accuracy: {clf_metrics['Accuracy']:.2%}") print(f" - F1 Score: {clf_metrics['F1_Score']:.4f}") print(f" - Precision (Up): {clf_metrics['Precision_Up']:.2%}") # Step 4: Aggregate Results print("\n" + "=" * 80) print("FINAL RESULTS (AGGREGATED ACROSS ALL FOLDS)") print("=" * 80) # Convert to numpy arrays all_predictions_reg = np.array(all_predictions_reg) all_predictions_clf = np.array(all_predictions_clf) all_true_reg = np.array(all_true_reg) all_true_clf = np.array(all_true_clf) # Calculate final metrics evaluator = ModelEvaluator() # Price prediction metrics final_rmse = np.sqrt(mean_squared_error(all_true_reg, all_predictions_reg)) final_mae = mean_absolute_error(all_true_reg, all_predictions_reg) # Direction prediction metrics final_accuracy = accuracy_score(all_true_clf, all_predictions_clf) final_f1 = f1_score(all_true_clf, all_predictions_clf, average='weighted') final_cm = confusion_matrix(all_true_clf, all_predictions_clf) print("\n📊 REGRESSION PERFORMANCE (5-Day Price Prediction)") print("-" * 40) print(f"Root Mean Squared Error (RMSE): {final_rmse:.4f}") print(f"Mean Absolute Error (MAE): {final_mae:.4f}") # Calculate average metrics across folds avg_price_rmse = np.mean([r['Price_RMSE'] for r in all_results_reg]) avg_price_mae = np.mean([r['Price_MAE'] for r in all_results_reg]) print(f"Average Price RMSE: ${avg_price_rmse:.2f}") print(f"Average Price MAE: ${avg_price_mae:.2f}") print("\n📊 CLASSIFICATION PERFORMANCE (5-Day Direction Prediction)") print("-" * 40) print(f"Overall Accuracy: {final_accuracy:.2%}") print(f"F1 Score: {final_f1:.4f}") print("\n📊 Confusion Matrix:") print("-" * 40) print(" Predicted") print(" Down Up") print(f"Down {final_cm[0, 0]:4d} {final_cm[0, 1]:4d}") print(f"Up {final_cm[1, 0]:4d} {final_cm[1, 1]:4d}") # Calculate additional metrics if final_cm[1, 1] + final_cm[0, 1] > 0: precision_up = final_cm[1, 1] / (final_cm[1, 1] + final_cm[0, 1]) print(f"\nPrecision for Up moves: {precision_up:.2%}") if final_cm[1, 1] + final_cm[1, 0] > 0: recall_up = final_cm[1, 1] / (final_cm[1, 1] + final_cm[1, 0]) print(f"Recall for Up moves: {recall_up:.2%}") # Visualize results print("\n📊 Generating visualization...") evaluator.plot_results(all_true_reg, all_predictions_reg, all_true_clf, all_predictions_clf) print("\n" + "=" * 80) print("MODEL EVALUATION COMPLETE") print("=" * 80) return { 'regression': { 'rmse': final_rmse, 'mae': final_mae, 'predictions': all_predictions_reg, 'true_values': all_true_reg }, 'classification': { 'accuracy': final_accuracy, 'f1_score': final_f1, 'confusion_matrix': final_cm, 'predictions': all_predictions_clf, 'true_values': all_true_clf } } # ================================================================================ # EXECUTE MAIN PIPELINE # ================================================================================ if __name__ == "__main__": # Run the complete pipeline results = main() print("\n🎯 Model training and evaluation completed successfully!") print("The ensemble model combines:") print(" • Transformer (attention-based pattern recognition)") print(" • GRU (sequential dependency modeling)") print(" • LightGBM (gradient boosting for tabular features)") print(" • Random Forest (robust baseline)") print("\nAll integrated through meta-learning for optimal performance.")Key Architectural Decisions & Justifications
1. Asset Selection: NVIDIA (NVDA)
- High volatility in the AI/semiconductor sector provides rich patterns
- Massive liquidity ensures price discovery and reduces market microstructure noise
- News sensitivity creates exploitable momentum and mean-reversion opportunities
2. Feature Engineering Philosophy
- Multi-scale features: Capture patterns across different time horizons (5, 10, 20, 50 days)
- Market microstructure: Kyle's lambda and Amihud illiquidity for execution cost awareness
- Regime indicators: Volatility clustering and correlation features for market state detection
- Technical indicators: Well-established signals that many market participants follow
3. Ensemble Architecture Rationale
- Transformer: Self-attention mechanism captures non-local dependencies and regime changes
- GRU: Efficient sequential modeling with fewer parameters than LSTM
- LightGBM: Superior performance on tabular data with built-in regularization
- Random Forest: Robust baseline resistant to outliers and overfitting
- Meta-learning stacking: Learns optimal combination weights dynamically
4. Walk-Forward Validation
- Expanding window: Uses all available historical data (realistic for production)
- Fixed test window: Consistent evaluation period across folds
- Gap parameter: Accounts for execution delay in real trading
5. Dual-Task Learning
- Simultaneous regression and classification improves feature representations
- Classification helps identify market regimes that affect price prediction accuracy
- Shared representations reduce overfitting through multi-task regularization
This implementation represents a production-ready, institutional-grade forecasting system that combines modern deep learning with traditional quantitative finance techniques, all while maintaining strict time-series discipline to prevent lookahead bias.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.