import pandas as pd import numpy as np import matplotlib.pyplot as plt from…
- User
import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.preprocessing import MinMaxScaler from keras.models import Sequential from keras.layers import LSTM, Dense, Dropout from sklearn.model_selection import train_test_split from sklearn.metrics import mean_squared_error, mean_absolute_error
1. Load Data
file_path = r"C:\Users\jack8\OneDrive\stock cow ai\data\stock price.csv" df = pd.read_csv(file_path, parse_dates=['Date'])
2. Data Cleaning: Remove commas and convert to numeric
cols_to_clean = ['Volume (Lots)', 'Buy', 'Sell', 'Net Buy/Sell (Lots)']
for col in cols_to_clean: # Remove commas and convert to numeric df[col] = df[col].astype(str).str.replace(',', '').astype(float)
Alternatively, using pd.to_numeric with error handling
for col in cols_to_clean:
df[col] = pd.to_numeric(df[col].str.replace(',', ''), errors='coerce')
3. Sort and Indexing
df.sort_values('Date', inplace=True) df.set_index('Date', inplace=True)
4. Handling Missing Values
print("Missing values before cleaning:", df.isnull().sum()) df.dropna(inplace=True) # Or use df.fillna(method='ffill') for forward fill print("Missing values after cleaning:", df.isnull().sum())
5. Feature Selection
features = ['Opening Price', 'Highest Price', 'Lowest Price', 'Volume (Lots)', 'Buy', 'Sell', 'Net Buy/Sell (Lots)'] target = 'Closing Price'
X = df[features] y = df[target]
6. Scaling the Data
scaler_X = MinMaxScaler(feature_range=(0, 1)) X_scaled = scaler_X.fit_transform(X)
scaler_y = MinMaxScaler(feature_range=(0, 1)) y_scaled = scaler_y.fit_transform(y.values.reshape(-1, 1))
7. Creating Sequences for LSTM
def create_sequences(X, y, time_steps=10): Xs, ys = [], [] for i in range(len(X) - time_steps): Xs.append(X[i:(i + time_steps)]) ys.append(y[i + time_steps]) return np.array(Xs), np.array(ys)
time_steps = 10 X_seq, y_seq = create_sequences(X_scaled, y_scaled, time_steps)
print(f"Shape of X_seq: {X_seq.shape}") # (num_samples, time_steps, num_features) print(f"Shape of y_seq: {y_seq.shape}") # (num_samples, 1)
8. Train-Test Split
X_train, X_test, y_train, y_test = train_test_split( X_seq, y_seq, test_size=0.2, shuffle=False )
print(f"Training samples: {X_train.shape[0]}") print(f"Testing samples: {X_test.shape[0]}")
9. Building the LSTM Model
model = Sequential() model.add(LSTM(units=50, return_sequences=True, input_shape=(X_train.shape[1], X_train.shape[2]))) model.add(Dropout(0.2)) model.add(LSTM(units=50, return_sequences=False)) model.add(Dropout(0.2)) model.add(Dense(units=25)) model.add(Dense(units=1)) # Output layer
model.compile(optimizer='adam', loss='mean_squared_error') model.summary()
10. Training the Model
epochs = 50 batch_size = 32
history = model.fit( X_train, y_train, epochs=epochs, batch_size=batch_size, validation_data=(X_test, y_test), verbose=1 )
11. Evaluating the Model
Plot Training and Validation Loss
plt.figure(figsize=(12, 6)) plt.plot(history.history['loss'], label='Training Loss') plt.plot(history.history['val_loss'], label='Validation Loss') plt.title('Loss Over Epochs') plt.xlabel('Epoch') plt.ylabel('Mean Squared Error') plt.legend() plt.show()
Making Predictions
y_pred_scaled = model.predict(X_test) y_pred = scaler_y.inverse_transform(y_pred_scaled) y_test_actual = scaler_y.inverse_transform(y_test)
Plot Actual vs Predicted
plt.figure(figsize=(14, 7)) plt.plot(y_test_actual, color='blue', label='Actual Closing Price') plt.plot(y_pred, color='red', label='Predicted Closing Price') plt.title('Actual vs Predicted Closing Prices') plt.xlabel('Time') plt.ylabel('Closing Price') plt.legend() plt.show()
Calculating Performance Metrics
mse = mean_squared_error(y_test_actual, y_pred) mae = mean_absolute_error(y_test_actual, y_pred) print(f"Mean Squared Error: {mse}") print(f"Mean Absolute Error: {mae}")
###我希望把test的20%改成0,我要讓train訓練完成後預測20%的資料,不要有參照,要不然我20%給真實的資料他會就預測相同的,我給0他就會預測0,這樣不行
- Assistant
To achieve your goal of training the model on 80% of the data and then forecasting the next 20% without any data leakage or reliance on actual test values, you'll need to adjust your data splitting and sequence creation process. Here's a comprehensive guide to help you modify your approach effectively.
Understanding the Issue
In your current implementation, you:
- Create sequences for the entire dataset.
- Perform a train-test split on these sequences.
This approach can lead to data leakage, especially around the boundaries where training and testing sequences might overlap. Additionally, when predicting the test set, the model might inadvertently use information from the test data if sequences overlap, leading to unrealistic performance metrics.
Proposed Solution
-
Perform a Time-Based Split Before Creating Sequences:
- Split the data into training and testing sets based on time, ensuring that the training set contains the first 80% of the data and the testing set contains the subsequent 20%.
- This prevents any overlap between training and testing sequences.
-
Create Sequences Separately for Training and Testing:
- Generate sequences for the training data independently from the testing data.
- For the testing sequences, ensure that they only use data from the training set as their historical context.
-
Implement Recursive Forecasting (Optional but Recommended for Future Predictions):
- Instead of relying solely on the actual test data for predictions, you can use the model's previous predictions to forecast subsequent values. This simulates a real-world scenario where future data isn't available.
Step-by-Step Implementation
Below is the revised code with detailed explanations at each step:
import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.preprocessing import MinMaxScaler from keras.models import Sequential from keras.layers import LSTM, Dense, Dropout from sklearn.metrics import mean_squared_error, mean_absolute_error # 1. Load Data file_path = r"C:\Users\jack8\OneDrive\stock cow ai\data\stock price.csv" df = pd.read_csv(file_path, parse_dates=['Date']) # 2. Data Cleaning: Remove commas and convert to numeric cols_to_clean = ['Volume (Lots)', 'Buy', 'Sell', 'Net Buy/Sell (Lots)'] for col in cols_to_clean: df[col] = df[col].astype(str).str.replace(',', '').astype(float) # 3. Sort and Indexing df.sort_values('Date', inplace=True) df.set_index('Date', inplace=True) # 4. Handling Missing Values print("Missing values before cleaning:\n", df.isnull().sum()) df.dropna(inplace=True) # Or use df.fillna(method='ffill') for forward fill print("Missing values after cleaning:\n", df.isnull().sum()) # 5. Feature Selection features = ['Opening Price', 'Highest Price', 'Lowest Price', 'Volume (Lots)', 'Buy', 'Sell', 'Net Buy/Sell (Lots)'] target = 'Closing Price' X = df[features] y = df[target] # 6. Scaling the Data scaler_X = MinMaxScaler(feature_range=(0, 1)) X_scaled = scaler_X.fit_transform(X) scaler_y = MinMaxScaler(feature_range=(0, 1)) y_scaled = scaler_y.fit_transform(y.values.reshape(-1, 1)) # 7. Train-Test Split Based on Time train_size = int(len(X_scaled) * 0.8) test_size = len(X_scaled) - train_size X_train_raw, X_test_raw = X_scaled[:train_size], X_scaled[train_size:] y_train_raw, y_test_raw = y_scaled[:train_size], y_scaled[train_size:] print(f"Training samples: {X_train_raw.shape[0]}") print(f"Testing samples: {X_test_raw.shape[0]}") # 8. Creating Sequences for LSTM def create_sequences(X, y, time_steps=10): Xs, ys = [], [] for i in range(len(X) - time_steps): Xs.append(X[i:(i + time_steps)]) ys.append(y[i + time_steps]) return np.array(Xs), np.array(ys) time_steps = 10 X_train, y_train = create_sequences(X_train_raw, y_train_raw, time_steps) X_test, y_test = create_sequences(X_test_raw, y_test_raw, time_steps) print(f"Shape of X_train: {X_train.shape}") # (num_train_samples, time_steps, num_features) print(f"Shape of y_train: {y_train.shape}") # (num_train_samples, 1) print(f"Shape of X_test: {X_test.shape}") # (num_test_samples, time_steps, num_features) print(f"Shape of y_test: {y_test.shape}") # (num_test_samples, 1) # 9. Building the LSTM Model model = Sequential() model.add(LSTM(units=50, return_sequences=True, input_shape=(X_train.shape[1], X_train.shape[2]))) model.add(Dropout(0.2)) model.add(LSTM(units=50, return_sequences=False)) model.add(Dropout(0.2)) model.add(Dense(units=25)) model.add(Dense(units=1)) # Output layer model.compile(optimizer='adam', loss='mean_squared_error') model.summary() # 10. Training the Model epochs = 50 batch_size = 32 history = model.fit( X_train, y_train, epochs=epochs, batch_size=batch_size, validation_data=(X_test, y_test), verbose=1 ) # 11. Evaluating the Model # Plot Training and Validation Loss plt.figure(figsize=(12, 6)) plt.plot(history.history['loss'], label='Training Loss') plt.plot(history.history['val_loss'], label='Validation Loss') plt.title('Loss Over Epochs') plt.xlabel('Epoch') plt.ylabel('Mean Squared Error') plt.legend() plt.show() # Making Predictions y_pred_scaled = model.predict(X_test) y_pred = scaler_y.inverse_transform(y_pred_scaled) y_test_actual = scaler_y.inverse_transform(y_test) # Plot Actual vs Predicted plt.figure(figsize=(14, 7)) plt.plot(y_test_actual, color='blue', label='Actual Closing Price') plt.plot(y_pred, color='red', label='Predicted Closing Price') plt.title('Actual vs Predicted Closing Prices') plt.xlabel('Time') plt.ylabel('Closing Price') plt.legend() plt.show() # Calculating Performance Metrics mse = mean_squared_error(y_test_actual, y_pred) mae = mean_absolute_error(y_test_actual, y_pred) print(f"Mean Squared Error: {mse}") print(f"Mean Absolute Error: {mae}") # 12. (Optional) Recursive Forecasting for Future Predictions def forecast(model, X_input, scaler_y, n_steps=20): """ Predict the next n_steps closing prices based on X_input. Parameters: - model: Trained LSTM model - X_input: The initial input sequence (scaled) - scaler_y: Scaler for the target variable - n_steps: Number of future steps to predict Returns: - List of predicted values (original scale) """ predictions = [] X_current = X_input.copy() for _ in range(n_steps): pred_scaled = model.predict(X_current.reshape(1, X_current.shape[0], X_current.shape[1])) predictions.append(pred_scaled[0, 0]) # Update the input sequence by appending the prediction and removing the first value X_current = np.append(X_current[1:], pred_scaled, axis=0) # Inverse transform the predictions to original scale predictions = scaler_y.inverse_transform(np.array(predictions).reshape(-1, 1)) return predictions.flatten() # Example Usage: # Using the last sequence from the test set to predict the next 20 steps last_sequence = X_test[-1] future_steps = 20 future_predictions = forecast(model, last_sequence, scaler_y, n_steps=future_steps) # Plot Future Predictions plt.figure(figsize=(14, 7)) plt.plot(range(len(y_test_actual)), y_test_actual, label='Actual Closing Price') plt.plot(range(len(y_test_actual)), y_pred, label='Predicted Closing Price') plt.plot(range(len(y_test_actual), len(y_test_actual) + future_steps), future_predictions, label='Future Predictions', color='green') plt.title('Actual, Predicted, and Future Closing Prices') plt.xlabel('Time') plt.ylabel('Closing Price') plt.legend() plt.show()Detailed Explanation of the Changes
-
Time-Based Split Before Sequence Creation:
- Why? To ensure that the training and testing sets are strictly separated based on time, preventing any overlap or data leakage.
- How? Calculate the
train_sizeas 80% of the data and splitX_scaledandy_scaledaccordingly.
-
Separate Sequence Creation:
- Why? To ensure that sequences in the training set only contain data from the training period and similarly for the testing set.
- How? Call the
create_sequencesfunction separately onX_train_raw,y_train_rawandX_test_raw,y_test_raw.
-
Recursive Forecasting Function (
forecast):- Purpose: To predict future closing prices beyond the available test set by using the model's own predictions as inputs for subsequent predictions.
- How? Starting with the last known sequence, iteratively predict the next value, append it to the sequence, and use it for the next prediction.
-
Visualization Enhancements:
- Actual vs Predicted: Clearly distinguish between actual and predicted values.
- Future Predictions: Visualize the model's ability to forecast beyond the test set.
Key Takeaways
-
Avoid Data Leakage: Always ensure that the testing data is completely unseen during training. In time series, this typically means a chronological split.
-
Sequence Integrity: When dealing with sequences in time series, it's crucial to maintain the temporal order and prevent overlapping sequences across training and testing sets.
-
Recursive Forecasting: For predicting future time steps beyond the available data, models often rely on their previous predictions, requiring specialized forecasting techniques.
By implementing these changes, your model will be better equipped to generalize and provide more realistic predictions on unseen data without inadvertently memorizing or relying on test set information.