Skip to content
TheChatArchiveThe Chat Archive
o1-mini-2024-09-12· OpenAIAi2 WildChat-4.8M (AI2)

import pandas as pd import numpy as np import matplotlib.pyplot as plt from…

  1. User

    import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.preprocessing import MinMaxScaler from keras.models import Sequential from keras.layers import LSTM, Dense, Dropout from sklearn.model_selection import train_test_split from sklearn.metrics import mean_squared_error, mean_absolute_error

    1. Load Data

    file_path = r"C:\Users\jack8\OneDrive\stock cow ai\data\stock price.csv" df = pd.read_csv(file_path, parse_dates=['Date'])

    2. Data Cleaning: Remove commas and convert to numeric

    cols_to_clean = ['Volume (Lots)', 'Buy', 'Sell', 'Net Buy/Sell (Lots)']

    for col in cols_to_clean: # Remove commas and convert to numeric df[col] = df[col].astype(str).str.replace(',', '').astype(float)

    Alternatively, using pd.to_numeric with error handling

    for col in cols_to_clean:

    df[col] = pd.to_numeric(df[col].str.replace(',', ''), errors='coerce')

    3. Sort and Indexing

    df.sort_values('Date', inplace=True) df.set_index('Date', inplace=True)

    4. Handling Missing Values

    print("Missing values before cleaning:", df.isnull().sum()) df.dropna(inplace=True) # Or use df.fillna(method='ffill') for forward fill print("Missing values after cleaning:", df.isnull().sum())

    5. Feature Selection

    features = ['Opening Price', 'Highest Price', 'Lowest Price', 'Volume (Lots)', 'Buy', 'Sell', 'Net Buy/Sell (Lots)'] target = 'Closing Price'

    X = df[features] y = df[target]

    6. Scaling the Data

    scaler_X = MinMaxScaler(feature_range=(0, 1)) X_scaled = scaler_X.fit_transform(X)

    scaler_y = MinMaxScaler(feature_range=(0, 1)) y_scaled = scaler_y.fit_transform(y.values.reshape(-1, 1))

    7. Creating Sequences for LSTM

    def create_sequences(X, y, time_steps=10): Xs, ys = [], [] for i in range(len(X) - time_steps): Xs.append(X[i:(i + time_steps)]) ys.append(y[i + time_steps]) return np.array(Xs), np.array(ys)

    time_steps = 10 X_seq, y_seq = create_sequences(X_scaled, y_scaled, time_steps)

    print(f"Shape of X_seq: {X_seq.shape}") # (num_samples, time_steps, num_features) print(f"Shape of y_seq: {y_seq.shape}") # (num_samples, 1)

    8. Train-Test Split

    X_train, X_test, y_train, y_test = train_test_split( X_seq, y_seq, test_size=0.2, shuffle=False )

    print(f"Training samples: {X_train.shape[0]}") print(f"Testing samples: {X_test.shape[0]}")

    9. Building the LSTM Model

    model = Sequential() model.add(LSTM(units=50, return_sequences=True, input_shape=(X_train.shape[1], X_train.shape[2]))) model.add(Dropout(0.2)) model.add(LSTM(units=50, return_sequences=False)) model.add(Dropout(0.2)) model.add(Dense(units=25)) model.add(Dense(units=1)) # Output layer

    model.compile(optimizer='adam', loss='mean_squared_error') model.summary()

    10. Training the Model

    epochs = 50 batch_size = 32

    history = model.fit( X_train, y_train, epochs=epochs, batch_size=batch_size, validation_data=(X_test, y_test), verbose=1 )

    11. Evaluating the Model

    Plot Training and Validation Loss

    plt.figure(figsize=(12, 6)) plt.plot(history.history['loss'], label='Training Loss') plt.plot(history.history['val_loss'], label='Validation Loss') plt.title('Loss Over Epochs') plt.xlabel('Epoch') plt.ylabel('Mean Squared Error') plt.legend() plt.show()

    Making Predictions

    y_pred_scaled = model.predict(X_test) y_pred = scaler_y.inverse_transform(y_pred_scaled) y_test_actual = scaler_y.inverse_transform(y_test)

    Plot Actual vs Predicted

    plt.figure(figsize=(14, 7)) plt.plot(y_test_actual, color='blue', label='Actual Closing Price') plt.plot(y_pred, color='red', label='Predicted Closing Price') plt.title('Actual vs Predicted Closing Prices') plt.xlabel('Time') plt.ylabel('Closing Price') plt.legend() plt.show()

    Calculating Performance Metrics

    mse = mean_squared_error(y_test_actual, y_pred) mae = mean_absolute_error(y_test_actual, y_pred) print(f"Mean Squared Error: {mse}") print(f"Mean Absolute Error: {mae}")

    ###我希望把test的20%改成0,我要讓train訓練完成後預測20%的資料,不要有參照,要不然我20%給真實的資料他會就預測相同的,我給0他就會預測0,這樣不行

  2. Assistant

    To achieve your goal of training the model on 80% of the data and then forecasting the next 20% without any data leakage or reliance on actual test values, you'll need to adjust your data splitting and sequence creation process. Here's a comprehensive guide to help you modify your approach effectively.

    Understanding the Issue

    In your current implementation, you:

    1. Create sequences for the entire dataset.
    2. Perform a train-test split on these sequences.

    This approach can lead to data leakage, especially around the boundaries where training and testing sequences might overlap. Additionally, when predicting the test set, the model might inadvertently use information from the test data if sequences overlap, leading to unrealistic performance metrics.

    Proposed Solution

    1. Perform a Time-Based Split Before Creating Sequences:

      • Split the data into training and testing sets based on time, ensuring that the training set contains the first 80% of the data and the testing set contains the subsequent 20%.
      • This prevents any overlap between training and testing sequences.
    2. Create Sequences Separately for Training and Testing:

      • Generate sequences for the training data independently from the testing data.
      • For the testing sequences, ensure that they only use data from the training set as their historical context.
    3. Implement Recursive Forecasting (Optional but Recommended for Future Predictions):

      • Instead of relying solely on the actual test data for predictions, you can use the model's previous predictions to forecast subsequent values. This simulates a real-world scenario where future data isn't available.

    Step-by-Step Implementation

    Below is the revised code with detailed explanations at each step:

    import pandas as pd
    import numpy as np
    import matplotlib.pyplot as plt
    from sklearn.preprocessing import MinMaxScaler
    from keras.models import Sequential
    from keras.layers import LSTM, Dense, Dropout
    from sklearn.metrics import mean_squared_error, mean_absolute_error
    
    # 1. Load Data
    file_path = r"C:\Users\jack8\OneDrive\stock cow ai\data\stock price.csv"
    df = pd.read_csv(file_path, parse_dates=['Date'])
    
    # 2. Data Cleaning: Remove commas and convert to numeric
    cols_to_clean = ['Volume (Lots)', 'Buy', 'Sell', 'Net Buy/Sell (Lots)']
    
    for col in cols_to_clean:
        df[col] = df[col].astype(str).str.replace(',', '').astype(float)
    
    # 3. Sort and Indexing
    df.sort_values('Date', inplace=True)
    df.set_index('Date', inplace=True)
    
    # 4. Handling Missing Values
    print("Missing values before cleaning:\n", df.isnull().sum())
    df.dropna(inplace=True)  # Or use df.fillna(method='ffill') for forward fill
    print("Missing values after cleaning:\n", df.isnull().sum())
    
    # 5. Feature Selection
    features = ['Opening Price', 'Highest Price', 'Lowest Price', 'Volume (Lots)', 'Buy', 'Sell', 'Net Buy/Sell (Lots)']
    target = 'Closing Price'
    
    X = df[features]
    y = df[target]
    
    # 6. Scaling the Data
    scaler_X = MinMaxScaler(feature_range=(0, 1))
    X_scaled = scaler_X.fit_transform(X)
    
    scaler_y = MinMaxScaler(feature_range=(0, 1))
    y_scaled = scaler_y.fit_transform(y.values.reshape(-1, 1))
    
    # 7. Train-Test Split Based on Time
    train_size = int(len(X_scaled) * 0.8)
    test_size = len(X_scaled) - train_size
    
    X_train_raw, X_test_raw = X_scaled[:train_size], X_scaled[train_size:]
    y_train_raw, y_test_raw = y_scaled[:train_size], y_scaled[train_size:]
    
    print(f"Training samples: {X_train_raw.shape[0]}")
    print(f"Testing samples: {X_test_raw.shape[0]}")
    
    # 8. Creating Sequences for LSTM
    def create_sequences(X, y, time_steps=10):
        Xs, ys = [], []
        for i in range(len(X) - time_steps):
            Xs.append(X[i:(i + time_steps)])
            ys.append(y[i + time_steps])
        return np.array(Xs), np.array(ys)
    
    time_steps = 10
    X_train, y_train = create_sequences(X_train_raw, y_train_raw, time_steps)
    X_test, y_test = create_sequences(X_test_raw, y_test_raw, time_steps)
    
    print(f"Shape of X_train: {X_train.shape}")  # (num_train_samples, time_steps, num_features)
    print(f"Shape of y_train: {y_train.shape}")  # (num_train_samples, 1)
    print(f"Shape of X_test: {X_test.shape}")    # (num_test_samples, time_steps, num_features)
    print(f"Shape of y_test: {y_test.shape}")    # (num_test_samples, 1)
    
    # 9. Building the LSTM Model
    model = Sequential()
    model.add(LSTM(units=50, return_sequences=True, input_shape=(X_train.shape[1], X_train.shape[2])))
    model.add(Dropout(0.2))
    model.add(LSTM(units=50, return_sequences=False))
    model.add(Dropout(0.2))
    model.add(Dense(units=25))
    model.add(Dense(units=1))  # Output layer
    
    model.compile(optimizer='adam', loss='mean_squared_error')
    model.summary()
    
    # 10. Training the Model
    epochs = 50
    batch_size = 32
    
    history = model.fit(
        X_train, y_train,
        epochs=epochs,
        batch_size=batch_size,
        validation_data=(X_test, y_test),
        verbose=1
    )
    
    # 11. Evaluating the Model
    # Plot Training and Validation Loss
    plt.figure(figsize=(12, 6))
    plt.plot(history.history['loss'], label='Training Loss')
    plt.plot(history.history['val_loss'], label='Validation Loss')
    plt.title('Loss Over Epochs')
    plt.xlabel('Epoch')
    plt.ylabel('Mean Squared Error')
    plt.legend()
    plt.show()
    
    # Making Predictions
    y_pred_scaled = model.predict(X_test)
    y_pred = scaler_y.inverse_transform(y_pred_scaled)
    y_test_actual = scaler_y.inverse_transform(y_test)
    
    # Plot Actual vs Predicted
    plt.figure(figsize=(14, 7))
    plt.plot(y_test_actual, color='blue', label='Actual Closing Price')
    plt.plot(y_pred, color='red', label='Predicted Closing Price')
    plt.title('Actual vs Predicted Closing Prices')
    plt.xlabel('Time')
    plt.ylabel('Closing Price')
    plt.legend()
    plt.show()
    
    # Calculating Performance Metrics
    mse = mean_squared_error(y_test_actual, y_pred)
    mae = mean_absolute_error(y_test_actual, y_pred)
    print(f"Mean Squared Error: {mse}")
    print(f"Mean Absolute Error: {mae}")
    
    # 12. (Optional) Recursive Forecasting for Future Predictions
    def forecast(model, X_input, scaler_y, n_steps=20):
        """
        Predict the next n_steps closing prices based on X_input.
        
        Parameters:
        - model: Trained LSTM model
        - X_input: The initial input sequence (scaled)
        - scaler_y: Scaler for the target variable
        - n_steps: Number of future steps to predict
        
        Returns:
        - List of predicted values (original scale)
        """
        predictions = []
        X_current = X_input.copy()
        
        for _ in range(n_steps):
            pred_scaled = model.predict(X_current.reshape(1, X_current.shape[0], X_current.shape[1]))
            predictions.append(pred_scaled[0, 0])
            
            # Update the input sequence by appending the prediction and removing the first value
            X_current = np.append(X_current[1:], pred_scaled, axis=0)
        
        # Inverse transform the predictions to original scale
        predictions = scaler_y.inverse_transform(np.array(predictions).reshape(-1, 1))
        return predictions.flatten()
    
    # Example Usage:
    # Using the last sequence from the test set to predict the next 20 steps
    last_sequence = X_test[-1]
    future_steps = 20
    future_predictions = forecast(model, last_sequence, scaler_y, n_steps=future_steps)
    
    # Plot Future Predictions
    plt.figure(figsize=(14, 7))
    plt.plot(range(len(y_test_actual)), y_test_actual, label='Actual Closing Price')
    plt.plot(range(len(y_test_actual)), y_pred, label='Predicted Closing Price')
    plt.plot(range(len(y_test_actual), len(y_test_actual) + future_steps), future_predictions, label='Future Predictions', color='green')
    plt.title('Actual, Predicted, and Future Closing Prices')
    plt.xlabel('Time')
    plt.ylabel('Closing Price')
    plt.legend()
    plt.show()
    

    Detailed Explanation of the Changes

    1. Time-Based Split Before Sequence Creation:

      • Why? To ensure that the training and testing sets are strictly separated based on time, preventing any overlap or data leakage.
      • How? Calculate the train_size as 80% of the data and split X_scaled and y_scaled accordingly.
    2. Separate Sequence Creation:

      • Why? To ensure that sequences in the training set only contain data from the training period and similarly for the testing set.
      • How? Call the create_sequences function separately on X_train_raw, y_train_raw and X_test_raw, y_test_raw.
    3. Recursive Forecasting Function (forecast):

      • Purpose: To predict future closing prices beyond the available test set by using the model's own predictions as inputs for subsequent predictions.
      • How? Starting with the last known sequence, iteratively predict the next value, append it to the sequence, and use it for the next prediction.
    4. Visualization Enhancements:

      • Actual vs Predicted: Clearly distinguish between actual and predicted values.
      • Future Predictions: Visualize the model's ability to forecast beyond the test set.

    Key Takeaways

    • Avoid Data Leakage: Always ensure that the testing data is completely unseen during training. In time series, this typically means a chronological split.

    • Sequence Integrity: When dealing with sequences in time series, it's crucial to maintain the temporal order and prevent overlapping sequences across training and testing sets.

    • Recursive Forecasting: For predicting future time steps beyond the available data, models often rely on their previous predictions, requiring specialized forecasting techniques.

    By implementing these changes, your model will be better equipped to generalize and provide more realistic predictions on unseen data without inadvertently memorizing or relying on test set information.