Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0125-preview· openaiHugging Face Suzhen/CodeChat

i have following code to train my NN model : %% from sklearn.preprocessing import…

  1. User

    i have following code to train my NN model :

    %%

    from sklearn.preprocessing import StandardScaler import pandas as pd import numpy as np from tensorflow import keras import joblib

    def calculate_targets_scaling_params(file_path): scaler = StandardScaler() for chunk in pd.read_csv(file_path, chunksize=10000): # Adjust chunksize based on your memory capacity filtered_chunk = chunk[[‘y_High_5d’,‘y_Low_5d’,‘y_Priority_5d’]]

    scaler.partial_fit(filtered_chunk) # Accumulate means and variances

    return scaler.mean_, scaler.var_

    %%

    x_scaler_loaded = joblib.load(‘nn_x_scaler.sav’) y_scaler_loaded = joblib.load(‘nn_y_hlp5_scaler.sav’)

    def data_generator(file_path, batch_size, x_scaler, y_scaler): chunksize = batch_size

    while True: # Loop forever, so the generator never terminates for chunk in pd.read_csv(file_path, chunksize=chunksize):

    filtered_c = chunk.drop([‘Date’, ‘Symbol’], axis=1)

    feature_data = filtered_c.drop([ ‘y_High_1d’, ‘y_Low_1d’, ‘y_Priority_1d’, ‘y_High_2d’, ‘y_Low_2d’, ‘y_Priority_2d’, ‘y_High_3d’, ‘y_Low_3d’, ‘y_Priority_3d’, ‘y_High_5d’, ‘y_Low_5d’, ‘y_Priority_5d’], axis=1) target_data = filtered_c[[‘y_High_5d’, ‘y_Low_5d’, ‘y_Priority_5d’]]

    feature_data_scaled = pd.DataFrame(x_scaler.transform(feature_data), columns=feature_data.columns)

    Assuming target_data also needs to be scaled, apply scaler separately

    target_data_scaled = pd.DataFrame(y_scaler.transform(target_data), columns=target_data.columns)

    Now, feature_data_scaled and target_data_scaled are both DataFrames, scaled and ready to use

    yield feature_data_scaled.values, target_data_scaled.values

    %%

    from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Dropout, Input import tensorflow as tf from tensorflow.keras.layers import BatchNormalization

    def build_model(): input_shape = (6427,)

    model = Sequential([ Dense(6427, activation=‘relu’, input_shape = input_shape), Dropout(0.17), BatchNormalization(), Dense(4096, activation=‘relu’), Dropout(0.15), BatchNormalization(), Dense(2048, activation=‘relu’), Dropout(0.12), BatchNormalization(), Dense(1024, activation=‘relu’), Dropout(0.10), BatchNormalization(), Dense(512, activation=‘relu’), Dropout(0.05), BatchNormalization(), Dense(256, activation=‘relu’), BatchNormalization(), Dense(128, activation=‘relu’), BatchNormalization(), Dense(64, activation=‘relu’), BatchNormalization(), Dense(32, activation=‘relu’), BatchNormalization(), Dense(3), ])

    optimizer = keras.optimizers.Adam(learning_rate=0.006) model.compile(optimizer=‘adam’, loss=‘mse’, # Use Mean Squared Error for regression metrics=[‘mae’]) # Mean Absolute Error as an additional metric return model

    %%

    Instantiate the model

    model = build_model()

    model.summary()

    %%

    import warnings warnings.filterwarnings(action=‘ignore’, message=‘X has feature names, but StandardScaler was fitted without feature names’) train_generator = data_generator(file_path, batch_size,x_scaler=x_scaler_loaded,y_scaler=y_scaler_loaded)

    total_samples = 301617 # Assuming same example size train_samples = int(total_samples)

    steps_per_epoch = train_samples // batch_size

    Modify the model fitting call to include validation data

    model.fit( train_generator, steps_per_epoch=steps_per_epoch, epochs=45, )

    currently my model not using any validation dataset when training update the code properly and implement val set

  2. Assistant

    Adding validation to your model training process is crucial to understand whether your model is learning generalized patterns or merely memorizing the data it's been trained on. In deep learning, this process typically involves dividing your dataset into three subsets: training, validation, and sometimes a separate test dataset. The training set is used to train the model, the validation set is used to evaluate it during training, and the test set is used to evaluate the model after training has finished.

    To incorporate a validation dataset into your existing code, you need to create another data generator for the validation data and specify the validation_data parameter in your model.fit() call. This typically involves splitting your main dataset file beforehand into two separate files: one for training and one for validation. Alternatively, if you want to split your data on the fly, you would have to adjust your data_generator function to allow splitting the data dynamically, which could complicate the process, especially with large datasets handled in chunks.

    For simplicity, let's proceed with the presumption that you have already split your dataset into two CSV files: one for training (train_file_path) and another for validation (val_file_path). Here's how you can adjust your existing code for incorporating a validation set:

    1. Load Scalers (Assuming you've already done this correctly)

    x_scaler_loaded = joblib.load('nn_x_scaler.sav')
    y_scaler_loaded = joblib.load('nn_y_hlp5_scaler.sav')
    

    2. Update the Data Generator (No changes required)

    The current data_generator function you have is fine. It can be used for both training and validation data.

    3. Instantiate Training and Validation Generators

    Before fitting your model, create both training and validation generators:

    batch_size = 128  # Or any size that fits your model and memory capabilities
    train_file_path = 'path_to_your_train_dataset.csv'
    val_file_path = 'path_to_your_validation_dataset.csv'
    
    train_generator = data_generator(train_file_path, batch_size, x_scaler=x_scaler_loaded, y_scaler=y_scaler_loaded)
    val_generator = data_generator(val_file_path, batch_size, x_scaler=x_scaler_loaded, y_scaler=y_scaler_loaded)
    

    4. Calculate Steps per Epoch for Both Training and Validation

    You need to calculate steps_per_epoch (the number of batch iterations before a training epoch is considered finished) and validation_steps (the same but for validation data) based on your training and validation data size, respectively:

    total_train_samples = 301617  # Adjust based on your actual training data size
    total_val_samples = 75404  # Adjust based on your actual validation data size
    
    steps_per_epoch = total_train_samples // batch_size
    validation_steps = total_val_samples // batch_size
    

    5. Modify Model's fit() Method to Include Validation Data

    Finally, make sure to include the validation_data and validation_steps in your model.fit() call:

    history = model.fit(
        train_generator,
        steps_per_epoch=steps_per_epoch,
        epochs=45,
        validation_data=val_generator,
        validation_steps=validation_steps,
    )
    

    This approach ensures your model is being evaluated against unseen data (validation set) at the end of each epoch, giving you insights into how well your model generalizes. Remember to adjust total_train_samples and total_val_samples to match the actual sizes of your respective datasets.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.