i have following code to train my NN model : %% from sklearn.preprocessing import…
- User
i have following code to train my NN model :
%%
from sklearn.preprocessing import StandardScaler import pandas as pd import numpy as np from tensorflow import keras import joblib
def calculate_targets_scaling_params(file_path): scaler = StandardScaler() for chunk in pd.read_csv(file_path, chunksize=10000): # Adjust chunksize based on your memory capacity filtered_chunk = chunk[[‘y_High_5d’,‘y_Low_5d’,‘y_Priority_5d’]]
scaler.partial_fit(filtered_chunk) # Accumulate means and variances
return scaler.mean_, scaler.var_
%%
x_scaler_loaded = joblib.load(‘nn_x_scaler.sav’) y_scaler_loaded = joblib.load(‘nn_y_hlp5_scaler.sav’)
def data_generator(file_path, batch_size, x_scaler, y_scaler): chunksize = batch_size
while True: # Loop forever, so the generator never terminates for chunk in pd.read_csv(file_path, chunksize=chunksize):
filtered_c = chunk.drop([‘Date’, ‘Symbol’], axis=1)
feature_data = filtered_c.drop([ ‘y_High_1d’, ‘y_Low_1d’, ‘y_Priority_1d’, ‘y_High_2d’, ‘y_Low_2d’, ‘y_Priority_2d’, ‘y_High_3d’, ‘y_Low_3d’, ‘y_Priority_3d’, ‘y_High_5d’, ‘y_Low_5d’, ‘y_Priority_5d’], axis=1) target_data = filtered_c[[‘y_High_5d’, ‘y_Low_5d’, ‘y_Priority_5d’]]
feature_data_scaled = pd.DataFrame(x_scaler.transform(feature_data), columns=feature_data.columns)
Assuming target_data also needs to be scaled, apply scaler separately
target_data_scaled = pd.DataFrame(y_scaler.transform(target_data), columns=target_data.columns)
Now, feature_data_scaled and target_data_scaled are both DataFrames, scaled and ready to use
yield feature_data_scaled.values, target_data_scaled.values
%%
from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Dropout, Input import tensorflow as tf from tensorflow.keras.layers import BatchNormalization
def build_model(): input_shape = (6427,)
model = Sequential([ Dense(6427, activation=‘relu’, input_shape = input_shape), Dropout(0.17), BatchNormalization(), Dense(4096, activation=‘relu’), Dropout(0.15), BatchNormalization(), Dense(2048, activation=‘relu’), Dropout(0.12), BatchNormalization(), Dense(1024, activation=‘relu’), Dropout(0.10), BatchNormalization(), Dense(512, activation=‘relu’), Dropout(0.05), BatchNormalization(), Dense(256, activation=‘relu’), BatchNormalization(), Dense(128, activation=‘relu’), BatchNormalization(), Dense(64, activation=‘relu’), BatchNormalization(), Dense(32, activation=‘relu’), BatchNormalization(), Dense(3), ])
optimizer = keras.optimizers.Adam(learning_rate=0.006) model.compile(optimizer=‘adam’, loss=‘mse’, # Use Mean Squared Error for regression metrics=[‘mae’]) # Mean Absolute Error as an additional metric return model
%%
Instantiate the model
model = build_model()
model.summary()
%%
import warnings warnings.filterwarnings(action=‘ignore’, message=‘X has feature names, but StandardScaler was fitted without feature names’) train_generator = data_generator(file_path, batch_size,x_scaler=x_scaler_loaded,y_scaler=y_scaler_loaded)
total_samples = 301617 # Assuming same example size train_samples = int(total_samples)
steps_per_epoch = train_samples // batch_size
Modify the model fitting call to include validation data
model.fit( train_generator, steps_per_epoch=steps_per_epoch, epochs=45, )
currently my model not using any validation dataset when training update the code properly and implement val set
- Assistant
Adding validation to your model training process is crucial to understand whether your model is learning generalized patterns or merely memorizing the data it's been trained on. In deep learning, this process typically involves dividing your dataset into three subsets: training, validation, and sometimes a separate test dataset. The training set is used to train the model, the validation set is used to evaluate it during training, and the test set is used to evaluate the model after training has finished.
To incorporate a validation dataset into your existing code, you need to create another data generator for the validation data and specify the
validation_dataparameter in yourmodel.fit()call. This typically involves splitting your main dataset file beforehand into two separate files: one for training and one for validation. Alternatively, if you want to split your data on the fly, you would have to adjust yourdata_generatorfunction to allow splitting the data dynamically, which could complicate the process, especially with large datasets handled in chunks.For simplicity, let's proceed with the presumption that you have already split your dataset into two CSV files: one for training (
train_file_path) and another for validation (val_file_path). Here's how you can adjust your existing code for incorporating a validation set:1. Load Scalers (Assuming you've already done this correctly)
x_scaler_loaded = joblib.load('nn_x_scaler.sav') y_scaler_loaded = joblib.load('nn_y_hlp5_scaler.sav')2. Update the Data Generator (No changes required)
The current
data_generatorfunction you have is fine. It can be used for both training and validation data.3. Instantiate Training and Validation Generators
Before fitting your model, create both training and validation generators:
batch_size = 128 # Or any size that fits your model and memory capabilities train_file_path = 'path_to_your_train_dataset.csv' val_file_path = 'path_to_your_validation_dataset.csv' train_generator = data_generator(train_file_path, batch_size, x_scaler=x_scaler_loaded, y_scaler=y_scaler_loaded) val_generator = data_generator(val_file_path, batch_size, x_scaler=x_scaler_loaded, y_scaler=y_scaler_loaded)4. Calculate Steps per Epoch for Both Training and Validation
You need to calculate
steps_per_epoch(the number of batch iterations before a training epoch is considered finished) andvalidation_steps(the same but for validation data) based on your training and validation data size, respectively:total_train_samples = 301617 # Adjust based on your actual training data size total_val_samples = 75404 # Adjust based on your actual validation data size steps_per_epoch = total_train_samples // batch_size validation_steps = total_val_samples // batch_size5. Modify Model's
fit()Method to Include Validation DataFinally, make sure to include the
validation_dataandvalidation_stepsin yourmodel.fit()call:history = model.fit( train_generator, steps_per_epoch=steps_per_epoch, epochs=45, validation_data=val_generator, validation_steps=validation_steps, )This approach ensures your model is being evaluated against unseen data (validation set) at the end of each epoch, giving you insights into how well your model generalizes. Remember to adjust
total_train_samplesandtotal_val_samplesto match the actual sizes of your respective datasets.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.