i have following code to train my NN model : %% from sklearn.preprocessing import…
- User
i have following code to train my NN model :
%%
from sklearn.preprocessing import StandardScaler import pandas as pd import numpy as np from tensorflow import keras import joblib
def calculate_targets_scaling_params(file_path): scaler = StandardScaler() for chunk in pd.read_csv(file_path, chunksize=10000): # Adjust chunksize based on your memory capacity filtered_chunk = chunk[[‘y_High_5d’,‘y_Low_5d’,‘y_Priority_5d’]]
scaler.partial_fit(filtered_chunk) # Accumulate means and variances
return scaler.mean_, scaler.var_
%%
x_scaler_loaded = joblib.load(‘nn_x_scaler.sav’) y_scaler_loaded = joblib.load(‘nn_y_hlp5_scaler.sav’)
def data_generator(file_path, batch_size, x_scaler, y_scaler): chunksize = batch_size
while True: # Loop forever, so the generator never terminates for chunk in pd.read_csv(file_path, chunksize=chunksize):
filtered_c = chunk.drop([‘Date’, ‘Symbol’], axis=1)
feature_data = filtered_c.drop([ ‘y_High_1d’, ‘y_Low_1d’, ‘y_Priority_1d’, ‘y_High_2d’, ‘y_Low_2d’, ‘y_Priority_2d’, ‘y_High_3d’, ‘y_Low_3d’, ‘y_Priority_3d’, ‘y_High_5d’, ‘y_Low_5d’, ‘y_Priority_5d’], axis=1) target_data = filtered_c[[‘y_High_5d’, ‘y_Low_5d’, ‘y_Priority_5d’]]
feature_data_scaled = pd.DataFrame(x_scaler.transform(feature_data), columns=feature_data.columns)
Assuming target_data also needs to be scaled, apply scaler separately
target_data_scaled = pd.DataFrame(y_scaler.transform(target_data), columns=target_data.columns)
Now, feature_data_scaled and target_data_scaled are both DataFrames, scaled and ready to use
yield feature_data_scaled.values, target_data_scaled.values
%%
from tensorflow.keras.models import Sequential from tensorflow.keras.layers import Dense, Dropout, Input import tensorflow as tf from tensorflow.keras.layers import BatchNormalization
def build_model(): input_shape = (6427,)
model = Sequential([ Dense(6427, activation=‘relu’, input_shape = input_shape), Dropout(0.17), BatchNormalization(), Dense(4096, activation=‘relu’), Dropout(0.15), BatchNormalization(), Dense(2048, activation=‘relu’), Dropout(0.12), BatchNormalization(), Dense(1024, activation=‘relu’), Dropout(0.10), BatchNormalization(), Dense(512, activation=‘relu’), Dropout(0.05), BatchNormalization(), Dense(256, activation=‘relu’), BatchNormalization(), Dense(128, activation=‘relu’), BatchNormalization(), Dense(64, activation=‘relu’), BatchNormalization(), Dense(32, activation=‘relu’), BatchNormalization(), Dense(3), ])
optimizer = keras.optimizers.Adam(learning_rate=0.006) model.compile(optimizer=‘adam’, loss=‘mse’, # Use Mean Squared Error for regression metrics=[‘mae’]) # Mean Absolute Error as an additional metric return model
%%
Instantiate the model
model = build_model()
model.summary()
%%
import warnings warnings.filterwarnings(action=‘ignore’, message=‘X has feature names, but StandardScaler was fitted without feature names’) train_generator = data_generator(file_path, batch_size,x_scaler=x_scaler_loaded,y_scaler=y_scaler_loaded)
total_samples = 301617 # Assuming same example size train_samples = int(total_samples)
steps_per_epoch = train_samples // batch_size
Modify the model fitting call to include validation data
model.fit( train_generator, steps_per_epoch=steps_per_epoch, epochs=45, )
currently my model not using any validation dataset when training change the code properly for and implement val set
- Assistant
To incorporate a validation dataset into your training process, you'll need to adjust your approach slightly. You can create a separate generator function for your validation data or split your existing data generator to yield both training and validation data depending on a parameter. However, to keep things simple and to follow the structure of your existing setup, I'll demonstrate incorporating the validation set by creating a separate generator for validation data.
Here are the steps you'll follow:
- Create a Validation Generator: Similar to your
data_generatorfunction, you'll create a separate generator to provide validation data. This can be done by either splitting your dataset beforehand and preparing a separate file for validation data or by introducing logic in your data generator to separate a chunk of data as validation data. - Load Validation Data: If you're using a separate file or a predefined portion of your file for validation, ensure that part of the data is not used in training.
- Modify Model Fitting: Update the
model.fit()call to include your validation data.
Below is an adjusted version of your code incorporating these steps:
Create Validation Data Generator
First, ensure you've prepared validation data. Here, I'm assuming you're generating validation data in a similar manner to your training data but from a possibly separate dataset or a split portion. The function might remain mostly the same, so I'll skip directly to using it since the definition will mimic
data_generator.Implement Validation Data in Training
file_path_train = 'train_data.csv' # Adjust with actual training dataset path file_path_val = 'validation_data.csv' # Adjust with actual validation dataset path batch_size = 64 # Or any batch size you find appropriate train_generator = data_generator(file_path_train, batch_size, x_scaler=x_scaler_loaded, y_scaler=y_scaler_loaded) val_generator = data_generator(file_path_val, batch_size, x_scaler=x_scaler_loaded, y_scaler=y_scaler_loaded) # Assuming the number of samples in your validation dataset total_val_samples = 50000 # Update with the correct amount val_steps = total_val_samples // batch_size # Fit the model model.fit( train_generator, steps_per_epoch=steps_per_epoch, epochs=45, validation_data=val_generator, validation_steps=val_steps, )By specifying both
validation_dataandvalidation_steps, Keras will evaluate your model on the validation data after each epoch, giving you further insight into its performance on unseen data.Ensure that the validation generator doesn't loop indefinitely like the training generator to avoid stepping through validation data endlessly during each epoch's validation phase. The provided
fitmethod setup assumes both your generators yield batches indefinitely, which is fine for training due to thesteps_per_epochparameter but ensure your validation batches cover your entire validation set exactly once per epoch. - Create a Validation Generator: Similar to your
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.