WHY THIS ERROR ? PS C:\Users\siddh> &…
- User
WHY THIS ERROR ? PS C:\Users\siddh> & C:/Users/siddh/AppData/Local/Programs/Python/Python311/python.exe c:/Users/swarn/python_codes/python_try/hugging/predictive_32.py C:\Users\siddh\AppData\Local\Programs\Python\Python311\Lib\site-packages\yfinance\base.py:48: FutureWarning: The default dtype for empty Series will be 'object' instead of 'float64' in a future version. Specify a dtype explicitly to silence this warning. _empty_series = pd.Series() Program started... Do you want to 'Train' the models or 'Use' existing models? Enter 'Train' or 'Use': use Enter the stock symbols separated by commas (e.g., AAPL, GOOGL, MSFT): NSE:SUZLON Enter the prediction horizon in days (e.g., 5): 15 Enter the number of years of data to use for training (e.g., 5 or 10): 5 Training the Models on 2 stocks data Estimated Time in minutes = 2 Time of start = 2024-09-23 00:12:04.286726 Loading saved models... RandomForestRegressor model loaded from c:/users/swarn/python_codes/python_try/hugging/new_models/predictive_32_RandomForestRegressor_model.pkl XGBoostRegressor model loaded from c:/users/swarn/python_codes/python_try/hugging/new_models/predictive_32_XGBoostRegressor_model.pkl Scaler loaded from scaler.pkl Label encoder loaded from label_encoder.pkl
Available models:
- Random Forest Regressor - RMSE: 0.116375
- XGBoost Regressor - RMSE: 0.096497 Enter the number corresponding to the model you want to use: 1 You have selected: Random Forest Regressor Fetching stock data... [*******************100%%] 1 of 1 completed | 0/2 [00:00<?, ?it/s] [*******************100%%] 1 of 1 completed█████████████ | 1/2 [00:02<00:02, 2.78s/it] 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:02<00:00, 1.45s/it] Calculating technical indicators on recent data... 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:00<00:00, 197.96it/s] Traceback (most recent call last): File "C:\Users\siddh\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\utils_encode.py", line 225, in _encode return _map_to_integer(values, uniques) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\siddh\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\utils_encode.py", line 165, in _map_to_integer return np.array([table[v] for v in values]) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\siddh\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\utils_encode.py", line 165, in return np.array([table[v] for v in values]) ~~~~~^^^ File "C:\Users\siddh\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\utils_encode.py", line 159, in missing raise KeyError(key) KeyError: 'WAAREERTL.BO'
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "c:\Users\swarn\python_codes\python_try\hugging\predictive_32.py", line 919, in main_machine_learning() File "c:\Users\swarn\python_codes\python_try\hugging\predictive_32.py", line 800, in main_machine_learning recent_dataset['Symbol_Encoded'] = le.transform(recent_dataset['Symbol']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\siddh\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\preprocessing_label.py", line 137, in transform return encode(y, uniques=self.classes) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\siddh\AppData\Local\Programs\Python\Python311\Lib\site-packages\sklearn\utils_encode.py", line 227, in _encode raise ValueError(f"y contains previously unseen labels: {str(e)}") ValueError: y contains previously unseen labels: 'WAAREERTL.BO'
- Assistant
The error you're encountering:
ValueError: y contains previously unseen labels: 'WAAREERTL.BO'occurs because the
LabelEncoderyou're using has encountered a label ('WAAREERTL.BO') that it wasn't trained on. Here's a breakdown of why this happens and how you can fix it.
Understanding the Error
1. Label Encoding in Machine Learning:
- LabelEncoder: Used to convert categorical text data into numerical labels.
- Fitting vs. Transforming:
le.fit(data): Learns all unique categories (labels) from the data.le.transform(data): Converts categories in the data to numerical labels based on what was learned during fitting.
2. Cause of the Error:
- Unseen Labels: When you attempt to transform labels that the encoder hasn't seen during fitting, it raises a
ValueError. - In your case, 'WAAREERTL.BO' is a stock symbol that wasn't present in the data used to fit the
LabelEncoder.
Why 'WAAREERTL.BO' is Causing the Issue
- Unexpected Symbol: Although you entered
'NSE:SUZLON'as the stock symbol, the dataset being processed includes 'WAAREERTL.BO'. - Data Fetching Logic: It's possible that your data fetching mechanism is pulling additional stock symbols besides the ones you specified.
How to Fix the Issue
Option 1: Update the Label Encoder
Refit the LabelEncoder with New Labels
If new symbols are expected, you should refit the
LabelEncoderto include them.-
Retrieve All Unique Symbols:
- Combine the symbols from the original training data and the new data.
- Example:
all_symbols = np.unique(np.concatenate((le.classes_, recent_dataset['Symbol'].unique())))
-
Refit the LabelEncoder:
- Fit the encoder with the updated list of symbols.
le.fit(all_symbols)
- Fit the encoder with the updated list of symbols.
-
Transform the Symbol Column:
- Now, transform the 'Symbol' column without encountering unseen labels.
recent_dataset['Symbol_Encoded'] = le.transform(recent_dataset['Symbol'])
- Now, transform the 'Symbol' column without encountering unseen labels.
Note: Re-training the encoder may require re-training your models to ensure consistency.
Option 2: Handle Unseen Labels at Prediction Time
Use a Custom Encoder that Can Handle Unseen Labels
You can modify your label encoding step to assign a default value to unseen labels.
-
Create a Custom Transformer:
from sklearn.preprocessing import LabelEncoder import numpy as np class CustomLabelEncoder(LabelEncoder): def transform(self, y): # Map unknown labels to a specific value, e.g., -1 unknown_label = -1 known = np.in1d(y, self.classes_) transformed = np.where(known, super().transform(y), unknown_label) return transformed -
Use the CustomLabelEncoder:
- Replace your existing
LabelEncoderwithCustomLabelEncoder. - Load the encoder as before:
le = joblib.load('label_encoder.pkl') le = CustomLabelEncoder() le.classes_ = np.load('classes.npy', allow_pickle=True) # Assuming you saved the classes - Transform the data:
recent_dataset['Symbol_Encoded'] = le.transform(recent_dataset['Symbol'])
- Replace your existing
Note: Assigning a default value to unseen labels may affect your model's predictions. Ensure your model can handle such cases appropriately.
Option 3: Ensure Only Known Symbols are Processed
Filter the Dataset to Include Only Known Symbols
-
Get Known Symbols from the Encoder:
known_symbols = le.classes_ -
Filter the Recent Dataset:
recent_dataset = recent_dataset[recent_dataset['Symbol'].isin(known_symbols)] -
Proceed with Transformation and Prediction:
recent_dataset['Symbol_Encoded'] = le.transform(recent_dataset['Symbol'])
Note: This approach ignores data with unknown symbols, which might be acceptable depending on your application's requirements.
Verify Your Data Fetching Logic
Since the symbol 'WAAREERTL.BO' appears unexpectedly, it's important to:
-
Check Data Sources: Ensure that the data fetching function only retrieves data for the symbols you've specified.
-
Inspect
recent_dataset:- Print out the unique symbols to see which ones are included:
print(recent_dataset['Symbol'].unique())
- Print out the unique symbols to see which ones are included:
-
Adjust Fetching Parameters: Modify your data fetching code if it's pulling in extra symbols unintentionally.
Example Fix
Here's how you might adjust your code incorporating the custom encoder:
# Load your model and encoder model = joblib.load('your_model.pkl') le = joblib.load('label_encoder.pkl') # Create and configure the custom encoder class CustomLabelEncoder(LabelEncoder): def transform(self, y): unknown_label = -1 known = np.in1d(y, self.classes_) transformed = np.where(known, super().transform(y), unknown_label) return transformed le_custom = CustomLabelEncoder() le_custom.classes_ = le.classes_ # Fetch recent data # ... your data fetching code ... # Transform the Symbol column recent_dataset['Symbol_Encoded'] = le_custom.transform(recent_dataset['Symbol']) # Check for any unknown labels assigned as -1 if (recent_dataset['Symbol_Encoded'] == -1).any(): print("Warning: Some symbols were not recognized by the LabelEncoder and have been set to -1.") # Proceed with prediction only for known symbols known_data = recent_dataset[recent_dataset['Symbol_Encoded'] != -1] # Make predictions # ... your prediction code ...
Final Recommendations
- Consistency is Key: Ensure the symbols used during training and prediction are consistent.
- Update Your Models When Necessary: If new symbols are becoming common, consider retraining your model to include them.
- Robust Error Handling: Implement checks and warnings to handle unexpected data gracefully.
Let me know if you need further clarification or assistance with modifying your code.