can you modify below code to incorporate with 8 input features for prediction and in…
- User
can you modify below code to incorporate with 8 input features for prediction and in conversion to onnx format ?
import pickle import os from skl2onnx import convert_sklearn from skl2onnx.common.data_types import FloatTensorType import numpy as np
def convert_pickle_to_onnx(): """ Converts a scikit-learn model saved as a pickle file to ONNX format compatible with BigQuery.
Steps: 1. Load the scikit-learn model from a pickle file. 2. Define sample input data to determine the input shape and type. 3. Convert the model to ONNX format with a targeted operatorset version. 4. Save the ONNX model to a specified file path. """ # ------------------------- # Configuration Parameters # ------------------------- # Path to the input pickle file containing the scikit-learn model pickle_path = 'C:\\dsprojects_folder_jupyter\\Discharge-Prediction\\EPIC_REG1.pkl' # <-- Replace with your pickle file path # Path where the ONNX model will be saved onnx_path = 'bins_model1.onnx' # <-- Replace with your desired ONNX file path # Sample input data to define the input shape and type # Modify this based on the number of features your model expects sample_input = [0,33.0,13,0,0,0,146,20,10] # Example for a model expecting 4 features # Target opset version compatible with BigQuery (IR version 8) target_opset = 8 # ------------------------- # Conversion Process # ------------------------- # Check if the pickle file exists if not os.path.exists(pickle_path): raise FileNotFoundError(f"Pickle file not found: {pickle_path}") # Load the scikit-learn model from the pickle file with open(pickle_path, 'rb') as file: model = pickle.load(file) print(f"Loaded model from {pickle_path}") # Prepare the sample input for determining input types sample_input_np = np.array([sample_input], dtype=np.float32) # Shape: (1, n_features) # Define the initial type based on the sample inputinitial_type = [('input', FloatTensorType([None, sample_input_np.shape[1]]))]
initial_types = [ ('Gender', FloatTensorType([None, 1])), ('AgeAtAdmission', FloatTensorType([None, 1])), ('LevelOfCare', FloatTensorType([None, 1])), ('ICDProcedureID', FloatTensorType([None, 1])), ('AdmitType', FloatTensorType([None, 1])), ('SurgicalCaseFlag', FloatTensorType([None, 1])), ('AdmittingDiagnosisCode', FloatTensorType([None, 1])), ('PatService', FloatTensorType([None, 1])), ] # Convert the scikit-learn model to ONNX format with target operatorset version try: onnx_model = convert_sklearn( model, initial_types=initial_types, target_opset=target_opset, options = {'zipmap': False} ) print(f"Model successfully converted to ONNX format with target_opset={target_opset}.") except Exception as e: print(f"Error during conversion: {e}") return # Save the ONNX model to the specified file try: with open(onnx_path, "wb") as f: f.write(onnx_model.SerializeToString()) print(f"ONNX model saved at: {onnx_path}") except Exception as e: print(f"Error saving ONNX model: {e}")def verify_onnx_model(onnx_path, sample_input): """ Verifies the ONNX model by loading it and performing a test inference.
Parameters: onnx_path (str): Path to the ONNX model file. sample_input (list or np.ndarray): Sample input data for verification. """ import onnx import onnxruntime as ort # Load the ONNX model try: model = onnx.load(onnx_path) onnx.checker.check_model(model) print(f"ONNX model at {onnx_path} is valid.") except Exception as e: print(f"Invalid ONNX model: {e}") return # Initialize ONNX Runtime session try: session = ort.InferenceSession(onnx_path) print("ONNX Runtime session initialized.") except Exception as e: print(f"Failed to initialize ONNX Runtime session: {e}") return # Prepare input data input_name = session.get_inputs()[0].name input_data = np.array([sample_input], dtype=np.float32) # Perform inference try: result = session.run(None, {input_name: input_data}) print("Inference result:", result) except Exception as e: print(f"Error during inference: {e}")-------------------------
Execute the Conversion
-------------------------
if name == "main": try: convert_pickle_to_onnx()
# Optional: Verify the converted ONNX model # Replace with actual paths and sample inputs as needed onnx_model_path = 'path/to/save_model.onnx' # Ensure this matches 'onnx_path' in conversion sample_input = [5.1, 3.5, 1.4, 0.2] # Must match the sample_input used during conversion verify_onnx_model(onnx_model_path, sample_input) except Exception as e: print(f"An error occurred: {e}") - Assistant
Certainly! Below is the modified code tailored to handle 8 input features for prediction and convert the scikit-learn model to the ONNX format. Additionally, I've corrected the sample inputs and ensured consistency between the conversion and verification steps.
Key Modifications:
-
Eight Input Features:
- Updated the
sample_inputin bothconvert_pickle_to_onnxandverify_onnx_modelto include 8 features. - Ensured that
initial_typescorrectly correspond to the 8 feature names.
- Updated the
-
Consistency in Paths and Sample Inputs:
- Ensured that the
onnx_model_pathin the verification step matches theonnx_pathused during conversion. - Updated the sample input used in verification to match the 8-feature setup.
- Ensured that the
-
Code Enhancements:
- Added descriptive comments for clarity.
- Ensured proper exception handling and informative print statements.
Revised Code:
import pickle import os from skl2onnx import convert_sklearn from skl2onnx.common.data_types import FloatTensorType import numpy as np def convert_pickle_to_onnx(): """ Converts a scikit-learn model saved as a pickle file to ONNX format compatible with BigQuery. Steps: 1. Load the scikit-learn model from a pickle file. 2. Define sample input data to determine the input shape and type. 3. Convert the model to ONNX format with a targeted operatorset version. 4. Save the ONNX model to a specified file path. """ # ------------------------- # Configuration Parameters # ------------------------- # Path to the input pickle file containing the scikit-learn model pickle_path = 'C:\\dsprojects_folder_jupyter\\Discharge-Prediction\\EPIC_REG1.pkl' # <-- Replace with your pickle file path # Path where the ONNX model will be saved onnx_path = 'bins_model1.onnx' # <-- Replace with your desired ONNX file path # Sample input data to define the input shape and type # Updated to 8 features sample_input = [0, 33.0, 13, 0, 0, 0, 146, 20] # Example for a model expecting 8 features # Target opset version compatible with BigQuery (IR version 8) target_opset = 8 # ------------------------- # Conversion Process # ------------------------- # Check if the pickle file exists if not os.path.exists(pickle_path): raise FileNotFoundError(f"Pickle file not found: {pickle_path}") # Load the scikit-learn model from the pickle file with open(pickle_path, 'rb') as file: model = pickle.load(file) print(f"Loaded model from {pickle_path}") # Prepare the sample input for determining input types sample_input_np = np.array([sample_input], dtype=np.float32) # Shape: (1, n_features) # Define the initial types based on feature names and input shape # Replace the feature names with actual feature names used in your model initial_types = [ ('Gender', FloatTensorType([None, 1])), ('AgeAtAdmission', FloatTensorType([None, 1])), ('LevelOfCare', FloatTensorType([None, 1])), ('ICDProcedureID', FloatTensorType([None, 1])), ('AdmitType', FloatTensorType([None, 1])), ('SurgicalCaseFlag', FloatTensorType([None, 1])), ('AdmittingDiagnosisCode', FloatTensorType([None, 1])), ('PatService', FloatTensorType([None, 1])), ] # Convert the scikit-learn model to ONNX format with target operatorset version try: onnx_model = convert_sklearn( model, initial_types=initial_types, target_opset=target_opset, options={'zipmap': False} # Adjust options as needed ) print(f"Model successfully converted to ONNX format with target_opset={target_opset}.") except Exception as e: print(f"Error during conversion: {e}") return # Save the ONNX model to the specified file try: with open(onnx_path, "wb") as f: f.write(onnx_model.SerializeToString()) print(f"ONNX model saved at: {onnx_path}") except Exception as e: print(f"Error saving ONNX model: {e}") def verify_onnx_model(onnx_path, sample_input): """ Verifies the ONNX model by loading it and performing a test inference. Parameters: onnx_path (str): Path to the ONNX model file. sample_input (list or np.ndarray): Sample input data for verification. """ import onnx import onnxruntime as ort # Load the ONNX model try: model = onnx.load(onnx_path) onnx.checker.check_model(model) print(f"ONNX model at {onnx_path} is valid.") except Exception as e: print(f"Invalid ONNX model: {e}") return # Initialize ONNX Runtime session try: session = ort.InferenceSession(onnx_path) print("ONNX Runtime session initialized.") except Exception as e: print(f"Failed to initialize ONNX Runtime session: {e}") return # Prepare input data # Assuming the model expects inputs as separate features # Create a dictionary where each key corresponds to a feature name input_names = [input.name for input in session.get_inputs()] if len(input_names) != len(sample_input): print("Mismatch between number of inputs and sample input features.") return input_data = {name: np.array([[value]], dtype=np.float32) for name, value in zip(input_names, sample_input)} # Perform inference try: result = session.run(None, input_data) print("Inference result:", result) except Exception as e: print(f"Error during inference: {e}") # ------------------------- # Execute the Conversion # ------------------------- if __name__ == "__main__": try: convert_pickle_to_onnx() # Optional: Verify the converted ONNX model # Replace with actual paths and sample inputs as needed onnx_model_path = 'bins_model1.onnx' # Ensure this matches 'onnx_path' in conversion sample_input = [0, 33.0, 13, 0, 0, 0, 146, 20] # Must match the sample_input used during conversion verify_onnx_model(onnx_model_path, sample_input) except Exception as e: print(f"An error occurred: {e}")Detailed Explanation of Changes:
-
Sample Input Update:
- Conversion Function (
convert_pickle_to_onnx):sample_input = [0, 33.0, 13, 0, 0, 0, 146, 20] # 8 features - Verification Function (
verify_onnx_model):sample_input = [0, 33.0, 13, 0, 0, 0, 146, 20] # 8 features
- Conversion Function (
-
initial_typesDefinition:- Defined as a list of tuples, each representing a feature name and its corresponding data type.
- Ensure that the feature names (
'Gender','AgeAtAdmission', etc.) match the actual feature names used in your scikit-learn model.initial_types = [ ('Gender', FloatTensorType([None, 1])), ('AgeAtAdmission', FloatTensorType([None, 1])), ('LevelOfCare', FloatTensorType([None, 1])), ('ICDProcedureID', FloatTensorType([None, 1])), ('AdmitType', FloatTensorType([None, 1])), ('SurgicalCaseFlag', FloatTensorType([None, 1])), ('AdmittingDiagnosisCode', FloatTensorType([None, 1])), ('PatService', FloatTensorType([None, 1])), ]
-
Consistency in File Paths:
- In the
verify_onnx_modelfunction call within the__main__block, ensured thatonnx_model_pathmatches theonnx_pathspecified during conversion.onnx_model_path = 'bins_model1.onnx' # Matches 'onnx_path' in conversion
- In the
-
Inference Input Preparation:
-
Modified the
verify_onnx_modelfunction to handle multiple input features correctly by creating a dictionary mapping each feature name to its value. -
This assumes that the ONNX model expects each feature as a separate input. If your model expects a single tensor input with all features, you might need to adjust this accordingly.
input_names = [input.name for input in session.get_inputs()] if len(input_names) != len(sample_input): print("Mismatch between number of inputs and sample input features.") return input_data = {name: np.array([[value]], dtype=np.float32) for name, value in zip(input_names, sample_input)} -
Alternative: If your model expects a single tensor input (e.g., a 2D array with shape
(1, 8)), modify the input preparation as follows:input_name = session.get_inputs()[0].name input_data = {input_name: np.array([sample_input], dtype=np.float32)}
-
-
Error Handling and Informative Messages:
- Added checks to ensure that the number of input features matches between the sample input and the model's expected inputs.
- Enhanced print statements for better traceability during conversion and verification.
Additional Recommendations:
-
Feature Names Consistency: Ensure that the feature names specified in
initial_typesexactly match the feature names used during the model training phase. Mismatched names can lead to inference issues. -
Operatorset Version: While
target_opset=8is used for compatibility with BigQuery, ensure that this version supports all operators used in your scikit-learn model. You can refer to the ONNX Operatorset Documentation for more details. -
Model Verification: After conversion, it's crucial to verify the ONNX model's predictions against the original scikit-learn model to ensure consistency.
-
Environment Setup: Ensure that all necessary packages (
skl2onnx,onnx,onnxruntime, etc.) are installed in your Python environment.
Running the Script:
-
Set the Correct Paths:
- Update
pickle_pathwith the path to your scikit-learn pickle file. - Update
onnx_pathif you wish to save the ONNX model to a different location.
- Update
-
Execute the Script:
- Run the script in your Python environment. It will perform the conversion and then verify the generated ONNX model using the provided sample input.
-
Verify Output:
- The script will print messages indicating the success or failure of each step, along with the inference results from the ONNX model.
By following these modifications and recommendations, you should be able to successfully convert your scikit-learn model with 8 input features to the ONNX format and verify its functionality.
-