Make this script onnx campatible at least schema import numpy as np import pandas as pd…
- User
Make this script onnx campatible at least schema
import numpy as np import pandas as pd from sklearn.base import BaseEstimator, TransformerMixin from sklearn.pipeline import Pipeline from sklearn.compose import ColumnTransformer from sklearn.preprocessing import StandardScaler === Transformer qui reshape en (n_samples, 2, size) ===
class ReshapeBack(BaseEstimator, TransformerMixin): def init(self, n_samples): self.n_samples = n_samples # on passe n_samples à l'init
def fit(self, X, y=None): self.n_features = X.shape[1] # devrait être 2 self.size = X.shape[0] // self.n_samples return self
def transform(self, X): reshaped = X.reshape(self.n_samples, self.size, self.n_features) return reshaped.transpose(0, 2, 1) # (n_samples, 2, size)
=== Exemple de données ===
df = pd.DataFrame({ 't10': [[1, 2, 3], [4, 5, 6], [7, 8, 9]], 'temp': [[10, 20, 30], [40, 50, 60], [70, 80, 90]], }) === Pipeline ===
pipe = Pipeline([ ('flatten_and_scale', Pipeline([ ('flatten', ColumnTransformer([ ('feature', FlattenListsPaired(), ['t10', 'temp']), ])), ('scaler', StandardScaler()), ])), ('reshape_back', ReshapeBack(n_samples=len(df))) ]) === Exécution ===
result = pipe.fit_transform(df)
print(result.shape) # (3, 2, 3) print(result)
Assistant
To make the provided script ONNX-compatible, we need to address several key aspects:
- Custom Transformers: The
ReshapeBackclass needs a custom ONNX converter. - Input Data Handling: The initial DataFrame contains lists, which are not directly compatible with ONNX. We'll separate the flattening step.
- ONNX Schema Compliance: Ensure all operations can be represented in the ONNX schema.
Here’s the revised script with ONNX compatibility:
import numpy as np import pandas as pd from sklearn.base import BaseEstimator, TransformerMixin from sklearn.pipeline import Pipeline from sklearn.compose import ColumnTransformer from sklearn.preprocessing import StandardScaler from skl2onnx import convert_sklearn, update_registered_converter from skl2onnx.common.data_types import FloatTensorType from skl2onnx.algebra.onnx_ops import OnnxReshape, OnnxTranspose from onnx import TensorProto # === Custom Transformer: Flatten Lists (Python Preprocessing) === class FlattenListsPaired(BaseEstimator, TransformerMixin): def fit(self, X, y=None): return self def transform(self, X): if isinstance(X, pd.DataFrame): X = X.values n_features = X.shape[1] flattened = np.concatenate([X[:, i].ravel() for i in range(n_features)], axis=0).reshape(-1, n_features) return flattened.astype(np.float32) # === Custom Transformer: ReshapeBack === class ReshapeBack(BaseEstimator, TransformerMixin): def __init__(self, n_samples): self.n_samples = n_samples def fit(self, X, y=None): self.n_features = X.shape[1] self.size = X.shape[0] // self.n_samples return self def transform(self, X): return X.reshape(self.n_samples, self.size, self.n_features).transpose(0, 2, 1) # === ONNX Converter for ReshapeBack === def reshape_back_converter(scope, operator, container): op = operator.raw_operator op_type = 'Reshape' op_name = scope.get_unique_operator_name(op_type) # Calculate target shape: (n_samples, size, n_features) target_shape = np.array([op.n_samples, op.size, op.n_features], dtype=np.int64) # Add target shape as initializer shape_name = scope.get_unique_variable_name('target_shape') container.add_initializer(shape_name, TensorProto.INT64, target_shape.shape, target_shape) # Reshape operation reshaped = scope.get_unique_variable_name('reshaped') container.add_node(op_type, [operator.input_full_names[0], shape_name], [reshaped], name=op_name) # Transpose operation (0, 2, 1) attrs = {'name': scope.get_unique_operator_name('Transpose'), 'perm': [0, 2, 1]} container.add_node('Transpose', reshaped, operator.output_full_names, **attrs) # Register the converter update_registered_converter(ReshapeBack, 'CustomReshapeBack', reshape_back_converter) # === Example Data === df = pd.DataFrame({ 't10': [[1, 2, 3], [4, 5, 6], [7, 8, 9]], 'temp': [[10, 20, 30], [40, 50, 60], [70, 80, 90]], }) # === Separate Flattening (Python Preprocessing) === flatten_transformer = ColumnTransformer([ ('flatten', FlattenListsPaired(), ['t10', 'temp']), ]) X_flattened = flatten_transformer.fit_transform(df) # === ONNX-Compatible Pipeline === onnx_pipe = Pipeline([ ('scaler', StandardScaler()), ('reshape_back', ReshapeBack(n_samples=len(df))) ]) # Fit and transform using flattened data result_sklearn = onnx_pipe.fit_transform(X_flattened) # === ONNX Conversion === initial_type = [('float_input', FloatTensorType([X_flattened.shape[0], X_flattened.shape[1]]))] onnx_model = convert_sklearn(onnx_pipe, initial_types=initial_type, target_opset=11) # Save the model with open('pipeline.onnx', 'wb') as f: f.write(onnx_model.SerializeToString()) print("ONNX model saved to 'pipeline.onnx'")Key Changes and Explanations:
- FlattenListsPaired:
- This transformer converts DataFrame columns containing lists into a flat NumPy array.
- Note: This step must be done in Python as ONNX cannot handle DataFrames with list-like objects directly.
- ReshapeBack Converter:
- A custom converter is registered using
update_registered_converter. - The converter performs two ONNX operations:
Reshape: Transforms the input into(n_samples, size, n_features).Transpose: Rearranges dimensions to(n_samples, n_features, size).
- A custom converter is registered using
- Pipeline Separation:
- The original pipeline is split into two parts:
- Python Preprocessing:
FlattenListsPaired(handled outside ONNX). - ONNX Pipeline:
StandardScalerandReshapeBack(converted to ONNX).
- Python Preprocessing:
- The original pipeline is split into two parts:
- ONNX Conversion:
- The
convert_sklearnfunction generates an ONNX model from the compatible pipeline. - The model is saved in the ONNX format.
- The
Usage Notes:
- Input Data: The ONNX model expects flattened numerical data as input (shape:
(total_elements, n_features)). - Inference: To use the ONNX model, load it with an ONNX runtime (e.g.,
onnxruntime) and pass the flattened data.
This approach ensures the core processing pipeline is ONNX-compatible while handling the initial data transformation in Python.
- Custom Transformers: The
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.