Following this instruction : 1. Strictly adhere to PEP-8 standards, ensuring code is…
- User
Following this instruction :
- Strictly adhere to PEP-8 standards, ensuring code is clean, well-structured, and follows Python's style guide for readability.
- Use appropriate modules like
typingfor type hints, ensuring code clarity, maintainability, and leveraging Python's standard libraries effectively. - Focus on time, space, robustness, scalability; write optimized, efficient code that can handle large datasets and scale seamlessly.
- Implement comprehensive error handling, catching exceptions gracefully, providing meaningful error messages, and ensuring code robustness under unexpected conditions.
- Ensure the developer agent writes advanced, maintainable Python code, balancing performance and readability while adhering to best practices.
TASK :
Please process the input data provided in one of the following formats:
- List with shape (batch_size, int)
- List with shape (batch_size, channel, height, width)
- List with shape (batch_size, channel, depth, height, width)
- List with shape (batch_size, seq_length)
Ensure that the data is appropriately preprocessed and formatted for model ingestion, handling each format's specific dimensional requirements. Advanced Prompting Tips:
Specify Data Formats Clearly: When dealing with multiple input shapes, explicitly outline each possible format. This ensures the model accurately identifies and processes the data based on its structure. Incorporate Conditional Instructions: Use conditional language to guide the model on different processing paths depending on the input format. For example, "If the input has three dimensions, treat it as image data; if it has five, handle it as volumetric data," etc. Provide Contextual Examples: Enhance understanding by including examples for each data format. This helps the model recognize patterns and apply the correct preprocessing steps for each specific input type.
- Assistant
from typing import List, Any import numpy as np def preprocess_data(data: List[Any]) -> np.ndarray: """ Preprocesses input data based on its shape and formats it for model ingestion. Parameters: data (List[Any]): Input data in one of the following formats: 1. List with shape (batch_size, int) 2. List with shape (batch_size, channel, height, width) 3. List with shape (batch_size, channel, depth, height, width) 4. List with shape (batch_size, seq_length) Returns: np.ndarray: Preprocessed data suitable for model ingestion. Raises: ValueError: If input data does not match any of the expected formats. """ try: data_array = np.array(data) shape = data_array.shape if len(shape) == 2: # Could be (batch_size, int) or (batch_size, seq_length) if np.issubdtype(data_array.dtype, np.integer) and shape[1] == 1: # Format 1: (batch_size, int) preprocessed_data = data_array.astype(np.int32) else: # Format 4: (batch_size, seq_length) preprocessed_data = preprocess_sequences(data_array) elif len(shape) == 4: # Format 2: (batch_size, channel, height, width) preprocessed_data = preprocess_images(data_array) elif len(shape) == 5: # Format 3: (batch_size, channel, depth, height, width) preprocessed_data = preprocess_volumetric(data_array) else: raise ValueError(f"Unexpected data shape: {shape}") return preprocessed_data except Exception as e: raise ValueError(f"Error preprocessing data: {e}") from e def preprocess_sequences(data_array: np.ndarray) -> np.ndarray: """ Preprocesses sequence data by padding/truncating to a fixed length. Parameters: data_array (np.ndarray): Array with shape (batch_size, seq_length) Returns: np.ndarray: Preprocessed sequence data. """ max_length = 100 # Example fixed sequence length batch_size = data_array.shape[0] # Initialize an array of the fixed sequence length preprocessed_data = np.zeros((batch_size, max_length), dtype=data_array.dtype) for i, sequence in enumerate(data_array): seq_length = min(len(sequence), max_length) preprocessed_data[i, :seq_length] = sequence[:seq_length] return preprocessed_data def preprocess_images(data_array: np.ndarray) -> np.ndarray: """ Preprocesses image data by normalizing pixel values. Parameters: data_array (np.ndarray): Array with shape (batch_size, channel, height, width) Returns: np.ndarray: Preprocessed image data. """ # Normalize pixel values to [0, 1] preprocessed_data = data_array.astype(np.float32) / 255.0 return preprocessed_data def preprocess_volumetric(data_array: np.ndarray) -> np.ndarray: """ Preprocesses volumetric data by normalizing and resizing. Parameters: data_array (np.ndarray): Array with shape (batch_size, channel, depth, height, width) Returns: np.ndarray: Preprocessed volumetric data. """ # Normalize volumetric data preprocessed_data = data_array.astype(np.float32) max_value = np.max(preprocessed_data) if max_value > 0: preprocessed_data /= max_value # Example: Resize depth dimension if necessary (omitted for brevity) return preprocessed_dataExplanation:
-
Imports and Type Hints: The code uses the
typingmodule for type hints andnumpyfor efficient numerical computations. -
Main Function (
preprocess_data):- Converts the input list to a NumPy array for easier shape inspection and manipulation.
- Determines the shape of the input data to identify its format.
- Uses conditional logic to handle each data format appropriately.
- Includes comprehensive error handling to catch and report issues during preprocessing.
-
Preprocessing Functions:
- Sequences (
preprocess_sequences):- Pads or truncates sequences to a fixed length (
max_length) for uniformity. - Initializes a zero-filled array and copies each sequence up to
max_length.
- Pads or truncates sequences to a fixed length (
- Images (
preprocess_images):- Normalizes pixel values to the range [0, 1] by dividing by 255.
- Converts data to
float32for consistency in model processing.
- Volumetric Data (
preprocess_volumetric):- Normalizes the data based on the maximum value to scale all inputs between 0 and 1.
- Placeholder comments indicate where additional preprocessing (e.g., resizing) can occur.
- Sequences (
-
Error Handling:
- The
try-exceptblock inpreprocess_datacaptures any exceptions raised during processing. - Errors provide clear messages indicating the nature of the issue, aiding in debugging and robustness.
- The
-
Scalability and Efficiency:
- Utilizes vectorized operations with NumPy for efficient computation over large datasets.
- Avoids unnecessary data copies and conversions to conserve memory.
- Designed to handle large batches by processing data in a batched manner.
-
PEP-8 Compliance:
- Follows standard naming conventions and code structure for readability.
- Includes docstrings for all functions, detailing parameters, returns, and potential exceptions.
This modular approach allows for easy maintenance and extension of the code. Additional preprocessing steps can be added to each function as needed, and new data formats can be handled by adding more conditional branches and corresponding functions.
-