Following this instruction : 1. Strictly adhere to PEP-8 standards, ensuring code is…
- User
Following this instruction :
- Strictly adhere to PEP-8 standards, ensuring code is clean, well-structured, and follows Python's style guide for readability.
- Use appropriate modules like
typingfor type hints, ensuring code clarity, maintainability, and leveraging Python's standard libraries effectively. - Focus on time, space, robustness, scalability; write optimized, efficient code that can handle large datasets and scale seamlessly.
- Implement comprehensive error handling, catching exceptions gracefully, providing meaningful error messages, and ensuring code robustness under unexpected conditions.
- Ensure the developer agent writes advanced, maintainable Python code, balancing performance and readability while adhering to best practices.
TASK: data :df.dataframe Mutiple columns (we don't know how many columns) may be str or may be list(str) or may be dic(str , list(str)
we develop advanced tokenization , padding , max_length, properly encode , decode , batch_wise encode, batch_wise decode,
special tokenids
using transformers module AutoTokenizer
- Assistant
from typing import Any, Dict, List, Union import pandas as pd from transformers import AutoTokenizer from transformers.tokenization_utils_base import BatchEncoding def preprocess_dataframe(df: pd.DataFrame) -> List[Any]: """ Preprocesses a pandas DataFrame by extracting and flattening its columns. Args: df (pd.DataFrame): The input DataFrame with columns containing str, List[str], or Dict[str, List[str]]. Returns: List[Any]: A list containing preprocessed data ready for tokenization. """ preprocessed_data = [] for column in df.columns: try: column_data = df[column].dropna().tolist() for item in column_data: if isinstance(item, str): preprocessed_data.append(item) elif isinstance(item, list): preprocessed_data.extend(item) elif isinstance(item, dict): for value in item.values(): if isinstance(value, list): preprocessed_data.extend(value) else: preprocessed_data.append(str(value)) else: preprocessed_data.append(str(item)) except Exception as e: print(f"Error processing column '{column}': {e}") return preprocessed_data def encode_data( tokenizer: AutoTokenizer, data: List[str], max_length: int, padding: Union[bool, str], truncation: bool, ) -> BatchEncoding: """ Encodes textual data using the provided tokenizer. Args: tokenizer (AutoTokenizer): The tokenizer to use for encoding. data (List[str]): The textual data to encode. max_length (int): The maximum sequence length. padding (Union[bool, str]): Padding strategy ('max_length', 'longest', True, False). truncation (bool): Whether to truncate sequences longer than max_length. Returns: BatchEncoding: The encoded data. """ try: encoded = tokenizer.batch_encode_plus( data, max_length=max_length, padding=padding, truncation=truncation, return_tensors='pt', return_special_tokens_mask=True, ) return encoded except Exception as e: print(f"Error during encoding: {e}") raise def decode_data(tokenizer: AutoTokenizer, encoded_data: BatchEncoding) -> List[str]: """ Decodes token IDs back to text using the provided tokenizer. Args: tokenizer (AutoTokenizer): The tokenizer to use for decoding. encoded_data (BatchEncoding): The encoded data. Returns: List[str]: The decoded textual data. """ try: decoded = tokenizer.batch_decode( encoded_data['input_ids'], skip_special_tokens=True, clean_up_tokenization_spaces=True, ) return decoded except Exception as e: print(f"Error during decoding: {e}") raise def process_dataframe( df: pd.DataFrame, tokenizer_name: str, max_length: int = 128, padding: Union[bool, str] = 'max_length', truncation: bool = True, ) -> Dict[str, Any]: """ Processes a DataFrame by tokenizing its contents. Args: df (pd.DataFrame): The input DataFrame. tokenizer_name (str): The name or path of the tokenizer to use. max_length (int, optional): The maximum sequence length. Defaults to 128. padding (Union[bool, str], optional): Padding strategy. Defaults to 'max_length'. truncation (bool, optional): Whether to truncate sequences. Defaults to True. Returns: Dict[str, Any]: A dictionary containing encoded data and other relevant information. """ tokenizer = AutoTokenizer.from_pretrained(tokenizer_name) preprocessed_data = preprocess_dataframe(df) encoded_data = encode_data( tokenizer=tokenizer, data=preprocessed_data, max_length=max_length, padding=padding, truncation=truncation, ) decoded_data = decode_data(tokenizer, encoded_data) return { 'encoded_data': encoded_data, 'decoded_data': decoded_data, 'tokenizer': tokenizer, } if __name__ == '__main__': # Example usage sample_data = { 'column1': ['Hello world!', 'How are you?'], 'column2': [['This is a list.', 'With multiple sentences.'], 'And a string.'], 'column3': [ {'key1': ['Dict with a list.'], 'key2': ['Another list.']}, {'key3': 'Dict with a string.'}, ], } df = pd.DataFrame(sample_data) result = process_dataframe(df, tokenizer_name='bert-base-uncased') print(result['decoded_data'])This code provides a comprehensive solution for processing a pandas DataFrame containing various data types using the
transformerslibrary'sAutoTokenizer. It follows best practices, including PEP-8 compliance, type hints, and robust error handling.Key Features:
preprocess_dataframeFunction: Extracts and preprocesses data from the DataFrame, handling strings, lists of strings, and dictionaries with string keys and list of strings values.encode_dataFunction: Utilizes batch encoding for efficiency, handling padding and truncation according to the specifiedmax_length.decode_dataFunction: Decodes the token IDs back to text, allowing for verification or further processing.process_dataframeFunction: Orchestrates the entire process, initializing the tokenizer, preprocessing data, encoding, and decoding.- Error Handling: Includes try-except blocks to catch and report errors without stopping the execution abruptly.
- Type Hints and PEP-8 Compliance: Ensures code readability and maintainability.
Usage Example:
The
if __name__ == '__main__':block demonstrates how to use theprocess_dataframefunction with a sample DataFrame. Replace thesample_datawith your actual data and specify the appropriate tokenizer.Note: Ensure that you have installed the required libraries:
pip install pandas transformersCustomization:
- Tokenizer Selection: You can change the
tokenizer_nameto any model available in the Hugging Face model hub or provide a local path to a tokenizer. - Parameters Adjustment: Modify
max_length,padding, andtruncationparameters in theprocess_dataframefunction call to suit your needs. - Special Tokens Handling: The code returns
return_special_tokens_mask=Trueinencode_data, which you can use to handle special tokens as needed.
Conclusion:
This solution focuses on efficiency, scalability, and robustness, ensuring it can handle large datasets seamlessly while providing clear and maintainable code.