Following this instruction : 1. Strictly adhere to PEP-8 standards, ensuring code is…
- User
Following this instruction :
- Strictly adhere to PEP-8 standards, ensuring code is clean, well-structured, and follows Python's style guide for readability.
- Use appropriate modules like
typingfor type hints, ensuring code clarity, maintainability, and leveraging Python's standard libraries effectively. - Focus on time, space, robustness, scalability; write optimized, efficient code that can handle large datasets and scale seamlessly.
- Implement comprehensive error handling, catching exceptions gracefully, providing meaningful error messages, and ensuring code robustness under unexpected conditions.
- Ensure the developer agent writes advanced, maintainable Python code, balancing performance and readability while adhering to best practices. t
TASK: we have generalised below code , best generalised , scalable, robustness, import numpy as np import pandas as pd import matplotlib.pyplot as plt import torch import torch.nn as nn import torchvision.transforms as T from datasets import load_dataset from transformers import AutoImageProcessor, ViTForImageClassification from transformers import Trainer, TrainingArguments import evaluate
dataset = load_dataset('pcuenq/oxford-pets') classes = dataset['train'].unique('label')
def show_samples(ds,rows=2,cols=4): samples = ds.shuffle().select(np.arange(rowscols)) fig = plt.figure(figsize=(cols4,rows4)) for i in range(rowscols): img = samples[i]['image'] label = samples[i]['label'] fig.add_subplot(rows,cols,i+1) plt.imshow(img) plt.title(label) plt.axis('off')
show_samples(dataset['train'],rows=3,cols=5) dataset = dataset['train'].train_test_split(train_size=0.8) model_name = "vit-base-patch16-224" model_checkpoint = f"google/{model_name}"
processor = AutoImageProcessor.from_pretrained(model_checkpoint) processor label2id = {c:idx for idx,c in enumerate(classes)} id2label = {idx:c for idx,c in enumerate(classes)} def transforms(batch): batch['image'] = [x.convert('RGB') for x in batch['image']] inputs = processor([x for x in batch['image']],return_tensors='pt') inputs['labels']=[label2id[y] for y in batch['label']] return inputs dataset = dataset.with_transform(transforms) def collate_fn(batch): return { 'pixel_values': torch.stack([x['pixel_values'] for x in batch]), 'labels': torch.tensor([x['labels'] for x in batch]) } accuracy = evaluate.load('accuracy') def compute_metrics(eval_preds): logits, labels = eval_preds predictions = np.argmax(logits,axis=1) acc = accuracy.compute(predictions=predictions, references=labels) return acc def print_trainable_parameters(model): trainable_params = 0 all_param = 0 for _, param in model.named_parameters(): all_param += param.numel() if param.requires_grad: trainable_params += param.numel() print( f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param:.2f}" ) from transformers import AutoModelForImageClassification, TrainingArguments, Trainer
model = AutoModelForImageClassification.from_pretrained( model_checkpoint, label2id=label2id, id2label=id2label, ignore_mismatched_sizes=True, ) print_trainable_parameters(model) from peft import LoraConfig, get_peft_model
config = LoraConfig( r=16, lora_alpha=16, target_modules=["query", "value"], lora_dropout=0.1, bias="none", modules_to_save=["classifier"], )
lora_model = get_peft_model(model, config) print_trainable_parameters(lora_model) batch_size = 128
args = TrainingArguments( f"{model_checkpoint}-finetuned-lora-oxford-pets", per_device_train_batch_size=batch_size, learning_rate=5e-3, num_train_epochs=5, per_device_eval_batch_size=batch_size, gradient_accumulation_steps=4, logging_steps=10, save_total_limit=2, evaluation_strategy="epoch", save_strategy="epoch", metric_for_best_model="accuracy", report_to='tensorboard', fp16=True, push_to_hub=True, remove_unused_columns=False, load_best_model_at_end=True, ) trainer = Trainer( model=model, args=args, data_collator=collate_fn, compute_metrics=compute_metrics, train_dataset=dataset["train"], eval_dataset=dataset["test"], tokenizer=processor ) trainer.train() repo_name = f"alanahmet/{model_name}-finetuned-lora-oxfordPets" lora_model.push_to_hub(repo_name) from peft import PeftConfig, PeftModel
config = PeftConfig.from_pretrained(repo_name) model = AutoModelForImageClassification.from_pretrained( config.base_model_name_or_path, label2id=label2id, id2label=id2label, ignore_mismatched_sizes=True, # provide this in case you're planning to fine-tune an already fine-tuned checkpoint )
Load the Lora model
inference_model = PeftModel.from_pretrained(model, repo_name) from PIL import Image import requests
url = "https://huggingface.co/datasets/alanahmet/LoRA-pets-dataset/resolve/main/shiba_inu_136.jpg" image = Image.open(requests.get(url, stream=True).raw) image image_processor = AutoImageProcessor.from_pretrained(model_checkpoint) encoding = image_processor(image.convert("RGB"), return_tensors="pt") print(encoding.pixel_values.shape) import torch
forward pass
with torch.no_grad(): outputs = inference_model(**encoding) logits = outputs.logits
predicted_class_idx = logits.argmax(-1).item() print("Predicted class:", inference_model.config.id2label[predicted_class_idx])
- Assistant
Certainly! Below is a refactored version of your script that adheres to the provided instructions. The code has been modularized for better readability and maintainability, includes type hints using the
typingmodule, implements comprehensive error handling, and follows PEP-8 standards. Additionally, it is optimized for scalability and robustness, ensuring it can handle large datasets efficiently.Key Improvements:
- Modular Design: The script is organized into classes and functions, separating concerns such as data handling, model setup, training, and inference.
- Type Hinting: Utilizes the
typingmodule to add type hints for better code clarity and maintainability. - Error Handling: Comprehensive try-except blocks are added to catch and handle potential exceptions gracefully.
- Configuration Management: Parameters are centralized in a configuration dataclass for easier adjustments and scalability.
- Logging: Implements logging to track the progress and debug if necessary.
- PEP-8 Compliance: Ensures the code follows Python's style guide for consistency and readability.
import logging from dataclasses import dataclass, field from typing import Any, Dict, List, Optional, Tuple import numpy as np import torch import torch.nn as nn import torch.optim as optim import torchvision.transforms as T from datasets import Dataset, load_dataset from matplotlib import pyplot as plt from PIL import Image from transformers import ( AutoImageProcessor, AutoModelForImageClassification, Trainer, TrainingArguments, ) import evaluate from peft import LoraConfig, PeftModel, get_peft_model import requests import sys # Configure logging logging.basicConfig( level=logging.INFO, format="%(asctime)s [%(levelname)s] %(name)s: %(message)s", handlers=[logging.StreamHandler(sys.stdout)], ) logger = logging.getLogger(__name__) @dataclass class Config: dataset_name: str = "pcuenq/oxford-pets" model_name: str = "vit-base-patch16-224" train_size: float = 0.8 batch_size: int = 128 learning_rate: float = 5e-3 num_epochs: int = 5 gradient_accumulation_steps: int = 4 logging_steps: int = 10 save_total_limit: int = 2 evaluation_strategy: str = "epoch" save_strategy: str = "epoch" metric_for_best_model: str = "accuracy" report_to: str = "tensorboard" fp16: bool = True push_to_hub: bool = True remove_unused_columns: bool = False load_best_model_at_end: bool = True lora_r: int = 16 lora_alpha: int = 16 lora_dropout: float = 0.1 lora_target_modules: List[str] = field( default_factory=lambda: ["query", "value"] ) lora_bias: str = "none" lora_modules_to_save: List[str] = field(default_factory=lambda: ["classifier"]) inference_image_url: str = ( "https://huggingface.co/datasets/alanahmet/LoRA-pets-dataset/resolve/main/shiba_inu_136.jpg" ) repository_name: str = "alanahmet/vit-finetuned-lora-oxford-pets" class OxfordPetsClassifier: def __init__(self, config: Config): self.config = config self.dataset: Optional[Dict[str, Dataset]] = None self.classes: Optional[List[str]] = None self.label2id: Dict[str, int] = {} self.id2label: Dict[int, str] = {} self.processor: Optional[AutoImageProcessor] = None self.model: Optional[AutoModelForImageClassification] = None self.trainer: Optional[Trainer] = None self.accuracy = evaluate.load("accuracy") logger.info("OxfordPetsClassifier initialized with config: %s", config) def load_data(self) -> None: try: logger.info("Loading dataset: %s", self.config.dataset_name) self.dataset = load_dataset(self.config.dataset_name) if self.dataset is None: raise ValueError("Dataset loading returned None.") self.classes = self.dataset["train"].unique("label") if not self.classes: raise ValueError("No classes found in the dataset.") self.label2id = {label: idx for idx, label in enumerate(self.classes)} self.id2label = {idx: label for idx, label in enumerate(self.classes)} logger.info("Dataset loaded successfully with %d classes.", len(self.classes)) except Exception as e: logger.error("Failed to load dataset: %s", e) raise def show_samples(self, ds: Dataset, rows: int = 2, cols: int = 4) -> None: try: samples = ds.shuffle(seed=42).select(np.arange(rows * cols)) fig = plt.figure(figsize=(cols * 4, rows * 4)) for i in range(rows * cols): img = samples[i]["image"] label = samples[i]["label"] fig.add_subplot(rows, cols, i + 1) plt.imshow(img) plt.title(self.id2label.get(label, "Unknown")) plt.axis("off") plt.tight_layout() plt.show() logger.info("Displayed sample images.") except Exception as e: logger.error("Failed to display samples: %s", e) raise def preprocess_data(self) -> None: try: if not self.dataset: raise ValueError("Dataset not loaded.") logger.info("Splitting dataset into train and test sets.") self.dataset = self.dataset["train"].train_test_split( train_size=self.config.train_size, seed=42 ) logger.info( "Dataset split into train (%d) and test (%d).", len(self.dataset["train"]), len(self.dataset["test"]), ) model_checkpoint = f"google/{self.config.model_name}" self.processor = AutoImageProcessor.from_pretrained(model_checkpoint) logger.info("Image processor loaded from checkpoint: %s", model_checkpoint) def transform(batch: Dict[str, Any]) -> Dict[str, Any]: batch["image"] = [x.convert("RGB") for x in batch["image"]] inputs = self.processor( [x for x in batch["image"]], return_tensors="pt", ) inputs["labels"] = [self.label2id[y] for y in batch["label"]] return inputs self.dataset = self.dataset.with_transform(transform) logger.info("Data transformations applied.") except Exception as e: logger.error("Failed to preprocess data: %s", e) raise @staticmethod def collate_fn(batch: List[Dict[str, Any]]) -> Dict[str, torch.Tensor]: try: pixel_values = torch.stack([x["pixel_values"] for x in batch]) labels = torch.tensor([x["labels"] for x in batch]) return {"pixel_values": pixel_values, "labels": labels} except Exception as e: logger.error("Collate function failed: %s", e) raise @staticmethod def compute_metrics(eval_preds: Tuple[np.ndarray, np.ndarray]) -> Dict[str, float]: try: logits, labels = eval_preds predictions = np.argmax(logits, axis=1) acc = evaluate.load("accuracy").compute(predictions=predictions, references=labels) return acc except Exception as e: logger.error("Metric computation failed: %s", e) raise def print_trainable_parameters(self, model: nn.Module) -> None: try: trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad) all_params = sum(p.numel() for p in model.parameters()) trainable_percent = 100 * trainable_params / all_params if all_params else 0 logger.info( "Trainable params: %d || All params: %d || Trainable%%: %.2f", trainable_params, all_params, trainable_percent, ) except Exception as e: logger.error("Failed to print trainable parameters: %s", e) raise def setup_model(self) -> None: try: model_checkpoint = f"google/{self.config.model_name}" self.model = AutoModelForImageClassification.from_pretrained( model_checkpoint, label2id=self.label2id, id2label=self.id2label, ignore_mismatched_sizes=True, ) logger.info("Base model loaded from checkpoint: %s", model_checkpoint) self.print_trainable_parameters(self.model) lora_config = LoraConfig( r=self.config.lora_r, lora_alpha=self.config.lora_alpha, target_modules=self.config.lora_target_modules, lora_dropout=self.config.lora_dropout, bias=self.config.lora_bias, modules_to_save=self.config.lora_modules_to_save, ) self.model = get_peft_model(self.model, lora_config) logger.info("LoRA configuration applied to the model.") self.print_trainable_parameters(self.model) except Exception as e: logger.error("Failed to set up the model: %s", e) raise def configure_trainer(self) -> None: try: training_args = TrainingArguments( output_dir=f"{self.config.model_name}-finetuned-lora-oxford-pets", per_device_train_batch_size=self.config.batch_size, learning_rate=self.config.learning_rate, num_train_epochs=self.config.num_epochs, per_device_eval_batch_size=self.config.batch_size, gradient_accumulation_steps=self.config.gradient_accumulation_steps, logging_steps=self.config.logging_steps, save_total_limit=self.config.save_total_limit, evaluation_strategy=self.config.evaluation_strategy, save_strategy=self.config.save_strategy, metric_for_best_model=self.config.metric_for_best_model, report_to=self.config.report_to, fp16=self.config.fp16, push_to_hub=self.config.push_to_hub, remove_unused_columns=self.config.remove_unused_columns, load_best_model_at_end=self.config.load_best_model_at_end, ) logger.info("TrainingArguments configured.") self.trainer = Trainer( model=self.model, args=training_args, data_collator=self.collate_fn, compute_metrics=self.compute_metrics, train_dataset=self.dataset["train"], eval_dataset=self.dataset["test"], tokenizer=self.processor, ) logger.info("Trainer initialized.") except Exception as e: logger.error("Failed to configure trainer: %s", e) raise def train(self) -> None: if not self.trainer: raise ValueError("Trainer is not configured.") try: logger.info("Starting training.") self.trainer.train() logger.info("Training completed.") except Exception as e: logger.error("Training failed: %s", e) raise def push_model_to_hub(self) -> None: try: if not self.model: raise ValueError("Model is not initialized.") logger.info("Pushing model to hub: %s", self.config.repository_name) self.model.push_to_hub(self.config.repository_name) logger.info("Model successfully pushed to hub.") except Exception as e: logger.error("Failed to push model to hub: %s", e) raise def load_inference_model(self) -> PeftModel: try: logger.info("Loading inference model from hub: %s", self.config.repository_name) config = PeftConfig.from_pretrained(self.config.repository_name) base_model = AutoModelForImageClassification.from_pretrained( config.base_model_name_or_path, label2id=self.label2id, id2label=self.id2label, ignore_mismatched_sizes=True, ) inference_model = PeftModel.from_pretrained(base_model, self.config.repository_name) logger.info("Inference model loaded successfully.") return inference_model except Exception as e: logger.error("Failed to load inference model: %s", e) raise def infer(self, model: PeftModel) -> None: try: logger.info("Performing inference on the sample image.") image = Image.open(requests.get(self.config.inference_image_url, stream=True).raw) encoding = self.processor(image.convert("RGB"), return_tensors="pt") logger.info("Image encoding shape: %s", encoding.pixel_values.shape) with torch.no_grad(): outputs = model(**encoding) logits = outputs.logits predicted_class_idx = logits.argmax(-1).item() predicted_class = self.id2label.get(predicted_class_idx, "Unknown") logger.info("Predicted class: %s", predicted_class) print(f"Predicted class: {predicted_class}") except Exception as e: logger.error("Inference failed: %s", e) raise def run(self) -> None: try: self.load_data() self.show_samples(self.dataset["train"], rows=3, cols=5) self.preprocess_data() self.setup_model() self.configure_trainer() self.train() self.push_model_to_hub() inference_model = self.load_inference_model() self.infer(inference_model) except Exception as e: logger.error("An error occurred during execution: %s", e) raise def main() -> None: config = Config() classifier = OxfordPetsClassifier(config) classifier.run() if __name__ == "__main__": main()Explanation of the Refactored Code
-
Configuration Management:
- A
Configdataclass encapsulates all configurable parameters, making it easy to adjust settings without modifying the core logic.
- A
-
Logging:
- The
loggingmodule provides detailed logs at various stages, aiding in debugging and monitoring the process.
- The
-
OxfordPetsClassifier Class:
- Initialization: Sets up initial variables and loads the evaluation metric.
- Data Loading (
load_data): Loads the dataset and prepares label mappings with error handling. - Sample Visualization (
show_samples): Displays a grid of sample images from the dataset. - Data Preprocessing (
preprocess_data): Splits the dataset, applies image transformations, and prepares it for the model. - Collate Function (
collate_fn): Batches the data appropriately for the model. - Metrics Calculation (
compute_metrics): Calculates the accuracy metric for evaluation. - Trainable Parameters Display (
print_trainable_parameters): Logs the number of trainable parameters in the model. - Model Setup (
setup_model): Loads the base model, applies LoRA configuration, and logs relevant information. - Trainer Configuration (
configure_trainer): Sets up theTrainerwith all necessary arguments and datasets. - Training (
train): Executes the training loop. - Model Upload (
push_model_to_hub): Pushes the fine-tuned model to Hugging Face Hub. - Inference Model Loading (
load_inference_model): Loads the model for inference. - Inference (
infer): Performs inference on a sample image and prints the predicted class. - Execution (
run): Coordinates the entire workflow by calling the above methods in sequence.
-
Error Handling:
- Each method contains try-except blocks to catch and log exceptions, ensuring that errors are handled gracefully and informative messages are provided.
-
Type Hinting:
- All methods and functions include type hints, enhancing code clarity and assisting with static type checking.
-
PEP-8 Compliance:
- The code follows PEP-8 guidelines, including proper naming conventions, spacing, and structure.
-
Scalability and Maintainability:
- The modular design allows for easy extension and modification.
- The use of configuration and logging ensures that the system can be scaled and maintained effectively.
Dependencies
Ensure that the following packages are installed in your environment. You can install them using
pip:pip install numpy pandas matplotlib torch torchvision transformers datasets evaluate peft pillow requestsRunning the Script
Simply execute the script using Python:
python oxford_pets_classifier.pyReplace
oxford_pets_classifier.pywith the filename you saved the script as.Conclusion
This refactored script provides a robust, scalable, and maintainable framework for training and deploying an image classification model on the Oxford Pets dataset using a Vision Transformer with LoRA fine-tuning. It adheres to best practices in Python development, ensuring that the code is clean, efficient, and easy to understand.