Following this instruction : 1. Strictly adhere to PEP-8 standards, ensuring code is…
- User
Following this instruction :
- Strictly adhere to PEP-8 standards, ensuring code is clean, well-structured, and follows Python's style guide for readability.
- Use appropriate modules like
typingfor type hints, ensuring code clarity, maintainability, and leveraging Python's standard libraries effectively. - Focus on time, space, robustness, scalability; write optimized, efficient code that can handle large datasets and scale seamlessly.
- Implement comprehensive error handling, catching exceptions gracefully, providing meaningful error messages, and ensuring code robustness under unexpected conditions.
- Ensure the developer agent writes advanced, maintainable Python code, balancing performance and readability while adhering to best practices.
TASK : Generalised robustness metrics Pipeline (logits or predictions or loss values)
NOTE : using the only torch module we have implement below scratch
Problem Statement:
- Image Classification Accuracy: Percentage of correctly classified images. Precision: Ratio of true positives to the sum of true positives and false positives. Recall (Sensitivity): Ratio of true positives to the sum of true positives and false negatives. F1 Score: Harmonic mean of precision and recall. AUC-ROC: Area under the receiver operating characteristic curve; measures the trade-off between true positive rate and false positive rate. Confusion Matrix: Matrix showing the performance of the model in predicting each class. Top-k Accuracy: Accuracy of the model when the true label is within the top-k predicted labels. Log Loss: Measures the uncertainty of predictions; lower values indicate better performance. Mean Average Precision (mAP): Average precision across all classes; useful for multi-class problems. Class Distribution: Analyzing the distribution of predictions across classes to detect any bias.
- Assistant
Certainly! Below is a comprehensive implementation of a Generalized Robustness Metrics Pipeline for image classification using only the
torchmodule. This pipeline calculates various metrics such as Accuracy, Precision, Recall, F1 Score, AUC-ROC, Confusion Matrix, Top-k Accuracy, Log Loss, Mean Average Precision (mAP), and Class Distribution.The implementation adheres strictly to PEP-8 standards, utilizes type hints for clarity, and incorporates robust error handling to ensure reliability and scalability. The code is structured in a maintainable and efficient manner, suitable for handling large datasets.
Implementation Overview
- MetricsPipeline Class: Central class that accumulates predictions and labels and computes the required metrics.
- Initialization: Validates inputs and initializes internal structures for metric computation.
- Update Method: Allows incremental updates with batches of predictions and labels.
- Compute Methods: Individual methods to compute each metric based on the accumulated data.
- Error Handling: Comprehensive checks and informative error messages to handle unexpected inputs gracefully.
- Type Hints: Utilizes the
typingmodule to enhance code clarity and maintainability.
Code Implementation
import torch from torch import Tensor from typing import Optional, Tuple, Dict, List import math class MetricsPipeline: """ A pipeline to compute various robustness metrics for image classification tasks. """ def __init__(self, num_classes: int, top_k: Optional[int] = 5) -> None: """ Initializes the MetricsPipeline. Args: num_classes (int): Number of classes in the classification task. top_k (Optional[int]): The 'k' in Top-k Accuracy. Defaults to 5. Raises: ValueError: If num_classes is not a positive integer. """ if not isinstance(num_classes, int) or num_classes <= 0: raise ValueError("num_classes must be a positive integer.") if top_k is not None and (not isinstance(top_k, int) or top_k <= 0): raise ValueError("top_k must be a positive integer.") self.num_classes: int = num_classes self.top_k: int = top_k if top_k is not None else 1 self.reset() def reset(self) -> None: """ Resets all stored predictions and labels. """ self.predictions: List[Tensor] = [] self.labels: List[Tensor] = [] self.class_counts: Tensor = torch.zeros(self.num_classes, dtype=torch.long) self.confusion_matrix: Tensor = torch.zeros( self.num_classes, self.num_classes, dtype=torch.long ) self.true_positive: Tensor = torch.zeros(self.num_classes, dtype=torch.float) self.false_positive: Tensor = torch.zeros(self.num_classes, dtype=torch.float) self.false_negative: Tensor = torch.zeros(self.num_classes, dtype=torch.float) self.logits: List[Tensor] = [] def update(self, logits: Tensor, labels: Tensor) -> None: """ Updates the pipeline with a new batch of logits and labels. Args: logits (Tensor): Logits output from the model (batch_size, num_classes). labels (Tensor): Ground truth labels (batch_size). Raises: ValueError: If logits and labels have mismatched dimensions or invalid shapes. """ if not isinstance(logits, Tensor): raise TypeError("Logits must be a torch.Tensor.") if not isinstance(labels, Tensor): raise TypeError("Labels must be a torch.Tensor.") if logits.dim() != 2: raise ValueError("Logits must be a 2D tensor of shape (batch_size, num_classes).") if labels.dim() != 1: raise ValueError("Labels must be a 1D tensor of shape (batch_size).") if logits.size(0) != labels.size(0): raise ValueError( "Batch size of logits and labels must be the same." ) if logits.size(1) != self.num_classes: raise ValueError( f"Logits should have {self.num_classes} classes." ) with torch.no_grad(): self.logits.append(logits) self.labels.append(labels) _, preds = torch.max(logits, dim=1) self.predictions.append(preds) self.labels.append(labels) for cls in range(self.num_classes): true_cls = labels == cls pred_cls = preds == cls self.true_positive[cls] += torch.sum(true_cls & pred_cls).item() self.false_positive[cls] += torch.sum(~true_cls & pred_cls).item() self.false_negative[cls] += torch.sum(true_cls & ~pred_cls).item() self.confusion_matrix[cls] += torch.bincount( preds[true_cls], minlength=self.num_classes ) self.class_counts[cls] += torch.sum(true_cls).item() def compute_accuracy(self) -> float: """ Computes the overall accuracy. Returns: float: Accuracy as a percentage. """ correct = sum( (pred == label).sum().item() for pred, label in zip(self.predictions, self.labels) ) total = sum(label.size(0) for label in self.labels) return 100.0 * correct / total if total > 0 else 0.0 def compute_precision(self) -> Optional[float]: """ Computes the macro-precision across all classes. Returns: Optional[float]: Macro-precision. Returns None if undefined. """ precision = 0.0 count = 0 for cls in range(self.num_classes): tp = self.true_positive[cls] fp = self.false_positive[cls] if tp + fp > 0: precision += tp / (tp + fp) count += 1 return (precision / count) * 100.0 if count > 0 else None def compute_recall(self) -> Optional[float]: """ Computes the macro-recall across all classes. Returns: Optional[float]: Macro-recall. Returns None if undefined. """ recall = 0.0 count = 0 for cls in range(self.num_classes): tp = self.true_positive[cls] fn = self.false_negative[cls] if tp + fn > 0: recall += tp / (tp + fn) count += 1 return (recall / count) * 100.0 if count > 0 else None def compute_f1_score(self) -> Optional[float]: """ Computes the macro F1 score across all classes. Returns: Optional[float]: Macro F1 score. Returns None if undefined. """ precision = self.compute_precision() recall = self.compute_recall() if precision is None or recall is None or precision + recall == 0: return None return 2 * (precision * recall) / (precision + recall) def compute_confusion_matrix(self) -> Tensor: """ Computes the confusion matrix. Returns: Tensor: Confusion matrix of shape (num_classes, num_classes). """ return self.confusion_matrix.clone() def compute_top_k_accuracy(self, k: Optional[int] = None) -> float: """ Computes the Top-k accuracy. Args: k (Optional[int]): The 'k' in Top-k Accuracy. Defaults to self.top_k. Returns: float: Top-k accuracy as a percentage. Raises: ValueError: If k is not a positive integer. """ if k is None: k = self.top_k if not isinstance(k, int) or k <= 0: raise ValueError("k must be a positive integer.") correct = 0 total = 0 for logits_batch, labels_batch in zip(self.logits, self.labels): topk = torch.topk(logits_batch, k, dim=1).indices labels_expanded = labels_batch.view(-1, 1).expand_as(topk) correct += torch.sum(topk == labels_expanded).item() total += labels_batch.size(0) return 100.0 * correct / total if total > 0 else 0.0 def compute_log_loss(self) -> Optional[float]: """ Computes the log loss (cross-entropy loss). Returns: Optional[float]: Log loss. Returns None if undefined. """ total_loss = 0.0 total_samples = 0 for logits_batch, labels_batch in zip(self.logits, self.labels): # Apply log softmax for numerical stability log_probs = torch.log_softmax(logits_batch, dim=1) # Gather the log probabilities for the true classes true_log_probs = log_probs[torch.arange(labels_batch.size(0)), labels_batch] total_loss += -torch.sum(true_log_probs).item() total_samples += labels_batch.size(0) return total_loss / total_samples if total_samples > 0 else None def compute_auc_roc(self) -> Optional[Dict[int, float]]: """ Computes the AUC-ROC for each class in a one-vs-rest manner. Returns: Optional[Dict[int, float]]: A dictionary mapping class indices to their AUC-ROC scores. Returns None if undefined. """ try: # Concatenate all logits and labels logits = torch.cat(self.logits, dim=0) labels = torch.cat(self.labels, dim=0) probabilities = torch.softmax(logits, dim=1) auc_scores: Dict[int, float] = {} for cls in range(self.num_classes): # Binary labels for the current class binary_labels = (labels == cls).float() scores = probabilities[:, cls] # Sort scores and corresponding labels sorted_scores, sorted_indices = torch.sort(scores, descending=True) sorted_labels = binary_labels[sorted_indices] tp = 0.0 fp = 0.0 prev_score = -math.inf auc = 0.0 f_prev = 0.0 t_prev = 0.0 for score, label in zip(sorted_scores, sorted_labels): if score != prev_score: auc += (fp - f_prev) * (t_prev + tp) / 2 f_prev = fp t_prev = tp prev_score = score if label == 1.0: tp += 1.0 else: fp += 1.0 auc += (fp - f_prev) * (t_prev + tp) / 2 total_positive = binary_labels.sum().item() total_negative = binary_labels.size(0) - total_positive if total_positive > 0 and total_negative > 0: auc_scores[cls] = auc / (total_positive * total_negative) else: auc_scores[cls] = float('nan') return auc_scores except Exception: return None def compute_mean_average_precision(self) -> Optional[float]: """ Computes the Mean Average Precision (mAP) across all classes. Returns: Optional[float]: mAP value. Returns None if undefined. """ try: # Concatenate all logits and labels logits = torch.cat(self.logits, dim=0) labels = torch.cat(self.labels, dim=0) probabilities = torch.softmax(logits, dim=1) average_precisions: List[float] = [] for cls in range(self.num_classes): # Binary labels for the current class binary_labels = (labels == cls).float() scores = probabilities[:, cls] # Sort scores and corresponding labels sorted_scores, sorted_indices = torch.sort(scores, descending=True) sorted_labels = binary_labels[sorted_indices] # Compute precision at each true positive cumulative_true = torch.cumsum(sorted_labels, dim=0) precision_at_k = sorted_labels * cumulative_true / (torch.arange(1, sorted_labels.size(0) + 1).float()) average_precision = torch.sum(precision_at_k) / sorted_labels.sum() if sorted_labels.sum() > 0 else float('nan') average_precision = average_precision.item() if not math.isnan(average_precision): average_precisions.append(average_precision) if len(average_precisions) == 0: return None mean_ap = sum(average_precisions) / len(average_precisions) return mean_ap except Exception: return None def compute_class_distribution(self) -> Dict[int, int]: """ Computes the distribution of predictions across classes. Returns: Dict[int, int]: A dictionary mapping class indices to their predicted counts. """ distribution = {} total_predictions = sum([pred.size(0) for pred in self.predictions]) for cls in range(self.num_classes): count = int(self.true_positive[cls] + self.false_positive[cls]) distribution[cls] = count return distribution def get_all_metrics(self) -> Dict[str, Optional[float]]: """ Computes and returns all metrics. Returns: Dict[str, Optional[float]]: A dictionary containing all computed metrics. """ metrics = { "Accuracy": self.compute_accuracy(), "Precision": self.compute_precision(), "Recall": self.compute_recall(), "F1_Score": self.compute_f1_score(), "Top-k_Accuracy": self.compute_top_k_accuracy(), "Log_Loss": self.compute_log_loss(), "Mean_Average_Precision": self.compute_mean_average_precision(), "AUC_ROC": self.compute_auc_roc(), "Confusion_Matrix": self.compute_confusion_matrix(), "Class_Distribution": self.compute_class_distribution(), } return metrics # Example Usage if __name__ == "__main__": try: num_classes = 3 pipeline = MetricsPipeline(num_classes=num_classes, top_k=2) # Simulated logits and labels logits_batch1 = torch.tensor([[2.5, 0.3, 0.2], [0.1, 2.1, 0.4], [0.2, 0.3, 2.2], [2.0, 1.0, 0.5]]) labels_batch1 = torch.tensor([0, 1, 2, 0]) logits_batch2 = torch.tensor([[1.5, 2.2, 0.3], [0.3, 0.2, 2.5], [2.1, 0.4, 0.5], [0.2, 2.3, 1.0]]) labels_batch2 = torch.tensor([1, 2, 0, 1]) # Update pipeline with batches pipeline.update(logits_batch1, labels_batch1) pipeline.update(logits_batch2, labels_batch2) # Compute all metrics metrics = pipeline.get_all_metrics() # Display metrics for metric, value in metrics.items(): if metric == "Confusion_Matrix": print(f"{metric}:\n{value}\n") elif metric == "AUC_ROC": print(f"{metric}:") for cls, auc in value.items(): print(f" Class {cls}: {auc:.4f}") print() elif metric == "Class_Distribution": print(f"{metric}: {value}\n") else: print(f"{metric}: {value:.2f}%\n" if isinstance(value, float) else f"{metric}: {value}\n") except Exception as e: print(f"An error occurred: {e}")Explanation of the Implementation
-
Class Initialization:
- The
MetricsPipelineclass is initialized with the number of classes and an optionaltop_kparameter for Top-k Accuracy. - It validates the input parameters and initializes necessary tensors and lists to store predictions, labels, and intermediate counts required for metric computations.
- The
-
Reset Method:
- The
resetmethod clears all stored data, allowing the pipeline to be reused for different datasets or evaluations.
- The
-
Update Method:
- The
updatemethod accepts batches of logits and labels. - It performs validation on the input tensors to ensure they have appropriate shapes and compatible dimensions.
- It then processes the logits to obtain predictions and updates internal counts for true positives, false positives, false negatives, and the confusion matrix.
- The
-
Metric Computation Methods:
- Accuracy: Calculates the percentage of correctly classified samples.
- Precision: Computes the macro-average precision across all classes.
- Recall: Computes the macro-average recall across all classes.
- F1 Score: Calculates the harmonic mean of precision and recall.
- Confusion Matrix: Constructs a matrix showing the performance of the model per class.
- Top-k Accuracy: Determines if the true label is within the top-k predicted probabilities.
- Log Loss: Measures the uncertainty of predictions using cross-entropy loss.
- AUC-ROC: Computes the Area Under the Receiver Operating Characteristic curve for each class in a one-vs-rest manner.
- Mean Average Precision (mAP): Calculates the average precision across all classes, useful for multi-class problems.
- Class Distribution: Analyzes the distribution of predictions across classes to detect any bias.
-
Error Handling:
- Each method includes comprehensive error handling to manage unexpected inputs or computational issues gracefully, ensuring robustness.
-
Type Hints and PEP-8 Compliance:
- The code utilizes type hints from the
typingmodule for enhanced readability and maintainability. - It adheres to PEP-8 standards, ensuring a clean and well-structured codebase.
- The code utilizes type hints from the
-
Example Usage:
- The
__main__section demonstrates how to use theMetricsPipelinewith simulated logits and labels. - It showcases the updating of the pipeline with multiple batches and the computation of all metrics, printing them in a readable format.
- The
Conclusion
This implementation provides a robust and scalable pipeline for evaluating various robustness metrics in image classification tasks using only the
torchmodule. It ensures high performance, maintainability, and adherence to best coding practices, making it a valuable tool for developers and researchers working in deep learning and computer vision.