Is AUC sensitive to data imbalance ?
- User
Is AUC sensitive to data imbalance ?
Assistant
Let me explain AUC (Area Under the ROC Curve) and its relationship with class imbalance:
AUC is relatively robust to class imbalance compared to metrics like accuracy. Here's why:
- AUC evaluates model performance across all possible classification thresholds
- It looks at the relationship between true positive rate (sensitivity) and false positive rate (1-specificity), which are both ratio metrics that inherently account for class sizes
- Each class is treated separately in the calculations, so the size difference between classes doesn't directly impact the metric
However, there are some considerations:
- While AUC is less sensitive to imbalance than accuracy, extreme imbalance can still affect model training and therefore indirectly impact AUC
- A high AUC score doesn't necessarily mean the model will perform well at a specific classification threshold
- For highly imbalanced datasets, metrics like PR-AUC (Precision-Recall AUC) might be more informative as they focus more on the minority class
Would you like me to elaborate on any of these points?
- User
ok imagine I have loaded in my notebook prediction probas y_pred and the labels y_test. Can you write a function that plots side by side the ROC and PR curve, giving respectively the associated scores ?
Assistant
Here's a function that plots both ROC and PR curves side by side using matplotlib and sklearn:
def plot_roc_pr_curves(y_test, y_pred): from sklearn.metrics import roc_curve, precision_recall_curve, auc, average_precision_score import matplotlib.pyplot as plt # Create figure with two subplots side by side fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5)) # ROC Curve fpr, tpr, _ = roc_curve(y_test, y_pred) roc_auc = auc(fpr, tpr) ax1.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.2f})') ax1.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--') ax1.set_xlim([0.0, 1.0]) ax1.set_ylim([0.0, 1.05]) ax1.set_xlabel('False Positive Rate') ax1.set_ylabel('True Positive Rate') ax1.set_title('Receiver Operating Characteristic (ROC)') ax1.legend(loc="lower right") ax1.grid(True) # Precision-Recall Curve precision, recall, _ = precision_recall_curve(y_test, y_pred) pr_auc = average_precision_score(y_test, y_pred) ax2.plot(recall, precision, color='darkgreen', lw=2, label=f'PR curve (AP = {pr_auc:.2f})') ax2.set_xlim([0.0, 1.0]) ax2.set_ylim([0.0, 1.05]) ax2.set_xlabel('Recall') ax2.set_ylabel('Precision') ax2.set_title('Precision-Recall Curve') ax2.legend(loc="lower left") ax2.grid(True) plt.tight_layout() plt.show() return roc_auc, pr_aucYou can use it like this:
roc_auc, pr_auc = plot_roc_pr_curves(y_test, y_pred)This function:
- Creates two subplots side by side
- Plots the ROC curve with its AUC score on the left
- Plots the PR curve with its Average Precision score on the right
- Returns both scores for further use if needed
- Includes grids, legends, and proper labels for better visualization
The plots will help you visually assess the model's performance, while the scores provide quantitative metrics.
- User
can you add a baseline for the PR curve ? it corresponds roughly to the proportion of class 1 (i.e. a random guess based on the population proportions)
Assistant
Here's the modified function that adds a baseline to the PR curve:
def plot_roc_pr_curves(y_test, y_pred): from sklearn.metrics import roc_curve, precision_recall_curve, auc, average_precision_score import matplotlib.pyplot as plt import numpy as np # Create figure with two subplots side by side fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5)) # ROC Curve fpr, tpr, _ = roc_curve(y_test, y_pred) roc_auc = auc(fpr, tpr) ax1.plot(fpr, tpr, color='darkorange', lw=2, label=f'ROC curve (AUC = {roc_auc:.2f})') ax1.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--') ax1.set_xlim([0.0, 1.0]) ax1.set_ylim([0.0, 1.05]) ax1.set_xlabel('False Positive Rate') ax1.set_ylabel('True Positive Rate') ax1.set_title('Receiver Operating Characteristic (ROC)') ax1.legend(loc="lower right") ax1.grid(True) # Precision-Recall Curve precision, recall, _ = precision_recall_curve(y_test, y_pred) pr_auc = average_precision_score(y_test, y_pred) # Calculate baseline for PR curve (proportion of positive class) baseline = np.mean(y_test) ax2.plot(recall, precision, color='darkgreen', lw=2, label=f'PR curve (AP = {pr_auc:.2f})') ax2.axhline(y=baseline, color='navy', linestyle='--', label=f'Baseline ({baseline:.2f})') ax2.set_xlim([0.0, 1.0]) ax2.set_ylim([0.0, 1.05]) ax2.set_xlabel('Recall') ax2.set_ylabel('Precision') ax2.set_title('Precision-Recall Curve') ax2.legend(loc="lower left") ax2.grid(True) plt.tight_layout() plt.show() return roc_auc, pr_auc, baselineThe main changes are:
- Added a baseline calculation using
np.mean(y_test) - Added a horizontal line in the PR curve plot showing this baseline
- Updated the return values to include the baseline
You can use it the same way:
roc_auc, pr_auc, baseline = plot_roc_pr_curves(y_test, y_pred)This baseline helps you better interpret the PR curve, especially in imbalanced cases. A curve closer to the top-right corner and well above the baseline indicates good performance.
- Added a baseline calculation using
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.