Does cross-entropy loss only consider the predicted probability -- as in a vector of…
- User
Does cross-entropy loss only consider the predicted probability -- as in a vector of logits -- at the correct class?
Assistant
No, cross-entropy loss considers the predicted probabilities at all classes, not just the correct one. Here's a more detailed explanation:
The cross-entropy loss is defined as:
-sum(y * log(p))Where:
- y is the one-hot encoded vector representing the true class label
- p is the vector of predicted probabilities across all classes
- sum() sums over all classes
So for each class, it computes:
-y_i * log(p_i)Where
y_iis 1 for the true class, and 0 otherwise, andp_iis the predicted probability for classi.Intuitively, it penalizes:
- High predicted probability for incorrect classes (since y=0 for those classes, the log term goes to 0)
- Low predicted probability for the correct class (since that makes the log term more negative)
So it takes into account the predicted probabilities across all classes when calculating the loss, not just the probability assigned to the correct class. Maximizing cross-entropy loss leads to high confidence (probability near 1) in the true class.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.