Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-2.1· AnthropicHugging Face Arena Human Preference 55K

Does cross-entropy loss only consider the predicted probability -- as in a vector of…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-1’s answer.
  1. User

    Does cross-entropy loss only consider the predicted probability -- as in a vector of logits -- at the correct class?

  2. Claude Assistant

    No, cross-entropy loss considers the predicted probabilities at all classes, not just the correct one. Here's a more detailed explanation:

    The cross-entropy loss is defined as:

    -sum(y * log(p))

    Where:

    • y is the one-hot encoded vector representing the true class label
    • p is the vector of predicted probabilities across all classes
    • sum() sums over all classes

    So for each class, it computes:

    -y_i * log(p_i)

    Where y_i is 1 for the true class, and 0 otherwise, and p_i is the predicted probability for class i.

    Intuitively, it penalizes:

    • High predicted probability for incorrect classes (since y=0 for those classes, the log term goes to 0)
    • Low predicted probability for the correct class (since that makes the log term more negative)

    So it takes into account the predicted probabilities across all classes when calculating the loss, not just the probability assigned to the correct class. Maximizing cross-entropy loss leads to high confidence (probability near 1) in the true class.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt