Skip to content
TheChatArchiveThe Chat Archive
Perplexitysonar-pro-high· PerplexityHugging Face LMArena Search Arena 24K

what's cross entropy formula

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4o-search-preview-high’s answer.
  1. User

    what's cross entropy formula

  2. Perplexity Assistant

    Cross Entropy Formula

    Cross-entropy is a measure from information theory and machine learning that quantifies the difference between two probability distributions: the true distribution PP and the predicted (or estimated) distribution QQ. It is widely used as a loss function for classification tasks.

    General Cross-Entropy Formula

    For two discrete probability distributions PP (true) and QQ (predicted) over the same set XX, the cross-entropy is defined as:

    H(P,Q)=−∑x∈XP(x)log⁡Q(x)H(P, Q) = -\sum_{x \in X} P(x) \log Q(x)
    • P(x)P(x): True probability of event xx
    • Q(x)Q(x): Predicted probability of event xx
    • The logarithm is usually natural log (ln⁡\ln), but sometimes base 2 is used (resulting in units of bits)[1][2][5].

    Cross-Entropy as a Loss Function

    In machine learning, especially for classification, the cross-entropy loss for a single sample is:

    L=−∑i=1Cyilog⁡(y^i)L = -\sum_{i=1}^{C} y_i \log(\hat{y}_i)
    • CC: Number of classes
    • yiy_i: True label (often one-hot encoded, so yi=1y_i=1 for the correct class, 00 otherwise)
    • y^i\hat{y}_i: Predicted probability for class ii[3][5][6]

    For a dataset with NN samples, the average cross-entropy loss is:

    L=−1N∑n=1N∑i=1Cynilog⁡(y^ni)L = -\frac{1}{N} \sum_{n=1}^{N} \sum_{i=1}^{C} y_{ni} \log(\hat{y}_{ni})

    Binary Cross-Entropy (Log Loss)

    For binary classification (two classes), the cross-entropy loss simplifies to:

    L=−1N∑i=1N[yilog⁡(pi)+(1−yi)log⁡(1−pi)]L = -\frac{1}{N} \sum_{i=1}^{N} \left[ y_i \log(p_i) + (1 - y_i) \log(1 - p_i) \right]
    • yiy_i: True label (0 or 1)
    • pip_i: Predicted probability for class 1[4][5][6][7]

    Categorical Cross-Entropy (Multi-Class)

    For multi-class classification with one-hot encoded labels:

    L=−∑i=1Cyilog⁡(y^i)L = -\sum_{i=1}^{C} y_i \log(\hat{y}_i)

    If yiy_i is one-hot, only the term for the correct class contributes, so for the correct class kk:

    L=−log⁡(y^k)L = -\log(\hat{y}_k)

    Summary Table

    ScenarioFormula
    General cross-entropyH(P,Q)=−∑x∈XP(x)log⁡Q(x)H(P, Q) = -\sum_{x \in X} P(x) \log Q(x)
    Binary cross-entropyL=−[ylog⁡(p)+(1−y)log⁡(1−p)]L = -[y \log(p) + (1-y) \log(1-p)]
    Categorical cross-entropyL=−∑i=1Cyilog⁡(y^i)L = -\sum_{i=1}^{C} y_i \log(\hat{y}_i)

    Key Points

    • Cross-entropy measures how well the predicted probabilities match the true distribution.
    • Lower cross-entropy indicates better predictions.
    • Used as a loss function to train classification models by minimizing the difference between predicted and true labels[1][2][5][6][7].

    References:
    [1] Machine Learning Mastery
    [2] Wikipedia
    [3] Data Science Stack Exchange
    [4] Towards Data Science
    [5] V7 Labs
    [6] 365 Data Science
    [7] DataCamp

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.