what's cross entropy formula
- User
what's cross entropy formula
Assistant
Cross Entropy Formula
Cross-entropy is a measure from information theory and machine learning that quantifies the difference between two probability distributions: the true distribution and the predicted (or estimated) distribution . It is widely used as a loss function for classification tasks.
General Cross-Entropy Formula
For two discrete probability distributions (true) and (predicted) over the same set , the cross-entropy is defined as:
- : True probability of event
- : Predicted probability of event
- The logarithm is usually natural log (), but sometimes base 2 is used (resulting in units of bits)[1][2][5].
Cross-Entropy as a Loss Function
In machine learning, especially for classification, the cross-entropy loss for a single sample is:
- : Number of classes
- : True label (often one-hot encoded, so for the correct class, otherwise)
- : Predicted probability for class [3][5][6]
For a dataset with samples, the average cross-entropy loss is:
Binary Cross-Entropy (Log Loss)
For binary classification (two classes), the cross-entropy loss simplifies to:
- : True label (0 or 1)
- : Predicted probability for class 1[4][5][6][7]
Categorical Cross-Entropy (Multi-Class)
For multi-class classification with one-hot encoded labels:
If is one-hot, only the term for the correct class contributes, so for the correct class :
Summary Table
Scenario Formula General cross-entropy Binary cross-entropy Categorical cross-entropy Key Points
- Cross-entropy measures how well the predicted probabilities match the true distribution.
- Lower cross-entropy indicates better predictions.
- Used as a loss function to train classification models by minimizing the difference between predicted and true labels[1][2][5][6][7].
References:
[1] Machine Learning Mastery
[2] Wikipedia
[3] Data Science Stack Exchange
[4] Towards Data Science
[5] V7 Labs
[6] 365 Data Science
[7] DataCamp
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.