Skip to content
TheChatArchiveThe Chat Archive
gpt-4-0125-preview· OpenAIHugging Face Arena Human Preference 55K

Is it true that when the predicated distribution is the same as the true distribution,…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Mistralmistral-medium’s answer.
  1. User

    Is it true that when the predicated distribution is the same as the true distribution, the cross entropy is 0?

  2. Assistant

    No, that statement is not accurate. When the predicted distribution is exactly the same as the true distribution, the cross-entropy is not 0, but it reaches its minimum possible value, which corresponds to the entropy of the true distribution itself.

    Cross-entropy is a measure of the difference between two probability distributions for a given random variable or set of events. It's often used in machine learning to quantify the difference between the predicted probabilities and the actual distribution (e.g., in classification problems). The formula for cross-entropy (H) between two distributions p (true distribution) and q (predicted distribution) over the same underlying set of events is:

    [ H(p, q) = -\sum_{x} p(x) \log q(x) ]

    Where (p(x)) is the probability of an event x in the true distribution, and (q(x)) is the probability of an event x in the predicted distribution.

    If the predicted distribution (q) perfectly matches the true distribution (p) for all events (x), then we are calculating the entropy of (p), not achieving a cross-entropy of 0. The entropy (H) of a distribution (p) is defined as:

    [ H(p) = -\sum_{x} p(x) \log p(x) ]

    Entropy measures the amount of uncertainty or disorder in a distribution. So, when (p = q), the cross-entropy equals the entropy of (p), which represents the minimum possible value that the cross-entropy can achieve, indicating the lowest possible level of surprise or uncertainty (given that the predictions perfectly match the true distribution). This value is not necessarily 0, except in trivial cases where the distribution is deterministic (e.g., always producing the same outcome with probability 1).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.