Skip to content
TheChatArchiveThe Chat Archive
o4-mini-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

What is entropy when talking about LLMs?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o3-2025-04-16’s answer.
  1. User

    What is entropy when talking about LLMs?

  2. Assistant

    Entropy (in the context of LLMs) is just the Shannon‐information measure of how “spread out” or “uncertain” a token-prediction distribution is. In plain English, it tells you how surprised you’d be on average when sampling from a model’s next-token probabilities.


    1. Definition

    Given a probability distribution p over your vocabulary V (for “next token”), the entropy H is:

    H(p) = – ∑[x ∈ V] p(x) · log₂ p(x)   (units: bits)
    

    – p(x) is the model’s probability of token x.
    – log₂ p(x) measures “surprise” of seeing token x.
    – Summing p(x)·(surprise) gives the average surprise.

    If you use natural logarithms instead (ln), you’ll get nats instead of bits.


    2. Intuition

    • Low entropy ⇒ the distribution is “peaky” (one or two tokens dominate).
    • High entropy ⇒ the model is uncertain and spreads mass across many tokens.

    Example:
    • p(“dog”) = 0.90, p(others) = 0.10 total → low entropy
    • p(“dog”)=p(“cat”)=p(“mouse”)=…=1/1000 → high entropy


    3. Why it matters in LLMs

    1. Sampling control
      – Temperature scaling (T): dividing logits by T changes entropy.
      – Nucleus (top-p) or top-k sampling: we trim the tail so the remaining distribution has lower entropy, yielding less “random” text.

    2. Perplexity
      – Perplexity = 2^H (if H in bits)
      – It’s a common evaluation metric—the lower the entropy, the lower the perplexity, the better the model “fits” the data.

    3. Loss function
      – Training minimizes cross-entropy between the true next-token distribution (usually a one-hot) and the model’s predicted p.
      – Cross-entropy is effectively the entropy of the true distribution plus the KL-divergence to the model’s.


    4. Quick example

    Suppose V = {“a”, “b”, “c”} and model outputs p = {0.5, 0.25, 0.25}.

    H(p) = –[0.5·log₂0.5 + 0.25·log₂0.25 + 0.25·log₂0.25]
    = –[0.5·(–1) + 0.25·(–2) + 0.25·(–2)]
    = 0.5 + 0.5 + 0.5 = 1.5 bits

    That means, on average, you need 1.5 bits of information to describe the next token.


    TL;DR

    Entropy quantifies how “uncertain” your model is about its next token. Lower ⇒ more confident/peaky; higher ⇒ more random/spread-out. It underpins sampling strategies, perplexity, and the cross-entropy loss used during training.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.