What is entropy when talking about LLMs?
- User
What is entropy when talking about LLMs?
- Assistant
Entropy (in the context of LLMs) is just the Shannon‐information measure of how “spread out” or “uncertain” a token-prediction distribution is. In plain English, it tells you how surprised you’d be on average when sampling from a model’s next-token probabilities.
1. Definition
Given a probability distribution p over your vocabulary V (for “next token”), the entropy H is:
H(p) = – ∑[x ∈ V] p(x) · log₂ p(x) (units: bits)– p(x) is the model’s probability of token x.
– log₂ p(x) measures “surprise” of seeing token x.
– Summing p(x)·(surprise) gives the average surprise.If you use natural logarithms instead (ln), you’ll get nats instead of bits.
2. Intuition
- Low entropy ⇒ the distribution is “peaky” (one or two tokens dominate).
- High entropy ⇒ the model is uncertain and spreads mass across many tokens.
Example:
• p(“dog”) = 0.90, p(others) = 0.10 total → low entropy
• p(“dog”)=p(“cat”)=p(“mouse”)=…=1/1000 → high entropy
3. Why it matters in LLMs
-
Sampling control
– Temperature scaling (T): dividing logits by T changes entropy.
– Nucleus (top-p) or top-k sampling: we trim the tail so the remaining distribution has lower entropy, yielding less “random” text. -
Perplexity
– Perplexity = 2^H (if H in bits)
– It’s a common evaluation metric—the lower the entropy, the lower the perplexity, the better the model “fits” the data. -
Loss function
– Training minimizes cross-entropy between the true next-token distribution (usually a one-hot) and the model’s predicted p.
– Cross-entropy is effectively the entropy of the true distribution plus the KL-divergence to the model’s.
4. Quick example
Suppose V = {“a”, “b”, “c”} and model outputs p = {0.5, 0.25, 0.25}.
H(p) = –[0.5·log₂0.5 + 0.25·log₂0.25 + 0.25·log₂0.25]
= –[0.5·(–1) + 0.25·(–2) + 0.25·(–2)]
= 0.5 + 0.5 + 0.5 = 1.5 bitsThat means, on average, you need 1.5 bits of information to describe the next token.
TL;DR
Entropy quantifies how “uncertain” your model is about its next token. Lower ⇒ more confident/peaky; higher ⇒ more random/spread-out. It underpins sampling strategies, perplexity, and the cross-entropy loss used during training.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.