What's the point of perplexity if it's basically computed from cross-entropy ?
- User
What's the point of perplexity if it's basically computed from cross-entropy ?
Assistant
You're right to question this - perplexity is indeed just an exponential transformation of cross-entropy, so mathematically they contain the same information. But perplexity persists for several practical reasons:
Interpretability
Perplexity has a beautifully intuitive interpretation: it represents the "effective vocabulary size" the model is choosing from at each step. A perplexity of 10 means the model is roughly as uncertain as if it were choosing uniformly from 10 options. This is much easier to grasp than "cross-entropy of 2.3 nats."
Human-Friendly Scale
Cross-entropy grows logarithmically, while perplexity grows linearly. If a model improves from 50% to 90% accuracy on binary classification:
- Cross-entropy: 1.0 → 0.15 (harder to interpret the magnitude)
- Perplexity: 2.0 → 1.16 (clearly shows we went from "choosing between 2 options" to "almost certain")
Historical Momentum
In NLP, perplexity became the standard metric before deep learning. Many benchmarks and papers use it, so it persists for comparison. It's like how we still use BLEU scores despite their limitations.
Different Communities
- ML researchers often prefer cross-entropy (it's the actual loss being optimized)
- NLP practitioners often prefer perplexity (it connects to linguistic intuitions)
You're absolutely right that we could just use cross-entropy everywhere. Perplexity is essentially a "user interface" choice - it doesn't add information, but it can make that information more digestible.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.