Definition
Perplexity is a measurement of how well a probability model predicts a sample. For language models, it represents the model’s uncertainty when predicting the next token: lower perplexity means the model is less “perplexed” or more confident. Mathematically, perplexity is the exponentiated average negative log-likelihood per token.
Why it matters
Perplexity is a fundamental evaluation metric:
- Model comparison: compare different models on same dataset
- Training monitoring: track improvement during training
- Domain assessment: measure how well model fits specific text
- Quantization impact: evaluate quality loss from compression
- Interpretable scale: can be understood as effective vocabulary size
Perplexity helps answer: “How surprised is the model by this text?”
How it works
┌────────────────────────────────────────────────────────────┐
│ PERPLEXITY │
├────────────────────────────────────────────────────────────┤
│ │
│ Formula: PPL = exp(-1/N × Σ log P(token_i | context)) │
│ │
│ Example: "The cat sat" │
│ ────────────────────── │
│ │
│ Token P(token|context) log P │
│ ───────────────────────────────────────── │
│ "The" 0.10 -2.30 │
│ "cat" 0.25 -1.39 │
│ "sat" 0.40 -0.92 │
│ │
│ Average log P = (-2.30 + -1.39 + -0.92) / 3 = -1.54 │
│ Perplexity = exp(1.54) = 4.66 │
│ │
│ ┌────────────────────────────────────────────────┐ │
│ │ INTERPRETATION: │ │
│ │ │ │
│ │ PPL ≈ "effective choices per position" │ │
│ │ │ │
│ │ PPL = 1: Model is 100% certain │ │
│ │ PPL = 10: ~10 equally likely options │ │
│ │ PPL = 50k: Random (vocab size) = no learning│ │
│ └────────────────────────────────────────────────┘ │
│ │
│ PERPLEXITY SCALE: │
│ ─────────────────── │
│ │◄────────────────────────────────────────────►│ │
│ 1 10 50 100 1000 50000 │
│ Perfect Great Good Okay Poor Random │
│ │
│ TYPICAL VALUES: │
│ ─────────────── │
│ GPT-4 on common text: ~10-20 │
│ Small model: ~50-100 │
│ Domain mismatch: ~100-500 │
│ Untrained model: ~vocab size │
│ │
└────────────────────────────────────────────────────────────┘Perplexity benchmarks:
| Range | Quality | Interpretation |
|---|---|---|
| 1-10 | Excellent | Highly predictable text |
| 10-30 | Very good | Typical for strong LLMs |
| 30-50 | Good | Reasonable model |
| 50-100 | Moderate | May need improvement |
| 100+ | Poor | Significant issues or domain mismatch |
Common questions
Q: What’s a “good” perplexity score?
A: It depends on the dataset and model size. State-of-the-art models achieve perplexity ~15-25 on standard benchmarks like WikiText. Within a project, focus on relative improvements rather than absolute numbers.
Q: Can perplexity compare models with different tokenizers?
A: Not directly, different tokenizers produce different numbers of tokens for the same text. Compare models using the same tokenizer, or normalize by character/word count instead of token count.
Q: Why might perplexity be low but generation quality poor?
A: Perplexity measures average prediction accuracy, not output quality. A model can have low perplexity by predicting common words well while failing at coherent long-form generation. Use perplexity alongside other metrics.
Q: How does perplexity relate to cross-entropy loss?
A: Perplexity = exp(cross-entropy). They measure the same thing on different scales. Cross-entropy is typically used during training (easier gradient computation); perplexity is more interpretable for reporting.
Related terms
- Loss Function: training objective
- LLM: language models being evaluated
- Tokenization: affects perplexity calculation
- Fine-tuning: improves domain perplexity
References
Jelinek et al. (1977), “Perplexity—a measure of the difficulty of speech recognition tasks”, JASA. [Foundational paper]
Merity et al. (2017), “Regularizing and Optimizing LSTM Language Models”, ICLR. [1,500+ citations]
Brown et al. (2020), “Language Models are Few-Shot Learners”, NeurIPS. [15,000+ citations]
Radford et al. (2019), “Language Models are Unsupervised Multitask Learners”, OpenAI. [10,000+ citations]