o4-mini-2025-04-16· OpenAI LMArena Human Preference 140K
In the context of AI, what are cardinal vs. ordinal benchmarks?
- User
In the context of AI, what are cardinal vs. ordinal benchmarks?
- Assistant
In AI evaluation, “cardinal” vs. “ordinal” benchmarks refer to two fundamentally different ways of measuring and comparing system performance:
-
Cardinal Benchmarks
- What they measure
Absolute scores on a numeric (interval) scale. - Key property
Differences are meaningful: a system scoring 80 vs. 70 did “twice as badly” as one scoring 90 vs. 80. - Examples
• Accuracy (%) on a classification task
• BLEU / ROUGE scores in machine translation or summarization
• Perplexity in language modeling
• F1-score, ROC-AUC, MSE, etc. - Pros & Cons
- Pros: You can average them, compute deltas, rank with meaningful gaps
– Cons: Often sensitive to scale, can be gamed by optimizing the metric itself
- Pros: You can average them, compute deltas, rank with meaningful gaps
- What they measure
-
Ordinal Benchmarks
- What they measure
Only the order or preference among models. You know A > B > C, but not by how much. - Key property
Invariant under any strictly monotonic transformation of the score scale—only the ranking matters. - Examples
• Pairwise human preference judgments (“Which summary do you prefer?”)
• Win-rate in self-play tournaments (e.g. AlphaGo vs. AlphaZero)
• Leaderboards that just list model ranks without publishing raw scores - Pros & Cons
- Pros: More robust to calibration issues; less sensitive to the exact numeric scale
– Cons: You can’t say “Model X is 10 points better than Y,” only “X beats Y”
- Pros: More robust to calibration issues; less sensitive to the exact numeric scale
- What they measure
Comparison Table
Aspect Cardinal Ordinal Scale type Interval/ratio (differences matter) Rank/order only (differences ×) Meaning of gaps Meaningful (e.g. 10-point gap) Not meaningful (just > or <) Aggregation Can average or sum Only counts of wins/placements Sensitivity Sensitive to scale/calibration Invariant to monotonic transforms Typical use cases Automated metrics (accuracy, BLEU, PPL) Human preference, tournament ranks When to use which?
- Cardinal metrics are great for quick iteration, hyperparameter tuning, model selection if you trust the metric’s scale.
- Ordinal evaluations are often preferred when human judgment or robust pairwise comparisons are needed, and you care less about “how much” and more about “who wins.”
Does that help clarify things? Or would you like examples from a particular subfield (e.g. vision, language, RL)?
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.