Skip to content
The Chat Archive
o4-mini-2025-04-16· OpenAI LMArena Human Preference 140K

In the context of AI, what are cardinal vs. ordinal benchmarks?

  1. User

    In the context of AI, what are cardinal vs. ordinal benchmarks?

  2. Assistant

    In AI evaluation, “cardinal” vs. “ordinal” benchmarks refer to two fundamentally different ways of measuring and comparing system performance:

    1. Cardinal Benchmarks

      • What they measure
        Absolute scores on a numeric (interval) scale.
      • Key property
        Differences are meaningful: a system scoring 80 vs. 70 did “twice as badly” as one scoring 90 vs. 80.
      • Examples
        • Accuracy (%) on a classification task
        • BLEU / ROUGE scores in machine translation or summarization
        • Perplexity in language modeling
        • F1-score, ROC-AUC, MSE, etc.
      • Pros & Cons
        • Pros: You can average them, compute deltas, rank with meaningful gaps
          – Cons: Often sensitive to scale, can be gamed by optimizing the metric itself
    2. Ordinal Benchmarks

      • What they measure
        Only the order or preference among models. You know A > B > C, but not by how much.
      • Key property
        Invariant under any strictly monotonic transformation of the score scale—only the ranking matters.
      • Examples
        • Pairwise human preference judgments (“Which summary do you prefer?”)
        • Win-rate in self-play tournaments (e.g. AlphaGo vs. AlphaZero)
        • Leaderboards that just list model ranks without publishing raw scores
      • Pros & Cons
        • Pros: More robust to calibration issues; less sensitive to the exact numeric scale
          – Cons: You can’t say “Model X is 10 points better than Y,” only “X beats Y”

    Comparison Table

    AspectCardinalOrdinal
    Scale typeInterval/ratio (differences matter)Rank/order only (differences ×)
    Meaning of gapsMeaningful (e.g. 10-point gap)Not meaningful (just > or <)
    AggregationCan average or sumOnly counts of wins/placements
    SensitivitySensitive to scale/calibrationInvariant to monotonic transforms
    Typical use casesAutomated metrics (accuracy, BLEU, PPL)Human preference, tournament ranks

    When to use which?

    • Cardinal metrics are great for quick iteration, hyperparameter tuning, model selection if you trust the metric’s scale.
    • Ordinal evaluations are often preferred when human judgment or robust pairwise comparisons are needed, and you care less about “how much” and more about “who wins.”

    Does that help clarify things? Or would you like examples from a particular subfield (e.g. vision, language, RL)?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.