Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

Can you tell me about the different leaderboards on lmarena.ai and how to interpret them?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Qwenqwq-32b’s answer.
  1. User

    Can you tell me about the different leaderboards on lmarena.ai and how to interpret them?

  2. Assistant

    Below is a quick “field guide” to the leaderboards you’ll run into on lmarena.ai, what each one is actually measuring, and a few rules of thumb for reading the numbers without getting tripped up ☕️📊


    1. Chat-Arena (a.k.a. “Arena Elo”)

    What it is
    • Live, crowdsourced A/B “battles.” A user chats, two models answer, voters pick a winner (or tie).
    • The site fits a Bradley-Terry/Elo-style model to thousands of these pairwise preferences.
    • The resulting rating is called Arena Elo (with a ±95 % conf. interval).

    How to read it
    • Higher Elo ⇒ more likely to win an average battle.
    • 50 pt gap ≈ 57 % win-prob.; 100 pt ≈ 64 %; 200 pt ≈ 76 %.
    • Ignore small gaps when CIs overlap – those models are “statistically tied.”
    • Check “Battles” count: < ~200 battles means the rating will still jump around.

    Helpful tabs/filters
    All | Open-Source | Base-only | Under 20B | New | Small-sample.
    Great when you care about real-time user preference, writing style, harmlessness, etc.


    1. Benchmark / Hub Leaderboard (static scores, a.k.a. “PW-E score”)

    What it is
    • Aggregates classic academic test-sets: MMLU, ARC-Chal, HellaSwag, TruthfulQA, GSM-8K, etc.
    • The “Hub score” or “Power-Weighted Elo (PW-E)” is just a weighted average of those columns.

    How to read it
    • Each column is % accuracy; bold = best in that column.
    • PW-E tries to combine them, but missing values (“--”) skew things, so compare rows with similar coverage.
    • Great for knowledge & reasoning under closed-book, non-chat conditions.


    1. Single-Dataset Views (“Eval Sets”)

    Pick “MMLU” or “GSM8K” in the drop-down and the board re-sorts by that one metric.
    • Useful for spotting specialist models (math-tuned, code-tuned, med-tuned, …).
    • Scores are directly comparable because every model gets the same prompt template.


    1. Live Battle Feed

    A fire-hose of the most recent A/B duels.
    • You’ll see raw win/loss counts, prompt text, and voter comments.
    • Fun for qualitative inspection, but don’t over-interpret single fights.


    1. Per-Model “Card”

    Click a model name → mini-dashboard:
    • Elo trajectory over time
    • Win-rate broken down by question category (coding, reasoning, casual chat…)
    • Model metadata (size, license, release date, system prompt, etc.)


    Putting it all together

    When choosing or comparing models:

    ✓ Look at Arena Elo for general chat UX.
    ✓ Check static benchmarks if you need proven accuracy on standardized tests.
    ✓ Mind the ±CI and the battle count.
    ✓ Filter by openness/size if deployment constraints matter.
    ✓ Remember that a 100-pt Elo gap is meaningful, but two models whose intervals overlap are effectively tied.

    Finally, the site is live and ratings change daily—always note the time-stamp in the header (“Last updated: …”). Happy model hunting! 🏹

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.

Identical first prompt