Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-opus-4-20250514-thinking-16k· AnthropicHugging Face LMArena Human Preference 140K

How do these leaderboard on LMArena work? The number of votes and the place models are…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Googlegemma-3-27b-it’s answer.
  1. User

    How do these leaderboard on LMArena work? The number of votes and the place models are do not seem correct

  2. Claude Assistant

    You've made a great observation! The rankings on LMArena (and similar platforms like LMSYS Chatbot Arena) can seem confusing because they don't work on simple vote counting. Here's what's actually happening:

    🎮 How Arena-Style LLM Leaderboards Work

    Elo Rating System (Not Vote Counting)

    LMArena uses an Elo rating system (like chess rankings), not raw vote counts. This means:

    • Quality of wins matters more than quantity
    • Beating a strong model gives you more points than beating a weak one
    • Losing to a weak model hurts your rating more than losing to GPT-4

    The Battle Process

    1. User gets two anonymous responses (Model A vs Model B)
    2. User picks the better response (or tie)
    3. Models' identities are revealed after voting
    4. Elo scores update based on:
      • Who won
      • The rating difference between models
      • The "K-factor" (how much ratings change per battle)

    🤔 Why Rankings Look "Wrong"

    1. Unequal Battle Distribution

    • GPT-4 might have 50,000 battles
    • Blacktooth might have 2,000 battles
    • But if Blacktooth wins 65% against strong opponents, it could rank higher!

    2. Opponent Strength Matters

    Example scenario:

    • Model A: 10,000 votes, mostly against weak models = Lower Elo
    • Model B: 1,000 votes, mostly against GPT-4/Claude = Higher Elo

    3. Temporal Effects

    • Newer models start with fewer battles but can climb fast
    • Older models might have "stale" ratings from when they were stronger relative to competition

    4. Statistical Confidence

    Models need a minimum number of battles (~500-1000) before their ratings stabilize. You might see:

    • High variance in new models
    • Some models temporarily "overshoot" their true rating

    📊 Common Confusions

    "This model has more wins but ranks lower!"

    • It probably beat mostly weak opponents
    • Or it has a poor win rate against top models

    "This new model shot to #3 with only 500 battles!"

    • It's beating strong models consistently
    • Rating might still be unstable (check back in a week)

    "Why do some models have way more battles?"

    • Popular models get selected more often
    • Some platforms weight selection to test new models
    • User bias (people want to test against GPT-4)

    🎯 How to Read the Leaderboard Correctly

    1. Check Confidence Intervals: Many arenas show error bars (±50 Elo points)
    2. Look at Battle Count: <1000 battles = take ranking with grain of salt
    3. Check Recent Performance: Some sites show 7-day or 30-day trends
    4. Category Rankings: Overall vs Coding vs Creative Writing can differ vastly

    🔧 Potential Issues with Arena Systems

    • Prompt Bias: Some models excel at short casual chats (what most users test)
    • Length Bias: Users often prefer longer, detailed responses
    • Style Bias: Formal/verbose models can win even if less accurate
    • Gaming: Some models might be optimized specifically for arena-style evaluation

    Bottom line: The "votes" you see are individual battle results, but the ranking comes from the Elo calculation that weights WHO you beat/lost to. A model with 90% win rate against weak models will rank below one with 60% win rate against top-tier models.

    Does this explain the discrepancy you noticed?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.