How do these leaderboard on LMArena work? The number of votes and the place models are…
- User
How do these leaderboard on LMArena work? The number of votes and the place models are do not seem correct
Assistant
You've made a great observation! The rankings on LMArena (and similar platforms like LMSYS Chatbot Arena) can seem confusing because they don't work on simple vote counting. Here's what's actually happening:
🎮 How Arena-Style LLM Leaderboards Work
Elo Rating System (Not Vote Counting)
LMArena uses an Elo rating system (like chess rankings), not raw vote counts. This means:
- Quality of wins matters more than quantity
- Beating a strong model gives you more points than beating a weak one
- Losing to a weak model hurts your rating more than losing to GPT-4
The Battle Process
- User gets two anonymous responses (Model A vs Model B)
- User picks the better response (or tie)
- Models' identities are revealed after voting
- Elo scores update based on:
- Who won
- The rating difference between models
- The "K-factor" (how much ratings change per battle)
🤔 Why Rankings Look "Wrong"
1. Unequal Battle Distribution
- GPT-4 might have 50,000 battles
- Blacktooth might have 2,000 battles
- But if Blacktooth wins 65% against strong opponents, it could rank higher!
2. Opponent Strength Matters
Example scenario:
- Model A: 10,000 votes, mostly against weak models = Lower Elo
- Model B: 1,000 votes, mostly against GPT-4/Claude = Higher Elo
3. Temporal Effects
- Newer models start with fewer battles but can climb fast
- Older models might have "stale" ratings from when they were stronger relative to competition
4. Statistical Confidence
Models need a minimum number of battles (~500-1000) before their ratings stabilize. You might see:
- High variance in new models
- Some models temporarily "overshoot" their true rating
📊 Common Confusions
"This model has more wins but ranks lower!"
- It probably beat mostly weak opponents
- Or it has a poor win rate against top models
"This new model shot to #3 with only 500 battles!"
- It's beating strong models consistently
- Rating might still be unstable (check back in a week)
"Why do some models have way more battles?"
- Popular models get selected more often
- Some platforms weight selection to test new models
- User bias (people want to test against GPT-4)
🎯 How to Read the Leaderboard Correctly
- Check Confidence Intervals: Many arenas show error bars (±50 Elo points)
- Look at Battle Count: <1000 battles = take ranking with grain of salt
- Check Recent Performance: Some sites show 7-day or 30-day trends
- Category Rankings: Overall vs Coding vs Creative Writing can differ vastly
🔧 Potential Issues with Arena Systems
- Prompt Bias: Some models excel at short casual chats (what most users test)
- Length Bias: Users often prefer longer, detailed responses
- Style Bias: Formal/verbose models can win even if less accurate
- Gaming: Some models might be optimized specifically for arena-style evaluation
Bottom line: The "votes" you see are individual battle results, but the ranking comes from the Elo calculation that weights WHO you beat/lost to. A model with 90% win rate against weak models will rank below one with 60% win rate against top-tier models.
Does this explain the discrepancy you noticed?
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.