how score calculated on lmarean.ai?
- User
how score calculated on lmarean.ai?
- Assistant
I’m assuming you’re talking about the Chatbot Arena leaderboard on lmsys.org (sometimes people type it as “lmarena.ai” or “lmarean”) where different large-language models duel and end up with an “Arena score”.
If that’s not what you meant, just let me know and I’ll adjust. 🙂How the Arena score is produced ──────────────────────────────
-
Two-model duel
• When you start an Arena chat you actually talk to two anonymous models (A and B).
• After a few turns you’re asked to pick which one did better, or choose “tie / both bad”. -
Convert that pick into a game result
• A wins → A = 1 point, B = 0 points.
• B wins → B = 1, A = 0.
• Tie → both get 0.5.
• “Both bad” → both get 0 (this doesn’t move ratings much, but it’s allowed). -
Update each model’s Elo rating
• The site uses the classic Elo formula, same as chess:Expected score for A
EA = 1 / (1 + 10^((RB − RA)/400))New rating
RA′ = RA + K × (SA − EA)– RA, RB … current ratings
– SA, SB … actual outcome (1, 0.5, 0)
– K … learning rate (Arena currently uses K ≈ 16 once a model has many games; early on it’s ~32 so new models settle faster). -
Repeat for every vote
All games are replayed in chronological order once per day (batch recomputation) so the published leaderboard is consistent even if users leave mid-conversation, etc. -
Confidence interval & visible “Arena score”
• The public column “Arena Elo ± CI” is just the mean Elo plus a bootstrapped 95 % confidence interval.
• If a model has very few votes/samples, you’ll see a wider interval and a ⚠️ icon.
Little details people often ask about ───────────────────────────────────── • Starting rating: everyone begins at 1200 Elo.
• Abandoned chats: discarded (duel only counted if the voter submits a choice).
• Multi-turn weighting: a single vote counts the same whether the chat had 2 turns or 10.
• Same-model matchups: never shown—the system guarantees A ≠ B.
• Strong models plateau: once two models are both ~200 points apart, the higher rated one is expected to win 75 % of the time, so Elo changes become tiny.Quick example ───────────── Let’s say model X is 1600 Elo, model Y is 1500.
Expected score for X
EX = 1 / (1 + 10^((1500−1600)/400)) ≈ 0.64A voter prefers X ⇒ SX = 1.
If K = 16:
ΔX = 16 × (1 − 0.64) ≈ +5.8 → X becomes 1606
ΔY = 16 × (0 − 0.36) ≈ −5.8 → Y becomes 1494Why your favorite model’s score keeps wobbling ────────────────────────────────────────────── • New votes come in 24/7.
• Newly added models may steal rating points if they beat incumbents.
• The confidence interval narrows slowly as N grows (law of large numbers).In short, the Arena score is a straight-up Elo rating based on pairwise user votes, with the usual chess-style math plus a daily batch recompute to keep things tidy.
Anything else you’re curious about (e.g., how ties affect K, how many votes a model has, etc.)?
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.