Skip to content
TheChatArchiveThe Chat Archive
o3-2025-04-16· OpenAIHugging Face LMArena Human Preference 140K

how score calculated on lmarean.ai?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with o4-mini-2025-04-16’s answer.
  1. User

    how score calculated on lmarean.ai?

  2. Assistant

    I’m assuming you’re talking about the Chatbot Arena leaderboard on lmsys.org (sometimes people type it as “lmarena.ai” or “lmarean”) where different large-language models duel and end up with an “Arena score”.
    If that’s not what you meant, just let me know and I’ll adjust. 🙂

    How the Arena score is produced ──────────────────────────────

    1. Two-model duel
      • When you start an Arena chat you actually talk to two anonymous models (A and B).
      • After a few turns you’re asked to pick which one did better, or choose “tie / both bad”.

    2. Convert that pick into a game result
      • A wins → A = 1 point, B = 0 points.
      • B wins → B = 1, A = 0.
      • Tie → both get 0.5.
      • “Both bad” → both get 0 (this doesn’t move ratings much, but it’s allowed).

    3. Update each model’s Elo rating
      • The site uses the classic Elo formula, same as chess:

      Expected score for A
      EA = 1 / (1 + 10^((RB − RA)/400))

      New rating
      RA′ = RA + K × (SA − EA)

      – RA, RB … current ratings
      – SA, SB … actual outcome (1, 0.5, 0)
      – K … learning rate (Arena currently uses K ≈ 16 once a model has many games; early on it’s ~32 so new models settle faster).

    4. Repeat for every vote
      All games are replayed in chronological order once per day (batch recomputation) so the published leaderboard is consistent even if users leave mid-conversation, etc.

    5. Confidence interval & visible “Arena score”
      • The public column “Arena Elo ± CI” is just the mean Elo plus a bootstrapped 95 % confidence interval.
      • If a model has very few votes/samples, you’ll see a wider interval and a ⚠️ icon.

    Little details people often ask about ───────────────────────────────────── • Starting rating: everyone begins at 1200 Elo.
    • Abandoned chats: discarded (duel only counted if the voter submits a choice).
    • Multi-turn weighting: a single vote counts the same whether the chat had 2 turns or 10.
    • Same-model matchups: never shown—the system guarantees A ≠ B.
    • Strong models plateau: once two models are both ~200 points apart, the higher rated one is expected to win 75 % of the time, so Elo changes become tiny.

    Quick example ───────────── Let’s say model X is 1600 Elo, model Y is 1500.

    Expected score for X
    EX = 1 / (1 + 10^((1500−1600)/400)) ≈ 0.64

    A voter prefers X ⇒ SX = 1.
    If K = 16:
    ΔX = 16 × (1 − 0.64) ≈ +5.8 → X becomes 1606
    ΔY = 16 × (0 − 0.36) ≈ −5.8 → Y becomes 1494

    Why your favorite model’s score keeps wobbling ────────────────────────────────────────────── • New votes come in 24/7.
    • Newly added models may steal rating points if they beat incumbents.
    • The confidence interval narrows slowly as N grows (law of large numbers).

    In short, the Arena score is a straight-up Elo rating based on pairwise user votes, with the usual chess-style math plus a daily batch recompute to keep things tidy.

    Anything else you’re curious about (e.g., how ties affect K, how many votes a model has, etc.)?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.