Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-7-sonnet-20250219-thinking-32k· AnthropicHugging Face LMArena Human Preference 140K

How is the elo calculated on LM Arena?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Mistralmistral-medium-2505’s answer.
  1. User

    How is the elo calculated on LM Arena?

  2. Claude Assistant

    The Elo rating system used in LM Arena is a clever adaptation of the classic chess rating method for comparing language models. Here's how it works:

    Basic Elo Calculation

    1. Starting Point: All models begin with a base Elo rating (typically 1200).

    2. Win Probability Formula: When two models with ratings R₁ and R₂ face each other, the expected probability that Model 1 will win is:

      E₁ = 1 / (1 + 10^((R₂ - R₁)/400))
      
    3. Rating Update: After a user votes for one model over another, the ratings update:

      R₁_new = R₁ + K × (S₁ - E₁)
      

      Where:

      • K is the "K-factor" (how much a single match affects ratings)
      • S₁ is the actual outcome (1 for win, 0 for loss)
      • E₁ is the expected outcome calculated above

    LM Arena Specifics

    The LMSYS team has implemented several important refinements to make Elo work well for language model evaluation:

    • Dynamic K-factor: Models with fewer votes have higher K-factors, meaning their ratings change more dramatically with each new vote. As models accumulate more votes, their ratings stabilize.

    • Confidence Intervals: The leaderboard shows not just the Elo rating but also confidence intervals (typically displayed as a range or error bars), indicating the statistical certainty of a model's position.

    • Tie Handling: When users indicate both models performed equally well, this is handled as a tie (0.5 points for each model).

    • Vote Quality Control: They implement various measures to filter out low-quality or biased votes.

    • Rating Calibration: Periodically, they may recalibrate the entire leaderboard to maintain consistency as new models are added.

    What makes this system particularly elegant is that:

    1. It naturally accounts for the strength of the opponent (beating a stronger model earns more points)
    2. A 100-point Elo difference corresponds to roughly a 64% win probability
    3. It can accommodate new models being added at any time

    This system has proven remarkably effective at converting hundreds of thousands of individual human judgments into a meaningful, stable ranking that closely aligns with other evaluation methods.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.