Skip to content
TheChatArchiveThe Chat Archive
Mistralmistral-medium-2505· Mistral AIHugging Face LMArena Human Preference 140K

How is the elo calculated on LM Arena?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219-thinking-32k’s answer.
  1. User

    How is the elo calculated on LM Arena?

  2. Mistral Assistant

    The Elo rating system used in the LMSYS Chatbot Arena is a modified version of the classic Elo system, originally designed for chess but adapted for comparing large language models (LLMs) based on user votes. Here’s how it works in the Arena:


    1. Basic Elo Formula (Adapted for LMs)

    The Elo system assigns a numerical rating to each model, which increases or decreases based on match outcomes (user votes). The key formula for updating ratings after a comparison is:

    [ E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}} ] [ E_B = \frac{1}{1 + 10^{(R_A - R_B)/400}} ]

    Where:

    • (R_A) and (R_B) are the current Elo ratings of Model A and Model B.
    • (E_A) is the expected probability that Model A wins against Model B.
    • (E_B) is the expected probability that Model B wins against Model A.

    After a vote, the ratings are updated as: [ R_A' = R_A + K \times (S_A - E_A) ] [ R_B' = R_B + K \times (S_B - E_B) ]

    Where:

    • (S_A = 1) if Model A wins, (0) if it loses (and vice versa for (S_B)).
    • (K) is the K-factor, which determines how much ratings change per vote (more on this below).

    2. Key Adaptations for the Chatbot Arena

    The LMSYS Arena modifies the classic Elo system in a few important ways:

    A. K-Factor (Volatility Control)

    • The K-factor determines how much a model’s rating changes after a vote.
    • In chess, (K) is often fixed (e.g., 32 for top players), but in the Arena, it’s dynamically adjusted based on:
      • Model uncertainty: Newer models or those with fewer votes have a higher (K) (more volatile ratings).
      • Vote consistency: If a model’s performance is stable, (K) decreases over time.
    • This prevents early fluctuations from skewing ratings permanently.

    B. Tie Handling (Draws)

    • Unlike chess, the Arena allows ties (users can vote "both are equally good").
    • If a tie occurs, both models gain or lose a small fraction of points (or none at all, depending on implementation).

    C. Bayesian Adjustments (Optional)

    • Some versions of the Arena use Bayesian Elo, which incorporates prior distributions to stabilize ratings when data is sparse.
    • This helps avoid extreme swings for new models with few votes.

    D. Leaderboard Smoothing

    • The displayed Elo ratings are often smoothed (e.g., using exponential moving averages) to reduce noise from small vote batches.

    3. How Votes Affect Elo

    • If a higher-rated model wins: It gains a few points (since it was expected to win).
    • If a lower-rated model wins: It gains many more points (an "upset" leads to a bigger rating jump).
    • New models start with a default rating (e.g., 1000) and quickly rise or fall based on early votes.

    4. Example Calculation

    Suppose:

    • Model A (Elo = 1200) vs. Model B (Elo = 1000).
    • A user votes that Model B is better.

    Step 1: Compute Expected Scores [ E_A = \frac{1}{1 + 10^{(1000 - 1200)/400}} = \frac{1}{1 + 10^{-0.5}} \approx 0.76 ] [ E_B = \frac{1}{1 + 10^{(1200 - 1000)/400}} = \frac{1}{1 + 10^{0.5}} \approx 0.24 ]

    Step 2: Update Ratings (Assume K = 32)

    • Since Model B won ((S_B = 1), (S_A = 0)): [ R_A' = 1200 + 32 \times (0 - 0.76) = 1200 - 24.32 = 1175.68 ] [ R_B' = 1000 + 32 \times (1 - 0.24) = 1000 + 24.32 = 1024.32 ]

    Result: Model B’s rating increases significantly (an upset), while Model A’s drops.


    5. Why Elo Works Well for LMs

    • Dynamic: Adjusts quickly to new models (e.g., Mixtral’s rapid rise in late 2023).
    • Fair: Rewards consistent performance, not just raw vote counts.
    • Interpretable: Higher Elo = better model, with clear margins between tiers.

    6. Limitations

    • Positional Bias: Users might favor the first model they see (the Arena randomizes order to mitigate this).
    • Task Dependency: Some models excel at coding, others at chat—Elo averages across all tasks.
    • Voter Subjectivity: Human preferences vary, but large vote volumes smooth this out.

    Conclusion

    The LMSYS Chatbot Arena’s Elo system is a modified, dynamic ranking method that balances responsiveness to new data with stability. It’s why you see models like GPT-4-Turbo, Claude 2.1, and Mixtral at the top—they consistently win head-to-head comparisons.

    For the latest rankings and live Elo updates, check the LMSYS Arena Leaderboard.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.