How is the elo calculated on LM Arena?
- User
How is the elo calculated on LM Arena?
Assistant
The Elo rating system used in the LMSYS Chatbot Arena is a modified version of the classic Elo system, originally designed for chess but adapted for comparing large language models (LLMs) based on user votes. Here’s how it works in the Arena:
1. Basic Elo Formula (Adapted for LMs)
The Elo system assigns a numerical rating to each model, which increases or decreases based on match outcomes (user votes). The key formula for updating ratings after a comparison is:
[ E_A = \frac{1}{1 + 10^{(R_B - R_A)/400}} ] [ E_B = \frac{1}{1 + 10^{(R_A - R_B)/400}} ]
Where:
- (R_A) and (R_B) are the current Elo ratings of Model A and Model B.
- (E_A) is the expected probability that Model A wins against Model B.
- (E_B) is the expected probability that Model B wins against Model A.
After a vote, the ratings are updated as: [ R_A' = R_A + K \times (S_A - E_A) ] [ R_B' = R_B + K \times (S_B - E_B) ]
Where:
- (S_A = 1) if Model A wins, (0) if it loses (and vice versa for (S_B)).
- (K) is the K-factor, which determines how much ratings change per vote (more on this below).
2. Key Adaptations for the Chatbot Arena
The LMSYS Arena modifies the classic Elo system in a few important ways:
A. K-Factor (Volatility Control)
- The K-factor determines how much a model’s rating changes after a vote.
- In chess, (K) is often fixed (e.g., 32 for top players), but in the Arena, it’s dynamically adjusted based on:
- Model uncertainty: Newer models or those with fewer votes have a higher (K) (more volatile ratings).
- Vote consistency: If a model’s performance is stable, (K) decreases over time.
- This prevents early fluctuations from skewing ratings permanently.
B. Tie Handling (Draws)
- Unlike chess, the Arena allows ties (users can vote "both are equally good").
- If a tie occurs, both models gain or lose a small fraction of points (or none at all, depending on implementation).
C. Bayesian Adjustments (Optional)
- Some versions of the Arena use Bayesian Elo, which incorporates prior distributions to stabilize ratings when data is sparse.
- This helps avoid extreme swings for new models with few votes.
D. Leaderboard Smoothing
- The displayed Elo ratings are often smoothed (e.g., using exponential moving averages) to reduce noise from small vote batches.
3. How Votes Affect Elo
- If a higher-rated model wins: It gains a few points (since it was expected to win).
- If a lower-rated model wins: It gains many more points (an "upset" leads to a bigger rating jump).
- New models start with a default rating (e.g., 1000) and quickly rise or fall based on early votes.
4. Example Calculation
Suppose:
- Model A (Elo = 1200) vs. Model B (Elo = 1000).
- A user votes that Model B is better.
Step 1: Compute Expected Scores [ E_A = \frac{1}{1 + 10^{(1000 - 1200)/400}} = \frac{1}{1 + 10^{-0.5}} \approx 0.76 ] [ E_B = \frac{1}{1 + 10^{(1200 - 1000)/400}} = \frac{1}{1 + 10^{0.5}} \approx 0.24 ]
Step 2: Update Ratings (Assume K = 32)
- Since Model B won ((S_B = 1), (S_A = 0)): [ R_A' = 1200 + 32 \times (0 - 0.76) = 1200 - 24.32 = 1175.68 ] [ R_B' = 1000 + 32 \times (1 - 0.24) = 1000 + 24.32 = 1024.32 ]
Result: Model B’s rating increases significantly (an upset), while Model A’s drops.
5. Why Elo Works Well for LMs
- Dynamic: Adjusts quickly to new models (e.g., Mixtral’s rapid rise in late 2023).
- Fair: Rewards consistent performance, not just raw vote counts.
- Interpretable: Higher Elo = better model, with clear margins between tiers.
6. Limitations
- Positional Bias: Users might favor the first model they see (the Arena randomizes order to mitigate this).
- Task Dependency: Some models excel at coding, others at chat—Elo averages across all tasks.
- Voter Subjectivity: Human preferences vary, but large vote volumes smooth this out.
Conclusion
The LMSYS Chatbot Arena’s Elo system is a modified, dynamic ranking method that balances responsiveness to new data with stability. It’s why you see models like GPT-4-Turbo, Claude 2.1, and Mixtral at the top—they consistently win head-to-head comparisons.
For the latest rankings and live Elo updates, check the LMSYS Arena Leaderboard.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.