How is the elo calculated on LM Arena?
- User
How is the elo calculated on LM Arena?
Assistant
The Elo rating system used in LM Arena is a clever adaptation of the classic chess rating method for comparing language models. Here's how it works:
Basic Elo Calculation
-
Starting Point: All models begin with a base Elo rating (typically 1200).
-
Win Probability Formula: When two models with ratings R₁ and R₂ face each other, the expected probability that Model 1 will win is:
E₁ = 1 / (1 + 10^((R₂ - R₁)/400)) -
Rating Update: After a user votes for one model over another, the ratings update:
R₁_new = R₁ + K × (S₁ - E₁)Where:
- K is the "K-factor" (how much a single match affects ratings)
- S₁ is the actual outcome (1 for win, 0 for loss)
- E₁ is the expected outcome calculated above
LM Arena Specifics
The LMSYS team has implemented several important refinements to make Elo work well for language model evaluation:
-
Dynamic K-factor: Models with fewer votes have higher K-factors, meaning their ratings change more dramatically with each new vote. As models accumulate more votes, their ratings stabilize.
-
Confidence Intervals: The leaderboard shows not just the Elo rating but also confidence intervals (typically displayed as a range or error bars), indicating the statistical certainty of a model's position.
-
Tie Handling: When users indicate both models performed equally well, this is handled as a tie (0.5 points for each model).
-
Vote Quality Control: They implement various measures to filter out low-quality or biased votes.
-
Rating Calibration: Periodically, they may recalibrate the entire leaderboard to maintain consistency as new models are added.
What makes this system particularly elegant is that:
- It naturally accounts for the strength of the opponent (beating a stronger model earns more points)
- A 100-point Elo difference corresponds to roughly a 64% win probability
- It can accommodate new models being added at any time
This system has proven remarkably effective at converting hundreds of thousands of individual human judgments into a meaningful, stable ranking that closely aligns with other evaluation methods.
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.