Skip to content
TheChatArchiveThe Chat Archive
gpt-4.1-2025-04-14· OpenAIHugging Face LMArena Human Preference 140K

There are two rating systems in which the same players play. Why are they different?…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Metallama-4-maverick-17b-128e-instruct’s answer.
  1. User

    There are two rating systems in which the same players play. Why are they different? Namely.

    Sytstem A      System B
    500	           1500	
    600	           1600	
    700	           1700	
    800	           1800	
    900	           1900	
    1000	       2000
    1100	       2100
    1150	       2150
    1200	       2200
    1250	       2250
    1300	       2300
    1350	       2350
    1400	       2400
    1450	       2450
    1500	       2500
    1550	       2550
    1600	       2600
    1650	       2650
    1700	       2700
    1750	       2750
    1800	       2800
    1850	       2850
    1900	       2900
    1950	       2950
    2000	       3000
    2100	       3100
    2200	       3200
    2300	       3300
    2400	       3400
    2500	       3500
    2600	       3600
    2700	       3700
    2800	       3800
    2900	       3900
    3000	       4000
    

    Why wouldn't the two systems converge with the same rating? Note that in system A the starting rating is 1500, in system B it is 2500. Some say because the two systems are identical, only they are scaled of a factor of 1000 points between each other.

    Yet two other different rating systems where more or less the same players play in both converge! chess.com blitz has a starting rating around 1000 while lichess blitz has a starting rating around 1500, and they have those rating ranges.

    Chess.com  Lichess Blitz
    Blitz    (+/- 75)
               N=2620
    500        945
    600        1030
    700        1115
    800        1205
    900        1305
    1000    1395
    1100    1485
    1150    1520
    1200    1560
    1250    1600
    1300    1630
    1350    1685
    1400    1720
    1450    1760
    1500    1795
    1550    1825
    1600    1855
    1650    1895
    1700    1925
    1750    1960
    1800    2000
    1850    2025
    1900    2060
    1950    2100
    2000    2120
    2100    2205
    2200    2270
    2300    2330
    2400    2405
    2500    2495
    2600    2585
    2700    2630
    2800    2715
    2900    2830
    3000    2925
    

    So why one systems does not converge and the other does? It feels like wrong.

  2. Assistant

    Great question! This is a subtle but important point about rating systems and how their scales and calibrations work.

    Let's break it down:


    1. Theoretical Example: System A vs System B

    In your first example, System A and System B are identical except for a constant offset:

    • System B = System A + 1000

    So, a player rated 1000 in A is 2000 in B, 1500 in A is 2500 in B, etc.

    If the only difference is this offset, then the systems are not fundamentally different—they are just using a different "zero point" or "origin." The relative ratings and gaps between players are identical. The absolute numbers are different, but the meaning is the same.

    This is like measuring temperature in Celsius vs Kelvin:

    • 0°C = 273.15 K
    • The difference between two temperatures is the same in both scales.

    So, these systems will never "converge" to the same numbers, because they are just shifted by a constant.


    2. Real-World Example: Chess.com vs Lichess Blitz

    Here, the situation is different. The two systems have:

    • Different starting ratings (Chess.com: ~1000, Lichess: ~1500)
    • Different rating pools (not all the same players, but a lot of overlap)
    • Different rating algorithms (slightly different implementations of Elo/Glicko, different K-factors, etc.)
    • Different player populations (Lichess tends to have a slightly stronger pool, etc.)

    But, over time, the ratings for the same player tend to be similar (not identical, but close) on both sites.
    For example, a 1500 on Chess.com Blitz is roughly a 1795 on Lichess Blitz, but the gap is not a constant 500 points—it varies across the range.

    Why do these systems "converge" (i.e., have similar numbers for similar skill)?

    • Both systems are calibrated so that the average active player is around the starting rating (1000 or 1500).
    • Both systems have rating inflation/deflation mechanisms to keep the pool centered.
    • Both systems have large, active, and overlapping player bases.
    • Both systems use similar rating algorithms (Elo/Glicko variants).

    Over time, the rating numbers "settle" so that a given skill level corresponds to a similar rating number, even if the starting points were different.


    3. Why Doesn't the First Example "Converge"?

    Because in your first example, the entire system is just shifted by a constant.

    • There is no mechanism for the numbers to "drift" toward each other.
    • The relative skill levels are preserved, but the absolute numbers are always offset by 1000.

    In contrast, in real-world systems like Chess.com and Lichess:

    • The rating numbers are not just shifted; they are calibrated to reflect the actual skill distribution of the player pool.
    • The systems are designed so that, for example, 1500 means "average player" (or whatever the target is), regardless of the starting point.

    4. Summary Table

    System TypeOffset Only?Calibrated to Player Pool?Will Ratings "Converge"?
    System A vs BYesNoNo (always offset)
    Chess.com vs LichessNoYesYes (roughly similar)

    5. Key Point

    • If two rating systems are just shifted by a constant, they will never "converge" to the same numbers.
    • If two rating systems are both calibrated to the same player pool and use similar algorithms, their numbers will "converge" to similar values, even if their starting points were different.

    TL;DR

    • System A and B are just shifted by a constant, so they never "converge."
    • Chess.com and Lichess ratings are both calibrated to their player pools, so their numbers "converge" to similar values for similar skill levels.
    • Convergence happens when systems are calibrated to the same population, not when they are just offset.

    If you want more technical details about how rating systems are calibrated, or about rating inflation/deflation, let me know!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.