Skip to content
The Chat Archive
llama-4-maverick-17b-128e-instruct· Meta LMArena Human Preference 140K

There are two rating systems in which the same players play. Why are they different?…

  1. User

    There are two rating systems in which the same players play. Why are they different? Namely.

    Sytstem A      System B
    500	           1500	
    600	           1600	
    700	           1700	
    800	           1800	
    900	           1900	
    1000	       2000
    1100	       2100
    1150	       2150
    1200	       2200
    1250	       2250
    1300	       2300
    1350	       2350
    1400	       2400
    1450	       2450
    1500	       2500
    1550	       2550
    1600	       2600
    1650	       2650
    1700	       2700
    1750	       2750
    1800	       2800
    1850	       2850
    1900	       2900
    1950	       2950
    2000	       3000
    2100	       3100
    2200	       3200
    2300	       3300
    2400	       3400
    2500	       3500
    2600	       3600
    2700	       3700
    2800	       3800
    2900	       3900
    3000	       4000
    

    Why wouldn't the two systems converge with the same rating? Note that in system A the starting rating is 1500, in system B it is 2500. Some say because the two systems are identical, only they are scaled of a factor of 1000 points between each other.

    Yet two other different rating systems where more or less the same players play in both converge! chess.com blitz has a starting rating around 1000 while lichess blitz has a starting rating around 1500, and they have those rating ranges.

    Chess.com  Lichess Blitz
    Blitz    (+/- 75)
               N=2620
    500        945
    600        1030
    700        1115
    800        1205
    900        1305
    1000    1395
    1100    1485
    1150    1520
    1200    1560
    1250    1600
    1300    1630
    1350    1685
    1400    1720
    1450    1760
    1500    1795
    1550    1825
    1600    1855
    1650    1895
    1700    1925
    1750    1960
    1800    2000
    1850    2025
    1900    2060
    1950    2100
    2000    2120
    2100    2205
    2200    2270
    2300    2330
    2400    2405
    2500    2495
    2600    2585
    2700    2630
    2800    2715
    2900    2830
    3000    2925
    

    So why one systems does not converge and the other does? It feels like wrong.

  2. Assistant

    The difference in ratings between two systems can be attributed to several factors, including the rating system's design, the pool of players, and the scaling or calibration of the ratings. Let's dive into the specifics of the provided examples.

    System A and System B

    The first example presents two rating systems, A and B, with a linear relationship between their ratings. System A ranges from 500 to 3000, and System B ranges from 1500 to 4000. The difference between corresponding ratings in the two systems is not constant but rather maintains an offset that scales linearly.

    System A      System B     Difference
    500           1500         1000
    600           1600         1000
    ...           ...          ...
    3000          4000         1000
    

    The consistent difference of 1000 points between the corresponding ratings in System A and System B indicates that they are indeed scaled versions of each other. The starting ratings (500 in System A and 1500 in System B) and the linear progression suggest that the systems are equivalent but shifted.

    Chess.com and Lichess Blitz Ratings

    The second example compares Chess.com and Lichess Blitz ratings. Despite different starting ratings (around 1000 for Chess.com and 1500 for Lichess), the ratings seem to converge relatively well, especially when considering the standard deviation.

    The key observation here is that while there's a general correlation between the ratings on the two platforms, the relationship isn't perfectly linear across the entire range. However, for the majority of players, the ratings correspond reasonably well.

    Why One Converges and the Other Doesn't

    1. Design and Calibration: The primary reason for the difference in convergence lies in how the rating systems are designed and calibrated. System A and System B appear to be identical in structure but scaled. In contrast, Chess.com and Lichess might use different rating algorithms (e.g., variations of Elo) that, despite their differences, manage to somewhat align their ratings through similar player pools and competitive dynamics.

    2. Rating System Algorithms: The algorithms used by Chess.com and Lichess might incorporate different factors such as rating deviation, volatility, or use different K-factors, which can affect how ratings change over time. These differences can lead to variations in how ratings are assigned and updated.

    3. Player Pool and Behavior: The pool of players and their behavior on the two platforms can influence the ratings. If the same players are active on both platforms, their relative skill levels are likely to be similarly assessed. However, differences in player pool composition, activity levels, and even the platforms' user interfaces can affect ratings.

    4. Scaling and Offset: System A and B have a clear linear relationship with a consistent offset. The Chess.com and Lichess ratings, while correlated, don't have as straightforward a relationship. The scaling isn't perfectly linear, and there might be more variability in how ratings are distributed.

    Conclusion

    The convergence or divergence of rating systems depends on their underlying design, the algorithms used to update ratings, the pool of players, and how the systems are calibrated. While System A and B are scaled versions of each other and thus don't converge to the same absolute values, Chess.com and Lichess ratings converge relatively well due to similarities in their underlying structures and the shared player base, despite differences in their rating systems.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.