Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-2024-11-20· OpenAIHugging Face LMArena Human Preference 140K

There are two rating systems in which the same players play. Why are they different?…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Geminigemini-2.0-flash-thinking-exp-01-21’s answer.
  1. User

    There are two rating systems in which the same players play. Why are they different? Namely.

    Sytstem A      System B
    500	           1500	
    600	           1600	
    700	           1700	
    800	           1800	
    900	           1900	
    1000	       2000
    1100	       2100
    1150	       2150
    1200	       2200
    1250	       2250
    1300	       2300
    1350	       2350
    1400	       2400
    1450	       2450
    1500	       2500
    1550	       2550
    1600	       2600
    1650	       2650
    1700	       2700
    1750	       2750
    1800	       2800
    1850	       2850
    1900	       2900
    1950	       2950
    2000	       3000
    2100	       3100
    2200	       3200
    2300	       3300
    2400	       3400
    2500	       3500
    2600	       3600
    2700	       3700
    2800	       3800
    2900	       3900
    3000	       4000
    

    Why wouldn't the two systems converge with the same rating? Note that in system A the starting rating is 1500, in system B it is 2500. Some say because the two systems are identical, only they are scaled of a factor of 1000 points between each other.

    Yet two other different rating systems where more or less the same players play in both converge! chess.com blitz has a starting rating around 1000 while lichess blitz has a starting rating around 1500, and they have those rating ranges.

    Chess.com  Lichess Blitz
    Blitz    (+/- 75)
               N=2620
    500        945
    600        1030
    700        1115
    800        1205
    900        1305
    1000    1395
    1100    1485
    1150    1520
    1200    1560
    1250    1600
    1300    1630
    1350    1685
    1400    1720
    1450    1760
    1500    1795
    1550    1825
    1600    1855
    1650    1895
    1700    1925
    1750    1960
    1800    2000
    1850    2025
    1900    2060
    1950    2100
    2000    2120
    2100    2205
    2200    2270
    2300    2330
    2400    2405
    2500    2495
    2600    2585
    2700    2630
    2800    2715
    2900    2830
    3000    2925
    

    So why one systems does not converge and the other does? It feels like something is not working. As a additional point, imagine lichess starting with a rating of 3000. I could not believe it would converge to the chess.com ratings after a while because the points in the system (for each player) will be just massive.

  2. Assistant

    The key difference between the two scenarios lies in how the rating systems are designed, particularly in terms of their scaling, starting points, and rating dynamics (e.g., how points are distributed and adjusted after games). Let’s break this down step by step to explain why one pair of systems converges while the other does not.


    1. System A vs. System B: Why They Don’t Converge

    In System A and System B, the ratings are scaled differently, with a fixed offset of 1000 points between them. For example:

    • A player rated 500 in System A is rated 1500 in System B.
    • A player rated 1000 in System A is rated 2000 in System B.

    This suggests that System B is simply a linear transformation of System A, where: [ \text{Rating in System B} = \text{Rating in System A} + 1000 ]

    If the two systems are identical except for this offset, they will never "converge" to the same ratings because the offset is built into the system. The ratings in System B will always be 1000 points higher than in System A, regardless of how many games are played. This is because the scaling and offset are fixed properties of the system.

    In this case, the two systems are not truly independent; they are just different representations of the same underlying rating system. The offset ensures that the ratings will always differ by 1000 points.


    2. Chess.com Blitz vs. Lichess Blitz: Why They Converge

    In contrast, the Chess.com and Lichess Blitz systems are independent rating systems. While they may have different starting points (e.g., Chess.com starts at 1000, Lichess starts at 1500), the ratings in both systems are determined dynamically based on player performance. Over time, the ratings in both systems reflect the same underlying player pool and skill levels, which leads to convergence.

    Here’s why this happens:

    • Dynamic Adjustments: Both systems use similar rating algorithms (likely based on Elo or Glicko), which adjust ratings based on game outcomes. Over time, players’ ratings stabilize around their true skill levels, regardless of the starting point.
    • Shared Player Pool: Many players participate in both systems, so their ratings in each system are influenced by the same set of opponents and results. This creates a natural alignment between the two systems.
    • No Fixed Offset: Unlike System A and System B, there is no fixed offset or scaling factor between Chess.com and Lichess ratings. The ratings are free to adjust independently, allowing them to converge over time.

    For example:

    • A player rated 1200 on Chess.com might stabilize around 1560 on Lichess because the two systems use different baselines and distributions, but the ratings still reflect the same skill level.

    3. What Happens if Lichess Starts at 3000?

    If Lichess were to start players at 3000 instead of 1500, the ratings would likely not converge with Chess.com ratings. Here’s why:

    • Inflation: Starting at 3000 would introduce a massive amount of points into the system. Even if players lose games and their ratings drop, the overall rating pool would remain inflated because the starting point is so high.
    • Different Scaling: The ratings would no longer reflect the same skill levels as Chess.com because the scaling would be completely different. For example, a 3000-rated player on Lichess might correspond to a 2000-rated player on Chess.com, but the relationship would not be consistent across the entire range of ratings.
    • Lack of Convergence: The two systems would diverge because the starting point and rating dynamics would create fundamentally different distributions of ratings.

    This is similar to the issue with System A and System B: if one system is scaled or offset differently, the ratings will not converge.


    4. Key Factors That Determine Convergence

    For two rating systems to converge, the following conditions must be met:

    1. Dynamic Adjustments: Both systems must adjust ratings based on player performance using similar algorithms (e.g., Elo or Glicko).
    2. Shared Player Pool: The same players (or a significant overlap) must participate in both systems, so the ratings reflect the same skill levels.
    3. No Fixed Offset or Scaling: The systems must not have a fixed offset or scaling factor that artificially separates the ratings.

    If these conditions are not met, the systems will not converge. For example:

    • System A and System B fail to converge because of the fixed offset.
    • Chess.com and Lichess Blitz converge because they meet all three conditions.

    5. Conclusion

    The reason System A and System B do not converge is that they are not truly independent systems; they are just scaled versions of each other with a fixed offset. In contrast, Chess.com and Lichess Blitz are independent systems that dynamically adjust ratings based on player performance, allowing them to converge over time.

    If Lichess were to start at 3000, the ratings would likely diverge from Chess.com because the starting point would introduce inflation and disrupt the alignment between the two systems. This highlights the importance of starting points, scaling, and rating dynamics in determining whether two systems will converge.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.