Skip to content
The Chat Archive
grok-3-preview-02-24· xAI LMArena Human Preference 140K

There are two rating systems in which the same players play. Why are they different?…

  1. User

    There are two rating systems in which the same players play. Why are they different? Namely.

    Sytstem A      System B
    500	           1500	
    600	           1600	
    700	           1700	
    800	           1800	
    900	           1900	
    1000	       2000
    1100	       2100
    1150	       2150
    1200	       2200
    1250	       2250
    1300	       2300
    1350	       2350
    1400	       2400
    1450	       2450
    1500	       2500
    1550	       2550
    1600	       2600
    1650	       2650
    1700	       2700
    1750	       2750
    1800	       2800
    1850	       2850
    1900	       2900
    1950	       2950
    2000	       3000
    2100	       3100
    2200	       3200
    2300	       3300
    2400	       3400
    2500	       3500
    2600	       3600
    2700	       3700
    2800	       3800
    2900	       3900
    3000	       4000
    

    Why wouldn't the two systems converge with the same rating? Note that in system A the starting rating is 1500, in system B it is 2500. Some say because the two systems are identical, only they are scaled of a factor of 1000 points between each other.

    Yet two other different rating systems where more or less the same players play in both converge! chess.com blitz has a starting rating around 1000 while lichess blitz has a starting rating around 1500, and they have those rating ranges.

    Chess.com  Lichess Blitz
    Blitz    (+/- 75)
               N=2620
    500        945
    600        1030
    700        1115
    800        1205
    900        1305
    1000    1395
    1100    1485
    1150    1520
    1200    1560
    1250    1600
    1300    1630
    1350    1685
    1400    1720
    1450    1760
    1500    1795
    1550    1825
    1600    1855
    1650    1895
    1700    1925
    1750    1960
    1800    2000
    1850    2025
    1900    2060
    1950    2100
    2000    2120
    2100    2205
    2200    2270
    2300    2330
    2400    2405
    2500    2495
    2600    2585
    2700    2630
    2800    2715
    2900    2830
    3000    2925
    

    So why one systems does not converge and the other does? It feels like something is not working. As a additional point, imagine lichess starting with a rating of 3000. I could not believe it would converge to the chess.com ratings after a while because the points in the system (for each player) will be just massive.

  2. Assistant

    This is a fascinating question about rating systems, and it touches on several important concepts in rating system design, including scaling, inflation, convergence, and the mechanics of how ratings are updated. Let’s break this down step by step to explain why the two systems in your first example (System A and System B) do not converge, while the two systems in your second example (Chess.com Blitz and Lichess Blitz) do, and why starting ratings and system mechanics matter.


    1. Understanding Rating Systems

    Rating systems like Elo (used in chess and many other competitive domains) are designed to measure relative skill levels among players. The key features of such systems include:

    • Relative Nature: Ratings are meaningful only in comparison to other players within the same system. A rating of 1500 in one system does not inherently mean the same thing as a rating of 1500 in another system unless the systems are designed to align.
    • Starting Rating: The initial rating assigned to new players (e.g., 1500 in System A, 2500 in System B) is arbitrary and serves as an anchor for the system. Over time, the distribution of ratings evolves based on game outcomes and the system's mechanics.
    • Rating Updates: Ratings are updated after games based on the expected outcome (derived from the rating difference between players) and the actual outcome. The formula typically looks like this for Elo systems: [ R_{\text{new}} = R_{\text{old}} + K \cdot (\text{Actual Score} - \text{Expected Score}) ] where (K) is a constant that controls the magnitude of rating changes, and the expected score is a function of the rating difference.
    • Convergence: Two rating systems are said to "converge" if, over time, the ratings of players in both systems become highly correlated and can be mapped to each other with a simple transformation (e.g., a linear scaling).

    2. System A vs. System B: Why They Don’t Converge

    In your first example, System A and System B have a consistent relationship: System B ratings are exactly 1000 points higher than System A ratings for all players. This suggests that the two systems are identical in their mechanics, except for a simple offset in the starting rating (1500 in System A vs. 2500 in System B). Here’s why they don’t converge:

    Reason: They Are Already Converged (But Offset)

    • The two systems are not converging to the same numerical values because they are already equivalent in terms of relative skill measurement, but they use different scales. Specifically, System B is just System A shifted upward by 1000 points.
    • This offset is preserved because the mechanics of rating updates (e.g., the Elo formula) are scale-invariant. In an Elo system, the expected score between two players depends only on the difference in their ratings, not their absolute values. For example:
      • In System A, if Player 1 has a rating of 1500 and Player 2 has a rating of 1600, the rating difference is 100, and the expected score is calculated based on that difference.
      • In System B, the same players have ratings of 2500 and 2600, respectively, but the rating difference is still 100, so the expected score and rating updates are identical in relative terms.
    • As a result, the two systems will never "converge" to the same numerical values because the offset of 1000 points is baked into the system from the start and preserved by the rating update mechanics.

    Why It Feels Like They Should Converge

    You might expect the systems to converge to the same numerical values because, intuitively, you think of ratings as absolute measures of skill. However, ratings are not absolute; they are relative and system-specific. The fact that System B starts at 2500 and System A starts at 1500 is arbitrary, and the systems are already "converged" in the sense that there is a perfect linear mapping between them: ( \text{System B Rating} = \text{System A Rating} + 1000 ).

    What Would Cause Convergence?

    For System A and System B to converge to the same numerical values, there would need to be a mechanism that actively adjusts ratings to eliminate the offset. For example:

    • If one system had a different (K)-factor (causing larger or smaller rating changes), the systems might drift apart or converge over time.
    • If one system had rating deflation or inflation (e.g., due to new players entering at a different average skill level), the offset might change.
    • If the systems were periodically "recalibrated" to align their averages or medians, they could converge.

    However, in your example, none of these mechanisms exist, so the offset of 1000 points is preserved indefinitely.


    3. Chess.com Blitz vs. Lichess Blitz: Why They Converge

    In your second example, Chess.com Blitz and Lichess Blitz ratings do not have a simple offset like System A and System B. Instead, the relationship between the ratings is more complex, and the systems appear to "converge" in the sense that the ratings of players in both systems become highly correlated and can be approximated by a non-linear mapping. Here’s why this happens:

    Reason 1: Different System Mechanics

    Chess.com and Lichess use slightly different rating systems, even though both are based on Elo or similar logistic models. Differences in mechanics can include:

    • Starting Ratings: Chess.com starts new players at around 1000, while Lichess starts them at around 1500. This creates an initial offset, but it’s not preserved because of other differences.
    • K-Factor: The (K)-factor, which controls how much ratings change after a game, may differ between the two systems. A higher (K)-factor leads to more volatile ratings, while a lower (K)-factor stabilizes ratings over time. If the (K)-factors differ, the systems will not maintain a simple offset.
    • Rating Floors: Some systems impose rating floors (e.g., Chess.com may prevent ratings from dropping below a certain value), which can compress the lower end of the rating distribution.
    • Player Pool Dynamics: The influx of new players, their skill levels, and how quickly they improve can cause rating inflation or deflation, which affects the overall distribution of ratings differently in each system.

    These differences mean that the two systems are not simply scaled versions of each other, unlike System A and System B. Instead, they evolve independently, and the relationship between ratings in the two systems is determined by the actual skill levels of the players and how those skill levels are reflected in the rating updates.

    Reason 2: Convergence Due to Shared Player Pool

    Despite the differences in mechanics, Chess.com and Lichess ratings converge in the sense that they both measure the same underlying skill (blitz chess ability) for a largely overlapping player pool. Over time, as players play many games in both systems, their ratings in each system stabilize to reflect their true skill relative to the rest of the player pool. This leads to a high correlation between ratings in the two systems, even if the numerical values differ.

    The data you provided shows that the relationship between Chess.com and Lichess ratings is not a simple offset but rather a more complex mapping. For example:

    • At the lower end (e.g., Chess.com 500 ≈ Lichess 945), the difference is around 445 points.
    • At the middle (e.g., Chess.com 1500 ≈ Lichess 1795), the difference is around 295 points.
    • At the higher end (e.g., Chess.com 3000 ≈ Lichess 2925), the difference is around -75 points.

    This non-linear relationship arises because the two systems have different rating distributions and mechanics, but they are still measuring the same underlying skill. The "convergence" you observe is not numerical convergence (i.e., the ratings becoming equal) but rather a stable, predictable mapping between the two systems.

    Reason 3: Rating Compression and Expansion

    The non-linear relationship between Chess.com and Lichess ratings also reflects differences in how ratings are distributed:

    • Lower Ratings: Lichess ratings are more compressed at the lower end, meaning that a small difference in skill corresponds to a smaller difference in rating compared to Chess.com. This is why a Chess.com rating of 500 maps to a Lichess rating of 945, while a Chess.com rating of 1000 maps to a Lichess rating of 1395 (a difference of 500 vs. 450).
    • Higher Ratings: At the higher end, Lichess ratings are more expanded, meaning that top players on Lichess have ratings that are closer to their Chess.com counterparts (e.g., Chess.com 3000 ≈ Lichess 2925).

    This compression and expansion are caused by differences in the (K)-factor, rating floors, and the overall player pool dynamics in each system.


    4. What If Lichess Started at 3000?

    You asked whether Lichess ratings would converge to Chess.com ratings if Lichess started new players at 3000 instead of 1500. The answer is no, they would not converge to the same numerical values, but they would still converge in the sense of having a predictable mapping.

    Here’s why:

    • If Lichess started new players at 3000, the entire rating distribution would be shifted upward by 1500 points compared to the current Lichess system. For example, a player who is currently 1500 on Lichess would be 3000 in the new system, a player who is currently 2000 would be 3500, and so on.
    • However, the relative differences between players’ ratings would remain the same, and the rating updates would still be based on those differences. As a result, the system would function identically to the current Lichess system, just with all ratings increased by 1500.
    • The relationship between Chess.com and the new Lichess system would still be a predictable mapping, but the numerical values would be even further apart. For example, a Chess.com rating of 1500 might map to a new Lichess rating of 3295 (1795 + 1500), and a Chess.com rating of 3000 might map to a new Lichess rating of 4425 (2925 + 1500).

    In short, changing the starting rating does not cause numerical convergence because the starting rating is arbitrary and does not affect the relative skill measurements within the system. Convergence (in the sense of a stable mapping) occurs because both systems measure the same underlying skill, not because of the numerical values of the ratings.


    5. Why It Feels Like Something Is Not Working

    The intuition that "something is not working" likely stems from the expectation that ratings in different systems should converge to the same numerical values. However, this expectation is based on a misunderstanding of how rating systems work. Ratings are not absolute measures of skill; they are relative measures within a specific system. Two systems can be perfectly equivalent in measuring skill (i.e., have a perfect correlation) without having the same numerical values.

    In the case of System A and System B, the systems are equivalent but offset by 1000 points, so they don’t need to converge numerically—they are already converged in terms of skill measurement. In the case of Chess.com and Lichess, the systems are not identical (due to differences in mechanics), but they converge in the sense of having a stable, predictable mapping because they measure the same skill in a shared player pool.


    6. Key Takeaways

    • System A vs. System B: These systems do not converge numerically because they are identical except for a 1000-point offset, which is preserved by the scale-invariant mechanics of rating updates. They are already "converged" in the sense of having a perfect linear mapping.
    • Chess.com vs. Lichess: These systems converge in the sense of having a stable, predictable (but non-linear) mapping because they measure the same skill in a shared player pool, despite differences in mechanics (e.g., starting ratings, (K)-factors, rating floors).
    • Starting Ratings: Changing the starting rating (e.g., Lichess starting at 3000) does not cause numerical convergence because starting ratings are arbitrary and do not affect relative skill measurements. It only shifts the entire rating distribution.
    • Convergence: True convergence (numerical or otherwise) requires either identical mechanics (as in System A and System B) or a shared player pool and consistent skill measurement (as in Chess.com and Lichess). Numerical convergence would require an active mechanism to align the systems, such as recalibration.

    7. Final Thought

    If you want two rating systems to converge numerically (i.e., have the same ratings for the same players), you would need to design them with identical mechanics and periodically recalibrate them to align their averages or medians. Without such a mechanism, the systems will either maintain a fixed offset (as in System A and System B) or develop a stable but non-linear mapping (as in Chess.com and Lichess).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.