Skip to content
The Chat Archive
gpt-4.1-2025-04-14· OpenAI LMArena Human Preference 140K

can i use it with more than one rater?

  1. User

    can i use it with more than one rater?

  2. Assistant

    Great question! Lin’s Concordance Correlation Coefficient (CCC) is primarily designed to assess agreement between two raters or two measurement methods. It evaluates both precision (how well the data fit a straight line) and accuracy (how close the data are to the line of perfect concordance).

    What if you have more than two raters?

    • Standard Lin’s CCC is not directly applicable to more than two raters.
    • For multiple raters (or more than two methods), you need to use extensions or alternative statistics.

    Options for Multiple Raters

    1. Generalized Concordance Correlation Coefficient (GCCC):

    2. Intraclass Correlation Coefficient (ICC):

      • The ICC is the most commonly used statistic for assessing agreement among more than two raters.
      • It measures the proportion of variance due to differences between subjects, relative to the total variance (including rater differences and error).
      • ICC is widely used in reliability studies.

    Summary Table

    Number of RatersRecommended Statistic
    2Lin’s CCC
    >2Generalized CCC or ICC

    In summary:

    • Lin’s CCC: Use for two raters/methods.
    • More than two raters: Use generalized CCC (if you want a direct extension) or, more commonly, the Intraclass Correlation Coefficient (ICC).

    If you need help with the formulas or R code for these, let me know!

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.