Skip to content
TheChatArchiveThe Chat Archive
Claudeclaude-3-7-sonnet-20250219-thinking-32k· AnthropicHugging Face LMArena Human Preference 140K

I am running an experiment where I am trying to obtain a subjective ranking of 30…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with gpt-4.1-2025-04-14’s answer.
  1. User

    I am running an experiment where I am trying to obtain a subjective ranking of 30 concepts (e.g. design principles). I always present two principles to a participant and they can say which one is more important, or that they are equally important. I want to calculate the required sample size of these pairwise comparisons to obtain a ranking that is stable. How do I approach this problem?

  2. Claude Assistant

    To calculate the required sample size for stable rankings in your pairwise comparison experiment:

    1. Bradley-Terry modeling approach: This statistical model converts pairwise comparisons into a continuous scale of "worth" for each item. You can run simulations with increasing sample sizes until parameter estimates stabilize (standard errors fall below a threshold you determine acceptable).

    2. Ranking stability analysis: Start with a moderate number of comparisons (perhaps 100-150), generate an initial ranking, then:

      • Add batches of additional comparisons
      • Recalculate rankings after each batch
      • Measure stability using Kendall's tau correlation between consecutive rankings
      • Stop when stability reaches your desired threshold (e.g., tau > 0.95)
    3. Power analysis: If you have estimates of the expected difference in importance between adjacent items, you can calculate sample size based on detecting this difference with sufficient power.

    For 30 items (435 possible pairs), you'll likely need 3-5 comparisons per pair for reasonable stability, but this varies based on how close items are in actual importance.

    Would you like me to elaborate on any of these approaches?

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.