I am running an experiment where I am trying to obtain a subjective ranking of 30…
- User
I am running an experiment where I am trying to obtain a subjective ranking of 30 concepts (e.g. design principles). I always present two principles to a participant and they can say which one is more important, or that they are equally important. I want to calculate the required sample size of these pairwise comparisons to obtain a ranking that is stable. How do I approach this problem?
- Assistant
Great question! You're essentially running a pairwise comparison experiment to derive a global ranking of 30 concepts based on subjective judgments. Determining the required sample size (i.e., the number of pairwise comparisons) for a stable ranking is a classic problem in psychometrics, statistics, and preference learning.
Let's break down the approach:
1. Understanding the Problem
- 30 concepts → There are ( \binom{30}{2} = 435 ) unique pairs.
- For each pair, a participant chooses one as more important, or says "equally important."
- You want to know: How many comparisons (total, or per pair) are needed for a stable ranking?
2. Key Factors
- Stability: Usually means that if you repeated the experiment, the ranking would not change much.
- Noise: Human judgments are noisy; more data reduces noise.
- Model: You need a way to aggregate pairwise data into a ranking (e.g., Bradley-Terry-Luce (BTL) model, Elo rating, or simple win counts).
3. Approaches to Sample Size Estimation
A. Full Pairwise Matrix
If you want to compare every pair at least once, you need 435 comparisons. But that's just one comparison per pair, which is usually not enough for stability.
B. Multiple Judgments per Pair
- If you want ( k ) judgments per pair, you need ( 435 \times k ) comparisons.
- More judgments per pair → more stable estimates.
C. Statistical Power Analysis
- Power analysis for pairwise comparison models (like BTL) is complex, but the general idea is:
- The more comparisons per pair, the more confident you are in the estimated "strength" of each concept.
- For stable rankings, 3–5 judgments per pair is a common minimum in practice, but more is better.
D. Simulation Approach (Recommended)
Because the stability depends on the actual distribution of preferences (some pairs may be very close, others obvious), simulation is often used:
- Simulate a "true" ranking and generate synthetic pairwise data with noise.
- Aggregate the data using your chosen model.
- Repeat for different numbers of comparisons per pair.
- Measure stability (e.g., how often the top-5 concepts change, or Kendall's tau correlation between runs).
- Choose the sample size where the ranking stabilizes.
4. Practical Guidelines
- Minimum: 3–5 judgments per pair (so 1,305–2,175 total comparisons).
- Better: 10 judgments per pair (4,350 total).
- If you can't do all pairs: Use an incomplete design (e.g., each concept is compared to a subset of others), but ensure the comparison graph is connected.
5. References and Tools
- Bradley-Terry Model: Wikipedia
- Sample size for pairwise comparisons:
- R package:
prefmodorBradleyTerry2for modeling and simulation.
6. Summary Table
Judgments per pair Total comparisons Stability (rule of thumb) 1 435 Very unstable 3 1,305 Minimum for rough ranking 5 2,175 Decent stability 10 4,350 Good stability
7. Final Recommendations
- Aim for at least 3–5 judgments per pair for a stable ranking.
- If resources are limited, consider incomplete block designs (not all pairs, but enough to connect the network).
- Simulate your specific scenario if possible, to empirically determine stability for your data and model.
If you tell me more about your constraints (number of participants, time per participant, etc.), I can help you design an efficient experiment!
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.