Skip to content
The Chat Archive
GeminiGemini (model unspecified)· GoogleHugging Face ShareLM (browser-extension contributions)

I have model judgements between two responses with 5 options: response a/b is…

  1. User

    I have model judgements between two responses with 5 options: response a/b is slightly/strongly better than response b/a, and a tie. To what numerical values it is common to map these judgments?

  2. Gemini Assistant

    When mapping 5-point pairwise comparison data (Likert-style model judgments) to numerical values, the "correct" mapping depends heavily on your downstream application—whether you are calculating simple aggregate metrics, computing Elo ratings, or training a Reward Model for RLHF.Here are the three most common mapping schemes used in the industry.1. The Zero-Centered "Reward" Scale (+2 to -2)This is the most intuitive scale for visualization and simple aggregation. It treats the comparisons as a spectrum of "goodness" centered around a neutral point. It assumes the semantic distance between "Tie" and "Slightly" is the same as between "Slightly" and "Strongly" (linearity).JudgmentValueResponse A is strongly better+2Response A is slightly better+1Tie0Response B is slightly better-1Response B is strongly better-2Export to SheetsBest for: Calculating an average "score" for a model to see if it tends to be better or worse than a baseline on average.Math: If you average these values across a dataset and the result is positive, Model A is generally superior.2. The Probability / Win-Rate Scale (0 to 1)This mapping normalizes judgments into a range between 0 and 1. This is very common when feeding data into Elo rating systems or Bradley-Terry models, which interpret results as probabilities of winning.JudgmentValue (Scheme A)Value (Scheme B)A Strongly Better1.01.0A Slightly Better0.751.0 (binary collapse)Tie0.50.5B Slightly Better0.250.0 (binary collapse)B Strongly Better0.00.0Export to SheetsScheme A (0.75/0.25): Assigns "partial credit" for slight wins. This preserves the nuance that a slight win is less certain or less dominant than a strong win.Scheme B (Binary Collapse): Treats "Strongly" and "Slightly" identically as a "Win." This is common in LMSYS Chatbot Arena style Elo calculations, where the magnitude of the win is often ignored in favor of the raw win/loss outcome to reduce noise.3. The Ordinal Integer Scale (1 to 5)Standard in psychometrics and survey analysis (Likert scales), but less common in modern LLM evaluation unless you are treating the output as a regression target for a neural network.A Strongly Better: 5A Slightly Better: 4Tie: 3B Slightly Better: 2B Strongly Better: 1Critical Nuance: Is the scale linear?A major challenge in mapping these judgments is that the distance between "Tie" and "Slightly Better" is often much smaller than the distance between "Slightly Better" and "Strongly Better."If you are training a Reward Model, researchers often find that mapping "Strongly" to a much higher value helps the model learn stronger signals.Linearly: 1 vs 2Non-linearly: 1 (Slightly) vs 3 or 4 (Strongly)Recommendation: If you are just starting your analysis, use the Zero-Centered Scale (+2 to -2). It is the easiest to interpret: a positive mean implies A is better, a negative mean implies B is better, and the magnitude tells you how dominant the difference is.Would you like me to show you how to calculate a weighted Elo rating using these values?