Skip to content
TheChatArchiveThe Chat Archive
Coherecommand-a-03-2025· CohereHugging Face LMArena Human Preference 140K

There is a benchmark that says that 30% of the questions are new, while the rest of the…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-7-sonnet-20250219’s answer.
  1. User

    There is a benchmark that says that 30% of the questions are new, while the rest of the questions are in the public domain, answers included. The top contestants in that benchmark score from high 60% to 80%.

    Is it plausible that contestants are mostly solving the known questions rather than the new ones (that are unknown) ?

  2. Cohere Assistant

    Yes, it is plausible that contestants are primarily solving the known questions rather than the new ones. Here’s why:

    1. Public Domain Questions Dominate: Since 70% of the questions are in the public domain with answers included, contestants can prepare extensively for these questions. They can study known solutions, practice similar problems, and memorize answers, giving them a significant advantage on this portion of the benchmark.

    2. New Questions Are Unpredictable: The 30% of new questions are, by definition, unknown and cannot be prepared for in advance. These questions require on-the-spot problem-solving, creativity, and critical thinking, which are generally more challenging than recalling or applying known solutions.

    3. Scoring Distribution: The top contestants score between 60% and 80%. Given that 70% of the questions are known, it is reasonable to assume that contestants are scoring very high on these known questions (likely close to 90-100%) and much lower on the new questions (possibly below 50%). This would still result in an overall score within the 60-80% range.

    4. Efficiency of Preparation: Preparing for known questions is more efficient and yields higher returns in terms of score improvement compared to preparing for unknown, unpredictable questions. Contestants may therefore focus their efforts on mastering the known questions.

    5. Risk Aversion: Contestants might prioritize minimizing errors on known questions rather than taking risks on new, unfamiliar ones, especially if the benchmark rewards consistency and accuracy.

    In summary, the dominance of known questions, combined with their predictability and the efficiency of preparation, makes it highly plausible that contestants are relying heavily on solving these questions to achieve their high scores, rather than excelling on the new, unknown ones.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.