Skip to content
The Chat Archive
gemini-2.5-flash-preview-04-17-grounding· Google LMArena Search Arena 24K

gpqa vs gpqa diamond

  1. User

    gpqa vs gpqa diamond

  2. Assistant

    GPQA and GPQA Diamond are both challenging datasets used to evaluate the capabilities of large language models (LLMs), particularly in understanding and reasoning about complex scientific topics. GPQA stands for "Graduate-Level Google-Proof Q&A Benchmark"[1][2][3]. The questions are designed to be very difficult, even for skilled non-experts with internet access, and require deep domain expertise[2][4][5].

    Here's a breakdown of the differences:

    • GPQA (Main Set): This is the primary dataset consisting of 448 multiple-choice questions across biology, physics, and chemistry[6][2][7]. These questions were written by domain experts (individuals with or pursuing PhDs in the relevant fields)[2][8]. While challenging, this set includes questions where even experts might make mistakes or there could be slight ambiguity[9]. Experts achieved around 65% accuracy on the main set, while skilled non-experts scored only about 34%[6][2].

    • GPQA Diamond: This is a subset of the main GPQA dataset and is considered the highest quality and most challenging subset[9][10]. It contains 198 questions that meet stricter criteria[1][10]. Specifically, a question is included in the Diamond set only if:

      • Both domain experts who reviewed the question answered it correctly.
      • A majority of non-expert validators answered it incorrectly (typically no more than one out of three non-experts got it right)[9][11][12].
      • Even if one expert initially answered incorrectly, the question could still be included if the expert clearly identified their mistake or understanding after reviewing the question writer's explanation[9].

    In essence, GPQA Diamond represents the most difficult and unambiguous questions from the GPQA dataset, where expert consensus is high and non-expert performance is particularly low[11][12][13]. This makes it a rigorous benchmark for testing the limits of advanced AI systems on challenging, graduate-level scientific reasoning[10][14].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.