deepseek v3 0324 gpqa benchmark?
- User
deepseek v3 0324 gpqa benchmark?
- Assistant
DeepSeek-V3 has demonstrated strong performance on the GPQA-Diamond benchmark, achieving a Pass@1 score of 59.1%. This score surpasses that of Claude-3.5, which scored 49.9%, and Qwen2.5, which scored 51.1%, indicating DeepSeek-V3's superior precision on high-difficulty QA tasks[1].
To ensure that this high performance is not due to data contamination, METR conducted a verification by paraphrasing the GPQA-Diamond questions and evaluating DeepSeek-V3 on these paraphrased questions, as well as on a held-out set of GPQA questions. The results showed that DeepSeek-V3's performance remained consistent across these datasets, suggesting that its high benchmark scores are not the result of memorization from training data[2].
In summary, DeepSeek-V3's performance on the GPQA-Diamond benchmark reflects its robust capabilities in handling complex question-answering tasks.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.