what's QWEN-2.5 coder 3B / 7B score onlivecodebench, codeforces, livebench?
- User
what's QWEN-2.5 coder 3B / 7B score onlivecodebench, codeforces, livebench?
Assistant
Based on the search results, here's the information regarding the scores for Qwen2.5 Coder models on the specified benchmarks:
LiveCodeBench:
- The search results frequently mention the Qwen2.5-7B-Instruct model achieving a score of 63.7% pass@1 on LiveCodeBench[1].
- Another source mentions the Qwen2.5-Coder-32B-Instruct model achieved the best performance among open-source models on LiveCodeBench (as of its release), showing competitive performance with GPT-4o[2][3]. However, a specific numerical score for this model on LiveCodeBench is not provided in the snippets.
- A source evaluating the Qwen2.5-72B-Instruct model (not the Coder variant) shows a score of 55.5 on LiveCodeBench (version 2305-2409)[4].
- There is no specific score mentioned for Qwen2.5-Coder-3B or Qwen2.5-Coder-7B (non-Instruct) on LiveCodeBench in the provided results.
Codeforces:
- LiveCodeBench uses problems from platforms including Codeforces, LeetCode, and AtCoder[5].
- A separate benchmark called CodeElo uses the Codeforces platform to evaluate LLMs, providing Elo ratings[6][7][8].
- The CodeElo benchmark tested models like Qwen/Qwen2.5-Coder-7B-Instruct, but the specific Elo rating achieved by this model is not listed in the provided snippets[8]. One result mentions QwQ-32B-Preview (an o1-like reasoning model, potentially related to Qwen) achieved an Elo rating of 1261 on CodeForces[6][7].
- Some sources note that LLM performance on Codeforces can drop after their training cut-off date, suggesting potential benchmark contamination issues[9].
- No direct score for Qwen2.5 Coder 3B or 7B on a "Codeforces benchmark" (other than its use within LiveCodeBench or CodeElo) was found.
LiveBench:
- LiveBench is presented as a distinct benchmark designed to be resistant to test set contamination, covering categories like math, coding, reasoning, language, instruction following, and data analysis[10][11][12]. It uses frequently updated questions from recent sources[10][12][13].
- The LiveBench leaderboard is mentioned, but the snippets do not list specific scores for any Qwen2.5 Coder models (3B or 7B) on it[10]. The leaderboard snippet shows top scores from OpenAI models[10].
- One source mentions a LiveBench score for Qwen2.5-7B-Instruct in a comparison graph, but the specific value isn't readable in the text snippet[1]. Another source evaluating the general Qwen2.5-72B-Instruct model also shows a LiveBench score in a graph, again without a specific number in the text[1].
In Summary:
- LiveCodeBench: Qwen2.5-7B-Instruct scored 63.7% pass@1[1]. Qwen2.5-Coder-32B-Instruct is noted as achieving top performance among open-source models at the time of its release, competitive with GPT-4o[2][3].
- Codeforces: Used as a source for problems in LiveCodeBench[5] and the basis for the CodeElo benchmark[6][7][8]. No specific Elo or score for Qwen2.5 Coder 3B/7B is readily available from the snippets.
- LiveBench: This is a separate, broader benchmark[10][11][12]. Scores for Qwen2.5 Coder models are not listed in the text of the provided snippets, though some graphs may contain this information[1].
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.