Find me an up-to-date roleplay benchmark that compares the capabilities of lots of LLM…
- User
Find me an up-to-date roleplay benchmark that compares the capabilities of lots of LLM models
- Assistant
As of March 2025, RPBench-Auto is a prominent benchmark designed to evaluate the role-playing capabilities of large language models (LLMs). Introduced by Boson AI in August 2024, RPBench-Auto offers an automated evaluation pipeline that assesses LLMs in both character-based and scene-based role-playing scenarios[1].
Key Features of RPBench-Auto:
-
Character-Based Role-Play: This setting involves interactions with 80 unique personae created by the online community, simulating platforms like character.ai.
-
Scene-Based Role-Play: This setting includes contextual elements such as plot recaps, character profiles, scene objectives, and progression criteria, allowing users to influence plot progression through dialogue.
-
Automated Evaluation: Utilizing models like GPT-4o as the baseline and GPT-4-Turbo as the judger, RPBench-Auto conducts multi-turn conversations to compare responses and assess performance.
For the most current leaderboard and detailed information, you can visit the RPBench Leaderboard[1].
Additionally, the LLM Leaderboard 2025 provides comprehensive performance benchmarks for various LLMs, including metrics like tokens per second and cost per token[2].
For a detailed comparison of 25 state-of-the-art LLMs, including QwQ, through 59 MMLU-Pro CS benchmark runs, you can refer to the Hugging Face blog post[3].
These resources should assist you in comparing the role-playing capabilities of various LLMs.
-
- User
Find more.
- Assistant
Certainly, here are additional benchmarks that evaluate the role-playing capabilities of various large language models (LLMs):
1. PingPong Benchmark
The PingPong Benchmark is designed to assess LLMs in role-playing scenarios. It evaluates models based on several criteria, including:
- Final Score: Overall performance rating.
- Length Normalized Score: Adjusts for response length.
- Refusal Ratio: Frequency of the model refusing to engage.
- Stay in Character Score: Ability to maintain character consistency.
- Language Fluency Score: Proficiency in language use.
- Entertainment Score: Engagement and entertainment value.
For instance, the benchmark lists models like Claude 3 Opus with a final score of 8.24 and WizardLM 2 8x22B with a score of 8.03[1].
2. Ayumi's LLM Role Play & ERP Ranking (Version 3)
Ayumi's ranking focuses on evaluating LLMs for role-playing and erotic role-playing (ERP) scenarios. It utilizes automated benchmarks to rate models based on:
- ALC-IQ3: Measures understanding of character cards, with a maximum score of 100.
- ERP3 Score: Average ratio of lewd words to total words in a response.
- Variety Score: Diversity of lewd words used across responses.
For example, Neural Chat V3 16k 7B achieved an ALC-IQ3 score of 89.33 and an ERP3 Score of 30.92[2].
3. LLM Performance Benchmarks – November 2024 Update
This benchmark provides a comprehensive evaluation of various LLMs, including their role-playing capabilities. It assesses models based on multiple metrics, such as:
- Performance Scores: Overall effectiveness in various tasks.
- Cost Efficiency: Cost per token.
- Response Speed: Tokens per second.
For instance, Claude 3 Haiku scored 64 in performance with a cost of €0.08 per token and a response speed of 0.52 responses per second[3].
These benchmarks offer valuable insights into the role-playing capabilities of various LLMs, assisting in selecting the most suitable model for specific applications.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.