Skip to content
TheChatArchiveThe Chat Archive
gpt-4o-search-preview· OpenAIHugging Face LMArena Search Arena 24K

Find me an up-to-date roleplay benchmark that compares the capabilities of lots of LLM…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Perplexitysonar-reasoning’s answer.
  1. User

    Find me an up-to-date roleplay benchmark that compares the capabilities of lots of LLM models

  2. Assistant

    As of March 2025, RPBench-Auto is a prominent benchmark designed to evaluate the role-playing capabilities of large language models (LLMs). Introduced by Boson AI in August 2024, RPBench-Auto offers an automated evaluation pipeline that assesses LLMs in both character-based and scene-based role-playing scenarios[1].

    Key Features of RPBench-Auto:

    • Character-Based Role-Play: This setting involves interactions with 80 unique personae created by the online community, simulating platforms like character.ai.

    • Scene-Based Role-Play: This setting includes contextual elements such as plot recaps, character profiles, scene objectives, and progression criteria, allowing users to influence plot progression through dialogue.

    • Automated Evaluation: Utilizing models like GPT-4o as the baseline and GPT-4-Turbo as the judger, RPBench-Auto conducts multi-turn conversations to compare responses and assess performance.

    For the most current leaderboard and detailed information, you can visit the RPBench Leaderboard[1].

    Additionally, the LLM Leaderboard 2025 provides comprehensive performance benchmarks for various LLMs, including metrics like tokens per second and cost per token[2].

    For a detailed comparison of 25 state-of-the-art LLMs, including QwQ, through 59 MMLU-Pro CS benchmark runs, you can refer to the Hugging Face blog post[3].

    These resources should assist you in comparing the role-playing capabilities of various LLMs.

  3. User

    Find more.

  4. Assistant

    Certainly, here are additional benchmarks that evaluate the role-playing capabilities of various large language models (LLMs):

    1. PingPong Benchmark

    The PingPong Benchmark is designed to assess LLMs in role-playing scenarios. It evaluates models based on several criteria, including:

    • Final Score: Overall performance rating.
    • Length Normalized Score: Adjusts for response length.
    • Refusal Ratio: Frequency of the model refusing to engage.
    • Stay in Character Score: Ability to maintain character consistency.
    • Language Fluency Score: Proficiency in language use.
    • Entertainment Score: Engagement and entertainment value.

    For instance, the benchmark lists models like Claude 3 Opus with a final score of 8.24 and WizardLM 2 8x22B with a score of 8.03[1].

    2. Ayumi's LLM Role Play & ERP Ranking (Version 3)

    Ayumi's ranking focuses on evaluating LLMs for role-playing and erotic role-playing (ERP) scenarios. It utilizes automated benchmarks to rate models based on:

    • ALC-IQ3: Measures understanding of character cards, with a maximum score of 100.
    • ERP3 Score: Average ratio of lewd words to total words in a response.
    • Variety Score: Diversity of lewd words used across responses.

    For example, Neural Chat V3 16k 7B achieved an ALC-IQ3 score of 89.33 and an ERP3 Score of 30.92[2].

    3. LLM Performance Benchmarks – November 2024 Update

    This benchmark provides a comprehensive evaluation of various LLMs, including their role-playing capabilities. It assesses models based on multiple metrics, such as:

    • Performance Scores: Overall effectiveness in various tasks.
    • Cost Efficiency: Cost per token.
    • Response Speed: Tokens per second.

    For instance, Claude 3 Haiku scored 64 in performance with a cost of €0.08 per token and a response speed of 0.52 responses per second[3].

    These benchmarks offer valuable insights into the role-playing capabilities of various LLMs, assisting in selecting the most suitable model for specific applications.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.