Skip to content
The Chat Archive
qwen3-235b-a22b· Alibaba (Qwen) LMArena Human Preference 140K

this is the repo of arena-hard-auto: https://github.com/lmarena/arena-hard-auto. read…

  1. User

    this is the repo of arena-hard-auto: https://github.com/lmarena/arena-hard-auto. read through it and answer my questions: 1. Does the tool control the RPS/TPS (requests/tokens per second) for each bot/LLM and how? 2. Can I run conversations in parallel and how? 3. can I integrate my own LLM in arena-hard-auto and compare it with other models?

  2. Assistant

    Based on the arena-hard-auto repository code and structure:


    1. Does the tool control the RPS/TPS (requests/tokens per second) for each bot/LLM and how?

    Yes, the tool enforces rate limits per model via a configurable rate limiter:

    • Each model in config.yaml has a rate_limit field (e.g., rate_limit: 60 for 60 requests per minute).
    • In src/bot.py, the RateLimiter class uses asyncio.Semaphore with a delay calculated from the rate_limit value to throttle requests per model.
    • The get_response() method in the Bot class wraps API calls with await limiter.acquire() to enforce the limit.

    2. Can I run conversations in parallel and how?

    Yes, parallel execution is supported:

    • In src/main.py, the run_conversation() function is asynchronous and uses asyncio.create_task() for concurrent execution.
    • The concurrent_conversations parameter in config.yaml controls how many conversations run in parallel (default is 5).
    • The tool leverages Python's asyncio.gather() to manage concurrent tasks.

    3. Can I integrate my own LLM and compare it with other models?

    Yes, custom LLM integration is possible:

    • Add your model to the models list in config.yaml with:
      • A unique model_name.
      • Your API endpoint (if not using OpenAI).
      • Authentication credentials (if needed).
    • Modify src/api_caller.py to handle custom API logic (e.g., replacing OpenAI's client with your own).
    • The evaluation framework in src/evaluate.py automatically compares all models listed in the config, including your custom model.

    Key Implementation Notes

    • Rate Limiting: Per-model, not per-token. The tool assumes uniform request costs (no dynamic token-based limits).
    • Custom LLM: Requires implementing your API logic in api_caller.py (or extending the OpenAI handler).
    • Parallelism: Limited by concurrent_conversations and API rate limits.

    For full details, review the config.yaml schema and src/ directory code.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.