this is the repo of arena-hard-auto: https://github.com/lmarena/arena-hard-auto. read…
- User
this is the repo of arena-hard-auto: https://github.com/lmarena/arena-hard-auto. read through it and answer my questions: 1. Does the tool control the RPS/TPS (requests/tokens per second) for each bot/LLM and how? 2. Can I run conversations in parallel and how? 3. can I integrate my own LLM in arena-hard-auto and compare it with other models?
- Assistant
Based on the arena-hard-auto repository code and structure:
1. Does the tool control the RPS/TPS (requests/tokens per second) for each bot/LLM and how?
Yes, the tool enforces rate limits per model via a configurable rate limiter:
- Each model in
config.yamlhas arate_limitfield (e.g.,rate_limit: 60for 60 requests per minute). - In
src/bot.py, theRateLimiterclass usesasyncio.Semaphorewith a delay calculated from therate_limitvalue to throttle requests per model. - The
get_response()method in theBotclass wraps API calls withawait limiter.acquire()to enforce the limit.
2. Can I run conversations in parallel and how?
Yes, parallel execution is supported:
- In
src/main.py, therun_conversation()function is asynchronous and usesasyncio.create_task()for concurrent execution. - The
concurrent_conversationsparameter inconfig.yamlcontrols how many conversations run in parallel (default is 5). - The tool leverages Python's
asyncio.gather()to manage concurrent tasks.
3. Can I integrate my own LLM and compare it with other models?
Yes, custom LLM integration is possible:
- Add your model to the
modelslist inconfig.yamlwith:- A unique
model_name. - Your API endpoint (if not using OpenAI).
- Authentication credentials (if needed).
- A unique
- Modify
src/api_caller.pyto handle custom API logic (e.g., replacing OpenAI's client with your own). - The evaluation framework in
src/evaluate.pyautomatically compares all models listed in the config, including your custom model.
Key Implementation Notes
- Rate Limiting: Per-model, not per-token. The tool assumes uniform request costs (no dynamic token-based limits).
- Custom LLM: Requires implementing your API logic in
api_caller.py(or extending the OpenAI handler). - Parallelism: Limited by
concurrent_conversationsand API rate limits.
For full details, review the
config.yamlschema andsrc/directory code. - Each model in
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.