how does lmarena work with private versus public releases
- User
how does lmarena work with private versus public releases
Assistant
LM Arena (Large Model Systems Arena), run by LMSys.org, is a platform for crowdsourced, blind, pairwise comparison of large language models (LLMs). Users are presented with two anonymous model outputs for the same prompt and vote for which one is better. This data is used to calculate Elo ratings and generate a leaderboard.
Here's how it generally works with public versus private/unreleased models:
1. Publicly Released Models:
- How they get there:
- Well-known models: The LMSys team proactively adds popular and newly released public models (e.g., GPT-4 versions, Claude versions, Llama series, Mixtral, Gemma, etc.). These are models whose existence is widely known, and they are typically accessible via public APIs or have open-source weights.
- Community suggestions: The community can also suggest models to be added.
- Identification: Once a model has enough votes and its Elo rating stabilizes, its identity is usually revealed on the leaderboard with its commonly known public name (e.g., "GPT-4-Turbo," "Claude 3 Opus," "Llama-2-70b-chat").
- Purpose: For the public, it provides a dynamic, human-preference-based ranking. For model developers, it's a way to see how their public model stacks up against others in real-world scenarios.
2. Private / Unreleased / Experimental Models:
- How they get there:
- Collaboration with developers: Model developers (companies or research labs) who have a new, unreleased, or private model can collaborate with the LMSys team to include their model in the Arena for evaluation.
- Submission: The developers would need to provide LMSys with access to their model. This could be through:
- An API endpoint (if the model is hosted by the developers).
- Model weights and inference code (if LMSys is to host it, which requires trust and infrastructure capacity from LMSys).
- Identification (Anonymity Options):
- Initial Anonymity (Standard): All models in the Arena are anonymous to the voters during the pairwise comparison ("Model A" vs. "Model B"). This is core to the methodology.
- Continued Anonymity on Leaderboard (Optional): For private/unreleased models, developers can request that their model's true identity not be revealed on the public leaderboard, even after it has accrued enough votes. It might appear as "experimental_model_XYZ," "model_alpha_v2," or some other placeholder.
- Timed Reveal: Developers might agree to have the model's identity revealed at a later date, perhaps coinciding with a public launch or announcement.
- Internal Tracking: Even if not publicly named, the developers would know which anonymous entry corresponds to their model, allowing them to track its performance.
- Purpose for Private Models:
- Stealth Benchmarking: Developers can get unbiased human feedback on how their private model performs against state-of-the-art public models without prematurely revealing their model or its capabilities.
- Pre-release Testing: Gauge user preference and identify strengths/weaknesses before a public launch.
- Iterative Development: Use the feedback to improve the model.
- Academic Research: Researchers can evaluate novel architectures or training methods.
Key Mechanisms & Considerations:
- Anonymity for Voters: Voters never know which specific models they are comparing during the voting process. This helps reduce bias.
- Elo Rating System: This system, borrowed from chess, updates model ratings based on the outcome of pairwise comparisons. It's effective for ranking entities in a competitive setting.
- Infrastructure: LMSys needs to manage the inference for all models being evaluated or have reliable API access. Adding many private models can be resource-intensive.
- Trust and Agreement: For private model submissions, there's an element of trust between the model developer and LMSys regarding model access, data privacy, and the terms of anonymity/reveal.
In summary, LM Arena is designed to evaluate any LLM. Public models are added by LMSys or community suggestion and are typically identified publicly on the leaderboard. Private or unreleased models can be added through collaboration with their developers, who can opt for varying degrees of anonymity on the public leaderboard, allowing them to leverage the platform for confidential pre-release testing and benchmarking.
- How they get there:
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.