How to use LM arena.AI
- User
How to use LM arena.AI
- Assistant
Of course! Here is a comprehensive guide on how to use the Chatbot Arena (often found via
arena.ai, which directs tolmsys.org).1. What is Chatbot Arena?
Chatbot Arena is a research project and public platform created by LMSys.org (Large Model Systems Organization). Its primary purpose is to evaluate and rank Large Language Models (LLMs) in a real-world, unbiased way.
The core idea is a blind, randomized "taste test" for AI. You, the user, act as a judge in a battle between two anonymous AI models. Your votes help create a dynamic, crowdsourced leaderboard of the best-performing models.
2. The Core Concept: The Blind Battle
Understanding this is key to using the Arena effectively:
- Anonymous Models: You are given two chatbots, labeled "Model A" and "Model B." You do not know which AI model is which (e.g., you don't know if you're talking to GPT-4, Claude 3, Llama 3, or Gemini).
- Randomized Pairing: The two models are chosen randomly for each new battle.
- Single Prompt: You give both models the exact same prompt.
- Side-by-Side Comparison: You receive two different responses, one from each model.
- You are the Judge: You compare the two responses and vote for the one you think is better.
- The Reveal: Only after you vote, the platform reveals the true identities of Model A and Model B.
This process generates valuable data on human preferences, which is used to calculate an Elo rating for each model, similar to how chess players are ranked.
3. Step-by-Step Guide: How to Use the Arena
Here’s how to jump in and start contributing.
Step 1: Go to the Website
- Navigate to chat.lmsys.org. (Typing
arena.aiwill also redirect you here).
Step 2: Choose Your Arena Mode
On the homepage, you'll see several options. The main one for evaluation is the Arena.
-
Arena (Side-by-Side): This is the classic mode. You enter one prompt, get two answers, and vote. It's quick and perfect for one-shot questions. This is the best place to start.
-
Arena (Battle): This is a conversational mode. You can have a back-and-forth chat with both anonymous models. Once you feel you have enough information to judge, you can cast your vote. This is better for testing conversational flow, memory, and follow-up capabilities.
-
Direct Chat: This is not for voting. Here, you can select a specific, named model from a dropdown list to chat with directly. It's useful if you want to test a particular AI you've heard about.
Step 3: Enter Your Prompt
Click into the
Arena (Side-by-Side)mode. You'll see a single text box at the bottom. This is where you type your question or command.Tips for Good Prompts:
- Be Creative: Don't just ask "What is the capital of France?" Try tasks like:
- "Write a short, funny poem about a coffee-addicted squirrel."
- "Explain the concept of quantum entanglement to a 10-year-old."
- "Write a Python function that finds all prime numbers up to n."
- "Help me plan a 3-day itinerary for a trip to Tokyo on a budget."
- "Summarize this article for me [paste text]."
- Test Different Skills: Try prompts that test reasoning, creativity, coding, math, and factual recall.
Step 4: Compare the Responses
After you hit enter, Model A and Model B will generate their responses. They will appear side-by-side. Read both of them carefully.
Consider these criteria when judging:
- Helpfulness & Accuracy: Did it answer your question correctly?
- Clarity & Formatting: Was the answer easy to read? Did it use lists, bolding, or code blocks effectively?
- Completeness: Did it give a thorough answer or a shallow one?
- Creativity & Style: If you asked for something creative, which one was more engaging?
- Safety & Instruction Following: Did it follow all parts of your prompt? Did it refuse appropriately for unsafe requests?
Step 5: Cast Your Vote
Below the two responses, you will see the voting buttons:
[A] is better: Choose this if Model A's response was clearly superior.[B] is better: Choose this if Model B's response was clearly superior.Tie: Choose this if both responses were of roughly equal quality (either equally good or equally mediocre).Both are bad: Choose this if both models failed badly, gave nonsensical answers, or were completely unhelpful.
Step 6: See the Reveal
As soon as you vote, the names of the models will be revealed! This is the fun part where you learn which AIs you were testing. For example, it might show:
Model A: gpt-4oModel B: claude-3-opus-20240229Step 7: Repeat!
Simply click "New Round" to start over with a new prompt and a new random pair of models. Every vote you cast helps improve the quality of the data.
4. Exploring the Leaderboard
Your votes directly contribute to the Chatbot Arena Leaderboard.
- How to Read It: The leaderboard ranks models based on their Elo rating. A higher Elo means the model wins more "battles" against other models according to human voters.
- What it Shows: You'll see the model's rank, its name, its Elo score, and the total number of votes it has been a part of.
- Why It's Important: This leaderboard is considered one of the most credible and unbiased rankings of LLM performance because it's based on thousands of real human preferences, not just automated benchmarks.
5. Why Your Contribution Matters
- Reduces Bias: Automated tests can be "gamed." Human preference captures a more holistic view of a model's quality.
- Drives Research: The data collected is used by AI researchers around the world to understand the strengths and weaknesses of different models.
- Keeps Companies Honest: It provides an independent benchmark that can't be easily manipulated, pushing developers to create genuinely better models.
You are not just playing a game; you are actively participating in cutting-edge AI research every time you cast a vote.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.