Skip to content
TheChatArchiveThe Chat Archive
Geminigemini-2.5-pro-preview-03-25· GoogleHugging Face LMArena Human Preference 140K

Who founded LMArena? What's the story?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Claudeclaude-3-5-sonnet-20241022’s answer.
  1. User

    Who founded LMArena? What's the story?

  2. Gemini Assistant

    Okay, let's clarify the founding of LMArena (often referred to as Chatbot Arena).

    Who Founded LMArena?

    LMArena wasn't founded by a single individual in the traditional sense of a startup founder. It was developed and is maintained by the Large Model Systems Organization (LMSys Org).

    LMSys Org is a collaborative research organization focused on large language models (LLMs) and related systems. It involves researchers and students primarily affiliated with:

    • University of California, Berkeley (UC Berkeley)
    • Carnegie Mellon University (CMU)
    • University of California, San Diego (UC San Diego)
    • Supported by other institutions and industry partners.

    So, the "founder" is best described as the LMSys Org research collective.

    What's the Story?

    The story behind LMArena stems from a significant challenge in the field of Artificial Intelligence, specifically with Large Language Models (LLMs):

    1. The Evaluation Problem: As LLMs (like GPT-4, Claude, Llama, etc.) became incredibly powerful and capable of human-like conversation and complex instruction following, traditional automated benchmarks (like accuracy on specific question sets) weren't fully capturing their real-world performance, nuances, helpfulness, or potential issues like bias or harmlessness. Evaluating subjective qualities like creativity, writing style, or conversational flow is very difficult automatically.

    2. The Need for Human Feedback: Researchers realized that human judgment is crucial for evaluating these advanced models. Humans are good at determining which of two responses is "better," more helpful, more engaging, or safer, even if it's hard to quantify why.

    3. The Solution - Crowdsourced Comparison: LMSys Org devised Chatbot Arena as an innovative solution. The core idea is:

      • Blind Battles: Pit two anonymous LLMs against each other in a side-by-side comparison.
      • User Interaction: A user provides a prompt, and both anonymous models generate a response.
      • Human Vote: The user reads both responses and votes for which one they think is better (or declare a tie, or flag issues).
      • Elo Rating System: By collecting thousands of these anonymous head-to-head comparisons and votes, LMSys uses an Elo rating system (similar to chess rankings) to rank the models based purely on human preference.
    4. Launch and Impact: Chatbot Arena was launched publicly around August 2023. It quickly gained popularity because:

      • It provided a dynamic, constantly updated leaderboard reflecting real-world human preferences, often revealing insights that static benchmarks missed.
      • It offered a fun, interactive way for anyone to contribute to LLM evaluation.
      • The anonymity prevented brand bias, focusing purely on the quality of the output.
      • The collected data became a valuable resource for the AI research community to understand model strengths and weaknesses and guide future development.

    In essence, the story of LMArena is about tackling the challenge of evaluating sophisticated AI by cleverly crowdsourcing human judgment through a gamified, blind comparison platform, driven by a collaborative academic research effort (LMSys Org).

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.