Skip to content
The Chat Archive
sonar-reasoning-pro-high LMArena Search Arena 24K

who invented llm as a judge

  1. User

    who invented llm as a judge

  2. Assistant

    The concept of "LLM-as-a-judge" emerged following the release of GPT-4, which was the first LLM powerful enough to reliably evaluate the outputs of other language models. While there isn't a single inventor attributed in the search results, the approach was first formally explored in the paper "Sparks of Artificial General Intelligence: Early experiments with GPT-4," published less than ten days after GPT-4's release[3].

    The methodology was later more thoroughly developed and analyzed in the paper "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," which introduced two evaluation benchmarks: MT-Bench (a dataset of 80 multi-turn questions) and LMSys Chatbot Arena (a platform for comparative model evaluation)[9][5].

    Before GPT-4's release, evaluating complex LLM outputs was difficult using traditional metrics, as these metrics couldn't effectively assess responses to open-ended tasks. The emergence of powerful models like GPT-4 enabled researchers to use these models themselves as evaluators, creating a more scalable alternative to expensive and time-consuming human evaluation[3][5].

    Since its introduction, LLM-as-a-judge has become one of the most widely-used evaluation techniques for assessing LLM performance and alignment with human preferences[3].

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.