Skip to content
TheChatArchiveThe Chat Archive
Metallama-4-maverick-17b-128e-instruct· MetaHugging Face LMArena Human Preference 140K

Can you tell me about ROUGE metric?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Coherecommand-a-03-2025’s answer.
  1. User

    Can you tell me about ROUGE metric?

  2. Meta Assistant

    ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of metrics used to evaluate the quality of text summarization and machine translation systems. It was introduced by Chin-Yew Lin in 2004.

    ROUGE measures the overlap between a system-generated summary (or translation) and a set of human-generated reference summaries (or translations). The metric is based on the idea that a good summary should contain the same information as the reference summaries.

    There are several variants of ROUGE, including:

    1. ROUGE-N: measures the overlap of n-grams between the system-generated summary and the reference summaries. For example, ROUGE-1 measures the overlap of unigrams (individual words), while ROUGE-2 measures the overlap of bigrams (pairs of words).
    2. ROUGE-L: measures the overlap of the longest common subsequence (LCS) between the system-generated summary and the reference summaries. The LCS is a sequence of words that appears in both the system-generated summary and the reference summaries, in the same order, but not necessarily consecutively.
    3. ROUGE-W: a weighted version of ROUGE-L, which gives more importance to consecutive matches.
    4. ROUGE-S: measures the overlap of skip-bigrams (pairs of words that appear in the same order, but not necessarily consecutively) between the system-generated summary and the reference summaries.

    The ROUGE score is typically calculated as a recall score, which is the ratio of the number of overlapping n-grams (or LCS, etc.) to the total number of n-grams in the reference summaries.

    ROUGE has become a standard metric for evaluating text summarization systems, and is widely used in the natural language processing (NLP) community. Its advantages include:

    • Easy to compute: ROUGE is relatively simple to calculate, especially compared to other evaluation metrics that require more complex linguistic analysis.
    • Correlates well with human judgments: ROUGE has been shown to correlate well with human evaluations of summary quality.

    However, ROUGE also has some limitations:

    • Does not capture all aspects of summary quality: ROUGE focuses on measuring the overlap between the system-generated summary and the reference summaries, but does not capture other important aspects of summary quality, such as coherence, fluency, and readability.
    • Sensitive to reference summaries: ROUGE is sensitive to the quality and diversity of the reference summaries used for evaluation.

    Overall, ROUGE is a useful metric for evaluating text summarization systems, but it should be used in conjunction with other evaluation metrics to get a more comprehensive picture of summary quality.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.