Skip to content
TheChatArchiveThe Chat Archive
Metallama-13b· MetaHugging Face Arena Human Preference 55K

How to evaluate large langage models?

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Stability AIstablelm-tuned-alpha-7b’s answer.
  1. User

    How to evaluate large langage models?

  2. Meta Assistant

    It's not hard to evaluate a language model, but it is still a rather complex task. Here are the most important things to keep in mind when evaluating a large language model:

    1. Make sure you have the correct version of the model in use. This is an easy mistake to make, and can lead to bad results.
    2. Make sure the model is correctly pretrained. This is also an easy mistake to make, and can lead to bad results.
    3. Make sure the model is correctly fine-tuned. This is also an easy mistake to make, and can lead to bad results.
    4. Make sure the model is correctly evaluated on the right task. This is also an easy mistake to make, and can lead to bad results.
    5. Make sure the model is correctly used. This is also an easy mistake to make, and can lead to bad results.
    6. Make sure the model is correctly evaluated using the right metrics. This is also an easy mistake to make, and can lead to bad results.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.