How to evaluate large langage models?
- User
How to evaluate large langage models?
Assistant
It's not hard to evaluate a language model, but it is still a rather complex task. Here are the most important things to keep in mind when evaluating a large language model:
- Make sure you have the correct version of the model in use. This is an easy mistake to make, and can lead to bad results.
- Make sure the model is correctly pretrained. This is also an easy mistake to make, and can lead to bad results.
- Make sure the model is correctly fine-tuned. This is also an easy mistake to make, and can lead to bad results.
- Make sure the model is correctly evaluated on the right task. This is also an easy mistake to make, and can lead to bad results.
- Make sure the model is correctly used. This is also an easy mistake to make, and can lead to bad results.
- Make sure the model is correctly evaluated using the right metrics. This is also an easy mistake to make, and can lead to bad results.
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.