Skip to content
TheChatArchiveThe Chat Archive
alpaca-13b· StanfordHugging Face Arena Human Preference 55K

How should I calculate the sample size required for an experiment where I compare two…

From a Chatbot Arena battleThe same person sent this prompt to two anonymous models at once. Compare it with Databricksdolly-v2-12b’s answer.
  1. User

    How should I calculate the sample size required for an experiment where I compare two models on the same set of data?

  2. Assistant

    The formula for calculating the sample size required for an experiment where you compare two models on the same set of data is n = m+p, where m is the number of successes in the smaller sample, and p is the probability of success in the smaller sample.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.