i have N models. I can compare two models and get relative accuracy of one of them (get…
- User
i have N models. I can compare two models and get relative accuracy of one of them (get value representing how much one model performing bettern than another) how can i find the best models while i can only compare two
- Assistant
In order to find the best model among the N models using pairwise comparisons, you could employ a tournament-style approach. Here's a step-by-step process to do so:
-
Pairwise Comparisons: Compare every model with every other model in a round-robin fashion. For each comparison, record which model is better and by how much.
-
Scoring System: Assign points to each model based on the outcome of the comparisons. For example, a model could earn 1 point for each comparison it wins, and no points for a loss.
-
Ranking: After all comparisons have been made, rank the models based on the points they've accumulated. The model with the most points is the best model according to your comparisons. In case of ties, you could compare the tied models directly again, or use the sum of the relative accuracy values they've accumulated as a tiebreaker.
Here's an illustrative example:
- Suppose you have 4 models (A, B, C, D).
- You compare them as follows: A vs B, A vs C, A vs D, B vs C, B vs D, C vs D.
- You record the outcomes like this:
- A beats B
- C beats A
- A beats D
- C beats B
- D beats B
- C beats D
- You assign points for each win:
- A: 2 points (wins against B and D)
- B: 0 points
- C: 3 points (wins against A, B, and D)
- D: 1 point (wins against B)
- According to the points, model C is the best model.
However, if you have a large number of models, this method can be quite inefficient, as the number of comparisons required would be N*(N-1)/2. For a large N, this may not be practical.
An alternative approach is a single-elimination tournament:
-
Create a Bracket: Pair up the models in a bracket format, similar to a sports tournament. If N is not a power of two, some models may need to receive a bye in the first round.
-
Tournament Rounds: Have each pair of models compete, and the winner of each pair moves on to the next round.
-
Repeat: Continue this process until only one model remains, which will be your best model.
This approach requires fewer comparisons (N-1 comparisons if
-
Other models, same or similar prompt
Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.