Skip to content
The Chat Archive
Gemini (model unspecified)· Google ShareLM (browser-extension contributions)

Do you recognize any problem in the following power analysis simulation? We have a…

  1. User

    Do you recognize any problem in the following power analysis simulation? We have a dataset of user queries. For each query qq, we collect responses rir_i from nn distinct Large Language Models (LLMs), where i∈[n]i \in [n]. Our goal is to rank these models. A direct pairwise comparison of all models would require O(n2)O(n^2) comparisons, which is computationally expensive. To mitigate this, we employ an "anchor model" approach. We select one model to serve as the anchor and compare all other models against it. For a given query, the comparison cc between a target model and the anchor results in one of five outcomes:  slight/strong preference for the model, slight/strong preference for the anchor, or a tie. We map these outcomes to a numerical scale S∈{2,1,0,−1,−2}S \in \{2, 1, 0, -1, -2\} respectively. To compare two distinct models MaM_a and MbM_b, we calculate the difference between their scores against the anchor. Let SaS_a and SbS_b be the scores of MaM_a and MbM_b relative to the anchor on a specific query. The relative performance score XX is defined as:X=Sa−SbX = S_a - S_b Consequently, the domain of XX is {−4,−3,−2,−1,0,1,2,3,4}\{-4, -3, -2, -1, 0, 1, 2, 3, 4\}. We aim to determine the sample size NN (number of queries) required to achieve a statistically significant result using a Wilcoxon Signed-Rank Test between the two models, ensuring sufficient statistical power. \section*{Power Analysis} We assume access to a set of KK empirical distributions representing score differences, denoted as D={D(1),D(2),…,D(K)}\mathcal{D} = \{D^{(1)}, D^{(2)}, \dots, D^{(K)}\}, where each D(k)D^{(k)} is a distribution over the domain {−4,…,4}\{-4, \dots, 4\}.The following algorithm estimates the statistical power for a given sample size NN. Inputs: \begin{enumerate}     \item NN: Sample size (number of queries).     \item α\alpha: Significance level (e.g., 0.05).     \item pp: Minimum probability threshold to consider a distribution "positive" (ensuring we only test power on distributions where Ma>MbM_a > M_b).     \item MM: Number of Monte Carlo iterations. \end{enumerate} \subsection*{} % \begin{algorithmic} % \caption{Estimate Power for Sample Size NN} \begin{algorithmic}[1] \Function{EstimatePower}{N,α,p,M,DN, \alpha, p, M, \mathcal{D}}\State counter←0counter \gets 0 \State rejects←0rejects \gets 0 \State K←∣D∣K \gets |\mathcal{D}| \Comment{Number of available distributions}\For{m←1m \gets 1 to MM}     \State Sample an index k∼Uniform{1,…,K}k \sim \text{Uniform}\{1, \dots, K\}     \State Let D←D(k)D \gets D^{(k)}          \     \If{Px∼D(X>0)>pP_{x \sim D}(X > 0) > p} \Comment{Only consider TP cases}         \State counter←counter+1counter \gets counter + 1         \State Generate sample set S={x1,…,xN}S = \{x_1, \dots, x_N\} where xj∼Dx_j \sim D                  \State pval←WilcoxonSignedRankTest(S)pval \gets \text{WilcoxonSignedRankTest}(S)                  \If{pval<αpval < \alpha}             \State rejects←rejects+1rejects \gets rejects + 1         \EndIf     \EndIf \EndFor \If{counter=0counter = 0}     \Return 00 \Else     \Return rejectscounter\frac{rejects}{counter} \EndIf \EndFunction \end{algorithmic} \paragraph{} If the algorithm outputs a value (1−β1-\beta) that is larger than 0.800.80, then NN is large enough.

  2. Assistant

    There are three critical flaws in this simulation design that will lead to incorrect estimates of statistical power.The most dangerous issue is that your simulation calculates the "average power across a database of historical distributions" rather than the power to detect a specific Minimum Detectable Effect (MDE). This will likely lead to an underpowered study for subtle differences between models.Here is a breakdown of the problems and how to fix them.1. The "Positive Distribution" Condition is FlawedThe Problem: The condition used to select "positive" distributions is mathematically insufficient to determine if Model A is actually better than Model B. You use the filter: Px∼D​(X>0)>p This only checks the frequency of positive outcomes, not the central tendency (median or mean).Example: Imagine a distribution D where:P(X=4)=0.3 (30% of the time, A is much better)P(X=−4)=0.7 (70% of the time, B is much better)If your threshold p=0.2, this distribution passes your filter (0.3>0.2). However, the median is likely negative, and Model B is clearly the superior model. Your simulation will then run a statistical test. If the test correctly identifies that B>A (rejecting the null), your code counts this as a "Success" (rejects + 1) for the power of detecting A>B.The Consequence: You are counting "correct rejections in the wrong direction" (Type III errors) as statistical power.2. Failure to Check Test DirectionalityThe Problem: The standard WilcoxonSignedRankTest returns a p-value for a two-sided test (usually H0​:median=0). Your code checks:Pythonif pval < alpha: rejects = rejects + 1 This counts any significant difference as power. In a power analysis for ranking, you only care if the test allows you to conclude Model A > Model B.If the sample leads to a statistically significant conclusion that Model B > Model A, your code currently treats this as a successful detection of Model A's superiority.3. Aggregating Power over Disparate DistributionsThe Problem: You average the power over all K distributions in your historical set. Estimated Power=∑Valid Iterations∑Rejections​Power is not a single number derived from a "bag of distributions." Power is a function of Effect Size.If your set D contains mostly "easy" comparisons (e.g., GPT-4 vs. a 1-parameter model), your calculated power will be nearly 100%.If you then apply this N to a "hard" comparison (e.g., two very similar models), your actual power might be 10%.Averaging them gives you a meaningless number (e.g., 55%) that ensures sufficiency for neither. You must solve for N based on the Minimum Detectable Effect (MDE) you wish to support.Recommended SolutionTo fix this, you should bin your historical distributions by their Effect Size (e.g., the Common Language Effect Size or standardized median difference) and calculate power specifically for the "worst-case" effect size you want to be able to detect.Here is the corrected algorithm:Revised InputsInstead of a raw probability threshold p, define a Minimum Detectable Effect (MDE). For the Wilcoxon test, the estimand is often the pseudomedian, but in this discrete context, let's look at the simple mean or median difference required.Corrected Algorithm LogicPythonFunction EstimatePower(N, alpha, min_effect_size, M, D): counter = 0 rejects = 0

    # 1. Pre-filter distributions that represent the MDE we care about
    # We define Effect Size here as the mean of the distribution
    valid_distributions = [d for d in D if Mean(d) >= min_effect_size]
    
    if len(valid_distributions) == 0:
        Error("No historical distributions match this effect size.")
    
    For m = 1 to M:
        # Sample from the filtered list of distributions
        D_target = RandomChoice(valid_distributions)
        
        # Generate Sample
        S = {x_1, ..., x_N} sampled from D_target
        
        # Run Test (Two-sided)
        # statistic is usually sum of signed ranks
        statistic, pval = WilcoxonSignedRankTest(S, alternative='two-sided')
        
        # 2. Check Significance AND Directionality
        # For Wilcoxon, we must ensure the sum of ranks favors positive values
        # (Implementation dependent, but generally check if mean(S) > 0)
        is_significant = (pval < alpha)
        is_correct_direction = (Median(S) > 0) 
        
        if is_significant and is_correct_direction:
            rejects = rejects + 1
            
        counter = counter + 1
    
    Return rejects / counter
    

    Additional Note on the "Anchor" MetricYou are using a derived metric X=Sa​−Sb​. This metric suffers from saturation. If Sa​=2 (Strong Preference for A vs Anchor) and Sb​=2 (Strong Preference for B vs Anchor), then X=0. The test treats these models as identical. In reality, A might still be better than B.Impact: This reduces the variance of X and artificially lowers the apparent power.Fix: Ensure your anchor is strong enough that it rarely loses strongly to both models, or accept that this simulation only measures the power to rank models relative to the anchor's discriminative ability, not "ground truth."

  3. User

    I do want to allow both model_A > mode_B and model_B > model_A

  4. Assistant

    If you want to detect significant differences in either direction (i.e., you want to know if Model A is better or if Model B is better), you are performing a Two-Sided Hypothesis Test.This simplifies the rejection logic (you don't need to check direction), but it makes your input filter (how you select distributions to test) even more critical.Here is the problem with the current code in a two-sided context, followed by the corrected algorithm.The New Problem: Contaminating Power with "Null" CasesTo estimate Power (1−β), you must only simulate scenarios where a true difference actually exists.If your dataset D contains pairs of models that are effectively equal (ties), and you include them in your simulation, two things happen:If the test rejects: You have committed a Type I error (False Positive).If the test does not reject: You have correctly identified the null.However, your code counts "failures to reject" as "low power."If you include a "tied" distribution in your simulation loop, the test should fail to reject 95% of the time (assuming α=0.05).Your algorithm will incorrectly interpret this as "5% Power," dragging your average power down drastically.You must filter your historical distributions to only include those that represent the Minimum Detectable Effect (MDE) you care about, regardless of direction.The Corrected Algorithm (Two-Sided)To allow for both A>B and B>A, we change the filter from a probability threshold (p) to an Effect Size threshold (δ).Getty ImagesExploreThe image above illustrates Effect Size. To measure power, we must ensure the distance between the means (the gap between curves) is large enough to be "real." We cannot test power on curves that overlap perfectly.Algorithm InputsN: Sample size.α: Significance level (0.05).δ: Minimum Absolute Mean Difference (The smallest score difference you consider "practically significant").Example: If δ=0.1, we only test power on model pairs whose average score difference is at least 0.1 or -0.1.Revised CodePythonFunction EstimatePower(N, alpha, delta, M, D_list): counter = 0 rejects = 0

    # 1. Filter Logic: TWO-SIDED
    # We only care about distributions where a meaningful difference exists.
    # It doesn't matter if Mean is positive (A>B) or negative (B>A).
    # We filter out "Ties" where Mean is approx 0.
    
    valid_distributions = []
    For each D in D_list:
        true_mean = CalculateMean(D) # Expected value of the distribution
        if Abs(true_mean) >= delta:
            valid_distributions.append(D)
            
    If valid_distributions is empty:
        Return Error("No historical data matches this effect size.")
    
    # 2. Simulation Loop
    For m = 1 to M:
        # Pick a scenario where a difference is KNOWN to exist
        D_target = RandomChoice(valid_distributions)
        
        # Generate N samples
        S = {x_1, ..., x_N} sampled from D_target
        
        # Run Wilcoxon (Two-Sided)
        # We don't need to check direction manually. 
        # If p < alpha, we successfully detected the difference.
        pval = WilcoxonSignedRankTest(S, mode='two-sided')
        
        If pval < alpha:
            rejects = rejects + 1
            
        counter = counter + 1
    
    Return rejects / counter
    

    Why this fixes your problemSymmetry: By using Abs(true_mean) >= delta, you automatically include historical cases where Model A crushed Model B, AND cases where Model B crushed Model A.Purity of Signal: You exclude cases where the models were tied. This prevents "correct non-rejections" from falsely lowering your power estimate.Realism: This tells you: "If there is a real difference of at least magnitude δ, what is the probability my sample size N will find it?"A Note on the "Effect Size" for your specific metricSince your domain is discrete {−4,…,4} and derived from an anchor, "Mean Difference" is the most intuitive metric for δ.δ=0.0: Invalid for power analysis (this is for calculating False Positive Rate).δ=0.1: Represents a very subtle preference. Requires large N.δ=0.5: Represents a noticeable preference. Requires smaller N.You should likely run this function for a few different values of δ (e.g., "Hard", "Medium", and "Easy" detection tasks) to give stakeholders a range of required sample sizes.

  5. User

    can you refer to the Minimum Detectable Effect (MDE) too?

  6. Assistant

    Yes, incorporating the Minimum Detectable Effect (MDE) is actually the best way to standardize your "distribution filtering" step.Instead of guessing which historical distributions are "good enough" to test, you explicitly define an MDE (e.g., "I want to detect any model improvement greater than 0.1 points"). You then only simulate power using historical cases that match or exceed that difference.Here is how to integrate MDE directly into your power analysis logic.1. Defining MDE for Your MetricSince your metric X is a discrete score difference {−4,…,4}, the most intuitive MDE is the Mean Score Difference.MDE = 0.05: We want to detect very tiny improvements (requires massive N).MDE = 0.20: We want to detect noticeable improvements (requires moderate N).MDE = 0.50: We only care if one model is drastically better (requires small N).2. Revised Algorithm with MDEThis version of the algorithm answers the question: "If I have a sample size of N, what is the probability I will successfully detect a real difference of size MDE?"Pythonimport numpy as np from scipy.stats import wilcoxon

    def estimate_power_with_mde(N, alpha, mde, M, historical_distributions): """ N: Sample size (number of queries) alpha: Significance level (0.05) mde: Minimum Detectable Effect (e.g., 0.1 mean score difference) M: Monte Carlo iterations historical_distributions: List of arrays, each containing historical score diffs """ rejects = 0 valid_iterations = 0

    # 1. PRE-FILTER: Only test on historical cases that represent this MDE or larger
    # We take absolute value because we care about detecting difference in EITHER direction
    valid_distributions = [
        dist for dist in historical_distributions 
        if abs(np.mean(dist)) >= mde
    ]
    
    if not valid_distributions:
        return 0.0  # Or raise error: "No historical data found with this effect size"
    
    for _ in range(M):
        # 2. SAMPLING: Pick a distribution that represents a "Real Effect"
        dist = valid_distributions[np.random.randint(len(valid_distributions))]
        
        # Bootstrap a sample of size N from this distribution
        # "replace=True" simulates drawing new queries from this same underlying performance profile
        sample_diffs = np.random.choice(dist, size=N, replace=True)
        
        # 3. TEST: Wilcoxon Signed-Rank Test
        # We use 'pratt' or 'zsplit' to handle zeros (ties) if strictly necessary, 
        # but standard 'wilcoxon' usually discards zeros by default.
        # mode='two-sided' checks for difference in ANY direction.
        try:
            stat, p_value = wilcoxon(sample_diffs, alternative='two-sided')
            
            # Check significance
            if p_value < alpha:
                rejects += 1
                
        except ValueError:
            # Handles edge case where all differences are 0 (test fails)
            pass
            
        valid_iterations += 1
    
    return rejects / valid_iterations if valid_iterations > 0 else 0
    

    3. The "Reverse" Question: Finding MDE for a fixed NOften, you cannot change N (it's fixed by budget). In that case, you want to know: "With my 500 queries, what is the smallest difference I can reliably detect?"You can find this by running the simulation above for different MDE values and finding where Power crosses 80%.MDE (Mean Diff)Estimated Power (N=500)Conclusion0.0523%Underpowered (You will miss real diffs)0.1065%Risky0.1582%Acceptable MDE0.2095%OverpoweredExport to SheetsStakeholder Summary: "With 500 queries, we are blind to differences smaller than 0.15 points. If Model A is better than Model B by 0.1, we will likely claim they are tied."