Skip to content
The Chat Archive
gemini-2.5-flash· Google LMArena Human Preference 140K

I've been looking lately at confidence intervals. They're kind of a mess. I'm wondering…

  1. User

    I've been looking lately at confidence intervals. They're kind of a mess. I'm wondering about their history, how the early writers described them and what they wanted out of them. Are there key papers I should turn to?

  2. Assistant

    You're absolutely right to say confidence intervals (CIs) are "kind of a mess." Their history is intertwined with deep philosophical debates in statistics, primarily the frequentist vs. Bayesian divide, and the difficulty of communicating nuanced statistical concepts. Understanding their origins helps clarify why they are the way they are.

    Let's dive into their history, how early writers described them, what they wanted, and some key papers.


    The Pre-Confidence Interval Era: "Probable Error" and Standard Errors

    Before Neyman coined "confidence interval," statisticians used other measures of uncertainty:

    1. Probable Error (PE): Popular in the 19th and early 20th centuries, especially with astronomers and physicists. The probable error was defined as the value such that there was a 50% chance the true value lay within ±PE\pm PE of the observed estimate.

      • How it was described: Often seen as the range where the true value probably lies. It implicitly carried a "Bayesian" flavor, even without formal Bayesianism, because it gave a probability statement about the parameter itself.
      • What they wanted: A simple, intuitive measure of the "precision" of an estimate.
      • Limitations: It required assuming a normal distribution for the error, and its interpretation was often muddled, blurring the line between probabilities of data given a parameter and probabilities of a parameter given data.
    2. Standard Error (SE): Ronald Fisher, among others, championed the use of the standard error, particularly in the context of Maximum Likelihood Estimation (MLE). He showed that MLEs were asymptotically normal, and their variance could be estimated.

      • How it was described: A measure of the variability of the sampling distribution of an estimator. For example, "the estimate is 10 with a standard error of 2."
      • What they wanted: A more theoretically grounded measure of precision, tied to the sampling distribution of the estimator.
      • Limitations: While more rigorous than PE, a standard error alone doesn't directly provide an interval statement unless you multiply it by some factor (e.g., 2 for an approximate 95% interval if normally distributed), and that still leaves the interpretation open.

    The Birth of Confidence Intervals: Jerzy Neyman (The Architect)

    The concept of confidence intervals as we know them was primarily developed by Jerzy Neyman in the 1930s. His work was a deliberate effort to create an objective, frequentist approach to interval estimation, avoiding what he saw as the subjective pitfalls of prior probabilities in Bayesian inference (which was less formalized then but implicitly present in many interpretations) and the conceptual difficulties of Fisher's "fiducial inference."

    Jerzy Neyman's Vision and What He Wanted:

    Neyman's key insight was to shift the focus from the unknown parameter being a random variable to the interval itself being a random variable (because it depends on the random sample).

    1. Objectivity and Frequentist Interpretation: Neyman wanted a procedure that, when applied repeatedly to independent samples from the same population, would contain the true (fixed, but unknown) parameter value a specified proportion of the time (e.g., 95% of the time).

      • Crucial point: For Neyman, the confidence level (e.g., 95%) is a property of the procedure, not a probability that the true parameter lies within a specific, already calculated interval. Once you've calculated an interval, the true parameter is either in it or it isn't – there's no probability involved for that single, fixed interval.
      • How he described them: As "confidence limits" or "confidence intervals" that define a "confidence belt" or "confidence band." He emphasized the performance of the method over many hypothetical repetitions.
    2. Avoiding Priors: A major driver for Neyman (and his collaborator Egon Pearson, in the context of hypothesis testing) was to develop statistical methods that did not rely on subjective prior probabilities for the unknown parameters. Bayesian methods require specifying such priors, which were (and often still are) seen as arbitrary or non-objective by frequentists.

    3. General Applicability: Neyman sought a general method for constructing such intervals, applicable to various estimation problems, without needing to assume specific prior distributions or relying on Fisher's somewhat opaque fiducial argument.

    The Rival: R.A. Fisher and Fiducial Inference

    While Neyman was developing confidence intervals, Ronald A. Fisher was working on his concept of "fiducial inference." This is often a source of confusion because Fisher's fiducial intervals sometimes numerically coincided with Neyman's confidence intervals for simpler problems, but their underlying philosophy was very different.

    • Fisher's Fiducial Inference: Fisher aimed to provide a probability distribution for the parameter given the observed data, but without invoking a prior distribution. He considered the observed sample statistic as fixed and then reasoned about the parameter's distribution.
      • How he described them: As a "fiducial distribution" of the parameter, allowing probability statements like "the probability is 0.95 that the true parameter lies between X and Y."
      • What he wanted: A way to make probability statements about parameters based solely on the data, without priors.
      • Limitations: Fiducial inference proved to be highly controversial, difficult to generalize, and often led to inconsistencies. It didn't gain widespread acceptance and has largely fallen out of favor because it's not clear what "fiducial probability" means or how to interpret it consistently in a frequentist sense.

    The Key Distinction:

    • Neyman's Confidence Interval: The true parameter is fixed. The interval is random. The probability is about the procedure covering the true parameter in the long run.
    • Fisher's Fiducial Interval: The data is fixed. The parameter has a "fiducial distribution." The probability is about the parameter being in the interval (a statement similar to a Bayesian posterior, but without an explicit prior).

    Why They're "Messy" and Why They Persist

    1. The Interpretation Hurdle: The single biggest reason CIs are "messy" is the common misinterpretation. Most people intuitively want to say, "There's a 95% chance the true parameter is in this specific interval I just calculated." This is a Bayesian interpretation of a frequentist interval, and it's incorrect under Neyman's framework. The correct frequentist interpretation (the procedure's long-run coverage) is conceptually harder to grasp for a single experiment.
    2. Philosophical Debates: The frequentist vs. Bayesian debate is ongoing. Bayesian credible intervals do allow the statement "There's a 95% probability the true parameter lies in this interval," but they require a prior distribution. CIs avoid this, but at the cost of the more abstract interpretation.
    3. Computational Complexity (Historically): While straightforward for simple cases, constructing exact confidence intervals can be mathematically complex for more intricate models.
    4. Overlap with Hypothesis Testing: CIs are intimately related to hypothesis testing within the Neyman-Pearson framework. An estimated parameter and its CI often convey more information than just a p-value, showing the plausible range of values and indicating which values would (or wouldn't) be rejected in a hypothesis test.

    Despite these challenges, confidence intervals persist because they offer a standardized, objective (in the sense of not requiring subjective priors), and widely applicable method for quantifying uncertainty in a frequentist framework. They provide a range of plausible values for a parameter, which is often more informative than a single point estimate alone.


    Key Papers You Should Turn To:

    Reading the original papers can be challenging, as the notation and conceptual framing might differ from modern textbooks, but they are invaluable for understanding the original intent.

    1. The Cornerstone:

      • Neyman, J. (1937). Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability. Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences, 236(767), 333-380.
        • Why read it: This is the foundational paper where Neyman lays out his theory of confidence intervals in detail. He clearly defines the concept, distinguishes it from other approaches, and discusses its properties. Be prepared for a dense read, but it's essential for understanding the original frequentist vision.
    2. For Context on Fisher's Rival Approach:

      • Fisher, R. A. (1930). Inverse Probability. Proceedings of the Royal Society of London. Series A, Containing Papers of a Mathematical and Physical Character, 130(814), 22-54.
        • Why read it: While not directly about "fiducial intervals" by name in this paper, it introduces the concept of fiducial inference and Fisher's desire to reverse the probability statement without using Bayes' theorem. It's a key paper showing Fisher's attempt to draw inferences about parameters from observed data. It highlights the philosophical tension that Neyman was responding to.
    3. For a Modern Perspective and Historical Commentary:

      • Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., & Wagenmakers, E. J. (2016). The Fallacy of Placing Confidence in Confidence Intervals. Psychonomic Bulletin & Review, 23(1), 103-123.
        • Why read it: This is an excellent modern paper that dissects the common misinterpretations of confidence intervals and clearly explains Neyman's original intent. It's much more accessible than the historical papers and provides a good summary of why they're "messy" for many people. It also contrasts them with Bayesian credible intervals.
      • Lehmann, E. L. (1999). Fisher, Neyman, and the Creation of Classical Statistics. Statistical Science, 14(2), 177-182.
        • Why read it: A concise and insightful historical overview by a prominent statistician who specialized in hypothesis testing and estimation theory. It provides context on the relationship and intellectual rivalry between Fisher and Neyman.
      • Reid, N., & Cox, D. R. (2015). On the Relationship Between Confidence Intervals and Fiducial Intervals. Statistical Science, 30(2), 177-189.
        • Why read it: This paper specifically addresses the relationship and differences between CIs and fiducial intervals, offering a more contemporary and nuanced discussion of their respective strengths and weaknesses.

    General Recommendation: For a solid understanding of the mathematical and philosophical underpinnings, any good advanced textbook on statistical inference will cover confidence intervals in detail. Look for books by authors like Casella & Berger, or Wasserman, which typically lay out the frequentist theory clearly.

    Understanding confidence intervals requires embracing their frequentist nature – focusing on the long-run performance of the procedure rather than a probability statement about a single outcome. It's a subtle but crucial distinction that lies at the heart of their "messiness" and their power.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.