Skip to content
The Chat Archive
o3-mini· OpenAI LMArena Human Preference 140K

I saw a very nice idea for measuring the length of a confidence interval. If it includes…

  1. User

    I saw a very nice idea for measuring the length of a confidence interval. If it includes the target parameter then we take the length as usual. When the when the true value is below the lower limit, then it uses the difference between the UPPER limit and the true value. There is a similar widening when the true value is above the upper limit. Can you point me to articles that have used this measure and discussed it?

  2. Assistant

    The measure you describe is very similar to what is known in the forecasting literature as the interval score (sometimes also called the “weighted interval score”). This score is a proper scoring rule for evaluating interval forecasts—it rewards narrow intervals when they capture the true value and penalizes intervals that miss the true value, with extra penalty depending on how far off the true value is.

    In more detail, if you have a central (1 – α) prediction interval with lower limit L and upper limit U, and an observed value y, the interval score is defined as

    [ S_\alpha(L, U; y) = (U - L) + \frac{2}{\alpha}(L - y),\mathbf{1}{y < L} + \frac{2}{\alpha}(y - U),\mathbf{1}{y > U},. ]

    That is, when the target (or observation) y falls inside [L, U] the score is simply the length of the interval, but when y falls outside, there’s an additional penalty proportional to the distance between y and the missed endpoint. (In some contexts the constant factor may differ, but the idea is the same.)

    There are several articles and books that discuss this kind of measure. Here are a few key references:

    1. Gneiting and Raftery (2007)

      • Reference: Gneiting, T., & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378.
      • Notes: This is one of the classic references on scoring rules. Section 2 of the paper introduces and motivates the interval score as a proper scoring rule for interval forecasts.
    2. Winkler’s Early Work

      • Reference: Winkler, R. L. (1981). Scoring Rules and the Evaluation of Probabilistic Forecasts.
      • Notes: Winkler’s work from the early 1980s laid much of the groundwork for evaluating probabilistic forecasts and discusses scoring rules that have a similar flavor. Although the focus might be broader, the ideas carry over to the kind of penalty for noncoverage you mention.
    3. Forecast Evaluation in Practice

      • In various applied fields (for example, weather forecasting and epidemic forecasting), the interval score has been used extensively. For instance, you might want to look at more recent articles such as:
        • Bracher, J., Ray, E. L., Gneiting, T., & Reich, N. G. (2021). Beyond Precaution: Forecast Evaluation for Predictive Uncertainty.
      • Notes: This and similar papers apply the interval score in contexts where it is important not just to have narrow intervals, but also to “punish” forecasts that miss the observed value.
    4. Books on Forecasting and Statistical Methods

      • There are also several textbooks and monographs on forecast evaluation and probabilistic forecasting that discuss proper scoring rules (including the interval score) in depth. For example, chapters in books on forecasting by Tim Brooks or in texts discussing probabilistic weather forecasting will typically include discussion of these ideas.

    These resources should help you both understand the rationale behind such a measure and see numerous applications in the literature. If you have access to a university library or online journal services, a search for “interval score” or “proper scoring rules for prediction intervals” will turn up many additional papers that discuss both theoretical properties and applications of this measure.

Responses were produced in different, uncontrolled contexts (system prompts, settings, prior turns). This is not a benchmark.