why do we diveide by n-1 and not n?
- User
why do we diveide by n-1 and not n?
- Assistant
Great question! The reason we divide by n-1 instead of n when calculating the sample variance (or sample standard deviation) has to do with getting an unbiased estimate of the population variance.
Background:
- Suppose you have a population with some true variance (\sigma^2).
- When you take a sample of size (n) from this population, you want to estimate (\sigma^2).
- The sample variance is calculated as: [ s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2 ] where (\bar{x}) is the sample mean.
Why divide by (n-1) and not (n)?
-
Using the sample mean introduces bias: The variance is the average squared deviation from the true mean (which is usually unknown). But since we don't know the true population mean, we use the sample mean instead. This means our deviations ((x_i - \bar{x})) are measured from a statistic estimated from the data, not the true mean.
-
Degrees of freedom: When we estimate the mean from the sample, it "consumes" one degree of freedom, because the sample mean is computed from the data points themselves and constrains their sum. That leaves only (n-1) independent pieces of information to estimate the variance.
-
Effect on the calculation: If you divided by (n), the resulting variance estimate tends to underestimate the true population variance — it's biased downwards.
-
Dividing by (n-1) corrects the bias: When you divide by (n-1), you get an unbiased estimator of the population variance. In expectation (over many samples), this estimator equals the true variance.
Intuition:
You can think of the sample variance as a "corrected" average squared difference. Because the sample mean itself "moves" to fit the data, the deviations tend to look smaller than they really are relative to the true mean. Dividing by (n-1) compensates for that and prevents underestimation.
Summary:
- Population variance (known mean): divide by (n).
- Sample variance (unknown mean, using sample mean): divide by (n-1) to get an unbiased estimate.
If you want, I can provide a mathematical proof or a numerical example to illustrate this. Just let me know!
- User
yes, please
- Assistant
Sure! Let me walk you through a simple mathematical proof that dividing by ( n-1 ) instead of ( n ) gives an unbiased estimator of the population variance.
Setup:
- You have a population with true mean ( \mu ) and true variance ( \sigma^2 = \mathrm{Var}(X) = \mathbb{E}[(X - \mu)^2] ).
- You draw a random sample ( X_1, X_2, \dots, X_n ), i.i.d. from this population.
- Sample mean is: [ \bar{X} = \frac{1}{n} \sum_{i=1}^n X_i ]
- Sample variance estimator (biased): [ S_n^2 = \frac{1}{n} \sum_{i=1}^n (X_i - \bar{X})^2 ]
- Adjusted sample variance estimator: [ S_{n-1}^2 = \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar{X})^2 ]
Goal: Show that ( \mathbb{E}[S_n^2] \neq \sigma^2 ), but ( \mathbb{E}[S_{n-1}^2] = \sigma^2 ).
Step 1: Express ( \sum (X_i - \bar{X})^2 ) in terms of true mean ( \mu )
Using the bias-variance decomposition, we start from the identity:
[ \sum_{i=1}^n (X_i - \bar{X})^2 = \sum_{i=1}^n \left( (X_i - \mu) - (\bar{X} - \mu) \right)^2 ]
Expanding,
[ = \sum_{i=1}^n (X_i - \mu)^2 - 2(\bar{X} - \mu) \sum_{i=1}^n (X_i - \mu) + \sum_{i=1}^n (\bar{X} - \mu)^2 ]
Note that:
- (\sum_{i=1}^n (X_i - \mu) = n(\bar{X} - \mu)),
- (\sum_{i=1}^n (\bar{X} - \mu)^2 = n (\bar{X} - \mu)^2).
Substituting:
[ = \sum_{i=1}^n (X_i - \mu)^2 - 2n(\bar{X} - \mu)^2 + n(\bar{X} - \mu)^2 = \sum_{i=1}^n (X_i - \mu)^2 - n(\bar{X} - \mu)^2 ]
So: [ \sum_{i=1}^n (X_i - \bar{X})^2 = \sum_{i=1}^n (X_i - \mu)^2 - n(\bar{X} - \mu)^2 ]
Step 2: Take expectations on both sides
Taking expectation ( \mathbb{E} ) over all samples:
[ \mathbb{E} \left[ \sum_{i=1}^n (X_i - \bar{X})^2 \right] = \mathbb{E} \left[ \sum_{i=1}^n (X_i - \mu)^2 \right] - \mathbb{E} \left[ n(\bar{X} - \mu)^2 \right] ]
Since the (X_i) are i.i.d.,
[ \mathbb{E} \left[ \sum_{i=1}^n (X_i - \mu)^2 \right] = \sum_{i=1}^n \mathbb{E}[(X_i - \mu)^2] = n \sigma^2 ]
Also,
[ \mathbb{E} \left[ (\bar{X} - \mu)^2 \right] = \mathrm{Var}(\bar{X}) = \frac{\sigma^2}{n} ]
Therefore:
[ \mathbb{E} \left[ \sum_{i=1}^n (X_i - \bar{X})^2 \right] = n \sigma^2 - n \cdot \frac{\sigma^2}{n} = n \sigma^2 - \sigma^2 = (n-1) \sigma^2 ]
Step 3: Find expectation of ( S_n^2 ) and ( S_{n-1}^2 )
Recall:
[ S_n^2 = \frac{1}{n} \sum_{i=1}^n (X_i - \bar{X})^2 ]
Taking expectation:
[ \mathbb{E}[S_n^2] = \mathbb{E}\left[ \frac{1}{n} \sum_{i=1}^n (X_i - \bar{X})^2 \right] = \frac{1}{n} \mathbb{E} \left[ \sum_{i=1}^n (X_i - \bar{X})^2 \right] = \frac{1}{n} (n-1) \sigma^2 = \left(1 - \frac{1}{n}\right) \sigma^2 ]
So this shows:
[ \boxed{ \mathbb{E}[S_n^2] = \frac{n-1}{n} \sigma^2 < \sigma^2 } ]
That is, ( S_n^2 ) underestimates the true variance ((\sigma^2)) by a factor (\frac{n-1}{n}).
For:
[ S_{n-1}^2 = \frac{1}{n-1} \sum_{i=1}^n (X_i - \bar{X})^2 ]
we have:
[ \mathbb{E}[S_{n-1}^2] = \frac{1}{n-1} \mathbb{E} \left[ \sum_{i=1}^n (X_i - \bar{X})^2 \right] = \frac{1}{n-1} (n-1) \sigma^2 = \sigma^2 ]
Thus,
[ \boxed{ \mathbb{E}[S_{n-1}^2] = \sigma^2 } ]
That means ( S_{n-1}^2 ) is an unbiased estimator of the population variance.
Summary:
- Dividing by ( n ) yields a biased (systematically too low) estimate of the population variance.
- Dividing by ( n-1 ) corrects the bias, giving an unbiased estimate.
- The factor ( n-1 ) is called the degrees of freedom, reflecting the number of independent pieces of data after estimating the mean.
Optional: Numerical illustration
Suppose you have ( n=2 ) samples generated from a population with variance ( \sigma^2=4 ).
- For many samples, sample variance dividing by ( n=2 ) tends to be about (\frac{1}{2} \times (2-1) \times 4 = 2), i.e., biased low.
- Dividing by ( n-1 = 1 ) gives the correct expected variance ( 4 ).
If you want, I can also help with code to simulate this and see the bias in practice.
- User
Why \sum_{i=1}^n \mathbb{E}[(X_i - \mu)^2] = n \sigma^2 and \mathbb{E} \left[ (\bar{X} - \mu)^2 \right] = \mathrm{Var}(\bar{X}) = \frac{\sigma^2}{n}
?
- Assistant
Great question! Let's carefully explain these two steps:
1. Why is (\displaystyle \sum_{i=1}^n \mathbb{E}[(X_i - \mu)^2] = n \sigma^2)?
- Each (X_i) is an independent and identically distributed (i.i.d.) random variable from the population.
- The population variance is defined as: [ \sigma^2 = \mathrm{Var}(X_i) = \mathbb{E}[(X_i - \mu)^2] ]
- Because all (X_i) have the same variance (\sigma^2), [ \mathbb{E}[(X_i - \mu)^2] = \sigma^2 \quad \text{for each } i=1,\dots,n. ]
- When you sum the expectations: [ \sum_{i=1}^n \mathbb{E}[(X_i - \mu)^2] = \sum_{i=1}^n \sigma^2 = n \sigma^2. ]
Important: This uses the linearity of expectation, which means that: [ \mathbb{E}\left[\sum_{i=1}^n Y_i \right] = \sum_{i=1}^n \mathbb{E}[Y_i] ] for any random variables (Y_1, \dots, Y_n), regardless of whether they are independent. So we can split the expectation of the sum into a sum of expectations easily.
2. Why is (\displaystyle \mathbb{E} \left[ (\bar{X} - \mu)^2 \right] = \mathrm{Var}(\bar{X}) = \frac{\sigma^2}{n})?
- Recall the sample mean: [ \bar{X} = \frac{1}{n} \sum_{i=1}^n X_i ]
- Because the (X_i) are i.i.d. with mean (\mu) and variance (\sigma^2), the mean of (\bar{X}) is: [ \mathbb{E}[\bar{X}] = \frac{1}{n} \sum_{i=1}^n \mathbb{E}[X_i] = \frac{1}{n} \times n \mu = \mu ]
- The variance of the sample mean is: [ \mathrm{Var}(\bar{X}) = \mathrm{Var}\left(\frac{1}{n} \sum_{i=1}^n X_i \right). ]
- Because the (X_i) are independent, [ \mathrm{Var}(\bar{X}) = \frac{1}{n^2} \sum_{i=1}^n \mathrm{Var}(X_i) = \frac{1}{n^2} \times n \sigma^2 = \frac{\sigma^2}{n}. ]
- Recall that variance is defined as: [ \mathrm{Var}(\bar{X}) = \mathbb{E}\left[(\bar{X} - \mu)^2\right], ] so it follows that: [ \mathbb{E}\left[(\bar{X} - \mu)^2\right] = \frac{\sigma^2}{n}. ]
Summary:
- Since each (X_i) has variance (\sigma^2), sum of their variances is (n \sigma^2).
- Because the sample mean (\bar{X}) averages over (n) observations, its variance is reduced by a factor of (n), i.e., (\frac{\sigma^2}{n}).
Let me know if you'd like me to explain anything else!