Skip to content
VidiMaster it, module by module
Module 9/Estimation & the Law of Large Numbers

Confidence intervals & the law of large numbers

The normal approximation lets us say how close the observed frequency p^=Sn/n\hat p=S_n/n is to the true pp — the idea behind confidence intervals — while the law of large numbers guarantees p^→p\hat p\to p.

Before you start — give these a try

Attempting first primes your brain for the lesson — even if you miss. Nothing is graded or saved; it's just a warm-up.

For a 95%95\% interval with n=1000n=1000, the margin is ε=1.9621000\varepsilon=\dfrac{1.96}{2\sqrt{1000}}. Compute ε\varepsilon (to 3 d.p.).

With margin ε=0.02\varepsilon=0.02 and n=2500n=2500, evaluate 2εn2\varepsilon\sqrt{n} (the argument of Φ\Phi).

What you’ll be able to do

  • Use p^=Sn/n\hat p=S_n/n to estimate an unknown pp and bound the error via the normal approximation.
  • Apply P(∣p^−p∣<ε)≥2Φ(2εn)−1P(|\hat p-p|<\varepsilon)\ge 2\Phi(2\varepsilon\sqrt{n})-1 to find a sample size or a confidence interval.
  • State the law of large numbers and distinguish it from the gambler's fallacy.

In your course

· MATH2015 · Linear Algebra & Probability
§14.4 Applications of the normal approximation
  • Inequality (14.1)Lower bound for P(∣p^−p∣<ε)P(|\hat p-p|<\varepsilon)
    P(∣p^−p∣<ε)≥2Φ(2εn)−1P(|\hat p-p|<\varepsilon)\ge 2\Phi(2\varepsilon\sqrt{n})-1.
  • Theorem 14.3Law of large numbers for binomial random variables
    lim⁡n→∞P(∣Sn/n−p∣<ε)=1\lim_{n\to\infty}P(|S_n/n-p|<\varepsilon)=1.
  • Example 14.795% CI from 450/1000 successes
  • Example 14.6Sample size for a target accuracy
The notes' Example 14.6 contains an arithmetic slip (it divides by ε2\varepsilon^2 instead of 4ε24\varepsilon^2); the correct sample size for ε=0.05\varepsilon=0.05 at 99%99\% is n=666n=666, used here.
1

Estimating an unknown probability

To estimate an unknown pp, run nn independent trials, count successes SnS_n, and use the observed proportion p^=Sn/n\hat p=S_n/n. The larger nn, the better the estimate — but how far off might p^\hat p be?

2

A normal-approximation bound

Standardizing SnS_n and bounding p(1−p)≤14p(1-p)\le \tfrac14 gives, for margin ε>0\varepsilon>0,\n\nP(∣p^−p∣<ε) ≥ 2Φ(2εn)−1.(14.1)P(|\hat p-p|<\varepsilon)\ \ge\ 2\Phi\big(2\varepsilon\sqrt{n}\big)-1.\qquad(14.1)\n\nUse it two ways: solve for nn to hit a target confidence, or build an interval p^±ε\hat p\pm\varepsilon around an observed p^\hat p.

3

Interpreting a confidence interval

Once the data are in and p^\hat p is observed, p^\hat p and the true pp are both fixed numbers. We do not say ‘‘pp is in the interval with probability 0.950.95.’’ We say the interval is a 95%95\% confidence interval: intervals built this way contain the true pp in 95%95\% of samples.

4

The law of large numbers

The LLN says the observed frequency converges to the true probability: for any ε>0\varepsilon>0, P(∣p^−p∣<ε)→1P(|\hat p-p|<\varepsilon)\to 1 as n→∞n\to\infty. It justifies estimating probabilities by long-run frequencies — but says nothing about ‘‘evening out’’ in the short run.

Inequality (14.1) — error of the sample proportion

For Sn∼Bin(n,p)S_n\sim\mathrm{Bin}(n,p), p^=Sn/n\hat p=S_n/n, and ε>0\varepsilon>0,\n\nP(∣p^−p∣<ε) ≥ 2Φ(2εn)−1.P(|\hat p-p|<\varepsilon)\ \ge\ 2\Phi\big(2\varepsilon\sqrt{n}\big)-1.

Intuition. Worst case p=12p=\tfrac12 (largest variance) gives this uniform lower bound, independent of the unknown pp.
Theorem 14.3 — Law of large numbers (binomial)

Let 0<p<10<p<1 and Sn∼Bin(n,p)S_n\sim\mathrm{Bin}(n,p). Then for any ε>0\varepsilon>0,\n\nlim⁡n→∞P(∣Snn−p∣<ε)=1.\lim_{n\to\infty}P\big(|\tfrac{S_n}{n}-p|<\varepsilon\big)=1.

Intuition. The observed frequency settles onto the true probability as trials pile up.

Worked examples

Example 1

In n=1000n=1000 trials we observe 450450 successes. Give a 95%95\% confidence interval for pp.

  1. 1

    Set 2Φ(2εn)−1=0.952\Phi(2\varepsilon\sqrt{n})-1=0.95, so Φ(2εn)=0.975\Phi(2\varepsilon\sqrt{n})=0.975 and 2εn=1.962\varepsilon\sqrt{n}=1.96.

  2. 2

    ε=1.9621000≈0.031\varepsilon=\dfrac{1.96}{2\sqrt{1000}}\approx 0.031.

  3. 3

    p^=450/1000=0.45\hat p=450/1000=0.45, so the interval is 0.45±0.0310.45\pm 0.031.

Answer. (0.419, 0.481)(0.419,\ 0.481).
Example 2

How many trials give p^\hat p within ε=0.05\varepsilon=0.05 of pp with probability at least 95%95\%?