Expectation: The Mean of a Random Variable

The expectation (or mean) of a discrete random variable $X$ is $\mathbb{E}[X]=\sum_k k\,\mathbb{P}\{X=k\},$ the sum running over every possible value $k$ of $X$ (Definition 13.1). It is a weighted average of the outcomes, each value $k$ weighted by how likely it is, $\mathbb{P}\{X=k\}$: the more probability sits at a value, the harder it pulls the average toward itself. The expectation is also called the first moment and is written $\mu=\mathbb{E}[X]$. A fair die has $\mathbb{E}[X]=\tfrac16(1+2+3+4+5+6)=\tfrac{21}{6}=3.5$ (Example 13.1) — a decisive reminder that the mean need not be a value $X$ can take: no face shows $3.5$. A useful special case is the indicator $\mathbf{1}_A$, equal to $1$ when the event $A$ occurs and $0$ otherwise; since $\mathbb{P}\{\mathbf{1}_A=1\}=\mathbb{P}(A)$, we get $\mathbb{E}[\mathbf{1}_A]=\mathbb{P}(A)$, so every probability is itself an expectation. Whether the defining sum actually produces a number becomes a genuine question once $X$ has infinitely many values — the subject of Remark 13.1 below.

When $X$ is continuous with density $f$, averaging is still 'value times weight', but the weights come from the density and the sum turns into an integral: $\mathbb{E}[X]=\int_{-\infty}^{\infty} x\,f(x)\,dx$ (Definition 13.2), again written $\mu=\mathbb{E}[X]$. A thin slice near $x$ carries probability $f(x)\,dx$, and $x\,f(x)\,dx$ is its contribution to the average — exactly mirroring the term $k\,\mathbb{P}\{X=k\}$ in the discrete sum. The cleanest case is the uniform variable $X\sim\mathrm{Unif}[a,b]$, whose density is the constant $\tfrac{1}{b-a}$ on $[a,b]$: $\mathbb{E}[X]=\int_a^b x\cdot\frac{1}{b-a}\,dx=\frac{1}{b-a}\cdot\frac{b^2-a^2}{2}=\frac{a+b}{2}$ (Example 13.7) — the midpoint of the interval, just as symmetry suggests. The same integral can diverge: the density $f(x)=x^{-2}$ on $[1,\infty)$ is perfectly legitimate, yet $\int_1^{\infty} x\cdot x^{-2}\,dx=\int_1^{\infty}\tfrac1x\,dx=\infty$, so that variable has infinite mean (Example 13.8).

Often we want the average not of $X$ but of a function $g(X)$ — a squared value, a payoff, a cost. One route is to work out the distribution of the new variable $Y=g(X)$ and sum $\sum_j j\,\mathbb{P}\{Y=j\}$; the law of the unconscious statistician (LOTUS, Proposition 13.1) says we may skip that step and weight $g$ directly against the distribution of $X$: $\mathbb{E}[g(X)]=\sum_k g(k)\,\mathbb{P}\{X=k\}$ for discrete $X$ (and $\int_{-\infty}^{\infty} g(x)\,f(x)\,dx$ in the continuous case). The name is a joke — you use the p.m.f. of $X$ 'unconsciously', without re-deriving anything. Taking $g(x)=x^n$ produces the moments of $X$: the $n$-th moment is $\mathbb{E}[X^n]=\sum_k k^n\,\mathbb{P}\{X=k\}$ (Definition 13.3). The first moment ($n=1$) is just the mean $\mu$; the second moment $\mathbb{E}[X^2]$ is the key ingredient of the variance, taken up in a later lesson. Beware that $\mathbb{E}[g(X)]\neq g(\mathbb{E}[X])$ in general — for a fair die, LOTUS gives $\mathbb{E}[X^2]=\tfrac16(1+4+9+16+25+36)=\tfrac{91}{6}\approx15.17$, well above $(\mathbb{E}[X])^2=12.25$.

An expectation is a sum or integral, and those do not always converge, so $\mathbb{E}[X]$ need not exist as a finite number. Remark 13.1 fixes the vocabulary: $\mathbb{E}[X]$ is well-defined if its defining sum or integral has a definite value — a finite number, or $+\infty$, or $-\infty$. Two failure modes are worth seeing. First, the value can be infinite: in a doubling-prize coin game you win $2^n$ with probability $2^{-n}$, so $\mathbb{E}[X]=\sum_{n\ge1} 2^n\cdot 2^{-n}=\sum_{n\ge1}1=\infty,$ even though the prize is finite every single time you play (Example 13.5; the continuous Example 13.8 behaves similarly). Second, the value can be undefined: if the positive and negative parts each sum to infinity there is no consistent total — a net-reward game leads to $1-1+1-1+\cdots$, a series with no limit, so $\mathbb{E}[X]$ simply does not exist (Example 13.6). Finally, keep in mind that a mean is not a probability: it can be negative, and it need not lie in $[0,1]$.