Principal Component Analysis

PCA starts from raw measurements and strips away the parts that carry no shape information. Arrange the data as a matrix $M$ whose columns are the objects (subjects, samples) and whose rows are the features (Remark 15.9, step 1). The first normalization step is centering: from each feature (row) subtract its sample mean, $s_i'=s_i-\mu_i\mathbf 1,\qquad \mu_i=\tfrac1n\sum_{k}s_{ik},$ so every row of the resulting matrix has mean $0$. If the features live on very different scales (heights in cm versus dimensionless ratios, say), also divide each centered row by its sample standard deviation $\sigma_i$, giving rows of unit variance — the standardized version, which makes $C$ a correlation matrix. The normalized data is the matrix $A$. Centering is not optional: covariance is measured about the mean, and skipping it folds the mean vector into the first principal component and corrupts every eigenvalue.

To see how features move together we build the covariance matrix $C=\frac{1}{n-1}AA^{\mathsf T}\in\mathbb R^{d\times d}$ for $d$ features (Remark 15.9, step 3). Its entry $C_{ij}=\frac{1}{n-1}\sum_k a_{ik}a_{jk}$ is exactly the sample covariance $\operatorname{Cov}(s_i,s_j)$ of features $i$ and $j$ (Definition 15.7, §15.5), and the diagonal entries $C_{ii}$ are the feature variances. Dividing by $n-1$ rather than $n$ is the unbiased sample convention used in the course. Two structural facts drive everything that follows. First, $C$ is symmetric, $C^{\mathsf T}=C$, because $\operatorname{Cov}(s_i,s_j)=\operatorname{Cov}(s_j,s_i)$ — covariance is symmetric and bilinear, behaving like an inner product on centered features (Proposition 15.17, Remark 15.7). Second, $C=\frac{1}{n-1}AA^{\mathsf T}$ is positive semidefinite, so all its eigenvalues are $\ge 0$ — reassuring, since they will turn out to be variances. A large $|C_{ij}|$ signals redundancy between two features; a near-zero one means they are linearly unrelated.

The whole point of PCA is to change coordinates so the features become uncorrelated, i.e. so the covariance matrix becomes diagonal (Remark 15.8). Because $C$ is real and symmetric, the Spectral Theorem (Theorem 9.1) guarantees an orthonormal basis of eigenvectors and an orthogonal matrix $S$ — its columns the eigenvectors — with $C=SDS^{\mathsf T},\qquad D=\operatorname{diag}(\lambda_1,\dots,\lambda_d).$ Rewrite the data in the new basis via $A'=S^{\mathsf T}A$ (Remark 15.9, step 5; here $S^{-1}=S^{\mathsf T}$ since $S$ is orthogonal). The covariance matrix of the new features is $C'=\frac{1}{n-1}A'A'^{\mathsf T}=S^{\mathsf T}CS=D,$ which is diagonal: every covariance is now $0$. The principal components are these orthonormal eigenvectors of $C$ (the columns of $S$), sorted so that $\lambda_1\ge\lambda_2\ge\cdots\ge\lambda_d$. They are orthogonal by construction but not unique: an eigenvector is defined only up to sign (and, within a repeated-eigenvalue subspace, up to rotation), so $\mathbf u$ and $-\mathbf u$ are equally valid principal components.

In the diagonalized covariance matrix $D$, the eigenvalue $\lambda_k$ is the variance of the data along the $k$-th principal component. This gives a clean budget for information: the total variance is $\operatorname{tr}(C)=\sum_i C_{ii}=\sum_k\lambda_k,$ and the proportion of variance explained by component $k$ is $\lambda_k/\sum_j\lambda_j$. Dimensionality reduction keeps the components with the largest eigenvalues and discards the smallest: if $\lambda_r,\dots,\lambda_d$ are tiny, the matching new features barely vary across subjects and carry almost no information, so projecting onto the top components (the first rows of $A'$) loses very little (Remark 15.9, steps 6–7). A near-zero eigenvalue flags a redundant feature — in the course's example the covariance matrix of height, weight and BMI has an eigenvalue $\approx 0$ because BMI is determined by the other two. The same principal directions come directly from the singular value decomposition (Theorem 9.3): if $A=U\Sigma V^{\mathsf T}$, the left singular vectors (columns of $U$) are the principal components and $\lambda_k=\sigma_k^2/(n-1)$. Working from the SVD of $A$ avoids forming $C$ explicitly and is the numerically preferred route.