Orthogonal Projections & the Orthogonal Complement

A vector $\mathbf{x}\in\mathbb{R}^n$ is orthogonal to a subspace $V$ when it is perpendicular to every vector of $V$, i.e. $\mathbf{x}\cdot\mathbf{v}=0$ for all $\mathbf{v}\in V$ (Definition 8.8). Collecting all such vectors gives the orthogonal complement $V^{\perp}=\{\mathbf{x}\in\mathbb{R}^n:\mathbf{x}\cdot\mathbf{v}=0\text{ for all }\mathbf{v}\in V\},$ which is itself a subspace (Lemma 8.2). Checking orthogonality against infinitely many $\mathbf{v}\in V$ sounds hopeless, but Lemma 8.3 rescues us: $\mathbf{x}\in V^{\perp}$ iff $\mathbf{x}\cdot\mathbf{v}_i=0$ for each vector $\mathbf{v}_i$ of a basis of $V$, because any $\mathbf{v}=c_1\mathbf{v}_1+\cdots+c_k\mathbf{v}_k$ then gives $\mathbf{x}\cdot\mathbf{v}=\sum_i c_i(\mathbf{x}\cdot\mathbf{v}_i)=0$ by linearity. So a membership test in $V^{\perp}$ is just finitely many dot products.

Theorem 8.6 is the heart of the section: for any subspace $V\subseteq\mathbb{R}^n$, every $\mathbf{x}$ decomposes uniquely as $\mathbf{x}=\mathbf{x}_{\parallel}+\mathbf{x}_{\perp},\qquad \mathbf{x}_{\parallel}\in V,\ \ \mathbf{x}_{\perp}\in V^{\perp}.$ The in-subspace part $\mathbf{x}_{\parallel}$ is the orthogonal projection $\operatorname{proj}_V\mathbf{x}$, and the leftover $\mathbf{x}_{\perp}=\mathbf{x}-\operatorname{proj}_V\mathbf{x}$ is the residual. To compute it, take a basis of $V$ and demand that $\mathbf{x}_{\perp}$ be orthogonal to each basis vector; this pins the coefficients down exactly. With an orthonormal basis $\mathbf{u}_1,\dots,\mathbf{u}_k$ the answer is clean (Remark 8.5): $\operatorname{proj}_V\mathbf{x}=(\mathbf{x}\cdot\mathbf{u}_1)\mathbf{u}_1+\cdots+(\mathbf{x}\cdot\mathbf{u}_k)\mathbf{u}_k,$ and for a merely orthogonal basis $\mathbf{v}_1,\dots,\mathbf{v}_k$ each term is divided by $\|\mathbf{v}_i\|^{2}$: $\operatorname{proj}_V\mathbf{x}=\sum_i\frac{\mathbf{x}\cdot\mathbf{v}_i}{\|\mathbf{v}_i\|^{2}}\mathbf{v}_i$. The map $\mathbf{x}\mapsto\operatorname{proj}_V\mathbf{x}$ is linear.

Because $P_V$ is linear, it is multiplication by a matrix. Writing each term of the orthonormal formula as $(\mathbf{x}\cdot\mathbf{u}_i)\mathbf{u}_i=\mathbf{u}_i\mathbf{u}_i^{T}\mathbf{x}$ and summing gives Theorem 8.8: with $Q=[\,\mathbf{u}_1\;\cdots\;\mathbf{u}_k\,]$ (orthonormal columns), $\operatorname{proj}_V\mathbf{x}=P\mathbf{x},\qquad P=QQ^{T}.$ The matrix $P$ is symmetric ($P^{T}=P$) and idempotent ($P^{2}=P$) -- projecting twice does nothing new, since the first projection already lands in $V$. If your basis is linearly independent but not orthonormal, assemble it as the columns of $A$ and use the general formula $P=A(A^{T}A)^{-1}A^{T},$ which is well defined because $\ker A=\{\mathbf{0}\}$ makes $A^{T}A$ invertible (Corollary 8.1). When the columns happen to be orthonormal, $A^{T}A=I$ and this collapses back to $QQ^{T}$. In practice one solves the normal equations $A^{T}A\,\mathbf{c}=A^{T}\mathbf{x}$ for the coordinates $\mathbf{c}$ and sets $\operatorname{proj}_V\mathbf{x}=A\mathbf{c}$.

The projection is the closest point of $V$ to $\mathbf{x}$: since $\mathbf{x}-\operatorname{proj}_V\mathbf{x}$ is orthogonal to $V$, Pythagoras gives $\|\mathbf{x}\|^{2}=\|\operatorname{proj}_V\mathbf{x}\|^{2}+\|\mathbf{x}_{\perp}\|^{2}$, so $\|\operatorname{proj}_V\mathbf{x}\|\le\|\mathbf{x}\|$ with equality iff $\mathbf{x}\in V$ (Theorem 8.7), and the distance from $\mathbf{x}$ to $V$ is $\|\mathbf{x}-\operatorname{proj}_V\mathbf{x}\|$. Structurally, $V$ and $V^{\perp}$ are complementary partners (Theorem 8.9): $V\cap V^{\perp}=\{\mathbf{0}\}$, $\dim V+\dim V^{\perp}=n$, and $(V^{\perp})^{\perp}=V$ -- all immediate once you note $V=\operatorname{im}(P_V)$, $V^{\perp}=\ker(P_V)$ and apply rank--nullity. Finally, orthogonal complements connect to linear maps through Theorem 8.10: $\operatorname{im}(A)^{\perp}=\ker(A^{T}),$ since being perpendicular to every column of $A$ is exactly the condition $A^{T}\mathbf{x}=\mathbf{0}$. This is the backbone of least squares: the residual of the best approximation lies in $\ker(A^{T})$.