Hilbert projection theorem

Some intuition for the theorem can be obtained by considering the first order condition of the optimization problem.

Consider a finite dimensional real Hilbert space ${\displaystyle H}$ with a subspace${\displaystyle M}$ and a point ${\displaystyle x}$. If ${\displaystyle y\in M}$ is a minimizer (in ${\displaystyle z}$) of ${\displaystyle \lVert x-z\rVert }$, then derivative must be zero.

In matrix derivative notation[1]

${\displaystyle {\begin{aligned}\partial \lVert x-z\rVert ^{2}&=\partial \langle z-x,z-x\rangle \\&=2\langle z-x,\partial z\rangle \end{aligned}}}$

Since ${\displaystyle \partial z}$ represents an arbitrary tangent direction, that is a vector in ${\displaystyle M}$, we see that ${\displaystyle y-x}$ must be orthogonal to all of ${\displaystyle M}$.

• Let us show the existence of y:

Let δ be the distance between x and C, (y_n) a sequence in C such that the distance squared between x and y_n is below or equal to δ² + 1/n. Let n and m be two integers, then the following equalities are true:

${\displaystyle \|y_{n}-y_{m}\|^{2}=\|y_{n}-x\|^{2}+\|y_{m}-x\|^{2}-2\langle y_{n}-x\,,\,y_{m}-x\rangle }$

and

${\displaystyle 4\left\|{\frac {y_{n}+y_{m}}{2}}-x\right\|^{2}=\|y_{n}-x\|^{2}+\|y_{m}-x\|^{2}+2\langle y_{n}-x\,,\,y_{m}-x\rangle }$

We have therefore:

${\displaystyle \|y_{n}-y_{m}\|^{2}=2\|y_{n}-x\|^{2}+2\|y_{m}-x\|^{2}-4\left\|{\frac {y_{n}+y_{m}}{2}}-x\right\|^{2}}$

(Recall the formula for the median in a triangle - Median_(geometry)#Formulas_involving_the_medians'_lengths) By giving an upper bound to the first two terms of the equality and by noticing that the middle of y_n and y_m belong to C and has therefore a distance greater than or equal to δ from x, one gets :

${\displaystyle \|y_{n}-y_{m}\|^{2}\;\leq \;2\left(\delta ^{2}+{\frac {1}{n}}\right)+2\left(\delta ^{2}+{\frac {1}{m}}\right)-4\delta ^{2}=2\left({\frac {1}{n}}+{\frac {1}{m}}\right)}$

The last inequality proves that (y_n) is a Cauchy sequence. Since C is complete, the sequence is therefore convergent to a point y in C, whose distance from x is minimal.

• Let us show the uniqueness of y :

Let y₁ and y₂ be two minimizers. Then:

${\displaystyle \|y_{2}-y_{1}\|^{2}=2\|y_{1}-x\|^{2}+2\|y_{2}-x\|^{2}-4\left\|{\frac {y_{1}+y_{2}}{2}}-x\right\|^{2}}$

Since ${\displaystyle {\frac {y_{1}+y_{2}}{2}}}$ belongs to C, we have ${\displaystyle \left\|{\frac {y_{1}+y_{2}}{2}}-x\right\|^{2}\geq \delta ^{2}}$ and therefore

${\displaystyle \|y_{2}-y_{1}\|^{2}\leq 2\delta ^{2}+2\delta ^{2}-4\delta ^{2}=0\,}$

Hence ${\displaystyle y_{1}=y_{2}}$, which proves uniqueness.

• Let us show the equivalent condition on y when C = M is a closed subspace.

The condition is sufficient: Let ${\displaystyle z\in M}$ such that ${\displaystyle \langle z-x,a\rangle =0}$ for all ${\displaystyle a\in M}$. ${\displaystyle \|x-a\|^{2}=\|z-x\|^{2}+\|a-z\|^{2}+2\langle z-x,a-z\rangle =\|z-x\|^{2}+\|a-z\|^{2}}$ which proves that ${\displaystyle z}$ is a minimizer.

The condition is necessary: Let ${\displaystyle y\in M}$ be the minimizer. Let ${\displaystyle a\in M}$ and ${\displaystyle t\in \mathbb {R} }$.

${\displaystyle \|(y+ta)-x\|^{2}-\|y-x\|^{2}=2t\langle y-x,a\rangle +t^{2}\|a\|^{2}=2t\langle y-x,a\rangle +O(t^{2})}$

is always non-negative. Therefore, ${\displaystyle \langle y-x,a\rangle =0.}$

QED

• Walter Rudin, Real and Complex Analysis. Third Edition, 1987.

• Orthogonality principle