Cochran's theorem

We first show that the matrices B⁽ⁱ⁾ can be simultaneously diagonalized and that their non-zero eigenvalues are all equal to +1. We then use the vector basis that diagonalize them to simplify their characteristic function and show their independence and distribution.[3]

Each of the matrices B⁽ⁱ⁾ has rank r_i and thus r_i non-zero eigenvalues. For each i, the sum ${\displaystyle C^{(i)}\equiv \sum _{j\neq i}B^{(j)}}$ has at most rank ${\displaystyle \sum _{j\neq i}r_{j}=N-r_{i}}$. Since ${\displaystyle B^{(i)}+C^{(i)}=I_{N\times N}}$, it follows that C⁽ⁱ⁾ has exactly rank N − r_i.

Therefore B⁽ⁱ⁾ and C⁽ⁱ⁾ can be simultaneously diagonalized. This can be shown by first diagonalizing B⁽ⁱ⁾. In this basis, it is of the form:

${\displaystyle {\begin{bmatrix}\lambda _{1}&0&0&\cdots &\cdots &&0\\0&\lambda _{2}&0&\cdots &\cdots &&0\\0&0&\ddots &&&&\vdots \\\vdots &\vdots &&\lambda _{r_{i}}&&\\\vdots &\vdots &&&0&\\0&\vdots &&&&\ddots \\0&0&\ldots &&&&0\end{bmatrix}}.}$

Thus the lower ${\displaystyle (N-r_{i})}$ rows are zero. Since ${\displaystyle C^{(i)}=I-B^{(i)}}$, it follows that these rows in C⁽ⁱ⁾ in this basis contain a right block which is a ${\displaystyle (N-r_{i})\times (N-r_{i})}$ unit matrix, with zeros in the rest of these rows. But since C⁽ⁱ⁾ has rank N − r_i, it must be zero elsewhere. Thus it is diagonal in this basis as well. It follows that all the non-zero eigenvalues of both B⁽ⁱ⁾ and C⁽ⁱ⁾ are +1. Moreover, the above analysis can be repeated in the diagonal basis for ${\displaystyle C^{(1)}=B^{(2)}+\sum _{j>2}B^{(j)}}$. In this basis ${\displaystyle C^{(1)}}$ is the identity of an ${\displaystyle (N-r_{1})\times (N-r_{1})}$ vector space, so it follows that both B⁽²⁾ and ${\displaystyle \sum _{j>2}B^{(j)}}$ are simultaneously diagonalizable in this vector space (and hence also together with B⁽¹⁾). By iteration it follows that all B-s are simultaneously diagonalizable.

Thus there exists an orthogonal matrix ${\displaystyle S}$ such that for all ${\displaystyle i}$, ${\displaystyle S^{\mathrm {T} }B^{(i)}S\equiv B^{(i)\prime }}$ is diagonal, where any entry ${\displaystyle B_{x,y}^{(i)\prime }}$ with indices ${\displaystyle x=y}$, ${\displaystyle \sum _{j=1}^{i-1}r_{j}<x=y\leq \sum _{j=1}^{i}r_{j}}$, is equal to 1, while any entry with other indices is equal to 0.

Let ${\displaystyle U_{i}^{\prime }}$ denote some specific linear combination of all ${\displaystyle U_{i}}$ after transformation by ${\displaystyle S}$. Note that ${\displaystyle \sum _{i=1}^{N}(U_{i}^{\prime })^{2}=\sum _{i=1}^{N}U_{i}^{2}}$ due to the length preservation of the orthogonal matrix S, that the Jacobian of a linear transformation is the matrix associated with the linear transformation itself, and that the determinant of an orthogonal matrix has modulus 1.

The characteristic function of Q_i is:

${\displaystyle {\begin{aligned}\varphi _{i}(t)={}&(2\pi )^{-N/2}\int du_{1}\int du_{2}\cdots \int du_{N}e^{itQ_{i}}\cdot e^{-u_{1}^{2}/2}\cdot e^{-u_{2}^{2}/2}\cdots e^{-u_{N}^{2}/2}\\={}&(2\pi )^{-N/2}\left(\prod _{j=1}^{N}\int du_{j}\right)e^{itQ_{i}}\cdot e^{-\sum _{j=1}^{N}u_{j}^{2}/2}\\={}&(2\pi )^{-N/2}\left(\prod _{j=1}^{N}\int du_{j}^{\prime }\right)e^{it\cdot \sum _{m=r_{1}+\cdots +r_{i-1}+1}^{r_{1}+\cdots +r_{i}}(u_{m}^{\prime })^{2}}\cdot e^{-\sum _{j=1}^{N}{u_{j}^{\prime }}^{2}/2}\\={}&(2\pi )^{-N/2}\left(\int e^{u^{2}(it-{\frac {1}{2}})}du\right)^{r_{i}}\left(\int e^{-{\frac {u^{2}}{2}}}du\right)^{N-r_{i}}\\={}&(1-2it)^{-r_{i}/2}\end{aligned}}}$

This is the Fourier transform of the chi-squared distribution with r_i degrees of freedom. Therefore this is the distribution of Q_i.

Moreover, the characteristic function of the joint distribution of all the Q_is is:

${\displaystyle {\begin{aligned}\varphi (t_{1},t_{2},\ldots ,t_{k})&=(2\pi )^{-N/2}\left(\prod _{j=1}^{N}\int dU_{j}\right)e^{i\sum _{i=1}^{k}t_{i}\cdot Q_{i}}\cdot e^{-\sum _{j=1}^{N}U_{j}^{2}/2}\\&=(2\pi )^{-N/2}\left(\prod _{j=1}^{N}\int dU_{j}^{\prime }\right)e^{i\cdot \sum _{i=1}^{k}t_{i}\sum _{k=r_{1}+\cdots +r_{i-1}+1}^{r_{1}+\cdots +r_{i}}(U_{k}^{\prime })^{2}}\cdot e^{-\sum _{j=1}^{N}{U_{j}^{\prime }}^{2}/2}\\&=(2\pi )^{-N/2}\prod _{i=1}^{k}\left(\int e^{u^{2}(it_{i}-{\frac {1}{2}})}du\right)^{r_{i}}\\&=\prod _{i=1}^{k}(1-2it_{i})^{-r_{i}/2}=\prod _{i=1}^{k}\varphi _{i}(t_{i})\end{aligned}}}$

From this it follows that all the Q_is are independent.

If X₁, ..., X_n are independent normally distributed random variables with mean μ and standard deviation σ then

${\displaystyle U_{i}={\frac {X_{i}-\mu }{\sigma }}}$

is standard normal for each i. Note that the total Q is equal to sum of squared Us as shown here:

${\displaystyle \sum _{i}Q_{i}=\sum _{ijk}U_{j}B_{jk}^{(i)}U_{k}=\sum _{jk}U_{j}U_{k}\sum _{i}B_{jk}^{(i)}=\sum _{jk}U_{j}U_{k}\delta _{jk}=\sum _{j}U_{j}^{2}}$

which stems from the original assumption that ${\displaystyle B_{1}+B_{2}\ldots =I}$. So instead we will calculate this quantity and later separate it into Q_i's. It is possible to write

${\displaystyle \sum _{i=1}^{n}U_{i}^{2}=\sum _{i=1}^{n}\left({\frac {X_{i}-{\overline {X}}}{\sigma }}\right)^{2}+n\left({\frac {{\overline {X}}-\mu }{\sigma }}\right)^{2}}$

(here ${\displaystyle {\overline {X}}}$ is the sample mean). To see this identity, multiply throughout by ${\displaystyle \sigma ^{2}}$ and note that

${\displaystyle \sum (X_{i}-\mu )^{2}=\sum (X_{i}-{\overline {X}}+{\overline {X}}-\mu )^{2}}$

and expand to give

${\displaystyle \sum (X_{i}-\mu )^{2}=\sum (X_{i}-{\overline {X}})^{2}+\sum ({\overline {X}}-\mu )^{2}+2\sum (X_{i}-{\overline {X}})({\overline {X}}-\mu ).}$

The third term is zero because it is equal to a constant times

${\displaystyle \sum ({\overline {X}}-X_{i})=0,}$

and the second term has just n identical terms added together. Thus

${\displaystyle \sum (X_{i}-\mu )^{2}=\sum (X_{i}-{\overline {X}})^{2}+n({\overline {X}}-\mu )^{2},}$

and hence

${\displaystyle \sum \left({\frac {X_{i}-\mu }{\sigma }}\right)^{2}=\sum \left({\frac {X_{i}-{\overline {X}}}{\sigma }}\right)^{2}+n\left({\frac {{\overline {X}}-\mu }{\sigma }}\right)^{2}=\overbrace {\sum _{i}\left(U_{i}-{\frac {1}{n}}\sum _{j}{U_{j}}\right)^{2}} ^{Q_{1}}+\overbrace {{\frac {1}{n}}\left(\sum _{j}{U_{j}}\right)^{2}} ^{Q_{2}}=Q_{1}+Q_{2}.}$

Now ${\displaystyle B^{(2)}={\frac {J_{n}}{n}}}$ with ${\displaystyle J_{n}}$ the matrix of ones which has rank 1. In turn ${\displaystyle B^{(1)}=I_{n}-{\frac {J_{n}}{n}}}$ given that ${\displaystyle I_{n}=B^{(1)}+B^{(2)}}$. This expression can be also obtained by expanding ${\displaystyle Q_{1}}$ in matrix notation. It can be shown that the rank of ${\displaystyle B^{(1)}}$ is ${\displaystyle n-1}$ as the addition of all its rows is equal to zero. Thus the conditions for Cochran's theorem are met.

Cochran's theorem then states that Q₁ and Q₂ are independent, with chi-squared distributions with n − 1 and 1 degree of freedom respectively. This shows that the sample mean and sample variance are independent. This can also be shown by Basu's theorem, and in fact this property characterizes the normal distribution – for no other distribution are the sample mean and sample variance independent.[4]

The result for the distributions is written symbolically as

${\displaystyle \sum \left(X_{i}-{\overline {X}}\right)^{2}\sim \sigma ^{2}\chi _{n-1}^{2}.}$

${\displaystyle n({\overline {X}}-\mu )^{2}\sim \sigma ^{2}\chi _{1}^{2},}$

Both these random variables are proportional to the true but unknown variance σ². Thus their ratio does not depend on σ² and, because they are statistically independent. The distribution of their ratio is given by

${\displaystyle {\frac {n\left({\overline {X}}-\mu \right)^{2}}{{\frac {1}{n-1}}\sum \left(X_{i}-{\overline {X}}\right)^{2}}}\sim {\frac {\chi _{1}^{2}}{{\frac {1}{n-1}}\chi _{n-1}^{2}}}\sim F_{1,n-1}}$

where F_1,n − 1 is the F-distribution with 1 and n − 1 degrees of freedom (see also Student's t-distribution). The final step here is effectively the definition of a random variable having the F-distribution.

To estimate the variance σ², one estimator that is sometimes used is the maximum likelihood estimator of the variance of a normal distribution

${\displaystyle {\widehat {\sigma }}^{2}={\frac {1}{n}}\sum \left(X_{i}-{\overline {X}}\right)^{2}.}$

Cochran's theorem shows that

${\displaystyle {\frac {n{\widehat {\sigma }}^{2}}{\sigma ^{2}}}\sim \chi _{n-1}^{2}}$

and the properties of the chi-squared distribution show that

${\displaystyle {\begin{aligned}E\left({\frac {n{\widehat {\sigma }}^{2}}{\sigma ^{2}}}\right)&=E\left(\chi _{n-1}^{2}\right)\\{\frac {n}{\sigma ^{2}}}E\left({\widehat {\sigma }}^{2}\right)&=(n-1)\\E\left({\widehat {\sigma }}^{2}\right)&={\frac {\sigma ^{2}(n-1)}{n}}\end{aligned}}}$