Binary symmetric channel

The theorem can be proved directly with a probabilistic method. Consider an encoding function ${\displaystyle E:\{0,1\}^{k}\to \{0,1\}^{n}}$ that is selected at random. This means that for each message ${\displaystyle m\in \{0,1\}^{k}}$, the value ${\displaystyle E(m)\in \{0,1\}^{n}}$ is selected at random (with equal probabilities). For a given encoding function ${\displaystyle E}$, the decoding function ${\displaystyle D:\{0,1\}^{n}\to \{0,1\}^{k}}$ is specified as follows: given any received codeword ${\displaystyle y\in \{0,1\}^{n}}$, we find the message ${\displaystyle m\in \{0,1\}^{k}}$ such that the Hamming distance ${\displaystyle \Delta (y,E(m))}$ is as small as possible (with ties broken arbitrarily). (${\displaystyle D}$ is called a maximum likelihood decoding function.)

The proof continues by showing that at least one such choice ${\displaystyle (E,D)}$ satisfies the conclusion of theorem, by integration over the probabilities. Suppose ${\displaystyle p}$ and ${\displaystyle \epsilon }$ are fixed. First we show that, for a fixed ${\displaystyle m\in \{0,1\}^{k}}$ and ${\displaystyle E}$ chosen randomly, the probability of failure over ${\displaystyle {\text{BSC}}_{p}}$ noise is exponentially small in n. At this point, the proof works for a fixed message ${\displaystyle m}$. Next we extend this result to work for all messages ${\displaystyle m}$. We achieve this by eliminating half of the codewords from the code with the argument that the proof for the decoding error probability holds for at least half of the codewords. The latter method is called expurgation. This gives the total process the name random coding with expurgation.

Continuation of proof (sketch)

Fix ${\displaystyle p}$ and ${\displaystyle \epsilon }$. Given a fixed message ${\displaystyle m\in \{0,1\}^{k}}$, we need to estimate the expected value of the probability of the received codeword along with the noise does not give back ${\displaystyle m}$ on decoding. That is to say, we need to estimate: ${\displaystyle \mathbb {E} _{E}\left[\Pr _{e\in {\text{BSC}}_{p}}[D(E(m)+e)\neq m]\right].}$ Let ${\displaystyle y}$ be the received codeword. In order for the decoded codeword ${\displaystyle D(y)}$ not to be equal to the message ${\displaystyle m}$, one of the following events must occur: ${\displaystyle y}$ does not lie within the Hamming ball of radius ${\displaystyle (p+\epsilon )n}$ centered at ${\displaystyle E(m)}$. This condition is mainly used to make the calculations easier. There is another message ${\displaystyle m'\in \{0,1\}^{k}}$ such that ${\displaystyle \Delta (y,E(m'))\leqslant \Delta (y,E(m))}$. In other words, the errors due to noise take the transmitted codeword closer to another encoded message. We can apply the Chernoff bound to ensure the non occurrence of the first event; we get: ${\displaystyle Pr_{e\in {\text{BSC}}_{p}}[\Delta (y,E(m))>(p+\epsilon )n]\leqslant 2^{-{\epsilon ^{2}}n}.}$ This is exponentially small for large ${\displaystyle n}$ (recall that ${\displaystyle \epsilon }$ is fixed). For the second event, we note that the probability that ${\displaystyle E(m')\in B(y,(p+\epsilon )n)}$ is ${\displaystyle {\text{Vol}}(B(y,(p+\epsilon )n)/2^{n}}$ where ${\displaystyle B(x,r)}$ is the Hamming ball of radius ${\displaystyle r}$ centered at vector ${\displaystyle x}$ and ${\displaystyle {\text{Vol}}(B(x,r))}$ is its volume. Using approximation to estimate the number of codewords in the Hamming ball, we have ${\displaystyle {\text{Vol}}(B(y,(p+\epsilon )n))\approx 2^{H(p)n}}$. Hence the above probability amounts to ${\displaystyle 2^{H(p)n}/2^{n}=2^{H(p)n-n}}$. Now using the union bound, we can upper bound the existence of such an ${\displaystyle m'\in \{0,1\}^{k}}$ by ${\displaystyle \leq 2^{k+H(p)n-n}}$ which is ${\displaystyle 2^{-\Omega (n)}}$, as desired by the choice of ${\displaystyle k}$.

Continuation of proof (detailed)

From the above analysis, we calculate the probability of the event that the decoded codeword plus the channel noise is not the same as the original message sent. We shall introduce some symbols here. Let ${\displaystyle p(y|E(m))}$ denote the probability of receiving codeword ${\displaystyle y}$ given that codeword ${\displaystyle E(m)}$ was sent. Let ${\displaystyle B_{0}}$ denote ${\displaystyle B(E(m),(p+\epsilon )n).}$ ${\displaystyle {\begin{aligned}\Pr _{e\in {\text{BSC}}_{p}}[D(E(m)+e)\neq m]&=\sum _{y\in \{0,1\}^{n}}p(y|E(m))\cdot 1_{D(y)\neq m}\\&\leqslant \sum _{y\notin B_{0}}p(y|E(m))\cdot 1_{D(y)\neq m}+\sum _{y\in B_{0}}p(y|E(m))\cdot 1_{D(y)\neq m}\\&\leqslant 2^{-{\epsilon ^{2}}n}+\sum _{y\in B_{0}}p(y|E(m))\cdot 1_{D(y)\neq m}\end{aligned}}}$ We get the last inequality by our analysis using the Chernoff bound above. Now taking expectation on both sides we have, ${\displaystyle {\begin{aligned}\mathbb {E} _{E}\left[\Pr _{e\in {\text{BSC}}_{p}}[D(E(m)+e)\neq m]\right]&\leqslant 2^{-{\epsilon ^{2}}n}+\sum _{y\in B_{0}}p(y|E(m))\mathbb {E} [1_{D(y)\neq m}]\\&\leqslant 2^{-{\epsilon ^{2}}n}+\sum _{y\in B_{0}}\mathbb {E} [1_{D(y)\neq m}]&&\sum _{y\in B_{0}}p(y|E(m))\leqslant 1\\&\leqslant 2^{-{\epsilon ^{2}}n}+2^{k+H(p+\epsilon )n-n}&&\mathbb {E} [1_{D(y)\neq m}]\leqslant 2^{k+H(p+\epsilon )n-n}{\text{ (see above)}}\\&\leqslant 2^{-\delta n}\end{aligned}}}$ by appropriately choosing the value of ${\displaystyle \delta }$. Since the above bound holds for each message, we have ${\displaystyle \mathbb {E} _{m}\left[\mathbb {E} _{E}\left[\Pr _{e\in {\text{BSC}}_{p}}\left[D(E(m)+e)\right]\neq m\right]\right]\leqslant 2^{-\delta n}.}$ Now we can change the order of summation in the expectation with respect to the message and the choice of the encoding function ${\displaystyle E}$. Hence: ${\displaystyle \mathbb {E} _{E}\left[\mathbb {E} _{m}\left[\Pr _{e\in {\text{BSC}}_{p}}\left[D(E(m)+e)\right]\neq m\right]\right]\leqslant 2^{-\delta n}.}$ Hence in conclusion, by probabilistic method, we have some encoding function ${\displaystyle E^{*}}$ and a corresponding decoding function ${\displaystyle D^{*}}$ such that ${\displaystyle \mathbb {E} _{m}\left[\Pr _{e\in {\text{BSC}}_{p}}\left[D^{*}(E^{*}(m)+e)\neq m\right]\right]\leqslant 2^{-\delta n}.}$ At this point, the proof works for a fixed message ${\displaystyle m}$. But we need to make sure that the above bound holds for all the messages ${\displaystyle m}$ simultaneously. For that, let us sort the ${\displaystyle 2^{k}}$ messages by their decoding error probabilities. Now by applying Markov's inequality, we can show the decoding error probability for the first ${\displaystyle 2^{k-1}}$ messages to be at most ${\displaystyle 2\cdot 2^{-\delta n}}$. Thus in order to confirm that the above bound to hold for every message ${\displaystyle m}$, we could just trim off the last ${\displaystyle 2^{k-1}}$ messages from the sorted order. This essentially gives us another encoding function ${\displaystyle E'}$ with a corresponding decoding function ${\displaystyle D'}$ with a decoding error probability of at most ${\displaystyle 2^{-\delta n+1}}$ with the same rate. Taking ${\displaystyle \delta '}$ to be equal to ${\displaystyle \delta -{\tfrac {1}{n}}}$ we bound the decoding error probability to ${\displaystyle 2^{-\delta 'n}}$. This expurgation process completes the proof.

Forney constructed a concatenated code ${\displaystyle C^{*}=C_{\text{out}}\circ C_{\text{in}}}$ to achieve the capacity of the noisy-channel coding theorem for ${\displaystyle {\text{BSC}}_{p}}$. In his code,

• The outer code ${\displaystyle C_{\text{out}}}$ is a code of block length ${\displaystyle N}$ and rate ${\displaystyle 1-{\frac {\epsilon }{2}}}$ over the field ${\displaystyle F_{2^{k}}}$, and ${\displaystyle k=O(\log N)}$. Additionally, we have a decoding algorithm ${\displaystyle D_{\text{out}}}$ for ${\displaystyle C_{\text{out}}}$ which can correct up to ${\displaystyle \gamma }$ fraction of worst case errors and runs in ${\displaystyle t_{\text{out}}(N)}$ time.
• The inner code ${\displaystyle C_{\text{in}}}$ is a code of block length ${\displaystyle n}$, dimension ${\displaystyle k}$, and a rate of ${\displaystyle 1-H(p)-{\frac {\epsilon }{2}}}$. Additionally, we have a decoding algorithm ${\displaystyle D_{\text{in}}}$ for ${\displaystyle C_{\text{in}}}$ with a decoding error probability of at most ${\displaystyle {\frac {\gamma }{2}}}$ over ${\displaystyle {\text{BSC}}_{p}}$ and runs in ${\displaystyle t_{\text{in}}(N)}$ time.

For the outer code ${\displaystyle C_{\text{out}}}$, a Reed-Solomon code would have been the first code to have come in mind. However, we would see that the construction of such a code cannot be done in polynomial time. This is why a binary linear code is used for ${\displaystyle C_{\text{out}}}$.

For the inner code ${\displaystyle C_{\text{in}}}$ we find a linear code by exhaustively searching from the linear code of block length ${\displaystyle n}$ and dimension ${\displaystyle k}$, whose rate meets the capacity of ${\displaystyle {\text{BSC}}_{p}}$, by the noisy-channel coding theorem.

The rate ${\displaystyle R(C^{*})=R(C_{\text{in}})\times R(C_{\text{out}})=(1-{\frac {\epsilon }{2}})(1-H(p)-{\frac {\epsilon }{2}})\geq 1-H(p)-\epsilon }$ which almost meets the ${\displaystyle {\text{BSC}}_{p}}$ capacity. We further note that the encoding and decoding of ${\displaystyle C^{*}}$ can be done in polynomial time with respect to ${\displaystyle N}$. As a matter of fact, encoding ${\displaystyle C^{*}}$ takes time ${\displaystyle O(N^{2})+O(Nk^{2})=O(N^{2})}$. Further, the decoding algorithm described takes time ${\displaystyle Nt_{\text{in}}(k)+t_{\text{out}}(N)=N^{O(1)}}$ as long as ${\displaystyle t_{\text{out}}(N)=N^{O(1)}}$; and ${\displaystyle t_{\text{in}}(k)=2^{O(k)}}$.

A natural decoding algorithm for ${\displaystyle C^{*}}$ is to:

• Assume ${\displaystyle y_{i}^{\prime }=D_{\text{in}}(y_{i}),\quad i\in (0,N)}$
• Execute ${\displaystyle D_{\text{out}}}$ on ${\displaystyle y^{\prime }=(y_{1}^{\prime }\ldots y_{N}^{\prime })}$

Note that each block of code for ${\displaystyle C_{\text{in}}}$ is considered a symbol for ${\displaystyle C_{\text{out}}}$. Now since the probability of error at any index ${\displaystyle i}$ for ${\displaystyle D_{\text{in}}}$ is at most ${\displaystyle {\tfrac {\gamma }{2}}}$ and the errors in ${\displaystyle {\text{BSC}}_{p}}$ are independent, the expected number of errors for ${\displaystyle D_{\text{in}}}$ is at most ${\displaystyle {\tfrac {\gamma N}{2}}}$ by linearity of expectation. Now applying Chernoff bound, we have bound error probability of more than ${\displaystyle \gamma N}$ errors occurring to be ${\displaystyle e^{\frac {-\gamma N}{6}}}$. Since the outer code ${\displaystyle C_{\text{out}}}$ can correct at most ${\displaystyle \gamma N}$ errors, this is the decoding error probability of ${\displaystyle C^{*}}$. This when expressed in asymptotic terms, gives us an error probability of ${\displaystyle 2^{-\Omega (\gamma N)}}$. Thus the achieved decoding error probability of ${\displaystyle C^{*}}$ is exponentially small as the noisy-channel coding theorem.

We have given a general technique to construct ${\displaystyle C^{*}}$. For more detailed descriptions on ${\displaystyle C_{\text{in}}}$ and ${\displaystyle C_{\text{out}}}$ please read the following references. Recently a few other codes have also been constructed for achieving the capacities. LDPC codes have been considered for this purpose for their faster decoding time.[4]