Residual neural network

The brain has structures similar to residual nets, as cortical layer VI neurons get input from layer I, skipping intermediary layers.[6] In the figure this compares to signals from the apical dendrite (3) skipping over layers, while the basal dendrite (2) collects signals from the previous and/or same layer.[note 1][7] Similar structures exists for other layers.[8] How many layers in the cerebral cortex compare to layers in an artificial neural network is not clear, nor whether every area in the cerebral cortex exhibits the same structure, but over large areas they appear similar.

For single skips, the layers may be indexed either as ${\textstyle \ell -2}$ to ${\textstyle \ell }$ or as ${\textstyle \ell }$ to ${\textstyle \ell +2}$. (Script ${\textstyle \ell }$ used for clarity, usually it is written as a simple l.) The two indexing systems are convenient when describing skips as going backward or forward. As signal flows forward through the network it is easier to describe the skip as ${\textstyle \ell +k}$ from a given layer, but as a learning rule (back propagation) it is easier to describe which activation layer you reuse as ${\textstyle \ell -k}$, where ${\textstyle k-1}$ is the skip number.

Given a weight matrix ${\textstyle W^{\ell -1,\ell }}$for connection weights from layer ${\textstyle \ell -1}$ to ${\textstyle \ell }$, and a weight matrix ${\textstyle W^{\ell -2,\ell }}$ for connection weights from layer ${\textstyle \ell -2}$ to ${\textstyle \ell }$, then the forward propagation through the activation function would be (aka HighwayNets)

${\displaystyle {\begin{aligned}a^{\ell }&:=\mathbf {g} (W^{\ell -1,\ell }\cdot a^{\ell -1}+b^{\ell }+W^{\ell -2,\ell }\cdot a^{\ell -2})\\&:=\mathbf {g} (Z^{\ell }+W^{\ell -2,\ell }\cdot a^{\ell -2})\end{aligned}}}$

where

${\textstyle a^{\ell }}$ the activations (outputs) of neurons in layer ${\textstyle \ell }$,

${\textstyle \mathbf {g} }$ the activation function for layer ${\textstyle \ell }$,

${\textstyle W^{\ell -1,\ell }}$ the weight matrix for neurons between layer ${\textstyle \ell -1}$ and ${\textstyle \ell }$, and

${\textstyle Z^{\ell }=W^{\ell -1,\ell }\cdot a^{\ell -1}+b^{\ell }}$

Absent an explicit matrix ${\textstyle W^{\ell -2,\ell }}$ (aka ResNets), forward propagation through the activation function simplifies to

${\displaystyle a^{\ell }:=\mathbf {g} (Z^{\ell }+a^{\ell -2})}$

Another way to formulate this is to substitute an identity matrix for ${\textstyle W^{\ell -2,\ell }}$, but that is only valid when the dimensions match. This is somewhat confusingly called an identity block, which means that the activations from layer ${\textstyle \ell -2}$ are passed to layer ${\textstyle \ell }$ without weighting.

In the cerebral cortex such forward skips are done for several layers. Usually all forward skips start from the same layer, and successively connect to later layers. In the general case this will be expressed as (aka DenseNets)

${\displaystyle a^{\ell }:=\mathbf {g} \left(Z^{\ell }+\sum _{k=2}^{K}W^{\ell -k,\ell }\cdot a^{\ell -k}\right)}$.

During backpropagation learning for the normal path

${\displaystyle \Delta w^{\ell -1,\ell }:=-\eta {\frac {\partial E^{\ell }}{\partial w^{\ell -1,\ell }}}=-\eta a^{\ell -1}\cdot \delta ^{\ell }}$

and for the skip paths (nearly identical)

${\displaystyle \Delta w^{\ell -2,\ell }:=-\eta {\frac {\partial E^{\ell }}{\partial w^{\ell -2,\ell }}}=-\eta a^{\ell -2}\cdot \delta ^{\ell }}$.

In both cases

${\textstyle \eta }$ a learning rate (${\textstyle \eta <0)}$,

${\textstyle \delta ^{\ell }}$ the error signal of neurons at layer ${\textstyle \ell }$, and

${\textstyle a_{i}^{\ell }}$ the activation of neurons at layer ${\textstyle \ell }$.

If the skip path has fixed weights (e.g. the identity matrix, as above), then they are not updated. If they can be updated, the rule is an ordinary backpropagation update rule.

In the general case there can be ${\textstyle K}$ skip path weight matrices, thus

${\displaystyle \Delta w^{\ell -k,\ell }:=-\eta {\frac {\partial E^{\ell }}{\partial w^{\ell -k,\ell }}}=-\eta a^{\ell -k}\cdot \delta ^{\ell }}$

As the learning rules are similar, the weight matrices can be merged and learned in the same step.