Universal approximation theorem

One of the first versions of the arbitrary width case was proved by George Cybenko in 1989 for sigmoid activation functions.[7] Kurt Hornik showed in 1991[8] that it is not the specific choice of the activation function, but rather the multilayer feed-forward architecture itself which gives neural networks the potential of being universal approximators. Moshe Leshno et al in 1993[9] and later Allan Pinkus in 1999[10] showed that the universal approximation property,[11] is equivalent to having a nonpolynomial activation function.

The arbitrary depth case was also studied by number of authors, such as Zhou Lu et al in 2017,[12] Boris Hanin and Mark Sellke in 2018,[13] and Patrick Kidger and Terry Lyons in 2020.[14] The result minimal width per layer was refined in.[15]

Several extensions of the theorem exist, such as to discontinuous activation functions,[9] noncompact domains,[14] certifiable networks[16] and alternative network architectures and topologies.[14][17] A full characterization of the universal approximation property on general function spaces is given by A. Kratsios in.[11]

The classical form of the universal approximation theorem for arbitrary width and bounded depth is as follows.[7][8][18][19] It extends[10] the classical results of George Cybenko and Kurt Hornik.

Universal Approximation Theorem: Fix a continuous function ${\displaystyle \sigma$ :\mathbb {R} \rightarrow \mathbb {R} } (activation function) and positive integers ${\displaystyle d,D}$. The function ${\displaystyle \sigma }$ is not a polynomial if and only if, for every continuous function ${\displaystyle f:\mathbb {R} ^{d}\to \mathbb {R} ^{D}}$ (target function), every compact subset ${\displaystyle K}$ of ${\displaystyle \mathbb {R} ^{d}}$, and every ${\displaystyle \epsilon >0}$ there exists a continuous function ${\displaystyle f_{\epsilon }:\mathbb {R} ^{d}\to \mathbb {R} ^{D}}$ (the layer output) with representation

${\displaystyle f_{\epsilon }=W_{2}\circ \sigma \circ W_{1},}$

where ${\displaystyle W_{2},W_{1}}$ are composable affine maps and ${\displaystyle \circ }$ denotes component-wise composition, such that the approximation bound

${\displaystyle \sup _{x\in K}\,\|f(x)-f_{\epsilon }(x)\|<\varepsilon }$

holds for any ${\displaystyle \epsilon }$ arbitrarily small (distance from ${\displaystyle f}$ to ${\displaystyle f_{\epsilon }}$ can be infinitely small).

The theorem states that the result of first layer ${\displaystyle f_{\epsilon }}$ can approximate any well-behaved function ${\displaystyle f}$. Such a well-behaved function can also be approximated by a network of greater depth by using the same construction for the first layer and approximating the identity function with later layers.

The 'dual' versions of the theorem consider networks of bounded width and arbitrary depth. A variant of the universal approximation theorem was proved for the arbitrary depth case by Zhou Lu et al. in 2017.[12] They showed that networks of width n+4 with ReLU activation functions can approximate any Lebesgue integrable function on n-dimensional input space with respect to ${\displaystyle L^{1}}$ distance if network depth is allowed to grow. It was also shown that there was the limited expressive power if the width was less than or equal to n. All Lebesgue integrable functions except for a zero measure set cannot be approximated by ReLU networks of width n. In the same paper[12] it was shown that ReLU networks with width n+1 were sufficient to approximate any continuous function of n-dimensional input variables.[20] The following refinement, specifies the optimal minimum width for which such an approximation is possible and is due to [21]

Universal Approximation Theorem (L1 distance, ReLU activation, arbitrary depth, minimal width). For any Bochner-Lebesgue p-integrable function ${\displaystyle f:\mathbb {R} ^{n}\rightarrow \mathbb {R} ^{m}}$ and any ${\displaystyle \epsilon >0}$, there exists a fully-connected ReLU network ${\displaystyle F}$ width exactly ${\displaystyle d_{m}=\max\{{n+1},m\}}$, satisfying

${\displaystyle \int _{\mathbb {R} ^{n}}\left\|f(x)-F_{}(x)\right\|^{p}\mathrm {d} x<\epsilon }$.

Moreover, there exists a function ${\displaystyle f\in L^{p}(\mathbb {R} ^{n},\mathbb {R} ^{m})}$ and some ${\displaystyle \epsilon >0}$, for which there is no fully-connected ReLU network of width less than ${\displaystyle d_{m}=\max\{{n+1},m\}}$satisfying the above approximation bound.

Together, the central results of [14] and of [2] yield the following general universal approximation theorem for networks with bounded width, between general input and output spaces.

Universal Approximation Theorem (non-affine activation, arbitrary depth, Non-Euclidean). Let ${\displaystyle {\mathcal {X}}}$ be a compact topological space, ${\displaystyle ({\mathcal {Y}},d_{\mathcal {Y}})}$ be a metric space, ${\displaystyle \phi$ :{\mathcal {X}}\rightarrow \mathbb {R} ^{n}} be a continuous and injective feature map and let ${\displaystyle \rho$ :\mathbb {R} ^{m}\rightarrow {\mathcal {Y}}} be a continuous readout map, with a section, having dense image ${\displaystyle Im(\rho )}$ with (possibly empty) collared boundary. Let ${\displaystyle \sigma$ :\mathbb {R} \to \mathbb {R} } be any non-affine continuous function which is continuously differentiable at at-least one point, with non-zero derivative at that point. Let ${\displaystyle {\mathcal {N}}_{\phi ,\rho }^{\sigma }}$ denote the space of feed-forward neural networks with ${\displaystyle n}$ input neurons, ${\displaystyle m}$ output neurons, and an arbitrary number of hidden layers each with ${\displaystyle n+m+2}$ neurons, such that every hidden neuron has activation function ${\displaystyle \sigma }$ and every output neuron has the identity as its activation function, with input layer ${\displaystyle \phi }$, and output layer ${\displaystyle \rho }$. Then given any ${\displaystyle \varepsilon >0}$ and any ${\displaystyle f\in C({\mathcal {X}},{\mathcal {Y}})}$, there exists ${\displaystyle F\in {\mathcal {N}}_{\phi ,\rho }^{\sigma }}$ such that

${\displaystyle \sup _{x\in {\mathcal {X}}}\,d_{\mathcal {Y}}(F(x),f(x))<\varepsilon .}$

In other words, ${\displaystyle {\mathcal {N}}}$ is dense in ${\displaystyle C({\mathcal {X}};{\mathcal {Y}})}$ with respect to the uniform distance.

Certain necessary conditions for the bounded width, arbitrary depth case have been established, but there is still a gap between the known sufficient and necessary conditions.[12][13][22]

• Kolmogorov–Arnold representation theorem
• Representer theorem
• No free lunch theorem
• Stone–Weierstrass theorem
• Fourier series