Rectifier (neural networks)

GELU is a smooth approximation to the rectifier. It has a non-monotonic “bump” when x < 0, and it serves as the default activation for models such as BERT.[13]

${\displaystyle f(x)=x\cdot \Phi (x)}$,

where Φ(x) is the cumulative distribution function of the standard normal distribution.

The SiLU (Sigmoid Linear Unit) is another smooth approximation first introduced in the GELU paper. [13]

${\displaystyle f(x)=x\cdot \operatorname {sigmoid} (x)}$

A smooth approximation to the rectifier is the analytic function

${\displaystyle f(x)=\ln(1+e^{x}),}$

which is called the softplus[14][3] or SmoothReLU function.[15] For large negative ${\displaystyle x}$ it is about ${\displaystyle e^{x}}$ so just above 0, while for large positive ${\displaystyle x}$ about ${\displaystyle x+e^{-x}}$ so just above ${\displaystyle x}$.

A sharpness parameter ${\displaystyle k}$ may be included:

${\displaystyle f(x)={\frac {\ln(1+e^{kx})}{k}}}$

The derivative of softplus is the logistic function. Starting from the parametric version,

${\displaystyle f'(x)={\frac {e^{kx}}{1+e^{kx}}}={\frac {1}{1+e^{-kx}}}}$

The logistic sigmoid function is a smooth approximation of the derivative of the rectifier, the Heaviside step function.

The multivariable generalization of single-variable softplus is the LogSumExp with the first argument set to zero:

${\displaystyle \operatorname {LSE_{0}} ^{+}(x_{1},...,x_{n}):=\operatorname {LSE} (0,x_{1},...,x_{n})=\log \left(1+e^{x_{1}}+\cdots +e^{x_{n}}\right).}$

The LogSumExp function is

${\displaystyle \operatorname {LSE} (x_{1},\dots ,x_{n})=\log \left(e^{x_{1}}+\cdots +e^{x_{n}}\right),}$

and its gradient is the softmax; the softmax with the first argument set to zero is the multivariable generalization of the logistic function. Both LogSumExp and softmax are used in machine learning.

Leaky ReLUs allow a small, positive gradient when the unit is not active.[8]

${\displaystyle f(x)={\begin{cases}x&{\text{if }}x>0,\\0.01x&{\text{otherwise}}.\end{cases}}}$

Parametric ReLUs (PReLUs) take this idea further by making the coefficient of leakage into a parameter that is learned along with the other neural-network parameters.[16]

${\displaystyle f(x)={\begin{cases}x&{\text{if }}x>0,\\ax&{\text{otherwise}}.\end{cases}}}$

Note that for a ≤ 1, this is equivalent to

${\displaystyle f(x)=\max(x,ax)}$

and thus has a relation to "maxout" networks.[16]

Exponential linear units try to make the mean activations closer to zero, which speeds up learning. It has been shown that ELUs can obtain higher classification accuracy than ReLUs.[17]

${\displaystyle f(x)={\begin{cases}x&{\text{if }}x>0,\\a(e^{x}-1)&{\text{otherwise}},\end{cases}}}$

where ${\displaystyle a}$ is a hyper-parameter to be tuned, and ${\displaystyle a\geq 0}$ is a constraint.

The ELU can be viewed as a smoothed version of a shifted ReLU (SReLU), which has the form ${\displaystyle f(x)=\max(-a,x)}$ given the same interpretation of ${\displaystyle a}$.