Loss function

Leonard J. Savage argued that using non-Bayesian methods such as minimax, the loss function should be based on the idea of regret, i.e., the loss associated with a decision should be the difference between the consequences of the best decision that could have been made had the underlying circumstances been known and the decision that was in fact taken before they were known.

The use of a quadratic loss function is common, for example when using least squares techniques. It is often more mathematically tractable than other loss functions because of the properties of variances, as well as being symmetric: an error above the target causes the same loss as the same magnitude of error below the target. If the target is t, then a quadratic loss function is

${\displaystyle \lambda (x)=C(t-x)^{2}\;}$

for some constant C; the value of the constant makes no difference to a decision, and can be ignored by setting it equal to 1.

Many common statistics, including t-tests, regression models, design of experiments, and much else, use least squares methods applied using linear regression theory, which is based on the quadratic loss function.

The quadratic loss function is also used in linear-quadratic optimal control problems. In these problems, even in the absence of uncertainty, it may not be possible to achieve the desired values of all target variables. Often loss is expressed as a quadratic form in the deviations of the variables of interest from their desired values; this approach is tractable because it results in linear first-order conditions. In the context of stochastic control, the expected value of the quadratic form is used.

In statistics and decision theory, a frequently used loss function is the 0-1 loss function

${\displaystyle L({\hat {y}},y)=I({\hat {y}}\neq y),\,}$

where ${\displaystyle I}$ is the indicator function.

Both frequentist and Bayesian statistical theory involve making a decision based on the expected value of the loss function; however, this quantity is defined differently under the two paradigms.

We first define the expected loss in the frequentist context. It is obtained by taking the expected value with respect to the probability distribution, P_θ, of the observed data, X. This is also referred to as the risk function[7][8][9][10] of the decision rule δ and the parameter θ. Here the decision rule depends on the outcome of X. The risk function is given by:

${\displaystyle R(\theta ,\delta )=\operatorname {E} _{\theta }L{\big (}\theta ,\delta (X){\big )}=\int _{X}L{\big (}\theta ,\delta (x){\big )}\,\mathrm {d} P_{\theta }(x).}$

Here, θ is a fixed but possibly unknown state of nature, X is a vector of observations stochastically drawn from a population, ${\displaystyle \operatorname {E} _{\theta }}$ is the expectation over all population values of X, dP_θ is a probability measure over the event space of X (parametrized by θ) and the integral is evaluated over the entire support of X.

In a Bayesian approach, the expectation is calculated using the posterior distribution π^* of the parameter θ:

${\displaystyle \rho (\pi ^{*},a)=\int _{\Theta }L(\theta ,a)\,\mathrm {d} \pi ^{*}(\theta ).}$

One then should choose the action a^* which minimises the expected loss. Although this will result in choosing the same action as would be chosen using the frequentist risk, the emphasis of the Bayesian approach is that one is only interested in choosing the optimal action under the actual observed data, whereas choosing the actual frequentist optimal decision rule, which is a function of all possible observations, is a much more difficult problem.

• For a scalar parameter θ, a decision function whose output ${\displaystyle {\hat {\theta }}}$ is an estimate of θ, and a quadratic loss function (squared error loss)

${\displaystyle L(\theta ,{\hat {\theta }})=(\theta -{\hat {\theta }})^{2},}$

the risk function becomes the mean squared error of the estimate,

${\displaystyle R(\theta ,{\hat {\theta }})=\operatorname {E} _{\theta }(\theta -{\hat {\theta }})^{2}.}$

• In density estimation, the unknown parameter is probability density itself. The loss function is typically chosen to be a norm in an appropriate function space. For example, for L² norm,

${\displaystyle L(f,{\hat {f}})=\|f-{\hat {f}}\|_{2}^{2}\,,}$

the risk function becomes the mean integrated squared error

${\displaystyle R(f,{\hat {f}})=\operatorname {E} \|f-{\hat {f}}\|^{2}.\,}$

In economics, decision-making under uncertainty is often modelled using the von Neumann–Morgenstern utility function of the uncertain variable of interest, such as end-of-period wealth. Since the value of this variable is uncertain, so is the value of the utility function; it is the expected value of utility that is maximized.