Mixture distribution

A general linear combination of probability density functions is not necessarily a probability density, since it may be negative or it may integrate to something other than 1. However, a convex combination of probability density functions preserves both of these properties (non-negativity and integrating to 1), and thus mixture densities are themselves probability density functions.

Let X₁, ..., X_n denote random variables from the n component distributions, and let X denote a random variable from the mixture distribution. Then, for any function H(·) for which ${\displaystyle \operatorname {E} [H(X_{i})]}$ exists, and assuming that the component densities p_i(x) exist,

${\displaystyle {\begin{aligned}\operatorname {E} [H(X)]&=\int _{-\infty }^{\infty }H(x)\sum _{i=1}^{n}w_{i}p_{i}(x)\,dx\\&=\sum _{i=1}^{n}w_{i}\int _{-\infty }^{\infty }p_{i}(x)H(x)\,dx=\sum _{i=1}^{n}w_{i}\operatorname {E} [H(X_{i})].\end{aligned}}}$

The jth moment about zero (i.e. choosing H(x) = x^j) is simply a weighted average of the jth moments of the components. Moments about the mean H(x) = (x − μ)^j involve a binomial expansion:[1]

${\displaystyle {\begin{aligned}\operatorname {E} [(X-\mu )^{j}]&=\sum _{i=1}^{n}w_{i}\operatorname {E} [(X-\mu _{i}+\mu _{i}-\mu )^{j}]\\&=\sum _{i=1}^{n}w_{i}\sum _{k=0}^{j}\left({\begin{array}{c}j\\k\end{array}}\right)(\mu _{i}-\mu )^{j-k}\operatorname {E} [(X-\mu _{i})^{k}],\end{aligned}}}$

where μ_i denotes the mean of the ith component.

In the case of a mixture of one-dimensional distributions with weights w_i, means μ_i and variances σ_i², the total mean and variance will be:

${\displaystyle \operatorname {E} [X]=\mu =\sum _{i=1}^{n}w_{i}\mu _{i},}$

${\displaystyle {\begin{aligned}\operatorname {E} [(X-\mu )^{2}]&=\sigma ^{2}\\&=\operatorname {E} [X^{2}]-\mu ^{2}&(\mathrm {standard} \ \mathrm {variance} \ \mathrm {reformulation} )\\&=\left(\sum _{i=1}^{n}w_{i}(\operatorname {E} [X_{i}^{2}])\right)-\mu ^{2}\\&=\left(\sum _{i=1}^{n}w_{i}(\sigma _{i}^{2}+\mu _{i}^{2})\right)-\mu ^{2}&(\mathrm {from} \ \sigma _{i}^{2}=\operatorname {E} [X_{i}^{2}]-\mu _{i}^{2},\mathrm {therefore} \ ,\operatorname {E} [X_{i}^{2}]=\sigma _{i}^{2}+\mu _{i}^{2})\\&=\sum _{i=1}^{n}w_{i}(\sigma _{i}^{2}+\mu _{i}^{2}-\mu ^{2}).\end{aligned}}}$

These relations highlight the potential of mixture distributions to display non-trivial higher-order moments such as skewness and kurtosis (fat tails) and multi-modality, even in the absence of such features within the components themselves. Marron and Wand (1992) give an illustrative account of the flexibility of this framework.[2]

The question of multimodality is simple for some cases, such as mixtures of exponential distributions: all such mixtures are unimodal.[3] However, for the case of mixtures of normal distributions, it is a complex one. Conditions for the number of modes in a multivariate normal mixture are explored by Ray & Lindsay[4] extending the earlier work on univariate [5][6] and multivariate distributions (Carreira-Perpinan and Williams, 2003[7]).

Here the problem of evaluation of the modes of an n component mixture in a D dimensional space is reduced to identification of critical points (local minima, maxima and saddle points) on a manifold referred to as the ridgeline surface, which is the image of the ridgeline function

${\displaystyle x^{*}(\alpha )=\left[\sum _{i=1}^{n}\alpha _{i}\Sigma _{i}^{-1}\right]^{-1}\times \left[\sum _{i=1}^{n}\alpha _{i}\Sigma _{i}^{-1}\mu _{i}\right],}$

where α belongs to the n − 1 dimensional unit simplex ${\displaystyle {\mathcal {S}}_{n}=\{\alpha \in \mathbb {R} ^{n}:\alpha _{i}\in [0,1],\sum _{i=1}^{n}\alpha _{i}=1\}}$ and Σ_i ∈ R^{D × D}, μ_i ∈ R^D correspond to the covariance and mean of the i^th component. Ray & Lindsay[4] consider the case in which n − 1 < D showing a one-to-one correspondence of modes of the mixture and those on the elevation function h(α) = q(x*(α)) thus one may identify the modes by solving ${\displaystyle {\frac {dh(\alpha )}{d\alpha }}=0}$ with respect to α and determining the value x*(α).

Using graphical tools, the potential multi-modality of n = {2, 3} mixtures is demonstrated; in particular it is shown that the number of modes may exceed n and that the modes may not be coincident with the component means. For two components they develop a graphical tool for analysis by instead solving the aforementioned differential with respect to w₁ and expressing the solutions as a function Π(α), α ∈ [0, 1] so that the number and location of modes for a given value of w₁ corresponds to the number of intersections of the graph on the line Π(α) = w₁. This in turn can be related to the number of oscillations of the graph and therefore to solutions of ${\displaystyle {\frac {d\Pi (\alpha )}{d\alpha }}=0}$ leading to an explicit solution for a two component homoscedastic mixture given by

${\displaystyle 1-\alpha (1-\alpha )d_{M}(\mu _{1},\mu _{2},\Sigma )^{2}}$

where d_M(μ₁, μ₂, Σ) = (μ₂ − μ₁)^TΣ⁻¹(μ₂ − μ₁) is the Mahalanobis distance.

Since the above is quadratic it follows that in this instance there are at most two modes irrespective of the dimension or the weights.

Simple examples can be given by a mixture of two normal distributions. (See Multimodal distribution#Mixture of two normal distributions for more details.)

Given an equal (50/50) mixture of two normal distributions with the same standard deviation and different means (homoscedastic), the overall distribution will exhibit low kurtosis relative to a single normal distribution – the means of the subpopulations fall on the shoulders of the overall distribution. If sufficiently separated, namely by twice the (common) standard deviation, so ${\displaystyle \left|\mu _{1}-\mu _{2}\right|>2\sigma ,}$ these form a bimodal distribution, otherwise it simply has a wide peak.[8] The variation of the overall population will also be greater than the variation of the two subpopulations (due to spread from different means), and thus exhibits overdispersion relative to a normal distribution with fixed variation ${\displaystyle \sigma ,}$ though it will not be overdispersed relative to a normal distribution with variation equal to variation of the overall population.

Alternatively, given two subpopulations with the same mean and different standard deviations, the overall population will exhibit high kurtosis, with a sharper peak and heavier tails (and correspondingly shallower shoulders) than a single distribution.

• 
  Univariate mixture distribution, showing bimodal distribution
• 
  Multivariate mixture distribution, showing four modes

The following example is adapted from Hampel,[9] who credits John Tukey.

Consider the mixture distribution defined by

F(x)   =   (1 − 10⁻¹⁰) (standard normal)  +  l0⁻¹⁰ (standard Cauchy).

The mean of i.i.d. observations from F(x) behaves "normally" except for exorbitantly large samples, although the mean of F(x) does not even exist.

• mixture (probability)
• mixture model

• graphical model
• hierarchical Bayes model