Maximum entropy probability distribution

Suppose S is a closed subset of the real numbers R and we choose to specify n measurable functions f₁,...,f_n and n numbers a₁,...,a_n. We consider the class C of all real-valued random variables which are supported on S (i.e. whose density function is zero outside of S) and which satisfy the n moment conditions:

${\displaystyle \operatorname {E} (f_{j}(X))\geq a_{j}\quad {\mbox{ for }}j=1,\ldots ,n}$

If there is a member in C whose density function is positive everywhere in S, and if there exists a maximal entropy distribution for C, then its probability density p(x) has the following shape:

${\displaystyle p(x)=\exp \left(\sum _{j=0}^{n}\lambda _{j}f_{j}(x)\right)\quad {\mbox{ for all }}x\in S}$

where we assume that ${\displaystyle f_{0}(x)=1}$. The constant ${\displaystyle \lambda _{0}}$ and the n Lagrange multipliers ${\displaystyle {\boldsymbol {\lambda }}=(\lambda _{1},\ldots ,\lambda _{n})}$ solve the constrained optimization problem with ${\displaystyle a_{0}=1}$ (this condition ensures that ${\displaystyle p}$ integrates to unity): [4]

${\displaystyle \max _{\lambda _{0};{\boldsymbol {\lambda }}}\left\{\sum _{j=0}^{n}\lambda _{j}a_{j}-\int \exp \left(\sum _{j=0}^{n}\lambda _{j}f_{j}(x)\right)dx\right\}\quad \mathrm {subject\;to:\;\;} {\boldsymbol {\lambda }}\geq \mathbf {0} }$

Using the Karush–Kuhn–Tucker conditions, it can be shown that the optimization problem has a unique solution, because the objective function in the optimization is concave in ${\displaystyle {\boldsymbol {\lambda }}}$.

Note that if the moment conditions are equalities (instead of inequalities), that is,

${\displaystyle \operatorname {E} (f_{j}(X))=a_{j}\quad {\mbox{ for }}j=1,\ldots ,n,}$

then the constraint condition ${\displaystyle {\boldsymbol {\lambda }}\geq \mathbf {0} }$ is dropped, making the optimization over the Lagrange multipliers unconstrained.

Suppose S = {x₁,x₂,...} is a (finite or infinite) discrete subset of the reals and we choose to specify n functions f₁,...,f_n and n numbers a₁,...,a_n. We consider the class C of all discrete random variables X which are supported on S and which satisfy the n moment conditions

${\displaystyle \operatorname {E} (f_{j}(X))\geq a_{j}\quad {\mbox{ for }}j=1,\ldots ,n}$

If there exists a member of C which assigns positive probability to all members of S and if there exists a maximum entropy distribution for C, then this distribution has the following shape:

${\displaystyle \operatorname {Pr} (X=x_{k})=\exp \left(\sum _{j=0}^{n}\lambda _{j}f_{j}(x_{k})\right)\quad {\mbox{ for }}k=1,2,\ldots }$

where we assume that ${\displaystyle f_{0}=1}$ and the constants ${\displaystyle \lambda _{0},\;{\boldsymbol {\lambda }}=(\lambda _{1},\ldots ,\lambda _{n})}$ solve the constrained optimization problem with ${\displaystyle a_{0}=1}$:[5]

${\displaystyle \max _{\lambda _{0};{\boldsymbol {\lambda }}}\left\{\sum _{j=0}^{n}\lambda _{j}a_{j}-\sum _{k\geq 1}\exp \left(\sum _{j=0}^{n}\lambda _{j}f_{j}(x_{k})\right)\right\}\quad \mathrm {subject\;to:\;\;} {\boldsymbol {\lambda }}\geq \mathbf {0} }$

Again, if the moment conditions are equalities (instead of inequalities), then the constraint condition ${\displaystyle {\boldsymbol {\lambda }}\geq \mathbf {0} }$ is not present in the optimization.

In the case of equality constraints, this theorem is proved with the calculus of variations and Lagrange multipliers. The constraints can be written as

${\displaystyle \int _{-\infty }^{\infty }f_{j}(x)p(x)dx=a_{j}}$

We consider the functional

${\displaystyle J(p)=\int _{-\infty }^{\infty }p(x)\ln {p(x)}dx-\eta _{0}\left(\int _{-\infty }^{\infty }p(x)dx-1\right)-\sum _{j=1}^{n}\lambda _{j}\left(\int _{-\infty }^{\infty }f_{j}(x)p(x)dx-a_{j}\right)}$

where ${\displaystyle \eta _{0}}$ and ${\displaystyle \lambda _{j},j\geq 1}$ are the Lagrange multipliers. The zeroth constraint ensures the second axiom of probability. The other constraints are that the measurements of the function are given constants up to order ${\displaystyle n}$. The entropy attains an extremum when the functional derivative is equal to zero:

${\displaystyle {\frac {\delta J}{\delta p}}\left(p\right)=\ln {p(x)}+1-\eta _{0}-\sum _{j=1}^{n}\lambda _{j}f_{j}(x)=0}$

It is an exercise for the reader that this extremum is indeed a maximum. Therefore, the maximum entropy probability distribution in this case must be of the form (${\displaystyle \lambda _{0}:=\eta _{0}-1}$)

${\displaystyle p(x)=e^{-1+\eta _{0}}\cdot e^{\sum _{j=1}^{n}\lambda _{j}f_{j}(x)}=\exp \left(\sum _{j=0}^{n}\lambda _{j}f_{j}(x)\right)\;.}$

The proof of the discrete version is essentially the same.

Suppose ${\displaystyle p}$, ${\displaystyle p'}$ are distributions satisfying the expectation-constraints. Letting ${\displaystyle \alpha \in (0,1)}$ and considering the distribution ${\displaystyle q=\alpha \cdot p+(1-\alpha )\cdot p'}$ it is clear that this distribution satisfies the expectation-constraints and furthermore has as support ${\displaystyle \mathrm {supp} (q)=\mathrm {supp} (p)\cup \mathrm {supp} (p')}$. From basic facts about entropy, it holds that ${\displaystyle {\mathcal {H}}(q)\geq \alpha {\mathcal {H}}(p)+(1-\alpha ){\mathcal {H}}(p')}$. Taking limits ${\displaystyle \alpha \longrightarrow 1}$ and ${\displaystyle \alpha \longrightarrow 0}$ respectively yields ${\displaystyle {\mathcal {H}}(q)\geq {\mathcal {H}}(p),{\mathcal {H}}(p')}$.

It follows that a distribution satisfying the expectation-constraints and maximising entropy must necessarily have full support — i. e. the distribution is almost everywhere positive. It follows that the maximising distribution must be an internal point in the space of distributions satisfying the expectation-constraints, that is, it must be a local extreme. Thus it suffices to show that the local extreme is unique, in order to show both that the entropy-maximising distribution is unique (and this also shows that the local extreme is the global maximum).

Suppose ${\displaystyle p,p'}$ are local extremes. Reformulating the above computations these are characterised by parameters ${\displaystyle {\vec {\lambda }},{\vec {\lambda }}'\in \mathbb {R} ^{n}}$ via ${\displaystyle p(x)={\frac {e^{\langle {\vec {\lambda }},{\vec {f}}(x)\rangle }}{C({\vec {\lambda }})}}}$ and similarly for ${\displaystyle p'}$, where ${\displaystyle C({\vec {\lambda }})=\int _{x\in \mathbb {R} }e^{\langle {\vec {\lambda }},{\vec {f}}(x)\rangle }~dx}$. We now note a series of identities: Via the satisfaction of the expectation-constraints and utilising gradients/directional derivatives, one has ${\displaystyle D\log(C(\cdot ))\vert _{\vec {\lambda }}=\left.{\frac {DC(\cdot )}{C(\cdot )}}\right|_{\vec {\lambda }}=\mathbb {E} _{p}[{\vec {f}}(X)]={\vec {a}}}$ and similarly for ${\displaystyle {\vec {\lambda }}'}$. Letting ${\displaystyle u={\vec {\lambda }}'-{\vec {\lambda }}\in \mathbb {R} ^{n}}$ one obtains:

${\displaystyle 0=\langle u,{\vec {a}}-{\vec {a}}\rangle =D_{u}\log(C(\cdot ))\vert _{{\vec {\lambda }}'}-D_{u}\log(C(\cdot ))\vert _{\vec {\lambda }}=D_{u}^{2}\log(C(\cdot ))\vert _{\vec {\gamma }}}$

where ${\displaystyle {\vec {\gamma }}=\theta {\vec {\lambda }}+(1-\theta ){\vec {\lambda }}'}$ for some ${\displaystyle \theta \in (0,1)}$. Computing further one has

${\displaystyle {\begin{array}{rcl}0&=&D_{u}^{2}\log(C(\cdot ))\vert _{\vec {\gamma }}\\&=&\left.D_{u}\left({\frac {D_{u}C(\cdot )}{C(\cdot )}}\right)\right|_{\vec {\gamma }}\\&=&\left.{\frac {D_{u}^{2}C(\cdot )}{C(\cdot )}}\right|_{\vec {\gamma }}-\left.{\frac {(D_{u}C(\cdot ))^{2}}{C(\cdot )^{2}}}\right|_{\vec {\gamma }}\\&=&\mathbb {E} _{q}[(\langle u,{\vec {f}}(X)\rangle )^{2}]-\left(\mathbb {E} _{q}[\langle u,{\vec {f}}(X)\rangle ]\right)^{2}=\mathrm {Var} _{q}(\langle u,{\vec {f}}(X)\rangle )\\\end{array}}}$

where ${\displaystyle q}$ is similar to the distribution above, only parameterised by ${\displaystyle {\vec {\gamma }}}$. Assuming that no non-trivial linear combination of the observables is almost everywhere (a.e.) constant, (which e.g. holds if the observables are independent and not a.e. constant), it holds that ${\displaystyle \langle u,{\vec {f}}(X)\rangle }$ has non-zero variance, unless ${\displaystyle u=0}$. By the above equation it is thus clear, that the latter must be the case. Hence ${\displaystyle {\vec {\lambda }}'-{\vec {\lambda }}=u=0}$, so the parameters characterising the local extrema ${\displaystyle p,p'}$ are identical, which means that the distributions themselves are identical. Thus, the local extreme is unique and by the above discussion, the maximum is unique—provided a local extreme actually exists.

Note that not all classes of distributions contain a maximum entropy distribution. It is possible that a class contain distributions of arbitrarily large entropy (e.g. the class of all continuous distributions on R with mean 0 but arbitrary standard deviation), or that the entropies are bounded above but there is no distribution which attains the maximal entropy.[lower-alpha 1] It is also possible that the expected value restrictions for the class C force the probability distribution to be zero in certain subsets of S. In that case our theorem doesn't apply, but one can work around this by shrinking the set S.

The uniform distribution on the interval [a,b] is the maximum entropy distribution among all continuous distributions which are supported in the interval [a, b], and thus the probability density is 0 outside of the interval. This uniform density can be related to Laplace's principle of indifference, sometimes called the principle of insufficient reason. More generally, if we're given a subdivision a=a₀ < a₁ < ... < a_k = b of the interval [a,b] and probabilities p₁,...,p_k which add up to one, then we can consider the class of all continuous distributions such that

${\displaystyle \operatorname {Pr} (a_{j-1}\leq X<a_{j})=p_{j}\quad {\mbox{ for }}j=1,\ldots ,k}$

The density of the maximum entropy distribution for this class is constant on each of the intervals [a_j-1,a_j). The uniform distribution on the finite set {x₁,...,x_n} (which assigns a probability of 1/n to each of these values) is the maximum entropy distribution among all discrete distributions supported on this set.

The exponential distribution, for which the density function is

${\displaystyle p(x|\lambda )={\begin{cases}\lambda e^{-\lambda x}&x\geq 0,\\0&x<0,\end{cases}}}$

is the maximum entropy distribution among all continuous distributions supported in [0,∞) that have a specified mean of 1/λ.

The normal distribution N(μ,σ²), for which the density function is

${\displaystyle p(x|\mu ,\sigma )={\frac {1}{\sigma {\sqrt {2\pi }}}}e^{-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}},}$

has maximum entropy among all real-valued distributions supported on (−∞,∞) with a specified variance σ² (a particular moment). Therefore, the assumption of normality imposes the minimal prior structural constraint beyond this moment. (See the differential entropy article for a derivation.)

In the case of distributions supported on [0,∞), the maximum entropy distribution depends on relationships between the first and second moments. In specific cases, it may be the exponential distribution, or may be another distribution, or may be undefinable.[8]

Among all the discrete distributions supported on the set {x₁,...,x_n} with a specified mean μ, the maximum entropy distribution has the following shape:

${\displaystyle \operatorname {Pr} (X=x_{k})=Cr^{x_{k}}\quad {\mbox{ for }}k=1,\ldots ,n}$

where the positive constants C and r can be determined by the requirements that the sum of all the probabilities must be 1 and the expected value must be μ.

For example, if a large number N of dice are thrown, and you are told that the sum of all the shown numbers is S. Based on this information alone, what would be a reasonable assumption for the number of dice showing 1, 2, ..., 6? This is an instance of the situation considered above, with {x₁,...,x₆} = {1,...,6} and μ = S/N.

Finally, among all the discrete distributions supported on the infinite set {x₁,x₂,...} with mean μ, the maximum entropy distribution has the shape:

${\displaystyle \operatorname {Pr} (X=x_{k})=Cr^{x_{k}}\quad {\mbox{ for }}k=1,2,\ldots ,}$

where again the constants C and r were determined by the requirements that the sum of all the probabilities must be 1 and the expected value must be μ. For example, in the case that x_k = k, this gives

${\displaystyle C={\frac {1}{\mu -1}},\quad \quad r={\frac {\mu -1}{\mu }},}$

such that respective maximum entropy distribution is the geometric distribution.

For a continuous random variable ${\displaystyle \theta _{i}}$ distributed about the unit circle, the Von Mises distribution maximizes the entropy when the real and imaginary parts of the first circular moment are specified[9] or, equivalently, the circular mean and circular variance are specified.

When the mean and variance of the angles ${\displaystyle \theta _{i}}$ modulo ${\displaystyle 2\pi }$ are specified, the wrapped normal distribution maximizes the entropy.[9]

There exists an upper bound on the entropy of continuous random variables on ${\displaystyle \mathbb {R} }$ with a specified mean, variance, and skew. However, there is no distribution which achieves this upper bound, because ${\displaystyle p(x)=c\exp {(\lambda _{1}x+\lambda _{2}x^{2}+\lambda _{3}x^{3})}}$ is unbounded except when ${\displaystyle \lambda _{3}=0}$ (see Cover & Thomas (2006: chapter 12)).

However, the maximum entropy is ε-achievable: a distribution's entropy can be arbitrarily close to the upper bound. Start with a normal distribution of the specified mean and variance. To introduce a positive skew, perturb the normal distribution upward by a small amount at a value many σ larger than the mean. The skewness, being proportional to the third moment, will be affected more than the lower order moments.

Every distribution with log-concave density is a maximal entropy distribution with specified mean μ and Deviation risk measure D.[10]

In particular, the maximal entropy distribution with specified mean ${\displaystyle E(x)=\mu }$ and deviation ${\displaystyle D(x)=d}$ is:

• The normal distribution ${\displaystyle N(m,d^{2})}$, if ${\displaystyle D(x)={\sqrt {E[(x-\mu )^{2}]}}}$ is the standard deviation;
• The Laplace distribution, if ${\displaystyle D(x)=E(|x-\mu |)}$ is the average absolute deviation;[6]
• The distribution with density of the form ${\displaystyle f(x)=c\exp(ax+b{[x-\mu ]_{-}}^{2})}$ if ${\displaystyle D(x)={\sqrt {E[{(x-\mu )_{-}}^{2}]}}}$ is the standard lower semi-deviation, where ${\displaystyle [x]_{-}:=\max\{0,-x\}}$, and a,b,c are constants.[10]

In the table below, each listed distribution maximizes the entropy for a particular set of functional constraints listed in the third column, and the constraint that x be included in the support of the probability density, which is listed in the fourth column.[6][7] Several examples (Bernoulli, geometric, exponential, Laplace, Pareto) listed are trivially true because their associated constraints are equivalent to the assignment of their entropy. They are included anyway because their constraint is related to a common or easily measured quantity. For reference, ${\displaystyle \Gamma (x)=\int _{0}^{\infty }e^{-t}t^{x-1}dt}$ is the gamma function, ${\displaystyle \psi (x)={\frac {d}{dx}}\ln \Gamma (x)={\frac {\Gamma '(x)}{\Gamma (x)}}}$ is the digamma function, ${\displaystyle B(p,q)={\frac {\Gamma (p)\Gamma (q)}{\Gamma (p+q)}}}$ is the beta function, and γ_E is the Euler-Mascheroni constant.

Table of probability distributions and corresponding maximum entropy constraints

Distribution Name	Probability density/mass function	Maximum Entropy Constraint	Support
Uniform (discrete)	${\displaystyle f(k)={\frac {1}{b-a+1}}}$	None	${\displaystyle \{a,a+1,...,b-1,b\}\,}$
Uniform (continuous)	${\displaystyle f(x)={\frac {1}{b-a}}}$	None	${\displaystyle [a,b]\,}$
Bernoulli	${\displaystyle f(k)=p^{k}(1-p)^{1-k}}$	${\displaystyle \operatorname {E} (k)=p\,}$	${\displaystyle \{0,1\}\,}$
Geometric	${\displaystyle f(k)=(1-p)^{k-1}\,p}$	${\displaystyle \operatorname {E} (k)={\frac {1}{p}}\,}$	${\displaystyle \mathbb {N} \setminus \left\{0\right\}=\{1,2,3,...\}}$
Exponential	${\displaystyle f(x)=\lambda \exp \left(-\lambda x\right)}$	${\displaystyle \operatorname {E} (x)={\frac {1}{\lambda }}\,}$	${\displaystyle [0,\infty )\,}$
Laplace	${\displaystyle f(x)={\frac {1}{2b}}\exp \left(-{\frac {|x-\mu |}{b}}\right)}$	${\displaystyle \operatorname {E} (|x-\mu |)=b\,}$	${\displaystyle (-\infty ,\infty )\,}$
Asymmetric Laplace	${\displaystyle f(x)={\frac {\lambda \,e^{-(x-m)\lambda s\kappa ^{s}}}{\kappa +1/\kappa }}\,(s\!=\!\operatorname {sgn}(x\!-\!m))}$	${\displaystyle \operatorname {E} ((x-m)s\kappa ^{s})=1/\lambda \,}$	${\displaystyle (-\infty ,\infty )\,}$
Pareto	${\displaystyle f(x)={\frac {\alpha x_{m}^{\alpha }}{x^{\alpha +1}}}}$	${\displaystyle \operatorname {E} (\ln(x))={\frac {1}{\alpha }}+\ln(x_{m})\,}$	${\displaystyle [x_{m},\infty )\,}$
Normal	${\displaystyle f(x)={\frac {1}{\sqrt {2\pi \sigma ^{2}}}}\exp \left(-{\frac {(x-\mu )^{2}}{2\sigma ^{2}}}\right)}$	${\displaystyle \operatorname {E} (x)=\mu ,\,\operatorname {E} ((x-\mu )^{2})=\sigma ^{2}}$	${\displaystyle (-\infty ,\infty )\,}$
Truncated normal	(see article)	${\displaystyle \operatorname {E} (x)=\mu _{T},\,\operatorname {E} ((x-\mu _{T})^{2})=\sigma _{T}^{2}}$	${\displaystyle [a,b]}$
von Mises	${\displaystyle f(\theta )={\frac {1}{2\pi I_{0}(\kappa )}}\exp {(\kappa \cos {(\theta -\mu )})}}$	${\displaystyle \operatorname {E} (\cos \theta )={\frac {I_{1}(\kappa )}{I_{0}(\kappa )}}\cos \mu ,\,\operatorname {E} (\sin \theta )={\frac {I_{1}(\kappa )}{I_{0}(\kappa )}}\sin \mu }$	${\displaystyle [0,2\pi )\,}$
Rayleigh	${\displaystyle f(x)={\frac {x}{\sigma ^{2}}}\exp \left(-{\frac {x^{2}}{2\sigma ^{2}}}\right)}$	${\displaystyle \operatorname {E} (x^{2})=2\sigma ^{2},\operatorname {E} (\ln(x))={\frac {\ln(2\sigma ^{2})-\gamma _{\mathrm {E} }}{2}}\,}$	${\displaystyle [0,\infty )\,}$
Beta	${\displaystyle f(x)={\frac {x^{\alpha -1}(1-x)^{\beta -1}}{B(\alpha ,\beta )}}}$ for ${\displaystyle 0\leq x\leq 1}$	${\displaystyle \operatorname {E} (\ln(x))=\psi (\alpha )-\psi (\alpha +\beta )\,}$ ${\displaystyle \operatorname {E} (\ln(1-x))=\psi (\beta )-\psi (\alpha +\beta )\,}$	${\displaystyle [0,1]\,}$
Cauchy	${\displaystyle f(x)={\frac {1}{\pi (1+x^{2})}}}$	${\displaystyle \operatorname {E} (\ln(1+x^{2}))=2\ln 2}$	${\displaystyle (-\infty ,\infty )\,}$
Chi	${\displaystyle f(x)={\frac {2}{2^{k/2}\Gamma (k/2)}}x^{k-1}\exp \left(-{\frac {x^{2}}{2}}\right)}$	${\displaystyle \operatorname {E} (x^{2})=k,\,\operatorname {E} (\ln(x))={\frac {1}{2}}\left[\psi \left({\frac {k}{2}}\right)\!+\!\ln(2)\right]}$	${\displaystyle [0,\infty )\,}$
Chi-squared	${\displaystyle f(x)={\frac {1}{2^{k/2}\Gamma (k/2)}}x^{{\frac {k}{2}}\!-\!1}\exp \left(-{\frac {x}{2}}\right)}$	${\displaystyle \operatorname {E} (x)=k,\,\operatorname {E} (\ln(x))=\psi \left({\frac {k}{2}}\right)+\ln(2)}$	${\displaystyle [0,\infty )\,}$
Erlang	${\displaystyle f(x)={\frac {\lambda ^{k}}{(k-1)!}}x^{k-1}\exp(-\lambda x)}$	${\displaystyle \operatorname {E} (x)=k/\lambda ,\,\operatorname {E} (\ln(x))=\psi (k)-\ln(\lambda )}$	${\displaystyle [0,\infty )\,}$
Gamma	${\displaystyle f(x)={\frac {x^{k-1}\exp(-{\frac {x}{\theta }})}{\theta ^{k}\Gamma (k)}}}$	${\displaystyle \operatorname {E} (x)=k\theta ,\,\operatorname {E} (\ln(x))=\psi (k)+\ln(\theta )}$	${\displaystyle [0,\infty )\,}$
Lognormal	${\displaystyle f(x)={\frac {1}{\sigma x{\sqrt {2\pi }}}}\exp \left(-{\frac {(\ln x-\mu )^{2}}{2\sigma ^{2}}}\right)}$	${\displaystyle \operatorname {E} (\ln(x))=\mu ,\operatorname {E} ((\ln(x)-\mu )^{2})=\sigma ^{2}\,}$	${\displaystyle [0,\infty )\,}$
Maxwell–Boltzmann	${\displaystyle f(x)={\frac {1}{a^{3}}}{\sqrt {\frac {2}{\pi }}}\,x^{2}\exp \left(-{\frac {x^{2}}{2a^{2}}}\right)}$	${\displaystyle \operatorname {E} (x^{2})=3a^{2},\,\operatorname {E} (\ln(x))\!=\!1\!+\!\ln \left({\frac {a}{\sqrt {2}}}\right)\!-\!{\frac {\gamma _{\mathrm {E} }}{2}}}$	${\displaystyle [0,\infty )\,}$
Weibull	${\displaystyle f(x)={\frac {k}{\lambda ^{k}}}x^{k-1}\exp \left(-{\frac {x^{k}}{\lambda ^{k}}}\right)}$	${\displaystyle \operatorname {E} (x^{k})=\lambda ^{k},\operatorname {E} (\ln(x))=\ln(\lambda )-{\frac {\gamma _{\mathrm {E} }}{k}}\,}$	${\displaystyle [0,\infty )\,}$
Multivariate normal	${\displaystyle f_{X}({\vec {x}})=}$ ${\displaystyle {\frac {\exp \left(-{\frac {1}{2}}({\vec {x}}-{\vec {\mu }})^{\top }\Sigma ^{-1}\cdot ({\vec {x}}-{\vec {\mu }})\right)}{(2\pi )^{N/2}\left|\Sigma \right|^{1/2}}}}$	${\displaystyle \operatorname {E} ({\vec {x}})={\vec {\mu }},\,\operatorname {E} (({\vec {x}}-{\vec {\mu }})({\vec {x}}-{\vec {\mu }})^{T})=\Sigma \,}$	${\displaystyle \mathbb {R} ^{n}}$
Binomial	${\displaystyle f(k)={n \choose k}p^{k}(1-p)^{n-k}}$	${\displaystyle \operatorname {E} (x)=\mu ,f\in {\text{n-generalized binomial distribution}}}$[11]	${\displaystyle \left\{0,{\ldots },n\right\}}$
Poisson	${\displaystyle f(k)={\frac {\lambda ^{k}\exp(-\lambda )}{k!}}}$	${\displaystyle \operatorname {E} (x)=\lambda ,f\in {\infty }{\text{-generalized binomial distribution}}}$[11]	${\displaystyle \mathbb {N} \cup \left\{0\right\}}$