Empirical measure

Let ${\displaystyle X_{1},X_{2},\dots }$ be a sequence of independent identically distributed random variables with values in the state space S with probability distribution P.

Definition

The empirical measure P_n is defined for measurable subsets of S and given by

${\displaystyle P_{n}(A)={1 \over n}\sum _{i=1}^{n}I_{A}(X_{i})={\frac {1}{n}}\sum _{i=1}^{n}\delta _{X_{i}}(A)}$

where ${\displaystyle I_{A}}$ is the indicator function and ${\displaystyle \delta _{X}}$ is the Dirac measure.

Properties

• For a fixed measurable set A, nP_n(A) is a binomial random variable with mean nP(A) and variance nP(A)(1 − P(A)).
  • In particular, P_n(A) is an unbiased estimator of P(A).
• For a fixed partition ${\displaystyle A_{i}}$ of S, random variables ${\displaystyle X_{i}=nP_{n}(A_{i})}$ form a multinomial distribution with event probabilities ${\displaystyle P(A_{i})}$
  • The covariance matrix of this multinomial distribution is ${\displaystyle Cov(X_{i},X_{j})=nP(A_{i})(\delta _{ij}-P(A_{j}))}$.

Definition

${\displaystyle {\bigl (}P_{n}(c){\bigr )}_{c\in {\mathcal {C}}}}$ is the empirical measure indexed by ${\displaystyle {\mathcal {C}}}$, a collection of measurable subsets of S.

To generalize this notion further, observe that the empirical measure ${\displaystyle P_{n}}$ maps measurable functions ${\displaystyle f:S\to \mathbb {R} }$ to their empirical mean,

${\displaystyle f\mapsto P_{n}f=\int _{S}f\,dP_{n}={\frac {1}{n}}\sum _{i=1}^{n}f(X_{i})}$

In particular, the empirical measure of A is simply the empirical mean of the indicator function, P_n(A) = P_n I_A.

For a fixed measurable function ${\displaystyle f}$, ${\displaystyle P_{n}f}$ is a random variable with mean ${\displaystyle \mathbb {E} f}$ and variance ${\displaystyle {\frac {1}{n}}\mathbb {E} (f-\mathbb {E} f)^{2}}$.

By the strong law of large numbers, P_n(A) converges to P(A) almost surely for fixed A. Similarly ${\displaystyle P_{n}f}$ converges to ${\displaystyle \mathbb {E} f}$ almost surely for a fixed measurable function ${\displaystyle f}$. The problem of uniform convergence of P_n to P was open until Vapnik and Chervonenkis solved it in 1968.[1]

If the class ${\displaystyle {\mathcal {C}}}$ (or ${\displaystyle {\mathcal {F}}}$) is Glivenko–Cantelli with respect to P then P_n converges to P uniformly over ${\displaystyle c\in {\mathcal {C}}}$ (or ${\displaystyle f\in {\mathcal {F}}}$). In other words, with probability 1 we have

${\displaystyle \|P_{n}-P\|_{\mathcal {C}}=\sup _{c\in {\mathcal {C}}}|P_{n}(c)-P(c)|\to 0,}$

${\displaystyle \|P_{n}-P\|_{\mathcal {F}}=\sup _{f\in {\mathcal {F}}}|P_{n}f-\mathbb {E} f|\to 0.}$

The empirical distribution function provides an example of empirical measures. For real-valued iid random variables ${\displaystyle X_{1},\dots ,X_{n}}$ it is given by

${\displaystyle F_{n}(x)=P_{n}((-\infty ,x])=P_{n}I_{(-\infty ,x]}.}$

In this case, empirical measures are indexed by a class ${\displaystyle {\mathcal {C}}=\{(-\infty ,x]:x\in \mathbb {R} \}.}$ It has been shown that ${\displaystyle {\mathcal {C}}}$ is a uniform Glivenko–Cantelli class, in particular,

${\displaystyle \sup _{F}\|F_{n}(x)-F(x)\|_{\infty }\to 0}$

with probability 1.

• Poisson random measure

• Billingsley, P. (1995). Probability and Measure (Third ed.). New York: John Wiley and Sons. ISBN 0-471-80478-9.
• Donsker, M. D. (1952). "Justification and extension of Doob's heuristic approach to the Kolmogorov–Smirnov theorems". Annals of Mathematical Statistics. 23 (2): 277–281. doi:10.1214/aoms/1177729445.
• Dudley, R. M. (1978). "Central limit theorems for empirical measures". Annals of Probability. 6 (6): 899–929. doi:10.1214/aop/1176995384. JSTOR 2243028.
• Dudley, R. M. (1999). Uniform Central Limit Theorems. Cambridge Studies in Advanced Mathematics. 63. Cambridge, UK: Cambridge University Press. ISBN 0-521-46102-2.
• Wolfowitz, J. (1954). "Generalization of the theorem of Glivenko–Cantelli". Annals of Mathematical Statistics. 25 (1): 131–138. doi:10.1214/aoms/1177728852. JSTOR 2236518.