Hoeffding's lemma

In probability theory, Hoeffding's lemma is an inequality that bounds the moment-generating function of any bounded random variable.[1] It is named after the FinnishAmerican mathematical statistician Wassily Hoeffding.

The proof of Hoeffding's lemma uses Taylor's theorem and Jensen's inequality. Hoeffding's lemma is itself used in the proof of McDiarmid's inequality.

Statement of the lemma

Let X be any real-valued random variable with expected value , such that almost surely, i.e. with probability one. Then, for all ,

Note that the proof below is based on the assumption that the random variable has zero expectation (i.e. assuming that ), hence the and in the lemma must satisfy . For any random variable, which do not obey this assumption we can define , which obey the assumptions and apply the proof on .

A brief proof of the lemma


Since is a convex function of , we have

So,

Let , and

Then, since

Taking derivative of ,

for all h.

By Taylor's expansion,

Hence,

(The proof below is the same proof with more explanation.)

More detailed proof

First note that if one of or is zero, then and the inequality follows. If both are nonzero, then must be negative and must be positive.

Next, recall that is a convex function on the real line:

Applying to both sides of the above inequality gives us:

Let and define:

is well defined on , to see this we calculate:

The definition of implies

By Taylor's theorem, for every real there exists a between and such that

Note that:

Therefore,

This implies

See also

Notes


This article is issued from Wikipedia. The text is licensed under Creative Commons - Attribution - Sharealike. Additional terms may apply for the media files.