Information content

For a given probability space, the measurement of rarer events are intuitively more "surprising," and yield more information content, than more common values. Thus, self-information is a strictly decreasing monotonic function of the probability, or sometimes called an "antitonic" function.

While standard probabilities are represented by real numbers in the interval ${\displaystyle [0,1]}$, self-informations are represented by extended real numbers in the interval ${\displaystyle [0,\infty ]}$. In particular, we have the following, for any choice of logarithmic base:

• If a particular event has a 100% probability of occurring, then its self-information is ${\displaystyle -\log(1)=0}$: its occurrence is "perfectly non-surprising" and yields no information.
• If a particular event has a 0% probability of occurring, then its self-information is ${\displaystyle -\log(0)=\infty }$: its occurrence is "infinitely surprising."

From this, we can get a few general properties:

• Intuitively, more information is gained from observing an unexpected event—it is "surprising".
  • For example, if there is a one-in-a-million chance of Alice winning the lottery, her friend Bob will gain significantly more information from learning that she won than that she lost on a given day. (See also: Lottery mathematics.)
• This establishes an implicit relationship between the self-information of a random variable and its variance.

The Shannon information is closely related to the log-odds. In particular, given some event ${\displaystyle x}$, suppose that ${\displaystyle p(x)}$ is the probability of ${\displaystyle x}$ occurring, and that ${\displaystyle p(\lnot x)=1-p(x)}$ is the probability of ${\displaystyle x}$ not occurring. Then we have the following definition of the log-odds:

${\displaystyle {\text{log-odds}}(x)=\log \left({\frac {p(x)}{p(\lnot x)}}\right)}$

This can be expressed as a difference of two Shannon informations:

${\displaystyle {\text{log-odds}}(x)=I(\lnot x)-I(x)}$

In other words, the log-odds can be interpreted as the level of surprise if the event 'doesn't' happen, minus the level of surprise if the event 'does' happen.

The information content of two independent events is the sum of each event's information content. This property is known as additivity in mathematics, and sigma additivity in particular in measure and probability theory. Consider two independent random variables ${\textstyle X,\,Y}$ with probability mass functions ${\displaystyle p_{X}(x)}$ and ${\displaystyle p_{Y}(y)}$ respectively. The joint probability mass function is

$${\displaystyle p_{X,Y}\!\left(x,y\right)=\Pr(X=x,\,Y=y)=p_{X}\!(x)\,p_{Y}\!(y)}$$

because ${\textstyle X}$ and ${\textstyle Y}$ are independent. The information content of the outcome ${\displaystyle (X,Y)=(x,y)}$ is

$${\displaystyle {\begin{aligned}\operatorname {I} _{X,Y}(x,y)&=-\log _{2}\left[p_{X,Y}(x,y)\right]=-\log _{2}\left[p_{X}\!(x)p_{Y}\!(y)\right]\\&=-\log _{2}\left[p_{X}{(x)}\right]-\log _{2}\left[p_{Y}{(y)}\right]\\&=\operatorname {I} _{X}(x)+\operatorname {I} _{Y}(y)\end{aligned}}}$$

See § Two independent, identically distributed dice below for an example.

The corresponding property for likelihoods is that the log-likelihood of independent events is the sum of the log-likelihoods of each event. Interpreting log-likelihood as "support" or negative surprisal (the degree to which an event supports a given model: a model is supported by an event to the extent that the event is unsurprising, given the model), this states that independent events add support: the information that the two events together provide for statistical inference is the sum of their independent information.

Consider the Bernoulli trial of tossing a fair coin ${\displaystyle X}$. The probabilities of the events of the coin landing as heads ${\displaystyle H}$ and tails ${\displaystyle T}$ (see fair coin and obverse and reverse) are one half each, ${\textstyle p_{X}{(H)}=p_{X}{(T)}={\tfrac {1}{2}}=0.5}$. Upon measuring the variable as heads, the associated information gain is

$${\displaystyle \operatorname {I} _{X}(H)=-\log _{2}{p_{X}{(H)}}=-\log _{2}\!{\tfrac {1}{2}}=1,}$$

so the information gain of a fair coin landing as heads is 1 shannon.[2] Likewise, the information gain of measuring ${\displaystyle T}$ tails is

$${\displaystyle \operatorname {I} _{X}(T)=-\log _{2}{p_{X}{(T)}}=-\log _{2}\!{\tfrac {1}{2}}=1{\text{ shannon}}.}$$

Suppose we have a fair six-sided die. The value of a dice roll is a discrete uniform random variable ${\displaystyle X\sim \mathrm {DU} [1,6]}$ with probability mass function

$${\displaystyle p_{X}(k)={\begin{cases}{\frac {1}{6}},&k\in \{1,2,3,4,5,6\}\\0,&{\text{otherwise}}\end{cases}}}$$

The probability of rolling a 4 is ${\textstyle p_{X}(4)={\frac {1}{6}}}$, as for any other valid roll. The information content of rolling a 4 is thus

$${\displaystyle \operatorname {I} _{X}(4)=-\log _{2}{p_{X}{(4)}}=-\log _{2}{\tfrac {1}{6}}\approx 2.585\;{\text{shannons}}}$$

of information.

Suppose we have two independent, identically distributed random variables ${\textstyle X,\,Y\sim \mathrm {DU} [1,6]}$ each corresponding to an independent fair 6-sided dice roll. The joint distribution of ${\displaystyle X}$ and ${\displaystyle Y}$ is

$${\displaystyle {\begin{aligned}p_{X,Y}\!\left(x,y\right)&{}=\Pr(X=x,\,Y=y)=p_{X}\!(x)\,p_{Y}\!(y)\\&{}={\begin{cases}\displaystyle {1 \over 36},\ &x,y\in [1,6]\cap \mathbb {N} \\0&{\text{otherwise.}}\end{cases}}\end{aligned}}}$$

The information content of the random variate ${\displaystyle (X,Y)=(2,\,4)}$ is

$${\displaystyle {\begin{aligned}\operatorname {I} _{X,Y}{(2,4)}&=-\log _{2}\!{\left[p_{X,Y}{(2,4)}\right]}=\log _{2}\!{36}=2\log _{2}\!{6}\\&\approx 5.169925{\text{ shannons}},\end{aligned}}}$$

just as

$${\displaystyle {\begin{aligned}\operatorname {I} _{X,Y}{(2,4)}&=-\log _{2}\!{\left[p_{X,Y}{(2,4)}\right]}=-\log _{2}\!{\left[p_{X}(2)\right]}-\log _{2}\!{\left[p_{Y}(4)\right]}\\&=2\log _{2}\!{6}\\&\approx 5.169925{\text{ shannons}},\end{aligned}}}$$

as explained in § Additivity of independent events.

If we receive information about the value of the dice without knowledge of which die had which value, we can formalize the approach with so-called counting variables

$${\displaystyle C_{k}:=\delta _{k}(X)+\delta _{k}(Y)={\begin{cases}0,&\neg \,(X=k\vee Y=k)\\1,&\quad X=k\,\veebar \,Y=k\\2,&\quad X=k\,\wedge \,Y=k\end{cases}}}$$

for ${\displaystyle k\in \{1,2,3,4,5,6\}}$, then ${\textstyle \sum _{k=1}^{6}{C_{k}}=2}$ and the counts have the multinomial distribution

$${\displaystyle {\begin{aligned}f(c_{1},\ldots ,c_{6})&{}=\Pr(C_{1}=c_{1}{\text{ and }}\dots {\text{ and }}C_{6}=c_{6})\\&{}={\begin{cases}{\displaystyle {1 \over {18}}{1 \over c_{1}!\cdots c_{k}!}},\ &{\text{when }}\sum _{i=1}^{6}c_{i}=2\\0&{\text{otherwise,}}\end{cases}}\\&{}={\begin{cases}{1 \over 18},\ &{\text{when 2 }}c_{k}{\text{ are }}1\\{1 \over 36},\ &{\text{when exactly one }}c_{k}=2\\0,\ &{\text{otherwise.}}\end{cases}}\end{aligned}}}$$

To verify this, the 6 outcomes ${\textstyle (X,Y)\in \left\{(k,k)\right\}_{k=1}^{6}=\left\{(1,1),(2,2),(3,3),(4,4),(5,5),(6,6)\right\}}$ correspond to the event ${\displaystyle C_{k}=2}$ and a total probability of 1/6. These are the only events that are faithfully preserved with identity of which dice rolled which outcome because the outcomes are the same. Without knowledge to distinguish the dice rolling the other numbers, the other ${\textstyle {\binom {6}{2}}=15}$ combinations correspond to one die rolling one number and the other die rolling a different number, each having probability 1/18. Indeed, ${\textstyle 6\cdot {\tfrac {1}{36}}+15\cdot {\tfrac {1}{18}}=1}$, as required.

Unsurprisingly, the information content of learning that both dice were rolled as the same particular number is more than the information content of learning that one dice was one number and the other was a different number. Take for examples the events ${\displaystyle A_{k}=\{(X,Y)=(k,k)\}}$ and ${\displaystyle B_{j,k}=\{c_{j}=1\}\cap \{c_{k}=1\}}$for ${\displaystyle j\neq k,1\leq j,k\leq 6}$. For example, ${\displaystyle A_{2}=\{X=2{\text{ and }}Y=2\}}$and ${\displaystyle B_{3,4}=\{(3,4),(4,3)\}}$.

The information contents are

$${\displaystyle \operatorname {I} (A_{2})=-\log _{2}\!{\tfrac {1}{36}}=5.169925{\text{ shannons}}}$$

$${\displaystyle \operatorname {I} \left(B_{3,4}\right)=-\log _{2}\!{\tfrac {1}{18}}=4.169925{\text{ shannons}}}$$

Let ${\textstyle Same=\bigcup _{i=1}^{6}{A_{i}}}$ be the event that both dice rolled the same value and ${\displaystyle Diff={\overline {Same}}}$ be the event that the dice differed. Then ${\textstyle \Pr(Same)={\tfrac {1}{6}}}$ and ${\textstyle \Pr(Diff)={\tfrac {5}{6}}}$. The information contents of the events are

$${\displaystyle \operatorname {I} (Same)=-\log _{2}\!{\tfrac {1}{6}}=2.5849625{\text{ shannons}}}$$

$${\displaystyle \operatorname {I} (Diff)=-\log _{2}\!{\tfrac {5}{6}}=0.2630344{\text{ shannons}}.}$$

The probability mass or density function (collectively probability measure) of the sum of two independent random variables is the convolution of each probability measure. In the case of independent fair 6-sided dice rolls, the random variable ${\displaystyle Z=X+Y}$ has probability mass function ${\textstyle p_{Z}(z)=p_{X}(x)*p_{Y}(y)={6-|z-7| \over 36}}$, where ${\displaystyle *}$ represents the discrete convolution. The outcome ${\displaystyle Z=5}$ has probability ${\textstyle p_{Z}(5)={\frac {4}{36}}={1 \over 9}}$. Therefore, the information asserted is

$${\displaystyle \operatorname {I} _{Z}(5)=-\log _{2}{\tfrac {1}{9}}=\log _{2}{9}\approx 3.169925{\text{ shannons.}}}$$

Generalizing the § Fair dice roll example above, consider a general discrete uniform random variable (DURV) ${\displaystyle X\sim \mathrm {DU} [a,b];\quad a,b\in \mathbb {Z} ,\ b\geq a.}$ For convenience, define ${\textstyle N:=b-a+1}$. The p.m.f. is

$${\displaystyle p_{X}(k)={\begin{cases}{\frac {1}{N}},&k\in [a,b]\cap \mathbb {Z} \\0,&{\text{otherwise}}\end{cases}}.}$$

In general, the values of the DURV need not be integers, or for the purposes of information theory even uniformly spaced; they need only be equiprobable.[2] The information gain of any observation ${\displaystyle X=k}$is

$${\displaystyle \operatorname {I} _{X}(k)=-\log _{2}{\frac {1}{N}}=\log _{2}{N}{\text{ shannons}}.}$$

If ${\displaystyle b=a}$ above, ${\displaystyle X}$ degenerates to a constant random variable with probability distribution deterministically given by ${\displaystyle X=b}$ and probability measure the Dirac measure ${\textstyle p_{X}(k)=\delta _{b}(k)}$. The only value ${\displaystyle X}$ can take is deterministically ${\displaystyle b}$, so the information content of any measurement of ${\displaystyle X}$ is

$${\displaystyle \operatorname {I} _{X}(b)=-\log _{2}{1}=0.}$$

In general, there is no information gained from measuring a known value.[2]

Generalizing all of the above cases, consider a categorical discrete random variable with support ${\textstyle {\mathcal {S}}={\bigl \{}s_{i}{\bigr \}}_{i=1}^{N}}$ and p.m.f. given by

$${\displaystyle p_{X}(k)={\begin{cases}p_{i},&k=s_{i}\in {\mathcal {S}}\\0,&{\text{otherwise}}\end{cases}}.}$$

For the purposes of information theory, the values ${\displaystyle s\in {\mathcal {S}}}$ do not even have to be numbers at all; they can just be mutually exclusive events on a measure space of finite measure that has been normalized to a probability measure ${\displaystyle p}$. Without loss of generality, we can assume the categorical distribution is supported on the set ${\textstyle [N]=\left\{1,2,...,N\right\}}$; the mathematical structure is isomorphic in terms of probability theory and therefore information theory as well.

The information of the outcome ${\displaystyle X=x}$ is given

$${\displaystyle \operatorname {I} _{X}(x)=-\log _{2}{p_{X}(x)}.}$$

From these examples, it is possible to calculate the information of any set of independent DRVs with known distributions by additivity.