Pointwise mutual information

The PMI of a pair of outcomes x and y belonging to discrete random variables X and Y quantifies the discrepancy between the probability of their coincidence given their joint distribution and their individual distributions, assuming independence. Mathematically:

${\displaystyle \operatorname {pmi} (x;y)\equiv \log {\frac {p(x,y)}{p(x)p(y)}}=\log {\frac {p(x|y)}{p(x)}}=\log {\frac {p(y|x)}{p(y)}}.}$

The mutual information (MI) of the random variables X and Y is the expected value of the PMI (over all possible outcomes).

The measure is symmetric (${\displaystyle \operatorname {pmi} (x;y)=\operatorname {pmi} (y;x)}$). It can take positive or negative values, but is zero if X and Y are independent. Note that even though PMI may be negative or positive, its expected outcome over all joint events (MI) is positive. PMI maximizes when X and Y are perfectly associated (i.e. ${\displaystyle p(x|y)}$ or ${\displaystyle p(y|x)=1}$), yielding the following bounds:

${\displaystyle -\infty \leq \operatorname {pmi} (x;y)\leq \min \left[-\log p(x),-\log p(y)\right].}$

Finally, ${\displaystyle \operatorname {pmi} (x;y)}$ will increase if ${\displaystyle p(x|y)}$ is fixed but ${\displaystyle p(x)}$ decreases.

Here is an example to illustrate:

Using this table we can marginalize to get the following additional table for the individual distributions:

With this example, we can compute four values for ${\displaystyle pmi(x;y)}$. Using base-2 logarithms:

(For reference, the mutual information ${\displaystyle \operatorname {I} (X;Y)}$ would then be 0.2141709)

Pointwise Mutual Information has many of the same relationships as the mutual information. In particular,

${\displaystyle {\begin{aligned}\operatorname {pmi} (x;y)&=&h(x)+h(y)-h(x,y)\\&=&h(x)-h(x|y)\\&=&h(y)-h(y|x)\end{aligned}}}$

Where ${\displaystyle h(x)}$ is the self-information, or ${\displaystyle -\log _{2}p(X=x)}$.

Pointwise mutual information can be normalized between [-1,+1] resulting in -1 (in the limit) for never occurring together, 0 for independence, and +1 for complete co-occurrence.[2]

${\displaystyle \operatorname {npmi} (x;y)={\frac {\operatorname {pmi} (x;y)}{h(x,y)}}}$

Where ${\displaystyle h(x,y)}$ is the joint self-information, which is estimated as ${\displaystyle -\log _{2}p(X=x,Y=y)}$.

In addition to the above-mentioned npmi, PMI has many other interesting variants. A comparative study of these variants can be found in [3]

Like mutual information,[4] point mutual information follows the chain rule, that is,

${\displaystyle \operatorname {pmi} (x;yz)=\operatorname {pmi} (x;y)+\operatorname {pmi} (x;z|y)}$

This is easily proven by:

${\displaystyle {\begin{aligned}\operatorname {pmi} (x;y)+\operatorname {pmi} (x;z|y)&{}=\log {\frac {p(x,y)}{p(x)p(y)}}+\log {\frac {p(x,z|y)}{p(x|y)p(z|y)}}\\&{}=\log \left[{\frac {p(x,y)}{p(x)p(y)}}{\frac {p(x,z|y)}{p(x|y)p(z|y)}}\right]\\&{}=\log {\frac {p(x|y)p(y)p(x,z|y)}{p(x)p(y)p(x|y)p(z|y)}}\\&{}=\log {\frac {p(x,yz)}{p(x)p(yz)}}\\&{}=\operatorname {pmi} (x;yz)\end{aligned}}}$

In computational linguistics, PMI has been used for finding collocations and associations between words. For instance, countings of occurrences and co-occurrences of words in a text corpus can be used to approximate the probabilities ${\displaystyle p(x)}$ and ${\displaystyle p(x,y)}$ respectively. The following table shows counts of pairs of words getting the most and the least PMI scores in the first 50 millions of words in Wikipedia (dump of October 2015) filtering by 1,000 or more co-occurrences. The frequency of each count can be obtained by dividing its value by 50,000,952. (Note: natural log is used to calculate the PMI values in this example, instead of log base 2)

Good collocation pairs have high PMI because the probability of co-occurrence is only slightly lower than the probabilities of occurrence of each word. Conversely, a pair of words whose probabilities of occurrence are considerably higher than their probability of co-occurrence gets a small PMI score.

• Fano, R M (1961). "chapter 2". Transmission of Information: A Statistical Theory of Communications. MIT Press, Cambridge, MA. ISBN 978-0262561693.

• Demo at Rensselaer MSR Server (PMI values normalized to be between 0 and 1)