Lincoln index

Consider two observers who separately count the different species of plants or animals in a given area. If they each come back having found 100 species but only 5 particular species are found by both observers, then each observer clearly missed at least 95 species (that is, the 95 that only the other observer found). Thus, we know that both observers miss a lot. On the other hand, if 99 of the 100 species each observer found had been found by both, it is fair to expect that they have found a far higher percentage of the total species that are there to find.

The same reasoning applies to mark and recapture. If some animals in a given area are caught and marked, and later a second round of captures is done: the number of marked animals found in the second round can be used to generate an estimate of the total population.[2]

Another example arises in computational linguistics for estimating the total vocabulary of a language. Given two independent samples, the overlap between their vocabularies enables a useful estimate of how many more vocabulary items exist but did not happen to show up in either sample. A similar example involves estimating the number of typographical errors remaining in a text, from two proofreaders' counts.

The Lincoln Index formalizes this phenomenon. If E1 and E2 are the number of species (or words, or other phenomena) observed by two independent methods, and S is the number of observations in common, then the Lincoln Index is simply

${\displaystyle L={E_{1}E_{2} \over S}.}$

For values of S < 10, this estimate is rough, and becomes extremely rough for values of S < 5. In the case where S = 0 (that is, there is no overlap at all) the Lincoln Index is formally undefined. This can arise if the observers only find a small percentage of the actual species (perhaps by not looking hard enough or long enough), if the observers are using methods that are not statistically independent (for example if one looks only for large creatures and the other only for small), or in other circumstances.

The Lincoln Index is merely an estimate. For example, the species in a given area could tend to be either very common or very rare, or tend to be either very hard or very easy to see.[3] Then it would be likely that both observers would find a large share of the common species, and that both observers would miss a large share of the rare ones. Such distributions would throw off the consequent estimate. However, such distributions are unusual for natural phenomena, as suggested by Zipf's Law).

T. J. Gaskell and B. J. George propose an enhancement of the Lincoln Index that claims to reduce bias.[4]

• Sampling Theory
• Drake equation
• German tank problem
• Inter-rater reliability

• Lincoln, Frederick C. (May 1930). Calculating Waterfowl Abundance on the Basis of Banding Returns. Circular. 118. Washington, DC: United States Department of Agriculture. Retrieved 21 May 2013.
• Petersen, C. G. J. (1896). "The Yearly Immigration of Young Plaice Into the Limfjord From the German Sea", Report of the Danish Biological Station (1895), 6, 5–84.
• T. J. Gaskell; B. J. George (1972). "A Bayesian Modification of the Lincoln Index". Journal of Applied Ecology. 9 (2): 377–384. doi:10.2307/2402438.