Category utility

Gluck & Corter (1985) and Corter & Gluck (1992) mention that the category utility is equivalent to the mutual information. Here is a simple demonstration of the nature of this equivalence. Assume a set of entities each having the same ${\displaystyle n}$ features, i.e., feature set ${\displaystyle F=\{f_{i}\},\ i=1\ldots n}$, with each feature variable having cardinality ${\displaystyle m}$. That is, each feature has the capacity to adopt any of ${\displaystyle m}$ distinct values (which need not be ordered; all variables can be nominal); for the special case ${\displaystyle m=2}$ these features would be considered binary, but more generally, for any ${\displaystyle m}$, the features are simply m-ary. For the purposes of this demonstration, without loss of generality, feature set ${\displaystyle F}$ can be replaced with a single aggregate variable ${\displaystyle F_{a}}$ that has cardinality ${\displaystyle m^{n}}$, and adopts a unique value ${\displaystyle v_{i},\ i=1\ldots m^{n}}$ corresponding to each feature combination in the Cartesian product ${\displaystyle \otimes F}$. (Ordinality does not matter, because the mutual information is not sensitive to ordinality.) In what follows, a term such as ${\displaystyle p(F_{a}=v_{i})}$ or simply ${\displaystyle p(v_{i})}$ refers to the probability with which ${\displaystyle F_{a}}$ adopts the particular value ${\displaystyle v_{i}}$. (Using the aggregate feature variable ${\displaystyle F_{a}}$ replaces multiple summations, and simplifies the presentation to follow.)

For this demonstration, also assume a single category variable ${\displaystyle C}$, which has cardinality ${\displaystyle p}$. This is equivalent to a classification system in which there are ${\displaystyle p}$ non-intersecting categories. In the special case of ${\displaystyle p=2}$ there are the two-category case discussed above. From the definition of mutual information for discrete variables, the mutual information ${\displaystyle I(F_{a};C)}$ between the aggregate feature variable ${\displaystyle F_{a}}$ and the category variable ${\displaystyle C}$ is given by:

${\displaystyle I(F_{a};C)=\sum _{v_{i}\in F_{a}}\sum _{c_{j}\in C}p(v_{i},c_{j})\log {\frac {p(v_{i},c_{j})}{p(v_{i})\,p(c_{j})}}}$

where ${\displaystyle p(v_{i})}$ is the prior probability of feature variable ${\displaystyle F_{a}}$ adopting value ${\displaystyle v_{i}}$, ${\displaystyle p(c_{j})}$ is the marginal probability of category variable ${\displaystyle C}$ adopting value ${\displaystyle c_{j}}$, and ${\displaystyle p(v_{i},c_{j})}$ is the joint probability of variables ${\displaystyle F_{a}}$ and ${\displaystyle C}$ simultaneously adopting those respective values. In terms of the conditional probabilities this can be re-written (or defined) as

${\displaystyle {\begin{aligned}I(F_{a};C)&=\sum _{v_{i}\in F_{a}}\sum _{c_{j}\in C}p(v_{i},c_{j})\log {\frac {p(v_{i}|c_{j})}{p(v_{i})}}\\&=\sum _{v_{i}\in F_{a}}\sum _{c_{j}\in C}p(v_{i}|c_{j})p(c_{j})\left[\log p(v_{i}|c_{j})-\log p(v_{i})\right]\\&=\sum _{v_{i}\in F_{a}}\sum _{c_{j}\in C}p(v_{i}|c_{j})p(c_{j})\log p(v_{i}|c_{j})-\sum _{v_{i}\in F_{a}}\sum _{c_{j}\in C}p(v_{i}|c_{j})p(c_{j})\log p(v_{i})\\&=\sum _{v_{i}\in F_{a}}\sum _{c_{j}\in C}p(v_{i}|c_{j})p(c_{j})\log p(v_{i}|c_{j})-\sum _{v_{i}\in F_{a}}\sum _{c_{j}\in C}p(v_{i},c_{j})\log p(v_{i})\\&=\sum _{v_{i}\in F_{a}}\sum _{c_{j}\in C}p(v_{i}|c_{j})p(c_{j})\log p(v_{i}|c_{j})-\sum _{v_{i}\in F_{a}}\log p(v_{i})\sum _{c_{j}\in C}p(v_{i},c_{j})\\&=\sum _{v_{i}\in F_{a}}\sum _{c_{j}\in C}p(v_{i}|c_{j})p(c_{j})\log p(v_{i}|c_{j})-\sum _{v_{i}\in F_{a}}p(v_{i})\log p(v_{i})\\\end{aligned}}}$

If the original definition of the category utility from above is rewritten with ${\displaystyle C=\{c,{\bar {c}}\}}$,

${\displaystyle CU(C,F)=\sum _{f_{i}\in F}\sum _{c_{j}\in C}p(f_{i}|c_{j})p(c_{j})\log p(f_{i}|c_{j})-\sum _{f_{i}\in F}p(f_{i})\log p(f_{i})}$

This equation clearly has the same form as the (blue) equation expressing the mutual information between the feature set and the category variable; the difference is that the sum ${\displaystyle \textstyle \sum _{f_{i}\in F}}$ in the category utility equation runs over independent binary variables ${\displaystyle F=\{f_{i}\},\ i=1\ldots n}$, whereas the sum ${\displaystyle \textstyle \sum _{v_{i}\in F_{a}}}$ in the mutual information runs over values of the single ${\displaystyle m^{n}}$-ary variable ${\displaystyle F_{a}}$. The two measures are actually equivalent then only when the features ${\displaystyle \{f_{i}\}}$, are independent (and assuming that terms in the sum corresponding to ${\displaystyle p({\bar {f_{i}}})}$ are also added).

At least since the time of Aristotle there has been a tremendous fascination in philosophy with the nature of concepts and universals. What kind of entity is a concept such as "horse"? Such abstractions do not designate any particular individual in the world, and yet we can scarcely imagine being able to comprehend the world without their use. Does the concept "horse" therefore have an independent existence outside of the mind? If it does, then what is the locus of this independent existence? The question of locus was an important issue on which the classical schools of Plato and Aristotle famously differed. However, they remained in agreement that universals did indeed have a mind-independent existence. There was, therefore, always a fact to the matter about which concepts and universals exist in the world.

In the late Middle Ages (perhaps beginning with Occam, although Porphyry also makes a much earlier remark indicating a certain discomfort with the status quo), however, the certainty that existed on this issue began to erode, and it became acceptable among the so-called nominalists and empiricists to consider concepts and universals as strictly mental entities or conventions of language. On this view of concepts—that they are purely representational constructs—a new question then comes to the fore: "Why do we possess one set of concepts rather than another?" What makes one set of concepts "good" and another set of concepts "bad"? This is a question that modern philosophers, and subsequently machine learning theorists and cognitive scientists, have struggled with for many decades.

One approach to answering such questions is to investigate the "role" or "purpose" of concepts in cognition. Thus the answer to "What are concepts good for in the first place?" by Mill & 1843/1936, p. 425) harvtxt error: no target: CITEREFMill1843/1936 (help) and many others is that classification (conception) is a precursor to induction: By imposing a particular categorization on the universe, an organism gains the ability to deal with physically non-identical objects or situations in an identical fashion, thereby gaining substantial predictive leverage (Smith & Medin 1981; Harnad 2005). As J.S. Mill puts it (Mill & 1843/1936, pp. 466–468) harv error: no target: CITEREFMill1843/1936 (help),

The general problem of classification... [is] to provide that things shall be thought of in such groups, and those groups in such an order, as will best conduce to the remembrance and to the ascertainment of their laws... [and] one of the uses of such a classification that by drawing attention to the properties on which it is founded, and which, if the classification be good, are marks of many others, it facilitates the discovery of those others.

From this base, Mill reaches the following conclusion, which foreshadows much subsequent thinking about category goodness, including the notion of category utility:

The ends of scientific classification are best answered when the objects are formed into groups respecting which a greater number of general propositions can be made, and those propositions more important, than could be made respecting any other groups into which the same things could be distributed. The properties, therefore, according to which objects are classified should, if possible, be those which are causes of many other properties; or, at any rate, which are sure marks of them.

One may compare this to the "category utility hypothesis" proposed by Corter & Gluck (1992): "A category is useful to the extent that it can be expected to improve the ability of a person to accurately predict the features of instances of that category." Mill here seems to be suggesting that the best category structure is one in which object features (properties) are maximally informative about the object's class, and, simultaneously, the object class is maximally informative about the object's features. In other words, a useful classification scheme is one in which category knowledge can be used to accurately infer object properties, and property knowledge can be used to accurately infer object classes. One may also compare this idea to Aristotle's criterion of counter-predication for definitional predicates, as well as to the notion of concepts described in formal concept analysis.

A variety of different measures have been suggested with an aim of formally capturing this notion of "category goodness," the best known of which is probably the "cue validity". Cue validity of a feature ${\displaystyle f_{i}\ }$ with respect to category ${\displaystyle c_{j}\ }$ is defined as the conditional probability of the category given the feature (Reed 1972;Rosch & Mervis 1975;Rosch 1978), ${\displaystyle p(c_{j}|f_{i})\ }$, or as the deviation of the conditional probability from the category base rate (Edgell 1993;Kruschke & Johansen 1999), ${\displaystyle p(c_{j}|f_{i})-p(c_{j})\ }$. Clearly, these measures quantify only inference from feature to category (i.e., cue validity), but not from category to feature, i.e., the category validity ${\displaystyle p(f_{i}|c_{j})\ }$. Also, while the cue validity was originally intended to account for the demonstrable appearance of basic categories in human cognition—categories of a particular level of generality that are evidently preferred by human learners—a number of major flaws in the cue validity quickly emerged in this regard (Jones 1983;Murphy 1982;Corter & Gluck 1992, and others).

One attempt to address both problems by simultaneously maximizing both feature validity and category validity was made by Jones (1983) in defining the "collocation index" as the product ${\displaystyle p(c_{j}|f_{i})p(f_{i}|c_{j})\ }$, but this construction was fairly ad hoc (see Corter & Gluck 1992). The category utility was introduced as a more sophisticated refinement of the cue validity, which attempts to more rigorously quantify the full inferential power of a class structure. As shown above, on a certain view the category utility is equivalent to the mutual information between the feature variable and the category variable. It has been suggested that categories having the greatest overall category utility are those that are not only those "best" in a normative sense, but also those human learners prefer to use, e.g., "basic" categories (Corter & Gluck 1992). Other related measures of category goodness are "cohesion" (Hanson & Bauer 1989;Gennari, Langley & Fisher 1989) and "salience" (Gennari 1989).