Principle of transformation groups

Consider a problem where all you are told is that there is a coin, and it has a head (H) and a tail (T). Denote this information by I. You are then asked "what is the probability of Heads?". Call this problem 1 and denote the probability P(H|I). Consider another question "what is the probability of Tails?". Call this problem 2 and denote this probability by P(T|I).

Now from the information that was actually in the question, there is no distinction between heads and tails. The whole paragraph above could be re-written with "Heads" and "Tails" interchanged, and "H" and "T" interchanged, and the problem statement would not be any different. Using the desideratum then demands that

${\displaystyle P(H|I)=P(T|I)}$

The probabilities must add to 1, this means that

${\displaystyle P(H|I)+P(T|I)=1\rightarrow 2P(H|I)=1\rightarrow P(H|I)=0.5}$.

Thus we have a unique solution. This argument easily extents to N categories, to give the "flat" prior probability 1/N. This provides a consistency-based argument for the principle of indifference, which goes as follows: if someone is truly ignorant about a discrete/countable set of outcomes apart from their potential existence, but does not assign them equal prior probabilities, then they are assigning different probabilities when given the same information.

This can be alternatively phrased as: a person who does not use the principle of indifference to assign prior probabilities to discrete variables, is either not ignorant about them, or reasoning inconsistently.

This is the easiest example for continuous variables. It is given by stating one is "ignorant" of the location parameter in a given problem. The statement that a parameter is a "location parameter" is that the sampling distribution, or likelihood of an observation X depends on a parameter ${\displaystyle \mu }$ only through the difference

${\displaystyle p(X|\mu ,I)=f(X-\mu )}$

for some normalised, but otherwise arbitrary distribution f(.).

Note that the given information that f(.) is a normalised distribution is a significant prerequisite to obtaining the final conclusion of a uniform prior; because uniform probability distributions can only be normalised given a finite input domain. In other words, the assumption that f(.) is normalized is implicitly also requiring that the location parameter ${\displaystyle \mu }$ does not extend to infinity in any of its dimensions. Otherwise, the uniform prior would not be normalisable.

Examples of location parameters include the mean parameter of a normal distribution with known variance and median parameter of a Cauchy distribution with known inter-quartile range.

The two "equivalent problems" in this case, given one's knowledge of the sampling distribution ${\displaystyle p(X|\mu ,I)=f(X-\mu )}$, but no other knowledge about ${\displaystyle \mu }$, is simply given by a "shift" of equal magnitude in X and ${\displaystyle \mu }$. This is because of the relation:

${\displaystyle f(X-\mu )=f([X+b]-[\mu +b])=f(X^{(1)}-\mu ^{(1)})}$

So simply "shifting" all quantities up by some number b and solving in the "shifted space" and then "shifting" back to the original one should give exactly the same answer as if we just worked on the original space. Making the transformation from ${\displaystyle \mu }$ to ${\displaystyle \mu ^{(1)}}$ has a Jacobian of simply 1, and so the prior probability ${\displaystyle g(\mu )=p(\mu |I)}$ must satisfy the functional equation:

${\displaystyle g(\mu )=\left|{\partial \mu ^{(1)} \over \partial \mu }\right|g(\mu ^{(1)})=g(\mu +b)}$

And the only function that satisfies this equation is the "constant prior":

${\displaystyle p(\mu |I)\propto 1}$

Thus the uniform prior is justified for expressing complete ignorance of a normalized prior distribution on a finite, continuous location parameter.

As in the above argument, a statement that ${\displaystyle \sigma }$ is a scale parameter means that the sampling distribution has the functional form:

${\displaystyle p(X|\sigma ,I)={1 \over \sigma }f\left({X \over \sigma }\right)}$

Where, as before f(.) is a normalised probability density function. The requirement that probabilities be finite and positive forces the condition ${\displaystyle \sigma >0}$. Examples include the standard deviation of a normal distribution with known mean or the gamma distribution. The "symmetry" in this problem is found by noting that

${\displaystyle {X \over \sigma }={Xa \over \sigma a};a>0}$

and setting ${\displaystyle X^{(1)}=Xa}$ and ${\displaystyle \sigma ^{(1)}=\sigma a.}$ But, unlike in the location parameter case, the Jacobian of this transformation in the sample space and the parameter space is a, not 1. So the sampling probability changes to:

${\displaystyle p(X^{(1)}|\sigma ,I)={1 \over a}\cdot {1 \over \sigma }f\left({Xa \over \sigma a}\right)={1 \over \sigma ^{(1)}}f\left({X^{(1)} \over \sigma ^{(1)}}\right)}$

Which is invariant (i.e. has the same form before and after the transformation), and the prior probability changes to:

${\displaystyle p(\sigma |I)={1 \over a}p(\sigma ^{(1)}|I)={1 \over a}p\left({\sigma \over a}|I\right)}$

Which has the unique solution (up to a proportionality constant):

${\displaystyle p(\sigma |I)\propto {1 \over \sigma }\rightarrow p(\log(\sigma )|I)\propto 1}$

Which is the well-known Jeffreys prior for scale parameters, which is "flat" on the log scale, although it is derived using a different argument to that here, based on the Fisher information function. The fact that these two methods give the same results in this case does not imply it in general.

Edwin Jaynes used this principle to provide a resolution to Bertrand's Paradox[2] by stating his ignorance about the exact position of the circle. The details are available in the reference or in the link.