Bayesian probability

Bayesian methods are characterized by concepts and procedures as follows:

• The use of random variables, or more generally unknown quantities,[7] to model all sources of uncertainty in statistical models including uncertainty resulting from lack of information (see also aleatoric and epistemic uncertainty).
• The need to determine the prior probability distribution taking into account the available (prior) information.
• The sequential use of Bayes' formula: when more data become available, calculate the posterior distribution using Bayes' formula; subsequently, the posterior distribution becomes the next prior.
• While for the frequentist, a hypothesis is a proposition (which must be either true or false) so that the frequentist probability of a hypothesis is either 0 or 1, in Bayesian statistics, the probability that can be assigned to a hypothesis can also be in a range from 0 to 1 if the truth value is uncertain.

Broadly speaking, there are two interpretations of Bayesian probability. For objectivists, who interpret probability as an extension of logic, probability quantifies the reasonable expectation that everyone (even a "robot") who shares the same knowledge should share in accordance with the rules of Bayesian statistics, which can be justified by Cox's theorem.[2][8] For subjectivists, probability corresponds to a personal belief.[3] Rationality and coherence allow for substantial variation within the constraints they pose; the constraints are justified by the Dutch book argument or by decision theory and de Finetti's theorem.[3] The objective and subjective variants of Bayesian probability differ mainly in their interpretation and construction of the prior probability.

The term Bayesian derives from Thomas Bayes (1702–1761), who proved a special case of what is now called Bayes' theorem in a paper titled "An Essay towards solving a Problem in the Doctrine of Chances".[9] In that special case, the prior and posterior distributions were beta distributions and the data came from Bernoulli trials. It was Pierre-Simon Laplace (1749–1827) who introduced a general version of the theorem and used it to approach problems in celestial mechanics, medical statistics, reliability, and jurisprudence.[10] Early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes).[11] After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called frequentist statistics.[11]

In the 20th century, the ideas of Laplace developed in two directions, giving rise to objective and subjective currents in Bayesian practice. Harold Jeffreys' Theory of Probability (first published in 1939) played an important role in the revival of the Bayesian view of probability, followed by works by Abraham Wald (1950) and Leonard J. Savage (1954). The adjective Bayesian itself dates to the 1950s; the derived Bayesianism, neo-Bayesianism is of 1960s coinage.[12][13][14] In the objectivist stream, the statistical analysis depends on only the model assumed and the data analysed.[15] No subjective decisions need to be involved. In contrast, "subjectivist" statisticians deny the possibility of fully objective analysis for the general case.

In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods and the consequent removal of many of the computational problems, and to an increasing interest in nonstandard, complex applications.[16] While frequentist statistics remains strong (as seen by the fact that most undergraduate teaching is still based on it [17]), Bayesian methods are widely accepted and used, e.g., in the field of machine learning.[18]

The use of Bayesian probabilities as the basis of Bayesian inference has been supported by several arguments, such as Cox axioms, the Dutch book argument, arguments based on decision theory and de Finetti's theorem.

Following the work on expected utility theory of Ramsey and von Neumann, decision-theorists have accounted for rational behavior using a probability distribution for the agent. Johann Pfanzagl completed the Theory of Games and Economic Behavior by providing an axiomatization of subjective probability and utility, a task left uncompleted by von Neumann and Oskar Morgenstern: their original theory supposed that all the agents had the same probability distribution, as a convenience.[28] Pfanzagl's axiomatization was endorsed by Oskar Morgenstern: "Von Neumann and I have anticipated ... [the question whether probabilities] might, perhaps more typically, be subjective and have stated specifically that in the latter case axioms could be found from which could derive the desired numerical utility together with a number for the probabilities (cf. p. 19 of The Theory of Games and Economic Behavior). We did not carry this out; it was demonstrated by Pfanzagl ... with all the necessary rigor".[29]

Ramsey and Savage noted that the individual agent's probability distribution could be objectively studied in experiments. Procedures for testing hypotheses about probabilities (using finite samples) are due to Ramsey (1931) and de Finetti (1931, 1937, 1964, 1970). Both Bruno de Finetti[30][31] and Frank P. Ramsey[31][32] acknowledge their debts to pragmatic philosophy, particularly (for Ramsey) to Charles S. Peirce.[31][32]

The "Ramsey test" for evaluating probability distributions is implementable in theory, and has kept experimental psychologists occupied for a half century.[33] This work demonstrates that Bayesian-probability propositions can be falsified, and so meet an empirical criterion of Charles S. Peirce, whose work inspired Ramsey. (This falsifiability-criterion was popularized by Karl Popper.[34][35])

Modern work on the experimental evaluation of personal probabilities uses the randomization, blinding, and Boolean-decision procedures of the Peirce-Jastrow experiment.[36] Since individuals act according to different probability judgments, these agents' probabilities are "personal" (but amenable to objective study).

Personal probabilities are problematic for science and for some applications where decision-makers lack the knowledge or time to specify an informed probability-distribution (on which they are prepared to act). To meet the needs of science and of human limitations, Bayesian statisticians have developed "objective" methods for specifying prior probabilities.

Indeed, some Bayesians have argued the prior state of knowledge defines the (unique) prior probability-distribution for "regular" statistical problems; cf. well-posed problems. Finding the right method for constructing such "objective" priors (for appropriate classes of regular problems) has been the quest of statistical theorists from Laplace to John Maynard Keynes, Harold Jeffreys, and Edwin Thompson Jaynes. These theorists and their successors have suggested several methods for constructing "objective" priors (Unfortunately, it is not clear how to assess the relative "objectivity" of the priors proposed under these methods):

• Maximum entropy
• Transformation group analysis
• Reference analysis

Each of these methods contributes useful priors for "regular" one-parameter problems, and each prior can handle some challenging statistical models (with "irregularity" or several parameters). Each of these methods has been useful in Bayesian practice. Indeed, methods for constructing "objective" (alternatively, "default" or "ignorance") priors have been developed by avowed subjective (or "personal") Bayesians like James Berger (Duke University) and José-Miguel Bernardo (Universitat de València), simply because such priors are needed for Bayesian practice, particularly in science.[37] The quest for "the universal method for constructing priors" continues to attract statistical theorists.[37]

Thus, the Bayesian statistician needs either to use informed priors (using relevant expertise or previous data) or to choose among the competing methods for constructing "objective" priors.

• Bertrand paradox—a paradox in classical probability
• De Finetti's game—a procedure for evaluating someone's subjective probability
• QBism—an interpretation of quantum mechanics based on subjective Bayesian probability
• Reference class problem
• An Essay towards solving a Problem in the Doctrine of Chances
• Monty Hall problem

• Berger, James O. (1985). Statistical Decision Theory and Bayesian Analysis. Springer Series in Statistics (Second ed.). Springer-Verlag. ISBN 978-0-387-96098-2.
• Bessière, Pierre; Mazer, E.; Ahuacatzin, J.-M.; Mekhnacha, K. (2013). Bayesian Programming. CRC Press. ISBN 9781439880326.
• Bernardo, José M.; Smith, Adrian F.M. (1994). Bayesian Theory. Wiley. ISBN 978-0-471-49464-5.
• Bickel, Peter J.; Doksum, Kjell A. (2001) [1976]. Basic and selected topics. Mathematical Statistics. 1 (Second ed.). Pearson Prentice–Hall. ISBN 978-0-13-850363-5. MR 0443141. (updated printing, 2007, of Holden-Day, 1976)
• Davidson, Donald; Suppes, Patrick; Siegel, Sidney (1957). Decision-Making: an Experimental Approach. Stanford University Press.
• de Finetti, Bruno (1937). "La Prévision: ses lois logiques, ses sources subjectives" [Foresight: Its logical laws, its subjective sources]. Annales de l'Institut Henri Poincaré (in French).
• de Finetti, Bruno (September 1989) [1931]. "Probabilism: A critical essay on the theory of probability and on the value of science". Erkenntnis. 31. (translation of de Finetti, 1931)
• de Finetti, Bruno (1964) [1937]. "Foresight: Its logical laws, its subjective sources". In Kyburg, H.E.; Smokler, H.E. (eds.). Studies in Subjective Probability. New York, NY: Wiley. (translation of de Finetti, 1937, above)
• de Finetti, Bruno (1974–1975) [1970]. Theory of Probability: A critical introductory treatment. Translated by Machi, A.; Smith, AFM. Wiley. ISBN 0-471-20141-3., ISBN 0-471-20142-1, two volumes.
• DeGroot, Morris (2004) [1970]. Optimal Statistical Decisions. Wiley Classics Library. Wiley. ISBN 0-471-68029-X..
• Hacking, Ian (December 1967). "Slightly more realistic personal probability". Philosophy of Science. 34 (4): 311–325. doi:10.1086/288169. JSTOR 186120.
  Partly reprinted in Gärdenfors, Peter; Sahlin, Nils-Eric (1988). Decision, Probability, and Utility: Selected Readings. Cambridge University Press. ISBN 0-521-33658-9.
• Hajek, A.; Hartmann, S. (2010) [2001]. "Bayesian Epistemology". In Dancy, J.; Sosa, E.; Steup, M. (eds.). A Companion to Epistemology (PDF). Wiley. ISBN 978-1-4051-3900-7. Archived from the original (PDF) on 2011-07-28.
• Hald, Anders (1998). A History of Mathematical Statistics from 1750 to 1930. New York: Wiley. ISBN 978-0-471-17912-2.
• Hartmann, S.; Sprenger, J. (2011). "Bayesian Epistemology". In Bernecker, S.; Pritchard, D. (eds.). Routledge Companion to Epistemology (PDF). Routledge. ISBN 978-0-415-96219-3. Archived from the original (PDF) on 2011-07-28.
• "Bayesian approach to statistical problems", Encyclopedia of Mathematics, EMS Press, 2001 [1994]
• Howson, C.; Urbach, P. (2005). Scientific Reasoning: The Bayesian approach (3rd ed.). Open Court Publishing Company. ISBN 978-0-8126-9578-6.
• Jaynes, E.T. (2003). Probability Theory: The logic of science. C. University Press. ISBN 978-0-521-59271-0. ("Link to fragmentary edition of March 1996".
• McGrayne, S.B. (2011). The Theory that would not Die: How Bayes' rule cracked the Enigma code, hunted down Russian submarines, and emerged triumphant from two centuries of controversy. New Haven, CT: Yale University Press. ISBN 9780300169690. OCLC 670481486.
• Morgenstern, Oskar (1978). "Some Reflections on Utility". In Schotter, Andrew (ed.). Selected Economic Writings of Oskar Morgenstern. New York University Press. pp. 65–70. ISBN 978-0-8147-7771-8.
• Peirce, C.S. & Jastrow J. (1885). "On Small Differences in Sensation". Memoirs of the National Academy of Sciences. 3: 73–83.
• Pfanzagl, J (1967). "Subjective Probability Derived from the Morgenstern-von Neumann Utility Theory". In Martin Shubik (ed.). Essays in Mathematical Economics In Honor of Oskar Morgenstern. Princeton University Press. pp. 237–251.
• Pfanzagl, J.; Baumann, V. & Huber, H. (1968). "Events, Utility and Subjective Probability". Theory of Measurement. Wiley. pp. 195–220.
• Ramsey, Frank Plumpton (2001) [1931]. "Chapter VII: Truth and Probability". The Foundations of Mathematics and other Logical Essays. Routledge. ISBN 0-415-22546-9. "Chapter VII: Truth and Probability" (PDF). Archived from the original (PDF) on 2008-02-27.
• Stigler, S.M. (1990). The History of Statistics: The Measurement of Uncertainty before 1900. Belknap Press; Harvard University Press. ISBN 978-0-674-40341-3.
• Stigler, S.M. (1999). Statistics on the Table: The history of statistical concepts and methods. Harvard University Press. ISBN 0-674-83601-4.
• Stone, J.V. (2013). Bayes' Rule: A tutorial introduction to Bayesian analysis. England: Sebtel Press. "Chapter 1 of Bayes' Rule".
• Winkler, R.L. (2003). Introduction to Bayesian Inference and Decision (2nd ed.). Probabilistic. ISBN 978-0-9647938-4-2. Updated classic textbook. Bayesian theory clearly presented