Feeds:
Posts
Comments

Archive for October, 2013

10 Nice things about Bayes’ theorem

Posted in Mathematics, tagged on October 26, 2013| 13 Comments »

As background for some future posts, I need to catalogue a few facts about Bayes’ theorem. This is all standard probability theory. I’ll be roughly following the discussion and notation of Jaynes.

Probability

We start with propositions, represented by A, B, etc. These propositions are in fact true or false, though we may not know which. We assume that we can do Boolean things with these propositions, in particular:

  • Conjunction: AB means “both A and B are true”

  • Disjunction: A+B means “at least one of the propositions A, B is true”

  • Negation: \bar{A} means “not A” (i.e. A is false)

We want to assign probabilities to propositions to represent how likely they are to be true, given the information in other propositions. The function p(A | B) assigns a probability to the proposition A, using only the information in B. Read “p” as “the probability of” and the vertical bar “ | ” as “given”, so p(A | B) reads “the probability of A given B”. There are no “raw” probabilities, p(A).

What is p? The older approach to probability of Kolmogorov et al. requires that P satisfy certain axioms. Roughly,

A1. p(A | B) is a non-negative number.

A2. One means certain: p(A | B) = 1 if B \Rightarrow A i.e. if A is certain, given B.

A3. Or means add: if at most one of (countable) A_i is true (i.e. disjoint), then p(A_1 + A_2 + \ldots | B) = \sum_{i = 1} p(A_i | B).

Since the publication of Cox’s Theorem, an alternative approach to probability has gained popularity. Cox (again I’m following Jaynes) proposes the following desiderata of rationality, not as arbitrary axioms but as expectations of any rational approach to reasoning in the face of uncertainty.

D1. Probabilities are represented by real numbers.

D2. Probabilities change in common sense ways. For example, if learning C makes B more likely, but doesn’t change how likely A is, then learning C should make AB more likely.

D3. If a conclusion can be reasoned out in more than one way, then every possible way must lead to the same result.

D4. Information must not be arbitrarily ignored. All given evidence must be taken into account.

D5. Identical states of knowledge (except perhaps for the labeling of the propositions) should result in identical assigned probabilities.

The great advance of Cox’s theorem was to show that one can start with these desiderata and arrive at Kolmogorov’s axioms (at least, for the finite version of A3). Thus, the traditional laws of probability can be applied to more than just frequencies.

Conjunction and Total Probability

Beginning with the desiderata, we can derive more than Kolmogorov’s axioms. There are a number of useful probability identities. Remember that an identity is a formula that holds for any propositions we may substitute in. Here are two preliminary identities, before we get to Bayes’ theorem.

Conjunction: If A = A_1 A_2 then, (more…)

Advertisement

Read Full Post »