Partition of a Sample Space and Total Probability

The reverse-probability problem

Suppose Bag I contains 2 white and 3 red balls, Bag II contains 4 white and 5 red balls, and one ball is drawn from a randomly chosen bag. Questions in the forward direction are easy: given the bag, what is the chance of white? But exams love the reverse question: given that the ball turned out white, what is the chance it came from Bag II? Here the evidence comes after the cause, and we must reason backwards from effect to cause. The tool for this is Bayes' theorem — and it is built on two simpler ideas: partitions and total probability.

Partition of a sample space

A set of events $E_1, E_2, \ldots, E_n$ is a partition of the sample space $S$ if

(a) they are pairwise disjoint: $E_i \cap E_j = \phi$ for $i \neq j$,

(b) together they cover everything: $E_1 \cup E_2 \cup \cdots \cup E_n = S$, and

(c) each has nonzero probability: $P(E_i) > 0$ for all $i$.

In words: exactly one of the $E_i$ must occur, and each is genuinely possible. The simplest partition is $\{E, E'\}$ for any event $E$ with $0 < P(E) < 1$ — either $E$ happens or it doesn't. Another useful one: for any two events $E$ and $F$, the four pieces $\{E \cap F,\ E \cap F',\ E' \cap F,\ E' \cap F'\}$ partition $S$. A sample space has many different partitions — you choose the one that matches the "causes" in your problem (which bag, which machine, which route).

Theorem of total probability

Theorem. If $\{E_1, E_2, \ldots, E_n\}$ is a partition of $S$ with each $P(E_i) > 0$, then for any event $A$, $$P(A) = P(E_1)\,P(A|E_1) + P(E_2)\,P(A|E_2) + \cdots + P(E_n)\,P(A|E_n) = \sum_{j=1}^{n} P(E_j)\,P(A|E_j)$$

Why it works (the 3-line proof):

Step 1 — slice $A$ along the partition: since the $E_i$ cover $S$, $$A = A \cap S = (A \cap E_1) \cup (A \cap E_2) \cup \cdots \cup (A \cap E_n)$$

Step 2 — the slices are disjoint (they sit inside the disjoint $E_i$), so probabilities add: $$P(A) = P(A \cap E_1) + P(A \cap E_2) + \cdots + P(A \cap E_n)$$

Step 3 — apply the multiplication rule to each slice: $P(A \cap E_i) = P(E_i)\,P(A|E_i)$. $\blacksquare$

The intuition: $P(A)$ is a weighted average of the conditional probabilities $P(A|E_i)$, weighted by how likely each cause $E_i$ is.

A worked total-probability computation

A person has undertaken a construction job. The probability of a strike is $0.65$; the job finishes on time with probability $0.80$ if there is no strike and $0.32$ if there is a strike. Find the probability that the job finishes on time.

Step 1 — choose the partition: let $B$ = "there is a strike", so $\{B, B'\}$ partitions $S$ with $P(B) = 0.65$, $P(B') = 0.35$.

Step 2 — record the conditionals: with $A$ = "completed on time", $P(A|B) = 0.32$ and $P(A|B') = 0.80$.

Step 3 — apply total probability: $$P(A) = P(B)\,P(A|B) + P(B')\,P(A|B') = 0.65 \times 0.32 + 0.35 \times 0.80 = 0.208 + 0.280 = 0.488$$

Answer: $P(\text{on time}) = 0.488$ — between the two conditional values $0.32$ and $0.80$, and closer to $0.32$ because the strike is the likelier cause. A total-probability answer should always land between the smallest and largest $P(A|E_i)$; use this as a sanity check.

Bayes' Theorem

The statement

Bayes' theorem. If $E_1, E_2, \ldots, E_n$ are nonempty events forming a partition of $S$, and $A$ is any event with $P(A) \neq 0$, then for each $i$, $$P(E_i|A) = \frac{P(E_i)\,P(A|E_i)}{\displaystyle\sum_{j=1}^{n} P(E_j)\,P(A|E_j)}$$

Proof (two lines, and worth 2 marks on its own):

Step 1 — definition of conditional probability, then the multiplication rule on the numerator: $$P(E_i|A) = \frac{P(A \cap E_i)}{P(A)} = \frac{P(E_i)\,P(A|E_i)}{P(A)}$$

Step 2 — expand $P(A)$ by the theorem of total probability: $$P(E_i|A) = \frac{P(E_i)\,P(A|E_i)}{\sum_{j=1}^{n} P(E_j)\,P(A|E_j)} \qquad \blacksquare$$

How to read the formula. The denominator is the total probability of the observed evidence $A$, summed over every possible cause. The numerator is the share of that total contributed by the one cause $E_i$ you are asked about. So Bayes' theorem is simply

$$P(\text{cause}\,|\,\text{evidence}) = \frac{\text{this cause's contribution}}{\text{all causes' contributions}}$$

Once the numerator is written down, the denominator is the sum of all terms of the same shape — no new computation, just bookkeeping.

Terminology

When Bayes' theorem is applied, the events $E_1, \ldots, E_n$ are called hypotheses, $P(E_i)$ is the priori probability of the hypothesis, and $P(E_i|A)$ is its posteriori probability — the updated probability after the evidence $A$ is seen. Bayes' theorem is also called the formula for the probability of causes: exactly one of the $E_i$ occurs, and the formula tells us which cause is how likely, given what we observed.

The board-exam template (follow it every time)

Step 1 — name the hypotheses: define $E_1, E_2, \ldots$ (which bag / machine / route / speaker honesty) and check they form a partition.

Step 2 — write the priors: $P(E_1), P(E_2), \ldots$ — these must add to 1.

Step 3 — define the evidence $A$ and write every conditional $P(A|E_i)$.

Step 4 — state Bayes' theorem, substitute, simplify. Keep every value as a fraction; decimals invite arithmetic slips.

Step 5 — answer in words ("the probability that the defective bolt came from machine B is $\frac{28}{69}$").

Board marking schemes award marks for each of these steps — writing the events and priors correctly earns credit even before the formula is touched.

Common mistakes to avoid

Mistake 1 — inverting the wrong way. $P(E_i|A)$ and $P(A|E_i)$ are different numbers. The question gives you $P(A|E_i)$ (forward: cause to evidence) and asks for $P(E_i|A)$ (backward: evidence to cause). If your "Bayes" fraction has no sum in the denominator, you have answered the forward question.

Mistake 2 — priors that don't add to 1. If the problem says machines produce 25%, 35%, 40% of output, the priors are $0.25, 0.35, 0.40$. Forgetting a hypothesis (or double-counting one) shows up instantly as priors summing to something other than 1.

Mistake 3 — dropping a zero term too early. If some $P(A|E_i) = 0$ (the doctor is never late by the fourth mode of transport), the term contributes 0 to the denominator — but write it and then drop it, so the examiner sees the full partition.

Mistake 4 — ignoring base rates. In rare-disease problems the tiny prior ($0.001$) crushes even a $90\%$-accurate test: the posterior comes out near $0.08$, not $0.9$. Trust the formula over intuition — this counterintuitive outcome is the point of the question.

Solved Examples

Example 1 — The two-bag classic

Bag I contains 3 red and 4 black balls; Bag II contains 5 red and 6 black balls. One ball is drawn at random from one of the bags and it is found to be red. Find the probability that it was drawn from Bag II.

Step 1 — hypotheses and priors: let $E_1$ = "Bag I chosen", $E_2$ = "Bag II chosen"; $P(E_1) = P(E_2) = \frac{1}{2}$.

Step 2 — evidence conditionals: with $A$ = "red ball drawn", $P(A|E_1) = \frac{3}{7}$ and $P(A|E_2) = \frac{5}{11}$.

Step 3 — Bayes' theorem: $$P(E_2|A) = \frac{P(E_2)\,P(A|E_2)}{P(E_1)\,P(A|E_1) + P(E_2)\,P(A|E_2)} = \frac{\frac{1}{2} \times \frac{5}{11}}{\frac{1}{2} \times \frac{3}{7} + \frac{1}{2} \times \frac{5}{11}}$$

Step 4 — simplify: the $\frac{1}{2}$'s cancel; $\frac{5}{11} \div \left(\frac{3}{7} + \frac{5}{11}\right) = \frac{5}{11} \div \frac{33 + 35}{77} = \frac{5}{11} \times \frac{77}{68} = \frac{35}{68}$.

Answer: $P(\text{Bag II} \mid \text{red}) = \dfrac{35}{68}$.

Example 2 — Three boxes of coins

Three identical boxes each hold two coins: box I two gold, box II two silver, box III one gold and one silver. A box is chosen at random and a coin drawn; it is gold. What is the probability that the other coin in that box is also gold?

Step 1 — translate the question: "the other coin is also gold" happens exactly when the chosen box is box I. So we need $P(E_1|A)$ where $E_i$ = "box $i$ chosen" and $A$ = "gold coin drawn".

Step 2 — priors and conditionals: $P(E_1) = P(E_2) = P(E_3) = \frac{1}{3}$; $P(A|E_1) = 1$, $P(A|E_2) = 0$, $P(A|E_3) = \frac{1}{2}$.

Step 3 — Bayes: $$P(E_1|A) = \frac{\frac{1}{3} \times 1}{\frac{1}{3} \times 1 + \frac{1}{3} \times 0 + \frac{1}{3} \times \frac{1}{2}} = \frac{\frac{1}{3}}{\frac{1}{3} + \frac{1}{6}} = \frac{\frac{1}{3}}{\frac{1}{2}} = \frac{2}{3}$$

Answer: $\dfrac{2}{3}$ — not $\frac{1}{2}$! Seeing gold makes the all-gold box twice as likely as the mixed box, because it has twice as many gold coins to show you.

Example 3 — The rare-disease test

A test detects a certain infection $90\%$ of the time when it is present, but $1\%$ of uninfected people also test positive. Only $0.1\%$ of a large population is infected. A random person tests positive. What is the probability that the person is actually infected?

Step 1 — hypotheses: $E$ = "actually infected", $E'$ = "not infected"; $P(E) = 0.001$, $P(E') = 0.999$.

Step 2 — conditionals: with $A$ = "tests positive", $P(A|E) = 0.9$ and $P(A|E') = 0.01$.

Step 3 — Bayes: $$P(E|A) = \frac{0.001 \times 0.9}{0.001 \times 0.9 + 0.999 \times 0.01} = \frac{0.0009}{0.0009 + 0.00999} = \frac{90}{1089} = \frac{10}{121} \approx 0.083$$

Answer: only about $8.3\%$. The false positives from the huge uninfected majority ($0.999 \times 0.01$) swamp the true positives from the tiny infected minority ($0.001 \times 0.9$). This base-rate effect is the most famous application of Bayes' theorem.

Example 4 — Which machine made the defective bolt?

Machines A, B, C manufacture $25\%$, $35\%$ and $40\%$ of a factory's bolts; $5\%$, $4\%$ and $2\%$ of their outputs are defective. A random bolt is defective. Find the probability that it came from machine B.

Step 1 — priors: $P(B_1) = 0.25$, $P(B_2) = 0.35$, $P(B_3) = 0.40$ (they add to 1 ✓).

Step 2 — conditionals for $E$ = "defective": $P(E|B_1) = 0.05$, $P(E|B_2) = 0.04$, $P(E|B_3) = 0.02$.

Step 3 — Bayes: $$P(B_2|E) = \frac{0.35 \times 0.04}{0.25 \times 0.05 + 0.35 \times 0.04 + 0.40 \times 0.02} = \frac{0.0140}{0.0125 + 0.0140 + 0.0080} = \frac{0.0140}{0.0345} = \frac{28}{69}$$

Answer: $\dfrac{28}{69}$. Note how machine C, despite making the most bolts, contributes the least defect probability — the posterior weighs production share times defect rate.

Example 5 — The late doctor

A doctor visits a patient by train, bus, scooter or other transport with probabilities $\frac{3}{10}, \frac{1}{5}, \frac{1}{10}, \frac{2}{5}$. The probabilities of being late are $\frac{1}{4}, \frac{1}{3}, \frac{1}{12}$ by train, bus and scooter, and $0$ by other transport. The doctor arrives late. Find the probability that the doctor came by train.

Step 1 — priors: $P(T_1) = \frac{3}{10}$, $P(T_2) = \frac{1}{5}$, $P(T_3) = \frac{1}{10}$, $P(T_4) = \frac{2}{5}$.

Step 2 — conditionals for $E$ = "late": $P(E|T_1) = \frac{1}{4}$, $P(E|T_2) = \frac{1}{3}$, $P(E|T_3) = \frac{1}{12}$, $P(E|T_4) = 0$.

Step 3 — denominator (total probability of being late): $$P(E) = \frac{3}{10} \cdot \frac{1}{4} + \frac{1}{5} \cdot \frac{1}{3} + \frac{1}{10} \cdot \frac{1}{12} + \frac{2}{5} \cdot 0 = \frac{3}{40} + \frac{1}{15} + \frac{1}{120} = \frac{9 + 8 + 1}{120} = \frac{18}{120} = \frac{3}{20}$$

Step 4 — Bayes: $$P(T_1|E) = \frac{\frac{3}{10} \times \frac{1}{4}}{\frac{3}{20}} = \frac{\frac{3}{40}}{\frac{3}{20}} = \frac{1}{2}$$

Answer: $\dfrac{1}{2}$.

Example 6 — The truth-teller and the die

A man speaks the truth 3 out of 4 times. He throws a die and reports that it shows a six. Find the probability that it is actually a six.

Step 1 — hypotheses about the die: $S_1$ = "six occurred", $S_2$ = "six did not occur"; $P(S_1) = \frac{1}{6}$, $P(S_2) = \frac{5}{6}$.

Step 2 — conditionals for $E$ = "he reports a six": he reports six either by telling the truth when it is six, or by lying when it is not: $P(E|S_1) = \frac{3}{4}$, $P(E|S_2) = \frac{1}{4}$.

Step 3 — Bayes: $$P(S_1|E) = \frac{\frac{1}{6} \times \frac{3}{4}}{\frac{1}{6} \times \frac{3}{4} + \frac{5}{6} \times \frac{1}{4}} = \frac{\frac{3}{24}}{\frac{3}{24} + \frac{5}{24}} = \frac{3}{8}$$

Answer: $\dfrac{3}{8}$. Even a mostly-honest report of a rare event is more likely false than true — the same base-rate logic as the disease test.

Example 7 — Total probability with a changing urn

An urn contains 5 red and 5 black balls. A ball is drawn at random, its colour noted, and it is returned along with 2 more balls of the same colour; then a second ball is drawn. What is the probability that the second ball is red?

Step 1 — partition by the first draw: $R_1$ (first red) and $B_1$ (first black), each with probability $\frac{5}{10} = \frac{1}{2}$.

Step 2 — conditionals for the second draw (urn now has 12 balls): after red, it holds 7 red and 5 black, so $P(R_2|R_1) = \frac{7}{12}$; after black, 5 red and 7 black, so $P(R_2|B_1) = \frac{5}{12}$.

Step 3 — total probability: $$P(R_2) = \frac{1}{2} \times \frac{7}{12} + \frac{1}{2} \times \frac{5}{12} = \frac{7 + 5}{24} = \frac{1}{2}$$

Answer: $\dfrac{1}{2}$ — the same as the first draw. The two lopsided conditionals average out perfectly by symmetry.

Example 8 — Know or guess?

On a multiple-choice test, a student knows the answer with probability $\frac{3}{4}$ and guesses with probability $\frac{1}{4}$. A guess is correct with probability $\frac{1}{4}$; knowing gives a correct answer for sure. Given that the student answered correctly, what is the probability that the student actually knew the answer?

Step 1 — hypotheses: $K$ = "knows" ($P = \frac{3}{4}$), $G$ = "guesses" ($P = \frac{1}{4}$).

Step 2 — conditionals for $C$ = "answers correctly": $P(C|K) = 1$, $P(C|G) = \frac{1}{4}$.

Step 3 — Bayes: $$P(K|C) = \frac{\frac{3}{4} \times 1}{\frac{3}{4} \times 1 + \frac{1}{4} \times \frac{1}{4}} = \frac{\frac{3}{4}}{\frac{3}{4} + \frac{1}{16}} = \frac{\frac{12}{16}}{\frac{13}{16}} = \frac{12}{13}$$

Answer: $\dfrac{12}{13}$. A correct answer is strong (but not conclusive) evidence of knowledge, because guessing rarely produces one.