Bayes' Theorem

Partition of a Sample Space

To understand Bayes' Theorem, we first need the idea of a partition of the sample space.

A set of events $E_1, E_2, \dots, E_n$ is said to form a partition of the sample space $S$ if:

1. They are pairwise disjoint: $$E_i \cap E_j = \varnothing \quad \text{for } i \ne j$$

2. They are exhaustive: $$E_1 \cup E_2 \cup \cdots \cup E_n = S$$

3. Each event has non-zero probability: $$P(E_i) > 0 \quad \text{for all } i$$

Thus, every outcome of the experiment belongs to exactly one of the events $E_1, E_2, \dots, E_n$.

Theorem on Total Probability

Let $E_1, E_2, \dots, E_n$ be a partition of the sample space $S$, and let $A$ be any event. Then the Theorem of Total Probability states that $$P(A) = P(E_1)P(A|E_1) + P(E_2)P(A|E_2) + \cdots + P(E_n)P(A|E_n)$$

Equivalently, $$P(A) = \sum_{j=1}^{n} P(E_j)P(A|E_j)$$

This theorem is useful when an event $A$ can occur through several mutually exclusive and exhaustive causes $E_1, E_2, \dots, E_n$.

Bayes' Theorem (Inverse Probability)

While the Theorem of Total Probability computes the probability of an observed event $A$, Bayes' Theorem computes the probability that a particular cause $E_i$ was responsible for the occurrence of $A$.

If $E_1, E_2, \dots, E_n$ form a partition of the sample space and $A$ is an event with $P(A) > 0$, then $$P(E_i|A) = \frac{P(E_i)P(A|E_i)}{\sum_{j=1}^{n} P(E_j)P(A|E_j)}$$

Here:

• $P(E_i)$ are the prior probabilities.
• $P(A|E_i)$ are the likelihoods.
• $P(E_i|A)$ is the posterior probability.

Example 1: The Three Urns Problem

Bag I contains 3 red and 4 black balls while Bag II contains 5 red and 6 black balls. One ball is drawn at random from one of the bags and it is found to be red. Find the probability that it was drawn from Bag II.

Solution: Let $$E_1 = \text{choosing Bag I}, \qquad E_2 = \text{choosing Bag II}$$ Since the bag is chosen at random, $$P(E_1) = \frac{1}{2}, \qquad P(E_2) = \frac{1}{2}$$

Let $$A = \text{drawing a red ball}$$ Then $$P(A|E_1) = \frac{3}{7}, \qquad P(A|E_2) = \frac{5}{11}$$

By Bayes' Theorem, $$P(E_2|A)=\frac{P(E_2)P(A|E_2)}{P(E_1)P(A|E_1)+P(E_2)P(A|E_2)}$$ Substituting, $$P(E_2|A)=\frac{\left(\frac{1}{2}\right)\left(\frac{5}{11}\right)}{\left(\frac{1}{2}\right)\left(\frac{3}{7}\right)+\left(\frac{1}{2}\right)\left(\frac{5}{11}\right)}$$ Cancelling $\frac{1}{2}$ from numerator and denominator, $$P(E_2|A)=\frac{\frac{5}{11}}{\frac{3}{7}+\frac{5}{11}}=\frac{\frac{5}{11}}{\frac{33+35}{77}}=\frac{\frac{5}{11}}{\frac{68}{77}}=\frac{5}{11}\cdot\frac{77}{68}=\frac{35}{68}$$

Answer: $$\boxed{\frac{35}{68}}$$

Example 2: Medical Diagnostic Test

A laboratory blood test is 99% effective in detecting a certain disease when it is, in fact, present. However, the test also yields a false positive result for 0.5% of healthy persons tested. If 0.1% of the population actually has the disease, what is the probability that a person has the disease given that the test result is positive?

Solution: Let $$E_1 = \text{person has the disease}, \qquad E_2 = \text{person is healthy}$$ Then $$P(E_1)=0.001, \qquad P(E_2)=0.999$$

Let $$A = \text{test result is positive}$$ Given, $$P(A|E_1)=0.99, \qquad P(A|E_2)=0.005$$

By Bayes' Theorem, $$P(E_1|A)=\frac{P(E_1)P(A|E_1)}{P(E_1)P(A|E_1)+P(E_2)P(A|E_2)}$$ So, $$P(E_1|A)=\frac{(0.001)(0.99)}{(0.001)(0.99)+(0.999)(0.005)}$$ $$P(E_1|A)=\frac{0.00099}{0.00099+0.004995}=\frac{0.00099}{0.005985}$$ Multiplying numerator and denominator by $100000$, $$P(E_1|A)=\frac{99}{598.5}=\frac{198}{1197}=\frac{22}{133}$$ Hence, $$P(E_1|A)=\frac{22}{133} \approx 0.1654$$

Answer: $$\boxed{\frac{22}{133} \approx 0.1654}$$

Example 3: Manufacturing Defects

In a factory, machine A produces 60% of the items and machine B produces 40%. 2% of items from A are defective and 1% from B are defective. If an item is chosen at random and found to be defective, find the probability that it was produced by machine B.

Solution: Let $$E_A = \text{item produced by machine A}, \qquad E_B = \text{item produced by machine B}$$ Then $$P(E_A)=0.6, \qquad P(E_B)=0.4$$

Let $$D = \text{item is defective}$$ Then $$P(D|E_A)=0.02, \qquad P(D|E_B)=0.01$$

First, by the Theorem of Total Probability, $$P(D)=P(E_A)P(D|E_A)+P(E_B)P(D|E_B)$$ $$P(D)=(0.6)(0.02)+(0.4)(0.01)=0.012+0.004=0.016$$

Now, by Bayes' Theorem, $$P(E_B|D)=\frac{P(E_B)P(D|E_B)}{P(D)}=\frac{(0.4)(0.01)}{0.016}=\frac{0.004}{0.016}=\frac{1}{4}$$

Answer: $$\boxed{\frac{1}{4}}$$

Example 4: Student Exam Performance

In an entrance test, a student either knows the answer, guesses, or copies. The probability that he knows is $\frac{1}{2}$, guesses is $\frac{1}{4}$, and copies is $\frac{1}{4}$. The probability that the answer is correct given he knows is $1$, given he guesses is $\frac{1}{4}$, and given he copies is $\frac{1}{8}$. If he answered correctly, find the probability he knew the answer.

Solution: Let $$E_1 = \text{he knows the answer}, \quad E_2 = \text{he guesses}, \quad E_3 = \text{he copies}$$ Then $$P(E_1)=\frac{1}{2}, \qquad P(E_2)=\frac{1}{4}, \qquad P(E_3)=\frac{1}{4}$$

Let $$C = \text{answer is correct}$$ Then $$P(C|E_1)=1, \qquad P(C|E_2)=\frac{1}{4}, \qquad P(C|E_3)=\frac{1}{8}$$

First, $$P(C)=\frac{1}{2}\cdot 1+\frac{1}{4}\cdot\frac{1}{4}+\frac{1}{4}\cdot\frac{1}{8}$$ $$P(C)=\frac{1}{2}+\frac{1}{16}+\frac{1}{32}=\frac{16+2+1}{32}=\frac{19}{32}$$

Therefore, $$P(E_1|C)=\frac{P(E_1)P(C|E_1)}{P(C)}=\frac{\frac{1}{2}\cdot 1}{\frac{19}{32}}=\frac{1}{2}\cdot\frac{32}{19}=\frac{16}{19}$$

Answer: $$\boxed{\frac{16}{19}}$$

Example 5: Insurance Risk

An insurance company insured 2000 scooter drivers, 4000 car drivers and 6000 truck drivers. The probability of an accident is 0.01, 0.03 and 0.15 respectively. One of the insured persons meets with an accident. What is the probability that he is a scooter driver?

Solution: Let $$E_S = \text{scooter driver}, \qquad E_C = \text{car driver}, \qquad E_T = \text{truck driver}$$ Total number insured is $$2000+4000+6000=12000$$ Hence, $$P(E_S)=\frac{2000}{12000}=\frac{1}{6}, \qquad P(E_C)=\frac{4000}{12000}=\frac{1}{3}, \qquad P(E_T)=\frac{6000}{12000}=\frac{1}{2}$$

Let $$A = \text{person meets with an accident}$$ Then, $$P(A|E_S)=0.01=\frac{1}{100}, \qquad P(A|E_C)=0.03=\frac{3}{100}, \qquad P(A|E_T)=0.15=\frac{15}{100}=\frac{3}{20}$$

Now, $$P(A)=P(E_S)P(A|E_S)+P(E_C)P(A|E_C)+P(E_T)P(A|E_T)$$ $$P(A)=\frac{1}{6}\cdot\frac{1}{100}+\frac{1}{3}\cdot\frac{3}{100}+\frac{1}{2}\cdot\frac{3}{20}$$ $$P(A)=\frac{1}{600}+\frac{1}{100}+\frac{3}{40}$$ Taking LCM $600$, $$P(A)=\frac{1+6+45}{600}=\frac{52}{600}=\frac{13}{150}$$

Therefore, $$P(E_S|A)=\frac{P(E_S)P(A|E_S)}{P(A)}=\frac{\frac{1}{600}}{\frac{52}{600}}=\frac{1}{52}$$

Answer: $$\boxed{\frac{1}{52}}$$

Example 6: Card Lost from Deck

A card from a pack of 52 cards is lost. From the remaining cards, two cards are drawn and are found to be diamonds. Find the probability that the lost card was a diamond.

Solution: Let $$E_1 = \text{lost card is a diamond}, \qquad E_2 = \text{lost card is not a diamond}$$ Then $$P(E_1)=\frac{13}{52}=\frac{1}{4}, \qquad P(E_2)=\frac{39}{52}=\frac{3}{4}$$

Let $$A = \text{the two drawn cards are diamonds}$$ If the lost card was a diamond, then among the remaining $51$ cards there are $12$ diamonds. So, $$P(A|E_1)=\frac{\binom{12}{2}}{\binom{51}{2}}$$ If the lost card was not a diamond, then among the remaining $51$ cards there are still $13$ diamonds. So, $$P(A|E_2)=\frac{\binom{13}{2}}{\binom{51}{2}}$$

Using Bayes' Theorem, $$P(E_1|A)=\frac{P(E_1)P(A|E_1)}{P(E_1)P(A|E_1)+P(E_2)P(A|E_2)}$$ Substituting, $$P(E_1|A)=\frac{\frac{1}{4}\cdot\frac{\binom{12}{2}}{\binom{51}{2}}}{\frac{1}{4}\cdot\frac{\binom{12}{2}}{\binom{51}{2}}+\frac{3}{4}\cdot\frac{\binom{13}{2}}{\binom{51}{2}}}$$ Cancel the common factor $\frac{1}{4\binom{51}{2}}$: $$P(E_1|A)=\frac{\binom{12}{2}}{\binom{12}{2}+3\binom{13}{2}}$$ Now, $$\binom{12}{2}=66, \qquad \binom{13}{2}=78$$ So, $$P(E_1|A)=\frac{66}{66+3(78)}=\frac{66}{300}=\frac{11}{50}$$

Answer: $$\boxed{\frac{11}{50}}$$

Example 7: Truth Tellers

A man is known to speak truth 3 out of 4 times. He throws a die and reports that it is a six. Find the probability that it is actually a six.

Solution: Let $$E_1 = \text{the die actually shows } 6, \qquad E_2 = \text{the die does not show } 6$$ Then $$P(E_1)=\frac{1}{6}, \qquad P(E_2)=\frac{5}{6}$$

Let $$A = \text{the man reports that it is a six}$$ Since he speaks truth with probability $\frac{3}{4}$, $$P(A|E_1)=\frac{3}{4}$$ If the die is not a six, then he reports six only when he lies. Thus, $$P(A|E_2)=\frac{1}{4}$$

Therefore, $$P(E_1|A)=\frac{P(E_1)P(A|E_1)}{P(E_1)P(A|E_1)+P(E_2)P(A|E_2)}$$ $$P(E_1|A)=\frac{\frac{1}{6}\cdot\frac{3}{4}}{\frac{1}{6}\cdot\frac{3}{4}+\frac{5}{6}\cdot\frac{1}{4}}$$ Multiplying numerator and denominator by $24$, $$P(E_1|A)=\frac{3}{3+5}=\frac{3}{8}$$

Answer: $$\boxed{\frac{3}{8}}$$

Example 8: Multiple Choice Questions

In a multiple-choice test, an examinee either knows the answer or guesses. Let $p$ be the probability that he knows. The probability that an examinee who guesses will be correct is $\frac{1}{m}$. What is the probability that he knows the answer given that he answered it correctly?

Solution: Let $$K = \text{knows the answer}, \qquad G = \text{guesses}$$ Then, $$P(K)=p, \qquad P(G)=1-p$$

Let $$C = \text{answer is correct}$$ Then, $$P(C|K)=1, \qquad P(C|G)=\frac{1}{m}$$

By Bayes' Theorem, $$P(K|C)=\frac{P(K)P(C|K)}{P(K)P(C|K)+P(G)P(C|G)}$$ $$P(K|C)=\frac{p\cdot 1}{p\cdot 1+(1-p)\cdot\frac{1}{m}}$$ Multiply numerator and denominator by $m$: $$P(K|C)=\frac{mp}{mp+1-p}$$ Hence, $$P(K|C)=\frac{mp}{1+(m-1)p}$$

Answer: $$\boxed{\frac{mp}{1+(m-1)p}}$$

Example 9: Item Transferred between Bags

Bag A contains 2 white and 3 red balls and Bag B contains 4 white and 5 red balls. One ball is transferred from Bag A to Bag B and then a ball is drawn from Bag B. The ball so drawn is found to be white. Find the probability that the transferred ball was white.

Solution: Let $$E_W = \text{the transferred ball is white}, \qquad E_R = \text{the transferred ball is red}$$ Since Bag A has $2$ white and $3$ red balls, $$P(E_W)=\frac{2}{5}, \qquad P(E_R)=\frac{3}{5}$$

Let $$A = \text{the ball drawn from Bag B is white}$$ If a white ball is transferred, Bag B becomes $5$ white and $5$ red. Therefore, $$P(A|E_W)=\frac{5}{10}=\frac{1}{2}$$ If a red ball is transferred, Bag B becomes $4$ white and $6$ red. Therefore, $$P(A|E_R)=\frac{4}{10}=\frac{2}{5}$$

Now, by Bayes' Theorem, $$P(E_W|A)=\frac{P(E_W)P(A|E_W)}{P(E_W)P(A|E_W)+P(E_R)P(A|E_R)}$$ Substitute the values: $$P(E_W|A)=\frac{\frac{2}{5}\cdot\frac{1}{2}}{\frac{2}{5}\cdot\frac{1}{2}+\frac{3}{5}\cdot\frac{2}{5}}$$ $$P(E_W|A)=\frac{\frac{1}{5}}{\frac{1}{5}+\frac{6}{25}}=\frac{\frac{5}{25}}{\frac{11}{25}}=\frac{5}{11}$$

Answer: $$\boxed{\frac{5}{11}}$$

Example 10: Letter from a City

A letter is known to have come from either TATANAGAR or CALCUTTA. On the envelope, just two consecutive letters TA are visible. What is the probability that the letter came from TATANAGAR?

Solution: Let $$T = \text{letter came from TATANAGAR}, \qquad C = \text{letter came from CALCUTTA}$$ Assuming both are equally likely, $$P(T)=\frac{1}{2}, \qquad P(C)=\frac{1}{2}$$

Let $$A = \text{the visible consecutive letters are TA}$$

Now count the consecutive pairs.

For TATANAGAR: The word has $9$ letters, so it has $8$ consecutive pairs: $$TA,\ AT,\ TA,\ AN,\ NA,\ AG,\ GA,\ AR$$ The pair $TA$ appears $2$ times. Hence, $$P(A|T)=\frac{2}{8}=\frac{1}{4}$$

For CALCUTTA: The word has $8$ letters, so it has $7$ consecutive pairs: $$CA,\ AL,\ LC,\ CU,\ UT,\ TT,\ TA$$ The pair $TA$ appears $1$ time. Hence, $$P(A|C)=\frac{1}{7}$$

By Bayes' Theorem, $$P(T|A)=\frac{P(T)P(A|T)}{P(T)P(A|T)+P(C)P(A|C)}$$ $$P(T|A)=\frac{\frac{1}{2}\cdot\frac{1}{4}}{\frac{1}{2}\cdot\frac{1}{4}+\frac{1}{2}\cdot\frac{1}{7}}$$ Cancelling $\frac{1}{2}$, $$P(T|A)=\frac{\frac{1}{4}}{\frac{1}{4}+\frac{1}{7}}=\frac{\frac{1}{4}}{\frac{11}{28}}=\frac{7}{11}$$

Answer: $$\boxed{\frac{7}{11}}$$