Mathematics & Logic · Probability & statistics
Base rates and what a test result means
Count true and false alarms in a made-up factory to see why the meaning of a test result depends on how common the problem was before testing.

What you will learn
- Calculate true and false positives from stated group sizes and test rates
- Explain why a detection rate isn't the probability of a defect after a positive result
- Calculate how a higher base rate changes the share of positives that are true
- Explain why a test's performance alone can't tell you what a positive result means
Lesson 1 of 4 · 4 min
Why most alarms from a good detector can be false
Picture a factory that screens 10,000 parts. One in a hundred is defective, so 100 are defective and 9,900 are good. Its detector catches 90% of defects and wrongly flags 1% of good parts.
The alarm goes off for one part. How likely is that part to be defective? The tempting answer is 90%. The real one is under half, and you can see why by counting both ways a part can set off the alarm.
The detector catches 90% of the 100 defective parts, so expect 90 alarms. Those are true positives. It wrongly flags 1% of the 9,900 good parts, which makes 99 more alarms. Those are false positives.
That's 189 alarms, or positive results, and only 90 come from defective parts. So the chance that a flagged part is defective is 90 out of 189, about 47.6%. More of the alarms are false than true.
The 90% is the detector's sensitivity, the share of defective parts it catches. It starts from the defective parts and asks how many set off the alarm. Your question starts from the alarms and asks how many are defective. Swapping the direction changes the group you're counting in.
The swap fools experts. In 2007, 160 gynecologists were given a screening problem of the same shape. In it, 1% of women had breast cancer, and the test caught 90% of cancers and falsely flagged 9% of healthy women. About 1 in 10 positives is a real cancer. Only 21% chose that answer, and most said 81% or 90%.
How common a problem is before any testing is called the base rate. Here it's 1%. That makes the pile of good parts 99 times the size of the defective pile. So a 1% error rate on the big pile produces about as many alarms as a 90% catch rate on the small one.
The detector still earns its keep. An alarm lifts a part's chance of being defective from 1% to 47.6%. That's a big jump, just a smaller one than the 90% suggests.
Try to recallWhy is a detector's 90% sensitivity not the same as a 90% chance that a part that tests positive is defective?
Sensitivity counts positives among the defective parts. The chance of a defect after a positive counts defective parts among all the positives, false alarms included, and it depends on the base rate.
After a training session on turning the percentages into counts of women out of 1,000, 87% of those 160 gynecologists got the screening problem right.
To read a positive result, count every route into the positive group. The false alarms belong in the total too.
Lessons in this topic
Why most alarms from a good detector can be false
Count true and false positives in a made-up factory to see why a detector that catches 90% of defects can still be wrong about most of the parts it flags.
Why the same alarm means more in a worse batch
Run the same detector on a batch with ten times as many defects, and watch the meaning of a positive result jump from under half to over 90%.
When two positives don't mean twice the evidence
Compare a repeat test that makes its own errors with one that repeats the first test's mistake, and see why only the first kind counts as new evidence.
How much a negative result should reassure you
Count the defects a detector misses to see when a negative result is reassuring and when it isn't.
Three quick stories from this topic
When a negative result reassures
A negative test result is most trustworthy when the problem it looks for is rare
Take a made-up detector that misses 10% of defects. If 1% of parts are defective, a pass is wrong about once in a thousand. If half are defective, the same pass is wrong about once in eleven, because there are far more defects to miss.
Count where negatives come from
Negatives come from good parts the detector clears and defective parts it misses. At 1% defective, the detector misses 10 parts and clears 9,801, so 10 of 9,811 negatives are defective. At 50% it misses 500 and clears 4,950, so 500 of 5,450 negatives are defective. The detector never changed.
It's the mirror image of the positive case
A rare problem makes positives less reliable and negatives more reliable, and a common problem does the reverse. The rates here are invented and assume the detector performs the same on every batch, which a real test may not.
Doctors and positive mammograms
Given the numbers, only 21% of 160 gynecologists knew what a positive mammogram means
The 2007 problem said 1% of women had breast cancer, and the test caught 90% of cancers and falsely flagged 9% of healthy women. About 1 in 10 positives is then a real cancer. Most of the doctors answered 81% or 90%.
Count it out
Of 1,000 women, 10 have cancer and 9 of them test positive. Of the 990 without cancer, about 89 also test positive. So 9 of 98 positives are real.
Counts fixed it
After a training session on turning the percentages into counts like these, 87% of the same doctors chose the right answer.
A repeat test can add nothing
A second positive test adds nothing if it repeats the first test's mistake
With a made-up detector, one alarm means a 48% chance of a real defect. A second alarm from an independent test lifts that to about 99%. A second alarm from a scanner fooled by the same harmless mark leaves it at 48%.
A perfect repeat adds nothing new
If the second result always matches the first, it tells you nothing more once you know the first.
Check the assumption before multiplying
You can treat two false alarms as independent only if there's evidence their errors are independent among the good parts.
Sources
- 3.3 Two Basic Rules of Probability - Introductory Statistics 2e, OpenStax
- Seeing Theory - Bayesian Inference, Brown University, Seeing Theory
- Helping Doctors and Patients Make Sense of Health Statistics, Gigerenzer, Gaissmaier, Kurz-Milcke, Schwartz and Woloshin, Psychological Science in the Public Interest 8(2), copy held by the Max Planck Institute for Human Development





