Body & Health · Medicine & public health
Reading a claim about a treatment
Turn a medical headline into a clear question about comparisons, outcomes, harms and the people studied.

What you will learn
- Identify the population, outcome and comparator you need to interpret a treatment claim
- Distinguish a personal testimonial from a useful comparison of treatments
- Explain the different purposes of a comparison group and random allocation
- Explain how blinding can reduce bias without guaranteeing a flawless study
Lesson 1 of 4 · 4 min
What a headline about a treatment leaves out
In July 2002 a large US trial reported that hormone pills taken after menopause raised the risk of breast cancer by 26%. The number was accurate. Standing alone, it told a reader very little.
A risk 26% higher than what? In which women? Over how long? A claim about a treatment only means something once it names four things. Those are the people studied, the treatment, what it was compared with and the outcome that was measured.
Now fill in the four. The trial was the Women's Health Initiative. The people were 16,608 healthy postmenopausal women aged 50 to 79. The treatment was a daily pill of estrogen plus progestin. The comparison was a placebo, a dummy pill. The outcome was invasive breast cancer over an average of 5.2 years.
The 26% is a relative figure. It says how much the risk changed, whatever it was to begin with. In absolute terms the same result was 8 more invasive breast cancers per 10,000 women per year. Both numbers are true. The absolute one tells you how many people a change touches, which is what someone weighing a treatment needs.
And breast cancer was one outcome among several. The same pills meant 5 fewer hip fractures per 10,000 women a year. Any single headline number leaves part of the result out.
Now take a testimonial. Someone starts a treatment, feels better and says it works. They may be completely sincere, but their story has no comparison in it. The order of events can't tell you how they'd have done without the treatment.
The condition may have eased by itself. Other things in their life may have changed. Hope can shape how improvement feels. That's why trials compare groups assigned at random, like the placebo group in the hormone trial. Whatever else is going on happens to both groups, so a difference between them points to the treatment.
A story is still worth something, because it can raise a question worth testing. Just keep two claims apart when you read one. "This person reports feeling better" is about one person, and you can take it at face value. "This treatment caused it and will help others" is about the treatment, and it needs a comparison group.
Try to recallA testimonial says, "I tried it, then felt better, so it's the best treatment." What two missing pieces would you need to assess that broader claim?
First, a comparison. How did similar people do on another treatment, or on none? Second, a defined outcome measured in a group of people, because "felt better" in one person can't show a treatment is best for others. Who was studied and what harms turned up would help too. None of this means doubting that the person felt better.
When a claim says a treatment works, ask four questions. Who was studied? What was it compared with? What outcome was measured, and over how long? Then ask for the absolute numbers. A 26% higher risk and 8 more cases per 10,000 women a year were the same finding.
Lessons in this topic
What a headline about a treatment leaves out
Find the people, the comparison and the outcome behind a claim that a treatment works or harms, and see why one number is never the whole result.
Three ways a trial keeps a comparison fair
See what a control group, random allocation and blinding each do, starting with a naval surgeon who tested six scurvy cures on 12 sailors.
How far a trial's result can be stretched
See what the people studied and the length of follow-up leave unanswered, and why a benefit should always be read next to its harms.
Where the hype in health news comes from
A study of 462 university press releases traced exaggerated health news to its source. Learn the three kinds of exaggeration it measured and how to trace a headline back yourself.
Three quick stories from this topic
The scurvy trial of 1747
In 1747 a naval surgeon tested six scurvy cures on 12 sailors, and citrus won
James Lind gave pairs of sick sailors cider, vinegar, sea water, dilute sulfuric acid, a garlic and mustard paste, or two oranges and a lemon a day. One of the citrus pair was fit for duty in six days. Comparing the pairs showed what no single recovery could.
Kept alike on purpose
Lind chose sailors who were "as similar as I could have them," housed them in the same quarters and fed them the same basic diet. Only the remedy differed.
A 48-year wait
The British Admiralty didn't order lemon juice for the navy until 1795, 48 years after Lind's trial.
One result, two numbers
A 26% rise in breast cancer risk meant 8 extra cases a year for every 10,000 women
Both figures describe one result from the 2002 Women's Health Initiative trial of hormone pills taken after menopause. The 26% compares the pill group with the placebo group. The 8 in 10,000 says how many women were affected, which is what someone weighing the pills needs to know.
The same trial's other numbers
Per 10,000 women a year, the hormone group also had 7 more coronary events and 8 more strokes, with 5 fewer hip fractures and 6 fewer colorectal cancers.
Ask for both
A relative figure tells you how much a risk changed. An absolute figure tells you how many people that change touches.
Hype starts in the press release
In a UK study, most hype in health news was already in the university's press release
Researchers compared 462 press releases from 20 UK universities with the research papers behind them and the news that followed. When a release turned a correlation into a cause, 81% of news stories did too. When it didn't, 18% did.
Approved by the scientists
Most university press releases are drafted with the researchers and aren't sent out without their approval.
Hype didn't pay
Press releases that exaggerated were about as likely to get news coverage as those that stuck to the paper.
Sources
- Clinical trials, NHS
- Evaluating Health Information, MedlinePlus, National Library of Medicine
- Risks and benefits of estrogen plus progestin in healthy postmenopausal women: principal results from the Women's Health Initiative randomized controlled trial, JAMA (PubMed record)
- Who was James Lind, and what exactly did he achieve?, The James Lind Library, JLL Bulletin (Iain Milne, Royal College of Physicians of Edinburgh)
- The association between exaggeration in health related science news and academic press releases: retrospective observational study, Sumner et al., BMJ 2014; PMC full text





