Science · Scientific discovery & evidence
What makes an experiment convincing?
Set up a fair comparison, reduce bias, and judge how far an experiment's result can support a claim.

What you will learn
- Pick out the treatment, comparison and outcome in a simple experiment.
- Explain how a confounding difference weakens a conclusion about cause.
- Explain how random assignment reduces systematic differences between groups at the start.
- Explain how blinding can reduce measurement bias driven by expectations.
Lesson 1 of 4 · 3 min
What makes a comparison fair?
In the 1800s, the doctor Ignaz Semmelweis worked on a maternity ward where an unusually high share of new mothers died of childbed fever. One explanation blamed giving birth lying on the back. He had women labor on their sides, and the fever rate didn't fall.
Another blamed the doctors' hands, which often came straight from autopsies. He had the doctors wash with a strong disinfectant, and the rate plummeted. A comparison like that is built to pull rival explanations apart.
Say a school wants to know whether a new practice exercise works. The treatment is what you deliberately change, here the exercise. The outcome is what you measure to judge it. The comparison group gets something else, such as the usual practice.
Everything else should match as closely as the design allows, including time and materials. Then the exercise is the only difference left to explain a gap in scores.
Decide what the outcome means before you look at any results. "Learning" could mean a quiz score that afternoon, an explanation given a week later, or success on a new kind of problem. Those aren't interchangeable.
An operational definition pins it down. It says exactly how the idea will be measured, so that someone else can follow the procedure and repeat it.
A confound is a second difference between the groups that could explain the result. Say the new-method group also gets twice as much practice time. Now a higher score can't tell you which change mattered, the method or the time.
The design has bundled method and time together, and no amount of careful scoring afterward will separate them.
The fix belongs in the design. Give both groups the same practice time, and time can no longer explain a difference in scores.
No experiment controls every detail. A good one makes the main rival explanations less plausible, and it reports the differences it couldn't remove.
Try to recallOne group uses a new study method for an hour, and another uses the old method for ten minutes. Why can't the difference in their scores isolate the method's effect?
The method and the practice time changed together. The extra time could explain the score difference by itself, so the comparison confounds method with time.
Lessons in this topic
What makes a comparison fair?
Turn a question about cause into a comparison, and spot the second difference that can spoil it.
Why experiments assign by chance and hide the labels
See how random assignment and blinding each guard against a different kind of bias.
Why one result is only a start
Tell a consistent measurement from a valid one, then see what repeating a study adds and who a result applies to.
When you can't run the experiment
Read a correlation for its strength and direction, then look for a third variable before you believe in a cause.
Three quick stories from this topic
Semmelweis's two tests
Hand washing beat a rival theory that mothers died from giving birth on their backs
Ignaz Semmelweis tested both ideas on his 1800s maternity ward. Having women labor on their sides left childbed fever unchanged. Having doctors wash their hands with a strong disinfectant made the rate plummet.
Why the hands were suspect
Doctors on the ward often performed autopsies immediately before examining women in labor. If unclean hands caused the fever, washing them should lower the rate, and it did.
One change at a time
Each test changed a single thing and watched the same outcome. That's what lets a comparison tell rival explanations apart. Bundle two changes together and you can't say which one mattered.
A Nobel laureate's retraction
A Nobel laureate retracted her own paper in 2020 because the work couldn't be reproduced
Frances Arnold shared the 2018 Nobel Prize in chemistry. A paper she co-authored in Science the following year was withdrawn after efforts to repeat the experiments failed. She announced the retraction herself.
What replication is for
Replication means repeating a study to see whether the result shows up again. A success adds support, and repeated failures cast doubt on the original claim.
No result is exempt
A finding's standing depends on whether it holds up when repeated, whoever published it. That's why a study has to describe its methods clearly enough for others to try.
The full Moon myth
A review of nearly 40 studies found the full Moon doesn't change how people behave
Rates of odd behavior stay constant through the lunar cycle. People pay more attention to strange behavior when the Moon is full, so the nights that fit the belief are the ones that get noticed.
How a false pattern survives
An illusory correlation forms when people believe two things are related even though they aren't, often based on unsystematic observation. Confirmation bias keeps it going. A hunch leads people to notice the cases that fit and overlook the ones that don't, and examples that come easily to mind feel like evidence.
Why it matters beyond folklore
Research links illusory correlations to prejudice, when behaviors get wrongly attributed to whole groups. The remedy is the same as for any claimed relationship. Check it against systematic data before treating the pattern as real.
Sources
- Analyzing findings, OpenStax, Psychology 2e
- Testing scientific ideas, University of California Museum of Paleontology
- Nobel Prize-winning scientist Frances Arnold retracts paper, BBC News, January 3, 2020





