Science · Scientific discovery & evidence
What a statistical test actually decides
Set up a null and an alternative hypothesis and see which errors a test can make. Then read a result for the narrow claim it actually supports.

What you will learn
- Write a null and an alternative hypothesis for a claim stated in words
- Say why a test has only two possible decisions and why neither of them means accepting the null
- State the exact probability that a p-value reports
- Compare a p-value with a significance level to reach a decision
Lesson 1 of 4 · 5 min
The two hypotheses every statistical test starts with
Someone claims that more than 30% of registered voters in Santa Clara County voted in the primary. You have a sample of voters and want to check. Before any arithmetic, a hypothesis test makes you write the claim as two statements that contradict each other.
The first is the null hypothesis. It's a statement of no difference, no effect or no relationship, and it's usually the status quo. The second is the alternative hypothesis. It contradicts the null, and it's usually what the researcher is trying to establish.
One rule makes the pair easy to write. The null always uses a symbol with an equal sign in it, meaning equal to, at least or at most. The alternative never does. It uses not equal to, greater than or less than.
For the voters, the claim is "more than 30%", so that goes in the alternative. The null is that the proportion is at most 0.30.
Two more. To test whether mean grade point average differs from 2.0, the null is that it equals 2.0. To test whether students take less than five years to graduate, the null is that the mean is at least five years.
Because the two statements contradict each other, the sample has one job. It decides whether there's enough evidence to reject the null. Only two decisions are available. Reject the null if the sample favors the alternative, or do not reject it if the evidence is insufficient.
Notice what's missing. There's no option to accept the null, and that's deliberate. Failing to find evidence against a statement isn't the same as finding evidence for it. An inspector who finds no fault in a machine hasn't certified that the machine is perfect.
So the null is normally the boring statement, the one the test tries to knock down. It survives by default whenever the data are quiet. That's why the choice matters. Whatever you put in the null gets the benefit of the doubt.
Try to recallA colleague's hypothesis test found no significant difference between two groups, and they conclude that the groups are identical. Say precisely what's wrong.
The test offers only two decisions, reject the null or do not reject it, and there's no decision to accept the null. A result that doesn't reject means the sample information was insufficient to rule the null out, which isn't evidence that it's true. The two groups might well differ by an amount this sample couldn't detect, so the honest report is that no difference was demonstrated rather than that none exists.
So before any data arrive, the setup has fixed what the answer can look like. The null carries the equal sign and the benefit of the doubt. The alternative carries the claim being argued for.
And the verdict will be one of two phrases, reject or do not reject. "The null is true" isn't one of them.
Lessons in this topic
The two hypotheses every statistical test starts with
A test starts with two statements that can't both be true. How you write them decides what the test can and can't conclude.
What a p-value actually tells you
The p-value asks how surprising the data would be if the null were true, which is a narrower question than the one most people think it answers.
The two ways a statistical test can be wrong
A test has four possible outcomes, and two of them are mistakes. Deciding which mistake matters more is a judgment the statistics can't make for you.
What a test result doesn't tell you
A significant result is a narrow claim, and a non-significant one is narrower still. Reading either as more than that undoes the care taken to get it.
Three quick stories from this topic
Same error, different cost
One statistical error is trivial for climbing gear and fatal for a crash victim
Wrongly reject the assumption that climbing gear is safe, and you replace gear that was fine. Wrongly reject the assumption that a crash victim is alive, and nobody treats them. So how strict a test should be depends on what follows a mistake.
Two ways to be wrong
A Type I error rejects a starting assumption that was true. A Type II error keeps one that was false. Every test risks both.
Why the worse error flips
For the gear, the dangerous mistake is the other one, trusting equipment that's unsafe. Which error matters more depends on what someone does next, and the statistics can't decide that.
No difference found
"No significant difference" between two treatments doesn't mean they work equally well
A test has only two verdicts, reject the starting assumption or fail to reject it. Failing to reject means the evidence was too thin to tell, and a study that's too small will usually end there. It's like not finding your keys after a quick glance in the bag.
Only two verdicts
The test starts by assuming no difference and asks whether the data are surprising enough to reject that. It never calculates how likely the assumption is, so it can't confirm it.
Small studies miss real effects
The chance of detecting a real effect is called the power of a test, and a bigger sample can raise it. A low-powered study often comes back quiet whether or not the effect exists.
What a p-value is not
A p-value of 0.03 doesn't mean a 3% chance that the result is a fluke
It means that if there were truly no effect, results at least this extreme would turn up in 3% of samples. In 2016 the American Statistical Association issued a formal statement that p-values don't measure the probability that a result came from chance alone.
A p-value isn't the chance that the null is true
The p-value assumes the null and asks about the data. The probability that the null is true given the data is a different conditional probability, and the test never produces it.
The significance level is set in advance
Compare the p-value with it. If the p is low the null must go, and if the p is high the null must fly. Choosing the level after seeing the p-value amounts to choosing the answer.
Sources
- 9.1 Null and Alternative Hypotheses - Introductory Statistics 2e, OpenStax, Rice University
- 9.2 Outcomes and the Type I and Type II Errors - Introductory Statistics 2e, OpenStax, Rice University
- 9.4 Rare Events, the Sample, Decision and Conclusion - Introductory Statistics 2e, OpenStax, Rice University
- American Statistical Association releases statement on statistical significance and p-values, American Statistical Association, March 7, 2016





