Science · Scientific discovery & evidence

What a statistical test actually decides

Set up a null and an alternative hypothesis and see which errors a test can make. Then read a result for the narrow claim it actually supports.

4 short lessons · about 21 minutes · 4 sources · updated November 2026

A paper figure examines a machine of gears through a blue magnifying glass. A peach gear on the far side, outside the glass's view, has a small crack.

What you will learn

  • Write a null and an alternative hypothesis for a claim stated in words
  • Say why a test has only two possible decisions and why neither of them means accepting the null
  • State the exact probability that a p-value reports
  • Compare a p-value with a significance level to reach a decision

Lesson 1 of 4 · 5 min

The two hypotheses every statistical test starts with

Before you readSomeone claims that more than 30% of a county's voters voted in the primary. A test of this claim starts by assuming which statement until the data say otherwise?

Correct.Not quite. The answer is At most 30% voted.

The test assumes the null hypothesis, here that the proportion is at most 0.30. This lesson shows how to write the pair of statements a test starts with.

Someone claims that more than 30% of registered voters in Santa Clara County voted in the primary. You have a sample of voters and want to check. Before any arithmetic, a hypothesis test makes you write the claim as two statements that contradict each other.

The first is the null hypothesis. It's a statement of no difference, no effect or no relationship, and it's usually the status quo. The second is the alternative hypothesis. It contradicts the null, and it's usually what the researcher is trying to establish.

One rule makes the pair easy to write. The null always uses a symbol with an equal sign in it, meaning equal to, at least or at most. The alternative never does. It uses not equal to, greater than or less than.

For the voters, the claim is "more than 30%", so that goes in the alternative. The null is that the proportion is at most 0.30.

Two more. To test whether mean grade point average differs from 2.0, the null is that it equals 2.0. To test whether students take less than five years to graduate, the null is that the mean is at least five years.

Check yourselfA baker claims the mean height of their loaves is more than 15 centimeters. Which pair of hypotheses fits a test of that claim?

Correct.Not quite. The answer is The null says the mean is at most 15 centimeters, and the alternative says it's greater.

The claim being argued for goes into the alternative, and "more than" becomes "greater than", which has no equal sign. The null has to contradict it and carry the equal sign, which makes it at most 15 centimeters. A null of less than 15 has no equal sign and leaves out 15 itself.

Because the two statements contradict each other, the sample has one job. It decides whether there's enough evidence to reject the null. Only two decisions are available. Reject the null if the sample favors the alternative, or do not reject it if the evidence is insufficient.

Notice what's missing. There's no option to accept the null, and that's deliberate. Failing to find evidence against a statement isn't the same as finding evidence for it. An inspector who finds no fault in a machine hasn't certified that the machine is perfect.

So the null is normally the boring statement, the one the test tries to knock down. It survives by default whenever the data are quiet. That's why the choice matters. Whatever you put in the null gets the benefit of the doubt.

Try to recallA colleague's hypothesis test found no significant difference between two groups, and they conclude that the groups are identical. Say precisely what's wrong.

The test offers only two decisions, reject the null or do not reject it, and there's no decision to accept the null. A result that doesn't reject means the sample information was insufficient to rule the null out, which isn't evidence that it's true. The two groups might well differ by an amount this sample couldn't detect, so the honest report is that no difference was demonstrated rather than that none exists.

So before any data arrive, the setup has fixed what the answer can look like. The null carries the equal sign and the benefit of the doubt. The alternative carries the claim being argued for.

And the verdict will be one of two phrases, reject or do not reject. "The null is true" isn't one of them.

Lessons in this topic

  1. The two hypotheses every statistical test starts with

    A test starts with two statements that can't both be true. How you write them decides what the test can and can't conclude.

    Essentials · 5 min

  2. What a p-value actually tells you

    The p-value asks how surprising the data would be if the null were true, which is a narrower question than the one most people think it answers.

    Essentials · 5 min

  3. The two ways a statistical test can be wrong

    A test has four possible outcomes, and two of them are mistakes. Deciding which mistake matters more is a judgment the statistics can't make for you.

    Deeper · 6 min

  4. What a test result doesn't tell you

    A significant result is a narrow claim, and a non-significant one is narrower still. Reading either as more than that undoes the care taken to get it.

    Deeper · 5 min

Three quick stories from this topic

Same error, different cost

One statistical error is trivial for climbing gear and fatal for a crash victim

Wrongly reject the assumption that climbing gear is safe, and you replace gear that was fine. Wrongly reject the assumption that a crash victim is alive, and nobody treats them. So how strict a test should be depends on what follows a mistake.

Two ways to be wrong

A Type I error rejects a starting assumption that was true. A Type II error keeps one that was false. Every test risks both.

Why the worse error flips

For the gear, the dangerous mistake is the other one, trusting equipment that's unsafe. Which error matters more depends on what someone does next, and the statistics can't decide that.

No difference found

"No significant difference" between two treatments doesn't mean they work equally well

A test has only two verdicts, reject the starting assumption or fail to reject it. Failing to reject means the evidence was too thin to tell, and a study that's too small will usually end there. It's like not finding your keys after a quick glance in the bag.

Only two verdicts

The test starts by assuming no difference and asks whether the data are surprising enough to reject that. It never calculates how likely the assumption is, so it can't confirm it.

Small studies miss real effects

The chance of detecting a real effect is called the power of a test, and a bigger sample can raise it. A low-powered study often comes back quiet whether or not the effect exists.

What a p-value is not

A p-value of 0.03 doesn't mean a 3% chance that the result is a fluke

It means that if there were truly no effect, results at least this extreme would turn up in 3% of samples. In 2016 the American Statistical Association issued a formal statement that p-values don't measure the probability that a result came from chance alone.

A p-value isn't the chance that the null is true

The p-value assumes the null and asks about the data. The probability that the null is true given the data is a different conditional probability, and the test never produces it.

The significance level is set in advance

Compare the p-value with it. If the p is low the null must go, and if the p is high the null must fly. Choosing the level after seeing the p-value amounts to choosing the answer.

Sources

This topic is part of Science past the easy answer, a guided path through related topics.

  • What makes an experiment convincing?

    Set up a fair comparison, reduce bias, and judge how far an experiment's result can support a claim.

    4 lessons · 14 min

  • How science works

    Science tests ideas against the world and against other scientists. See how a guess becomes a test, what 'theory' really means, how sciences without labs test ideas, and how errors get caught.

    4 lessons · 16 min

  • Albert Einstein

    A patent examiner published four papers in 1905 that changed physics, then spent years on a new theory of gravity. See what he did, how he argued with quantum theory, and his life as a refugee and campaigner.

    4 lessons · 16 min

  • Food webs and ecological knock-on effects

    Read who eats whom, tell the flow of energy apart from the recycling of materials, and reason carefully about indirect ecological effects.

    4 lessons · 14 min

  • Atoms and elements

    All ordinary matter is made of atoms, and each element is one kind of atom. See what's inside an atom, why its proton count makes gold gold, and how we know atoms are real when no one can see one.

    4 lessons · 15 min

  • How we know the age of rocks

    Rock layers and fossils give the order of Earth's history, and radioactive decay gives the dates. See how the two are joined, and why Earth's own age had to come from a meteorite.

    4 lessons · 22 min

All science topics · Take the quiz