GeroStats
At dusk, a shepherd boy calls towards a village while a real wolf approaches his flock and the villagers turn away.

Statistical foundations · first of two connected guides

False alarms and missed detections.

Type I and Type II errors are the two ways a hypothesis-testing decision can be wrong.

Learning order: 1. Type I and Type II errors → 2. Multiple testing. Start here.
You will learn: false alarms, missed detections, α, β and statistical power.

A memorable story

False alarms can make a real warning harder to trust.

The boy who cried wolf

Day after day, a young shepherd watches his flock on the hillside above the village. The work is quiet, and one afternoon he decides to create some excitement. “Wolf!” he shouts. “A wolf is attacking the sheep!”

The villagers drop what they are doing and run up the steep path to help. When they arrive, there is no wolf. The boy laughs at how easily he fooled them. A few days later he calls “Wolf!” again. Once more the villagers hurry to the flock, and once more they find no danger. Their concern turns to anger. They warn him that an alarm means something only when people can trust it.

Then, near dusk, the bushes at the edge of the field begin to move. A real wolf steps onto the hillside and stalks towards the sheep. Terrified, the boy cries “Wolf! Please help. This time it is real!”

Down in the village, people hear him, but nobody starts up the path. They remember the earlier tricks and assume this is another false alarm. The boy’s warning is now true, yet it is treated as false. With no help coming, the wolf attacks the flock.

The decisive moment: a real wolf is present and the boy’s alarm is true, but the villagers’ earlier experience leads them to dismiss it.

Begin in plain English

What is a statistical test for?

A statistical test helps you judge whether an observed pattern is difficult to explain using a particular “no effect” model. It is a structured comparison, not a machine that announces whether a scientific claim is true.

  1. Write the question. Decide which population, outcome and comparison matter before looking for a small p-value.
  2. State H0 and H1. H0 supplies the model used for the comparison; H1 describes the departure the study is designed to detect.
  3. Turn the data into a test statistic. The statistic measures the observed difference or association relative to the variation expected in the data.
  4. Compare it with a null reference distribution. This shows how results would vary across repeated samples if H0 and the test’s assumptions were correct.
  5. Calculate a p-value and apply the planned rule. The p-value records how unusual the result is under that model. The rule says when the evidence is strong enough to reject H0.
  6. Return to the scientific question. Report the effect estimate, confidence interval, assumptions and practical importance. A test decision is only one part of the answer.

A statistical test does not calculate the probability that H0 is true, prove that H1 is true or tell you whether an effect is clinically important.

One test · four possibilities

Map reality against the decision.

Start with two hypotheses

H0: There is no wolf.
H1: There is a wolf.

The boy’s job is to perform a “wolf test”. Read the guide to null and alternative hypotheses if those statements are new to you.

RealityDecisionStatistical interpretation
No wolf“No wolf”Correct decision
No wolf“Wolf!”Type I error: false positive
Wolf present“Wolf!”Correct detection
Wolf present“No wolf”Type II error: false negative

Remember it this way:
Type I error = crying wolf when there is no wolf.
Type II error = failing to recognise the wolf when it really is there.

The boy’s final cry is a true positive, but the villagers make a false-negative decision when they treat the real wolf as absent. Repeated false alarms can make people more reluctant to respond later. Statistical error probabilities do not literally behave like human trust, but the story is a useful way into false alarms and missed detections.

The statistical underpinning

α controls false alarms; power describes detection.

Type I error · α

Reject a true null hypothesis.

The test raises an alarm even though H0 is true. A test run at α = 0.05 is designed so this long-run error probability is no greater than 5% under its assumptions.

Type II error · β

Fail to reject a false null hypothesis.

The test misses a real departure from H0. β depends on the specified alternative, effect size, variability, sample size and testing rule.

Power · 1 − β

Detect a specified real effect.

Power is the probability that the test rejects H0 when the particular alternative used in the calculation is true.

The trade-off

Changing the threshold has consequences.

With the same data and test, lowering α usually makes rejection harder and can increase β. Larger, better-designed studies can improve power without simply accepting more false alarms.

A non-significant result is not proof that H0 is true. It may reflect little effect, imprecision, low power, poor measurement or a model that does not answer the clinical question.

Epidemiology translation

Use the same table for disease screening.

Screening example

H0: the person does not have the disease.
Type I error: the test says disease when the person does not have it.
Type II error: the test says no disease when the person actually has it.

The consequences are not symmetric. A false positive may trigger anxiety and further investigation; a false negative may delay care. The acceptable balance therefore depends on the purpose of the test and the consequences of each decision.

Check your understanding

Name the error before moving on.

A screening test is positive in a person who does not have the disease. Which error is this?

Type I error. The test has raised a false-positive alarm while H0 is true.

A study fails to detect a clinically important effect that is really present. Which error is this?

Type II error. The study has failed to reject a false H0 for that specified effect.

Does p > 0.05 prove there is no effect?

No. Failure to reject H0 is not proof of no effect. Inspect the estimate, confidence interval, study design and power.

Go deeper · α, β, power and decision theory

Authoritative and foundational sources for the statistical ideas beneath the story.

Definitions · NIST

Significance, Type II error and power

NIST’s statistical handbook connects α to Type I error and β to Type II error for a specified alternative.

Read the NIST handbook
Operating characteristics · NIST

How statistical tests make decisions

A deeper account of rejection regions, errors of the second kind, operating-characteristic curves and sample size.

Read NIST’s testing guide
Clinical trials · FDA / ICH

Statistical principles for clinical trials

Regulatory guidance on controlling Type I error, planning power and limiting Type II error in confirmatory trials.

Open ICH E9
Foundations · 1933

Neyman and Pearson’s testing framework

The original paper underlying tests designed around controlled error rates and power against alternatives.

Open the original paper