Type I and Type II errors
A single test can raise a false alarm or miss a real effect. Those are decision errors for that one testing procedure.
Statistical foundations · second of two connected guides
What changes when you run many tests.
Multiple testing is a family-level problem: the more questions an analysis tests, the more opportunities it has to find at least one false-positive result by chance.
Learning order: 1. Type I and Type II errors → 2. Multiple testing. You are on step 2.
You will decide: which tests form a family, what error rate matters and how every result will be reported.
Connected, not identical
One test has decision errors. Many tests create a family problem.
Multiple testing
Running many tests gives the analysis many chances to produce at least one false-positive result. The family, not only each test, needs an error-control plan.
Learn Type I and Type II errors first if false positives, false negatives, α, β or power are unfamiliar. That separate guide uses the boy-who-cried-wolf story and a screening example.
The connection: multiple-testing methods usually focus on controlling false positives across a family. Making the rule more demanding can reduce power and increase missed detections, so the consequences of both error types still matter.
See the accumulation
More chances make at least one false alarm more likely.
Suppose 20 independent null hypotheses are all true and each test uses α = 0.05. Each test has a 5% false-positive probability, but the analysis now has 20 opportunities to produce at least one false alarm.
| Independent tests at α = 0.05 | Chance of at least one false positive |
|---|---|
| 1 | 5.0% |
| 5 | 22.6% |
| 10 | 40.1% |
| 20 | 64.2% |
Where 64.2% comes from
With 20 independent tests, the chance that none is falsely positive is 0.9520. Therefore the chance of one or more false positives is 1 − 0.9520 ≈ 0.642.
Real tests can be correlated, so this calculation is an illustration rather than a universal exact answer. The central lesson remains: the complete testing strategy matters, not only the smallest p-value.
Plan before looking
Define the family, then choose the protection.
- Name the primary question. A clear primary outcome prevents a study from treating every result as equally confirmatory.
- Define the family. Group the tests that support the same claim or decision. Do not decide the family after seeing which p-value is smallest.
- Choose the error rate. Family-wise error control asks whether any false positive occurs in the family. False-discovery-rate control limits the expected proportion of false findings among the declared discoveries.
- Choose a method in advance. Holm is a strong general family-wise option. Benjamini–Hochberg can suit larger exploratory families when controlling false discoveries is the stated aim.
- Report the whole family. Show all planned outcomes and analyses, the adjustment method, adjusted and unadjusted results where useful, effect estimates and confidence intervals.
Ask fewer, better questions.
Pre-specify one primary outcome and a small number of justified secondary outcomes.
Protect against any false claim.
Bonferroni uses α divided by the number of tests. Holm usually keeps the same strong protection with better power.
Allow exploration, label it honestly.
Benjamini–Hochberg controls a different quantity: the expected proportion of false discoveries among findings called discoveries.
Do not replace thinking with correction.
Show effect sizes, intervals, outcome definitions and clinical importance. Adjustment cannot repair selective reporting or a poor question.
Correction in plain English
A correction changes the decision rule, not the data.
“Correction” can sound as though the original data or results were wrong. They are not edited. A multiple-testing method accounts for the number and purpose of the tests by making the rule for declaring a finding more demanding, or by calculating adjusted p-values that represent that rule.
Share α across the family.
For 20 tests and a family-wise α of 0.05, compare each p-value with 0.05 ÷ 20 = 0.0025. Equivalently, multiply each raw p-value by 20, never allowing the adjusted value to exceed 1.
Start with the strongest result.
Sort the p-values from smallest to largest. Compare the smallest with 0.05 ÷ 20, the next with 0.05 ÷ 19, and so on. Stop when a comparison fails. Holm protects against any false positive in the family and is usually less conservative than plain Bonferroni.
Control the false-discovery rate.
Sort the p-values and give each a rank. Compare rank i with i ÷ 20 × the chosen false-discovery rate. Find the largest p-value that passes; that result and all smaller ones are called discoveries under the procedure.
Match the rule to the claim.
Family-wise methods suit a small set of confirmatory claims where any false claim matters. False-discovery-rate methods can suit larger exploratory searches. The family and method should be justified before the results are inspected.
A correction does not make false positives impossible, prove that retained findings are true or rescue an analysis that hid tests. Report the full family, effect estimates, confidence intervals, raw and adjusted results where useful, and the method’s assumptions.
Simple worked decision
Twelve biomarkers, one p-value of 0.03.
Is the smallest result enough?
- What happened
- A team tested 12 biomarkers separately and highlighted the only unadjusted p-value below 0.05: p = 0.03.
- Why caution is needed
- The study created 12 opportunities for a small p-value. Reporting only the winner hides the testing family.
- Bonferroni illustration
- A simple family-wise threshold would be 0.05 ÷ 12 ≈ 0.0042, so p = 0.03 would not pass that rule.
- Better report
- Show all 12 estimates and intervals, say the analysis was exploratory, state the planned adjustment and treat promising signals as needing validation.
Check your understanding
Spot the hidden extra chances.
Can one pre-specified test make Type I or Type II errors without being a multiple-testing problem?
Yes. Type I and Type II errors belong to the decision rule for one test. Multiple testing begins when a family contains several testing opportunities.
A study has one outcome but tests it in six subgroups. Is that still multiple testing?
Yes. Subgroup tests create several opportunities for a chance finding and should be planned, justified and reported together.
Does every study with several descriptive tables need a Bonferroni correction?
No. Multiplicity concerns depend on the inferential claims and decision strategy, not the mere number of numbers printed. Define which tests support which claim.
Can you choose Holm or Benjamini–Hochberg after seeing which gives the preferred answer?
No. Choose and justify the family and method before inspecting the results whenever the work is confirmatory.
Go deeper · family-wise error, false discoveries and corrections
Regulatory guidance, foundational papers and official software documentation for the statistical methods beneath this guide.
Multiplicity issues in clinical trials
Regulatory guidance on multiple outcomes, subgroups, analysis sets, interim analyses and confirmatory claims.
Open the EMA guidanceHolm’s sequential procedure
The original paper proving a stepwise method that controls the family-wise error rate.
Open Holm’s paperBenjamini and Hochberg’s FDR procedure
The foundational paper defining false-discovery-rate control and its step-up procedure.
Open the original paperAdjust p-values in R
Documentation for Bonferroni, Holm, Benjamini–Hochberg and other methods in p.adjust().