HolmesCo: Mineral Survey

Multiple comparisons — Week 6 handout with instructor notes

HolmesCo Mineral Survey

This is a fictional document for teaching purposes.

PRESS RELEASE — FOR IMMEDIATE DISTRIBUTION

Date: March 2026

Critical Mineral Discovery in Northern England

In response to the UK government’s Critical Minerals Strategy, HolmesCo has completed a comprehensive geochemical survey targeting minerals essential for the clean energy transition.

Our field teams collected soil samples from 20 sites spanning County Durham, Northumberland, Cumbria, and North Yorkshire. Each sample was analysed for enrichment in 20 critical minerals including lithium, cobalt, tungsten, niobium, and rare earth elements.

Using rigorous statistical testing at the industry-standard significance level (p < 0.05), we identified statistically significant enrichment for three critical minerals at one site near Alston, Cumbria.

“With global demand for critical minerals surging, this is a strategically important discovery. Three separate minerals showing significant enrichment at the same location strongly suggests a genuine mineralisation event — right here in the North Pennines.”

— Dr G. Holmes, Chief Scientific Officer, HolmesCo

HolmesCo recommends immediate follow-up drilling at the Alston site and has applied for an exploration licence under the Crown Estate’s critical minerals framework.

Investors interested in supporting UK mineral security should contact HolmesCo’s Business Development team.

Instructor notes

The numbers tell a different story.

Ask the class to work through the arithmetic:

  • 20 sites × 20 minerals = 400 tests
  • At α = 0.05, each test has a 5% false positive rate
  • Expected false positives (if nothing is there): 400 × 0.05 = 20
  • HolmesCo found 3 significant results

Three hits is fewer than expected by chance. Far from being evidence of a “genuine mineralisation event,” the result is actually underwhelming — you’d expect about 20 false positives from 400 tests, and they only found 3.

Key teaching points:

  1. The multiple comparisons problem. Every test you run is another chance for a false positive. Running 400 tests at α = 0.05 is almost guaranteed to produce “significant” results even when nothing is real.

  2. The family-wise error rate. P(at least one false positive) = 1 − 0.95400 ≈ 1.00 (virtually certain). HolmesCo’s individual p < 0.05 threshold is meaningless without correction.

  3. Three hits < 20 expected — the result is actually evidence against a real discovery, not for one. This subverts students’ expectation that significance = discovery.

  4. The press release framing. Notice how “three critical minerals at one site” sounds impressive — especially when wrapped in language about national strategy and the energy transition. The total number of tests (400) is not mentioned. This is a common pattern in misleading statistical claims: report the hits, hide the denominator. The policy urgency around critical minerals makes the reader want to believe it.

Discussion question: How would you correct for multiple comparisons?

  • Bonferroni correction: divide α by the number of tests. α_corrected = 0.05 / 400 = 0.000125. None of HolmesCo’s results would survive this correction.
  • ANOVA approach: test all sites simultaneously for each mineral, then use post-hoc tests with correction (like TukeyHSD).
  • Simply reporting the number of tests alongside the results would let the reader judge for themselves.