HolmesCo: Gold Assay

Base rate neglect — Week 6 handout with instructor notes

HolmesCo Gold Assay Results

This is a fictional document for teaching purposes.

PRESS RELEASE — FOR IMMEDIATE DISTRIBUTION

Date: March 2026

Major Gold Discovery in County Durham

HolmesCo is pleased to announce the results of a landmark geochemical survey across County Durham and the Northern Pennines.

Survey methodology

Our field teams collected soil and shallow rock samples from 10,000 locations across the survey area. Each sample was tested using HolmesCo’s proprietary AuDetect™ geochemical assay, which identifies gold-bearing rock with 95% accuracy:

  • If gold is present, the assay correctly identifies it 95% of the time (sensitivity)
  • If gold is absent, the assay correctly rules it out 95% of the time (specificity)

Results

Of the 10,000 samples tested, 50 returned positive results for gold mineralisation.

The positive samples are concentrated in the western part of the survey area, consistent with known geological structures in the Alston Block.

Interpretation

“Fifty confirmed gold hits across County Durham represents a highly significant discovery. Our AuDetect™ assay is 95% accurate — these results give us great confidence that there is genuine gold mineralisation in the region. We recommend immediate follow-up drilling at the most promising sites.”

— Dr G. Holmes, Chief Scientific Officer, HolmesCo

HolmesCo has applied for three exploration licences and is seeking investment partners to fund a Phase 2 drilling programme.

Investors interested in this opportunity should contact HolmesCo’s Business Development team.

This press release has been prepared by HolmesCo and has not been independently reviewed. AuDetect™ accuracy figures are based on laboratory validation against known reference samples. Field performance may vary. HolmesCo accepts no liability for investment decisions made on the basis of this announcement.

Instructor notes

The base rate destroys HolmesCo’s claim.

This handout goes with the base rate / conditional probability segment in Week 6 content (concept block 3). Present the handout first, then guide students through the arithmetic.

The question to pose

“HolmesCo says the assay is 95% accurate and they found 50 positives. How many of those 50 are likely to be real gold?”

Most students will initially guess around 47–48 (95% of 50).

The calculation

Ask: “How common is gold in County Durham?”

Gold mineralisation in the Northern Pennines is rare. A generous estimate is about 5 in 10,000 samples (0.05%) actually contain gold-bearing rock.

Work through a natural frequency table for 10,000 samples:

Truly gold-bearing (5) Not gold-bearing (9,995) Total
Assay positive 5 × 0.95 ≈ 5 9,995 × 0.05 ≈ 500 ~505
Assay negative 5 × 0.05 ≈ 0 9,995 × 0.95 ≈ 9,495 ~9,495

With a 95%-accurate test and a base rate of 0.05%, you’d expect roughly 505 positive results, of which only 5 are real.

P(gold | positive) = 5 / 505 ≈ 1%.

A positive result from HolmesCo’s assay means there is about a 1 in 100 chance the sample actually contains gold.

So what about HolmesCo’s 50 positives?

The maths predicts ~505 positives, but HolmesCo only found 50. This is actually fewer than expected from false positives alone (~500). Two possible explanations:

  1. The assay’s specificity is slightly better than 95% in the field (say 99.5%), which would give ~50 false positives — consistent with HolmesCo’s results.
  2. HolmesCo applied additional screening or filtering (and didn’t mention it in the press release).

Either way, the positive predictive value remains terrible. Even in the best case (99.5% specificity, 50 positives total), only ~5 would be real: PPV ≈ 5/55 ≈ 9%. Of HolmesCo’s 50 “confirmed” hits, probably 45–50 are false.

Key teaching points

  1. P(data | H₀) ≠ P(H₀ | data). The assay is 95% accurate (P(positive | gold) = 0.95). But what we need is P(gold | positive), which depends on the base rate. Students confuse these — that’s exactly the point.

  2. The base rate matters enormously. Gold is rare. When the thing you’re looking for is rare, even a very accurate test produces mostly false positives.

  3. “95% accurate” sounds reassuring but is misleading. HolmesCo’s press release trades on the intuition that 95% accuracy ≈ 95% chance of being right. It isn’t, when the base rate is low.

  4. Connection to p-values. A p-value tells you P(data | H₀) — how surprising the result is if the null is true. But what you want is P(H₀ | data) — is the null actually true given your data? The difference depends on the base rate: how plausible was the hypothesis before you tested? Testing an implausible hypothesis (gold in Durham) at α = 0.05 is the p-value equivalent of HolmesCo’s 95%-accurate assay on a 0.05% base rate.

  5. “How plausible was this before we tested?” This is the new refrain, joining “Is that a big number?” Students should develop the habit of asking about prior plausibility before interpreting any test result.

Discussion prompt

“When you run a t-test in your project next week, what’s your prior expectation? Is the effect you’re looking for plausible or a long shot? How should that change how you interpret the p-value?”

Connection to medical testing (optional warm-up)

Before revealing the HolmesCo numbers, you may want to run the classic medical test version:

  • A disease affects 1 in 1,000 people
  • A test is 99% sensitive, 99% specific
  • You test positive — what’s the probability you have the disease?

Answer: ~9% (not 99%). This builds the intuition; the HolmesCo version then cements it with a geological example that sticks.