Testing hypotheses

Research Methods — Week 6

Recap

Where we are

• Week 3: the logic of hypothesis testing (conceptual)
• Week 5: experimental design, controls, confounders

This week: the formal machinery. You’ll learn to run a t-test in R.

And then you’ll learn why it can mislead you.

• [5 min] Frame the session: first half is mechanics, second half is the most important lesson of the course.

Questions?

Submit questions anonymously:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

The t-test

🎓💻 Concept block 1

What does a t-test do?

Compares two group means and asks:

“Is this difference larger than we’d expect by chance?”

• [25 min block] The t-test, step by step, with a live demo.

The logic

1. Assume no difference (null hypothesis)
2. Calculate how far apart the means are, relative to the variability (→ t-statistic)
3. Ask: if there were really no difference, how often would we see a t this extreme? (→ p-value)

Live demo

🖥️ Switching to WebR

t.test(temperature ~ site, data = borehole)

• Use the borehole temperature dataset. Show the output: t-statistic, df, p-value, confidence interval.
• Interpret aloud: “The mean difference is X°C, with a 95% CI of [Y, Z]. The p-value is 0.03, meaning we’d see a difference this large about 3% of the time if there were truly no difference.”

Reading the output

t = 2.34, df = 18, p-value = 0.031
95% CI: [0.12, 2.34]

Element	Meaning
t-statistic	How many SEs apart are the means
p-value	How surprising, if H₀ is true
95% CI	Plausible range for the true difference

• Emphasise: the CI is often more useful than the p-value. It tells you the size of the effect, not just whether it exists.

Interpret a t-test

💬✏️ Exercise 1

Write one sentence

I’m showing you R output from a t-test.

Write one sentence interpreting this result for a policy audience.

Then we’ll compare.

• [10 min] Show pre-prepared output on screen. Students write, then share.
• Common mistake to watch for: “the p-value means there’s a 3% chance the null is true.” Flag this — it’s exactly the confusion block 3 will address.

A common mistake

“p = 0.03 means there’s a 3% chance the null is true.”

Wrong. But extremely common. We’ll come back to this.

• Plant the seed now. The base rate fallacy in block 3 will explain why this interpretation is wrong.

Assumptions

🎓 Concept block 2

Every test rests on assumptions

For the t-test:

1. Independence — observations don’t influence each other
2. Normality — data within each group are roughly normal
3. Equal variance — groups have similar spread (Student’s t; Welch’s t relaxes this)

• [20 min block] Assumptions, how to check, what to do when violated.

How to check

Assumption	Check
Normality	Histogram, QQ plot, shapiro.test()
Equal variance	Side-by-side boxplots, Levene’s test
Independence	Think about the study design

When assumptions are violated

A skewed variable can produce misleading p-values.

HolmesCo

“Geological Solutions Since 2019”

Statistical Analysis — Soil Permeability

t-test on raw values: p = 0.04 → “Significant difference!”

t-test on log-transformed values: p = 0.23 → “No significant difference.”

The “significant” result was an artefact of skewness.

• This is a vivid example. HolmesCo tested permeability (always log-normal) without transforming. The p-value changed from 0.04 to 0.23 — the “discovery” vanished.

Check your assumptions

✏️💻 Exercise 2

In WebR

Given a dataset:

1. Make histograms of each group
2. Decide whether to transform
3. Run t.test() before and after transformation
4. Compare: did the conclusion change?

• [10 min] Quick hands-on exercise. The WebR page has the scaffolded version with hints.

The base rate fallacy

🎓💬 Concept block 3

The most important 25 minutes of the course

I’m going to show you why a statistically significant result can still be wrong most of the time.

• [25 min block] Take your time. This is the intellectual centrepiece of the module. Let the numbers sink in.

Warm-up: Monty Hall

Some of you simulated the Monty Hall problem in first-year Python.

Switching wins 2/3 of the time. But your gut says 50/50.

Why? Because your gut ignores the information the host gave you when they opened a door.

That confusion — between P(win) and P(win | what you now know) — is exactly what this block is about.

• Brief connection. Don’t re-derive Monty Hall. The point: conditional probability is unintuitive, and this has serious consequences.

The medical test

A disease affects 1 in 1,000 people.

A test is 99% accurate (99% sensitivity, 99% specificity).

You test positive.

What’s the probability you have the disease?

Cast your vote

PollEv.com/geol

text geol to 07480 781235

• Activate the poll before showing this slide. Use a numeric open-ended response so answers appear as a histogram.
• Most students will cluster around 95–99%. A few may guess lower. The spread makes the reveal (“it’s 9%”) land much harder — they can see on screen that almost everyone got it wrong.
• Wait until responses plateau, then advance to the arithmetic.

The arithmetic

Out of 100,000 people:

	Has disease	No disease	Total
Test positive	99	999	1,098
Test negative	1	98,901	98,902
Total	100	99,900	100,000

P(disease | positive) = 99 / 1,098 ≈ 9%

Not 99%. Nine percent.

• Let this land. Pause. “Why is it so low?”

Why is it so low?

Because the disease is rare.

The base rate matters enormously.

Even a very accurate test produces many false positives when the condition is uncommon.

HolmesCo strikes gold

HolmesCo

“Geological Solutions Since 2019”

Press Release — Major Gold Discovery in County Durham!

“Our geochemical assay (95% accuracy) detected gold in 50 out of 10,000 soil samples. Confirmed multi-element anomaly!”

• Work through the numbers on the board or next slide.

The reality

Suppose 5 in 10,000 samples are actually gold-bearing.

	Gold present	No gold	Total
Assay positive	~5	~500	~505
Assay negative	~0	~9,495	~9,495

HolmesCo’s 50 positives? Almost certainly all false.

The press release is nonsense.

• The 50 “hits” is actually fewer than the ~500 expected false positives. If anything, the assay is underperforming.
• This is the Week 6 equivalent of the traitor reveal.

Connection to p-values

A p-value tells you: P(data | H₀)

How surprising is this data if the null is true?

What you actually want: P(H₀ | data)

How likely is the null to be true given this data?

These are not the same thing. The difference depends on the base rate.

• Key sentence: “If you’re testing an implausible hypothesis, a significant p-value is probably a false positive. If you’re testing something well-supported, a significant result is more convincing.”

The new refrain

“How plausible was this before we tested?”

• This is the Bayesian insight without the formula. The habit of asking about prior plausibility is what matters.
• This also sets up Week 7: if you run 20 tests, the base rate for any individual hypothesis being true is low → false positives accumulate.

Plan your test

✏️ Integrative exercise

For your project

Discuss with your group:

1. What comparison will you test?
2. What’s your null hypothesis?
3. What’s your prior expectation — plausible or a long shot?
4. What assumptions should you check?

Write 3–4 sentences.

• [10 min] This feeds directly into the application session.

Wrap-up

Key points

1. A t-test compares two groups — but check the assumptions first
2. A p-value is P(data | H₀), not P(H₀ | data)
3. The base rate determines whether a “significant” result is real
4. Always ask: “How plausible was this before I tested?”

Exit ticket

A t-test comparing two groups returns p = 0.04. Which statement is correct?

PollEv.com/geol

text geol to 07480 781235

• Correct answer: B — P(data | H₀), not P(H₀ | data).
• A is the classic confusion of P(data | H₀) with P(H₀ | data). Very common, even among experienced researchers.
• C conflates statistical significance with practical or scientific significance — and ignores the base rate argument from this session.
• D inverts the conditional probability (like A, but from the other direction).
• If many students answer A or D, revisit the medical test example: “If the disease has a 1-in-1000 base rate, a positive test is probably still a false positive — even though p < 0.05.”

Next time

Application session: “First tests”

You’ll run your first real tests on your project data.

Remember: a significant p-value is the start of the conversation, not the end.

• Students should come ready to work on their project data. Groups may need to do data cleaning before testing.