Comparing groups

Research Methods — Week 7

Recap

Where we are

Last week: t-tests for two groups. The base rate fallacy.

This week: what happens when you have more than two groups — and more than one test.

• [5 min] Quick recap. Any lingering questions from last week’s base rate discussion?

Questions?

Submit questions anonymously:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

Why not just do lots of t-tests?

🎓 Concept block 1

The problem

Three groups (e.g., wind farms in three regions).

You could run three pairwise t-tests: A vs B, A vs C, B vs C.

Each test has a 5% false positive rate.

Three tests: P(at least one false positive) = 1 − 0.95³ ≈ 14%

Ten tests: 40%

Twenty tests: 64%

• [20 min block] Multiple comparisons. Build to the HolmesCo mineral survey.

The escalation

Number of tests	P(≥1 false positive)
1	5%
3	14%
10	40%
20	64%
100	99.4%

Run enough tests and you’re guaranteed to “find” something.

HolmesCo’s mineral survey

HolmesCo

“Geological Solutions Since 2019”

Press Release — Multi-Element Anomaly Discovered!

“We tested soil samples from 20 sites for enrichment in 20 elements. We found statistically significant enrichment (p < 0.05) for three elements at one site!”

If there’s truly nothing there, how many significant results would you expect by chance?

• 20 elements × 20 sites = 400 tests × 0.05 = 20 expected false positives. Three hits is fewer than expected by chance. HolmesCo’s “discovery” actually underperforms random noise.

The answer

20 × 20 = 400 tests at α = 0.05

Expected false positives: 20

HolmesCo found 3.

That’s fewer than expected by chance. The “discovery” is noise.

• Let this land. The subversion — that the result is actually less than chance — makes the multiple comparisons problem viscerally clear.

How many tests?

💬 Exercise 1

Your project

How many comparisons are you planning to make?

If you run all of them at α = 0.05, what’s your family-wise false positive rate?

Formula: 1 − (1 − α)^k where k = number of tests

• [10 min] Students calculate for their own projects. Then discuss: “If you’re testing more than one thing, you need to account for it.”

ANOVA

🎓💻 Concept block 2

One test for multiple groups

ANOVA (Analysis of Variance) asks:

“Does any group differ from the others?”

Without specifying which.

• [20 min block] ANOVA mechanics, live demo, post-hoc tests.

The logic

1. Total variation = between groups + within groups
2. If between-group variation is large relative to within-group → at least one group is different
3. The F-statistic captures this ratio
4. The p-value tells you how surprising this F is under the null

Live demo

model <- aov(output ~ region, data = wind_df)
summary(model)

If significant → which pairs differ?

TukeyHSD(model)

• Live-code this in WebR. Show the output, read the pairwise comparisons. Point out the adjusted p-values.

Corrections for multiple comparisons

Method	How	When
Tukey’s HSD	All pairwise, designed for ANOVA	After a significant ANOVA
Bonferroni	Divide α by number of tests	Simple, conservative, any context

Both reduce the false positive rate by being stricter about what counts as significant.

• Don’t go deep into the math. The key idea: when you test multiple things, you raise the bar for each individual test.

Run an ANOVA

✏️💻 Exercise 2

In WebR

Provided dataset: wind farm output in four regions.

1. Visualise with boxplots
2. Run aov()
3. Is it significant?
4. Run TukeyHSD() — which pairs differ?
5. Write a one-sentence conclusion

• [15 min] Hands-on. The WebR exercise page has scaffolding.

Effect sizes

🎓 Concept block 3

“Significant” ≠ “important”

A p-value tells you whether an effect is detectable.

It does not tell you whether it’s big enough to matter.

• [20 min block] Effect sizes — the statistical formalisation of “Is that a big number?”

The HolmesCo temperature problem

HolmesCo

“Geological Solutions Since 2019”

Two boreholes, 1,000 readings each.

Mean difference: 0.02°C. p-value: 0.001.

“Highly significant temperature anomaly detected!”

Is 0.02°C meaningful for any practical purpose?

• With enough data, trivially small differences become “significant.” The p-value says “detectable,” not “important.”

Cohen’s d

A standardised measure of how far apart two groups are:

d	Interpretation
0.2	Small
0.5	Medium
0.8	Large

effectsize::cohens_d(y ~ group, data = df)

• d = (mean₁ − mean₂) / pooled SD. It answers “how many standard deviations apart are the groups?”

For ANOVA: η² (eta-squared)

Proportion of variance explained by the grouping variable.

η²	Interpretation
0.01	Small
0.06	Medium
0.14	Large

Always report both the p-value and an effect size.

A policy-maker needs to know “how big?” — not just “is it real?”

• Link to “Is that a big number?” from Week 2. Same habit, formalised.

HolmesCo’s report card

✏️💬 Integrative exercise

Three test results

HolmesCo

“Geological Solutions Since 2019”

Technical Summary

1. Test A: p = 0.002, Cohen’s d = 1.2
2. Test B: p = 0.04, Cohen’s d = 0.08
3. Test C: p = 0.08, 95% CI = [−0.3, 6.1]

For each: should we act on this? Why or why not?

• [15 min] A is significant with a large effect — act. B is significant but trivially small — maybe not. C is non-significant but the CI almost excludes zero — worth investigating further.

Wrap-up

Key points

1. Multiple tests inflate false positives — correct for them
2. ANOVA compares multiple groups in one test; TukeyHSD finds which differ
3. Significance ≠ importance — always report effect sizes
4. “Is that a big number?” now has a formal answer

Next time

Application session: “Deepening your analysis”

Apply ANOVA or refine your t-test with effect sizes and assumption checks.

• Not all groups need ANOVA. Groups with two-group comparisons should deepen their t-test analysis with CIs and effect sizes.