How confident should we be?

Research Methods — Week 3

Recap

Where we are

• Week 1: What makes a testable question
• Week 2: Summarising data, honest visualisation, ggplot2

You’ve produced figures showing biomass trends and emissions comparisons.

This week: how much should we trust those numbers?

• [5 min] Recap. Ask: “Did anyone look at their figures again after last session? Notice anything new?”

Questions?

Submit questions anonymously:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

Error and uncertainty

🎓 Concept block 1

All measurements have uncertainty

The question is not whether there’s uncertainty — it’s how much and what kind.

• [20 min block] Two types of error, then precision vs accuracy.

Two kinds of error

Systematic error (bias)

Your instrument is consistently wrong.

Example: emission factors that exclude the supply chain always underestimate true emissions.

Random error (noise)

Measurements vary each time.

Example: annual electricity generation fluctuates with weather, demand, and plant outages.

• Systematic error is the sneaky one — you can’t detect it by repeating the measurement. You need an independent check.
• Random error averages out with enough data. Systematic error doesn’t.

Precision vs accuracy

Precise but inaccurate

Measurements cluster tightly — but around the wrong value.

e.g., a miscalibrated thermometer

Accurate but imprecise

Measurements centre on the right value — but scatter widely.

e.g., noisy field readings

Which is worse? It depends on whether you can correct the bias.

• Geoscience examples: radiometric dating is precise but relies on assumptions about initial conditions (accuracy). Temperature records are accurate on average but have coverage gaps (precision).

What could go wrong?

💬✏️ Exercise 1

The biomass emission factor

Show the CO₂-per-MWh figure from last week.

“What are the sources of uncertainty in this number?”

Work in pairs. 5 minutes. Then share.

• [15 min] Expected answers: what’s included in lifecycle emissions? Transport? Land use change? Drying? Regrowth? Different studies give different numbers — why?
• Classify each as systematic or random.
• This is the pivot point: students start to realise the “zero” emission factor is a policy choice, not a measurement.

The logic of hypothesis testing

🎓 Concept block 2

The courtroom analogy

In a trial

• Start with “innocent”
• Look at the evidence
• Decide: enough to convict?

In statistics

• Start with “no effect” (null hypothesis)
• Look at the data
• Decide: enough to reject the null?

• [20 min block] This is conceptual — no formulae yet. The goal is to build intuition for why we need formal testing.

Two types of mistake

	Null is true	Null is false
Reject null	Type I error (false positive)	✓ Correct
Don’t reject	✓ Correct	Type II error (false negative)

Type I: Convicting an innocent person.

Claiming biomass is worse than coal when it isn’t.

Type II: Acquitting a guilty one.

Missing a real difference because your sample was too small.

• The significance level (α, usually 0.05) is our threshold for Type I error — how much surprise we require.
• Type II error depends on sample size, effect size, and variability. We’ll formalise this later; for now, the intuition is enough.

What does “overlap” look like?

Imagine two distributions: emissions from biomass plants and coal plants.

• If they’re far apart with narrow spread → easy to tell apart
• If they overlap a lot → hard to tell apart
• The question: is this difference bigger than we’d expect by chance?

• Sketch this on the board or show a simple graphic. Two bell curves overlapping. “Would you bet these are different?”
• No R code yet. Just visual intuition. The mechanics come in Week 6.

How much evidence is enough?

💬 Exercise 2

The scenario

You measure emissions from 5 biomass plants and 5 coal plants.

The biomass mean is lower. But there’s overlap.

Are you convinced?

• [10 min] Let students discuss in small groups, then collect responses.

What if…

• What if you measured 50 of each?
• What if the difference were twice as large?
• What if one measurement were an extreme outlier?

Sample size, effect size, and variability all matter.

We’ll formalise this in Week 6 — for now, trust your intuition that eyeballing isn’t good enough.

• The point: students should leave feeling “I can see why we need formal tests” — which motivates Week 6’s t-tests.
• Don’t resolve the disagreement neatly. The tension is productive.

Sampling and representativeness

🎓 Concept block 3

Where did your data come from?

And does it represent what you think it represents?

• [20 min block] Population vs sample, selection bias, survivorship bias, confounding.

Population vs sample

You have data from Drax — one power station.

Can you generalise to “biomass electricity”?

Drax produces ~86% of UK biomass electricity. Does that help or hurt?

• It helps (the sample is most of the population) but it also means Drax is unusual — it’s massive, purpose-built, and uses a specific supply chain. Other biomass plants may behave very differently.

Selection bias

If you only measure the biggest, best-known facility, your results may not generalise.

Survivorship bias: if failing biomass plants shut down and disappear from the data, the remaining ones look better than average.

Confounding: biomass plants might be newer than coal plants. Any efficiency difference might reflect age, not fuel.

• Geological parallel: sampling only exposed rock faces — weathered, accessible, atypical of the subsurface.
• These are the same issues HolmesCo will demonstrate in Week 5 (valley boreholes, convenience sampling).

Assumption audit

✏️💬 Integrative exercise

The carbon accounting rule

The standard framework: CO₂ from burning biomass is counted as zero at the point of combustion.

Work in pairs:

1. List the assumptions this framework makes
2. For each: is it testable? What evidence would challenge it?
3. Which assumptions, if wrong, would most change the conclusion?

• [15 min] This directly feeds Week 3 app (scenario analysis) and Week 4 (the briefing).
• Expected assumptions: trees regrow (timescale?), supply chain emissions are negligible, land use doesn’t change, the alternative fuel would have been burned anyway.
• The “if wrong, most changes the conclusion” question is the key one. It teaches students to prioritise assumptions by impact.

What did you find?

Collect 3–4 assumptions from the room.

• Draw these out and write them up. Students will need them for the application session (scenario analysis) and the Week 4 briefing.

“What assumptions are we making?”

This is the question that separates good analysis from bad.

Wrap-up

Key points

1. All measurements have uncertainty — the question is how much
2. Systematic error is more dangerous than random error
3. The biomass “zero” is an assumption, not a measurement
4. Next step: what happens when we change the assumptions?

Exit ticket

A lifecycle assessment reports that UK biomass electricity produces 0 gCO₂/kWh. What is the most important thing to check before accepting this number?

PollEv.com/geol

text geol to 07480 781235

• Correct answer: B — What system boundary and timescale the assessment used.
• The “zero” is not a measurement — it is a policy assumption that depends entirely on what is included (combustion only? transport? processing? regrowth?) and over what time horizon.
• D (peer review) is a reasonable instinct but wrong as the most important thing — a peer-reviewed paper can still use a narrow system boundary.
• This question directly tests whether students have internalised “What assumptions are we making?” as a reflexive question.

Next time

Application session: “Changing the assumptions”

You’ll take the data and ask: does the answer change when we change the inputs?

• Students should arrive ready to do arithmetic in R — multiplying emission factors by generation volumes under different scenarios.