Week 9: Checking Your Work

Reproducibility, reporting, and knowing what AI can’t do for you

Introduction

Your projects are nearly complete. Before you write up, it’s worth stepping back and checking your work with fresh eyes. This page has a few short exercises to help you:

  • Spot common reproducibility problems in code
  • Practice writing a proper results sentence
  • Think critically about what AI can and can’t do

These exercises are deliberately lighter than Week 8 — the real work this week happens with external AI tools and in your report.

Exercise 1: Spot the reproducibility problem

The script below is meant to load some data, run a t-test, and report the result. But if you tried to run it in a fresh R session, it would fail. There are four problems — can you find and fix them all?

NoteHint 1

The four problems are:

  1. dplyr::filter() is used but library(dplyr) is never called
  2. early and late are commented out — they were probably defined in the console but never saved to the script
  3. t.test(late, early) uses early and late, which don’t exist because of problem 2
  4. sample() with replace = TRUE produces random results — without set.seed(), you’ll get different numbers every time
NoteHint 2

Fixes:

  1. Add library(dplyr) at the top (or use base R subsetting instead)
  2. Uncomment the lines that define early and late
  3. (Fixed by step 2)
  4. Add set.seed(some_number) before replicate()
TipSolution 
library(dplyr)

wind_data <- read.csv("data/uk_electricity.csv")
wind_recent <- dplyr::filter(wind_data, year >= 2015)

early <- wind_data[wind_data$year < 2015, ]$wind_twh
late <- wind_data[wind_data$year >= 2015, ]$wind_twh

result <- t.test(late, early)
result

set.seed(2847)
boot_means <- replicate(1000, mean(sample(late, replace = TRUE)))
cat("Bootstrap 95% CI:", quantile(boot_means, c(0.025, 0.975)), "\n")

These are the four most common reproducibility problems:

  1. Missing packages — you loaded it once, forgot you needed it
  2. Console-only work — you defined something interactively but it’s not in your script
  3. Broken dependencies — downstream code relies on upstream steps that are missing
  4. Unseeded randomness — results change every time you run

If someone clones your repo and runs your script from scratch, all four will break. Script everything. Seed everything. Test from a fresh session.

Quick check: What belongs in a results sentence?

When reporting a t-test result for a policy audience, which of the following should you include?

    1. The p-value only
    1. The p-value and effect size
    1. Just “the result was statistically significant”
    1. The p-value, effect size, confidence interval, and a plain-language interpretation

Enter a, b, c, or d:

Exercise 2: Write the interpretation

Here is the output from a t-test comparing wind generation (TWh) in two periods: 2000–2014 vs 2015–2024.

    Welch Two Sample t-test

t = 10.7, df = 11.3, p-value = 5.2e-07
95 percent confidence interval:
 44.2  67.1
sample estimates:
mean of x  mean of y
    68.8      13.2

“Mean of x” is the 2015–2024 group. “Mean of y” is 2000–2014.

Write a one-sentence interpretation suitable for a policy report. Include: the direction of the effect, the size (mean difference or CI), and what it means in context.

NoteHint 1

The key numbers: mean difference ≈ 56 TWh (68.8 − 13.2), 95% CI [44, 67]. Wind generation was substantially higher in the recent period. How would you explain this to someone who doesn’t know statistics?

TipSolution

A good interpretation might be:

“UK wind generation was substantially higher in 2015–2024 than in 2000–2014, with a mean difference of approximately 56 TWh per year (95% CI: 44–67 TWh), reflecting the large-scale deployment of onshore and offshore wind capacity over this period.”

Key features: direction (higher), magnitude (56 TWh), uncertainty (CI), and context (capacity deployment). No “proves” — just evidence.

Quick check: AI strengths and weaknesses

AI tools like ChatGPT and Copilot are useful in research — but not for everything. Which task is AI most reliable at?

    1. Judging whether your sample size is adequate
    1. Spotting a syntax error in your R code
    1. Deciding which statistical test fits your research question
    1. Evaluating whether your effect size is meaningful

Enter a, b, c, or d:

Extension: Base rate revisited

This exercise is optional — for students who finish early.

Remember HolmesCo’s gold assay from Week 6? Their test is 95% accurate and they found 50 positive results from 10,000 samples. Suppose only 5 samples in 10,000 are truly gold-bearing.

Calculate the probability that a positive test result is actually correct (the positive predictive value). Would an AI tool necessarily get this right?

NoteHint 1

Sensitivity = 0.95: the test correctly identifies 95% of true gold-bearing samples. Specificity = 0.95: the test correctly identifies 95% of non-gold samples (so 5% of non-gold samples test positive by mistake).

NoteHint 2 
true_pos <- 5 * 0.95        # ≈ 4.75
false_pos <- 9995 * 0.05    # ≈ 499.75
total_pos <- true_pos + false_pos
ppv <- true_pos / total_pos * 100
TipSolution 
true_gold <- 5
true_pos <- true_gold * 0.95
non_gold <- 10000 - true_gold
false_pos <- non_gold * 0.05
total_pos <- true_pos + false_pos
ppv <- true_pos / total_pos * 100
ppv

The positive predictive value is about 0.94% — less than 1 in 100 positive results is real. HolmesCo’s 50 “confirmed hits” are almost all false positives.

This is the base rate fallacy in action. A 95% accurate test sounds reliable, but when the thing you’re looking for is very rare, false positives swamp true positives. The same logic applies to p-values: a “significant” result for an implausible hypothesis is probably a false positive.

When AI analyses this scenario, it often processes the numbers fluently but doesn’t spontaneously flag that the base rate makes the conclusion unreliable. That judgment — “how plausible was this before we tested?” — is exactly the skill this course has been building.

Save your work

If you completed the reproducibility exercise, make sure your own project script passes the same checks: does it run from scratch in a fresh session? Are all packages loaded? All variables defined?

Commit and push via GitHub Desktop.