AI, Reproducibility, and Integrity — Reference Sheet

Key concepts from Week 9. Useful when writing your report and reflecting on your use of AI.

Using AI effectively

What AI is good at

Generating first-draft code (“make a ggplot of X vs Y”)
Explaining error messages
Suggesting structure for a document
Spotting syntax mistakes
Summarising text

What AI is bad at

Knowing whether its answer is correct. AI is confident even when wrong.
Understanding your specific data. It doesn’t know what your columns mean, what’s missing, or what assumptions you’ve already checked.
Making methodological judgements. It won’t tell you your test is inappropriate for your data structure — or if it does, its reasoning may be generic rather than specific.
Citing sources reliably. AI frequently invents plausible-looking references that don’t exist.
Distinguishing meaningful results from spurious ones. AI rarely asks “Is that a big number?” or “How plausible was this before we tested?”

The rule of thumb

AI is a useful first draft, never a final answer. Treat its output the way you’d treat a suggestion from a colleague who hasn’t read your data — worth considering, but always check.

Reproducibility checklist

Could someone else clone your repo, run your code, and get the same results? Check each item:

All analysis is scripted. No steps that only happened in the console. If you typed it, it should be in a .R file.
File paths are relative, not absolute. read.csv("data/file.csv") works on any machine. read.csv("C:/Users/me/Desktop/file.csv") doesn’t.
Required packages are loaded explicitly. Every library() call is at the top of the script.
Decisions are documented. Why did you choose this test? Why did you exclude those rows? A brief comment is enough.
The script runs top-to-bottom without errors. Open a fresh R session, run the whole file, check the output.
Data files are in the repo (or clearly documented if they’re too large to include).
Random seeds are set if you use any random process (bootstrapping, simulation, cross-validation).

Why this matters

Your commit history is your reproducibility record. A policy-maker should be able to trace your conclusion back to your data through your code.

The integrity spectrum

Not all bad research practice is deliberate. The line between honest mistakes and misconduct is a spectrum:

Category	What it looks like	How it happens
Innocent errors	Wrong column, rounding mistakes, misread output	Carelessness. Peer review catches most of these.
Questionable research practices	p-hacking, HARKing, selective reporting	Often unintentional. You run many tests and only report the one that “worked.” You form the hypothesis after seeing the results and present it as if you predicted it.
Fabrication / falsification	Inventing data, altering results	Deliberate and rare, but easier to detect than people think (Benford’s law, duplicated figures, statistically impossible results).

Where AI fits

If AI writes your analysis and you don’t check it, you are responsible for the errors.
If AI generates data or figures that don’t reflect reality, that is fabrication — even if unintentional.
Using AI to help write code or structure a report is fine. Presenting AI-generated analysis as your own without verification is not.

The HolmesCo lesson

HolmesCo doesn’t fabricate data. But their corner-cutting — on design, on assumption-checking, on interpretation — produces conclusions that are functionally no better than fabrication. Integrity isn’t just “don’t lie.” It’s “do the work properly.”

Key terms

Term	Meaning
Reproducibility	Someone else can run your code on your data and get the same result
Replicability	Someone else can repeat your study with new data and reach the same conclusion
p-hacking	Running many tests and only reporting the significant ones
HARKing	Hypothesising After Results are Known — presenting a post-hoc finding as a prediction
Selective reporting	Omitting analyses that didn’t support your preferred conclusion
Replication crisis	Many published findings don’t hold up when others try to reproduce them