AI, reproducibility, and integrity

Research Methods — Week 9

Recap

Where we are

Your projects are nearly done. You’ve designed investigations, run tests, fitted models, and identified limitations.

This week: two forces that shape modern research —

AI and the pressure to produce results.

• [5 min] This session is more discursive than technical — more debate and reflection, less coding.

Questions?

Submit questions anonymously:

PollEv.com/geol

text geol to 07480 781235

• Open the Poll Everywhere Q&A activity before the session and leave it active throughout.
• Questions appear in your Poll Everywhere presenter view; upvoted questions rise to the top.
• Address during transitions or at the end of concept blocks, not mid-explanation.
• Reset (archive) the activity between sessions so questions don’t carry over from the previous week.

What AI does well — and badly

🎓💬 Concept block 1

AI is good at…

• Generating first-draft code
• Summarising text
• Suggesting structure
• Spotting syntax errors
• Explaining error messages

• [20 min block] Honest assessment of AI capabilities and limits.

AI is bad at…

• Knowing whether its answer is correct
• Understanding your specific data
• Making judgement calls about methodology
• Citing sources reliably
• Distinguishing meaningful from spurious results

Live demo (if possible)

🖥️ Give an AI the HolmesCo gold assay scenario

“We tested 10,000 soil samples with a 95%-accurate assay. 50 tested positive. What’s the probability of a true gold deposit?”

Watch what it does with the base rate.

• The AI will likely either get it wrong (ignoring the base rate) or give a generically correct but vague answer without applying it to the specific numbers.
• Have a backup screenshot in case the AI gives a perfect answer.
• The point: fluent ≠ thoughtful.

The key distinction

AI is fluent but not thoughtful.

It can write a convincing paragraph about any result — including a wrong one.

AI audit

💬 Exercise 1

Your experience

Think about a time in this module when you used AI (or were tempted to). What did it help with? Where did you have to override it?

Common patterns:

• Useful: debugging code, plot templates, explaining errors
• Unreliable: interpreting results, choosing methods

• [15 min] Open discussion or think-pair-share. Collect themes on the board. Students are generally honest about this — the goal is reflection, not confession.

Reproducibility

🎓 Concept block 2

The replication crisis

Many published findings don’t hold up when others try to reproduce them.

• [20 min block] Reproducibility as a practical skill.

Why?

• p-hacking — trying many analyses, reporting the one that “works”
• Underpowered studies — too few observations to detect real effects
• Selective reporting — filing away null results
• Data not shared — no one can check your work

Your commit history IS your reproducibility record

If someone cloned your repo and ran your code, would they get the same results?

Practical reproducibility habits

1. Script everything — no “I typed this in the console”
2. Document your decisions — why this test, why you excluded those data points
3. Record software versions — sessionInfo() in R
4. Share data and code — within ethical limits

• Connection to the policy report: a policy-maker should be able to trace your conclusion back to your data.

Reproducibility check

💬✏️ Exercise 2

Can you re-run your analysis?

Open your project repo. Start a fresh R session.

Run your analysis script from scratch.

Does it work? What breaks?

Common problems:

• Hardcoded file paths
• Missing libraries
• Steps done in the console but not saved

Fix what you can. 10 minutes.

• [15 min] This is surprisingly powerful. Most students will find at least one thing that breaks. That experience is more persuasive than any lecture.

Scientific integrity

🎓💬 Concept block 3

The spectrum

• [20 min block] Brief and frank. Not preachy.

Three levels

Level	What	Example
Innocent errors	Rounding, wrong column, misread output	Your peer reviewer catches these
Questionable practices	p-hacking, HARKing, selective reporting	Often unintentional — this is what base rate and multiple comparisons sessions were about
Fabrication	Making up or altering data	Rare, career-ending, easier to detect than you’d think

Where does AI fit?

If AI writes your analysis and you don’t check it → you are responsible for the errors.

If AI generates data or figures that don’t reflect reality → that’s fabrication, even if unintentional.

The HolmesCo connection

HolmesCo doesn’t fabricate data.

But their corner-cutting on design, analysis, and interpretation produces conclusions that are functionally no better than fabrication.

Integrity isn’t just “don’t lie.”

It’s “do the work properly.”

Open discussion

💬 Q&A

Your questions

This is a looser session. What’s on your mind?

• Questions about your reports?
• AI use policies?
• What counts as plagiarism when AI is involved?

• [10 min] Address what comes up. Students often have genuine questions about the AI boundary. The honest answer: use AI as a tool, check everything, and document what you used.

Wrap-up

Key points

1. AI is useful for mechanics; unreliable for judgement
2. Reproducibility = scripted, documented, version-controlled
3. Integrity = doing the work properly, not just not lying

Next time

Application session: “AI as critic”

You’ll feed your analysis to an AI and evaluate its feedback.

The question: how much should you trust it?

• Students should bring their analysis code and a draft results section to the application session.