Week 10: Common Pitfalls

HolmesCo’s greatest hits — and how to avoid them in your report

Introduction

You’re about to write up your final report. Before you do, let’s review six common pitfalls — the kind of mistakes that HolmesCo makes routinely and that peer reviewers should catch.

For each excerpt below, identify the problem and think about how you’d fix it. These are quick-fire: spend about 2 minutes on each.

Pitfall 1: Unsupported conclusion

“Our data shows that wind farm output varies by season, with lower generation in summer months. Therefore, wind energy is unreliable and should not receive government subsidies.”

What’s wrong with this conclusion? Type your answer below.

TipSolution

The conclusion goes far beyond the data. The analysis shows seasonal variation — that’s expected and well-known. But “varies by season” does not mean “unreliable,” and a data pattern alone cannot support a policy recommendation about subsidies. That would require cost-benefit analysis, comparison with other energy sources, and consideration of grid-level storage and demand patterns. The leap from observation to policy is unsupported.

Pitfall 2: Mystery figure

A report contains the following paragraph:

“As shown below, there is a clear trend in the data.”

[A scatter plot appears here with no title, no axis labels, no units, no caption, and no figure number. It is not referenced anywhere else in the text.]

List everything this figure is missing.

TipSolution

Missing elements:

• Axis labels with units (e.g., “Year” and “Generation (TWh)”)
• A descriptive title or caption explaining what the figure shows
• A figure number so it can be referenced
• A reference in the text (“see Figure 1”) rather than “below”
• Context — what “clear trend” means should be stated explicitly

A figure that can’t be understood without reading the surrounding text has failed at its job. Every figure should be self-contained: a reader should understand what it shows from the caption and labels alone.

Pitfall 3: P-value without effect size

“The difference in solar panel output between the two regions was highly significant (p = 0.001), confirming that location affects performance.”

What’s missing from this result?

TipSolution

How much does output differ? The p-value says the difference is unlikely to be zero, but it doesn’t say whether the difference is 0.1 watts or 100 watts. The sentence needs:

• The effect size (mean difference between regions)
• A confidence interval (how precisely is the difference estimated?)
• Context — is the difference large enough to matter for siting decisions?

Remember Week 7: “significant” means detectable, not important. “Is that a big number?”

Pitfall 4: Overstatement

“The results prove that biomass electricity is carbon-neutral under current accounting rules.”

What’s wrong with the word “prove”?

TipSolution

Science doesn’t “prove” things — it provides evidence that supports or contradicts hypotheses. Even if the data are consistent with carbon neutrality under certain accounting rules, the analysis can’t rule out alternative explanations or account for every uncertainty.

Better phrasing: “The analysis is consistent with…” or “The evidence suggests that, under current accounting rules…”

The distinction matters because “proven” shuts down further inquiry. Good science stays open to revision.

Pitfall 5: Vague methods

“We analysed the data using statistical tests in R and found significant results.”

Could you reproduce this analysis from this description?

TipSolution

Almost everything:

• Which data? (source, time period, variables)
• Which tests? (t-test? ANOVA? regression?)
• Why those tests? (what hypothesis, what comparison?)
• What assumptions were checked? (normality, equal variance?)
• How were the data cleaned? (exclusions, transformations?)

A methods section should give enough detail that someone could reproduce your analysis with just your description and your data. “Statistical tests in R” could mean anything.

Pitfall 6: No limitations

“Wind generation in the North East is 40% higher than the South East (p < 0.01, Cohen’s d = 1.2). We recommend prioritising all future wind farm development in the North East.”

The statistics are well-reported. So what’s the problem?

TipSolution

There are no limitations or caveats:

• What about confounders? (Different turbine types, terrain, existing capacity, grid connection costs?)
• Is the sample representative? (Which wind farms were compared?)
• What can’t this analysis tell us? (Future wind patterns may differ; land availability, planning constraints, and public acceptance aren’t in the data.)
• Does the conclusion match the evidence? (A regional difference in output doesn’t automatically mean all development should go there.)

A good report acknowledges what the analysis cannot tell you. Honest uncertainty is more trustworthy than false confidence.

Quick check: The course refrains

Over the past 10 weeks, five questions have come up again and again. They should be automatic by now — the habits that separate careful analysis from HolmesCo-style corner-cutting.

Match each refrain to the pitfall it would have caught:

Refrain	Week introduced
“Is that a big number?”	Week 2
“Compared to what?”	Week 2
“What assumptions are we making?”	Week 3
“How plausible was this before we tested?”	Week 6
“What is the model not capturing?”	Week 8

Which of these refrains would have caught Pitfall 3 (p-value without effect size)? Enter the refrain as a short phrase:

Self-assessment

Before you start writing your report, rate yourself honestly on each refrain. Which do you use automatically? Which do you need to consciously remember?

This isn’t graded — it’s for you.

Save your work

You’re about to write your final report. Keep these pitfalls in mind as you draft — and use the peer review template to check a classmate’s work in the application session.

Commit any remaining code to your repo via GitHub Desktop.