HolmesCo: Report Card
Effect sizes vs p-values — Week 7 handout with instructor notes
HolmesCo Technical Summary — Three Results to Evaluate
This is a fictional document for teaching purposes.
HolmesCo
Geological Consultants Since 2019
QUARTERLY TECHNICAL SUMMARY — Q1 2026
HolmesCo is pleased to present three key findings from our recent project portfolio. All results have been subjected to rigorous statistical testing.
Result 1: Wind farm output and elevation
HolmesCo compared annual energy output (GWh) for 15 wind farms on hilltop sites vs 15 wind farms in lowland areas across Northern England.
| Hilltop | Lowland | |
|---|---|---|
| Mean output | 48.2 GWh | 31.5 GWh |
| SD | 11.3 | 10.8 |
t-test: t(28) = 4.12, p < 0.001
95% CI for difference: 8.4 to 25.0 GWh
Cohen’s d = 1.51
“Hilltop wind farms produce significantly more energy. This confirms what we’ve always known — elevation matters.”
Result 2: Borehole temperature and county
HolmesCo measured temperatures at 12 m depth in 1,000 boreholes in County Durham and 1,000 boreholes in North Yorkshire.
| Durham | N. Yorkshire | |
|---|---|---|
| Mean temperature | 11.52°C | 11.50°C |
| SD | 0.8 | 0.9 |
t-test: t(1998) = 0.53, p = 0.001
95% CI for difference: 0.008 to 0.032°C
Cohen’s d = 0.02
“A highly significant difference in geothermal potential between the two counties. HolmesCo recommends prioritising County Durham for geothermal development.”
Result 3: Solar panel degradation rate
HolmesCo compared annual power output degradation rates for 20 monocrystalline vs 20 polycrystalline solar panels over 10 years.
| Mono | Poly | |
|---|---|---|
| Mean degradation | 0.45%/yr | 0.58%/yr |
| SD | 0.22 | 0.25 |
t-test: t(38) = 1.74, p = 0.09
95% CI for difference: −0.02 to 0.28 percentage points/yr
Cohen’s d = 0.55
“No significant difference between panel types. Both technologies degrade at essentially the same rate.”
Your task
For each result, answer: Should we act on this? Why or why not?
Consider:
- Is the result statistically significant?
- Is the effect big enough to matter for policy or engineering decisions?
- Does the confidence interval tell a different story from the p-value?
- Would you trust HolmesCo’s interpretation?
Instructor notes
Result 1: Significant with a LARGE effect — act on this
This is the genuine article. The effect is large (d = 1.51), the CI is entirely above zero and practically meaningful (8–25 GWh), and the p-value is very small. The conclusion is well-supported.
HolmesCo’s interpretation is unusually reasonable here. The only quibble: “confirms what we’ve always known” suggests they didn’t need the test — confirmation bias, but the evidence genuinely supports it.
Result 2: Significant with a TINY effect — don’t act
This is the key teaching example. The p-value is 0.001 (“highly significant!”), but Cohen’s d is 0.02 — negligibly small. The actual difference is 0.02°C. No one can detect, measure, or use a 0.02°C difference for geothermal planning.
The large sample size (n = 2000) made a trivially small difference “significant.” This is the statistical equivalent of measuring a flea and calling it a horse because your ruler is very precise.
HolmesCo’s recommendation to “prioritise County Durham” based on a 0.02°C difference is absurd — but this is exactly the kind of reasoning that appears in real consultancy reports when people don’t look past the p-value.
Key message: “Significant” ≠ “important.” Always report effect sizes.
Result 3: Non-significant but SUGGESTIVE — investigate further
The p-value (0.09) is above the conventional 0.05 threshold, so HolmesCo says “no difference.” But look at the effect size: d = 0.55 is a medium effect. The CI (−0.02 to 0.28) nearly excludes zero — most of the interval is above zero, suggesting mono panels probably do degrade less.
With only 20 panels per group, the study is likely underpowered — it may lack the statistical power to detect a real difference of this size. A larger study might well find significance.
HolmesCo’s “no difference” conclusion is premature. The honest conclusion is: “We can’t rule out an important difference; more data is needed.”
Key message: Non-significant ≠ no effect. The absence of evidence is not evidence of absence.