Statistics & the Art of Not Fooling Yourself

The hook

The easiest person to fool

"The first principle is that you must not fool yourself — and you are the easiest person to fool." — Richard Feynman, Caltech commencement, 1974

Feynman was talking about a kind of integrity that no formula can supply. The cruel twist of modern statistics is that the formulas, used the way humans naturally use them, become accomplices. They lend the warm authority of mathematics to whatever you were already hoping to find.

The villain has a clinical name: researcher degrees of freedom?Researcher degrees of freedom are the legitimate-looking analytic choices a researcher can make about data collection, cleaning, modeling, and reporting.. Every real analysis hides dozens of small, defensible decisions. Which outliers to exclude? Should you control for age? Sex? Both? Do you stop collecting data now, or run twenty more participants? You measured three outcomes — which do you report? Each choice, taken alone, is innocent. Taken together, made after you've glanced at how the numbers are leaning, they become a machine for minting false discoveries.

Simmons, Nelson & Simonsohn proved this with arithmetic before they proved it with the Beatles. By simulation, they showed that bundling just four ordinary degrees of freedom — peeking at the data and adding more if needed; choosing between two related outcome measures; adding or dropping a covariate like gender; dropping one of three experimental conditions — can inflate your chance of a false positive from the advertised 5% to as high as 61%. Flip enough innocent switches and a coin-flip's worth of noise becomes a near-certainty of "success."

The next panel lets you run that machine yourself — with a live simulation, not a canned animation. Every "study" inside it is pure static, generated with no real effect whatsoever. Watch what your freedom does to the truth.

The reference table below shows the same machine in static form: each added research freedom gives pure noise another route to look like a discovery.

The core model

What a p-value actually says (and the six lies people tell about it)

To not be fooled, you have to know precisely what the instrument reports. Many readers, including trained scientists and a worrying number of the statistics instructors teaching them, get this wrong. So, carefully:

Concrete version: suppose the null model says two groups have the same mean. If a difference this large or larger would occur 3% of the time under that model, then p = .03. That is not a 3% chance the null is true.

Read it twice. It is a statement about data, assuming a hypothesis — written P(data | null). It is emphatically not a statement about the hypothesis given the data, P(null | data). Confusing those two is the exact inversion Day 4 warned about with the disease-test problem: P(positive | sick) is not P(sick | positive), and no amount of staring turns one into the other without a prior. Here are the six misreadings to burn out of your reflexes — drawn from Greenland and colleagues' field guide to twenty-five of them:

• "p = .03 means there's a 3% chance the null is true."No. That's P(null | data) — a Bayesian quantity needing a prior. The p-value already assumes the null.
• "p = .05 means a 5% chance the result is just chance."No. Chance (the null) is the very assumption the calculation is built on; it can't also be the thing in doubt.
• "p > .05 means there is no effect."No. Absence of evidence is not evidence of absence — an underpowered study fails to detect real effects all the time.
• "1 − p is the probability the alternative is true."No. Neither p nor its complement is a probability about any hypothesis.
• "p tells you the result will replicate."No. A low p from one noisy study says little about the next study.
• "A significant result is a large or important one."No. With a big enough sample, a trivially small effect is "significant." Significance ≠ size ≠ importance.

That last one is the quiet killer, so it gets its own model.

Significance is not size: meet the effect size

A p-value entangles effect size, noise, sample size, and model assumptions. Pour in enough participants and even a microscopic, meaningless difference crosses the line. So grown-up statistics insists on reporting the effect size?An effect size reports how large a difference or association is, instead of only whether it crossed a significance threshold. separately: Cohen's d?Cohen’s d expresses a difference between two means in units of standard deviation. (a difference in means measured in standard deviations) or a correlation r. Jacob Cohen offered rough labels — d ≈ 0.2 small, 0.5 medium, 0.8 large — while cheerfully admitting they were arbitrary: "Although arbitrary, the proposed conventions will be found to be reasonable by reasonable people." The point is not the labels. The point is that a number with no effect size attached has told you almost nothing worth knowing.

The interval you've been misreading too

Confidence intervals?A confidence interval is a range produced by a method with a stated long-run coverage rate, such as 95%. are sold as the humane alternative to p-values, and they are better — but they hide their own trap.

So the natural sentence — "there's a 95% chance the true value is in this interval" — is, in the frequentist world, simply false. Your interval either contains the truth or it doesn't; the dice were thrown when you ran the study. (A Day 4 Bayesian credible interval does license that sentence — but only after you've committed to a prior. Different tool, different promise.) Nor does a CI mean 95% of your data live inside it, nor that values just outside are impossible. What it gives you, refreshingly, is a range of effect sizes compatible with your data — and that reframing, from "yes/no" to "how much, how precisely," is the heart of what reformers call the new statistics.

Why small studies can exaggerate what they detect

Two ways to be wrong: a Type I error?A Type I error is a false positive: rejecting a true null hypothesis. (false alarm — declaring an effect that isn't there) and a Type II error?A Type II error is a false negative: missing a real effect. (a miss — failing to see one that is). Statistical power?Statistical power is the probability that a study will detect an effect of a given size if that effect is real. is your chance of catching a real effect; more data buys more power. You'd assume an underpowered study just yields shrugs and "no result." Worse: it actively poisons the literature. When power is low, the only estimates lucky enough to clear the significance bar are the wildly overblown ones — the winner's curse?The winner’s curse is the tendency for selected significant estimates from noisy studies to exaggerate the true effect.. Pair that with publication practices that preferentially reward significant findings, and you manufacture a published record of effects that are both exaggerated and fragile. A sobering 2013 audit ("Power failure," Button et al.) put the median power across neuroscience at roughly 21%; a 2017 reanalysis by Nord and colleagues usefully qualifies the headline, arguing that power is not uniformly low across every neuroscience subfield. The lesson survives the caveat: where studies are underpowered, real effects are missed and the effects that surface are often inflated.

The deeper trap

You don't have to cheat to be fooled

Here is the part that should keep honest people up at night. You might read all of the above, swear off p-hacking, run exactly one pre-planned analysis, and report it faithfully — and still have an inflated false-positive rate. This is Andrew Gelman and Eric Loken's garden of forking paths?The garden of forking paths is the set of analyses a researcher might reasonably have chosen after seeing different patterns in the same data..

Their insight is subtle and a little haunting. The damage doesn't require running many analyses. It only requires that the single analysis you ran was contingent on the data you happened to see. Suppose your data had come out slightly differently — a touch noisier here, a cluster there. You'd have made a different, equally reasonable choice: controlled for age instead of income, compared women instead of the whole sample, used a median instead of a mean. The other analyses are invisible because you never ran them. But the universe of paths you would have taken, in nearby worlds, is exactly the multiplicity that inflates your error rate. As Gelman and Loken put it, you can get a false positive with no "fishing expedition" and no p-hacking at all, even with the hypothesis fixed in advance.

Notice how this rhymes with Day 5. There, the lurking decision was which variables to condition on — and conditioning on a collider?A collider is a variable influenced by two other variables; conditioning on it can create a misleading association between its causes. could manufacture a correlation out of nothing. The forking paths are the same hazard generalized: every defensible analytic choice is a junction, and the data quietly nudge you down the branch that flatters your hypothesis. The honest analyst and the p-hacker can produce the identical false result. The difference is only that one of them knows they did it.

The debate

Burn the threshold, or just lower it?

If p < .05 is so abusable, what should replace it? Here the field splits — not into crank versus expert, but into camps of serious statisticians who genuinely disagree. It helps to lay them on a single line, from "patch it" to "torch it."

What unites the whole spectrum, from cautious to incendiary, is a single migration: away from a yes/no verdict at a magic number, toward reporting how big an effect is, how uncertain you are, and how fragile the answer is to the choices you made. That last idea — fragility — is where the live 2020s frontier lives.

Open questions

What's genuinely unsettled

• Should "statistical significance" survive at all? Reformers are split between disciplining the threshold and abolishing it. Decisions still have to be made somewhere — a drug is approved or it isn't — and abolitionists owe an account of how.
• Can a multiverse ever yield a single honest verdict? Or is the choice of "reasonable specifications" an irreducibly subjective act that just relocates the forking paths one level up? PIMA and friends are early attempts; the jury is out.
• Is variation in conclusions the same as variation in reality? A subtle 2023 reanalysis noted that in some many-analyst studies the headline-grabbing disagreement was about significance, while the underlying effect sizes were quietly consistent and small. Disagreement about a verdict can exceed disagreement about the number. Don't over-learn "everything is hopeless."
• Do p-values or e-values deserve the future? The betting-based tools elegantly solve optional stopping but can demand more data and import a new modeling burden (which bet?). Coexistence looks likelier than conquest.
• And the question waiting in the AI block: when a model trained on millions of papers reports a "robust" result, has it learned to reason about evidence — or to imitate the very forking-path habits that got us here? (Days 138–145.)