Statistics & the Art of Not Fooling Yourself

§1 The original sin

A tool built from two incompatible blueprints

Here is a fact that explains an astonishing amount of misery: the standard procedure every science student is taught as “hypothesis testing” is a hybrid. It welds Fisherian evidence to Neyman-Pearson long-run decision rules, and that hybrid often causes confusion.

On one side stood Ronald Fisher. In Statistical Methods for Research Workers (1925), he offered the $p$-value as an informal, continuous gauge of evidence against a single null hypothesis — a measure of how strange your data would look if nothing were going on. He needed no alternative hypothesis, no rejection region, no long run. He suggested $p<.05$ only as a homely rule of thumb, the rough point past which a coincidence starts to feel implausible, and explicitly said you could pick a different line if you preferred. For Fisher, a small p was a prompt to think, not a verdict to obey. The null, he wrote, is never proved — only, occasionally, embarrassed.

On the other side stood Jerzy Neyman and Egon Pearson. In a series of papers culminating in 1933, they rebuilt the whole enterprise as a decision rule, not a measure of evidence. You specify two hypotheses up front, fix in advance your tolerable rate of false alarms (α, the Type I error rate) and misses (β, the Type II rate; power = $1-\beta$), draw a line, and thereafter behave: reject or accept, accepting that over the long run you’ll be wrong at the rates you chose. There is, pointedly, no measure of evidence for the case in front of you — only a procedure with known long-run properties. Neyman thought Fisher’s inductive “evidence” talk was mush; Fisher thought Neyman had turned science into factory quality-control. The feud was lifelong and genuinely vicious.

Once you see the seam, the whole replication crisis reads differently. We took a flexible Fisherian prompt-to-think, froze it into a Neyman–Pearson gate, stripped out the part of Neyman–Pearson that made the gate valid (pre-committed $\alpha$, adequate power, a real alternative), and then published only what got through. The wonder is not that so much fails to replicate. It’s that anything survives.

§2 Why repeated testing raises false alarms

The two facts that let repeated analysis manufacture false positives

The False-Positive Factory wasn’t a trick of clever programming. It rests on two mathematical facts so simple they’re almost insulting — and once you hold both, you understand why enough uncorrected looks can turn noise into apparently significant evidence.

Fact one: under the null, small p-values still arrive by chance

Run an exact test with continuous data where nothing is happening, under the assumptions of the null model, and the $p$-value you get is a uniform random number between 0 and 1. For discrete, composite, or conservative tests, valid $p$-values are often super-uniform: $P(p\le \alpha )\le \alpha$, not necessarily exactly equal. The practical warning survives that qualification. In the calibrated continuous case, exactly 5% of the interval [0, 1] lies below .05; in conservative cases the rate is no larger than that. But flip it around and the menace appears: even by-the-book, effect-free experiments sometimes hand you a “significant” result by raw luck. Run the experiment twenty ways — twenty outcomes, twenty covariates, twenty subgroups — and you’ve effectively bought many tickets on a small-probability prize. The expected number of “discoveries” in a drawer full of nothing is driven by how many times you looked.

Use the live panel to watch an idealized continuous exact test. With the true effect at zero, $p$-values lie flat, with 5% falling below the line. Turn up a real effect and the distribution leans toward smaller values: genuine signal makes very small significant $p$-values more common than barely significant ones. That surplus is a literature’s pulse; its absence is a warning sign.

Fact two: keep looking without correction, and the false-alarm risk grows

The optional-stopping toggle in the main factory hid a result that ought to be more famous than it is. Suppose you test after every new subject and stop the instant $p$ dips below .05. If there’s a real effect, fine. But if there isn’t — if the null is exactly true — and you’re allowed to keep sampling indefinitely, then the probability that you eventually cross .05 is not 5%; in the mathematical idealization, it approaches 100%. This is “sampling to a foregone conclusion,” proved by Anscombe in 1954 and hammered home by Armitage and colleagues in 1969: the wandering $p$-value, given enough patience, is liable to stumble below any fixed line at some point, purely by chance, even when nothing is there.

This is why peeking is not a venial sin but a structural exploit — and why Day 4’s e-values and anytime-valid tests are such a clean fix: they’re built so that the guarantee survives unlimited peeking. It’s also why real clinical trials, which must monitor accumulating data for safety, use alpha-spending and group-sequential designs — they pay a small statistical tax at each interim look so the total false-alarm budget still adds up to 5%. Optional stopping done lawlessly is cheating; done with a spending rule, it’s one of the triumphs of 20th-century statistics.

§3 Reading the fingerprints

The forensic turn

If a real effect leaves many very small significant $p$-values, while p-hacking leaves a pile-up just below .05, then you can audit a whole literature from its published numbers alone — without rerunning a single experiment. That realization launched a small, slightly gleeful field of statistical forensics.

The flagship tool is p-curve (Simonsohn, Nelson & Simmons, 2014 — yes, the same trio from the main lesson). Collect the significant $p$-values from a set of studies on one effect and look at their shape. A healthy p-curve has more very small significant $p$-values than barely significant ones: many .01s, fewer .04s, because real signal mostly yields convincingly small $p$-values. A literature with no underlying effect, propped up entirely by p-hacking, shows the suspicious pattern instead — an unnatural surplus of .04s and .049s, the telltale residue of researchers nudging results just over the line and stopping the moment they cleared it. A pile-up just below .05 is suspicious.

And then there are the sting operations — demonstrations built to make the multiple-comparisons problem impossible to wave away:

Hype filter: p-curve and these forensic tools are established as diagnostics — but they detect patterns, not guilt; an innocent literature with publication bias can mimic some signatures, so they flag suspicion, not proof.

§4 The full arithmetic

How wrong is a “significant” result, really?

Now for the number that should be printed on the first page of every statistics textbook and almost never is. You ran a clean study, no hacking, and got $p=.05$. Intuition whispers: “about a 5% chance this is a fluke.” Intuition is catastrophically wrong, and we can prove it with nothing more than Day 4’s Bayes.

A $p$-value lives in the world of $\mathrm{P}(\text{data}\mid \text{null})$. What you actually care about is $\mathrm{P}(\text{null}\mid \text{significant result})$ — the chance your shiny finding is bogus. Bridging the two requires a prior: how plausible was your hypothesis before the data? And the moment you supply a realistic prior, the picture darkens. The engine is the same one Ioannidis used in his famous Day 2 essay “Why Most Published Research Findings Are False”: combine the prior odds, the study’s power, and $\alpha$, and read off the false positive risk — the share of your “significant” results that are false alarms.

Use the calculator to see how often significant findings are wrong—even with a coin-flip prior and decent power, and especially when the hypothesis is a long shot.

Two refinements make the news worse, not better — and they’re the real essence here.

It’s grimmer than the grid suggests (Colquhoun). The calculator above uses the whole .05 tail as the false-positive rate. But David Colquhoun (2014, 2017) pointed out that a result sitting right at $p=.05$ is barely more likely under a real effect than under the null, so its calibrated false-positive risk is higher than the tail suggests. His verdict, for an even-odds prior: observe $p\approx .05$ and the minimum chance you’re wrong is around 26% — not 5%. Demand a genuine discovery rate, and $p<.05$ simply isn’t a high enough bar. This is the quantitative heart of the Benjamin et al. push toward .005 you met in the main lesson.

The null is almost never exactly true anyway (Meehl’s crud factor). Paul Meehl observed that in large social-science datasets, everything correlates a little with everything — birth order with politics, height with humor — because the world is a dense web of faint real connections he called the crud factor. So a point null of “exactly zero” is a straw man; with a big enough sample you can reject it for any pair of variables you like. “Statistically significant” then certifies only that your sample was large, not that you found anything that matters. It’s the deepest reason significance and importance are different animals.

§5 The better toolkit

What the people who know this reach for

Demolition is easy; the appendix owes you the rebuild. The estimation-era toolkit isn’t exotic — it’s a handful of moves that quietly fix the specific failures above.

Better errors: Type S and Type M

Andrew Gelman and John Carlin (2014) argue that in the low-power world we actually inhabit, the classic Type I / Type II framing asks the wrong question. Two sharper failures matter more. A Type S (“sign”) error is getting the direction wrong — concluding a treatment helps when it harms. A Type M (“magnitude”) error is the exaggeration ratio — how many times too large your “significant” estimate is. Their punchline ties straight to the main lesson’s winner’s curse: when power is low, the few estimates that reach significance aren’t just uncertain, they’re systematically inflated, often two- or threefold. The published effect isn’t a slightly noisy truth; it’s a funhouse-mirror enlargement of it.

Proving a negative: equivalence testing

The main lesson warned that "$p>.05$" does not mean “no effect.” But sometimes you genuinely want to conclude there’s nothing meaningful here — and you can, with equivalence testing (the TOST procedure; Lakens, 2017). You define the smallest effect you’d care about, then test whether the true effect is confidently smaller than that. It flips absence-of-evidence into evidence-of-absence, legitimately. A non-significant result plus a passed equivalence test is a real conclusion; a non-significant result alone is just a shrug.

Facing Multiplicity: Controlling the False Discovery Rate

When you really must run many tests (every gene, every brain region), the old fix was Bonferroni — divide $\alpha$ by the number of tests — which is correct but so brutal it leaves you no power. Benjamini and Hochberg’s false discovery rate (1995), one of the most-cited papers in all of science, changed the target: instead of guaranteeing zero false positives among all tests, control the expected proportion of false positives among the ones you flag. It’s the pragmatic bargain that made genomics and neuroimaging statistically livable — accept a known, small fraction of duds in exchange for actually finding things.

Renaming the interval to mean what it is

Finally, a small move with large clarifying power: Amrhein, Greenland and colleagues urge calling a confidence interval a compatibility interval — the range of effect sizes reasonably compatible with your data and model. The word kills two errors at once. It stops you treating values inside as proven and values outside as impossible, and it nudges you to report the whole interval as a statement about magnitude and uncertainty rather than a yes/no membership test for zero.

§6 The case for the defence

Significance testing, done severely

It would be too tidy to end with “p-values bad, estimation good.” Evenhandedness — and accuracy — demand a serious counterweight, and the philosopher Deborah Mayo supplies it. In Statistical Inference as Severe Testing (2018), she defends error-statistical reasoning against both the abolitionists and the Bayesians, with a single demanding principle.

You have evidence for a claim only to the degree it has passed a test it would probably have failed, had the claim been false.

That is severity, and notice it is Day 2’s Popper rendered quantitative: a result corroborates a hypothesis only if the test genuinely risked refuting it. On this view, the rot isn’t the $p$-value — it’s the abandonment of severity. A $p$-value squeezed out of a forking-paths fishing trip is worthless not because it’s a $p$-value but because the “test” was never severe: it could hardly have come out any other way. Mayo’s warning to the reformers is pointed: drop all thresholds and error-control, and you don’t get more nuanced science — you get less accountable science, with no public standard for when a claim has actually earned its keep. The synthesis the main lesson reached for lives here: not “abolish” or “obey,” but demand severity, report magnitude and uncertainty, and make the test one the hypothesis could have flunked.

§7 The bridge to tomorrow

A p-value, measured in coin flips

One last idea, and it’s a beauty — because it quietly turns the page to Day 7. Sander Greenland (2019) suggests we stop staring at the bare $p$-value and instead take its surprisal, borrowed straight from information theory: the S-value, defined as $S=-{\log }_{2}p$. This converts the p-value into bits of surprisal, expressing how incompatible the data are with the tested model under its assumptions — and bits, as we’ll see tomorrow, are the universal currency of surprise.

The translation is wonderfully concrete. Ask: how many fair coins would I have to flip and get all heads to be this surprised? That count is the S-value. A result at $p=.05$ carries about 4.3 bits — roughly the surprise of calling four or five coin tosses in a row. Stated that way, the spell breaks. Four heads in a row is mildly interesting; it is not a discovery, and you would never bet your reputation on it.

Move the slider to translate any $p$-value into Day 7’s information units.

The conversion translates common $p$-values into Day 7’s information units.

What to actually do

The working scientist’s defence kit

Distilled from everything above and in the main lesson — the moves that convert luck into knowledge:

• Preregister the confirmatory test. Freeze the confirmatory path before you see the data; label everything else exploratory.
• Report the effect size with a compatibility interval, always. The magnitude and its uncertainty are the result; the $p$-value is a footnote.
• Power the study for the smallest effect you'd care about — and remember that a “significant” result from an underpowered study is probably inflated (Type M).
• Translate your p into bits. If it’s only a few coin flips’ worth of surprise, say so, and don’t oversell.
• Check robustness across reasonable analyses (a specification curve), and ask whether the result survived a severe test or merely a convenient one.
• When you must test many things, control the false discovery rate — not the family-wise error into oblivion.
• If you want to claim "no effect," run an equivalence test. Absence of evidence isn’t evidence of absence until you’ve actually looked for absence.

Field map

Ten waves, sorted by the failure mode they attack

Filter the atlas by maturity. Green means the core theory is established; amber means serious and promising but still settling; red means potentially transformative and currently dangerous.

The atlas below sorts ten post-2020 methods by the failure mode they attack and labels their maturity.

§1 The p-value that survives the refresh button

Anytime-valid inference: the end of illegal peeking

The old fixed-sample test is fragile because it assumes you decided the sample size before seeing the data. But modern science does not behave like that. A/B tests stream in hourly. Platform teams refresh dashboards. Clinical trials have interim looks. Labs add subjects when the first batch is “almost significant.” The ordinary p-value is not built for this world.

Anytime-valid inference changes the contract. Instead of a p-value that is valid only at a pre-fixed stopping time, it uses e-values, e-processes, always-valid p-values, and confidence sequences: evidence measures and intervals designed to remain valid at arbitrary stopping times. You can look early, look often, stop when the evidence is decisive, or continue when it is ambiguous, without silently inflating the false-positive rate.

This is not a decorative upgrade. It attacks one of Day 6’s central machines for fooling yourself: optional stopping. The frontier after 2020 is the consolidation of a “game-theoretic statistics / safe anytime-valid inference” language. It treats evidence as a capital process in a game against nature. If the null is true, your evidence wealth cannot systematically grow without bound; Ville-style inequalities turn that into sequential tests and confidence sequences.

The hype filter: the mathematics is now serious and mature. The practical risk is cultural. People like dashboards but dislike changing the number they report. If a team runs a continuously monitored experiment and still reports ordinary fixed-horizon p-values, the UI may look modern while the inferential contract is 1930s paper-and-pencil.

§2 The guarantee that wraps a black box

Conformal prediction becomes conformal risk control

Conformal prediction is the simplest profound idea in the current statistics/ML interface. Train any predictive model. Hold out calibration data. Measure how wrong the model is on those held-out cases. Use the empirical quantile of those errors to form a prediction set for the next case. Under exchangeability, the set covers the true answer at the advertised rate in finite samples, regardless of whether the model is linear, neural, brilliant, or embarrassing.

A common simplification treats conformal prediction as “intervals for any model.” That is correct but incomplete. The post-2020 frontier is broader: conformal risk control asks not merely “does the true label fall in the set?” but “can we control a chosen risk function?” In vision and language tasks, the useful error metric may be false negative rate, graph distance, abstention error, retrieval miss-rate, or token-level F1. Conformal risk control extends the conformal idea to monotone losses and gives finite-sample control of the expected risk.

That matters for AI systems because the output is often not a scalar. A chatbot answer, segmentation mask, medical image region, or structured extraction is not a single yes/no label. Risk control lets you build a calibrated ship / abstain / escalate / ask for review wrapper around a black-box model. In the safest version, the model remains a prediction engine; the conformal wrapper becomes the trust policy.

The hype filter: marginal coverage has the advertised guarantee; the harder target is conditional coverage. Under distribution shift, the classical exchangeability guarantee weakens. Recent work handles weighted and non-exchangeable settings, conditional-style guarantees, and e-value variants, but the practitioner still has to ask whether the calibration data resemble deployment. Conformal methods make uncertainty explicit. They do not make a bad deployment context safe.

§3 Using AI predictions without swallowing their errors

Prediction-powered inference: the rectifier is the point

AI can label a million galaxies, satellite tiles, pathology patches, protein records, or clinical notes. But a statistical estimate from those predictions is not automatically valid, because model errors are not random clerical typos. They are structured. They follow the model’s blind spots.

Prediction-powered inference (PPI) solves the problem by refusing to trust the model alone. You combine a small labeled sample, where humans or instruments provide gold-standard truth, with a much larger unlabeled sample on which the model predicts. The labeled sample estimates a rectifier: the correction for the model’s bias on the estimand you actually care about. The unlabeled sample supplies cheap precision; the labeled sample supplies validity.

PPI++ improves the efficiency of this idea by adapting to prediction quality. If the model is useful, the interval shrinks. If the model is weak, the method falls back toward the labeled-only interval rather than letting bad predictions poison the estimate. Cross-prediction-powered inference goes further by distributing prediction and correction across folds. The architecture is exactly the incoming-wave pattern: use the black box for power, but put the guarantee in the wrapper.

The limitation is practical, not philosophical. You still need a gold-standard labeled sample drawn from the right target population. You still need to define the estimand before the model seduces you into measuring whatever it can label. And if the labeled set misses the subgroup where the model fails, the rectifier can inherit that blind spot.

§4 Machines for nuisance, humans for assumptions

Double / debiased machine learning for causal effects

Day 5’s warning was blunt: association is not intervention. To estimate a causal effect from observational data, you must adjust for confounders, and you must do it without imposing a fantasy linear model on a nonlinear world.

Double/debiased machine learning (DML) uses ML where ML is strongest: learning high-dimensional nuisance functions such as outcome regressions and propensity scores. Then it uses orthogonal scores and cross-fitting where statistics is strongest: protecting a low-dimensional target estimate from first-order errors in those nuisance fits. You let the random forest, gradient booster, or neural net approximate the messy parts, then construct the causal estimator so that small nuisance mistakes wash out.

The post-2020 change is not that DML was invented after 2020; it was not. The change is that software and applied practice caught up. DoubleML implementations in Python and R, causal-forest workflows, and biomedical heterogeneous-treatment-effect reviews have turned “orthogonalize the nuisance” from a seminar phrase into an applied workflow.

The hype filter is essential: DML weakens modeling assumptions; it does not remove causal identification assumptions. If the back-door set is wrong, if a confounder is unmeasured, if treatment assignment depends on hidden variables, a beautiful orthogonal score will compute a precise answer to the wrong causal question. DML solves flexible adjustment, not the metaphysics of causation.

§5 Explore first, infer second, pay the bill

Inference after discovery: post-selection, causal discovery, and data fission

The garden of forking paths is not going away. Modern analysis often begins with discovery: cluster the cells, find the subgroup, select the variables, learn the causal graph, choose the image region, tune the transformation, ask the AI analyst to “find what matters.” The fatal mistake is to use the same data to discover a target and then pretend the target was specified in advance.

Post-selection inference is the statistical attempt to account for that bill directly. The classical approaches condition on the selection event, split the data into discovery and confirmation sets, or build simultaneous intervals wide enough to survive selection. The post-2020 frontier adds three especially relevant ideas.

The hype filter: this is not easy, and it is not a substitute for preregistration when preregistration is possible. But it is the right frontier for exploratory science because “never explore” is not a serious rule. The serious rule is: if the data suggested the question, the inference must know that.

§6 When the released data are not the data

Differential privacy turns uncertainty into a public design problem

Open science wants data sharing. Human subjects, firms, hospitals, and governments want confidentiality. Differential privacy is the strongest modern language for this tradeoff: random noise is added so that one person’s presence or absence has bounded influence on the released output.

For Day 6, the statistical sting is this: privacy noise is not a file-format detail. If you analyze differentially private statistics or synthetic data as if they were ordinary observations, your intervals can be too narrow, your p-values too confident, and your null findings too frequent. The privacy mechanism is part of the data-generating process, so valid inference has to include it.

Post-2020 work has been building noise-aware confidence intervals, private bootstraps, inference from DP synthetic data, and guidelines for evaluating whether a claimed privacy guarantee means what users think it means. This is not glamorous, but it is central. In the next decade, many researchers will never touch the raw administrative or medical data. They will touch privacy-filtered outputs. The frontier question becomes: can we make those outputs scientifically useful without making people identifiable?

The hype filter: differential privacy protects individuals by making some analyses less precise. A valid private interval may be much wider. That is not failure; that is the price tag made visible. The dangerous failure is a narrow interval that forgot the noise.

§7 The model is a workflow, not a formula

Bayesian workflow and simulation-based calibration

Bayesian statistics is often sold as a clean alternative to p-values: put a prior on the unknowns, compute a posterior, report probabilities about parameters. That is attractive, but it can become another ritual if the prior is unexamined, the model is misspecified, and the sampler is silently broken.

The post-2020 Bayesian workflow movement says: the unit of trust is not the posterior; it is the workflow that produced and criticized the posterior. Prior predictive checks ask whether the model could generate plausible data before seeing the data. Posterior predictive checks ask what the fitted model still cannot reproduce. Cross-validation asks whether predictions survive held-out cases. Simulation-based calibration checks whether the inference algorithm can recover known parameters when the data are simulated from the model.

This belongs in the cutting-edge appendix because it is a direct answer to Day 1’s stopped-clock worry. A posterior can be numerically impressive for accidental reasons. A workflow makes the path visible: what was assumed, what was simulated, what failed, what changed, and what uncertainty remains.

The hype filter: model checking is not model proof. Posterior predictive checks can miss the discrepancy that matters. SBC validates computation under the assumed generative model, not the truth of the model itself. But compared with a lone posterior interval, a documented workflow is a much harder machine to fool.

§8 The error-correction system grows teeth

Statistical forensics and replicability triage at scale

Appendix I’s forensic toolkit was artisanal: GRIM checks, statcheck, p-curve, digit anomalies, spreadsheet oddities. The post-2020 change is that the stakes and scale have grown.

The Francesca Gino / Data Colada case showed both the power and the danger. Data Colada published detailed anomaly analyses in 2023. Harvard’s investigation concluded research misconduct; Gino denied wrongdoing and sued. In September 2024, a federal judge dismissed all claims against the Data Colada defendants; in May 2025 Harvard revoked Gino’s tenure and terminated her employment, while some legal disputes involving Harvard continued. The scientific lesson is not “bloggers should run universities.” It is that transparent forensic arguments can become part of the formal error-correction system.

The next layer is triage. We cannot replicate every claim. Decision markets have shown that scientists’ aggregated forecasts can identify high- and low-replicability findings: in one Nature Human Behaviour study, top-market studies replicated far more often than bottom-market studies. The Center for Open Science’s Predicting Replicability Challenge adds machine-learning models trained on thousands of prior replication outcomes. The early results are mixed: Round 1 was weak by some metrics; Round 2 improved. That nuance matters. Replicability prediction is a promising smoke alarm, not a truth machine.

The hype filter: forensic tools can be misused as accusation engines. A weird dataset is not automatically fraud; a low replicability forecast is not a verdict. The frontier needs transparent code, calibrated false-alarm rates, and institutional protections for both whistleblowers and accused researchers.

§9 Synthetic humans are not humans

Silicon samples and LLM social simulations

Silicon sampling asks an irresistible question: if a language model has absorbed enough social text, can it act as a simulated survey respondent or experimental participant? The first wave of results was exciting. Conditioned on demographic backstories, models sometimes reproduced aggregate opinion patterns. Agent-based simulations produced believable local interactions. Large-scale replications of psychology and management scenarios found that LLMs often matched the direction of main effects.

The second wave is the caution. LLMs can distort correlations even when margins look right. They often compress distributions toward modal answers. They may amplify effect sizes. They can perform worse on socially sensitive topics. Their “population” is a residue of training data, alignment tuning, prompt wording, and model version. That is not a sample frame. It is a machine-shaped mirror.

The defensible use case is therefore narrow but real: pilot instruments, debug ambiguous survey wording, estimate rough sample-size needs, test whether a manipulation is even legible, and generate hypotheses before spending human-subjects budgets. The reckless use case is confirmatory evidence without human validation.

Reporting standards should become severe: model name and version, date, system prompt, persona construction, temperature, sampling scheme, repeated-run variability, validation against human data, and a preregistered rule for when the simulation is allowed to count. Otherwise silicon samples become the garden of forking paths wearing a human mask.

§10 When the analyst is silicon too

AI analysts, automated science, and benchmark contamination

The most dangerous frontier is not AI as a model inside an analysis. It is AI as the entity that chooses the analysis.

Data-analysis agents can clean data, write code, choose models, run tests, generate plots, and narrate findings. New benchmarks such as StatLLM evaluate statistical code; data-agent benchmarks evaluate longer workflows. Studies of LLM data analysis show variation across models, prompts, temperatures, and repeated executions; even the same task and dataset can lead to meaningfully different estimates. At the far edge, autonomous research systems now generate ideas, write code, run experiments, plot results, draft papers, and perform self-review.

This does not make AI useless. It makes AI analysis statistically dangerous in a very familiar way. The many-analysts problem has become cheap enough to run in seconds. The “researcher degrees of freedom” are now prompt degrees of freedom, model degrees of freedom, retry degrees of freedom, tool-call degrees of freedom, and hidden chain-of-reasoning degrees of freedom.

AI benchmarking has the same pathology at field scale. Public leaderboards are repeated tests. Benchmark contamination happens when static benchmarks leak into training data. Developers can tune to the benchmark. Scores can rise because models generalize better, because they memorized more, because the harness changed, or because the benchmark itself is exhausted. The statistical object is no longer “accuracy on a test set”; it is a dynamic game between model builders, benchmark maintainers, and hidden evaluation data.

The responsible frontier looks like this: prompt logs, containerized code, repeated independent agent runs, multiverse reports, hidden holdout evaluations, contamination audits, preregistered analysis instructions, and human review of the decisions the system made. The dream is an AI analyst that exposes every fork. The nightmare is an AI analyst that silently chooses the fork that gets published.

The through-line

The incoming wave is not “trust AI.” It is “wrap discretion.”

The strongest post-2020 methods share a pattern. They accept that the model may be a black box, the analyst may explore, the experiment may be monitored continuously, the data may be privatized, and the research system may be too large for manual checking. Then they ask: where can a guarantee still be attached?

The original statistics crisis was not caused by evil people. It was caused by flexible humans using brittle procedures in incentive systems that rewarded clean stories. The next crisis will rhyme: flexible machines using brittle evaluations in incentive systems that reward impressive outputs. The cure is the same at a higher level: make the forks visible, attach guarantees where they belong, and refuse to mistake polish for contact with truth.