small-sample
Prediction intervals without a model
It is the modern era of machine learning, and models everywhere emit point predictions no one can trust the error bars on. You must give a 90% prediction interval for the very next observation — one that genuinely covers nine times in ten — using only forty past observations and NO distributional assumptions whatsoever. No bell curve to lean on, no fitted model to believe. The guarantee must hold in finite samples, provably, or it is worthless. Deliver a method whose coverage is a theorem, not a hope. Get it wrong and automated systems ship confident predictions with intervals that silently fail exactly when they matter — in medicine, in finance, in the tails where people get hurt.
Who this problem belongs to
The two figures whose methods fit it best, out of 35 in contention.
This is Barber's home problem, stated almost as an abstract of her papers. Conformal prediction — rank the new point's nonconformity among the forty, invoke exchangeability, read off an interval — delivers exactly 90% coverage as a finite-sample theorem with no distributional assumptions, and her work supplies the modern instruments: the jackknife+ (with Candès, Ramdas, Tibshirani) with its provable worst-case coverage when data are too few to split, conformal methods under covariate shift for when exchangeability itself bends, and precise impossibility results showing conditional coverage cannot be had assumption-free — the honest boundary a defender of the method must state. Her knockoffs work (with Candès) shows the same signature elsewhere: exact finite-sample error control from clever symmetry, no model believed. Every clause of the problem, including the defense under interrogation, is her published specialty. Named winner; scores accordingly.
Wasserman is a named winner and earns it on the published record. With Jing Lei, Max G'Sell, Ryan Tibshirani and others he developed distribution-free predictive inference for regression — split and full conformal methods whose 90% coverage at n=40 is an exact finite-sample theorem under exchangeability alone, precisely this problem's deliverable. He also proved the sobering complements: results with Lei and with Barber-line collaborators showing which distribution-free goals (conditional coverage, assumption-free confidence for densities) are impossible, so he can tell the commanders exactly what can and cannot be guaranteed — the problem's 'defend that the interval means what you claim' clause. His textbooks (All of Statistics, All of Nonparametric Statistics) built the statistics–ML bridge this modern-era problem sits on. Slightly behind Barber only because conformal's sharpest recent instruments carry her name.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
35 figures are scored on this problem. Draw it in a battle to see where you land.