classification
When 0.9 must mean ninety percent
It is the era when model scores start driving decisions no one double-checks, and a subtle failure surfaces: a classifier that says "0.9" is right only seven times in ten when it does. Its ranking may be excellent and its probabilities still be lies. Take a well-discriminating model and make its outputs honest — so that among all cases it scores 0.9, ninety percent truly belong to the class — using a held-out set and a monotone recalibration that preserves the ranking. The stakes are wherever a number is consumed as a probability: a weather forecast, a triage score, a bet. Get it wrong and a decision-maker multiplies costs by miscalibrated odds and is systematically, invisibly wrong.
Who this problem belongs to
The two figures whose methods fit it best, out of 71 in contention.
Barber's research on conformal prediction and knockoffs is the direct modern answer to exactly this problem: conformal methods construct prediction sets with a distribution-free, finite-sample guarantee using a held-out calibration set, precisely the monotone, guarantee-preserving recalibration this problem describes, without assuming the underlying model family is correct. Her work on inference after selection also addresses the honesty problem underlying miscalibration, that a score optimized to discriminate well is not automatically an honest probability. Conformal prediction's calibration step is essentially a rigorous generalization of what this problem asks for, holding out data specifically to make a model's outputs trustworthy. Her toolkit is arguably the single most historically appropriate one in this entire roster for this exact problem, built for precisely this purpose.
Efron's bootstrap, from 1979, gave statistics a general-purpose tool for estimating a statistic's sampling distribution by resampling, directly useful for constructing the kind of held-out validation and honest uncertainty quantification this calibration problem demands. His empirical Bayes work explicitly addresses correcting a naive estimate using information from related cases, methodologically close in spirit to recalibrating a model's raw scores against observed outcome frequencies. His career-long focus on honest, computationally grounded uncertainty quantification, rather than convenient but unverified theoretical claims, is exactly the discipline this problem's 0.9-must-mean-ninety-percent demand requires. He did not build isotonic regression or Platt scaling specifically, but his general resampling and empirical correction philosophy underlies much of the modern calibration toolkit.
Fought here
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
71 figures are scored on this problem. Draw it in a battle to see where you land.