small-sample
Fifty examples in the test set
It is the present, and a team is about to deploy a model whose accuracy they have measured on a held-out set of only fifty examples — because labeling more costs an expert's time no one will spend. From forty-eight correct out of fifty you must report an honest confidence interval on the true accuracy, using exact finite-sample methods rather than a large-sample approximation that lies at this size, and warn how little a single test set can promise about deployment. Get it wrong and a system ships on a point estimate that looks like 96% and could plausibly be 85%, failing in the field exactly where the tiny test set had no power to see it coming.
Who this problem belongs to
The two figures whose methods fit it best, out of 47 in contention.
Fisher's exact test, developed in the 1920s specifically because chi-squared approximations broke down on small contingency tables, is the direct ancestor of the exact binomial interval this problem demands: rather than approximating with a normal curve, you compute probabilities from the binomial distribution itself, which is precisely why the Clopper-Pearson exact interval descends from Fisher's small-sample philosophy. His broader insistence that inference should be exact wherever computation allows, not approximate for convenience, is exactly the discipline the problem wants applied to a fifty-example test set. He is not the literal author of the Clopper-Pearson construction, which is why the score falls just short of a perfect mark. That closeness to the method's historical origin is what keeps Ronald Fisher among the stronger carriers in this particular batch.
Vapnik's VC theory, developed with Chervonenkis from the 1960s onward, is the mathematical foundation for the entire modern concern this problem describes: how much can a model's measured performance on a finite test set actually be trusted to predict deployment behavior, and how does that trust scale with sample size. His generalization bounds formally quantify the gap between empirical and true accuracy, giving the theoretical backbone for why a fifty-example test accuracy is far less reliable than it looks. He did not himself derive the exact binomial interval Clopper and Pearson built, working instead one level up in learning-theoretic generalization, which keeps him just below Fisher. That closeness to the method's historical origin is what keeps Vladimir Vapnik among the stronger carriers in this particular batch.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
47 figures are scored on this problem. Draw it in a battle to see where you land.