information
The shortest description that predicts
It is the era when model selection needs a principle deeper than taste, and a beautiful idea arrives: the best model is the one that lets you describe the data in the fewest total bits — the bits to encode the model plus the bits to encode the data given it. Too simple a model leaves a long residual; too complex a model costs more to specify than it saves. Formalize this as a selection criterion, connect it to the deep notion that learning is compression, and use it to choose among competing hypotheses without a validation set. Get it wrong and you overfit by rewarding fit alone, or dress up a penalty as principle — description length ties model choice to the data's information.
Who this problem belongs to
The two figures whose methods fit it best, out of 32 in contention.
Kolmogorov's algorithmic information theory, developed in the 1960s alongside Solomonoff and Chaitin, defines the complexity of an object as the length of the shortest program that generates it, giving mathematical precision to exactly the intuition this problem asks to be formalized: description length as the measure of a model's true simplicity. His minimal description length is the theoretical bedrock beneath the practical criterion Rissanen later built for statistical model selection, and his earlier 1933 axiomatization of probability supplies the framework for reasoning rigorously about likelihoods at all. He did not himself build the two-part code selection criterion applied to real statistical model comparison, so a small deduction remains, but the core mathematical object the problem wants formalized is his invention.
Akaike's 1974 information criterion is the direct rival formalization of the same instinct this problem describes: that model selection needs a principled tradeoff between fit and complexity rather than raw likelihood maximization, which always favors the largest model. His AIC penalizes the number of free parameters using an information-theoretic argument grounded in Kullback-Leibler divergence between the fitted and true model, giving him deep, hands-on fluency with exactly the mathematics of balancing complexity against fit that MDL later reframes in coding-theoretic language. He built and popularized model-selection criteria used constantly without validation sets, precisely the task named here. The gap from full marks is that his criterion is asymptotic and likelihood-based rather than a two-part code, a related but distinct formalization.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
32 figures are scored on this problem. Draw it in a battle to see where you land.