information
Keep only the bits that matter
It is the era when data drowns in irrelevant features, and the question sharpens from "compress the input" to "compress the input while keeping exactly what predicts the target." Formalize the tradeoff: squeeze a representation of the input as small as possible, measured in bits, subject to preserving as much mutual information as possible about the label you care about. The optimal representation throws away everything that does not help predict and nothing that does. Derive it, and use it to understand what a good learned representation even is. Get it wrong and you keep noise that inflates the model, or discard signal you needed — this view of relevance underlies feature selection, sufficient statistics, and reading what deep networks learn.
Who this problem belongs to
The two figures whose methods fit it best, out of 35 in contention.
Shannon's 1948 definition of mutual information as the reduction in uncertainty about one variable given knowledge of another is the exact currency this problem trades in: 'preserving as much mutual information as possible about the label' is stated in his own vocabulary, and his broader information theory establishes why bits are the right unit for measuring what a representation keeps or discards. His rate-distortion theory, formalizing the tradeoff between compressing a signal and preserving fidelity to it, is the direct mathematical ancestor of the information bottleneck's compression-versus-relevance tradeoff, differing mainly in substituting a label for a reconstruction target. He is not scored higher only because the specific bottleneck formalization, with its Lagrangian tradeoff and iterative solution, arrived decades after his own work.
Cover's textbook with Joy Thomas systematized mutual information, the data processing inequality, and rate-distortion theory into the standard modern toolkit, and the data processing inequality specifically, that no processing of a signal can increase its information about a hidden variable, is the deep theorem underlying why a compressed representation can only lose relevant information relative to the raw input, never gain it. His research on universal prediction and portfolio theory shows the same instinct for extracting exactly the information that matters for a task and discarding the rest. He did not develop the information bottleneck formalization itself, but his textbook is likely where any practitioner, including the method's own inventors, learned the tools it depends on.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
35 figures are scored on this problem. Draw it in a battle to see where you land.