information
Compress the English novel
It is the 1950s, and Shannon has just measured something startling: the true entropy of English is far below what a letter-by-letter count suggests, because language is thick with predictable structure. Your task is to compress an English novel far below that naive entropy estimate by modeling that structure — and then to see the deep equivalence hiding underneath, that compressing text well and predicting the next symbol well are, mathematically, the same problem. The better your model of language, the fewer bits you need. Get the connection wrong and you treat compression and language modeling as separate crafts, missing that a great compressor IS a great language model — an identity that, decades later, becomes the engine of modern text prediction. Squeeze the bits by understanding the language.
Who this problem belongs to
The two figures whose methods fit it best, out of 35 in contention.
This is Shannon's own experiment. In 'Prediction and Entropy of Printed English' (1951) he had human subjects guess the next letter of text and used their guesses to bound the entropy of English at roughly one bit per character — far below the 4.7 bits of a uniform alphabet or the ~4.1 of a unigram count. The 1948 paper already supplied the machinery: n-gram approximations to English, the source coding theorem tying achievable compression to source entropy, and the explicit framing of a language source as a stochastic process. The problem's deep identity — a predictor and a compressor are the same object — is stated almost verbatim in his 1951 guessing-game construction, where a predictor is converted into a reversible code. No anachronism whatsoever; he owns every axis of this problem.
Jelinek spent the 1970s-90s at IBM building exactly the machinery this problem demands: statistical n-gram language models, trained on corpora, evaluated by perplexity — which is nothing but the per-symbol codelength of the model, the compression view made operational. His speech-recognition group treated language modeling as a source-coding problem from the start, drawing directly on the Shannon lineage (Jelinek's own early work was in information theory under Fano at MIT). Smoothing, back-off, and interpolated estimators — the craft of squeezing predictive probability out of finite text — are precisely how you compress a novel below its naive entropy. His toolkit postdates the 1950s framing, but the problem explicitly asks for the prediction-compression equivalence, and Jelinek is the person who industrialized it. He lacks only the neural machinery for the deepest squeeze.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
35 figures are scored on this problem. Draw it in a battle to see where you land.