information
What is a bit, exactly?
It is 1948 at Bell Labs, and "information" is a word engineers use loosely and no one has measured. Before channels and codes comes the founding act: pin down a single number that quantifies the uncertainty in a random source — the average surprise per symbol — and prove it is the only measure satisfying a few self-evident requirements. From a handful of axioms about how uncertainty behaves under combination and refinement, derive that the formula must be, essentially uniquely, the entropy. Then show it is not arbitrary but the operational limit on compression. Get it wrong and information theory rests on a made-up quantity — instead, this shows the bit is not a convention but a discovered constant of communication, forced by logic.
Who this problem belongs to
The two figures whose methods fit it best, out of 37 in contention.
This is Shannon's own 1948 paper, 'A Mathematical Theory of Communication,' written at Bell Labs precisely to pin down what 'information' should mean. He derives entropy from a small set of self-evident axioms about how uncertainty should behave, continuity in the probabilities, monotonic increase with more equally likely outcomes, and additivity when a choice is broken into successive choices, and shows the formula is essentially unique given those requirements. He then proves entropy is not arbitrary but the operational limit on lossless compression, the source coding theorem, precisely the closing claim this problem demands. He built both the axiomatic derivation and its operational meaning in the same paper. The tiny deduction from perfect marks reflects only that the axioms were slightly sharpened by later authors like Khinchin.
Cover's textbook with Joy Thomas gives the definitive modern treatment of Shannon's axiomatic derivation of entropy, walking through the uniqueness proof from the continuity, monotonicity, and grouping axioms with a rigor and clarity that has trained generations of students in exactly this argument. His own research on universal source coding and the operational interpretation of entropy as a compression limit extends and reinforces the second half of this problem, that entropy is not a convention but a discovered operational constant. He did not himself invent the entropy axioms or their uniqueness proof, arriving as the field's most authoritative systematizer and extender of Shannon's original 1948 argument rather than its originator, keeping him just below Shannon.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
37 figures are scored on this problem. Draw it in a battle to see where you land.