David MacKay

Life and career

David MacKay died in April 2016, twelve days short of his forty-ninth birthday, of stomach cancer. He had been knighted a few months earlier. He spent the last stretch of his illness blogging about his own treatment with the same clarity and arithmetic honesty he had brought to everything else, and the field lost, at a stroke, the person who was probably its best explainer and one of its most original synthesizers.

He was born in Stoke-on-Trent in 1967, read natural sciences at Trinity College, Cambridge, and then went to Caltech for a PhD, which he completed in 1991 under John Hopfield — a genealogical detail that explains a good deal about his intellectual posture. He was trained as a physicist, in a lab where neural computation was being treated as statistical mechanics, and he never stopped thinking of learning, coding, and inference as one subject seen from different angles. He returned to Cambridge as a Royal Society research fellow and spent the rest of his academic career in the Cavendish Laboratory, eventually as Professor of Natural Philosophy — a deliberately old-fashioned title he seems to have enjoyed.

MacKay's output was unusually varied for someone with a short career. In the early 1990s he produced the papers that put Bayesian inference on a practical footing for neural networks. In the mid-1990s, working with Radford Neal, he rediscovered and revived Robert Gallager's low-density parity-check codes, which had been invented in 1962 and abandoned as computationally infeasible; MacKay and Neal showed they performed close to the Shannon limit under iterative decoding, and LDPC codes are now in Wi-Fi, satellite television, and 5G. He built Dasher, a text-entry interface that lets someone type by steering through a zooming landscape of letters whose sizes are set by a language model — an application of arithmetic coding to human-computer interaction, designed so that people who can only control a gaze tracker or a single switch can write at usable speed. He wrote *Information Theory, Inference, and Learning Algorithms*, published in 2003 and made freely available, which is one of the very few textbooks that is genuinely a pleasure to read.

Then he changed fields. *Sustainable Energy — Without the Hot Air*, published in 2008 and also free online, applied the physicist's back-of-envelope method to Britain's energy debate: convert every claim to the same units — kilowatt-hours per person per day — and see which proposals could possibly add up. It is a devastating book precisely because it is not polemical; it simply insists on arithmetic, and finds that most of the participants in the debate had not done any. On the strength of it he served as Chief Scientific Advisor to the UK Department of Energy and Climate Change from 2009 to 2014, moving from a Cambridge chair into the machinery of government.

Key contributions

**The evidence framework for Bayesian model comparison (1992).** MacKay's practical Bayesian treatment of neural networks is his most quoted technical work. The idea is a two-level inference. At the first level, given hyperparameters (a weight-decay strength α and a noise level β), you infer the posterior over weights, which under a Gaussian approximation around the mode is a Laplace approximation requiring the Hessian. At the second level you infer the hyperparameters by maximizing the *evidence* — the marginal likelihood P(data | model), obtained by integrating out the weights. The critical consequence is that this integration automatically implements Occam's razor. A more flexible model spreads its prior predictive mass over more possible datasets, so it assigns less probability to the one you actually observed; the "Occam factor" is the ratio of posterior to prior volume in parameter space, and it penalizes complexity without any ad-hoc penalty term being inserted by hand. MacKay used this to set regularization strengths from the data alone, to compute an effective number of well-determined parameters, and — via automatic relevance determination, in which each input gets its own hyperparameter — to let the evidence prune irrelevant inputs.

**Active data selection (1992).** If your model gives calibrated uncertainty, you can ask where to measure next. MacKay's information-based objective functions choose the query that maximizes expected information gain about the model or its predictions. This is the direct conceptual ancestor of both modern active learning and Bayesian optimization: fit a probabilistic surrogate, get honest error bars, spend your next expensive observation where it buys the most.

**Gaussian processes as the honest limit.** MacKay was among the clearest early expositors of the fact that a Bayesian neural network with a suitable prior tends, in the wide limit, to a Gaussian process, and that GPs give exactly the calibrated nonparametric regression that the evidence framework wants. His tutorials taught a generation how covariance function choice encodes assumptions about smoothness and length scale.

**Low-density parity-check codes.** With Neal, the demonstration that sparse random parity-check matrices decoded by iterative belief propagation on the associated factor graph achieve performance near the Shannon limit. The theoretical texture here is what makes it interesting: decoding is exactly loopy belief propagation, its success or failure exhibits a sharp *threshold* in the channel noise level, and the analysis of that threshold is a phase transition in the statistical-physics sense. MacKay was ideally placed to see this, and he wrote about it in exactly those terms.

**The textbook.** *Information Theory, Inference, and Learning Algorithms* is a contribution in its own right because of the thesis it embodies: that data compression, error correction, statistical inference, and machine learning are one subject. Source coding, channel coding, Bayesian model comparison, Monte Carlo methods, message passing, and neural networks appear in the same volume with a shared vocabulary — which is why students who learn from it think differently about the relationship between compression and prediction than students who learn the topics separately.

In battle

MacKay is the closest thing this roster has to a genuine generalist, and the numbers are striking: mean 45.2, median 42, and only *nine* problems at or below 20 — with a floor of 10, higher than most players' median. He is rarely the best answer, and almost never a disaster.

His dominant band traces the unification. "Tune the un-differentiable" at 90 is Bayesian optimization, and the game's explanation calls it nearly a MacKay exam question: probabilistic surrogate, calibrated error bars, next query chosen by expected information gain, model complexity set by the evidence from scarce data. Every conceptual ingredient is his, and he is docked only because he did not publish on acquisition functions in the modern hyperparameter-tuning idiom. "Codes that kiss the limit" (88) is LDPC, his own revival. "The shortest description that predicts" (85) is minimum description length, where his textbook's central identity — that a good model is a good compressor — is the argument. "Which examples deserve labels?" (82) is active learning, his 1992 paper. "Two voices, two microphones" (82) is ICA, which he treated in the book as a latent-variable inference problem rather than a signal-processing trick. "Which of five models?" (78) is model comparison by evidence. "Phase transition at the threshold" (76) is the statistical physics of decoding. "Predict the ore grade underground" (76) is Gaussian process regression, which is kriging under a different name.

His category profile is what a synthesizer's looks like: search 73.0, information 71.2, causality 56.2, testing 53.0, regression 52.4, computability 51.0, small-sample 46.0, classification 45.3. He is above 40 in the majority of categories. Almost no one else on the deep-learning roster is above 40 in more than three.

Even his losses are mild, and they are all of one kind: *building things*. "A computer shared by fifty" at 10 is his floor — operating system scheduling and memory protection, where the packet notes there is simply no bridge from an inference-and-coding framework to interrupt handling. "Ship it to a hundred contributors" (18) is distributed software development process. His systems category average of 14.0 is the only one that collapses. The remainder of his weak set is mathematics he consumed rather than produced: "Cut through the interior" (14) and "The certificate of optimality" (13) are convex optimization, and "The grammar beneath the sentence" (15) is formal language theory. "The coefficient that flips sign" (15) is confounding in regression — the one genuinely statistical loss, and a reminder that being Bayesian about a model does not by itself make you careful about causal structure.

Play MacKay when you want a defensible answer with honest uncertainty on a problem you have never seen before. He is the safest pick in the game, and the price is that he almost never scores a 99.