systems
The model is five percent of the system
It is 2015 in a Mountain View war room, and the postmortem is embarrassing: the machine-learned model at the heart of the product is a few hundred lines, but it is wrapped in tens of thousands of lines of glue — feature pipelines nobody owns, data dependencies nobody declared, a training set that quietly diverged from what the servers see in production. Accuracy fell for a month before anyone noticed, because no test covers "the world changed." Engineer ML as a system: detect training-serving skew, version data like code, monitor the input distribution, and make every consumed signal a declared, tested dependency. Get it wrong and the organization ships a model that was excellent at training time into a production reality that has already moved on.
Who this problem belongs to
The two figures whose methods fit it best, out of 48 in contention.
This is Dean's own war room, almost literally. He was the senior technical figure behind Google's production machine-learning infrastructure through the exact period this problem describes, and the 2015 paper this scenario dramatizes, 'Hidden Technical Debt in Machine Learning Systems' by Sculley and colleagues, came directly out of the engineering culture Dean helped build at Google — the recognition that the model is a small core wrapped in glue code, undeclared data dependencies, and untested pipelines. His work on TensorFlow was explicitly designed to make training-serving parity, versioned data, and monitored pipelines first-class engineering concerns rather than afterthoughts. No one on this roster lived closer to the actual postmortem this problem describes. Score: near-maximal, docked only because the paper's authorship credit belongs to his colleagues, not to Dean personally.
Bottou's 'Counterfactual Reasoning and Learning Systems,' written from his years building production advertising and search-ranking systems, is close to a direct blueprint for this exact problem: he showed that deployed machine-learning systems create feedback loops where the model's own predictions change the data distribution it will next be trained on, so a model excellent at training time can silently diverge from the production reality it helped create. His large-scale learning work also insisted that the engineering substrate — how data flows, how it is logged, how the system as a whole behaves under continuous operation — matters as much as any single algorithm's accuracy. Few figures on this roster wrote more directly and rigorously about a production ML system quietly moving out from under its own model than Bottou did.
Fought here
48 figures are scored on this problem. Draw it in a battle to see where you land.