optimization
The million-parameter valley
It is the deep-learning era, and you must minimize a wildly non-convex loss surface over a million parameters — a landscape of valleys, ridges, and saddle points where the global optimum is unknowable and the true gradient over all data is too costly to compute each step. Second-order information (the Hessian) is unthinkable at this size. Make it converge anyway, in practice, using cheap noisy gradient estimates from small batches — trusting that the noise itself helps escape bad points. Get it wrong and training either crawls, stalls on a plateau, or diverges outright, and the model never learns — the entire modern practice of training huge networks rests on the surprising fact that stochastic first-order methods, crude as they seem, reliably find good solutions in these million-dimensional valleys.
Who this problem belongs to
The two figures whose methods fit it best, out of 68 in contention.
This problem is Bottou's life work stated as a puzzle. From his late-1980s neural network training at AT&T through the 2008 Bottou-Bousquet analysis of the tradeoffs of large-scale learning and his 2010 'Large-Scale Machine Learning with Stochastic Gradient Descent,' he built the modern case that cheap noisy gradients beat exact ones when data is abundant: optimization error is only one term in a budget that includes estimation error, so a crude fast step wins over a careful slow one. He co-trained the LeNet convolutional networks with SGD in the 1990s, when most of optimization considered the method disreputable. He knows the learning-rate schedules, the averaging tricks, and the theory of why the noise is a feature. No carrier in this batch is better matched.
Hinton spent four decades making exactly this kind of surface converge in practice. The 1986 backpropagation paper with Rumelhart and Williams supplied the cheap gradient; Boltzmann machine training made him fluent in stochastic, noise-driven learning; and his practical toolbox — momentum lore from the 1980s connectionist era, RMSProp introduced informally in his Coursera lectures, dropout, and unsupervised pre-training as an initialization trick during the 2006 deep-learning revival — is precisely the folk engineering that makes million-parameter descent work. He is weaker than Bottou or Bertsekas on convergence theory; his knowledge is empirical, accumulated by watching training curves stall and diverge since the era when a million parameters was absurd. But the problem asks for practice, and his practice defined the field's.
Fought here
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
68 figures are scored on this problem. Draw it in a battle to see where you land.