It is the deep-learning era, and you must minimize a wildly non-convex loss surface over a million parameters — a landscape of valleys, ridges, and saddle points where the global optimum is unknowable and the true gradient over all data is too costly to compute each step. Second-order information (the Hessian) is unthinkable at this size. Make it converge anyway, in practice, using cheap noisy gradient estimates from small batches — trusting that the noise itself helps escape bad points. Get it wrong and training either crawls, stalls on a plateau, or diverges outright, and the model never learns — the entire modern practice of training huge networks rests on the surprising fact that stochastic first-order methods, crude as they seem, reliably find good solutions in these million-dimensional valleys.
Chose Curiosity as compression progress — wrong. Brute-force deep nets at scale was the one that fit.
Schmidhuber's lab spent the 1990s and 2000s training deep recurrent networks by stochastic gradient descent when compute made every trick matter. He co-authored LSTM (1997), the architectural answer to Hochreiter's vanishing-gradient analysis, and the 1997 flat minima paper arguing that broad basins in weight space yield better generalization — a direct anticipation of why SGD's noisy dynamics find good solutions. His group's later GPU-trained networks (Ciresan et al., circa 2010-2012) demonstrated that plain online backpropagation at scale wins vision benchmarks, an empirical vindication of the problem's premise. He is not the theorist Bertsekas or Amari is, and his contributions are architectural more than algorithmic, but he was making nonconvex million-weight training converge in practice earlier than most of the deep-modern bench.
Santerre teaches this material at Berkeley, which is exactly the problem: he can lecture fluently on why SGD escapes saddle points, sketch Robbins-Monro on a whiteboard, and assign Bottou's papers as reading — and every one of those citations is a person in this bracket who actually did the thing. Bottou spent a career making stochastic descent work; Santerre spent a career making slides about Bottou making stochastic descent work. His practical contribution to the million-parameter valley is advising companies to hire someone like Hinton's students. In a batch containing the inventor of backprop, the analyst of its convergence, and the engineer of its infrastructure, the professor's toolkit — a well-organized syllabus and genuine enthusiasm — converges to zero faster than any learning rate schedule he could name.
Battle #35 · 8/9/2026, 6:24:36 PM · this result is deterministic: the same two personas on this problem always resolve the same way.