Leon Bottou

Life and career

For most of the twentieth century, stochastic gradient descent was regarded by serious optimization researchers as a crude device — noisy, slow to converge in any classical sense, and lacking the guarantees that made Newton and quasi-Newton methods respectable. Léon Bottou is the person most responsible for the fact that it now trains essentially every model in the world.

Bottou was born in France in 1965, trained at the École Polytechnique, and took his doctorate at the Université de Paris-Sud in 1991 on connectionist learning, with applications to speech recognition. It was at Orsay that he met Yann LeCun, and the connection took him to AT&T Bell Labs in New Jersey, into the Adaptive Systems group that was then in the middle of building the most successful applied neural network system of the era. Bottou was a central contributor to the handwriting recognition and check reading work — the 1998 document recognition paper that presents LeNet lists him as a co-author, and the graph transformer network portion of that paper, in which segmentation and recognition are trained jointly through a differentiable graph-structured pipeline, is very much in his line of interest.

He has an unusual professional profile for a research scientist: he has repeatedly moved between research labs, industrial engineering, and his own companies. He founded Neuristique, which produced neural network simulation software and later the Lush language he developed with LeCun; he co-created the DjVu document image compression format at AT&T, which for years was the practical way to put scanned documents on the early web; he worked at NEC Labs America, then at Microsoft, where he was embedded in the advertising system and confronted head-on the question of how to reason about interventions in a deployed feedback loop; and since 2015 he has been at Facebook AI Research.

That Microsoft period matters intellectually. Working on ad placement forces you to notice that a system trained on logged data is being asked a *counterfactual* question — what would have happened had we shown a different ad — and that ordinary supervised learning answers it wrong. Bottou's 2013 work on counterfactual reasoning and learning systems is a serious attempt to bring causal machinery into large-scale ML practice, and it prefigures his later insistence that the field's reliance on correlations is a limitation rather than a nuisance. That argument culminated in invariant risk minimization, developed with Arjovsky, Gulrajani and Lopez-Paz, which tries to learn representations whose optimal predictor is the same across environments, on the grounds that only such features are likely to be causal and to survive distribution shift. He was also a co-author on the Wasserstein GAN work that reframed adversarial training around a better-behaved distance between distributions.

Key contributions

**The large-scale learning trade-off (Bottou–Bousquet, 2007).** This is the argument that changed how the field thinks about optimization, and it is worth stating carefully because its logic is not obvious. When you fit a model you incur three errors: *approximation* error, from your hypothesis class not containing the truth; *estimation* error, from having finite data; and *optimization* error, from not solving the empirical problem exactly. Classical optimization theory studies the third term in isolation and concludes you should drive it to zero. Bottou and Bousquet observed that under a fixed *computation* budget these terms trade against each other. If your optimizer is cheap per step, you can afford to process far more data, which shrinks estimation error. If it is expensive per step, you solve the empirical problem beautifully — and overfit a smaller sample. In the large-data regime the correct choice is therefore a *worse* optimizer that is much faster, which is why SGD, with its O(1) per-step cost independent of dataset size and its sublinear convergence rate, beats methods with far better rates. The noise is not a defect to be tolerated; within the budget accounting, it is free.

**Practical SGD.** Bottou's "SGD tricks" and his large-scale learning papers codified the craft: learning rate schedules of the form η_t = η_0/(1 + λη_0 t) and their theoretical justification, the conditions under which averaging the iterates recovers optimal asymptotic rates, when to use second-order information cheaply, the importance of shuffling and of not cycling deterministically, and how to sanity-check that a run is converging. His later SIAM Review survey with Curtis and Nocedal — written with two of the most respected people in classical nonlinear optimization — is the document that formally reconciled the machine learning and optimization communities on this question.

**Counterfactual learning in deployed systems.** Logged data from a system that made its own choices is confounded by those choices. Bottou's framework applies importance weighting against the logging policy, together with careful variance control, to estimate what a different policy would have earned — off-policy evaluation, developed for a real advertising system with real money on it, and connected explicitly to causal graphs rather than treated as a bandit trick.

**Invariant risk minimization.** Rather than minimizing average risk over pooled data, seek a representation such that a single classifier is simultaneously optimal in every training environment. The motivation is that spurious correlations vary across environments while causal mechanisms do not, so invariance is a computable proxy for causality. Whether IRM delivers on this in practice is actively debated; the framing has been influential regardless.

**Wasserstein GAN.** With Arjovsky and Chintala, replacing the Jensen–Shannon objective — which is uninformative when the model and data distributions have disjoint support, as they generally do on a low-dimensional manifold — with an approximation to the earth mover's distance, producing gradients that remain useful and training that is markedly more stable.

In battle

Bottou is a broader player than most of his Bell Labs cohort: mean 31.1, median 26, only forty-two problems at or below 20. He also has the distinction of holding a near-maximum score on the single problem that most defines modern practice.

That problem is "The million-parameter valley" at 97, and the game calls it his life's work stated as a puzzle: the Bottou–Bousquet accounting for why cheap noisy gradients beat exact ones when data is abundant, the learning rate schedules, the averaging results, and the fact that he was training LeNet with SGD in the 1990s when the optimization community considered the method disreputable. Directly beside it is "Learn from the stream, one at a time" at 93 — online and single-pass learning, where the per-example update *is* the algorithm. "The model is five percent of the system" at 90 rewards a career spent inside deployed systems where the learning algorithm is a small component surrounded by logging, serving, feedback loops and drift. "The interpolator that should have failed" at 80 is the modern benign-overfitting and double-descent puzzle, where his generalization-versus-optimization accounting is the natural language for the question. And "Prove the descent" (72) reflects that he actually engages with convergence analysis rather than only reporting curves.

His secondary strength is worth naming because it is unusual for an optimization person: causality at 35.5, from the counterfactual and IRM work. Very few figures in the deep learning lineage score anything at all on causal problems, and Bottou is one of the reasons that is changing. His scores on distributed problems — "The cluster that iterates" (74), "Train across a thousand GPUs" (72), "Train on the phones, keep the secrets" (62) — come from the same source: he knows what it takes to make SGD run when the data does not fit on one machine.

The losses are almost entirely *engineering and formal* rather than statistical, which distinguishes him from the pure ML figures. "The grammar beneath the sentence" at 4 is his floor, and the packet flags it as an explicit floor case: proving separations in the Chomsky hierarchy is a formal-language result with no connection to numerical optimization. "Let the machine find the proof" (7) is automated deduction; "The scratch on the disc" (6) is coding theory; "What happened first?" (6), "The machine that overdosed" (5), and "The shopping cart that must not vanish" (5) are distributed clocks, safety-critical software, and availability under partition — real systems engineering, which he is adjacent to but has not contributed to. His computability average of 10.0 and search average of 16.8 mark the same boundary: discrete, combinatorial, and proof-theoretic work is outside his continuous, stochastic, budget-accounting worldview.

Play Bottou whenever the question is *how do I actually fit this, at this scale, within this budget* — or when logged data is being used to answer a question about a decision. Bench him on anything requiring a discrete proof or a fault-tolerant machine.