Richard Sutton

Life and career

Richard Sutton came to computer science through psychology. He took his undergraduate degree at Stanford in psychology in the late 1970s, at a moment when the animal-learning literature — classical conditioning, the Rescorla–Wagner model, secondary reinforcers — was full of precise quantitative theories about how organisms learn from delayed consequences, and almost none of that had been connected to machine learning. He went to the University of Massachusetts at Amherst for graduate work with Andrew Barto, and the two of them spent the next decade building the connection. Sutton's 1984 dissertation on temporal credit assignment is the founding document of modern reinforcement learning; his 1988 paper introducing TD(λ) and proving convergence for the tabular case is its central technical result.

He spent roughly a decade at GTE Laboratories doing industrial research, returned to UMass, and then worked at AT&T's Shannon Laboratory before moving in 2003 to the University of Alberta, where he built one of the world's strongest reinforcement-learning groups and helped make Edmonton an unlikely capital of the field. He has been a central figure in the Alberta Machine Intelligence Institute and spent several years affiliated with DeepMind's Alberta office. In 2024 he and Barto shared the Turing Award for their foundational work.

Sutton's textbook with Barto, *Reinforcement Learning: An Introduction*, is one of those rare books that both defined a field's vocabulary and remained the best introduction to it across two editions and twenty years. Its structure — bandits, then Markov decision processes, then dynamic programming, Monte Carlo, and temporal-difference methods laid side by side as points on a spectrum — is how essentially everyone now thinks about the subject.

He is also the field's most consistent and most uncomfortable philosopher. "The Bitter Lesson," a short essay he published in 2019, argues from seventy years of AI history that the approaches which ultimately win are general methods that leverage computation — search and learning — and that building human knowledge into systems produces short-term gains and long-term dead ends. Chess, Go, speech recognition and computer vision are marshalled as the same story told four times. The piece is short, unhedged, and enormously contentious, and it is genuinely his considered position rather than a provocation: his entire research program has been the pursuit of learning principles that are scale-free, incremental, and free of task-specific engineering. He has argued in the same spirit that reward maximization may be sufficient to produce the full range of intelligent abilities — a strong claim, stated as a hypothesis, that continues to draw serious argument.

Key contributions

**Temporal-difference learning.** The core idea and the one everything else builds on. Suppose you want to estimate the value of a state — the expected return from here on. Monte Carlo says: wait for the episode to end, then update toward the observed return. Dynamic programming says: update toward the reward plus the discounted value of the next state, using the current estimate of that next value, which requires a model of the transition dynamics. TD takes the second update but computes it from an *observed* transition instead of a model: V(s) ← V(s) + α[r + γV(s′) − V(s)]. That bracketed quantity is the TD error, and it is the single most important quantity in modern RL. The method is fully incremental, costs one small update per transition, requires no model, and does not wait for the outcome — properties that mattered enormously on 1984 hardware and matter again now for different reasons.

TD(λ) with eligibility traces generalizes it: maintain a decaying memory of recently visited states and distribute each TD error backward over them. λ=0 gives one-step bootstrapping; λ=1 gives Monte Carlo; intermediate values interpolate, trading bias against variance. Sutton's 1988 paper proved convergence in the mean for TD(0) in the tabular case and made the argument that bootstrapping is not a computational shortcut but a genuinely different and often better estimator.

**Actor–critic architectures.** With Barto and Charles Anderson, Sutton built systems in which one component (the critic) learns a value function by TD and another (the actor) adjusts a policy using the critic's TD error as a reinforcement signal. This decomposition — a learned evaluator training a learned controller — is the ancestor of A3C, PPO, SAC, and essentially every modern deep-RL algorithm in practical use.

**The policy gradient theorem.** With David McAllester, Satinder Singh and Yishay Mansour, Sutton established that the gradient of expected return with respect to policy parameters can be written without differentiating the state distribution — as an expectation of the score function times the action-value. This is what makes policy-gradient methods with function approximation legitimate rather than heuristic, and it is the theoretical basis for optimizing parameterized stochastic policies directly.

**Dyna and the integration of planning with learning.** Dyna makes a simple and powerful observation: if you learn a model of the environment from experience, you can generate simulated transitions from it and run the *same* TD updates on them. Planning, acting and learning become one algorithm distinguished only by where the transitions come from. This dissolves the model-free/model-based dichotomy into a question of how you spend compute.

**Options and temporal abstraction.** With Doina Precup and Singh, Sutton formalized options — closed-loop policies with initiation sets and termination conditions — within the semi-Markov decision process framework, giving a mathematically clean account of temporally extended actions that composes with all the standard value-based machinery. This is the foundation of hierarchical RL.

**Off-policy learning with function approximation.** Sutton and collaborators identified the "deadly triad" — function approximation, bootstrapping, and off-policy training together can diverge — and developed gradient-TD methods with convergence guarantees under all three. He also proposed the Horde architecture and general value functions, in which an agent learns many predictions about many signals in parallel as a form of scalable, self-supervised world knowledge.

In battle

Sutton's sheet is exactly what a field-founder's should look like: mean 35.3, median 24, fifteen problems above 80, and forty-five at or below 20. The distribution is violently bimodal because his contribution is one deep idea applied everywhere it fits and nowhere it doesn't. RL at 72.9 across eighteen problems is his stronghold.

"The long corridor of sparse reward" (98) is his dissertation as an exam question — credit must propagate backward across many steps to a distant goal, with cheap incremental updates and no model. TD is not an analogy here; it is the literal answer, at the literal institution and year. "Learn the optimum while acting otherwise" (88) is off-policy learning, where he shares the ground with Watkins. "The dice make it learnable" (97) rewards his instinct that stochasticity is not an obstacle but the thing that makes sample-based estimation work at all. "Dynamic programming's curse" (87) is the problem TD exists to route around: DP needs a model and a sweep over the whole state space, and sampling replaces both. "Teach the machine checkers" (93) and "The grandmaster's intuition" (88) put him behind Samuel and Silver respectively but very high, because both systems are TD learning with function approximation and self-play — Samuel anticipated it, Silver industrialized it, Sutton formalized it. "Which elevator answers the call" (95) is Crites and Barto's system, where Sutton is the method's author rather than the system's builder. "The two slot machines" (88) is the bandit problem and the exploration–exploitation tradeoff his textbook opens with.

The losses are total and uniformly statistical. "The corner that sparsity loves" (4) is the lasso's geometry — nonsmooth convex optimization for sparse estimation, with no contact point to value-function learning whatsoever. "Three species of iris" (5) is discriminant analysis, "The paradox in the admissions data" (5) is Simpson's paradox, "What happened first?" (5) is temporal causal ordering, "Five hundred stocks, one year of days" (6) is high-dimensional covariance estimation, and "A computer shared by fifty" (6) is time-sharing operating systems. His classification average is 6.0, systems 5.5, NLP 11.0, high-dimensional 14.0. Sutton's mathematics is stochastic approximation over value functions; supervised learning from a fixed dataset, statistical inference about a population, and systems engineering are all outside it.

The instructive weakness is the one his own philosophy predicts. Sutton's methods are designed to be general and scale-free, which means they are deliberately prior-free — and prior-free methods are sample-hungry. On any problem where the data budget is small and domain knowledge would have carried the day, he loses to carriers whose whole approach is encoding structure in advance. The Bitter Lesson is a claim about the long run and about compute trends; in a single matchup with a tight data budget, the bitter lesson is the one Sutton is on the wrong side of.