Andrew Barto

Life and career

Andrew Barto is the half of reinforcement learning's founding partnership that most people cannot name, which is a fact he seems to have made peace with long ago and which is also, in a small way, a lesson about how credit works in science. He studied at the University of Michigan, taking a degree in mathematics and then a doctorate in computer science there in the mid-1970s. He arrived at the University of Massachusetts at Amherst in the late 1970s as a postdoctoral researcher with Michael Arbib, whose group sat at the intersection of brain theory, control, and computation, and he stayed at UMass for his entire career, building and leading the laboratory where reinforcement learning as we know it was assembled.

The partnership with Richard Sutton, who arrived as his graduate student around 1980, produced the field. What Barto brought to it — and what distinguishes his intellectual signature from Sutton's — was a dual grounding: on one side, the psychology and neuroscience of animal learning, and on the other, the mathematics of optimal control and dynamic programming. In the early 1980s these were separate literatures that did not talk. Barto's persistent claim was that they were describing the same thing. An animal learning to press a lever for delayed food and an engineer solving a Bellman equation are both doing sequential decision-making under uncertainty; write the animal-learning theory in the control theorist's notation and you get a class of algorithms nobody had.

That framing paid off spectacularly in the 1990s, when Wolfram Schultz's recordings of dopamine neurons in the primate midbrain turned out to look strikingly like a temporal-difference error signal — firing at unexpected reward, shifting to the predictive cue once the association is learned, dipping below baseline when an expected reward fails to arrive. Barto had written explicitly about adaptive critics and the basal ganglia, and the reward-prediction-error hypothesis of dopamine, developed with Dayan, Montague and others, became one of computational neuroscience's most productive theories. It runs in both directions: RL gave neuroscience a formal model, and the neuroscience gave RL a claim to be describing something real.

Barto co-authored the standard textbook with Sutton, supervised a remarkable line of students who populate the field's senior ranks, and is now professor emeritus at UMass. In 2024 he and Sutton received the Turing Award. He has been publicly measured about the current moment in AI, and consistent about what his framework does and does not claim.

Key contributions

**The actor–critic architecture.** With Sutton and Charles Anderson, Barto published in 1983 the "neuronlike adaptive elements" paper that gave the field its first working nontrivial control result and its most durable architecture. Two learning elements: the *critic*, which learns a state-value function and converts sparse environmental reward into a dense per-step evaluation signal (the TD error), and the *actor*, which adjusts action probabilities using that signal. The critic solves the temporal credit-assignment problem so the actor does not have to. The demonstration was pole balancing — the cart-pole task, which became the field's fruit fly — learned from a failure signal delivered only when the pole fell. Every modern policy-gradient method with a learned baseline or value head is an actor–critic system.

**RL as approximate dynamic programming.** Barto's most consequential conceptual contribution may be the systematic identification of reinforcement learning with asynchronous, sampled, approximate dynamic programming. With Steven Bradtke and Satinder Singh he developed real-time dynamic programming: rather than sweeping the full state space as classical DP requires, back up values only along states the agent actually visits, in the order it visits them, and inherit convergence guarantees from the asynchronous DP theory of Bertsekas and Tsitsiklis under stated conditions. This did two things at once. It gave RL algorithms respectability and analytical traction by connecting them to a mature body of control theory, and it explained *why* they work: they are Bellman backups performed on a sampled subset of the state space, concentrating computation where the agent's own behavior takes it. That is the precise answer to the curse of dimensionality — you do not solve the whole MDP, you solve the part of it you inhabit.

**Elevator dispatching and function approximation at scale.** With his student Robert Crites, Barto built one of the earliest convincing demonstrations that RL could beat industrial-strength engineered heuristics on a real problem. A bank of elevators serving a busy building has a state space — car positions, directions, button states, waiting times — far too large to tabulate, so the value function was approximated by a neural network; the arrival process was continuous and irregular; and the agents learned decentralized dispatch policies with squared waiting time as the cost, a deliberate choice that penalizes leaving any single passenger stranded rather than optimizing the average. The trained controllers outperformed the best dispatch algorithms in the elevator industry's own literature. As a case study it remains close to ideal: a genuinely hard scheduling problem, no tractable model, and a clear baseline to beat.

**Intrinsic motivation and hierarchical learning.** In the 2000s, with Singh and Nuttapong Chentanez, Barto formalized intrinsically motivated reinforcement learning — agents that generate their own reward signals from novelty, surprise, or learning progress, and use them to build a repertoire of reusable skills in the absence of external task reward. This was drawn directly from developmental psychology, and it anticipated a large modern literature on exploration bonuses and curiosity-driven learning. He also contributed substantially to hierarchical RL, co-authoring an influential survey of temporal abstraction and options.

In battle

Barto's profile is a specialist's with a very high peak and a long fall: mean 30.7, median 18, eight problems above 80, but fifty-four at or below 20. RL at 67.4 across eighteen problems and games at 40.4 across sixteen are the only categories carrying real weight.

His top score, "Which elevator answers the call" (99), is a problem he did not merely have the tools for but literally solved: the Crites–Barto system, neural value approximation included, the squared-wait-time cost included, beating the industry's published heuristics. It is the purest ownership claim on his sheet, and it sits above Sutton's score on the same problem for exactly that reason — Sutton supplied the method, Barto built the thing.

"Balance the broomstick" (92) is the 1983 cart-pole paper, the demonstration that made actor–critic credible. "The long corridor of sparse reward" (93) is TD's founding problem, where he sits just under Sutton, whose dissertation it was. "Dynamic programming's curse" (84) is where his real-time DP work is the direct answer: sample the states you visit, back up asynchronously, inherit the convergence theory. "The dice make it learnable" (90), "Teach the machine checkers" (84), "The grandmaster's intuition" (84) and "Learn the optimum while acting otherwise" (82) fill in the rest of the RL and game-learning bench, generally a few points behind whoever owns the specific algorithm — Watkins on off-policy, Samuel on checkers, Silver on Go — but always in the fight.

One number is worth flagging: experimental design at 38.0 is unusually high for an RL carrier. Barto's grounding in animal learning means he thinks naturally about how an agent's own behavior determines the data it sees, which is closer to sequential experimentation than most of his peers get.

The floor is uncompromising and entirely statistical. "How high must the dike be?" (4) is extreme-value theory, "Why tall fathers have shorter sons" (4) is regression to the mean, "The therapy the trial reversed" (4) is confounding by indication in observational epidemiology, "The paradox in the admissions data" (5) is Simpson's paradox, and "Five hundred stocks, one year of days" (5) is high-dimensional covariance estimation. His regression average is 5.0, information 6.0, classification 7.0, high-dimensional 8.5, causality 12.9 across eleven problems. There is no part of Barto's toolkit that estimates a population quantity from a fixed sample or defends a causal claim against confounding, and the matrix scores that at close to zero.

"The machine that overdosed" (5) deserves a note because it looks like it should be his: it is the Therac-25 case, a safety-critical software engineering failure, and the fact that a control theorist scores 5 on it is the game correctly distinguishing *control* from *software verification*.

The strategic reading: Barto is a slightly deeper, slightly narrower Sutton, with an edge on problems that involve building a working controller for a messy real system and a corresponding disadvantage on problems that are about the abstract learning principle. And the packet's phrasing of his battle identity is not a joke — on most of his best problems, he is competing directly with his own former student, and frequently placing second by a few points.