rl
Which elevator answers the call
It is 1996 at the University of Massachusetts, and the test of reinforcement learning is a wait in the lobby: four elevators, ten floors, morning-rush traffic, and a dispatch question — which car answers which call — whose state space is astronomically large, whose passengers arrive at random, and whose events fall at irregular continuous times that break the neat discrete clock of textbook MDPs. Learn the dispatch policy from simulated traffic, with a neural network carrying the value function because no table could, and beat the best engineered controllers in the elevator industry's own literature. Squared wait time is the cost, so the policy must protect the unlucky, not the average. Get it wrong and RL stays a gridworld science; this building is the field's first industrial referee.
Who this problem belongs to
The two figures whose methods fit it best, out of 30 in contention.
Barto is not analogous to this problem, he is its author: with Robert Crites, his student at UMass Amherst, he built exactly this system in the mid-1990s, a team of RL agents each learning a decentralized dispatch policy for a simulated bank of elevators, with a neural network approximating the value function because the state space, car positions, directions, and passenger wait times combined, was too large for any table. Squared wait time was their cost signal by design, protecting the unlucky rider rather than the average one, and their trained controllers beat the best heuristic dispatch algorithms published in the elevator industry's own literature. Continuous, irregular passenger arrivals were handled by the same TD methods he and Sutton had spent a decade developing. No carrier in this pool owns the actual result as literally as Barto does.
Sutton did not run the elevator experiment himself, Crites and Barto did, but every mechanism the problem needs is his intellectual property as much as anyone's: temporal-difference learning, developed with Barto through the 1980s, is precisely what let the dispatch agents bootstrap value estimates from incomplete, irregularly timed episodes without waiting for a final outcome. His TD(lambda) formalism and his insistence that function approximation, not lookup tables, is how RL must scale to real problems are the load-bearing ideas behind the neural-network value function this problem specifies. He was UMass Amherst faculty alongside Barto during these years and shaped the lab's whole research program. The gap between him and full credit is authorship, not applicability; the toolkit is entirely his.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
30 figures are scored on this problem. Draw it in a battle to see where you land.