rl
The long corridor of sparse reward
It is 1984 at the University of Massachusetts, and the reinforcement-learning problem has a wound at its center: an agent gets reward only at a distant goal, and every step before it looks equally worthless at the moment it is taken. Learn value estimates that propagate credit backward through time — without a model of the environment, updating each state's estimate from the very next estimate rather than waiting for the far-off outcome. The method must be incremental, cheap, and provably convergent, because the ambition is a scale-free principle of learning, not a trick for one corridor. Get temporal credit assignment wrong and reinforcement learning stays where the critics say it belongs: unable to learn anything whose payoff is not immediate.
Who this problem belongs to
The two figures whose methods fit it best, out of 33 in contention.
This problem is Sutton's 1984 UMass dissertation and his 1988 Machine Learning paper almost verbatim. Temporal-difference learning is his answer to exactly this wound: update each state's value from the next state's estimate, bootstrapping so credit flows backward without waiting for the distant goal. He framed TD as a scale-free principle rather than a corridor trick — the eligibility-trace family TD(lambda) interpolates between one-step bootstrapping and Monte Carlo, and his 1988 paper proved convergence of TD(0) in the mean for the tabular case. The method is incremental and cheap by construction, one update per transition, which was the point on 1984 hardware. If any historical toolkit maps one-to-one onto a problem statement, it is Sutton's onto this one; the era, the institution, and the ambition are all his.
Barto ran the UMass Amherst lab where this problem lived in 1984, and the intellectual scaffolding is jointly his. The 1983 Barto, Sutton, and Anderson actor-critic paper on pole balancing already contained the adaptive critic: an element learning value estimates from successive predictions, propagating credit through time without an environment model. Barto's distinctive contribution was situating TD within the older literatures — Klopf's hedonistic neuron hypothesis, Widrow's adaptive elements, and later the connection to dynamic programming that made convergence analysis possible. He supplied the theoretical patience the field needed: insisting the method be understood as approximate DP, not folklore. He shades just below Sutton here because the specific incremental TD formulation and its first convergence proof were the student's thesis; but the problem was posed, funded, and defended in Barto's lab.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
33 figures are scored on this problem. Draw it in a battle to see where you land.