rl
Learn the optimum while acting otherwise
It is 1989 at Cambridge, and reinforcement learning has an awkward coupling at its core: the existing methods learn about the policy they are executing, so an agent that must explore — act badly on purpose, sometimes — is contaminating its own estimate of the best behavior. Cut the knot: design an update that learns the value of the optimal policy directly from experience generated by any sufficiently exploratory behavior, one cheap increment per step, no model of the environment required. Then earn the word 'learns': prove convergence to the optimal values under conditions you must state exactly. Get it wrong and off-policy learning remains a plausible heuristic that quietly diverges in exactly the deployments — cautious robots, safe trials — where exploration and exploitation must be different policies.
Who this problem belongs to
The two figures whose methods fit it best, out of 31 in contention.
Watkins's 1989 Cambridge PhD thesis introduced Q-learning, the exact off-policy, model-free algorithm this problem describes: learning the value of the optimal policy directly from experience generated by any sufficiently exploratory behavior policy, one cheap increment per step, with no environment model required. His original convergence analysis, later sharpened by Watkins and Dayan in 1992, proved convergence to the optimal action-value function under conditions on exploration and learning-rate schedules, precisely the "prove convergence under conditions you must state exactly" demand this problem makes. This is not adjacent expertise or a compatible framework; this is the literal person and the literal algorithm the problem describes, decoupling the policy being learned about from the policy generating behavior, making him the closest possible match to this problem's entire premise.
Sutton's temporal-difference learning, developed through the 1980s and formalized in his foundational 1988 paper, is the broader family of methods Watkins's Q-learning belongs to, and Sutton's textbook with Barto is the canonical treatment explaining exactly why off-policy learning matters, an agent that must explore, act suboptimally on purpose, needs to learn about the greedy policy while behaving differently, the precise coupling problem this scenario describes. His "bitter lesson," that general, scalable learning from experience beats hand-engineered heuristics, is the philosophy underneath Q-learning's model-free design. He did not personally prove Q-learning's convergence himself, that credit belongs to his student-era contemporary Watkins, so his score reflects deep foundational and pedagogical authorship of the surrounding framework just short of the specific proof.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
31 figures are scored on this problem. Draw it in a battle to see where you land.