rl
Ascend the policy itself
It is 2016 at Berkeley, and simulated robots are learning to walk — badly. Value-based methods stumble in continuous action spaces, so the alternative is direct: parameterize the policy, estimate the gradient of expected reward from sampled trajectories, and climb. The catch is variance: naive gradient estimates are so noisy that one lucky rollout can convince the learner a flailing gait is progress, and one large step can collapse a working policy into a heap that never recovers. Engineer the estimator — baselines to cut variance, trust regions to bound each update — and show stable learning of locomotion from reward alone. Get it wrong and policy search stays a lottery over random seeds, and the embarrassing reproducibility crisis of deep RL arrives on schedule.
Who this problem belongs to
The two figures whose methods fit it best, out of 34 in contention.
Abbeel's Berkeley lab is the exact institutional and intellectual home of this problem: his group produced trust-region policy optimization and generalized advantage estimation, the specific engineering fixes, baselines to cut variance, trust regions to bound each update, this scenario names directly, while working precisely on simulated robot locomotion in the mid-2010s, the exact application this problem describes. His own earlier apprenticeship-learning and inverse-reinforcement-learning research for real robots gave him deep firsthand experience with exactly the variance and instability problems naive policy-gradient estimates produce. This is not adjacent expertise; his lab is where this problem's engineering solutions were actually built, tested, and published, making him about as close a match as this roster contains for a 2016 Berkeley policy-gradient robotics problem.
Sutton's foundational contributions to policy-gradient methods, including the 1999 policy-gradient theorem with Barto, McAllester, and Singh, established the mathematical bedrock this problem's entire approach rests on: that the gradient of expected reward with respect to policy parameters can be estimated from sampled trajectories without differentiating through the environment. His broader reinforcement-learning framework, including actor-critic architectures that use a learned value function as a variance-reducing baseline, directly anticipates the "baselines to cut variance" fix this problem requires. He did not personally develop the specific trust-region engineering that made 2016-era deep policy search stable at Berkeley, so his score reflects deep foundational authorship of the mathematical theorem this problem's entire method is built on, just below the specific 2016 engineering.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
34 figures are scored on this problem. Draw it in a battle to see where you land.