rl
The robot learns to grasp
It is 2015 at Berkeley, and the simulation-to-reality gap is eating the field alive: policies that grasp flawlessly in physics engines fumble on real hardware, and real hardware is unforgiving — a robot arm collecting training data at physical speed gathers in a week what a simulator produces in a minute, wearing out its joints as it goes. A robot arm must learn manipulation from camera pixels and sparse successes, with real-hardware sample budgets: thousands of trials, not millions. Sample efficiency is not a nicety here; it is the entire constraint. The stakes are the difference between robot learning as a demo genre and robots that work in warehouses, hospitals, and homes — and whether embodied learning is possible at all outside the simulator.
Who this problem belongs to
The two figures whose methods fit it best, out of 30 in contention.
This problem is set in Abbeel's own building. In 2015 his Berkeley group, with Sergey Levine, published end-to-end training of deep visuomotor policies: a PR2 robot learning manipulation tasks directly from camera pixels, using guided policy search to keep real-hardware trial counts in the hundreds rather than millions — trajectory optimization supplying supervision so the network need not explore blindly. His earlier apprenticeship-learning work with Ng (autonomous helicopter aerobatics, mid-2000s) attacked the same constraint from the demonstration side: when real trials are expensive and dangerous, extract the policy from an expert. Sample-efficient deep RL for physical manipulation is not adjacent to his research program; it is the research program, at exactly this date and place. No other carrier's toolkit maps onto the problem statement this literally.
Silver co-authored DQN (2013-2015), which established that a deep network could learn control from raw pixels and sparse rewards — half of this problem solved in principle. His deterministic policy gradient work (2014) became DDPG, the workhorse algorithm for continuous-control robotics of exactly this era. The gap is the other half of the problem: Silver's landmark systems bought their performance with simulation. DQN consumed tens of millions of Atari frames; AlphaGo's self-play generated data at a rate no physical arm survives. His methods assume experience is cheap, and the problem's entire premise is that it is not. He has the deep-RL machinery the task demands, first-hand and at the highest level, but the sample-efficiency axis requires retooling his approach rather than applying it.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
30 figures are scored on this problem. Draw it in a battle to see where you land.