rl
Balance the broomstick
It is 1962, and the control problem of the age is keeping unstable things upright: rockets on launch pads, inertial platforms, and — the laboratory miniature of them all — an inverted pendulum on a moving cart. Keep the broomstick balanced using torque control, and do it both ways: from the equations when you have them, via the elegant machinery of state-space control and optimal estimation, and from raw experience when you don't — when the dynamics are unknown and the controller must learn from its own failures. The comparison is the point: what does knowing the model buy you, and what can trial and error genuinely recover? The engineers' version of the stakes is not academic. The thing being balanced, at full scale, is a Saturn V.
Who this problem belongs to
The two figures whose methods fit it best, out of 35 in contention.
This problem is set in Kalman's exact moment. Between 1960 and 1962 he published the papers that created modern state-space control: the discrete-time filter for optimal estimation under Gaussian noise, the continuous-time version with Bucy, and the algebraic treatment of controllability and observability that tells you precisely when an unstable plant like an inverted pendulum can be stabilized from limited measurements. The cart-pole is linearizable about the upright equilibrium, which is exactly where his machinery is strongest: check controllability of the linearized system, estimate the state from noisy cart-position measurements, feed back through an optimal gain. NASA adopted this apparatus for Apollo navigation within the decade. His toolkit says nothing about the model-free half of the problem — learning from failures was not his question — but on the model-based half he is the standard everyone else is measured against.
The cart-pole is Barto's signature testbed. The 1983 Barto, Sutton, and Anderson paper on neuronlike adaptive elements demonstrated the actor-critic architecture — an associative search element learning a control policy and an adaptive critic element learning to evaluate states — precisely on pole balancing with unknown dynamics, failure signals only, and a coarse state discretization inherited from Michie and Chambers's BOXES. That is the model-free half of this problem, executed and published. Barto also came from control theory adjacent territory: he and his students consistently framed reinforcement learning against dynamic programming and optimal control, so the comparison the problem demands — what the model buys you versus what trial and error recovers — is a question he spent a career articulating. He arrives two decades after 1962, but the problem's learned-control half is essentially his invention.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
35 figures are scored on this problem. Draw it in a battle to see where you land.