rl
Dynamic programming's curse
It is 1957 at RAND Corporation in Santa Monica, and the Air Force is paying for optimal policies over sequential decisions — logistics, targeting, inventory — computed on machines with a few thousand words of storage. The recursive principle is beautiful: solve the tail, back up the values. But exact sequential optimization dies exponentially with state dimension; the man who coined "dynamic programming" also coined "curse of dimensionality" as its epitaph, in the same breath. Show precisely where the exact method dies — count the states honestly — and what approximation rescues it: value functions represented compactly rather than tabulated, and the price paid in optimality. The stakes are whether sequential optimization is a theory of toy problems or a tool for the real ones the sponsors actually have.
Who this problem belongs to
The two figures whose methods fit it best, out of 30 in contention.
This is Bellman's own problem, stated in his own building. At RAND in the 1950s he formulated the principle of optimality, wrote the functional equation that bears his name, and coined both 'dynamic programming' and 'curse of dimensionality' — the latter appearing in his 1957 Dynamic Programming and again in Adaptive Control Processes (1961) as the honest epitaph the prompt demands. He did not stop at diagnosis: with Dreyfus he experimented with polynomial and quadrature approximations of the cost-to-go precisely because the JOHNNIAC-era machines he used had a few thousand words of storage, forcing him to count states before anyone else had to. Asking him to show where exact DP dies and what approximation rescues it is asking him to recite his collected works.
Bertsekas is the person who turned Bellman's epitaph into a rigorous engineering discipline. His Dynamic Programming and Optimal Control volumes give the exact state-counting arguments the prompt asks for, and Neuro-Dynamic Programming (with Tsitsiklis, 1996) is the definitive account of the rescue: value functions represented by parametric approximators, projected Bellman equations, and — crucially — error bounds quantifying the price paid in optimality, the part most practitioners wave away. He analyzed approximate value iteration, approximate policy iteration, and rollout, showing when each converges and how badly it can miss. The only gap is chronological: his synthesis arrives four decades after 1957, on machines RAND could not imagine. But the question posed to Bellman's sponsors is answered, completely, in Bertsekas's books.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
30 figures are scored on this problem. Draw it in a battle to see where you land.