rl
When to replace the engine
It is 1960, and the airlines have discovered that their most expensive decisions are not single decisions at all. Maintain a fleet of jet engines: inspect, repair, or replace, under uncertainty about hidden wear, with inspection costs, overhaul costs, and the catastrophic cost of an in-flight failure. The trap is optimizing each decision in isolation — replacing too early burns capital, too late burns worse. Formalize the problem as states, actions, transition probabilities, and costs, and optimize the policy, not the single decision: a complete rule mapping every observed condition to an action, evaluated over the long run. This is where sequential decision theory meets industrial reality; a policy wrong in the tail is an engine failing over the Atlantic.
Who this problem belongs to
The two figures whose methods fit it best, out of 31 in contention.
This problem is nearly a page out of Bellman's own book. Equipment replacement was one of the worked examples he used at RAND in the 1950s to demonstrate dynamic programming: define the machine's condition as state, inspect/repair/replace as actions, write the functional equation, and back up expected costs to obtain a complete policy rather than a myopic decision. His 1957 'Dynamic Programming' treats exactly this class — sequential decisions under uncertainty with per-stage costs and a catastrophic tail — and by 1960 he was extending the framework to adaptive control when transition probabilities are imperfectly known. The state space here is small enough that his tabular value iteration runs even on 1960 hardware, so his usual enemy, the curse of dimensionality, never shows up. Home turf, contemporaneous tools, canonical formulation.
Bertsekas wrote the definitive modern treatment of precisely this problem class: 'Dynamic Programming and Optimal Control' works through machine replacement and maintenance as standard examples, with discounted, average-cost, and infinite-horizon formulations all made rigorous. Crucially, the hidden-wear wrinkle — you observe inspections, not true condition — is a POMDP, and Bertsekas's texts give the belief-state reformulation that converts it back into a Markov decision problem over probability distributions, plus the sufficient-statistic arguments that justify it. His policy iteration and error-bound machinery would certify the resulting rule, and his later neuro-dynamic programming (1996, with Tsitsiklis) covers the case where the fleet state space grows too large to tabulate. He arrives decades after 1960, but he is the mathematician who made this exact industrial application airtight.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
31 figures are scored on this problem. Draw it in a battle to see where you land.