rl
The policy learned from old charts
It is 2018, and intensive-care databases hold thousands of complete sepsis trajectories — vitals, labs, fluids, vasopressors, outcomes — and a seductive proposal: learn the optimal treatment policy from the records alone, since no one can randomize dying patients over drug doses. Do it without fooling yourself, which is the entire problem: the logged actions came from clinicians reacting to what they saw and to what the record never captured, so naive off-policy learning confuses the signature of a doctor's alarm with the effect of the drug. Bound what the data can support, evaluate the learned policy without deploying it, and flag the recommendations that rest on regions the data barely visits. Get it wrong and the algorithm's confident dosing advice is confounding, laundered into a protocol.
Who this problem belongs to
The two figures whose methods fit it best, out of 36 in contention.
Murphy's research program is built for precisely this problem: she pioneered micro-randomized trials and statistical methods for reinforcement learning in mobile health, developing the machinery for learning sequential treatment policies from observational and quasi-experimental data while explicitly confronting confounding between clinician decisions and patient trajectories. Her work on dynamic treatment regimes formalizes exactly the sepsis question, what is the optimal sequence of fluids and vasopressors, using methods honest about what logged data can and cannot support, including when a policy recommendation rests on regions the data barely visits. She has published directly on off-policy evaluation for healthcare decision-making and on the danger of confusing a clinician's alarmed response to a crashing patient with the causal effect of a drug. No carrier in this pool is closer to the actual literature this problem describes.
Rubin's potential-outcomes framework, developed from the 1970s onward, is the formal language for exactly the danger this problem names: confounding between what a clinician observed and what they chose to do, the same structure that makes naive off-policy learning from sepsis records confuse a doctor's alarm with a drug's effect. His work on propensity scores and matching gives concrete tools for adjusting logged treatment data toward something closer to a randomized comparison, and his EM and multiple-imputation methods address the missingness inherent in incomplete medical records. What is less native to him is the sequential, multi-decision reinforcement-learning structure specifically, his causal framework is built primarily for single or few time-point treatment effects, and adapting it to long treatment trajectories draws on later extensions rather than his own original formalism.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
36 figures are scored on this problem. Draw it in a battle to see where you land.