optimization
Tune the un-differentiable
It is the era of expensive simulations and complex systems with knobs no one can differentiate — a chemical process, a machine-learning pipeline — where each single evaluation costs hours of compute or a physical experiment. You must optimize thirty hyperparameters of such a black box with a budget of only two hundred evaluations and no gradients to guide you. Random or grid search would exhaust the budget learning almost nothing. Build a surrogate model of the response and use it to choose each next evaluation where it teaches the most, balancing exploration against exploitation. Get it wrong and you squander a tiny, costly budget on uninformative trials and never find the good configuration — when every evaluation is precious, sample-efficient search decides whether the system gets tuned at all.
Who this problem belongs to
The two figures whose methods fit it best, out of 70 in contention.
This problem is Box's life work restated. Response surface methodology, which he created with Wilson in 1951 at ICI, was born from precisely this situation: a chemical process with many knobs, each run expensive, no gradients, and a need to find good operating conditions in few experiments. His answer, fit a local polynomial surrogate, design the next batch of runs where they are most informative, follow steepest ascent, then refine with second-order designs, is the direct ancestor of modern Bayesian optimization, and his EVOP brought the same sequential logic to running plants. His Bayesian work and the all-models-are-wrong dictum are the exact epistemology of surrogate modeling. Honest gaps: thirty dimensions strains classical fractional factorials, and Gaussian-process surrogates with expected improvement postdate him. But no one in this batch matches problem to method more perfectly.
This problem is nearly a MacKay exam question. His 1992 papers on Bayesian interpolation and information-based objective functions for active data selection are direct ancestors of modern Bayesian optimization: fit a probabilistic surrogate, get honest error bars, then choose the next datum where expected information gain is highest. He treated neural networks as Bayesian regression surrogates with calibrated uncertainty, used the evidence framework to pick model complexity from scarce data, and his textbook wove information theory and inference into exactly the exploration currency acquisition functions spend. His whole career, from Dasher to low-density parity-check codes to sustainable-energy arithmetic, was extracting maximum information per costly observation. Gaps are minor: he worked before the hyperparameter-tuning industrial era and did not publish on optimization-specific acquisition functions like expected improvement, but every conceptual ingredient is his.
Fought here
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
70 figures are scored on this problem. Draw it in a battle to see where you land.