high-dim
The eigenvalues are lying
It is 2006, and principal component analysis — the century-old workhorse — is quietly failing its heaviest users: a genomicist computes the top eigenvectors of a covariance matrix estimated from five hundred samples of ten thousand variables, and the leading eigenvalues come out impressively large even when the truth is pure identity noise. In the regime where dimension and sample size grow together, sample eigenvalues spread deterministically wider than the truth, and sample eigenvectors decorrelate from the population ones — below a sharp detection threshold, entirely. Characterize the phase transition: when is a genuine spike detectable at all, how biased is its sample estimate, and what corrections or sparsity assumptions restore consistency? Get it wrong and fields keep interpreting eigenstructure that is a theorem about noise.
Who this problem belongs to
The two figures whose methods fit it best, out of 60 in contention.
Donoho's career is built on exactly this failure mode: classical asymptotics (fixed dimension, sample size to infinity) breaking down when dimension and sample size grow together, and on finding the honest replacement theory. His work with Iain Johnstone on minimax estimation in high dimensions and his later formalization of the proportional-growth regime gave the field its vocabulary for phase transitions in exactly this setting, where sample eigenvalues of pure noise spread deterministically beyond their population values. His 'high-dimensional data analysis: the curses and blessings of dimensionality' lecture (2000) anticipated the genomics scenario directly, and his broader fifty-year retrospective on data science treats p much greater than n as the field's defining rupture. The only reason this does not reach the very top is that the sharpest spiked-covariance detection threshold itself was formalized slightly later by Johnstone, whom Donoho worked alongside.
Bickel spent the 2000s at Berkeley building the rigorous asymptotic theory for exactly this regime: his 2008 papers with Elizaveta Levina on regularizing large covariance and precision matrices by banding and thresholding directly confront the problem of a five-hundred-sample, ten-thousand-variable covariance matrix whose raw sample eigenstructure is unusable. His broader program on semiparametric efficiency gave him the tools to state precisely what is and is not estimable as dimension grows with sample size, rather than merely observing that something goes wrong. He treated the genomics and microarray applications this problem describes as his own working examples, not hypotheticals. His score falls just short of the top only because his emphasis was regularized estimation and consistency rather than the sharp phase-transition threshold itself, which is a companion strand of the same literature he helped establish.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
60 figures are scored on this problem. Draw it in a battle to see where you land.