high-dim
Five hundred stocks, one year of days
It is 2008, and portfolio engines across the financial industry share a buried flaw: optimizing five hundred assets requires their covariance matrix, estimated from roughly 250 trading days — fewer observations than dimensions, so the sample matrix is singular, its smallest eigenvalues pure artifact. Worse, the optimizer is a machine for finding exactly those artifacts: it loads hardest on the noise directions that promise impossible diversification. Estimate covariance honestly in this regime — shrinkage toward structured targets, factor models, sparsity in the inverse — and quantify what the optimizer's error costs in realized risk. The year makes the stakes vivid: portfolios built on fictitious correlations are about to discover, simultaneously, what their true tail dependence is. Get it wrong and diversification evaporates precisely when it was the entire point.
Who this problem belongs to
The two figures whose methods fit it best, out of 64 in contention.
Friedman, together with Hastie and Tibshirani, published the graphical lasso in 2008, the exact algorithm this problem asks for: an efficient method for estimating a sparse inverse covariance matrix by L1-penalizing its entries, precisely addressing the singular, p-approximately-n covariance matrices that portfolio optimizers exploit as fictitious diversification. His earlier work on CART and gradient boosting shows a career-long instinct for regularizing flexible models against overfitting to noise, the exact discipline missing from naive sample covariance in this scenario. The timing could not be more direct: the paper appeared in the same year, 2008, that this problem is set, addressing the financial-crisis-era covariance-estimation crisis as it was unfolding. His score reflects genuinely primary authorship of the specific technique this problem needs.
Hastie is a co-author, with Friedman and Tibshirani, of the 2008 graphical lasso paper that directly solves this problem: sparse estimation of the inverse covariance matrix via L1 penalization, engineered for exactly the p-approximately-n regime where five hundred assets and roughly 250 trading days leave the sample covariance singular. His Elements of Statistical Learning, co-written with the same collaborators, treats shrinkage and regularization in high dimensions as a central organizing theme, and his broader career applying statistical learning to genomics and finance-adjacent high-dimensional problems gives him deep comfort with exactly this failure mode. His score reflects primary co-authorship of the field's standard technical answer to the specific 2008 covariance-estimation crisis this problem describes.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
64 figures are scored on this problem. Draw it in a battle to see where you land.