Leo Breiman

Life and career

Leo Breiman's career has an unusual shape for a statistician: he made his name in pure probability, quit academia for thirteen years to work as a consultant, came back, and then spent the rest of his life telling his own discipline that it had taken a wrong turn.

Born in New York City in 1928 and raised largely in Los Angeles, he took his undergraduate degree in physics at Caltech and his PhD in mathematics at Berkeley in 1954, under Michel Loève. His early work was in probability theory — he wrote a well-regarded 1968 textbook, *Probability*, and is a co-namesake of the Shannon–McMillan–Breiman theorem, the almost-sure convergence result at the foundation of ergodic information theory. On that basis he became a professor at UCLA. By the mid-1960s he had, by his own later account, become dissatisfied with the distance between theorem-proving and anything he could see happening in the world.

So in 1967 he left. For thirteen years he worked as a freelance statistical consultant, on problems that had nothing to do with abstract measure theory: air pollution modeling for the California Air Resources Board, census-related methodology, highway and traffic data, weather modification experiments, and a long engagement with the practical question of how to make predictions that clients could actually act on. This period is the hinge of his intellectual life. Consulting clients did not want asymptotic efficiency; they wanted rules a physician or a loan officer could read and apply, and they wanted those rules to work on the next patient, not merely to fit the old ones.

He returned to academia in 1980, joining the Berkeley statistics department, where he stayed until his retirement in 1993 and remained active well past it. In 1984 he published, with Jerome Friedman, Richard Olshen, and Charles Stone, *Classification and Regression Trees* — CART — the book that gave decision trees a rigorous statistical footing. In the 1990s and 2000s, from formal retirement, he produced the ideas that most working data scientists now use daily: bagging (1996) and random forests (2001). In 2001 he also published "Statistical Modeling: The Two Cultures" in *Statistical Science*, an essay arguing that academic statistics had committed itself almost entirely to a "data modeling" culture — assume a stochastic model, estimate its parameters, interpret them — while an "algorithmic modeling" culture, judged by predictive accuracy on held-out data, was solving the problems that mattered. The paper was published with sharp discussion from senior statisticians. It reads today less as polemic than as prophecy.

Breiman was elected to the National Academy of Sciences and received major honors from the statistical profession. He died in 2005, at 77, in Berkeley.

Key contributions

**CART (1984).** Recursive binary partitioning was not invented from nothing — AID and similar procedures predate it — but CART is what made trees defensible. The algorithm greedily splits the feature space on the single variable and threshold that most reduces node impurity (Gini index, or the "twoing" criterion), recursing until nodes are small or pure. The genuinely important half is what comes next: *cost-complexity pruning*. Grow the tree deliberately too large, then define a penalized risk R_α(T) = R(T) + α|T| over subtrees, note that the sequence of minimizing subtrees as α increases is nested and finite, and choose α by cross-validated error. This converted trees from an ad hoc heuristic into a method with an honest complexity-control story. CART also introduced surrogate splits for missing data and a treatment of variable importance.

**Bagging (1996).** Breiman's diagnosis of trees was that they are *unstable*: small perturbations of the training set produce very different trees, which means high variance. His fix was bootstrap aggregating — draw B bootstrap samples, fit a predictor on each, average (regression) or vote (classification). Averaging leaves the bias roughly unchanged while cutting the variance, so bagging helps exactly for high-variance, low-bias base learners like deep unpruned trees, and does essentially nothing for stable ones like linear discriminant analysis. That conditional statement is the whole theory, and it is why bagging is taught as the canonical demonstration of the bias–variance decomposition in action.

**Random forests (2001).** Bagged trees are still correlated with each other, because a strong predictor gets chosen at the top split of nearly every tree, and averaging correlated predictors caps the variance reduction. Breiman's move was to decorrelate them deliberately: at each split, restrict the candidate variables to a random subset of size *m* out of *p*. Each tree gets slightly worse; the ensemble gets substantially better. He accompanied this with practical machinery that made forests unusually easy to deploy — out-of-bag error estimates, which give a nearly free held-out performance estimate from the roughly one-third of observations not in each bootstrap sample; permutation-based variable importance; and proximity measures. Random forests are still, in 2020s practice, one of the strongest off-the-shelf baselines for tabular data with modest sample sizes, and they require almost no tuning.

**Arcing and the boosting connection.** Breiman also studied AdaBoost empirically and theoretically, coining "arcing" (adaptive resampling and combining) and contributing to the understanding of why sequential reweighting of hard examples works — a line of thinking that connects his ensembles to Friedman's gradient boosting.

**"The Two Cultures" (2001).** The essay's claims are specific, not merely rhetorical. That the data-modeling culture's conclusions are about the *model*, not about nature, and are therefore only as good as an untested assumption. That goodness-of-fit tests and residual analysis are weak instruments for validating a model in high dimensions. That predictive accuracy on held-out data is the honest arbiter. That the Rashomon effect — many different models fitting the data about equally well while telling different stories — undermines confident interpretation of any one of them. And, importantly, that accuracy and interpretability trade off, a tension he owned rather than dodged: his own forests are far more accurate and far less readable than his own single trees.

In battle

Breiman's computed profile shows a persona with a very high ceiling in a specific region and a very hard floor everywhere else: mean 33 across a hundred problems, median 26, eight dominant scores at 80 or above, and forty problems where he scores 20 or under. He is a specialist who happens to specialize in the thing most applied problems actually are.

His peak is **P146 — The tree you can read**, at 99. The judge's note calls it "his desk in 1984," and that is right: impurity-based recursive partitioning plus cross-validated cost-complexity pruning, in a framing about rules a loan officer or physician can apply, is CART reproduced from the specification. The only point withheld is for his own later argument that single trees are unstable. Close behind is **P022 — Predict, or explain?** (97) — the Two Cultures essay as a battle problem — and **P026 — A committee of weak learners** (92), which is bagging and boosting territory. **P029 — The tilted scan** (92) and **P019 — Regression with lies in it** (82) reward the robustness of tree-based methods to messy, contaminated, non-Gaussian data, where a linear model with a normality assumption simply fails. **P039 — Bet with information theory** (86) draws on his probability-theory roots. **P097 — Does your pipeline reproduce?** (86) and **P298 — Explain the denial** (85) reflect his out-of-bag validation discipline and his variable-importance work — he is the person on this roster most associated with the question "how do you know your accuracy number is real?"

By category, information (71), fairness (60), classification (50), and regression (45) are his strong ground. Note that classification is his largest well-scored category with seventeen problems: this is the breadth that makes him a good general-purpose draft pick for anything tabular and supervised.

The losses are instructive precisely because they are so total. He scores 5 on **P246 — Small tools, loosely joined**, the Unix design-philosophy problem, and the explanation says there is simply no bridge between ensemble prediction and shell pipelines. He scores 8 on **P080 — The software that may not fail** (formal verification), 8 on **P083 — The grammar beneath the sentence** (syntactic structure, where a bag-of-features tree learns nothing about hierarchy), 7 on **P159 — Is this number prime?** and 9 on **P157 — The equation with no algorithm** — decidability and number theory, categories where his computability average is 8. He scores 10 on **P259 — The imitation game, scored**, a philosophy-of-mind framing his empiricism has no purchase on. His weakest categories overall are systems (6.5), computability (8), causality (12), and search (14.5).

The causality number deserves a flag for students: at 12, Breiman is among the worst-suited personas on this roster for causal-inference problems, and this is not an accident of scoring. His entire program judges a model by predictive accuracy on held-out data — a criterion that is indifferent to whether the associations it exploits are causal. Send him against Pearl or Rubin on an intervention problem and he will lose, badly, for a principled reason he would probably have conceded.

Draft Breiman for tabular prediction, noisy real-world data, ensembles, model validation, and any problem where the tension between accuracy and interpretability is the point. Keep him away from formal methods, systems engineering, language structure, and causal identification.