Bernhard Scholkopf

Life and career

Bernhard Schölkopf has had two careers, and the interesting thing is that he ended the first one himself. Through the 1990s and 2000s he did as much as anyone to turn kernel methods from a clever trick into a systematic discipline — kernel PCA, one-class SVMs, kernel mean embeddings, the textbook. Then, around the time deep learning made fixed-kernel machines look dated, he pivoted a large research empire toward causality, on the argument that the thing statistical learning could not do was tell you what would happen if you intervened, and that this was the deficiency that mattered.

He was born in Stuttgart and trained in physics and mathematics in Germany and Britain, taking a master's degree in mathematics in London before returning for a doctorate in computer science at the Technical University of Berlin. His doctoral work was supervised in collaboration with Vladimir Vapnik, then at AT&T Bell Labs — which is to say he learned statistical learning theory from the person who invented it, at exactly the moment the support vector machine was being built. The physics training is audible in his writing: a preference for asking what invariance or symmetry a method is exploiting, and a suspicion of methods that work without an account of why.

After his doctorate he worked at the German national research center GMD FIRST in Berlin and spent a period at Microsoft Research in Cambridge. In 2001 he became a director at the Max Planck Institute for Biological Cybernetics in Tübingen, and subsequently a founding director of the Max Planck Institute for Intelligent Systems, which he helped build into one of Europe's principal machine learning centers. He has been a driving force behind ELLIS, the European Laboratory for Learning and Intelligent Systems, a network founded on the conviction that Europe needed to retain its machine learning talent rather than export it. He is among the most decorated scientists in Germany, including recipient of the Gottfried Wilhelm Leibniz Prize, the country's foremost research award.

His applied collaborations are unusually far-flung and worth knowing about because they explain his methodological instincts. One of the more striking is in astronomy: with collaborators he developed **half-sibling regression** for removing instrument systematics from exoplanet transit light curves, exploiting the fact that noise shared across many detector pixels is causally attributable to the instrument while the transit signal is not. This is a causal argument doing real work on a real dataset, and it is the kind of thing he means when he says causality is not philosophy.

Key contributions

**Kernel methods, systematized.** The kernel trick — replace inner products with a positive-definite kernel and you implicitly work in a high- or infinite-dimensional feature space at the cost of the original one — was known before Schölkopf. What he did, with Alexander Smola, Klaus-Robert Müller, and a long list of collaborators, was recognize that essentially *any* algorithm expressible in inner products could be kernelized, and then systematically do it.

**Kernel PCA** was the demonstration case: principal component analysis in feature space, computed by eigendecomposition of the centered kernel matrix, yielding nonlinear principal components without ever forming the feature map. **One-class SVMs** and support vector novelty detection attacked a different problem — estimate a region containing most of the probability mass, without density estimation — by finding the hyperplane in feature space that separates the data from the origin with maximum margin, giving a principled anomaly detector with a tunable outlier fraction. His **ν-SVM** reparameterization replaced the uninterpretable regularization constant C with a parameter ν that directly bounds the fraction of margin errors and support vectors, which is what a practitioner can actually reason about. And his generalization of the **representer theorem** established the conditions under which the solution to a regularized risk functional in a reproducing kernel Hilbert space is guaranteed to lie in the span of the training data — the theorem that makes all of this computable.

*Learning with Kernels*, with Smola, remains the reference text for the whole apparatus.

**Kernel mean embeddings.** A further step with real consequences: embed an entire *probability distribution* as the mean of its feature map in an RKHS. For a characteristic kernel this embedding is injective, so distributions are distinguished by their embeddings, and distances between distributions become distances in Hilbert space. This yields the **maximum mean discrepancy** as a two-sample test statistic — a nonparametric test of whether two samples come from the same distribution — and, applied to joint versus product distributions, the **Hilbert-Schmidt Independence Criterion**, a kernel measure of statistical dependence. HSIC in particular became a workhorse, because independence testing is the primitive operation in constraint-based causal discovery, and this was Schölkopf's bridge from kernels to causality.

**Causal discovery in the bivariate case.** Standard constraint-based causal discovery cannot distinguish X → Y from Y → X: the two graphs are Markov equivalent and imply the same conditional independences. Schölkopf's Tübingen group attacked this head on. **Additive noise models** exploit an asymmetry: if Y = f(X) + N with N independent of X and f nonlinear, then generically no model of the form X = g(Y) + Ñ with Ñ independent of Y exists. So you fit both directions, test residual independence in each, and the direction that passes is the causal one. The group also assembled and curated the **cause-effect pairs benchmark** — real bivariate datasets with known ground-truth direction — which made the whole subfield empirically accountable.

**Independence of cause and mechanism.** The deeper principle behind all of it: the distribution of the cause, P(X), and the mechanism generating the effect from the cause, P(Y|X), are "independent" in the sense that nature did not tune one to the other — they contain no information about each other. Schölkopf has formalized this in several ways, including via algorithmic information theory. It has a striking practical corollary, developed in his work on **causal and anticausal learning**: semi-supervised learning should help when you are predicting a cause from an effect (anticausal), because then the marginal of the input carries information about the conditional you want, and should *not* help in the causal direction, where the two are independent by assumption. This retrodicts a great deal of otherwise confusing empirical experience with semi-supervised methods.

The same principle drives his account of **transfer and distribution shift**: if you have identified the true causal mechanisms, some of them will be invariant across environments while spurious statistical associations will not. This is the intellectual foundation of invariance-based approaches to out-of-distribution generalization, and of his more recent program on **causal representation learning** — the argument that the units a model should learn are causal variables, not arbitrary latent coordinates. *Elements of Causal Inference*, with Peters and Janzing, is the compact modern statement of the whole framework.

In battle

Schölkopf's profile is broad and strongly peaked in exactly the two places his career has been: mean 36.3, seven dominant problems, `causality` at 70.5 topping the category table, with `classification` at 50.5 and `high-dim` at 45.0 underneath.

**P219 — Which one is the cause?** at 96 is the clearest hit. Bivariate cause-effect determination via additive noise models and independence of mechanism, benchmarked against ground truth — that is his group's program down to the canonical altitude-and-temperature style example, and he co-authored the benchmark datasets themselves.

**P153 — Trained here, deployed there** at 90 is distribution shift, and here his causal work gives him an argument nobody else has: the invariant mechanisms are the ones that transfer, and identifying them is a causal problem, not a statistical one. He edges Bin Yu's 88 on the same problem — two very different justifications for the same instinct.

**P027 — Classify with 30 labels and 30,000 unlabeled** at 88 is semi-supervised learning, where he contributes both the kernel machinery and, distinctively, the causal-versus-anticausal analysis of *when it can work at all*. He lands just behind Niyogi's 95. **P023 — The line between two clouds** (87), **P020 — The smooth curve, honestly** (85), **P025 — Sixty thousand digits** (84), and **P024 — XOR** (82) are the kernel core: margin classification, kernel ridge regression, digit recognition, and the canonical nonlinearly-separable problem that a feature-space lift dissolves instantly. **P284 — Unroll the swiss roll** at 78 is kernel PCA's territory, though Niyogi's eigenmaps beat him there.

Note the pattern in his battle identity: he loses raw perception benchmarks to deep nets, then wins the "why does it transfer" postmortem. `perception` at 34.3 is exactly that — decent, never dominant. Fixed kernels do not learn features, and on the benchmarks where learned features won, he is the runner-up.

His losses are cleanly outside mathematics-of-data entirely. `systems` at 8.5 and `games` at 10.0 are the floor. **P246 — Small tools, loosely joined** (7) is Unix design. **P199 — Prune the game tree, provably** (8) is alpha-beta search, **P204 — Solve the game, completely** (12) is exhaustive game solving; nothing in kernel methods or causal graphs touches adversarial search. **P078 — Agreement among the unreliable** (10) is Byzantine consensus, **P158 — The problem all others reduce to** (10) is NP-completeness, and **P036 — Correct the corrupted block** (11) is error-correcting codes.

The tactical read: Schölkopf is one of the safer broad picks on this roster, because he owns two distinct toolkits from two distinct eras and the game credits both. Play him on any nonlinear supervised problem for a solid score, and on anything asking *which direction does the arrow point* or *why did this stop working when we moved it* for a near-certain win. Bench him on anything discrete, adversarial, or systems-shaped.