Frederick Jelinek

Life and career

The most quoted line in the history of natural language processing — "every time I fire a linguist, the performance of the speech recognizer goes up" — was said by a man who spent his career insisting he had never actually said it in that form, and who nonetheless meant it. Frederick Jelinek was not being cruel. He was making an empirical claim: that for the problem of turning acoustic waveforms into text, a probability model estimated from a large corpus outperformed any set of rules a human expert could write down. In 1975 this was heresy. By 1995 it was the field.

Jelinek was born in Kladno, Czechoslovakia, in 1932, into a Jewish family. His father, a dentist, died in Theresienstadt. Jelinek survived the war in Prague and emigrated with his mother and siblings to the United States in 1949, arriving as a teenager with limited English — a detail that gives a certain irony to the career that followed. He worked his way through school, took his degrees at MIT, and completed a PhD in electrical engineering there in 1962 under Robert Fano, in information theory. His early technical work was on coding theory: sequential decoding, tree codes, and the analysis of channel coding, and he wrote a well-regarded 1968 textbook, *Probabilistic Information Theory*.

He taught at Cornell from 1962 to 1974, and then made the move that defined him. IBM's Thomas J. Watson Research Center recruited him to lead a speech recognition group, and he stayed for twenty years, building what became the most influential research group in the history of the field. The IBM Continuous Speech Recognition group under Jelinek assembled a remarkable roster — Lalit Bahl, Robert Mercer, Peter Brown, the Della Pietra brothers, John Lafferty, and others — and their approach was uniform: treat the problem as noisy-channel decoding, estimate everything from data, and evaluate on held-out test sets. Several of those people later took the same statistical methods to Renaissance Technologies and made extraordinary amounts of money; Mercer and Brown ran the firm. That is a footnote to the science but a real one about how deeply the group believed in estimating from data.

In 1993 Jelinek left IBM for Johns Hopkins, where he founded and directed the Center for Language and Speech Processing. Hopkins under Jelinek became the training ground for a large fraction of the statistical NLP community, and he established the summer research workshops that brought together researchers from many institutions to work intensively on shared problems — an institution-building achievement that arguably matched his technical one.

Colleagues describe a demanding, dryly funny man who did not soften disagreements and who held students to standards he did not explain twice. He remained at Hopkins, working, until he died in his office there in September 2010, at seventy-seven.

Key contributions

**The noisy-channel formulation of speech recognition.** Jelinek's central move was to import Shannon's framework wholesale. Treat the spoken sentence *W* as a message sent through a noisy channel — the speaker's vocal tract, the room, the microphone — arriving as an acoustic signal *A*. Recognition is then decoding: find *W* maximizing P(*W* | *A*), which by Bayes is proportional to P(*A* | *W*) P(*W*). The first factor is the **acoustic model**, the second the **language model**, and they can be estimated separately from different data. This decomposition is the architecture of speech recognition for the next thirty years, and it generalizes: the same P(target)·P(source | target) structure organizes statistical machine translation, spelling correction, and part-of-speech tagging.

**Hidden Markov models as the acoustic model.** Jelinek's group made the HMM the workhorse of speech: phones as hidden states, spectral feature vectors as observations, transitions and emissions estimated by the forward–backward (Baum–Welch) algorithm, decoding by Viterbi. Neither the HMM nor Baum–Welch originated with him, but IBM's group is the reason the machinery became the default, and they contributed heavily to the practical engineering — tied states, context-dependent phone models, efficient decoding.

**n-gram language models and smoothing.** The language model P(*W*) was estimated as a product of conditional probabilities over short histories: trigrams, in the classic case. The hard part is not the counting but the zeros — most trigrams you will ever encounter never appeared in training. Jelinek and Robert Mercer's **deleted interpolation** (1980) interpolates the trigram, bigram, and unigram estimates with weights fitted on held-out data, using the EM algorithm to set them. This is a hierarchical shrinkage estimator, and it is one of the earliest large-scale practical uses of held-out estimation. Katz backoff, Good–Turing discounting, and eventually Kneser–Ney all address the same problem; Jelinek–Mercer smoothing is still a named baseline.

**Perplexity.** If you evaluate language models by their cross-entropy on held-out text, exponentiate it, and call the result perplexity — the effective average branching factor of the model — you have Jelinek's contribution to evaluation methodology. It sounds minor and is not. Perplexity gave the field a cheap, automatic, corpus-based metric that could be computed without running the full recognizer, which meant language modeling could progress by measurement rather than argument. The insistence on held-out evaluation, on test sets you do not touch, and on numerical comparison is as much Jelinek's legacy as any specific model.

**Statistical machine translation.** The 1990 and 1993 papers from Jelinek's group — Brown, Della Pietra, Della Pietra, Mercer and colleagues — took the Canadian Hansard parliamentary proceedings, a large corpus available in both English and French, and learned translation from parallel text with no hand-built grammar at all. The IBM Models 1 through 5 define a generative story in which a source sentence produces a target sentence through alignment, fertility, and distortion parameters, all estimated by EM. Word alignment as a latent variable learned from parallel corpora is the foundation on which phrase-based SMT and, eventually, neural MT were built.

In battle

Jelinek's profile is the sharpest instrument in the statistical NLP tradition: mean 30.6, median 20, nine dominant scores, and one perfect 100.

That 100 is **The parallel text is the teacher** (P256) — statistical machine translation from Hansard. This is not analogy; it is his group, his building, his year. He led the team that treated translation as noisy-channel decoding over millions of sentence pairs with no grammar rules, and his maxim about firing linguists is the literal defense of the approach the problem describes, delivered by the person who defended it publicly against linguists who called it barbarism. There is no closer fit available.

**Words from waveforms** (P073) at 98 is speech recognition itself — the same claim, made twenty years earlier and more painfully. **Compress the English novel** (P037) at 90 rewards a man trained in information theory who spent his career estimating the entropy of English by held-out cross-entropy; language modeling and compression are the same problem in different notation, and he lived on both sides of the identity. **Decode the hidden path** (P038) at 90 is Viterbi decoding over an HMM, the operational core of his systems. **A million parsed sentences** (P261) at 88 is the Penn Treebank and the statistical parsing era it enabled; **Translate Russian by machine** (P255) at 85 is the earlier MT effort his methods eventually vindicated; **Label every word** (P085) at 84 is HMM-based part-of-speech tagging, a direct transplant of his acoustic machinery to text.

The one that may surprise is **The one-in-a-million event** (P006) at 87 — estimating the probability of things you have never observed. That is the smoothing problem, and it is arguably the deepest statistical contribution of his group. His `small-sample` category mean of 50.5 comes almost entirely from this: he is one of the few figures in the roster whose central technical worry was the zero-count cell.

His category means read exactly as you would predict: `nlp` 56.1 across eighteen problems, `information` 42.6 across sixteen, `classification` 25.2, `perception` 24.2.

The losses are equally clean. He scores 5 on **Why tall fathers have shorter sons** (P133, regression to the mean), 5 on **Three species of iris** (P151, Fisher's discriminant), 5 on **The pixel you cannot see** (P152), 5 on **Beat the world champion** (P200, Deep Blue), 5 on **The machine that overdosed** (P248, the Therac-25 safety failure), and 2 on **The first randomized trial** (P113) — his floor. His `experimental-design` mean is 7.0, `systems` 5.5, `regression` 9.5, `search` 11.5, `games` 11.5.

The pattern is precise and worth stating for students, because it cuts against a lazy reading of "statistician." Jelinek is a *sequence* statistician working in the noisy-channel paradigm with abundant data. He is not a classical inferential statistician: randomization, experimental design, causal identification, and estimation theory were never his subject. And he is not a systems or search person at all — game tree search, software safety, and combinatorial optimization are foreign. Play him wherever there is a corpus, a sequence, and a probability to estimate. Against Fisher on trial design, or against Deep Blue's engineers on chess, the man who taught the field to measure everything has nothing to measure.