Noam Chomsky

Life and career

Two facts about Noam Chomsky sit awkwardly together for anyone in machine learning. He is the author of the formal result that every compiler course still teaches — the hierarchy of grammars, and the theorem that finite-state machines cannot generate the nested dependencies of natural language. He is also, for seventy years, the most persistent and least conciliatory critic of the statistical approach to language that now works spectacularly well. Both of these are him, and neither cancels the other.

Chomsky was born in Philadelphia in 1928 to Ashkenazi Jewish immigrant parents; his father was a Hebrew scholar who worked on medieval Hebrew grammar, which is either a coincidence or the whole explanation. He attended an experimental progressive school, then the University of Pennsylvania, where he studied under the structuralist linguist Zellig Harris — a mentor whose distributional methods Chomsky would ultimately reject, though the influence of Harris's political anarchism stuck. He spent 1951–1955 as a Harvard Junior Fellow and took his Pennsylvania doctorate in 1955 with work on transformational analysis.

He joined MIT in 1955, in the Research Laboratory of Electronics, and stayed for the rest of his academic career, eventually as Institute Professor. MIT is a strange home for a linguist, and the strangeness is productive: he was surrounded by information theorists, computer scientists, and engineers, and his early papers are written partly *for* them. "Three Models for the Description of Language" (1956) appeared in an IEEE information theory venue, not a linguistics journal. *Syntactic Structures* (1957) is a short book that reorganized a field.

The 1959 review of B. F. Skinner's *Verbal Behavior* is the other hinge. Chomsky argued that behaviorist accounts of language acquisition — reinforcement of verbal responses — could not in principle explain the productivity of language, the fact that speakers routinely produce and understand sentences they have never encountered. The review is widely credited with helping end behaviorism's dominance in psychology and with launching the cognitive revolution. Whatever one thinks of its argumentative fairness, its historical effect is not in dispute.

From the mid-1960s Chomsky became equally known as a political dissident, beginning with opposition to the Vietnam War, and has since written many volumes on U.S. foreign policy and media. He was arrested at antiwar protests and was on Nixon's enemies list. He is among the most cited scholars alive, across both bodies of work. Since 2017 he has been affiliated with the University of Arizona. He has continued to argue, well into his nineties and directly about large language models, that a system which learns the statistical structure of text is not thereby a theory of the human language faculty — a position that has hardened, not softened, as the systems have improved.

Key contributions

**The Chomsky hierarchy.** The 1956 paper defines classes of generative grammars by the form of their rewrite rules, and — the essential point — shows these classes are *strictly nested* in the languages they can generate. Type 3, regular grammars, correspond to finite automata. Type 2, context-free grammars, correspond to pushdown automata. Type 1, context-sensitive grammars, to linear bounded automata. Type 0, unrestricted rewriting, to Turing machines. The hierarchy is a theorem, not a taxonomy: there are languages a context-free grammar generates that no finite-state device can, and the standard demonstration — center embedding, "the rat the cat the dog chased bit died" — is a linguistic argument with a formal-language proof behind it. English contains such constructions; therefore no Markov model of English is adequate as a *competence* theory. That argument, aimed at Shannon-style n-gram models, is the founding shot of the entire structural-versus-statistical dispute.

For computer science the payoff was immediate and enormous. Context-free grammars, with Backus–Naur form as their notation, became the way programming language syntax is specified, and every parser generator — from the LR tables in yacc to modern recursive-descent frameworks — implements Chomsky Type 2. The Chomsky–Schützenberger theorem, characterizing context-free languages in terms of Dyck languages and regular sets, tied the linguistic classification to algebraic language theory. A graduate student today meets the hierarchy in automata theory and often never learns it came from a linguist arguing about English relative clauses.

**Generative grammar and transformations.** *Syntactic Structures* and, more fully, *Aspects of the Theory of Syntax* (1965) propose that a speaker's knowledge of language is a finite generative system producing an infinite set of sentences, and that surface forms are derived from underlying structures by transformations. *Aspects* introduced the deep structure / surface structure distinction and the **competence / performance** division — the claim that linguistics should model the idealized knowledge system, not the noisy behavior. That methodological choice is precisely what statistical NLP later rejected, and it explains much of the mutual incomprehension.

**Poverty of the stimulus and Universal Grammar.** The acquisition argument: children converge on complex, structure-dependent grammatical rules from data that is finite, noisy, and mostly lacking negative evidence, and they do so fast, uniformly, and without explicit instruction. Chomsky concluded that substantial constraints must be innate — a Universal Grammar delimiting the space of possible human languages. The Principles and Parameters framework (1980s) made this concrete: universal principles plus a small set of binary parameters set by exposure. The Minimalist Program (1990s onward) pushed toward a maximally spare account, with a single structure-building operation, Merge, doing most of the work. All of this is actively contested by usage-based and connectionist linguists; it is also the framework against which those alternatives define themselves.

**The critique of statistical language.** Chomsky's example sentence "colorless green ideas sleep furiously" was constructed to show that grammaticality and probability come apart: the sentence has essentially zero corpus probability and is unmistakably well-formed English, while a word-scrambled version has the same corpus probability and is not. Peter Norvig's widely read reply and the broader empirical turn in NLP are best understood as answers to this specific challenge, and the argument is still live in what a language model can be said to know.

In battle

Chomsky's profile is one of the most extreme peak-and-plain shapes in the game: mean 16.7, median 10, eighty-one problems at or below 20 — and a 99. That 99 is **The grammar beneath the sentence** (P083), and it is essentially his 1956 paper restated. The problem asks for a hierarchy of grammars with a proof that finite-state models cannot capture nested dependencies, addressed simultaneously to linguists and to compiler writers. He produced exactly that, in exactly that year, for exactly that dual audience. The only thing keeping him off 100 is that the full four-level formalization was finalized slightly later with Schützenberger.

His other dominant scores follow the same logic. **Translate Russian by machine** (P255) at 92 puts him at the Georgetown-era MT effort, where his structural argument was the strongest available diagnosis of why word-substitution translation failed. **A million parsed sentences** (P261) at 82 is the Penn Treebank: a corpus of syntactic annotations is unthinkable without his phrase-structure formalism, even though the statistical parsers trained on it embody a research program he opposes. Below that, **The imitation game, scored** (P259) and **A recognizer for a language of ten speakers** (P265) at 68, **Will it ever halt?** (P030) at 58, and **The truth it cannot prove** (P154) at 54 — the last two reflecting that his hierarchy sits inside computability theory, which is his best category at a mean of 49.

Note the shape of his `nlp` mean: 37.9 across eighteen problems. That is strong in absolute terms but far below what a naive reading of "greatest linguist of the century" would predict, and the reason is the whole point of his inclusion. The overwhelming majority of NLP problems in the game are statistical — alignment models, embeddings, language modeling, tagging with HMMs — and on those he is a bystander. He scores 47 on **What is in the training data?** (P266), a question about corpora he can characterize but not analyze.

The floor is brutal and diagnostic. He scores 4 on **Just look at the neighbors** (P145) — k-nearest neighbors — 4 on **The therapy the trial reversed** (P215), 4 on **Ascend the policy itself** (P223, policy gradient), 4 on **Depth for a robot on Mars** (P241, stereo vision), 3 on **The thousand-year flood from thirty years** (P107), and 3 on **Three species of iris** (P151), Fisher's discriminant. His `classification` mean is 8.3, `perception` 9.4 over nineteen problems, `causality` 5.5, `games` 5.5. There is no estimator anywhere in his toolkit. He has never fit a model to data in the sense this audience means.

The pedagogy here is the sharpest in the roster. Chomsky loses, repeatedly and by wide margins, to the statistical methods he spent his life arguing were the wrong theory of language — and he is simultaneously unbeatable on the one question of what languages a finite-state device can generate. A student who plays him will feel both facts within a few rounds, which is more than most textbook treatments manage.