David Marr

Life and career

David Marr wrote his most important book while dying. Diagnosed with acute leukemia in 1978, he spent his remaining time drafting *Vision*, working from a hospital bed and from home, finishing the manuscript before his death in November 1980 at the age of thirty-five. The book was published in 1982 with the assistance of his colleagues and his wife, Lucia Vaina. It is still assigned in vision courses forty years later, and its opening chapters are among the most-cited methodological arguments in cognitive science.

Marr was born in Essex, England, in 1945, educated at Rugby School, and read mathematics at Trinity College, Cambridge, where he took his degree in 1966. He then turned to neuroscience, completing a PhD in 1972 under Giles Brindley in the physiological laboratory. His doctoral and immediately post-doctoral work is remarkable in its own right and is often forgotten because of what came later: three theoretical papers on the cerebellum (1969), the neocortex (1970), and the hippocampus (1971), each proposing a computational function for a brain structure based on its known anatomy. The cerebellum paper — arguing that the cerebellar cortex is a learning machine in which climbing fibers act as a teaching signal modifying parallel-fiber synapses onto Purkinje cells — was developed independently by James Albus and is now known as the Marr–Albus theory. It remains a live hypothesis in motor learning. The hippocampus paper proposed autoassociative recurrent storage and recall, which is essentially the Hopfield network eleven years early.

In 1973 Marr moved to the MIT Artificial Intelligence Laboratory, invited by Marvin Minsky and Seymour Papert, and shifted from theoretical neuroanatomy to vision. MIT in the 1970s was the right and wrong place for him at once: it had the computational culture and the machine vision effort, but its dominant style was symbolic AI, and Marr's insistence that vision required *its own* rigorous theory before any programming began cut against the lab's engineer-first habits. He became a professor of psychology in 1980, months before his death. He worked closely with Tomaso Poggio, Ellen Hildreth, Keith Nishihara, and Shimon Ullman, and the collaborations produced most of the specific technical machinery in the book.

The Marr Prize, awarded at the International Conference on Computer Vision, is named for him and is the field's highest paper honor. A striking number of computer vision's leaders trace their framing of the discipline to a book written by a man who was dead before the field he named had produced a working system.

Key contributions

**The three levels of analysis.** This is the piece of Marr that has traveled furthest outside vision, into cognitive science, neuroscience, and increasingly into discussions of what a trained network is doing. Marr argued that any information-processing system must be understood at three distinct and largely independent levels:

The **computational theory** asks what problem is being solved and why — what is the input, what is the desired output, and what constraints from the structure of the world make the mapping well-posed at all. The **representation and algorithm** level asks how the input and output are represented and what procedure transforms one into the other. The **hardware implementation** level asks how that procedure is physically realized, in neurons or silicon.

The polemical force of this is in the claim of *priority*: you cannot understand a system by studying its implementation alone. Marr's famous illustration is that studying feathers will not teach you aerodynamics. A recording from a cortical cell tells you very little unless you already have a theory of what computation the cell participates in. This argument was aimed at neurophysiology in 1980; it is aimed, now, at anyone who thinks probing activations will by itself explain a neural network.

**The reconstructionist pipeline.** Marr's substantive theory of vision is that the visual system recovers a description of the three-dimensional world from images in stages, each producing an explicit representation. The **primal sketch** makes explicit the intensity changes in the image — edges, bars, blobs, terminations — and their local geometric organization. The **2.5-D sketch** is a viewer-centered representation of surface orientation, depth, and discontinuity: what the surfaces in front of you are doing, but not yet what objects they belong to. The **3-D model representation** is object-centered and hierarchical, built from generalized cylinders in a coordinate frame attached to the object itself, so that a recognized shape is invariant to viewpoint.

**Specific algorithms.** Marr and Hildreth's edge detector (1980) convolves the image with a Laplacian of Gaussian and locates **zero crossings** of the result — a principled scheme derived from the requirement that intensity changes be detected at multiple spatial scales, and one that Marr connected to the center-surround receptive fields of retinal ganglion cells. Marr and Poggio's cooperative stereo algorithm (1976, 1979) formalizes stereopsis by stating the constraints that make correspondence solvable — uniqueness (a point in one image matches at most one in the other) and continuity (disparity varies smoothly except at object boundaries) — and then finding a match satisfying them. Marr and Nishihara (1978) developed the generalized-cylinder object representation.

The methodological point runs through all of them: state what makes the problem well-posed, derive the algorithm from those constraints, and only then check whether the brain appears to implement it. This is the opposite of the tinkering that dominated machine vision at the time, and it is why the field regards him as its founder even though almost none of his specific algorithms survive in current practice.

In battle

Marr's profile is narrow, high, and short — a mean of 21.0 with a median of 15, sixty-five problems at or below 20, and exactly two dominant scores. He is a specialist in one modality, and outside it he has essentially nothing.

The peak is **Edges before objects** (P069) at 99. This is his program stated in his own words: the three levels, the primal sketch, zero crossings of the Laplacian of Gaussian, the 2.5-D sketch, tied to retinal and cortical physiology as constraint rather than decoration. Nothing on the roster matches the problem more precisely. The one point withheld reflects the honest caveats — the neuroscience in the framework was partly speculative, and his insistence on bottom-up early vision before top-down interpretation was contested afterward and has not entirely held up.

**Where did every pixel go?** (P234) at 93 is optical flow, where the constraint-derivation style he pioneered is exactly the right method: the aperture problem is a statement about what the data underdetermine, and the smoothness assumption is a stated constraint that makes the problem well-posed — Horn and Schunck's formulation is Marr's methodology even where it is not his paper. **Depth for a robot on Mars** (P241) at 75 is stereo, his and Poggio's direct contribution, though Kalman-style estimation beats him on the pose-tracking part of the same problem. **Let the images choose the basis** (P288) at 74 is eigenfaces and PCA on images, where he supplies the representational framing but not the statistics. Below those, **Rebuild the city from vacation photos** (P235) at 60 (structure from motion), **The pattern survives the shift** (P232) at 58 (translation invariance), **Cut the image into things** (P240) at 55 (segmentation), and **The memory that completes the pattern** (P242) at 47 — the last a nod to his 1971 hippocampus paper, which anticipated autoassociative memory.

His `perception` mean of 41.1 across nineteen problems is the highest category figure he has, and it is the whole of his strength. `high-dim` at 24.9 and `networks` at 22.0 reflect residual relevance from the neural-theory work.

The floor is sharp and worth reading carefully, because it is the source of the game's most pointed lesson about him. He scores 5 on **Learn from the stream, one at a time** (P150), 5 on **Trained here, deployed there** (P153), 5 on **The variable you removed is still there** (P289), 4 on **Eleven factors, twelve runs** (P114, fractional factorial design), 4 on **The certificate of optimality** (P177, LP duality), and 3 on **Three species of iris** (P151). His `experimental-design` mean is 5.7, `optimization` 7.0, `small-sample` 7.5, `games` 8.2, and — the telling one — `classification` 14.6 across seventeen problems.

That classification number is the point. Marr theorized vision and cannot classify. There is no learning algorithm anywhere in his framework: the constraints come from analysis of the physical world, the representations are designed, and the parameters are derived rather than fitted. When a vision problem is posed as "learn a mapping from labeled images," his machinery is silent, and he loses to figures with no theory of vision whatsoever but a working estimator.

This is exactly the confrontation his battle identity names. The end-to-end deep learning systems that now do everything his pipeline aimed at do not compute a primal sketch, do not build a 2.5-D sketch, and do not represent objects as generalized cylinders. They skip his levels. Marr's counter-argument — that a system which works is not yet a system that is understood, and that the computational-theory question remains unanswered even when performance is solved — is the strongest thing anyone has said against the current paradigm, and the game lets a student feel both sides of it in a few rounds.