perception
Segment anything
It is 2022 in a Meta AI lab, and segmentation has spent thirty years as a piecework trade: one model for tumors, one for pedestrians, one for cats, each demanding its own labeled dataset and its own months of engineering. The foundation-model era poses a heretical question: partition an arbitrary image into meaningful regions with no task-specific training at all — promptable, general-purpose segmentation, where a click, a box, or a word specifies what "meaningful" means this time. Success requires data engines that label millions of images by bootstrapping the model on itself, and an architecture that amortizes everything. The stakes: either segmentation becomes infrastructure — one model, every downstream task from radiology to robotics — or the foundation-model thesis fails its first real test outside language.
Who this problem belongs to
The two figures whose methods fit it best, out of 56 in contention.
Girshick is not merely applicable to this problem; historically, he is its answer — a core author of Segment Anything itself (2023, with Kirillov and colleagues at Meta AI). The entire arc of modern detection and segmentation runs through his hands: R-CNN (2014) proved deep features transfer to localization, Fast and Faster R-CNN made it end-to-end and real-time, and Mask R-CNN (2017, with He and Dollár) made high-quality per-object masks routine — the direct architectural ancestor of promptable mask prediction. The data engine's model-in-the-loop annotation extends practices his Detectron lineage normalized at FAIR. Prompt encoding, ambiguity-aware multiple masks, ViT backbones adapted to dense prediction: each is a design decision his group actually made. The only honest deduction is that SAM was a team's achievement, not one man's.
Malik owns the intellectual history of this problem. Normalized cuts (with Shi, 2000) made what is a meaningful region a precise optimization question; his Berkeley Segmentation Dataset and boundary benchmarks defined how the field measures the answer; gPb and his perceptual-organization program carried Gestalt grouping into computation; and his students and academic descendants populate the FAIR vision culture that built Segment Anything — the problem's own Win line names him. Even SAM's signature move, returning multiple nested masks for one ambiguous click, is his old observation that segmentation is ill-posed without a notion of scale and intent. He embraced deep learning rather than resisting it. What keeps him a notch below the actual builder: the transformer-era engineering and the industrial data engine are his descendants' craft more than his own hands-on practice.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
56 figures are scored on this problem. Draw it in a battle to see where you land.