perception
Name what you've never trained on
It is 2021 in San Francisco, and the standard vision recipe has a hidden invoice: every new task begins with a curated labeled dataset, so recognition is forever gated on annotation budgets and frozen category lists. The alternative uses supervision nobody had to buy — four hundred million image-text pairs scraped from the web — and trains two encoders to agree: image and caption pulled together, mismatched pairs pushed apart. The test is the radical part: classify standard benchmarks zero-shot, by asking which caption fits, matching supervised baselines without seeing one training label. Then audit what web-scale text smuggles in — its stereotypes ship with its supervision. Get it right and vision uncouples from the labeled dataset; the category list becomes a sentence anyone can rewrite.
Who this problem belongs to
The two figures whose methods fit it best, out of 61 in contention.
This is, almost without qualification, Radford's own paper: 'Learning Transferable Visual Models From Natural Language Supervision,' released by OpenAI in January 2021 and universally known as CLIP. He led the design of exactly this system: two encoders, one for images and one for text, trained contrastively on four hundred million web-scraped image-caption pairs so that matched pairs are pulled together in embedding space and mismatched pairs pushed apart. He also ran the zero-shot evaluation this problem describes, classifying ImageNet and dozens of other benchmarks by comparing an image embedding against candidate caption embeddings with no task-specific fine-tuning, and the paper's own limitations section already flags the bias-and-stereotype auditing this problem demands. It is difficult to imagine a more direct match between carrier and problem.
Sutskever is a listed co-author on the CLIP paper and, as OpenAI's chief scientist at the time, one of the people who backed the scaling hypothesis that made web-scale, weakly-labeled contrastive pretraining a plausible bet in the first place. His earlier AlexNet work with Krizhevsky and Hinton had already shown that raw compute plus enormous data could outperform carefully engineered features, the same wager CLIP makes by trading curated labels for four hundred million noisy web pairs. His sequence-to-sequence work also established the encoder-representation-learning intuitions the CLIP architecture depends on. He is not the paper's lead author or primary architect of the contrastive objective, which is why his score sits just below Radford's rather than matching it.
Fought here
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
61 figures are scored on this problem. Draw it in a battle to see where you land.