systems
Index the entire web
It is 2003 in Mountain View, and the arithmetic of scale has turned against heroism: the web is billions of pages, the index must be rebuilt continuously, and the only affordable hardware is racks of commodity machines so cheap that at any given hour, somewhere in the datacenter, several are dying. Build and refresh a searchable index of billions of pages on machines that fail daily — which means failure cannot be an exception; it must be the design's ordinary weather. Storage, computation, and coordination all need abstractions that let ordinary programmers write jobs that survive dying hardware without thinking about it. The stakes: whether the world's information is searchable at all, and whether planetary-scale computing becomes an engineering discipline or stays a priesthood.
Who this problem belongs to
The two figures whose methods fit it best, out of 36 in contention.
This is Dean's own building, his own year: MapReduce and the Google File System were designed and deployed at Google in exactly this 2003-2004 window to solve exactly this problem — indexing a rapidly growing web on racks of commodity machines that fail as routine weather, not exception. Dean co-authored the MapReduce paper that let ordinary programmers write map and reduce functions without touching failure handling, replication, or scheduling directly, which is precisely the abstraction the problem demands. No other carrier owns both the historical deployment and the specific engineering answer this completely. Even granting the era gap, Dean's standing on mapreduce, bigtable, tensorflow; large-scale ml systems is close enough to this problem's actual demands that a graduate student would expect a real, defensible showing rather than a token one.
Stoica's PhD work on Chord (2001) — a scalable, fault-tolerant distributed hash table for peer-to-peer lookup at internet scale — directly anticipates the storage-and-coordination abstractions this problem needs, and his later systems (Spark, Mesos, Ray) extend exactly this lineage of building programmer-friendly abstractions over unreliable commodity clusters. His major production systems (Spark, 2010) postdate this 2003 problem by several years, so credit is for a real, technically precise lineage into the problem's exact domain rather than contemporaneous deployment. Even granting the era gap, Stoica's standing on spark and ray: the compute substrate of modern ml is close enough to this problem's actual demands that a graduate student would expect a real, defensible showing rather than a token one.
Fought here
36 figures are scored on this problem. Draw it in a battle to see where you land.