systems
The cluster that iterates
It is 2010 at Berkeley, and machine-learning researchers are abusing the reigning distributed framework in a way its designers never intended: their algorithms are iterative — gradient steps, graph passes, alternating updates — and the framework writes the entire working set to disk between every iteration, making a hundred-pass computation a hundred times slower than it needs to be. Design the compute substrate for iterative analytics at cluster scale: keep working data in memory across passes, recover from machine failures without replicating everything, and expose an abstraction ordinary scientists can program without becoming distributed-systems experts. Get it wrong and large-scale ML remains the private craft of two companies' infrastructure teams — the analysis a lab can afford being decided by disk latency, not by ideas.
Who this problem belongs to
The two figures whose methods fit it best, out of 42 in contention.
This is Stoica's own desk, not a hypothetical. He was a Berkeley professor when Matei Zaharia, his student, built Spark in 2009-2010 specifically to fix the disk-thrashing problem this scenario describes: MapReduce's insistence on writing every intermediate result back to HDFS, which murdered iterative machine-learning workloads. Stoica co-designed the resilient distributed dataset abstraction that let a cluster keep working sets in memory across passes while still recovering from node failure by replaying a lineage graph instead of replicating data. He went on to found the AMPLab and later co-create Ray, purpose-built for exactly the iterative, fine-grained ML workloads this problem names. There is no daylight between the man and the machine here; he is the answer, not an analogy to it. Score: maximal, deservedly.
Dean did not build Spark, but he built the paradigm this problem is reacting against and then, at Google, the ones that came after it. MapReduce, which he co-designed with Sanjay Ghemawat in 2004, is the very disk-bound batch framework the scenario names as inadequate for iterative ML; Dean understood its limits from the inside and moved past them with Bigtable, and later led the systems work behind TensorFlow, which manages distributed, iterative, fault-tolerant computation graphs for machine learning at a scale Berkeley's 2010 cluster never approached. His instincts for fault tolerance without full replication, and for abstractions ordinary researchers can program against, transfer almost completely. He loses a few points only because his solutions were built inside one company's walls, not for the open, academic usability this problem also demands.
42 figures are scored on this problem. Draw it in a battle to see where you land.