systems
Train across a thousand GPUs
It is 2021, and the scaling laws have made their promise legible: more parameters, more data, more compute yield capabilities nothing smaller exhibits. The catch is engineering. Distribute the training of a hundred-billion-parameter model across a datacenter: the model does not fit on any single device, so it must be sliced across data, tensor, and pipeline dimensions; gradients must synchronize within a communication budget that the interconnect barely affords; and at thousand-GPU scale, hardware failure during a months-long run is a certainty, so checkpointing and recovery are load-bearing, not hygiene. A single training run costs millions of dollars and months of calendar. A bad parallelism strategy or a silent numerical fault doesn't just slow the run — it incinerates the budget and the year.
Who this problem belongs to
The two figures whose methods fit it best, out of 35 in contention.
This is Dean's actual career arc. He co-designed MapReduce and Bigtable for datacenter-scale fault-tolerant computation, then at Google Brain co-led the DistBelief and TensorFlow systems that pioneered data- and model-parallel training of neural networks across thousands of machines, directly anticipating the tensor, pipeline, and data-parallel slicing this problem demands. His 2010s papers on large-scale distributed deep learning explicitly confront synchronization budgets, straggler mitigation, and fault tolerance during long-running jobs -- exactly the checkpointing-as-load-bearing-not-hygiene stakes this problem names. He has spent two decades building the actual infrastructure this problem describes, at the actual scale, for the actual purpose, making him about as direct a match as this pool contains.
Stoica co-created Spark and then Ray, the latter explicitly built as a distributed-systems substrate for machine learning workloads: elastic scheduling across heterogeneous clusters, fault-tolerant task graphs, and the actor model used by modern large-model training and serving frameworks (Ray is the backbone under several LLM-training stacks). His Berkeley lab (RISELab, and its ML-systems focus) is a direct historical ancestor of the model-parallel, pipeline-parallel tooling this problem needs, and his career has run at exactly the datacenter scale and cost stakes described. The deduction is narrow: Ray and Spark are general distributed-compute substrates that others layer 3D-parallelism strategies onto, so credit for the specific parallelism schemes is shared rather than solely his.
35 figures are scored on this problem. Draw it in a battle to see where you land.