nlp
The gradient that vanishes
It is 1991 in Munich, and recurrent networks — the great hope for sequence learning — have a disease nobody has named: trained by gradients flowing backward through time, they learn dependencies a few steps long and nothing further, because the error signal shrinks geometrically with every step it travels. The diploma thesis on your desk proves it: the vanishing is not bad tuning, it is the architecture's mathematics. Diagnose the pathology rigorously, then design the cure — a memory cell whose error flows back undiminished, gated so the network learns what to store, what to forget, and when to speak. Get it wrong and sequence learning stays shackled to short contexts for a generation; get it right and speech, translation, and handwriting recognition all inherit the machinery.
Who this problem belongs to
The two figures whose methods fit it best, out of 53 in contention.
This problem is Hochreiter's own 1991 diploma thesis at the Technical University of Munich, supervised by Schmidhuber, which is exactly what this problem describes: a rigorous proof that recurrent networks trained by backpropagation through time suffer vanishing or exploding gradients as an inherent mathematical property of the architecture, not a tuning failure, because the error signal shrinks geometrically with each backward step. Hochreiter went on to co-invent, with Schmidhuber, the long short-term memory cell in 1997, a gated architecture whose constant error carousel lets gradients flow backward undiminished, exactly the cure this problem's closing challenge demands. He worked with the computational tools and theoretical framing available in early-1990s Munich, years before LSTM's eventual dominance in speech, translation, and handwriting recognition became apparent. This is not an application of Hochreiter's ideas; it is his actual thesis and his actual subsequent invention.
Schmidhuber supervised Hochreiter's 1991 diploma thesis diagnosing the vanishing gradient problem this problem describes, and the two co-invented the long short-term memory architecture in 1997, the gated memory cell whose constant error carousel is precisely this problem's demanded cure — a network that learns what to store, forget, and output through explicit gates rather than relying on an unmodified recurrent connection that inevitably loses gradient signal over many steps. His lab's decades-long insistence that recurrent networks were the correct architecture for general sequence learning, maintained through years of relative obscurity before deep learning's mainstream revival, was substantially vindicated once LSTM became foundational to speech, translation, and handwriting recognition systems. Schmidhuber worked in Munich on general-purpose sequence learning and credit assignment specifically. His relevance is direct co-authorship of both the diagnosis and the cure this problem describes.
Fought here
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
53 figures are scored on this problem. Draw it in a battle to see where you land.