nlp
The sentence in a single vector
It is 2014, and neural networks have conquered images while translation remains the stronghold of phrase-table engineering — a decade of accumulated pipeline, feature functions, and folklore. The proposal is radical simplification: one network reads the source sentence into a vector, another writes the translation out of it, trained end to end on sentence pairs with no alignment supervision at all. Make it work at scale, then find where the architecture chokes: the entire sentence squeezed through one fixed-size vector, long sentences degrading first — and design the mechanism that lets the decoder look back at the source as it writes. Get it wrong and neural MT is dismissed as a toy; get it right and the pipeline era of NLP has three years to live.
Who this problem belongs to
The two figures whose methods fit it best, out of 53 in contention.
This problem is Sutskever's own 2014 paper, 'Sequence to Sequence Learning with Neural Networks,' written with Vinyals and Le at Google. It proposed exactly this architecture: an LSTM encoder reads a source sentence into a single fixed-size vector, a second LSTM decodes a translation from that vector, trained end to end on sentence pairs with no alignment supervision. Sutskever's group found and documented the bottleneck this problem asks for: performance degraded sharply on long sentences because the entire meaning had to survive compression into one vector, a limitation the same broader research community resolved within a year via attention. He worked with the deep learning tools and GPU compute available in 2014, not earlier phrase-based pipelines. This is not analogy or transferred skill; it is the actual paper this problem describes.
Bengio, with Cho and Bahdanau, published the 2014-2015 papers that both demonstrated the encoder-decoder bottleneck this problem names and solved it: 'Neural Machine Translation by Jointly Learning to Align and Translate' introduced attention, letting the decoder look back at all source-sentence hidden states rather than relying on a single fixed vector. His earlier 2003 work on neural language models had already established that learned distributed representations could replace hand-built statistical features, the philosophical predicate for abandoning phrase-table engineering entirely. Bengio worked with the RNN and gated-unit architectures available in 2014, developing GRUs as part of this exact line of research. His contribution is not adjacent to this problem — the attention mechanism it asks for, letting the decoder escape the bottleneck, is substantially his and his students' invention.
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
53 figures are scored on this problem. Draw it in a battle to see where you land.