rl
The room with no reward
It is 2018, and one Atari game has become the field's public humiliation: Montezuma's Revenge, where the first reward sits beyond ladders, ropes, a key, and a locked door — hundreds of correct actions with zero feedback — and the agents that conquered every other game score nothing. Random exploration will not stumble through; the agent needs reasons to act before the world supplies any. Build intrinsic motivation that is not a bug generator: curiosity about the unpredicted, novelty bonuses over learned state abstractions — and confront the noisy-TV trap, where pure randomness is infinitely 'novel' and the curious agent stares at static forever. Get it wrong and RL remains a method for dense-reward games only, useless wherever feedback is rare and delayed — which is everywhere that matters.
Who this problem belongs to
The two figures whose methods fit it best, out of 30 in contention.
Schmidhuber invented formal artificial curiosity in 1990-91, proposing that an agent should generate its own intrinsic reward from how much a learned world-model's predictions improve, exactly the machinery later used to attack Montezuma's Revenge. His 2010 'formal theory of fun' explicitly frames novelty and compressibility gains as reward, anticipating both the curiosity-bonus approach and the noisy-TV failure mode this problem names, an agent addicted to unpredictable but uninformative stimuli is precisely the pathology his own compression-progress framework was designed to avoid by rewarding learning progress rather than raw surprise. Few researchers in this pool have a more direct or longer-running claim to the specific mechanism, self-generated intrinsic reward for sparse-feedback environments, this problem requires, and he has spent three decades insisting, sometimes loudly, that he did it first.
Silver led DeepMind's reinforcement-learning research through the years Montezuma's Revenge became the field's public embarrassment, and his group's Atari results, both the successes and the well-documented zero-score failures on sparse-reward games, directly motivated the intrinsic-motivation research this problem describes. His broader instinct, that learned value estimation must be paired with structured exploration to work at all, is central: dense-reward Atari games fell to his methods while sparse-reward ones like this one did not, exposing exactly the gap the problem asks to be closed. His own published work leans more on exploration bonuses derived from uncertainty in value estimates than on curiosity modules built from prediction error, a real but narrower slice of the intrinsic-motivation literature than Schmidhuber's or Pathak's.
Fought here
In the mind map
The same ideas, as concepts rather than history — in John's ML knowledge map.
30 figures are scored on this problem. Draw it in a battle to see where you land.