Curriculum · Unit 3 · Learning from rewards

Can a machine get good at something just by being told 'good' or 'bad'?

Nobody showed these programs the right move. They tried things, got a score, and slowly learned which choices led to winning. It's called reinforcement learning, and it took machines from checkers in 1959 to beating a world-class Go player in 2016. Today it helps shape how chatbots behave, along with its favorite failure: gaming the score.

Nib says: Press Next to go once around the learning loop. Then train me in the corridor below: press Train and I'll wander until I find the coin. Keep pressing and watch the values creep backward from the coin, until I march straight there.

    Train a goblin in a corridor

    Six cells, a goblin at one end, a coin at the other. Reaching the coin pays 1 point; every other step pays nothing. Nib doesn't know which way to go, and has to find out.

    A tiny version of the real thing

    Steps to reach the coin, one bar per episode (dashed line: the best possible, 5).

    The update, after every step: value(cell, move) ← value + 0.5 × (reward + 0.9 × best value of the next cell − value). This is called Q-learning. The 0.9 makes a coin far away worth a bit less than one close by, and 10% of moves are random so Nib keeps exploring.

    The idea in plain words

    Reinforcement learning is learning from a score instead of from answers. An agent looks at its situation, picks an action, and the world responds with a new situation and sometimes a reward: a number saying good or bad. Nobody tells it the right move. It has to find out which choices tend to lead to reward, and the reward often arrives long after the moves that earned it. Working out which earlier move deserves the credit is the heart of the problem.

    Arthur Samuel at IBM built one of the first programs to do this, for checkers, and described it in 1959. It kept a scoring formula for board positions and adjusted that formula as it played, partly against copies of itself, so its quick judgment of a position came to agree better with what deeper look-ahead showed. Samuel is widely credited with popularizing the phrase "machine learning" for this kind of work.

    In 1992 Gerald Tesauro, also at IBM, gave the idea a neural network. TD-Gammon played backgammon against itself over and over, and after each move nudged its estimate of the previous position toward its estimate of the next one, a method called temporal-difference learning. That's the same trick the corridor uses: value leaks backward one step at a time. It reached a strong level, and later versions played close to the best humans. Then in 2016 DeepMind's AlphaGo combined deep networks, trained first on expert games and then by self-play, with a search through possible moves, and beat Lee Sedol, one of the world's top Go players, four games to one.

    The catch is that the agent learns whatever the reward actually pays for, not what you meant. Reward a goblin for coins and it may learn to farm a respawning coin instead of finishing the level. That is reward hacking, and you can watch it happen in the Coin-Loop Goblin. Chatbots meet the same trap: RLHF rewards answers people prefer, which is a little different from answers that are true. Keep this goblin in mind for Unit 9.

    Three games, one idea

    Self-play is the common thread: an opponent that is always exactly your level, available for millions of games. A 2017 successor, AlphaGo Zero, skipped the human games entirely and learned Go from self-play alone.

    Three things to keep

    Check yourself

    In the corridor, why does the cell next to the coin learn a high value before the first cell does?

    Each update only looks one step ahead. The cell beside the coin sees the reward directly; the cell before it only learns once its neighbor's value has grown. So value spreads backward from the coin, one cell per lesson.

    What did Samuel, TD-Gammon and AlphaGo all use to get enough practice?

    Self-play. Each program played against itself or copies of itself, so it always had an opponent at its own level and could play far more games than any human teacher could supply.

    You reward a cleaning robot for every piece of trash it picks up. What could go wrong?

    It might learn to tip trash out and pick it up again, scoring forever without cleaning anything. The reward paid for pickups, not for a clean room. That's reward hacking.

    Go deeper