Curriculum · Unit 5 · Backprop and deep learning

How does a deep network work out which of its millions of weights to blame?

When a network gets an answer wrong, every one of its weights played some part. Backpropagation is the bookkeeping trick that hands each weight its exact share of the blame, in one sweep backward from the mistake. It's the engine under every model you've heard of, and it's small enough to watch by hand.

Nib says: Here's a network with just five neurons. Press Next to push one example through it, measure the mistake, send the blame backward and fix the weights, with every number worked out live on this page. Then scroll down and train it on XOR yourself with the +1, +10 and +100 buttons. The starting weights are ones I made up, so everything here is illustrative.

    Made-up starting weights, sigmoid neurons, loss = ½(output − target)², learning rate 2. All numbers are computed by this page and rounded for display. Illustrative.

    Try itTrain it on XOR

    XOR means "one or the other, but not both". A single neuron can't learn it. This one has a hidden layer, so let's see if backprop can find the trick. Each step looks at all four rows and nudges every weight a little.

    2 inputs → 2 hidden → 1 output
    ✓ when the output rounds to the right answer
    x₁x₂h₁h₂outputwant

    Same starting weights as above. Illustrative.

    The idea in plain words

    Training is a blame game. The network makes a guess, you measure how wrong it was with a single number called the loss, and then every weight needs to know: if I nudged myself up a little, would the loss go up or down, and by how much? That number is the weight's gradient. Once every weight has one, you move each a small step in the direction that lowers the loss, and repeat, many times.

    For a single neuron, like the one in Unit 2, the blame is easy to share out. The trouble starts with hidden layers. A hidden neuron has no right answer of its own, so how can it know whether it helped or hurt? In 1986 David Rumelhart, Geoffrey Hinton and Ronald Williams showed a clear way through, using the chain rule from calculus. Start at the output, where the error is known. Pass it backward through each connection, multiplying by the weight and by how sensitive each neuron was, and every hidden neuron receives its share. Others, including Paul Werbos in the 1970s, had worked out the underlying math earlier. The 1986 paper showed that it makes hidden layers learn useful features of their own.

    XOR is the classic test. One neuron can only draw one straight line between "yes" and "no", and XOR can't be split by one line. With two hidden neurons, the network can draw two lines and combine them. Train the toy above and look at the h₁ and h₂ columns: one hidden neuron usually ends up acting like "x₁ OR x₂", the other like "x₁ AND x₂", and the output learns "OR, but not AND". Nobody told it to. Backprop found it.

    Then it went to work. In 1989 Yann LeCun and colleagues used backprop to train a convolutional network, which reuses the same small feature detectors across an image, to read handwritten ZIP code digits from real mail. In 1997 Sepp Hochreiter and Jürgen Schmidhuber's LSTM added gated memory cells, so a network could carry information along a sequence of words or sounds without losing it. But for years networks stayed fairly small, because data and computing power were scarce.

    Two things changed that. ImageNet (2009) supplied millions of labeled photos, and GPUs, built for video games, turned out to be very good at a network's arithmetic. In 2012 AlexNet, a deep convolutional net trained on two gaming GPUs, won the ImageNet challenge by a huge margin, and the deep-learning boom began. Today's language models still learn by backprop, during pretraining and again during fine-tuning. There are just billions of weights instead of nine, and a great deal more compute.

    From one paper to a boom

    1. 1986

      Backprop

      Rumelhart, Hinton and Williams show that sending the error backward lets hidden layers learn.

    2. 1989

      Reading the mail

      LeCun's team trains a convolutional network on handwritten ZIP code digits from US mail.

    3. 2009

      ImageNet

      Deng, Fei-Fei Li and colleagues publish a huge database of labeled images to learn from.

    4. 2012

      AlexNet

      A deep convolutional net trained on GPUs wins the ImageNet challenge: about 15% top-5 error against about 26% for the runner-up.

    Along the way, in 1997, the LSTM gave networks a working memory for sequences. It powered speech recognition and translation until the transformer arrived in Unit 6.

    Three things to keep

    Check yourself

    A weight's gradient is negative. Should training make that weight bigger or smaller?

    Bigger. A negative gradient means that raising the weight would lower the loss, and the update rule is new = old − learning rate × gradient, so subtracting a negative number raises it. You can see this on the output weights in step 5.

    In step 4, why do the two weights leaving x₂ get no blame at all?

    Because x₂ was 0 in that example. A weight's gradient is its neuron's share of the blame times the input flowing through it, and anything times 0 is 0. Those weights didn't affect this answer, so they aren't changed by it. Other rows of XOR will move them.

    The backprop algorithm was published in 1986. Why did deep learning only take off around 2012?

    Networks need lots of examples and lots of arithmetic. Large labeled datasets like ImageNet and fast parallel hardware like GPUs arrived later, and AlexNet showed what happened when you put them together.

    Go deeper

    Read

    Watch

    Explore