Curriculum · Unit 2 · A neuron that learns

How can a pile of simple on/off units learn anything?

An artificial neuron is almost embarrassingly simple: add up some inputs, check the total against a line, say yes or no. In 1958 Frank Rosenblatt gave it a way to learn from its own mistakes. In 1969 a famous book showed the one thing a single layer can never learn.

Nib says: First press Next to see what one neuron does with its inputs. Then train a real perceptron below: press Step to watch one mistake get fixed, or Run epoch to go through all four examples. Try AND, then OR, then XOR, and see which one it can't crack.

    Train a perceptron

    Four points, two inputs each. The perceptron draws one straight line and says 1 on the shaded side, 0 on the other. Every time it gets a point wrong, it nudges its weights.

    Learn:
    x₁x₂targetsays

    The rule is the modern textbook form of Rosenblatt's: w ← w + lr · (target − output) · x, and the bias moves by lr · (target − output). Here lr (the learning rate) is 0.25, and the starting weights are made up.

    The idea in plain words

    In 1943 a neuroscientist and a young logician turned a brain cell into arithmetic. Warren McCulloch and Walter Pitts described a neuron as a tiny logic device: signals come in, they are added up, and if the total clears a threshold the neuron fires a 1; otherwise it stays at 0. Wire a few together with the right thresholds and you get AND, OR and NOT, the same building blocks Shannon found in switches. Their neurons were fixed, though. Someone had to set every connection by hand.

    Frank Rosenblatt's perceptron, published in 1958, added the missing piece: learning. Each input gets a weight, a number saying how much it counts. Show the perceptron an example; if it answers correctly, do nothing. If it's wrong, nudge each weight a little in the direction that would have made the answer right. Repeat. If any straight line can separate the yes-examples from the no-examples, this simple rule is guaranteed to find one eventually. Rosenblatt also built it as a machine that learned to tell simple shapes apart, and the newspapers got very excited.

    Then came the catch. Look at XOR, "one or the other but not both": the points (0,1) and (1,0) should say 1, while (0,0) and (1,1) should say 0. They sit on opposite corners of a square, and no single straight line can put the right pairs on each side. In their 1969 book Perceptrons, Marvin Minsky and Seymour Papert worked out precisely what single-layer perceptrons can and can't do, and XOR is the simplest example of the "can't". Interest and funding for neural networks dropped through the 1970s, and the book is often blamed for part of that slump.

    The fix was already known in principle: stack neurons in layers. One hidden neuron can learn OR, another can learn "not both", and a final neuron combines them into XOR. What nobody had was a good way to train the hidden layers, to decide which inner weight deserved the blame for a mistake. That answer, backpropagation, is Unit 5. Every modern model, including the transformers behind today's chatbots, is still built from billions of these weighted sums.

    One line can't separate XOR's corners, but two can: the 1s sit in the band between them. In network terms, one hidden neuron computes OR, another computes "not both", and an output neuron ANDs them together. Training those hidden neurons is the problem backpropagation solved.

    Three things to keep

    Check yourself

    What did Rosenblatt's perceptron add to the McCulloch–Pitts neuron?

    A way to learn. McCulloch and Pitts's neurons had connections fixed by hand; the perceptron adjusts its weights automatically, a little after every wrong answer.

    The perceptron outputs 1 on an example whose target is 0, and the first input is 1. What happens to the first weight?

    It goes down. The error is target − output = 0 − 1 = −1, so the weight changes by lr × (−1) × 1. Lowering it makes the sum smaller, which pushes the answer toward 0 next time.

    Why can't a single perceptron learn XOR, no matter how long it trains?

    It can only draw one straight line, and XOR's 1s, at (0,1) and (1,0), sit on opposite corners from its 0s, at (0,0) and (1,1). No line separates them. Adding a hidden layer fixes it, which is where Unit 5 picks up.

    Go deeper