About Teach a Neuron
How a machine learns from examples, built up from one neuron to a small network. A neuron here is a very small sum: multiply each input by a weight, add them up, and answer yes or no by the sign. Learning means changing the weights. The rack is drawn after the first machine built to do that, and every dial and cable on it is a real reading of the sums.
ONE NEURON: two kinds of dot on a plotting table. ClickTap to place a dot (again on a dot to remove it), or pick a set. STEP shows one example: the sum, the guess and, if it was wrong, the correction. RUN does as many passes as the setting says, at the speed you set. Beside it runs the theorem: how many corrections so far against the most it can ever need. XOR has no line, and the page shows why.
THE RETINA: a grid of 400 photocells and a bank of 400 dials, one for each. Teach it two letters, then draw on the retina, add speckle or move the letter, and see what it says. The two charts test it on fresh letters. ADD AS puts your own drawing into its lessons.
A SMALL NETWORK: two inputs, up to three hidden layers, one output. RUN trains until you pause it. The big picture shows the answer at every point of the square; each small square shows one neuron's own answer; each cable's thickness is its weight. Filled dots are the ones it learns from, ringed dots are held back as a test. Changing the shape, activation or seed fades the old result until you press RUN; the rate can change at any time.
INSIDE BACKPROP: a network of five neurons, small enough to show everything. STEP and BACK walk the forward pass, the loss, the slope for every line by the chain rule, and one update. The table sets each slope beside a numeric one, found by nudging that weight a little up and down and running the network again, with the gap between them.
THE STORY
In 1943 Warren McCulloch and Walter Pitts, in "A Logical Calculus of the Ideas Immanent in Nervous Activity", described idealised neurons that add up inputs against a threshold, and showed that networks of them can do logic. Nothing in their scheme learned. In The Organization of Behavior (1949) Donald Hebb proposed that a connection strengthens when one cell repeatedly helps to fire another: learning as a change in connections.
Frank Rosenblatt, at the Cornell Aeronautical Laboratory in Buffalo, described the perceptron in a 1957 report and in Psychological Review in 1958, first simulating it on an IBM 704. The rule is simple: when the answer is wrong, add the inputs to the weights or take them away. The Mark I Perceptron, built with US Navy funding and shown publicly in 1960, took its picture from 400 photocells in a 20 by 20 grid, with a plugboard for the wiring, and its weights were potentiometers turned by electric motors as it learned. It is now in the Smithsonian. The press ran ahead of it: the New York Times reported in 1958 that the Navy expected a machine that would one day walk, talk, see and write.
In 1962 Albert Novikoff proved the theorem on this page: if some line separates the examples, the rule makes at most (R/γ)² corrections, however the examples are ordered.
Marvin Minsky and Seymour Papert's Perceptrons (1969) is often said to have proved that neural networks cannot do XOR. That is not what it says. It is a careful mathematical study of a single layer of adjustable weights working on features that each look at only part of the picture. It proved such machines cannot compute things like parity (XOR is its two-input case) or whether a shape is connected without features that grow with the picture. It did not prove anything of the kind about networks with several adjustable layers. McCulloch and Pitts had already shown that fixed multi-layer networks can do any logic; what was missing was a way to train them. Minsky and Papert did write that they judged the extension to many layers likely to be sterile, and called that an intuitive judgement to be tested. Interest and funding fell away in the years after; how much of that was down to the book is still argued over. Rosenblatt died in a boating accident in 1971.
The training method came with backpropagation, made widely known by David Rumelhart, Geoffrey Hinton and Ronald Williams in Nature in 1986 ("Learning representations by back-propagating errors"). The idea had earlier roots: Seppo Linnainmaa's 1970 thesis on working derivatives backwards through a calculation, and Paul Werbos's 1974 thesis, which raised its use for neural networks.
HOW IT WORKS
The rule: sum = w₁x₁ + w₂x₂ + bias. A sum above 0 answers +, otherwise −. A sum of exactly 0 counts as wrong either way, so a blank machine has to learn. If an example is wrong, add its inputs (and 1 for the bias) to the weights if the answer was +, and take them away if it was −. Starting from zero, a rate would only scale everything, so there is none.
The bound: write each example as three numbers, (x₁, x₂, 1), and flip its sign if its answer is −. R is the length of the longest. γ is the margin: the best any set of weights can do at keeping all of them on the positive side, measured as a distance. It is 0 exactly when no line exists. This γ is measured with the bias included, so it isn't quite the gap you see on the table. The page works it out to about ten figures by finding the shortest vector in the convex hull of those points.
Why XOR can't work: for any weights, the sums at (0,1) and (1,0) add up to the same as the sums at (0,0) and (1,1). XOR wants the first two above 0 and the other two not, which is impossible. With a fixed order, once the weights end a pass where they ended an earlier one, everything will repeat, so the page stops and says so.
The retina here wires each photocell straight to a dial. The real machine had a layer of 512 randomly wired association units between them and 8 output units; those are left out. The letters are 10 by 10 drawings doubled up, so each square covers four cells. The lessons are the clean letter and 23 copies with 8% of cells flipped. Each chart tests fresh letters; the guessing line is at 50%. A neuron has one weight per cell and no idea that nearby cells are near, which is why moving a letter defeats it. MOVED TOO gives lessons up to 4 cells off centre: it does better within that range, and is still only adding up cells.
The network: hidden neurons use tanh or ReLU; the output is always a sigmoid, and the loss is cross-entropy. Training is plain gradient descent on batches of 10 dots, with no momentum, and the seed fixes the dots, the starting weights and the order. Each neuron has a bias too, which isn't drawn as a cable. The dots are made in the page; 70% are for learning, 30% are for the test. Two hidden neurons get stuck on XOR about half the time (the seeds tried in the tests: 50 in 100), and four solve it nearly always: more neurons give more ways down.
Inside backprop: the loss is half the squared gap, as in the 1986 paper, and that paper's hidden neurons were sigmoids; tanh is the default here because it learns faster. The examples are XOR's four in turn. The numeric slope uses a nudge of 0.00001 each way. It agrees to about ten places, except right at a ReLU's kink (a sum of exactly 0), where the slope isn't defined; the page says when that happens.
Everything is worked out in your browser. Your dots, drawings and training are never sent anywhere or kept. A copied link holds the settings, the set and the seed, and your own dots only up to 30 of them: beyond that the link leaves them out and says so.
FURTHER WATCHING AND READING