About Line of Best Fit
Points are pins on graph paper. The red string is the least-squares line: for each pin, take how far it sits from the line (its leftover, or residual), square it, and add them all up; this is the line that makes the total smallest. The page draws each squared leftover as a real square of tracing paper, so the line is the one that leaves the least paper on the board. The total is given in small squares of the graph paper.
Examples are along the top. NEW DRAW deals fresh random points for the same example; for the first two you can pick 20, 100 or 500 points. On the paper, click (or tap) an empty spot to pin a point, drag a pin to move it, or pick REMOVE PINS and click pins to take them out. Clicking a pin picks it out: the page gives its leverage and draws a dotted line showing the fit without it. With the paper focused, [ and ] pick the previous and next pin, the arrow keys move it (Shift for bigger steps), Delete removes it and Esc lets go.
TRY YOUR OWN LINE pins a green thread across the paper with two big pins to drag. The best line and its numbers are hidden until you press REVEAL, so you can see how close you got: the squares show your line's leftovers and its total area.
Distance chooses how a leftover is measured. VERTICAL (the usual one) predicts the up-and-down value from the across one. HORIZONTAL does it the other way round, and gives a different line. PERPENDICULAR measures straight to the line at a right angle: that is total least squares, the line along the long axis of the cloud. It depends on the scales of the two axes, so it uses them as drawn here. Least absolute line adds a violet dashed line that makes the total of the plain (unsquared) vertical distances smallest.
The strip underneath plots each pin's vertical leftover from the line against its across value, with a histogram of them at the end. Leftovers with no pattern mean a straight line suits the data; a hump or long runs mean it doesn't.
THE STORY
Least squares was first published by Adrien-Marie Legendre in 1805, in an appendix to a book on the orbits of comets. Carl Friedrich Gauss published his account in 1809 and said he had been using it since 1795; it was behind his famous prediction of where to find the dwarf planet Ceres, spotted by Giuseppe Piazzi at the start of 1801 and then lost in the Sun's glare. Using Gauss's orbit, astronomers found it again at the end of that year. Legendre was not pleased, and who got there first is still argued about.
Least absolute distances are older. In the 1750s Roger Boscovich, fitting a handful of measurements of the Earth's shape, chose the line with the smallest total distance (with the distances above and below balancing). It is harder to work out by hand, which is one reason squares won: squaring gives a formula. Squares also make a big miss count for a lot, which is why one stray point can swing a least-squares line so far.
The word regression comes from Francis Galton. In 1886 he compared the heights of more than 900 grown children with their parents' average (multiplying the women's heights by 1.08 first). Tall parents had tall children, but on average less far above the mean than the parents: about two thirds as far. He called it "regression towards mediocrity". It is not a force making people average; it happens whenever two things are linked but not perfectly, and it works backwards too. The HEIGHTS example is simulated to match his figures, not his real table.
The perpendicular line was described by Robert Adcock in 1878 and by Karl Pearson in 1901 ("On lines and planes of closest fit"), the idea behind what is now called principal component analysis. Pearson also gave the correlation coefficient r its usual formula in 1896, building on Galton. In 1973 Francis Anscombe made four small data sets with the same line and the same r but completely different pictures, to show why you should always look at the points and the leftovers, not just the numbers.
The app grew from the Python scripts scatter_plot.py (the NO LINK and STRONG LINK examples are its two pictures, made the same way), line_graph.py (the RANDOM WALK) and histogram.py (the histogram of leftovers).
HOW IT WORKS
For n points with averages x̄ and ȳ, the least-squares slope is b = Sxy ÷ Sxx, where Sxx adds up (x − x̄)² and Sxy adds up (x − x̄)(y − ȳ). The line always passes through the average point (x̄, ȳ), so the intercept is ȳ − b·x̄. The correlation is r = Sxy ÷ √(Sxx·Syy), between −1 and 1, and r² is the share of the spread in y that the line accounts for: the squares' total is (1 − r²) of what a flat line at ȳ would leave.
The horizontal line swaps the roles: its slope (drawn on the same axes) is Syy ÷ Sxy. The two lines agree only when r is ±1; the perpendicular line always lies between them. It runs along the direction where the points spread most, found from Sxx, Syy and Sxy.
The least absolute line has no formula. Some best line always passes through two of the points, and the best line through any one point has a weighted median of the slopes to the other points as its slope, so the page works that out for every point and keeps the line with the smallest total.
Leverage measures how much a pin's own height pulls the line at its own x: h = 1/n + (x − x̄)² ÷ Sxx. The leverages add up to 2, so the average is 2/n; pins far out to the side have the most. A pin with high leverage that also sits off the pattern moves the line a long way.
Every example except YOUR OWN uses made-up or random numbers. Most are dealt from a seed, so COPY LINK brings back the same points; a link carries pins you have moved only when there are 30 or fewer.
FURTHER WATCHING AND READING