Least squares

Least squares is a method of fitting the best line to a set of data points. We need a rule (a statistical method) to compare each candidate line and rank one as “best.” Least squares is probably the most common one out there, so common that it gets applied without even being named as least squares. In statistical contexts it’s also called ordinary least squares, or OLS.

This page builds it up in pieces and ends by running the whole thing on 400 real births, where the method works exactly as designed and still gives a misleading answer.

Why squares?

The most obvious option would be to just take the difference from each data point to the candidate line (the residual) and add them up. This quickly falls apart: a residual of +10 would immediately cancel the next residual of −10, so even a very bad line that balances its badness could have a low score.

The next obvious options are using absolute values, or just taking the square of the residual, and minimizing that. Squaring is what gives us least squares, and the figure below draws it literally. Each residual becomes a square whose side is the gap, so the score is a total area. The dot screen inside each square keeps the ink proportional to that area, and the corner keeps the running total. Drag the line to slide it, the round handles to tilt it, and the points themselves if you want to make trouble.

Try to get the total down. It starts at 29.5, and the best these twelve points allow is about 3.7. You’ll find yourself going after the big squares first and tolerating several small ones, which is exactly what the rule rewards: doubling a gap quadruples its square, so one badly missed point costs more than several slightly missed ones. Least squares is the line that makes that total area as small as possible, and that is the entire definition.

One detail in that picture matters later. Each gap is measured straight up and down, from the point to the line, not at right angles to the line itself. That is a choice, and it follows from treating x as known and y as the thing being predicted. Lesson 3 measures it the other way and gets a different line.

This is also the whole content of the complaint that least squares is sensitive to outliers. There is no bug to fix. We chose to measure error as an area, and areas grow fast.

Every line as a point

So far we’ve searched by hand, one line at a time. There’s a much better way to see the whole search at once.

A line is fully described by two numbers, slope and intercept, and any pair of numbers is a point on a plane. So every possible line corresponds to one point on a plane whose axes are slope and intercept. Each of those lines has a score, the total squared error we just defined. Attach that score to its point as a height, and the plane becomes a landscape. The figure below is that landscape seen from directly above, with ink standing in for height.

The two figures are linked. Drag the line in the first one and the marker moves here; drag the marker and the line moves there. The dots grow fat where the lines are bad and thin out to bare paper near the best one, and the rings are contour lines, exactly like a topographic map. The shape they outline is a bowl: a single lowest point, no secondary dips, curving smoothly everywhere.

That shape is why least squares is easy to compute. If the landscape had many local dips, finding the lowest would be a real search problem, and you would never be sure you had finished. Because it’s a simple bowl, you can start anywhere, walk downhill, and always arrive at the same place. For these twelve points that place is near slope 0.7 and intercept 0.4, scoring the 3.7 you may already have reached by hand.

Press absolute error on the first figure to see the alternative. The squares become sticks, since each gap now counts at face value instead of being squared, and the bowl develops a sharp crease along its floor. That puts the tradeoff on screen. Squaring buys a smooth bowl, which calculus can solve in a single step, and the same squaring is what makes outliers so influential. Absolute error is much less sensitive to outliers and gives up the smoothness, which makes it harder to optimize.

Legendre published the squared version in 1805, Gauss claimed he had been using it since 1795, and the priority fight between them was bitter. Two centuries later it is the default in every statistics package.

Solving it

Because the bowl is smooth and has one bottom, calculus can find that bottom directly: set both partial derivatives to zero and solve. For slope $\hat{m}$ and intercept $\hat{b}$:

$$\hat{m}=\frac{S_{xy}}{S_{xx}}\hat{b}=\overline{y}-\hat{m}\overline{x}$$

Both pieces have a plain reading. The slope is the covariance of x and y ($S_{xy}$) divided by the variance of x ($S_{xx}$): how much the two vary together, scaled by how much x varies on its own. The intercept then simply positions the line so that it passes through ($\overline{x}$, $\overline{y}$), the center of mass of the cloud. Every least-squares line goes through that point, whatever the data.

So the whole method reduces to two quantities: covariance and variance. That is why the next lesson is about nothing else.

How much does the fit wobble?

The twelve points are synthetic. I generated them by taking a line I chose and adding random noise, which means the true line exists and can be drawn. That almost never happens with real data.

Press show the true line, then resample several times. The true line stays put while the fitted line shifts around it, because each resample draws fresh noise and the fit responds to that noise. The size of that movement is what standard errors and confidence intervals exist to describe. Here you can just watch it directly.

400 real births

Now the same instrument on 400 births sampled from the 2014 US natality file, plotting gestational age against birthweight. The show the true line button is gone, because there is no known true line. All we have is the method and the definition of best we gave it.

The fitted slope is about 125 g per week, putting roughly half a kilo between a 36-weeker and a 40-weeker. If that seems low against the third-trimester growth velocities we quote at the bedside, it should. Those come from following the same fetus over time. This is a cross-section of different babies, each measured once, at whatever gestational age they happened to deliver at. Babies delivering at 30 weeks are not a random sample of babies, and many of them delivered early precisely because something was wrong. The regression is a correct answer to the question we typed in, which was not the question we cared about.

Two things to try. Drag one of the extreme preterm points around and watch how little the line moves: 400 other points are holding it in place, so a single outlier has less leverage than you might expect in a sample this size. Then look at the left edge, below about 32 weeks, where the cloud curves away from any straight line. Growth is not linear across this range. The method has no way to notice that, and no part of its output will say so. It fits a straight line through a curve and reports the same kind of number with the same apparent confidence.

That is least squares. It answers the question you set, and it does not tell you whether that was the right question. Judging the question is a separate skill, and it is what the next two lessons are for.