Principal components

Principal component analysis (typically written PCA) finds the directions a cloud of data points actually spreads along, and then describes each data point in those terms. Those directions are the principal components, and in linear algebra terms they're the eigenvectors of the covariance matrix.

In the second lesson we described a data cloud as an ellipse. PCA takes that further and figures out which way the ellipse is pointing in your coordinate system.

Casting a shadow

Start with any straight line drawn through the cloud. Not the best one yet, just any one.

Take a single data point and think of that line as the ground, with the sun directly overhead. The point casts a shadow straight down onto the line. Where that shadow lands is the point's projection onto the line. In geometric terms you get to it by walking from the point to the line at right angles.

Now do it for every point at once. The whole cloud collapses onto the line and becomes a row of shadows, and each shadow needs only one number to place it: how far along the line it sits. Every point has gone from two numbers to one.

That collapse is the point of the exercise. A growth scan gives you four measurements, a blood panel gives you fifteen, a gene expression study gives you twenty thousand, and none of those can be plotted or held in your head at once. Projecting the data down onto one line, or onto a flat plane made from two, cuts it to something you can actually look at. The cost is that shadows throw information away: two points at different heights above the ground cast the same shadow, and from the shadow alone you cannot tell them apart. So the question is which line to use, because the choice decides what survives the collapse.

Two rules

Lesson 1 had this same problem. Infinitely many candidate lines, and a need for a rule that scores each one so a winner can be picked. The rule there was to add up the squared distances from the points to the line and keep the line with the smallest total.

That rule works here too, with one change. Lesson 1 measured each distance straight up and down. Here we measure it at right angles to the line, which is the direction the shadow travels. Call this rule A: score every line by the total squared perpendicular distance from the points to it, and keep the smallest. It is least squares with the ruler turned.

There is a second rule, and it looks like a completely different idea. Instead of measuring the distances down to the line, look at the shadows themselves. A line running along the length of the cloud gives shadows that fan out across the whole length. A line running across the cloud crushes them into a tight clump, and points that were far apart end up almost on top of each other. Fanned-out shadows keep the points distinguishable, so rule B says score every line by the variance of its shadows, meaning how spread out along the line they are, and keep the largest.

Rule A wants a minimum. Rule B wants a maximum, of a different quantity. Turn the line below and watch both numbers.

At every angle the two numbers move in opposite directions, and their sum never changes. On this cloud it is 2.50, whatever you do to the line.

A right triangle is why. Take one point and measure from the center of the cloud. The distance along the line and the distance across to it are the two legs of a right triangle whose hypotenuse is the straight-line distance from the center out to the point. That hypotenuse does not depend on the angle of the line at all, so by Pythagoras the two squared legs have to add up to the same thing however the line is turned. Add that across every point and the variance along the line plus the variance across it always equals the total variance of the cloud.

So the two scores are not independent quantities. They are two shares of a fixed budget, and making one of them large is the same act as making the other small. Turn the line all the way around scoring by rule A and note the winning angle, then do it again scoring by rule B, and the winning angle is the same number. On this cloud that number is about 34 degrees, where the shadows carry 2.16 of the total 2.50 and the distances across account for the remaining 0.34.

That winning angle is PC1, the first principal component. The direction at right angles to it is PC2, and it gets whatever variance PC1 left behind.

Three lines, three questions

Lesson 1 noted, beside its figure of squares, that the gaps there were measured straight up and down rather than at right angles, and that this was a choice rather than a fact. Rule A above makes the other choice. The two produce different lines through identical points.

There are three lines worth drawing, not two. Least squares of y on x minimizes vertical error, treating x as known and y as the thing being predicted. Least squares of x on y minimizes horizontal error, swapping those two roles. PC1 minimizes perpendicular error, treating both variables as measurements with nothing being predicted at all.

The error marks say which question each line answered: vertical combs, horizontal combs, perpendicular combs. Switch the families on and off to see them one at a time.

The slopes come out at 0.53 for y on x, 1.07 for x on y and 0.67 for PC1, so the two regressions differ from each other by a factor of two on the same eighty points. None of them is a true line that the other two are failing to reach. They are answers to three different questions.

Two things hold whatever the cloud looks like. All three lines pass through the mean, which lesson 1 showed for the least-squares line. And PC1 always lands between the two regressions. Raise the correlation and the three converge; lower it and they fan apart. A paper reporting the regression line between two noisy measurements has quietly picked one of the three, usually without saying which.

Using both directions

Everything so far has used a single line. PCA uses both of them.

PC1 and PC2 sit at right angles to each other, which is exactly the relationship the original x and y axes had, so they can serve as axes instead. Rewriting every point's coordinates in terms of PC1 and PC2 is a change of frame of reference and nothing more. No point moves relative to any other, no distance between points changes, and the total variance is untouched. The only thing that changes is which pair of directions the numbers are quoted in.

Slide from data coordinates into PC coordinates and watch the covariance matrix in the corner.

The off-diagonal entry falls to zero, and that zero is most of what PCA is for. Covariance measures whether two coordinates move together, so a covariance of zero means these two carry no straight-line information about each other. The tilt that lesson 2 spent a whole section on has gone, and nothing was done to the cloud to remove it. The tilt was never really a property of the data. It was a property of the axes the data happened to be measured against, and changing the frame of reference made it disappear.

The two numbers left on the diagonal are the variances along PC1 and PC2. Those are the eigenvalues of the original covariance matrix, and the two directions are its eigenvectors, which is where the linear algebra name in the first paragraph comes from.

Four variables

With two variables you can just plot the data, so PCA has not earned much yet. It earns its keep when there are more variables than a plot can hold, which is most of the time.

A routine growth scan measures four things: biparietal diameter, head circumference, abdominal circumference and femur length, all in millimeters. Four variables give a 4×4 covariance matrix and four principal components instead of two. The same logic runs: find the direction of greatest spread, then the direction of greatest spread among whatever is left, and so on. The bars below show how the total spread divides among the four, with the weights that say what each component is made of.

Read the weights on PC1 in raw millimeters. Abdominal circumference gets +0.75 and head circumference +0.62, while biparietal diameter gets +0.18 and femur length +0.15. Read as a sentence, PC1 is mostly abdomen and head.

Now press standardize each variable. The four weights become +0.50, +0.50, +0.50 and +0.50, which is four measurements counting equally.

Nothing about the fetuses changed between those two readings. AC and HC dominated the first one because they are the numerically biggest measurements: abdominal circumference has a variance of about 3,500 mm² where femur length has about 140. PCA maximizes variance, variance is in squared units, and so the variable carrying the largest numbers wins by default. This is lesson 2's units trap with consequences attached. There, changing the units changed a number nobody was going to interpret. Here, changing the units changes which variables the method says your data is about.

Standardizing first is the usual fix, and it's what a lot of software does without mentioning it. It isn't automatically the right call. It declares that a millimeter of femur matters as much as a millimeter of abdomen, which is a claim about biology rather than about arithmetic.

Why is PC1 so large?

The first component takes just under 98 percent of the variance in raw millimeters, and 97.5 percent standardized. A number that large normally reads as a success, but here it is a sign that one obvious thing is driving every measurement at once.

Every pair of these four measurements correlates above 0.95: BPD with HC at 0.97, HC with AC at 0.96, and so on around all six pairs. The reason is not subtle. These scans span 20 to 40 weeks, and everything about a fetus gets bigger across that range. Gestational age is driving all four measurements at once, so a single everything-is-large direction absorbs almost all the spread. PC1 here is gestational age wearing a disguise, and it took a 4×4 matrix to rediscover a number already printed at the top of the report.

So what about the 2 percent left over? In raw millimeters PC2 is head against abdomen, HC +0.75 against AC −0.65. That is a tempting story, because head-sparing asymmetric growth is a real phenomenon and it would look exactly like that. But press standardize and PC2 turns into abdomen against everything else, AC +0.78 with BPD −0.51. The same data, one analyst decision apart, tells two different stories about a component holding around one percent of the variance. Neither belongs in a manuscript.

Size for dates

None of that makes PCA the wrong tool here. It makes raw millimeters the wrong input. Gestational age is not a discovery, it is the one thing already known about every scan on the list, so it should come out before the method is asked what else is going on.

The recipe is short. Fit each measurement against gestational age, keep the residual, and standardize it. Each fetus is then described by how large every part is for its dates rather than by how far along it is, which is the same move a centile chart makes. Those four columns are what the figure below runs PCA on.

The correlations fall from about 0.96 to about 0.3, and the spread divides 49, 19, 17 and 16 percent instead of 98, 2, 0 and 0. PC1 survives as a size component, four positive weights all near 0.5, but it now means large-for-dates instead of late-in-gestation, which is the version a clinician would recognize.

The last three components are a trap of a different kind. They hold 19, 17 and 16 percent, close enough to call equal, and when eigenvalues are equal their directions are not determined: any rotation inside that space fits the data exactly as well. In this simulated dataset they are equal by construction, because each fetus was given a shared size factor plus independent noise of the same size on every measurement. The weights printed for PC2, PC3 and PC4 are therefore a portrait of the random seed. Real data never arrives with that fact attached, which is why deciding how many components are real is its own literature and not something the bar chart answers.

Recap

A projection is a shadow: every point dropped onto a line at right angles, two numbers becoming one. PCA picks which line by scoring every angle, and the two obvious ways of scoring turn out to be one, because the variance along a line and the distance across to it are two shares of a fixed budget. The winning direction is PC1 and the perpendicular one is PC2. Using both as axes is a change of frame of reference that leaves the data untouched while driving the covariance between the new coordinates to zero.

All of that is honest arithmetic, and none of it knows what the variables mean. It found gestational age because gestational age was the biggest thing in the room, and it named abdominal circumference as the lead variable because abdominal circumference had the biggest numbers. Hand it standardized residuals instead and it answers a different question, correctly, and still says nothing about which question was worth asking.

That has been the pattern for three lessons. Lesson 1 fit a straight line through a curve and reported it with the same confidence as any other fit. Lesson 2 produced a covariance that moved by a factor of ten thousand when the units changed. Lesson 3 rediscovered gestational age. The method never fails, and choosing what to ask it is the part that is not automatic.