Covariance

Covariance is a single number that says whether two measurements move together: when x comes in above its average, does y tend to come in above its average too? Variance is the same idea for one variable on its own, describing how far it spreads around its average. Hold both in one object and you have the covariance matrix, also called the variance-covariance matrix and usually written Σ.

That pair is where lesson 1 left off. The least-squares slope turned out to be a covariance divided by a variance, so between the two of them they do all the work in probably the most-used formula in statistics.

What covariance measures

Take one data point. Measure how far it sits from the mean in x, and how far from the mean in y. Those two distances are its deviations. Multiply them together. Do that for every point and average the results. That average is the covariance.

The multiplying is what produces the sign. A point above average in both x and y contributes positive times positive, so a positive product. A point below average in both contributes negative times negative, also positive. But a point that is above average in x and below average in y contributes a negative product. So when the cloud tilts up to the right the positive products dominate and the covariance comes out positive; when it tilts down, negative; and when there is no tilt at all the two kinds of product roughly cancel and the covariance sits near zero.

Variance is the same calculation with y replaced by x: the average of a deviation times itself, which can never be negative. That’s why variance and covariance belong in one object. For two variables there are four combinations, x with x, x with y, y with x, and y with y, and stacking them in a 2×2 grid gives the covariance matrix. The variances sit on the diagonal and the covariance fills both off-diagonal slots, identical in each, because multiplying two deviations does not care about the order. The matrix is always symmetric.

Why an ellipse?

Those three distinct numbers, two variances and one covariance, pin down the shape of the cloud. They do not say where any individual point sits, only the overall extent: how wide, how tall, and how tilted. A shape with one width, one height and one tilt, symmetric about its center, is an ellipse. It takes three numbers to draw one and the matrix holds exactly three, so the ellipse is not a summary of the matrix but the same information drawn rather than listed.

The one below is drawn at two standard deviations (the square root of the variance, which puts it back in the original units), so most of the cloud falls inside it. Its long axis points along the direction in which the data varies most, its short axis sits at right angles to that, and the length of each axis is set by how much the data varies along it. Those two directions come out of the matrix itself, and finding them is exactly what lesson 3 does.

Below are eighty synthetic points, the ellipse, and the matrix. Grab a square handle to stretch the cloud, the diamond to tilt it, or drag the underlined matrix numbers up and down. The ρ slider sets the correlation directly. Worth knowing: these points are constructed so that the sample matches the dial exactly. Lesson 1 was where you felt sampling wobble, so this page switches it off and lets the geometry hold still.

Reading the four numbers: top-left is the variance of x, so stretching the cloud sideways makes it grow while the variance of y, at bottom-right, sits still. The covariance in the two off-diagonal slots moves whenever the tilt or either spread changes.

Now drag the tilt handle and watch all four entries move together. Tilt is not stored in any one cell. It only exists in the relationship between the covariance and the two variances, which is why reading a covariance matrix one cell at a time misses it entirely.

The units trap

A covariance changes value when you change the units, which matters as soon as the two variables are measured in different things. The next figure holds the same cloud, literally the same points, so dragging above moves it down here too. The only change is that x is now a length in millimeters and y a weight in grams. The buttons switch units, and that’s all they do.

Switch x to centimeters and every entry involving x drops by a factor of ten. Switch y to kilograms and its entries fall by a thousand. Do both and the covariance shrinks ten-thousand-fold, while not one data point has moved.

The picture squashes too, and that is deliberate. This plot gives one numeric unit of x the same number of pixels as one numeric unit of y, which is the only way a plot can show you the numbers honestly. Most software instead rescales each axis to fill the frame, which quietly hides the whole effect.

Covariance carries its units with it, here millimeter-grams, so changing the units has to change the number, the same way quoting a height in feet instead of meters changes the number without making anyone taller. The practical consequence is the important part: the size of a covariance means nothing on its own. You cannot look at 12,500 and say whether that is a strong relationship or a weak one.

What survives the unit change is r, the correlation coefficient, written out in full as Pearson’s r. It is the covariance divided by both standard deviations:

$$r=\frac{S_{xy}}{s_x·s_y}$$

The standard deviation of x carries the units of x, and the standard deviation of y carries the units of y, so dividing by both cancels everything and leaves a pure number. Dividing by the spreads also bounds it: r can only run from −1 to +1, where ±1 means the points sit exactly on a straight line and 0 means no straight-line relationship at all. That’s why r is the number to quote when you want “how tight is this” to mean the same thing in millimeters as in centimeters.

Worth being clear about what r does not do. It measures straight-line association only, so a strong curved relationship can produce a small r, and it says nothing about the slope. A steep relationship and a shallow one can both have r = 0.9.

Head circumference and fetal weight

Here is a pair in the units it actually arrives in: head circumference in millimeters against estimated fetal weight in grams. These 350 scans are simulated from the published INTERGROWTH-21st centiles (Papageorghiou 2014) rather than taken from patients. The figure says so on its face, and the data folder carries the full recipe. Millimeters against grams is not a contrived example here, it’s what an ultrasound report looks like. The covariance comes out in the tens of thousands of millimeter-grams, a quantity nobody has intuition for. The correlation is 0.93.

That 0.93 needs an asterisk, and it is the kind worth internalizing. Estimated fetal weight isn’t measured. It’s computed: the machine takes head circumference, abdominal circumference and femur length, puts them into a regression equation (usually Hadlock’s, from 1985), and reports the result. So when HC and EFW correlate at 0.93, part of that is fetal biology and part of it is arithmetic, because HC is one of the inputs to EFW. In this simulated dataset that’s literally how the EFW column was produced.

A variable computed from another will always correlate with it, and the correlation tells you about the formula as much as about the patient. Before reading anything into a correlation, it’s worth checking whether one of the two variables is partly made out of the other.

Three things are worth carrying into lesson 3: a covariance matrix is a shape, its numbers depend on the units so their size means nothing by itself, and r is what is left once the units are divided out. Lesson 3 goes looking for the directions a cloud genuinely spreads along, and the first decision it has to make is which units to look in.