Using the QR Decomposition to orthogonalize data

This is going to be a very short post, illustrating one idea with one example (yes, one, not five).

It turns out that there is another way to have Mathematica® orthogonalize a matrix: it’s called the QR decomposition. The matrix Q will contain the orthogonalized data… and the matrix R will specify the relationship between the original data and the orthogonalized.

That means we do not have to do the laborious computations described in this post. Understand, if we do not care about the relationship between the original data and the orthogonalized data, then I see no advantage in Mathematica to using the QR over using the Orthogonalize command.

(R will be upper triangular instead of lower triangular. Irrelevant. Far more important is that the R matrix handles the constant term, so we have a linear relationship rather than an affine one. I could have made the original linear by including the constant term, but I had chosen not to. Now it doesn’t matter.)

On the other hand, you might have a QR decomposition available to you, but not an Orthogonalize command. You’re not required to do anything with the R matrix if you don’t want to.

Let me show it to you, for the Hald data.

I set the path to the Hald data… and read it all in.

Having gotten independent variables identified as xi, and the dependent variable y, let me assign the obvious but utterly uninformative names to the independent variables:

Here’s what we have:

We’ve learned, however, that if we want to eliminate multicollinearity, then we need to get the design matrix – the independent variables plus the constant column – rather than just the independent variables. We have seen that if we can orthogonalize everything, then the resulting new data is completely free from multicollinearity.

Let me get the design matrix from a regression using all the data.

I’ll remind us that the quickest and most accurate assessment of multicollinearity is the variance inflation factors (VIF) – if they’re available to you – although I prefer to convert them to R^2 values, and I have a function to do so:

The relationship is simply

$R^2 = 1 - \frac{1}{\text{VIF}}\$,

which comes directly from the definition of the VIF:

$\text{VIF} = \frac{1}{1-R^2}\$.

(Perhaps I should remind you that the four R^2 in that list are: X1 fitted as a function of X2, X3, X4; X2 as a function of X1, X3, X4; X3 as a function of X1, X2, X4; and X4 as a function of X1, X2, X3.)

We see, as we expect from the Hald data, that all four variables are strongly related. (You may recall that their sum is almost exactly 100 – they were percentages, and there were additional variables which consistently added up to about 1.5 .)

Let me ask Mathematica for a QR decomposition. (It turns out that Mathematica returns the transpose Q’.) Note that I will use uppercase when I talk about the decomposition, but lower case for the actual numerical matrices. Here is the call, with output called qt and r… so I also set q = qt’, the transpose of the transpose. The following two matrices are Q and R:

You see that Q (the transpose of the answer supplied) is the same shape as X. Let me confirm that Q is column-orthonormal – i.e. that Q’Q = I.

Let me confirm that X = QR. Here’s the product…

… and is that the same as X? Yes:

So, we have a valid QR decomposition: one column-othonormal matrix Q and one upper triangular matrix R.

Fine, but how is Q related to the output of an Orthogonalize command?

The orthogonalize command works on rows, so I need to transpose the design matrix X first… Orthogonalize… then transpose again:

Recall Q:

Many – OK, all but one! – of the columns have different signs, but that’s OK: if vectors a and b are orthogonal, then so are -a and b, etc. Perhaps the fastest way to check that – apart from the signs of entire columns – that we have the same orthogonalized data – is to post-multiply q by a diagonal matrix to change the signs on all columns but #2:

Here’s the product… is it the same as Z?

Yes. We have gotten effectively the same orthogonalized data,

So: if there’s any chance that you will want the relationship between the original data and the orthogonalized data, I strongly recommend the QR decomposition.