## Regression 1 – Isolating & Identifying Linear Dependence

I almost named this post “Nailing Linear Dependence”. It’s so easy….

## Introduction

This post will show just how easy it is to isolate and identify linear dependence. Looking at subsets, as we did in this previous post, works – but what I’m about to show you is easier and faster. (Henceforth I will refer to that link as “the previous post” or once as “that post”.)

On the other hand, we will see a case where looking at subsets gives an equivalent answer that might be more appealing.

Now, I’m going to solve the same five examples as in the previous post. I am not going to duplicate the introductory material in the previous post, so if you need more context, please read that post. You might or might not want to spend much time looking at the details of examining subsets of the columns of X – that examination is what I’m about to replace by something more incisive.

As usual, I will use either XT or X’ to denote the transpose of X; and v’ or vt to denote the transpose of v.

## Example 1

Here’s a matrix.

$X = \left(\begin{array}{ccccc} -74 & 80 & 18 & -56 & -112 \\ 14 & -69 & 21 & 52 & 104 \\ 66 & -72 & -5 & 764 & 1528 \\ -12 & 66 & -30 & 4096 & 8192 \\ 3 & 8 & -7 & -13276 & -26552 \\ 4 & -12 & 4 & 8421 & 16842\end{array}\right)$

Detection.
We could, and did, look just at its singular values…

SingularValueList[X]//N = {36368.4, 170.701, 60.5332, 7.6019}

(As I often do, I have combined on one line the command and the output generated by Mathematica®.)

There are only 4 of them. This matrix is of rank 4 instead of rank 5. But there’s more:

It has a nullspace of rank 1 and the 5th column of v is a basis for the nullspace! (I discussed how columns of u and v were bases for the four fundamental subspaces back in this post.)

It will be necessary to get the entire Singular Value Decomposition (SVD) instead of just the list of singular values; we need both w and v. Recall

X = u w v’;

I remind you that u and v are orthogonal matrices: u’u = v’v = I = uu’ = vv’. That means that columns of v are orthonormal vectors.

Here is the command for getting the SVD, and here is the w matrix:

(A Mathematica aside: my SVD is computed using integers and the answers are rational, and huge… the //N on the display of w gets me real numbers. Down the road doing it this way will get me the convenience of integer zeros instead of real zeros. I’m sure there was another way to cope with the real zeros, but this is the method I found.)

Detection.
That rightmost column of zeros in w shows us graphically that there are only four nonzero singular values. And it’s that column of zeros that says X.v will have a column of zeros:

$X v = \left(\begin{array}{ccccc} -125.22 & -106.829 & 27.8455 & 3.49281 & 0. \\ 116.276 & 66.0346 & 31.9677 & -3.93586 & 0. \\ 1708.36 & 96.1829 & -17.073 & 4.72986 & 0. \\ 9158.93 & -62.7248 & -38.1859 & -2.72592 & 0. \\ -29686. & -5.2723 & -9.70055 & -0.319794 & 0. \\ 18829.9 & 12.3532 & 4.81722 & 0.440139 & 0.\end{array}\right)$

And the column of zeros in Xv explicitly shows that the fifth column of v is in the nullspace of X, because we see that the fifth column of v was mapped to the zero vector.

Identification and Isolation.
So just what is the fifth column of v? (No, you didn’t miss anything: I never showed you all of the matrix v. We don’t need all of v.)

Let’s divide by the smaller number (in absolute value):

which says that (and here’s where integer zeros came in handy)

-2 x4 + x5 = 0

i..e

x5 = 2 x4.

Perhaps I should elaborate on that. Let

n1 = {X1, X2, X3, X4, X5}

symbolize any row in our matrix X. Then

n1.t1 = -2 X4+X5

is the result of applying that row to our basis vector t1. But t1 is in the nullspace, so that dot product is zero… and I could choose to write X5 as…

In fact, I think I’ll let Mathematica do that for me from now on.

## Example 2

Now we are given a different matrix:

$X = \left(\begin{array}{ccccc} -74 & 80 & 18 & -56 & 116 \\ 14 & -69 & 21 & 52 & -27 \\ 66 & -72 & -5 & 764 & -82 \\ -12 & 66 & -30 & 4096 & 6 \\ 3 & 8 & -7 & -13276 & -6 \\ 4 & -12 & 4 & 8421 & -4\end{array}\right)$

Detection: is this matrix of less than full rank? Get the SVD… and display the matrix w of singular values:

Yes, there are only four nonzero singular values: column 5 is all zeros.

Isolation and Identification.
What is the fifth column of v?

## Example 3

Given this matrix…

$X = \left(\begin{array}{ccccc} -74 & 80 & 86 & -56 & -112 \\ 14 & -69 & -124 & 52 & 104 \\ 66 & -72 & -78 & 764 & 1528 \\ -12 & 66 & 120 & 4096 & 8192 \\ 3 & 8 & 19 & -13276 & -26552 \\ 4 & -12 & -20 & 8421 & 16842\end{array}\right)$

Here are its singular values, the matrix w:

Detection.
There are only 3 nonzero singular values, and two columns of zeros. This means that the two rightmost columns of v are a basis for the nullspace of X. We have two linear dependencies.

Isolation and Identification.
The two rightmost columns of v are:

Let’s rescale those… divide each column by its smallest element. Column 4, written as a row, becomes…

## Example 4

We are given this matrix:

$\left(\begin{array}{ccccc} -74 & 80 & 18 & 166 & 332 \\ 14 & -69 & 21 & -193 & -386 \\ 66 & -72 & -5 & -150 & -300 \\ -12 & 66 & -30 & 186 & 372 \\ 3 & 8 & -7 & 27 & 54 \\ 4 & -12 & 4 & -32 & -64\end{array}\right)$

Here’s its SVD, with only w displayed, as usual:

Detection and Isolation and Identification.
Three nonzero singular values; i.e. two columns of zeros. We have two linear dependencies again. (Rank = 3, but full rank would be 5.)

Here are the two rightmost columns of v:

Are both answers – the old pair and the new pair – right?

Yes. The difference is because v4 and v5 (not the visually appealing t4 and t5) are an orthonormal basis for the nullspace. The vectors t1 and t2 span the nullspace, but they are not orthonormal.

(Incidentally, let me remark that t4 and t5 are orthogonal, but not unit vectors, hence not orthonormal.)

The key is that in example 3, the two dependencies were specified as orthogonal. (I wrote them that way! To be more precise, the two dependencies involved different variables – and that guaranteed orthogonal vectors.)

In example 4, the two dependencies had X4 in common.

The fastest way to check both sets of answers is just to apply X to the vectors.

## Example 5

Here’s another matrix:

$X = \left(\begin{array}{ccccc} -74 & 80 & 18 & -56 & 113 \\ 14 & -69 & 21 & 52 & -30 \\ 66 & -72 & -5 & 764 & -85 \\ -12 & 66 & -30 & 4096 & 3 \\ 3 & 8 & -7 & -13276 & -9 \\ 4 & -12 & 4 & 8421 & -7\end{array}\right)$

Here are its singular values.

There are 5 of them. This matrix is not linearly dependent. Everything is fine.

But.

If I were to run a regression using these 5 variables as the independent variables, we would have a problem.

What would have changed?

We would add a column of 1s to those variables.

Instead of studying X itself, if we’re going to use it for regression, we need to study the design matrix.

Here’s the corresponding design matrix:

$M = \left(\begin{array}{cccccc} 1 & -74 & 80 & 18 & -56 & 113 \\ 1 & 14 & -69 & 21 & 52 & -30 \\ 1 & 66 & -72 & -5 & 764 & -85 \\ 1 & -12 & 66 & -30 & 4096 & 3 \\ 1 & 3 & 8 & -7 & -13276 & -9 \\ 1 & 4 & -12 & 4 & 8421 & -7\end{array}\right)$

Here are its singular values:

Detection.
That column of zeros says we’re lacking one nonzero singular value: the matrix is of rank 5 instead of rank 6.

Isolation and Identification.
What’s the rightmost column of v’?

## Summary

Get the SVD of the matrix X to be investigated…

X = u w v’.

Display the matrix w of singular values.

The number of columns of zeros in w is the number of columns of v (counting from the right) which are a basis for the nullspace. Each basis vector can be translated into an equation expressing linear dependence of the variables.

That’s it.

And it looks like it will work for multicollinearity, too – somewhat differently because we won’t actually have a nullspace….

Let me add a couple of remarks.

The columns of v will always give us an orthonormal basis; there might be “simpler” non-unit vectors which have a more intuitive explanation. In fact, I kept normalizing the vectors so that their components were integers. That was possible because my original matrices were composed of integers.

But we also saw in Example 4 that the two orthogonal vectors were not as simple a description as what we found by looking at subsets, so it may be worthwhile knowing that we can do it that way, too.