Regression 1: You inverted what matrix?

Edit: 2011 Nov 25: In “How the Hell” I have some negative signs. Find “edit”.

I want to show you something about LinearModelFit that shocked me. I will use the Toyota data. Let me point out that I am using version 7.0.1.0 of Mathematica@. Furthermore, this post is in two categories, OLS and linear algebra; the example comes from regression, but the key concept is the inverse of a matrix.

Regression 1: Linear Dependence, or Exact Multicollinearity

You want to read my comment of Feb 7 at the end of this post: there is an additional post you will want to read after you read this one.

Introduction

(Let me say up front that this was a long post to assemble, and I’m not at all sure that my editing got all of the “dependent” and “independent” right – so feel free to point out any mismatches.)

I had thought that I would begin this discussion by looking at the Hald data. It turns out, however, that we need to take a wider view before we specialize.

I have decided that it is a very good idea to begin an investigation of multicollinearity by investigating linear dependence. And I thought I didn’t like the term “exact multicollinearity” – except that it’s perfect for why I’m doing this post.

For one thing, “multicollinearity” has a vague definition: the term is used to describe set of vectors that are “close to” being linearly dependent… multicollinearity, then, is “approximate linear dependence”. But how close must it be before it’s an issue?

So let’s look at exact linear dependence first.

This is of more than pedagogical interest: what I am about to show you can be applied to multicollinearity.

Example: Is it a transition matrix? Part 1

This example comes from PCA / FA (principal component analysis, factor analysis), namely from Jolliffe (see the bibliography). But it illustrates some very nice linear algebra.

More precisely, the source of this example is:
Yule, W., Berger, M., Butler, S., Newham, V. and Tizard, J. (1969). The WPPSL: An empirical evaluation with a British sample. Brit. J. Educ. Psychol., 39, 1-13.

I have not been able to find the original paper. There is a problem here, and I do not know whether the problem lies in the original paper or in Jolliffe’s version of it. If anyone out there can let me know, I’d be grateful. (I will present 3 matrices, taken from Jolliffe; my question is, does the original paper contain the same 3 matrices?)

Like the previous post on this topic, this one is self-contained. In fact, it has almost nothing to do with PCA, and everything to do with finding — or failing to find! — a transition matrix relating two matrices.

Color: decomposing the RGB A transpose matrix

(abuse of) terminology

Sometimes I get tired of writing “xbar, ybar, zbar tables” — and I just write “xyz bar tables” or even “XYZ tables”. Similarly for rbar, gbar, bbar tables — rgb bar. I’m not talking about anything new, just abbreviating the names.

Introduction

This is the third post about the example on p. 160 of W&S. Once again, I am going to decompose the reflected spectrum into its fundamental and its residual.

This time, however, I’m going to use the rbar, gbar, bbar tables (RGB) instead of the xbar, ybar, zbar (XYZ) tables. They did not do this.

I’ll tell you now there is one little twist in these calculations. We will need the ybar table, because we still need to use it to scale our results.

In addition to showing, as I did previously, the dual basis spectra and the orthonormal basis spectra for the non-nullspace, I will display the orthonormal basis for the nullspace.

Color: decomposing the A transpose matrix and the reflected spectrum

This is the second post about the example on p. 160 of W&S. Here’s the first. (Incidentally, I am going to decompose the reflected spectrum into its fundamental and its residual. I do not think that they did this. Oh, this is what I did for Cohen’s toy example, but now it’s for real. Approximate, but for real.)

review the XYZ for this

Linear Algebra: the four fundamental subspaces

Edit 18 Sep 2009. Purely cosmetic. I have highlighted in blue the result describing the spans of the parts of the u and v matrices. I have found that I still need to refer to it rather than reconstruct it when I want it. In other words, it’s what I want to pick out quickly from this post.

Given a linear operator A, we are told that there are four fundamental subspaces associated with it. They are

• the nullspace of A
• the column space of A
• the nullspace of (i.e. of the transpose of A)
• the column space of .

Okay, they are easy enough to list.

Next, we need to realize that the column space of A is the range of A, and the column space of is the range of . That’s what we really care about — the ranges.

But why is involved? In particular, why do we care about the nullspace and the range of ?

Let me show you what they “really” are. (That is, this is how I understand them.)