Linear Algebra: the four fundamental subspaces

Edit 18 Sep 2009. Purely cosmetic. I have highlighted in blue the result describing the spans of the parts of the u and v matrices. I have found that I still need to refer to it rather than reconstruct it when I want it. In other words, it’s what I want to pick out quickly from this post.

Given a linear operator A, we are told that there are four fundamental subspaces associated with it. They are

• the nullspace of A
• the column space of A
• the nullspace of (i.e. of the transpose of A)
• the column space of .

Okay, they are easy enough to list.

Next, we need to realize that the column space of A is the range of A, and the column space of is the range of . That’s what we really care about — the ranges.

But why is involved? In particular, why do we care about the nullspace and the range of ?

Let me show you what they “really” are. (That is, this is how I understand them.)

I’m going to use an exercise from Strang’s “Linear Algebra and Its Applications”; see the bibliography.

This is exercise 2.4.2.. “Find the dimension and construct a basis for the four subspaces associated with…”

(Of course, since the second row is twice the first, we already know that the matrix is of rank 1.)

Now, Strang presents the four fundamental subspaces by using the LDU (LU) decomposition; I am going to use — what else? — the Singular Value Decomposition (SVD).

Here is the Mathematica command…

We know all about the SVD. In particular, we know that the SVD decomposes the matrix A as

,

where u and v are orthogonal matrices; and w is the same shape as A and, loosely speaking, as diagonal as it can be.

Here is the w matrix…

That we have only one nonzero value tells us, if we didn’t already know it, that the matrix A is of rank 1. ( This is a representation of the A matrix with respect to different bases, specified by u and v. As such, it is the same shape and of the same rank. Since w is of rank 1, so is A.)

Here is the v matrix…

The columns of v are an orthonormal basis for the domain of A. Let’s apply A to it, that is to each of these basis vectors.

Well! The three rightmost columns of v were mapped to zero by A; we conclude that they are a basis for the nullspace of A.

What about the leftmost (remaining) column of v? Its image, by definition, is in the range of A. So the first column of v is a basis for the pre-image of the range of A.

(Once we know what A does to a basis for its domain, we know what it does to any vector in its domain.)

That’s the pre-image of the range. What about the range of A? Well, let’s look at the matrix u:

We see that the first column of u is proportional to the image of the first column of v. They span the same space — which is the range of A. Look at the algebra instead of the numbers:

,

and the almost-diagonal form of w says that the columns of u w (equivalently, the columns of A v) are multiples of the columns of u. OK, some of them are zero multiples, but the real point is that the nonzero columns of A v are multiples of corresponding columns of u.

What about the other column of u? The matrix A maps from R^4 to R^2; we say that its codomain is R^2. And its codomain is split into two subspaces, each 1-dimensional:

• the range of A
• the part of the codomain that is not the range (“the part we cannot reach with A”).

The rightmost column of u is a basis for the rest of the codomain.

Okay…. We seem to have four fundamental subspaces, and we have orthonormal bases for them, which means we have orthogonal direct sums:

• domain of A = nullspace of A preimage of the range of A
• codomain of A = range of A the part we cannot reach with A.

The columns of v split naturally into bases for the nullspace and for the pre-image of the range of A; the columns of u split naturally into bases for the range of A and for the part we cannot reach with A. That is,

• nullspace of A is spanned by rightmost columns of v
• pre-image of range of A is spanned by leftmost columns of v
• range of A is spanned by leftmost columns of u
• the part we cannot reach is spanned by rightmost columns of u

But what about the range and nullspace of ? Now would be a good time to guess that

• the preimage of the range of A is the range of
• the part we cannot reach with A is the nullspace of ,

and that’s why shows up: it provides more convenient (and in a sense, symmetric) names for two of the four spaces.

Maybe we should write the SVD of ? From

,

we get

.

The roles of u and v have been swapped. (The point is that there are no new bases involved, still just the columns of u and v.)

Then our description of the spaces and bases of A translates to the following description of the spaces and bases of :

• nullspace of is spanned by rightmost columns of u
• pre-image of range of is spanned by leftmost columns of u
• range of is spanned by leftmost columns of v
• the part we cannot reach with is spanned by rightmost columns of v.

Our guess was right. The bases tell us that we have two names for the spaces spanned by each of the 4 partial bases.

• the rightmost columns of v span the nullspace of A, and equivalently the part we cannot reach with
• the leftmost columns of v span the pre-image of the range of A, and equivalently the range of
• the rightmost columns of u span the nullspace of , and equivalently the part we cannot reach with A
• the leftmost columns of u span the pre-image of the range of , and equivalently the range of A.
  

Therefore, instead of naming the orthogonal direct sums as

• domain of A = nullspace of A preimage of the range of A
• codomain of A = range of A the part we cannot reach with A

we call them

• domain of A = nullspace of A range of
• codomain of A = range of A the nullspace of

(If you think about it, those are the only two combinations of “range” and “nullspace” that make sense, i.e. that are subspaces of the domain and codomain respectively.)

BTW, we should actually solve the exercise (!) and write down the bases.

Recall v.

For the nullspace of A, the 3 rightmost columns of v:

Strang’s basis is equivalent:

Check mine by applying A to each column…

For the nullspace of , recall u:

then the rightmost columns of u are:

Strang’s basis is equivalent: .

Check mine by applying to it:

For the range of A, the first column of u:

Strang’s basis is equivalent: .

For the range of , the leftmost column of v:

Again, his basis is equivalent:.

I should point out that Strang’s bases for the domain and codomain (found from the LDU decomposition) are orthogonal but not orthonormal. Mine are orthonormal.

I should also point out that I really, really like orthonormal bases. Sometimes there’s a good reason not to use them, but barring such a reason – orthonormal. And we will be using this material for light spectra.

So, as I see it, we have two principles here.

First, the 4 fundamental subspaces “really” are

• an orthogonal decomposition of the domain of A:
  • nullspace of A
  • preimage of the range of A
• an orthogonal decomposition of the codomain of A:
  • the range of A
  • the subspace we cannot reach with A

Second, we can describe two of those subspaces in terms of ,

• the range of = preimage of the range of A
• the nullspace of = the subspace we cannot reach with A.

Thus, more compactly, we have the 4 fundamental subspaces as

• an orthogonal decomposition of the domain of A:
  • nullspace of A
  • range of
• an orthogonal decomposition of the codomain of A:
  • the range of A
  • nullspace of

As Paul Harvey used to say, “And now you know the rest of the story.”