Example: Is it a transition matrix? Part 2

We had three matrices from Jolliffe, P, V, and Q. They were allegedly a set of principal components P, a varimax rotation V of P, and a quartimin “oblique rotation” Q.

I’ll remind you that when they say “oblique rotation” they mean a general change-of-basis. A rotation preserves an orthonormal basis; a rotation cannot transform an orthonormal basis to a non-orthonormal basis, and that’s what they mean — a transformation from an orthonormal basis to a non-orthonormal basis, or possibly a transformation from a merely orthogonal basis to a non-orthogonal one. In either case, the transformation cannot be a rotation.

(It isn’t that complicated! If you change the lengths of basis vectors, it isn’t a rotation; if you change the angles between the basis vectors, it isn’t a rotation.)

Anyway, we showed in Part 1 that V and Q spanned the same 4D subspace of .

Now, what about V and P? Let me recall them:

As before, I suppose there is a transition matrix T relating them; each column of can be computed by applying a transition matrix T to the corresponding column of :

V’ = T P’

i.e.

V = P T’.

Then I imagine briefly that I can “solve” for T’…

,

but then quickly replace the non-existent inverse of Q by the appropriate pseudo-inverse

.

I compute T’, and then display its transpose T:

and then I test whether

V = P T’ (?).

The largest difference (element by element) between the matrices V and P T’ is 0.170627, which is rather larger (by a factor of 23) than the differences we saw in Part 1. Here’s the full blown matrix — where I chose to round off first, then subtract (i.e. what we’d compute if we were subtracting published, already rounded-off, matrices):

I am inclined to think there is a probem. But we don’t know much at all about it.

Let’s try one of our alternatives. We construct a projection operator onto the subspace spanned by P.

Get the SVD (Singular Value Decomposition) of P…

.

Then, since u is 10×10 and V is of rank 4, the leftmost 4 columns (u1) of u….

are an orthonormal basis for the range of P (i.e. the column space of P). We construct a projection operator, as we have before, by

.

We check it first by confirming that R R = R, i.e. it is idempotent, hence a projection operator. We check it further by applying it to the column vectors in P: we expect that R P = P; i.e. it actually projects onto the column space of P (the space spanned by the columns of P).

Now, having checked that R is a projection operator onto P, we apply it to V, and compute V – R V:

Let me round that to the nearest .01…

I should put that array of differences into perspective. Recall V:

This reinforces my belief that we have a problem. Some of the differences we found are larger than some of the values of V itself. I can see that V is not in the column space of P, and while it’s not extremely far away, it’s not close either. (This seems more definitive than “the hypothetical transition matrix T doesn’t quite work”.)

As before, however, we are tip-toeing around the real question: Is every column vector in V a linear combination of the columns of P?

Easy enough. Do a regression. Actually 4 of them, one for each column of V.

Let me name the individual columns of the P matrix, and let me set the first column of V (v1) as “y”, and the columns of V as independent variables.

(Some smaller pictures follow, in which you can see that the final option is “IncludeConstantBasis -> False”.) The results are:

Not bad at all. In fact, rather close to perfect.

What was the first row of T?

{0.905145, -0.386677, 0.0882059, -0.141151}.

Yes, those are the (“Estimates”) for that regression; first row of T is first column of T’ is regression coefficients for the first column of V as the dependent variable.

Let me emphasize another relationship. For the first regression, yhat should be the projection of v1 (the 1st column of V) onto the column space of P. But we have that projection directly as R v1:

R v1 = {0.477784,0.476228,0.355644,0.257779,0.487767,
0.058964,0.209419,0.0529866,0.123281,0.19278}

… and we have the yhat from the regression:

yhat = {0.477784,0.476228,0.355644,0.257779,0.487767,
0.058964,0.209419,0.0529866,0.123281,0.19278}

Good, they are the same. So if regression works better for you than the projection operator, then use regression. Personally, I prefer the projection operator because I can apply it to all of V at once. (And because sometimes I have a use for the orthonormal basis u1.)

Largely to display the R^2, let me run the other three regressions. (Oh, I also reduced the magnification in my Mathematica notebook, so the pictures are smaller and the right side of the regression command isn’t cut off, at least on my screen.)

Here’s v2 as the dependent variable:

Here’s v3 as the dependent variable:

Here’s v4 as the dependent variable:

We see that the is over 99% for v2, falls to 95% for v3, and is 97.6% for v4. Those last two are just not large enough, not for this question.

I’m certain that V and P do not span the same subspace, but that’s about all I know.

We could, however, have made the dual choice, projecting P onto V instead of V onto P. Are they the same?

No.

We already know — from the T matrix, from the projection operator, or from the regressions — that the P and V matrices do not represent the same data in two different coordinate systems. But there might be — and in this case, there is! — more information available by making the other choice.

Let’s get the projection onto V instead of onto P. We need the SVD of V…

.

We need the 4 leftmost columns (u1) of u…

We construct the projection operator…

Now we apply it to P, and compute the difference P – R P:

So what?

Round it off:

Ah ha! It’s time for Robin to squeak out, “Holy moly, Batman!”

The fourth column of P does not lie in the subspace spanned by the 4 columns of V, while the first three do. That is the problem. I don’t why or where it happened, but I know exactly what’s wrong: the 4th column of P is the only problem.

Somebody blew it somewhere. (OK, since I don’t know how a varimax rotation is computed, I have to entertain the possibility that V is not supposed to be P under a change of basis. But I can’t believe that, because V and Q have a common column space, and 3 of the 4 columns of P lie in that common column space. If that’s an accident, what does a deliberate act look like?)

Is the mistake in the orignal paper? Were P and V computed from the same correlation matrix but by independent algorithms? Or was V computed incorectly from P (and then Q computed from V)?

Or did someone copy P incorrectly? (P should be primary, but then how the heck does the 4th column of P fail to be in the column space of V? I would sooner have expected an error in V, given P.)

Or did they rotate the first 3 PCs, and then apply that transformation to the 4th column? (No. I’ve checked. You can too.)

Let me emphasize that we would get the same result from four regressions each with one column of P as the dependent variable and all the columns of V as the independent variables. The first three regressions would be extremely good fits, but the fourth would not be. We would, again, conclude that the fourth column of P was not exactly — or even to a fair approximation — a linear combination of the columns of V.

Summary: theory

Given two matrices, say P and V, which are alleged to be the same data in different coordinate systems, we may check that assertion very quickly (if both are of full rank):

Define

.

If we find it true that

V’ = T P’,

then T is a transition matrix between two bases, and P and V are indeed the same data in different coordinate systems. We’re done.

If, however, the equality fails, ,

then T is not a transition matrix, and P and V are not the same data in different coordinate systems. Unless we want more information, we’re done.

Finally, we have seen that, numerically, we may have T “close to” a transition matrix. (In Part 1, V and Q are very nearly the same data; in the color posts, the xyx bar and rgb bar tables are very nearly the same data.)

Summary: practice

Computing T tells us if something is wrong, but not much more. It’s quick, easy, and gives a simple answer: “good” or “no good”.

Computing both projection operators P onto V and V onto P (not just one of them) would likewise tell us whether or not something is wrong, and, in addition, the projection operators might isolate the problem.

We could use regression instead of computing explicit projection operators.

Summary: this example

We found that V and Q are the same data in two different coordinate systems, but that P is not. In addition, we actually know that the problem is limited to the 4th column of P: it does not lie in the space spanned by the columns of V (i.e. it does not lie in the column space of V, which is also the column space of Q).

Question:
Can someone tell me if these P, V and Q match the original paper by Yule et al.?

And there’s a question you might have.
How many examples did I have to search through to find such a good (because it was such a bad) example?

One. This was the very first case I checked, outside of the 1931 CIE tables.

That’s scary.

Color: re-doing Cohen’s example

Cohen’s example again

Let me now show you how I would do Cohen’s example. (His computations, pretty much, were the previous post.) I cannot over-emphasize that he deserves a lot of credit for getting the mathematics right, even if he didn’t name it correctly or do it beautifully.

I start with the A matrix:

This, as I have said, is a toy example of real color matching functions. Each column corresponds to one of the tri-stimulus values; each row represents a wavelength.

My summary of Cohen began with a different matrix — the E matrix — only because it was easier to enter into the computer. A is primary.

The next issue we have to cope with is that we really care about the transpose rather than about A. A’ will be applied to a spectrum, and will deliver three tri-stimulus values for it. I’m not convinced that any notational choice I make now will be without problems.

(As I said in the previous post, I use A’ for the transpose whenever it is convenient.)

I am going to let

.

B represents the linear operator I wish to study. Thus, B is

It maps to . At most, it can be of rank 3; and then it would have a four dimensional null space.

What shall we do with it? All together now, class: find its singular value decomposition (SVD).

,

where u and v are orthogonal (hence square) matrices, and w is as close to diagonal as it can be, given that it is the same shape as B.

The Mathematica command is

{u,w,v}=SingularValueDecomposition[B];

where the final semi-colon suppresses printing.

It is convenient to look first at the singular values themselves, the nonzero “diagonal” elements of the matrix w:

.

That all three are “significantly” larger than zero tells us that the matrix B (and therefore its transpose, A) is, indeed, of rank 3.

In case this is new to you, the w matrix is

(And if this is new to you, check the tag cloud and the categories. There’s a lot examples of the Singular Value Decomposition. I could almost rename this blog “Applications of the SVD”.)

What about v?

It is 7×7. We know from the discussion of the four fundamental subspaces (look at the blue text) that the 4 rightmost columns of v are an orthonormal basis for the null space of B, and the 3 leftmost columns are an orthonormal basis for the pre-image of the range of B.

That is, writing a partitioned matrix [v] = [v1 v2], we have

We should confirm that B.v2 = 0, i.e. every column of v2 is in the null space of B:

Once having a basis, v1, for the pre-image of the range, we know that we can construct a projection operator as v1 v1′:

Because I didn’t call it R, I can recall Cohen’s R matrix, computed using a pseudo-inverse. I assure you that it is the same, we have P = R.

On the other side, since B is of full rank, its range is equal to its codomain, both being , so the entire matrix u is an orthonormal basis for the range of B.

Now the thing to do is, as before, split some given spectrum into its fundamental and its residual (metameric black). We only really have one good test, an equal energy spectrum.

We get the fundamental of our test spectrum…

…by applying our projection operator to it:

and that is exactly what we got before (come on, P = R; we have the same projection operator.)

Then we get the residual n as the difference…

and that, too, is the same as before.

As a refesher, let’s compute the components of the fundamental wrt the v1 basis. Since v1 is orthonormal, we may find the components of the fundamental by taking dot products, the fundamental dotted into each basis vector in v1. But that’s just

which says that f can be written as the sum

of the following three multiples of the basis vectors– I’m going to round off for clarity –

.

The sum does indeed (almost) match our fundamental. (The discrepancy comes from my rounding off.)

.

The last thing we need to get is the dual basis E for the rows of B. I just go look it up in the previous post…

:

Check that B E = I; it does.

And why do we care about the dual basis, i.e. the columns of E? Well, if we apply B to our test spectrum, we get

.

What are those three numbers? They are components of our test spectrum wrt the dual basis E!

That is, the sum of the following three vectors (which are multiples of the basis vectors in E)…

is

… which, once again, differs from our fundamental

only because of the rounding used to make the numbers readable.

To say that again: the dual basis E gives us a geometric interpretation of the tri-stimulus values we computed by applying B (i.e. A’) to a spectrum. The tri-stimulus values are components of the spectrum (equivalently, of its fundamental) wrt the E basis.

So. What did Cohen and I have that was the same? The matrices A and E, and the projection operator R. But note that where he computed R as

R = E’A,

and that’s short for ,

I would use

.

I’ll admit that’s not much of a savings if we have both A and E.)

What did we have that was related? Where he had two orthonormal bases F1 and F2, I have one, v1. All three of these bases span the same subspace of .

Digression: ways to show that F1, F2, and v1 span the same subspace of R7

One way to show that is to show that the columns of F1 and F2 are preserved by the projection operator cobstructed from v1. We have

and we compute

And yes, their difference is the zero matrix.

Similarly for F2.

Alternatively, F1 and F2 should generate the same projection operator as v1 did.

I assure you that is the same as P and R. And similarly for F2.

Alternatively, again, we could simply write the columns of F1 (then F2) as a linear combination of the columns of v1, and solve for the components.

Alternatively, yet again, we could try to find a transition matrix relating F1 and v1, then F2 and v2. That’s awkward because they’re not square matrices. Guess what? Use a pseudo-inverse! (Very likely the next math post.)

I have no idea what is special about F1 and F2. They are orthonormal bases, like v1, for the pre-image of the range of B. Since they span the same space, they are related by rotations (and possibly a reflection).

end digression, return to main thread

Finally, what did I do that Cohen did not? The singular value decomposition, which got me a basis, v2, for the null space of B — which he did not have. And I daresay that v1 was easier to get than either F1 or F2.

I think having v2 is incredibly useful. I get a basis for metameric blacks.

(Of course, Cohen had computed the matrices Ma and Me, and Ga and Ge, but I don’t need them because I wasn’t computing F1 and F2.)

Let me summarize that.

Cohen

• A and E;
• Ma = A’A, Me = E’E, where Me = Ma^-1;
• Ga and Ge, Cholesky decompositions of Ma and Me;
• orthonormal bases F1 = A Ge and F2 = E Ga;
• the projection operator .

Rip

• A;
• A’ = B = u w v’;
• [v] = [v1] [v2] where both v1 and v2 are orthonormal bases;
• the projection operator R = v1 v1′;
• the dual basis E.

(Strictly speaking, A is the dual basis to E, but the relationship is symmetric. Nevertheless, the rows of A’ live in the dual space V*, while the columns of E live in the original space V.)

I dispensed with everything leading to F1 and F2. I use v1 in place of F1 and/or F2. I get a basis, v2, for the null space.

I think the key point is using the projection operator to split a spectrum into fundamental and null vectors. It’s a unique orthogonal decomposition into a vector which has the same tri-stimulus values as the original spectrum, and a vector in the null space (metameric black, with tri-stimulus values = {0,0,0}.)

(The decomposition is unique to A, but A itself is not unique. More to follow.)

PCA / FA Example 4: Davis. R-mode & Q-mode via the SVD.

let’s finally do this using the SVD. i need to do one terrible thing: where davis writes U and V, we need v and u, resp. (look, i couldn’t reliably keep translating davis’ equations in my head, so i had to use his notation; by the same token, i can’t reliably translate the SVD over and over again. thank god he used UC, upper case.)

if the correspondence had been u ~ U and v ~ V, the translation would have been trivial. unfortunately, the correspondence is

u ~ V

v ~ U.

(from the SVD posts, you recall that given we conclude that v is an eigenvector matrix for , and u is an eigenvector matrix for . and, just as important, the nonzero values of w are the nonzero .)

from the SVD of X,

and the definition of

i conjecture that we could define .

from the definition of

i conjecture that we could define .

then from

we get ,

since v is orthogonal.

and from

we get ,

since u is orthogonal.

(almost) finally, from and

we get ,

which is the first dual relationship between Q-mode and R-mode.

finally, from and

we get ,

which is the other dual relationship.

so we see that davis’ duality delationships make sense. 

let me summarize the SVD equations for the As and Qs:

.

BTW, i’ll compute using and because they involve two matrices instead of three.

it’s time to actually compute the SVD.

here’s u, v, and w:

(we could check that u and v are orthogonal, and in fact i did check that. OTOH, in contrast to eigenvector matrices, mathematica says u and v will always be orthogonal.)

we should compare u with V; here’s V:

most interesting. the 3rd and 4th eigenvectors are different although the 3rd one isn’t very different. what’s going on? well, they are two eigenvectors associated with the same eigenvalue (0), and in fact they span the null space of . (i think this is true for any repeated eigenvalue, that we could select any orthonormal basis for the multi-dimensional eigenspace, but i need to confrm that. in any case, it’s certainly ok here because the eigenvalue is zero and we’re dealing with the null space.)

what is significant, however, is that u and V have different signs on the two columns with nonzero eigenvalues, the 1st and 2nd.

now compare v and U:

they differ only in the sign of one column that matters (because it has nonzero eigenvalue), the 1st. and that’s the problem: U and V are incompatible in the sign of the 2nd column. we don’t have to match davis; we just have to change the sign of one column if we want the expected correspondence between U ~ v and V ~ u.

let me just show the incompatibiity. from the SVD ,

i can always write .

the question is: given U and V computed separately by eigendecompositions, can we use them in the SVD? let’s see what happens if we substitute U for v and V for u. that is, we compute

.

 

x x

=

close. there’s just one little problem: the nonzero elements of w are supposed to be positive. this is what i mean when i say that U and V need not be compatible in general, and in this case they are not. after all, U and V came from independent eigendecompositions.

moving on, we compute

x

=

we should compare that to what i got before.

oh, dear. this emphasizes that i changed the definition of when i went from an eigendecomposition to the SVD. OTOH, so did davis. 

if is defined as the weighted eigenvector matrix U, then it must be the same size as U. we would get it by post-multiplying U by a 3×3 diagonal matrix of eigenvalues. instead, mine is bigger than 3×3, while davis’ is smaller.

but if it is defined as , then is not the same size as v (or U) because w is not square. 

i think this is just a matter of terminolgy. conceptually, i can best see the duality between Q-mode and R-mode using the SVD. 

OTOH, conceptually, if i want the -weighted eigenvector matrix, i just post-multiply v by a diagonal matrix. in practice, i would just cut down to get something the same size as v.

to put that another way, if we use davis’ square matrix of , then we still see the duality and his is the nonzero -weighted eigenvector matrix.

(BTW, we’ve seen that square matrix of before, in the derivation of the SVD. we needed to invert it. maybe i will start using it, too.)

if this is getting confusing, fall back on: what we have are u, w, and v; everything else follows from them.

moving on again, we compute :

 

x

=

then we compute :

x

=

(i see no point to confirming that ; that was our definition.)

now we compute :

x

=

we’ve done all the computations. (that’s an understatement: we done them three or four different ways!) 

next we will look at the interpretations.

PCA / FA example 4: Davis. R-mode FA, eigenvalues

from davis’ “statistics and data analysis in geology”, we take the following extremely simple example (p. 502). his data matrix is

where each column is a variable. he now centers the data, by subtracting each column mean from the values in the column.

let me do that. i compute the column means…

and subtract each one from the appropriate column, getting

now let’s just follow him. he introduces the SVD (the singular value decomposition), by way of explanation, and we will use it, but not quite yet. he performs what’s called an R-mode factor analysis as follows.

compute :

then he computes the eigendecomposition. i get that the eigenvector matrix is… 

and that the diagonal matrix of eigenvalues is …

we run into our first notational quagmire. he has shown us the SVD , and what we have here is , the eigenvalues of (see “an SVD of X ~ eigenstructures of XTX and XXT”).

what he wants are the singular values w. no problem, you think: just take the square roots. sheesh, that’s what harman did, and maybe we’re beginning to see why.

unfortunately, davis is also using a 2×2 diagonal matrix instead of the 3×3 i get. to be more specific, i get w as a 4×3 and as a 3×3. to be completely specific, i get w is… 

and therefore is…

which is, as it should be, the 3×3 matrix of eigenvalues.

well, having said that he wants the square roots, i’ll give them to him, but i will modify his notation. i take to be a 3×3 diagonal matrix of square roots of eigenvalues, one of which is zero…

and i will use for his 2×2 matrix of nonzero square roots (which he called ).

and, i will need the 2×2 submatrix of eigenvalues:

perhaps i should point out that i have no intention of throwing away any parts of these matrices in my own work. but i need to make sure that i am following davis correctly, so i am carrying out his computations in parallel with my own.

besides, you might be trying to read davis even as i speak, another reason for me to display his work, too.

at the end of all that, all we have gotten is 2 positive eigenvalues of (84 and 12), and we have their two positive square roots; we have these two pairs of numbers laid out in both 2×2 and 3×3 diagonal matrices. we have far more notation that mathematics.

so much for the eigenvalues. 

next we’ll dig into the eigenvectors.

the SVD: preface

Rather than edit the following SVD posts – unless I have to – let me put some prefatory material here. I am also going to answer a couple of email questions in comments, partly just to check out their mechanics.

The following posts in the math SVD category should be read in the order displayed, not starting from the back.

That may have been a bad idea on my part, but let’s see how it works out. The difficulty is that most of what I do out here will be available in the usual blog-order, latest first. Eventually it may be awkward if one category is in reverse order wrt everything else.

I would like to add two pieces of vocabulary. For the SVD the columns of u and v are called the left singular vectors of X and the right singular vectors of X, respectively.

I have also discovered that the latex interpreter works in comments. Cool! My understanding is that it does not work in forums; they would be the appropriate place to post code – and have it display as code! After all, that’s where I found the key starting points for latex.

the SVD (singular value decomposition) stuff

ok, i’ve published everything i planned to put out there for the SVD. not to say i won’t need to add things, if anyone ever has any questions about what i meant….  

yes, i need to put out a bibliography, too. but i’ve been at this all day, and it’s time to stop. incidentally, the SVD stuff is a category of its own – not a page as i originally set up the first (last!) installment.

i should probably emphasize that the SVD category is posted in the order i intend it to be read. 

it looks like quite a pile of stuff, and it’s scary to think that it’s all only background for principal components analysis!

one advantage to doing as much work as i did today is that i’ve managed to streamline the process. take a piece of the latex output from mathematica®, drop it into TextWrangler, edit it massively there, then copy a preliminary version into the wordpress editor, and finally edit there until it works.

well, i’m thrilled to have found a way to play with the SVD, after all the years i’ve meant to. i need to actually write down the list of all the other things i’ve been meaning to look at.

the Singular Value Decomposition (SVD)

The singular value decomposition (SVD) is marvelous. It says that any real matrix X can be decomposed as

[singular value decomposition]

where u and v are orthogonal (i.e. rotations) and w is as close to diagonal as possible.To be more precise,

where in fact is not only a diagonal matrix but it has only positive elements. Finally, we order the elements of w from largest to smallest(don’t sweat it if some are equal).
(the theorem is true for any complex matrix, provided the transpose is replaced by the conjugate transpose, and u and v are unitary instead of orthogonal. i just want to stick with the transpose. Or you could simply interpret as the conjugate transpose of v in everything i do.)
My main reference for this is Stewart: short and sweet, with some history and the derivation. Strang has a gentler discussion, but no derivation.

the SVD generalizes eigenstructure – one correction 5/22/08

I’ve seen two things like this. First, the diagonalization of a square matrix A:

[diagonalization]

where 

D is diagonal

correction: if A is symmetric, P may be chosen orthogonal.

We recall that not every matrix can be diagonalized; instead they can be brought to Jordan Canonical form. OTOH the set of matrices which cannot be diagonalized is a subset of measure zero. Nevertheless, for some applications (e.g. differential equations), matrices which cannot be diagonalized play a significant role. I’m sure i’ll talk about this someday.   

Looks a little different from the SVD, you say? we always have to pay attention to whether we decomposed a matrix or transformed it. Rewrite the transformation as a decomposition of A: 

and then as

[eigenvector decomposition]

since P was chosen orthogonal.

Now that resembles the SVD:

.

A is square, X is not (in general); D is diagonal most of the time, while w is as close to diagonal as a rectangular matrix can get; and instead of one orthogonal matrix P, we have two orthogonal matrices u and v.

The SVD is a generalization of the eigenvector decomposition, from square matrices to matrices of arbitrary shape.      

Among other things, this means that any square matrix can be diagonalized by u and v of its SVD; it just can’t always be done with u = v = P.

To put that another way, if we can change the bases independently in the domain and codomain, then we can diagonalize any square matrix; but if werequire the same one change of basis in both the domain and codomain, then we may not be able to diagonalize the matrix.

the SVD specializes change-of-bases

the general formula relating two matrix representations of a single linear operator L is:

[change of bases]

where P and Q are invertible (hence square) matrices (of the appropriate sizes). the best reference i’ve ever seen for this is — are you ready?

the Schaum’s outline for linear algebra, by Lipschutz.

that equation is written as a transformation. We have a linear operator

L: V –>W,

and we have its matrix A WRT bases in each space V and W. then we do a change of basis P on the domain V, and a change of basis Q on the codomain W, and we find that the matrix B of L WRT the new bases is given by the change of bases equation.

we may rewrite it as a decomposition:

.

we can’t replace by because P is an arbitrary invertible matrix: we may have made any legal change-of-basis, not the special case of a rotation.
The SVD 

 

is a special case of a change of basis: X and w represent the same linear operator WRT different bases. in particular, since v is orthogonal, we can replace by :

• X ~ A
• w ~ B
• u ~ Q
• v ~ P 

So the SVD tells us that any matrix can be represented by a nearly diagonal matrix.
Why don’t we end up with a possible jordan canonical form for the SVD? or to put it another way, why does the SVD of a jordan canonical form still turn out to have nonzero terms only on the diagonal?
Because we have two matrices u and v instead of one P, to effect the decomposition.
(Of course, the eigenvector diagonalization is also a special case of the change-of-bases equation, with only one basis to be changed.)

the SVD provides real-valued rank

Suppose that X has a singular value decomposition 

;

n nonzero singular values (nonzero elements of w);

the smallest nonzero singular value is .

Then,

X is of rank n

X differs from a matrix Y of rank n-1 by ;

no matrix of rank n-1 differs from X by less than ; 

we may compute one such matrix Y as follows: 

the matrix

is of rank n-1 and differs from X by   , where w2 is w with  replaced by 0.

If, for example, X had 5 nonzero singular values and the smallest one was .15, then we could say that the rank of X was 4.15 . and if we replace the .15 in w by 0 in w2 and compute Y, we get a matrix Y of rank 4 which differs from X by as little as possible.
i remark that i do this by changing a value in the matrix w, leaving u and v alone; hence Y is the same shape as X. There are those who would drop a row and column from w, then drop a column from each of u and v. (see below under alternative SVD.)