PCA/FA Answers to some Basilevsky questions

Let us look at three of the questions I asked early in February, and answer two of them.

First, what do we know? What have we done?

We assume that we have data X, with the variables in columns, as usual. In fact, we assume that the data is at least centered, and possibly standardized.

We compute the covariance matrix

,

then its eigendecomposition

,

where is the diaginal matrix of eigenvalues. We define the -weighted matrix

.

Finally, we use A as a transition matrix to define new data Z:

.

We discovered two things. One, the matrix A is the cross covariance between Z and X:

.

I find this interesting, and I suspect that it would jump off the page at me out of either Harman or Jolliffe; that is, I suspect it is written there but it didn’t register.

Two, we discovered that we could find a matrix Ar which is the cross covariance between Zc and Xs. That is, the general relation

specializes to both centered data Xc and standardized data Xs, and gives us

.

In addition, however, we found a matrix Ar such that

,

using the new centered data Zc and the old standardized data Xs. Oh, the definition of the matrix Ar was:

,

where was a diagonal matrix whose entries were the inverse standard deviations of the centered data Xc.

That is, each row of Ac was weighted by the inverse standard deviation to define Ar. And it all worked out because the definition of Ar

was dual to the definition of standardized data, which weighted the columns of Xc:

.

I will remind you that I find this interesting, but I’m not convinced it is important. I cannot forgive Ar for the fact that it is not a matrix of eigenvectors. I assure you, I could be missing something about Ar.

Anyway, that is what we did.

What were the questions?

1. when is A a covariance matrix?
2. is v a covariance matrix?
3. can we do something similar for constant row sums?

For question 3, I have the feeling there’s something we can do, but I haven’t put the pieces together. It can’t be same as for question 1, because a matrix that weights the rows of the data matrix is not the same size as one that weights the columns (in general). Perhaps we need to look at XXT instead of XTX. But then things don’t drop out if A and Z are defined the same way, so…. This is still open.

For question 1, two partial answers are:
If , i.e. A is formed by weighting the columns of an orthonormal eigenvector matrix v by the , then A is a covariance matrix of the same form as the one from which we got v.

And the construction that gave us Ar (weighting the rows of A) works for arbitrary weights.

I’ll explain.

Suppose we have a data matrix X. Suppose we compute a “dispersion matrix”, by which I mean a multiple of :

.

(Of course, is usually 1/(N-1).)

Suppose we do an eigendecomposition of c,

;

then define a matrix

,

and finally define new data using A as the transition matrix:

.

(I’ll remind you that our observations are columns of and ; and a transition matrix applied to new components gives us old components. If you need to, look here.)

I claim that

,

i.e., that A is the “cross dispersion matrix” between X and Z.

Please note, in particular, that the data X need not be centered.

The proof goes just the way it did before.

We will need a couple of things. From

,

we get

.

And also from

we get

(because v is orthogonal and is diagonal).

Then

.

QED.

(If we had started with instead of , we would have ended up with , which is a good thing, since . Things are just a little different because the cross-dispersion in general, and the cross-covariance in particular, are not symmetric.)

This result, that A is the corresponding dispersion matrix between X and Z means that we can say something about Z without ever computing it.

(I’m not particularly fond of that, but then I haven’t been trying to deal with thousands of observations and a hundred variables. I’m used to datasets small enough to hold in my hand, as it were.)

Example

We could verify this, in particular, for the raw data that we used for the previous Basilevsky computations. We knew it was true for centered data or for standardized data, i.e. that it was true for the covariance matrix or the correlation matrix; we have seen now that it is true for an arbitrary dispersion matrix

.

It didn’t depend on 1/(N-1) or on X being centered. If we do an eigendecomposition of , then .

Well, let’s do it. Here’s the raw data that we used here:

I will take , just so that it can’t be mistaken for anything else (I need some number; let’s make sure that if anything interesting happens, we see it. OK, nothing interesting will happen, not that kind of interesting, anyway.)

We compute

Now that I’ve written it once, let me emphasize: the weird expression occurs only because I needed a number for my abstract scaling factor . In retrospect, perhaps I should not have used greek letters for both abstract and numerical.)

We get an eigendecomposition, but all I want right now is , the diagonal matrix of . OK, the eigenvalues are

and is

On second thought, I really should give you the orthonormal eigenvector matrix for reference:

The matrix A is

The new data Z is given by

Finally, the cross-dispersion between Z and X is given by

And that is the same as A, as it should be.

We continue with the answer to question 1. We found that if we do an eigendecomposition of , then .

What about the matrix Ar, found by weighting the rows of A by the inverse standard deviations?

There was nothing special about the inverse standard deviations. If we weight the columns of X by a diagonal matrix , we can find a corresponding matrix Ar.

Let’s work that out. I am going to take weights (1,2,4} in place of the inverse standard deviations. That is, I take the diagonal matrix

and scale the columns of X and the rows of A:

(Perhaps now is a good time to remind you that we can scale the columns of a matrix by post-multiplying it by a diagonal matrix of the scale factors; and we can scale the rows of a matrix by pre-multiplying it by a diagonal matrix.)

i expect that

.

Compute the RHS:

and that is, indeed, .

So, the construction of Ar using the inverse standard deviations could be done with any set of weights instead, and applied to any dispersion matrix, not just the A = Ac from centered data.

In other words, I know two kinds of transition matrices that are dispersion matrices: A between Z and X, Ar (Ad in this case) between a newer Z and the older X.

I do not have an “iff” result. I can’t guarantee that these are the only kinds of matrices that work.

Question 2. Is v a dispersion matrix?

No, at least not between its new data Z and the old X, i.e. not the way A is. (I wonder if there’s a way to make it happen. Would it ever be worthwhile if it were possible?)

This time, let’s first confirm the failure numerically.

We need new data from v instead of from A. Rather than destroy the Z’s, call the new data Y. We should have

,

i.e.

,

i.e.

.

That last equality corresponds to , so we’re good.

Here’s the new data Y from old data X using the transition matrix v:

now compute , i.e. in this case,

and that is to be compared with v – but I’ve already done the algebra, and I know they should have proportional columns, so I took the ratio instead of the difference, i.e. here is , term by term:

Those should look familiar: they are, as they should be, the eigenvalues of :

So the cross-dispersion matrix between X and Y is a column-weighted version of v, and the weights are the eigenvalues.

Time to do the algebra. This time, Y = Xv. We start with the dispersion matrix…

and that’s what we got: the cross-dispersion is the column-weighted v, and the weights are the eigenvalues .

PCA / FA Basilevsky: with data

Introduction

I am looking into Basilevsky because he did something I didn’t understand: he normalized the rows of the Ac matrix (which I denoted Ar). We discussed that, and we illustrated the computations, in the previous two posts. But we did those computations without having any data. I want to take a closer look, with data.

In contrast to As and Ac, which are eigenvector matrices, his Ar matrix is not. Nevertheless, as I said, his Ar is not without some redeeming value. In fact, all three of the A matrices have the same redeeming value.

I will show, first by direct computation and then by proof, that each of these A matrices is the cross covariance between X data and Z data.

(That doesn’t mean I want to use his Ar matrix; all it means is that I learned something new about an A matrix.)

Recall the raw data of example 9. We’re about to go get the A matrices and the new data Z for both standardized and centered data.

Yes, we’ve seen all these forms of the data before, but I want this post to be self-contained.

Now, I’m going to switch to Basilevsky’s notation, using Z for F^T. If you will, we’re mixing Harman’s model

where Z = X^T, and F^T is the new data (“scores”) WRT the transition matrix (“loadings”) A; and Jolliffe’s model

where Z is the new data (“scores”)….

We start with Harman’s model

.

eliminate Z:

,

and we know that F^T is the new data corresponding to X WRT the transition matrix A .

Now we reintroduce Z, redefining it as Jolliffe uses it, for the new data; that is, we let

and get

.

If I had to do it all over again, I would probably not have used Basilevsky’s notation. This is the notation I had for the previous two posts, but i’ll admit, it’s confusing.

But for this post, as for the previous two, that is our model.

First, let’s go get the standardized and the centered data (“X”), the corresponding A matrices, and the corresponding new data (“Z”).

Basilevsky Standardized

If you are comfortable with these calculations – which we’ve seen before for this very data – you should move right along. If, on the other hand, you need a refresher, here it is. And if you need to see the calculations explained, here

I standardize the data and call it Xs:

Get the eigenvalues of the correlation matrix:

Get the SVD (Singular Value Decomposition) of Xs:

.

Form the diagonal matrix of …

Form the weighted eigenvector matrix

Get the scores FT as the first three columns of 2u (where 2 = Sqrt[N-1], N=5). This is as close to the u matrix itself as I need to get. But instead of calling these F^T, we are calling them Z:

Check it by confirming that we have

.

(We do.)

Basilevsky centered

Recall the raw data:

I will center the data and call it Xc. Get the column means…

and subtract them from each column to get Xc:

Proceeding as before, we get the SVD (Singular Value Decomposition) and display w…

Get the eigenvalues of the covariance matrix (and that’s the only change we make after switching from standardized to centered data):

Get the diagonal matrix of of the covariance matrix…

Get the weighted eigenvector matrix

Get the scores F^T as the first three columns of 2u (where 2 = Sqrt[N-1], N=5). except that we are calling them Z, as Basilevsky does.

Check it by confirming that we have

.

(We do.)

Key: the A matrices are cross-covariances

We have As and Ac:

We have new data Zs and old Xs:

and old:

Now, what is the cross-covariance between Xs and Zs? I need either

or

,

and N-1 = 4. Note that they are the transposes of each other. We compute the former:

Not symmetric. But it’s a cross-covariance, between two different variables, not the covariance of one variable.

Now recall As:

They are the same.

For centered data:

and we recall

Again, they are the same.

Now, get Ar, by normalizing the rows of Ac:

and compute the cross-covariance between Xs and Zc:

0nce again, they are the same. We have good reason, now, to believe that

,

where Z is the new data WRT A (i.e. the scores, as most people define them).

That’s what I set out to show.

We have seen it for As, relating standardized data Xs to the scores Zs; for Ac, relating centered data Xc to the scores Zc; and finally for Ar, relating standardized data Xs to the scores Zc.

Proving it

Let’s take a look at the proofs. The question in the back of my mind is: when is a transition matrix also a cross-covariance matrix? (I will confess that I do not have a universal answer; it appears that I can generalize from the Ar matrix somewhat.)

That is, given old data X and new data Z WRT transition matrix T,

.

when can we conclude that T is the covariance between X and Z?

For either standardized data or centered data X, one argument works. We have

,

where is the diagonal matrix of and V is an orthogonal eigenvector matrix of the covariance matrix c; that is, we have the eigendecomposition

We have the decomposition of X:

.

For simplicity, i’m going to assume that A is invertible (i.e. that is invertible; i.e. that all the eigenvalues are nonzero): then

and I note that

.

Let me name those three equations: we have an eigendecomposition of the covariance matrix, the decomposition of X, and the definition of A.

Now we compute the cross-covariance. There may be simpler ways, and there are certainly other paths to the same answer, but this works for me. As I said, I’m going to assume that A is invertible, because for one thing we can cope if it is not; and for another, if it is not, we would have fewer Z variables than X variables.

(the decomposition of X)

(definition of the covariance matrix)

(the eigendecomposition of c and definition of A)

(simplify)

(definition of A)

QED.

The eigendecomposition is crucial. Without it, we are left hanging at

,

Now let us try to prove it for Ar. It’s a surprisingly simple consequence of what we just did. We need only one new equation, the relationship between standardized and centered data:

,

which says that we get standardized data from centered data by dividing each column by its standard deviation; and we recall that scaling columns can be accomplished by post-multiplication by a diagonal matrix. (After all, that’s how we get the A matrix from the V matrix.)

What we actually need is the transpose:

.

Let us compute the cross covariance of Xs and Zc.

(Xs in terms of Xc)

(by the previous result for Ac!)

(by definition of Ar)

QED.

The key, then, was that Ar was to Ac as Xs was to Xc (OK, not exactly: scale the rows of one and the columns of the other). I don’t know of any other suitable relationships, but if we found one to use, we should apply it to Xc and Ac.

Let me be more specific. We never really used the fact that was a diagonal matrix of standard deviations. If we had constructed completely different data Xd by scaling the columns of centered data Xc, using an arbitrary diagonal matrix , so that

,

and if we had defined a new A matrix by scaling the rows of Ac using the same diagonal matrix:

,

then we would find that Ad was the cross covariance of Xd and Zc.

I need to look at this some more.

PCA / FA Basilevsky: what he did.

Introduction

The previous post discussed something interesting that Basilevsky did. (Search the bibliography, he’s there.) I can’t say I like it — because it leads to a basis which is non-orthogonal, and whose vectors are not eigenvectors of either the correlation or covariance matrices.

But I had to understand it.

I don’t know how widespread it is nowadays, but even if Basilevsky is the only author who does it, it’s another example of the lack of standardization (no pun intend, I swear) in PCA / FA. This branch of applied statistics is like the mythical Wild West: everybody’s got a gun and there are bullets flying all over the place. Law and order have not arrived yet.

OTOH, it’s nice to find something different in just about every book I open.

Let me set the stage again. What we have is the following 3 models.

.

where X is data (variables in columns, observations in rows, and there are N rows), Z is principal components (PCs), and A is constructed from an eigenvector matrix V using the diagonal matrix of .

Now is a good time, before we need to be reminded of it, to emphasize that all three models are of the same form:

.

In the first model,

,

Xc is centered data, and

,

where Vc is an orthogonal eigenvector matrix of the covariance matrix :

.

In the third model,

,

Xs is standardized data, and

,

where Vs is an orthogonal eigenvector matrix of the correlation matrix :

.

In the second model,

we have the standardized data Xs as in the third model, but the PCs Zc from the first model. The transition matrix Ar may be found by normalizing the rows of Ac:

,

where is a diagonal matrix of the standard deviations () of Xc.

The example

Basilevsky gives us a covariance matrix c. (So we have no data, for this example.)

We will want both the correlation matrix r and the diagonal matrix , so let’s get them. is to be a diagonal matrix of the square roots of the variances, hence of the square roots of the diagonal of c. (i should call it s, because it’s made up of sample standard deviations, but I don’t want “s” as well as “As”, “Vs”, etc.) Here it is:

Once we have , the fastest way for me to get the correlation matrix is

:

From the covariance matrix:

So let’s do the eigendecompositions. First from the covariance matrix, which will give us the first model. We get eigenvalues…

an eigenvector matrix…

These are not quite Basilevsky’s numbers. The 4th eigenvector is off, by as much as .0014 . ah, the 2nd is off too, in two terms. Not great, but not a big deal. (Actually, I want to check his numbers, but not today. I will be using my numbers for eigenvectors and eigenvalues.)

(I suspect that most packages which do PCA / FA use iterative schemes to find one eigenvector after another; I’m getting used to seeing more and more errors as we move to the right in the eigenvector matrices. I think they’re suffering from accumulated round-off errors. Of course, I generally take it for granted that my answers are the right ones, but see below.)

Now I’m going change the signs to agree with Basilevsky:

We will discover that Basilevsky should have changed the sign of the second column, too, but he didn’t so I won’t either.

Let’s keep going. We get the diagonal matrix of …

and

Basilevsky then displays equations for the variables, using Vc not Ac:

What he has written is

i.e.

,

where the Zu are unstandardized, in contrast to Zc and Zs which are standardized. (The Zu were computed from Vc, not from Ac.) I remark that Basilevsky did not compute Ac.

From the correlation matrix:

Now let’s do the correlation matrix, which will give us the third model. Here are the eigenvalues…

and eigenvectors…

Let’s stop there for a moment.

I want to compare the two V’s: recall Vc…

He says, “… even though is positive as a covariance loading, it becomes highly negative as a correlations loadings [sic].”

He correctly points out that the 2nd column of Vc and the second column of Vs would have the same signs if one were multiplied by negative 1, and correctly concludes that we can take care of the sign of by changing the sign of the second column. (At least, I hope that what he said. And is his notation for an element of what i’m calling V.)

Unfortunately, he missed the fact that the 3rd columns cannot be made to have the same signs. The 3rd column of Vc has 2 negative signs; the 3rd column of Vs has only 1.

That is, even though is positive, is negative, when all other signs are the same.

This serves as a reminder that Vc and Vs, eigenvectors of the covariance matrix and of the correlation matrix, do not have a simple relationship to each other.

i will also point out that both Ac and Ar will be derived from Vc by scaling by positive numbers, and As will be derived from Vs by scaling by positive numbers, so this sign discrepancy between the (2,3) elements of Vs and Vc will propagate to the (2,3) elements of Ac and Ar versus As.

Now, lets get back to doing our thing. Get …

and As…

Yes and yes, Basilevsky and I agree. Even to all his signs.

We use As and get the equations

These Xs are standardized, and so are the Zs, in contrast to the Xc and Zu of the first set of equations:

Basilevsky’s model:

Basilevsky has shown us that if we define

,

then we would have the second model,

.

here’s Ar…

And here’s the corresponding equations…

followed immediately by the second set computed earlier…

There are some satisfying similarities. The first columns differ significantly only in the last entry, and even that is .2 vs .36 . For the second column, as we said, if we had changed its sign, all the signs would match up, and in particular the single large terms would then be +0.9324 and +0.9575 .

I should emphasize that he calls both Ar and As, “correlation loadings”, at one point or another.

We could also severely round both As and Ar to confirm their similarity:

As he said, we shoulda changed the sign of the second column of Vc.

Without data, there isn’t a whole lot more we can do here. But I do want to show you one last thing: let’s take a closer look at Ar. We computed it from

and the rows of Ar are supposed to be of unit length. As usual, the fastest way to compute the lengths (more precisely, the squared lengths) of vectors in an array A, is to compute either or . The former gives us dot products of all the rows with each other, including with themselves, while the latter gives us all the dot products among the columns.

So if we compute , the diagonal elements should be 1. We see that they are:

In fact, we see that is equal to our old friend the correlation matrix r:

Don’t be distracted, however, by his scaling the rows. The new basis – which gives us the PCs Zc from Xs – is specified by the columns of Ar, just as the columns of Ac and As were the bases that gave us Zc from Xc, and Zs from Xs. (As I said, all three models are of the same form: . And as I’ve said before, we should keep our eyes on the linear algebra.)

So what does the basis Ar look like? (I’ve said some terrible things about it; let me justify them.) For this, we compute :

That the diagonals are not 1 says the columns are not of unit length; that the off-diagonal elements are not zero says the columns are not mutually orthogonal.

Ar is not a very nice basis at all.

I think it has some redeeming social value, but we’ll look at that when we have data.

Looking at eigenvectors

Let’s check one more thing. The columns of Vc are eigenvectors of the covariance matrix: we should have

.

Do we? We do. (When I compute the difference, I get zero, to within .)

And columns of Ac are eigenvectors of the covariance matrix, too:

.

Similarly, columns of Vs and As are eigenvectors of r, and we should have both

and

.

We do.

(This is why I’m so sure my answers are right: I’ve checked the eigendecompositions.)

What about Ar? Even supposing that its columns are eigenvectors of c, we don’t know what the eigenvalues are. No problem, just see if

r Ar

is proportional to Ar. (That is, see if columns are proportional; see below.)

We have

and when we divide, element by element, by Ar, we get

so we conclude that the columns of Ar are not eigenvectors of r. (Put that the other way: if columns of Ar are eigenvectors, then applying r to them should yield columns which are proportional to the eigenvectors, i.e. proportional to Ar.)

Are the columns of Ar eigenvectors of c? here’s c Ar…

divided by Ar…

Again, no; the columns of Ar are not eigenvectors of c.

Oh, do you need to see how that calculation works when it comes out right? here’s c Ac…

and dividing by Ac gives us…

which says the 4 eigenvalues for the 4 columns are

and that’s exactly what we got a long time ago:

The purpose of this post was to demonstrate Basilevsky’s computations. Next time, I’ll do these kinds of calculations with data.

PCA / FA Basilevsky on standardizing. Discussion.

Introduction and review

Basilevsky presents an extremely interesting idea. For all I know, it’s become common in the last 10-20 years, but I hadn’t seen it in any of the other books we’ve looked at.

I’ll tell you up front that he’s going to normalize the rows of an A matrix, specifically the A matrix computed from the eigendecomposition of the covariance matrix.

I’ll also tell you up front that I don’t see any good reason for doing it, but I’m not averse to finding such a reason someday.

Suppose we have centered data X with variables in columns, and N rows (observations). The covariance matrix is ), and we find its eigendecomposition. We get a matrix V having the eigenvectors as columns, and we construct a diagonal matrix whose entries are the square roots of the eigenvalues.

That is, we have the decomposition

.

Then we define the matrix : each column is an eigenvector, of length .

For the next few posts, I will use a notation of Basilevsky (and Jolliffe): the principal components will be denoted by Z. (Yes, Harman and I almost invariably use , but I need to abandon that notation for a while. Sorry.)

This coincides with Jolliffe’s notation, which was to write

,

with Z as the principal components, X the data, and V the orthogonal eigenvector matrix.

As we have observed many times, the starting point for making sense of that is the change-of-basis equation for a vector:

,

where and are column vectors, x is the old components, z is the new components, and V is the transition matrix. (Its columns are the old components of the new basis vectors. Using the symbol V is not an accident: the matrix V from the eigendecomposition is a transition matrix.)

Applied to the entire matrix X, the change-of-basis equation is

Transpose:

Solve for Z:

But V is orthogonal, , so :

.

Among the very first things we learned in PCA (in Harman) were two: first, that the Z’s we just defined have maximally redistributed the variance of the X’s: the 1st Z variable has the largest variance of any linear combination of the X’s; the 2nd Z variable has the 2nd largest variance of any other linear combination of the X’s, etc. (All subject to the constraint that the total variance of the Z’s is equal to the total variance of the X’s.)

Second, we learned that if we used the transition matrix , then the Z’s defined by it would be both standardized and uncorrelated.

In our present notation (our is Harman’s Z, and our is Harman’s F), we would write Harman’s model as

which, column by column, is precisely the change-of-basis equation

,

with transition matrix A instead of V.

Or we could write

.

(I will try not to mix these up.)

Let me emphasize that the two Z’s in

and

are different. In what follows, I will distinguish them by subscripts (Zu for the first, Zc for the second.)

(It is true that the second set of Z’s could have been found from the first set: just standardize the first set! I had never thought to ask that or check it before, but it would be rather annoying if it were not true. See below.)

We will, in fact, be dealing with both centered data Xc and the corresponding standardized data Xs; we will have corresponding orthogonal eigenvector matrices Vc and Vs; corresponding A matrices Ac and As, and a third one Ar; we will have 3 Z matrices, although a fourth exists in principle.

Zu will be the unstandardized Z from Vc:

Zc will be the standardized Z from Ac:

Zs will be the standardized Z from As: .

(I have tried to minimize the occurence of Z’s as opposed to Zs, and X’s versus Xs, but I can’t reduce the number to zero. Zs and Xs are specific matrices, as opposed to collections of Z’s and X’s, which have apostrophes)

A new basis

We will show that

,

where Ar can be computed from Ac.

That last is a mix of apples and oranges: standardized Xs, but standardized Zc from the covariance matrix, instead of either standardized Xs and standardized Zs from the correlation matrix, or centered Xc and standardized Zc from the covariance matrix.

As I intimated at the beginning, the Ar is Ac with its rows normalized to length 1.

Let’s work that out. The key is that each row of Ac has a sample std deviation as its length.

Let’s see that. To compute the mutual dot products of rows of Ac, we could compute

.

The diagonals of the result would be the squared lengths of rows of Ac (the dot products of rows with themselves).

OK. Let me drop the subscript c for the moment:

,

where c is the covariance matrix.

(the eigendecomposition was .)

So the diagonal elements of are not only the squared lengths of the rows, but also the sample variances of Xc.

We want to scale each row of Ac by its length. To scale columns of V, we post-multiplied by the diagonal matrix . To scale rows of Ac, we will premultiply by a diagonal matrix of inverse standard deviations.

So let = diagonal matrix of square roots of variances, and let be its inverse. (Ignore the possibility of zero variances; after all, we do that every time we talk about the correlation matrix.)

We want to scale the rows of Ac to unit length to get Ar, so we define

.

We have the relationship between Xc and Zc:

.

We standardize Xc by scaling columns by standard deviations, so

.

FInally, transposing

,

QED.

Let me lay out all three equations again (I really like to see them all together):

.

That’s how I would present Basilevsky’s work. He never introduced Ac, and he computed Zc from the unstandardized Zu:

Standardizing Zu

Yes, as I said, the standardized Zc can be computed by standardizing the Zu; and the variances of the Zu are given by the eigenvalues of the covariance matrix . It’s easy enough to confirm.

From our usual definition of Zc,

, so ,

and the old definition of Zu

,

we could write

i.e.

,

QED.

Summary and query

Overall, then, I learned two things. One, that standardized Zc and unstandardized Zu, defined by

and

were related in the sensible way; we could get Zc by standardizing Zu.

Two, and more importantly, I learned that we could write

,

where

.

Fine.

Now, why would we want to?

I don’t know.

Basilevsky got there by looking at

and saying he wanted to standardize both Z and X.

But we already know how to do that:

,

with standardized Xs and standardized Zs.

Why doesn’t he do that?

He does.

And he knows and shows that , and , so by using Ar he has gotten standardized Zc which are different from standardized Zs.

I am perfectly happy with Zs derived from Xs. I am perfectly happy with Zc derived from Xc. I have no idea why we would want to use Ar to relate Zc to Xs. It’s true, but why do we care?

OK, Ac is not an orthonormal basis, but neither is Ar, so that’s not an improvement. OK, the columns of Ac do not have unit length, but they are mutually orthogonal and they are eigenvectors of the covariance matrix. In contrast, the columns of Ar are neither mutually orthogonal , nor are they eigenvectors (of Xc or of Xs).

I find myself wondering if there is some reason for liking this non-orthogonal basis Ar. Does its failure to be orthogonal tell us something? But it didn’t give us anything new. It says (once again) that

,

but Zc is found from Xc and the eigenvectors of the covariance matrix:

,

and Xs is found by standardizing Xc. We don’t need Ar to get the two things it relates.

I think Ar is cute, but I need some reason for going to a basis which isn’t at least orthogonal. Heck, I need a reason for going from the orthogonal Vc to a basis (Ac) whose vectors are not of unit length, but the reason is that it provides standardized Zc.

Finally, I need some reason for abandoning the eigenvectors. I’m in no hurry to substitute Ar for either Ac or As. The eigenvectors have significant properties; what does Ar have to offer that makes up for losing those properties?

If anyone knows….