PCA / FA Example 4: Davis. R-mode & Q-mode via the SVD.

let’s finally do this using the SVD. i need to do one terrible thing: where davis writes U and V, we need v and u, resp. (look, i couldn’t reliably keep translating davis’ equations in my head, so i had to use his notation; by the same token, i can’t reliably translate the SVD over and over again. thank god he used UC, upper case.)
if the correspondence had been u ~ U and v ~ V, the translation would have been trivial. unfortunately, the correspondence is
u ~ V
v ~ U.
(from the SVD posts, you recall that given X = u \ w \ v^T we conclude that v is an eigenvector matrix for X^T\ X, and u is an eigenvector matrix for XX^T. and, just as important, the nonzero values of w are the nonzero \sqrt{\text{eigenvalues}}.)
from the SVD of X, X = u \ w \ v^T
and the definition of A^R = U \ \sqrt{\text{eigenvalues}}
i conjecture that we could define A^R = v \ w^T.
from the definition of A^Q = V \ \sqrt{\text{eigenvalues}}
i conjecture that we could define A^Q = u \ w .
then from S^R = X \ A^R
we get S^R = u\ w\ v^T\ v \ w^T = u \ w \ w^T ,
since v is orthogonal.
and from S^Q = X^T\ A^Q
we get S^Q = v \ w^T \ u^T \ u \ w = v \ w^T \ w,
since u is orthogonal.
(almost) finally, from u \ w = \ A^Q \ and S^R = u \ w \ w^T
we get S^R = A^Q\ w^T\ ,
which is the first dual relationship between Q-mode and R-mode.
finally, from v \ w^T = A^R and S^Q =v \ w^T\ w
we get S^Q = A^R\ w,
which is the other dual relationship.
so we see that davis’ duality delationships make sense.
let me summarize the SVD equations for the As and Qs:
A^R = v \ w^T
A^Q = u \ w
S^R = u \ w \ w^T = A^Q\ w^T
S^Q = v \ w^T\ w = A^R\ w.
BTW, i’ll compute using S^R = A^Q\ w^T and S^Q = A^R\ w because they involve two matrices instead of three.
it’s time to actually compute the SVD.
here’s u, v, and w:
u = \left(\begin{array}{cccc} -0.801784&0&0.597511&0.0110879\\ 0.267261&-0.816497&0.351732&0.371738\\ 0.&0.408248&-0.016937&0.912714\\ 0.534522&0.408248&0.720401&-0.169237\end{array}\right)
v = \left(\begin{array}{ccc} 0.816497&0&0.57735\\ -0.408248&-0.707107&0.57735\\ -0.408248&0.707107&0.57735\end{array}\right)
w = \left(\begin{array}{ccc} 9.16515&0&0\\ 0.&3.4641&0\\ 0.&0&0\\ 0.&0&0\end{array}\right)
(we could check that u and v are orthogonal, and in fact i did check that. OTOH, in contrast to eigenvector matrices, mathematica says u and v will always be orthogonal.)
we should compare u with V; here’s V:
V = \left(\begin{array}{cccc} 0.801784&0&-0.597614&0\\ -0.267261&0.816497&-0.358569&-0.365148\\ 0.&-0.408248&0&-0.912871\\ -0.534522&-0.408248&-0.717137&0.182574\end{array}\right)
most interesting. the 3rd and 4th eigenvectors are different although the 3rd one isn’t very different. what’s going on? well, they are two eigenvectors associated with the same eigenvalue (0), and in fact they span the null space of XX^T. (i think this is true for any repeated eigenvalue, that we could select any orthonormal basis for the multi-dimensional eigenspace, but i need to confrm that. in any case, it’s certainly ok here because the eigenvalue is zero and we’re dealing with the null space.)
what is significant, however, is that u and V have different signs on the two columns with nonzero eigenvalues, the 1st and 2nd.
now compare v and U:
v = \left(\begin{array}{ccc} 0.816497&0&0.57735\\ -0.408248&-0.707107&0.57735\\ -0.408248&0.707107&0.57735\end{array}\right)
U = \left(\begin{array}{ccc} 0.816497&0&-0.57735\\ -0.408248&-0.707107&-0.57735\\ -0.408248&0.707107&-0.57735\end{array}\right)
they differ only in the sign of one column that matters (because it has nonzero eigenvalue), the 1st. and that’s the problem: U and V are incompatible in the sign of the 2nd column. we don’t have to match davis; we just have to change the sign of one column if we want the expected correspondence between U ~ v and V ~ u.
let me just show the incompatibiity. from the SVD X = u \ w \ v^T,
i can always write w = u^T\ X\ v.
the question is: given U and V computed separately by eigendecompositions, can we use them in the SVD? let’s see what happens if we substitute U for v and V for u. that is, we compute
V^T\ X\ U.
V^T\ X\ U = \left(\begin{array}{cccc} 0.801784&-0.267261&0&-0.534522\\ 0.&0.816497&-0.408248&-0.408248\\ -0.597614&-0.358569&0&-0.717137\\ 0.&-0.365148&-0.912871&0.182574\end{array}\right)
x \left(\begin{array}{ccc} -6&3&3\\ 2&1&-3\\ 0.&-1&1\\ 4&-3&-1\end{array}\right) x \left(\begin{array}{ccc} 0.816497&0&-0.57735\\ -0.408248&-0.707107&-0.57735\\ -0.408248&0.707107&-0.57735\end{array}\right)
= \left(\begin{array}{ccc} -9.16515&0&0\\ 0.&-3.4641&0\\ 0.&0&0\\ 0.&0&0\end{array}\right)
close. there’s just one little problem: the nonzero elements of w are supposed to be positive. this is what i mean when i say that U and V need not be compatible in general, and in this case they are not. after all, U and V came from independent eigendecompositions.
moving on, we compute A^R = v \ w^T
v \ w^T = \left(\begin{array}{ccc} 0.816497&0&0.57735\\ -0.408248&-0.707107&0.57735\\ -0.408248&0.707107&0.57735\end{array}\right) x \left(\begin{array}{cccc} 9.16515&0&0&0\\ 0&3.4641&0&0\\ 0&0&0&0\end{array}\right)
= \left(\begin{array}{cccc} 7.48331&0.&0.&0.\\ -3.74166&-2.44949&0.&0.\\ -3.74166&2.44949&0.&0.\end{array}\right)
we should compare that to what i got before.
\left(\begin{array}{ccc} 7.48331&0.&0.\\ -3.74166&-2.44949&0.\\ -3.74166&2.44949&0.\end{array}\right)
oh, dear. this emphasizes that i changed the definition of A^R when i went from an eigendecomposition to the SVD. OTOH, so did davis.
if A^R is defined as the \sqrt{\text{eigenvalue}} weighted eigenvector matrix U, then it must be the same size as U. we would get it by post-multiplying U by a 3×3 diagonal matrix of eigenvalues. instead, mine is bigger than 3×3, while davis’ is smaller.
but if it is defined as A^R = v \ w^T, then A^R is not the same size as v (or U) because w is not square.
i think this is just a matter of terminolgy. conceptually, i can best see the duality between Q-mode and R-mode using the SVD.
OTOH, conceptually, if i want the \sqrt{\text{eigenvalue}}-weighted eigenvector matrix, i just post-multiply v by a diagonal matrix. in practice, i would just cut down A^R to get something the same size as v.
to put that another way, if we use davis’ square matrix of \sqrt{\text{eigenvalues}}, then we still see the duality and his A^R is the nonzero \sqrt{\text{eigenvalue}}-weighted eigenvector matrix.
(BTW, we’ve seen that square matrix of \sqrt{\text{eigenvalues}} before, in the derivation of the SVD. we needed to invert it. maybe i will start using it, too.)
if this is getting confusing, fall back on: what we have are u, w, and v; everything else follows from them.
moving on again, we compute A^Q = u \ w :
u \ w = \left(\begin{array}{cccc} -0.801784&0&0.597511&0.0110879\\ 0.267261&-0.816497&0.351732&0.371738\\ 0.&0.408248&-0.016937&0.912714\\ 0.534522&0.408248&0.720401&-0.169237\end{array}\right)
x \left(\begin{array}{ccc} 9.16515&0&0\\ 0.&3.4641&0\\ 0.&0&0\\ 0.&0&0\end{array}\right)
= \left(\begin{array}{ccc} -7.34847&0.&0.\\ 2.44949&-2.82843&0.\\ 0.&1.41421&0.\\ 4.89898&1.41421&0.\end{array}\right)
then we compute S^R = A^Q\ w^T:
A^Q\ w^T = \left(\begin{array}{ccc} -7.34847&0.&0.\\ 2.44949&-2.82843&0.\\ 0.&1.41421&0.\\ 4.89898&1.41421&0.\end{array}\right) x \left(\begin{array}{cccc} 9.16515&0&0&0\\ 0.&3.4641&0&0\\ 0.&0&0&0\end{array}\right)
= \left(\begin{array}{cccc} -67.3498&0.&0.&0.\\ 22.4499&-9.79796&0.&0.\\ 0.&4.89898&0.&0.\\ 44.8999&4.89898&0.&0.\end{array}\right)
(i see no point to confirming that S^R = X \ A^R ; that was our definition.)
now we compute S^Q = A^R\ w:
A^R\ w = \left(\begin{array}{cccc} 7.48331&0.&0.&0.\\ -3.74166&-2.44949&0.&0.\\ -3.74166&2.44949&0.&0.\end{array}\right) x \left(\begin{array}{ccc} 9.16515&0&0\\ 0.&3.4641&0\\ 0.&0&0\\ 0.&0&0\end{array}\right)
= \left(\begin{array}{ccc} 68.5857&0.&0.\\ -34.2929&-8.48528&0.\\ -34.2929&8.48528&0.\end{array}\right)
we’ve done all the computations. (that’s an understatement: we done them three or four different ways!)
next we will look at the interpretations.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: