I hope to put out a summary of multi-collinearity this Monday… but, if necessary, I already have a different post written – a sketchy introduction to group theory. Finishing it off last weekend didn’t leave much time for other mathematics.

I did pick up chemical reactions again… namely, finding mechanisms – sequences of elementary reactions – that explain observed reaction rates… specifically, the dependence of rates on composition. It’s beginning to make sense, now that I’ve looked at it after a long layoff.

I also spent a little while looking through my books on spectrum analysis (that is, frequency domain analysis of time series). For this, I feel like one of the mythical 3 blind men trying to understand an elephant. I like having more than one way to come at something, but for spectrum analysis, there seem to be too many different ways of approaching it. (I know, first understand one, or maybe two together, and then move on to more.)

A 2nd book by Vladimir I. Arnold came in: “catastrophe theory”. The back of the book says that it “…provides a concise, non-mathematical review of the less controversial results in catastrophe theory.”

I beg to differ. One might call it “non-rigorous” but it is hardly nonmathematical.

The most significant thing I got out of it was: “On odd-dimensional manifolds there can be no symplectic structures, but instead there are contact structures.”

The point is that symplectic structures can only be defined on even-dimensional spaces – I knew that… the prototype is the even-dimensional phase space of Hamiltonian mechanics. And I have heard of contact structures… without realizing that there was any relationship. At this point, that’s about all I know.

So let me go learn some more about something.

• Can we fit a 7th degree polynomial to our 8 data points? Yes.
• We can do it using regression.
• We can do it using Lagrange Interpolation.
• Did Draper & Smith use the same orthogonalized data? Yes, but not normalized.
• How did Draper & Smith get their values? They looked them up.
• Were their values samples of Lagrange polynomials? No.

The bottom line is that starting with half-integral values of x, all I need is the Orthogonalize command, to apply Gram-Schmidt to the powers of x. I did that here. I don’t need to look up a set of equations or a pre-computed table of orthogonal vectors. Furthermore, I can handle arbitrary data which is not equally-spaced.

Can we fit a 7th degree polynomial to our 8 data points? Yes.
We can do it using regression.

Let me finish by showing you what happens if I add x^7. Here’s the result of centering the 8 years 1986..1993:

Here’s the dependent variable again:

Now compute powers of x, up through x^7, and append the y values… and set names:

Run stepwise… (I’ll remark that I almost always run stepwise rather than a single specific model. It gives me context for the fit of interest, in general.)

Oh, come on! That should work.

It did. We still get an error on the last regression, but we already know there is no problem with the first six fits (other than multicollinearity). Let’s check the inversion on the last fit: no warnings.

What’s going on?

Let’s try to look at the details of this regression.

Ah ha! That 1/0 occurs because n – k = 0: we have 8 observations and 8 variables. Notice the “1″ sitting between two “Missing”? That’s the RSquared… it was computed just fine, thank you.

Since R^2 = 1, the sum of the squared errors must be zero, and if the sum of non-negative numbers is zero, then each one of them – each one of the squared errors – must be zero. We have a fitted equation with zero errors – but we also have n – k = 0 messing up many of our usual output results.

(I had originally run the previous posts using powers thru x^7… but the error message because n = k has nothing to do with the error messages that came from extreme multicollinearity, and I didn’t want to proliferate issues.)

We did indeed put a 7th degree polynomial exactly thru 8 points. Here’s the fitted equation… and replace Xi by X^i (and name the resulting polynomial “g”)…

Let’s take a look at it.. and the data points, too:

I wouldn’t really forecast using this… but let’s go four years out anyway. That is, instead of drawing the polynomial between -7/2 and +7/2, I go out to +15/2.

Yikes! That’s what x^7 will do to you. The original data is completely blown away by the absurdly large (negative) growth of the high-degree polynomial.

In summary,
we can use LinearModelFit to exactly fit the data
n = k will mess up some of the usual outputs

Oh, I could have just called for the one fit:

We are spared the error messages… except that they’ll show up as soon as we call for anything involving n-k:

(So it’s my fault they showed up at all: stepwise asks for information involving n-k, and that’s what generated the messages.)

We can do it using Lagrange Interpolation.

You may recall something called Lagrange Interpolation. It’s a formula for a polynomial that goes exactly thru a set of points. (It is crucial for Lagrange Interpolation that the x values be distinct. Similarly, the least-squares fit cannot go through all 8 points if the x values are not distinct.)

Mathematica® will do a Lagrange Interpolation for us. Here are the (x,y) pairs:

And here’s the command and the output for a fitted polynomial… and the answer rewritten… and, for comparison, the answer from the OLS fit (that’s why I named it – so I could refer to it here):

They are the same, as they should be.

Here is a a video showing Lagrange Interpolation. (As I said in the diary post of Jan 21, I have found an error in a different video lecture by this speaker, so keep your eyes open.)

Did Draper & Smith use the same orthogonalized data? Yes, but not normalized.

Let me show you what D&S used. The following matrix is one of several tables; it’s specifically for 8 observations. It is accompanied by equations for “orthogonal polynomials”. Basically, someone sat down and worked out equations rather than simply orthogonalizing the powers of x numerically.

It has mutually orthogonal – but not orthonormal – columns; computing Z’Z gets me all the dot products of the columns with each other… since all off-diagonal terms are zero, the columns are orthogonal to each other.

We understand that the square roots of the diagonal entries are the lengths of the vectors. Its second column

Is there any significant difference between their data matrix and mine?

Only the scaling. Take theirs and divide by mine, and square. Here is my orthogonal matrix from the previous post; its starting point was x = {-(7/2),-(5/2),-(3/2),-(1/2),1/2,3/2,5/2,7/2}…

The Orthogonalize command (applied to the transpose) gave us…

Divide their matrix by my matrix, term by term, and then square the entries:

Those values are precisely the diagonal elements we saw when we computed Zo’Zo: {8,168,168,264,616,2184,264}.

In other words, if I normalize their columns to unit vectors, I will get precisely the same matrix as if I orthogonalize my powers of x – provided I started with the centered data.

(There were any number of ways to show that relationship. The most straight-forward might have been to normalize each of their columns, and shown that the result matched my data…. Instead I chose to simply divide their data by my data, element by element.)

How did Draper & Smith get their values? They looked them up.

They have gone to a lot of trouble to get equations. It turns out they are general in one respect – they work for any number of observations. But they suffer from two defects. One, we have only been given equations through degree 6: if we want higher degrees, we’ll have to work them out or look them up. Two, the equations require that the x values be evenly spaced. But orthogonalizing the data will work whether or not the values are evenly spaced.

Here are the equations they wrote. You will see that each has a multiplicative constant . That’s part of what makes the equations general. (They cite many references for the following information, including a source of Fortran code; they might have computed it all themselves. In any case, I have scanned these equations and two subsequent tables from Draper & Smith.)

Here is the prescription for n = 8:

The numbers on the last row are the . That “2″ under their first column says that the column of integers was obtained from x (half-integers!) by multiplying by 2.

Here’s the code I wrote to implement the equations for n = 8. (It includes the , so don’t worry about reading them from the image.)

(I chose to include the constant term.)

Here’s my output. First, each new power is a new row… then I transpose that answer so that each power is a column… then I compute Z’Z to confirm orthogonality of columns:

Let me show you something very important. Although the equations are general in that they contain “n”, the number of observations, they are implicitly specific in that they require that the entries in X be separated by 1.

Huh?

What happens if we start with 2x (integers) instead of x (half-integers? The resulting matrix does not have orthogonal columns.

So, not only do we have that list of to pre-multiply the equations, but we have to start with integers for n odd, half-integers for n even.

I have a slightly better idea. I can get rid of the – because I can figure out what they must be. (And I could let the code decide whether I need integers or half integers to start… and I could even let the code decide whether I already had half-integers! but I’m only going to show you how to figure out the .

For 7 observations, I need centered integers:

Here’s the output: first with powers in rows, then transposed. In between, I multiplied by an identity matrix…. because I’m going to change some of the diagonal entries:

Well, column 2 has a common factor of 2… as do columns 4 and 6 – whoa! column 4 has a common factor of 4… and column 7 needs to be multiplied by 7… so I change entries in the diagonal matrix, and get:

In other words, I have scaled down the answers as far as possible while still having integers.

Did I get what they would have? Here’s their table for n = 7:

Finally, I had already multiplied column 2 by 2 in the code for n=8… after I divide by 2, I’ve got that . I don’t need his table of … and I could even rewrite the code to remove the lambdas from it.

But I’m not going to bother.

As I said, what they so laboriously construct could be accomplished on a case-by-case basis simply by orthogonalizing the data matrix (starting with integers or half-integers for equally-spaced data). And such a numerical orthogonalization will work even if the data is not evenly spaced. (I would try for approximately integral separation, but I can’t predict how amenable the data will be.)

Were their values samples of Lagrange polynomials? No.

Here, for example, is the Draper & Smith data for the quadratic term:

and they look like this:

Here is a graph of the second-degree Legendre polynomial:

Its integral from -1 to 1 is zero; this means it is orthogonal to the constant function = 1.

Let’s change the horizontal and vertical scales on the polynomial… and show the data points we actually have:

So: our data points are not samples of the corresponding Legendre polynomial.

Now let’s sample the Legendre polynomial. Here is a graph of the Legendre polynomial and 8 equally-spaced samples.

Fine. but what I want to know is: can I scale these samples to be orthogonal to the constant? NO: their sum would have to be zero. If it isn’t already, I can’t make it zero by multiplying by any nonzero scale factor.

Oh, Rip, don’t be silly. You don’t need to scale it – you need to shift it up or down! With a sum of 8 for the eight samples, I need to subtract 1 from each sample:

Now scale the result so that I have integers – without common factors:

OK, I just reproduced the Draper & Smith values.

Unfortunately, this process will not work for the quartic data, so as far as I can tell so far, there’s no easy way to get the discrete orthogonal polynomials from the continuous (Legendre) polynomials.

I’ll repeat the bottom line. If we start with integral or half-integral data, for odd or even number of observations, the Gram-Schmidt orthogonalization algorithm (the Orthogonalize command) will generate effectively the same answers as the equations or tables.

And the Orthogonalize command will work even if the x-values are not equally-spaced.

So just orthgonalize. Don’t go looking for pre-computed tables.

I had been struggling with circuit theory… specifically with RLC circuits. Yes, I solved them long ago in college – although, in fact, I didn’t actually study elementary differential equations until I was a graduate student. (I hadn’t had them before I transferred to Caltech as a sophomore… where they had been covered freshman year.) I had also never seen Laplace transforms until I was a graduate student. I learned both during the 1st course for which I was a TA.

And here I was, all bollixed up, unable to get what I expected.

On the other hand, I had gotten there after deciding that I needed to solve one of the simplest possible LRC circuits, rather than the more complicated ones to which I was trying to apply a slightly more sophisticated method.

Anyway, I woke up one morning dreaming about it, and determined to forget all the complicated stuff… just solve the second-order differential equation for current… and then solve the equation using Laplace transforms. In fact, I solved the voltage balance rather than the second-order ODE using Laplace transforms – and there is one little tricky detail….

A piece of cake. All I had to do was pay attention. Yes, I learned a couple of things. The first was about me and circuits: although I certainly know better, I have a tendency to think of the second-order equation for current as though it were a voltage balance. It isn’t remotely. Grrr.

The second was that if we do want to use a voltage balance and Laplace transforms, then the initial voltage on each capacitor looks for all the world as though the capacitor had been replaced by a constant voltage source.

Which I had read, and not understood. So going back to the beginning had cleared up one of the more advanced issues. Cool.

Okay, it was all elementary… but I’m still delighted to have gotten it all to work out. This is, after all, something my undergraduate is trying to make sense of. When you get confused, simplify the problem. Now I’m ready to tackle more complicated circuits.

Let me close by reviewing the performance of this blog. I should have done it a few weeks ago but I was playing a computer game instead.

From its inception in November 2007 to December 31, 2011… the blog had had 143,500 hits, almost 53,000 spam… and a total of 372 posts.

During 2011 alone, the blog got more than 50,000 hits – more than 4000 almost every month, more than 5000 three times… with a single-day maximum of 309 hits. (The top line is 6000, then we have 5000, and 4000.)

7 posts each had more than 1000 hits in 2011… 2 of them were new: compressed sensing and 5 card draw poker hands – and each is a runaway bestseller.

8 posts have over 2000 hits for all time… one would not expect that list to change quickly… 7 of those posts were the top 7 listed in December 2010… compressed sensing is the new kid on the block.

And with that, let me get about doing and/or writing mathematics.

Oh, I got a new book this week: “Topological Methods in Hydrodynamics”, by Arnold and Khesin….

I have some mathematics done for a post for this Monday… but I keep thinking about other things I might add to it. If I can’t stabilize the content, I may find it difficult to put the post out on time. Duh.

So who’s in charge? The managing editor wants to publish… but the mathematician isn’t ready to call it quits. We’ll see what happens.

While looking for an online reference to a particular formula – I’m horrified that I couldn’t find it in my own library! – I found the following site. It appears to deal primarily with undergraduate numerical mathematics. (And it did have the formula I was seeking.)

There are 2 points I need to emphasize. One, it has an index to a collection of YouTube videos – mathematics, of course.

Two, I have only looked at five of the videos… and I have found a mistake – 2 of them actually – in one “slide”. Unfortunately, I did not find a blog post corresponding to the video… and I didn’t see any way to attach comments to the video. Ah, I did just send an email.

Here’s a freeze-frame:

The full video is here.

The lecturer asserts that a steeper slope implies a higher R^2 because the vertical distance between the data and the fitted line will be larger. Yes, but the vertical distance between the data and its mean value is also larger. The R^2 and the adjusted R^2 will not change. What will change? The estimated variance.

He also said that if the x values were more spread out, then the R^2 would be higher. That’s interesting, because if the x values are more spread out, then the computed slope would be lower… and according to the first point, the R^2 would be lower. In fact, the R^2 will be the same.

It is conceivable that I have completely misunderstood what he means, so let me be explicit. Take a 2-variable data set, x and y, and fit a regression. Now multiply y by 10 and fit another regression: you will get the same R^2 and adjusted R^2; the estimated variance will be 100 times larger.

Now multiply x by 10 (still using the 10 times y) and fit a 3rd regression: you will get the same R^2 and adjusted R^2, and the same estimated variance as for the second case.

So I’m saying that the R^2 and adjusted R^2 are not affected by vertical or horizontal scaling of the data. (We’ve seen that the R^2 and adjusted R^2 are the same for the Hald data and the standardized Hald data – and standardizing data is a change of scale. And I standardized everything, including the dependent variable, so I also had a change of vertical scale.)

(The numerical stability might be affected by changes of scale! We’ve just seen that taking powers of x = 1.986, 1.987, …, 1.993 leads to horrendous inversions of X’X, while centering the x values (i.e. rescaling to -7/2, -5/2, …, 7/2) eliminates the numerical inaccuracies.)

In other words, just as I tell you to be careful reading my posts, I tell you to be careful reading his, or watching his videos. But you might well find something of interest among them….

I started this before I turned my kid and undergraduate loose, so let me get about their play and work… before I turn to Monday’s post.

What we used last time was

That is, I had divided the year by 1000… because, as messy as our results were, they would have been a little worse using the years themselves.

But there’s a simple transformation that we ought to try – and it will have a nice side effect.

Just center the data. Start with the years themselves, and subtract the mean:

I’ll observe that if we wanted to work with integers, we could just multiply by 2. In either case, our new x is not a unit vector.

Oh, the nice side effect? Our centered data is orthogonal to a constant vector.

Let’s see what happens.

As before, I compute the powers of x up to x^6, and I run both forward and backward selection.

Forward and backward did not agree in the previous post – but now they do. Maybe this is promising.

Let’s check the last inversion: no warning, and the identity matrix looks perfect.

This is a nice improvement. I at least am convinced that centering the years is a better way to treat them.

Nevertheless, having been shocked by the previous post, let me look at the singular values of the data matrix…

Considerably better.

Now let’s drop the last two regressions, and see what our criteria would choose.

Regressions 1, 3, 4. Let’s also print 5, and 6 because we couldn’t include them in the selection…. Heck, let’s print all 6… as two sets of three:

The third regression has all t-statistics significant… in fact, x^2 is more significant with x^4 added.

In regression 4, x^6 enters with a t-statistic whose absolute value is greater than 1, but less than 2… so the Adjusted R^2 went up. (Note that I am printing R^2 rather than Adjusted R^2… because Draper & Smith talked about the best R^2 they could get. But we can see the Adjusted R^2 in the output from the forward and backward selections.)

In regression 5, x^3 enters with a low t-statistic, but everything else remains significant. Not until we add x^5 do we see other t-statistics fall below 1. And yet, the degradation is fairly slow.

Do we still have multicollinearity? Yes, indeed! Here are the R^2 computed from the Variance Inflation Factors:

So, just a better choice for x gave us computationally nicer results.

Let’s look at the 3-variable fit. First the predicted values… then the fitted equation… then that equation with X2 replaced by X^2, etc.:

Here’s a picture: the equation and the fitted points. (Hmm. I should have shown the data, too, but I didn’t. In that respect, this fit is as good as what we had last time.)

Visually, that’s pretty satisfying. I wouldn’t forecast very far out, though. Let’s go four years out…

That’s the problem with powers of x – they grow quickly.

orthogonalize

Now, we should expect that orthogonalizing will eliminate the multicollinearity. And we hope that the we’ll still have nicely behaved computations.

As usual, transpose the design matrix… then orthogonalize… transpose again… and confirm that Z is orthogonal… (I use the backward selection because it keeps the variables in the original order.)

Here’s what the orthogonalized data looks like… in particular, we have a constant first column, and a second column which is proportional to the centered data.

Drop the first (constant) column… and append the dependent variable…

Run forward… and backward selection…

They agree.

As before, take only the first four regressions (i.e. 5 variables including the constant), and see what our criteria choose:

Also as before, let’s go ahead and look at all 6 regressions, in two subsets:

We see pretty much the same thing. The third regression has all t-statistics significant… the fourth regression raises the Adjusted R^2 when it brings in a variable with t-statistic greater than 1… then the last two variables enter with insignificant t-statistics.

Orthogonalization hasn’t changed the quality of the fit – in one sense… but it has eliminated multicollinearity.

OK, fine, we’ve eliminated the multicollinearity… but at what cost? Our new data Z are linear combinations of powers of x… how can we look at what we’ve got? More importantly, how would we forecast using the orthogonalized fit?

Let me show the answer first. Here’s the previously fitted polynomial in X… with four predictions from the orthogonalized fit overlaid. That is, the curve comes from the powers of X, while the four points come from the fit using Z. They appear to coincide – and I will confirm that.

OK, let’s do a forecast using the Z fit. Given four new values of x, I want to compute the appropriate values of Z. This isn’t perfectly straight-forward… because I don’t want – I can’t do! – another fit. I want to compute new observations based on the existing definition of Z.

Well, we know how to do that. I let K be the matrix of “old” data (the powers of x, excluding the constant):

Then I let J be the matrix of “new” data (the orthogonal matrix Z, excluding the constant):

We know that the transition matrix T can be computed as

T’ = J’ K

(OK, OK, that’s the transpose T’ of T.)

We also recall that orthogonalizing the data was an affine transformation rather than a linear one; we have

K = J T’ – M.

Well, the fastest way to get M is to compute

M = K – J T’,

which is:

It’s certainly nice to see that M has constant columns. If it didn’t, then we would conclude that there was something wrong with the transformation.

(I also saved one row of M – because I’m going to work with vectors in a moment.)

You may well be wondering what the hell I’m doing. If I have a new observation, say, x = 9/2 – that is, the next number after 7/2 – then I want to know what Z values it generates. Well, first let’s just get the powers of our new x… there’s got to be a slick way of having Mathematica® generate them….

Those are the powers of 9/2.

What are the corresponding Z values? From

J T’ = K – M

we solve for J:

Check it. start with K… get J? I take our first observation… subtract M (actually M1, one row of M)… post-multiply by the inverse of T’… and compare that to the first observation of the J matrix:

OK.

Oh, when we get a new row for the J matrix, we’re going to want to use our orthogonalized fit. Just what is it?

I hope you see the problem: I don’t know how Z2 is related to Z, or how Z3 is related to Z and Z2. When we were working with X, it was trivial: X2 = X^2, X3 = X^3 etc.

Now, take our first forecast, x = 9/2. We subtract M1 from its powers… post-multiply by the inverse of T’… and we now have Z1, Z2, and Z4, so plug them into the equation.

(Yes, we also have Z3, Z5, and Z6, but they’re not used in the equation.)

The final number is the predicted value using our orthogonalized fit.

Now, there’s probably a cleaner way to do three more predictions, but I decided to take it easy and just do them one after another. My next three “old” values are 11/2, 13/2, and 15/2:

Get the set of Z (J) values and the predicted yhat for 11/2:

… and for 13/2:

… and for 15/2:

Now assemble points from those predictions. We know the x-value and we have the predicted y-value, and we get pairs of coordinates.

Now, let us return to our fitted polynomial in powers of X. Let’s see what it predicts for the same four values (9/2, 11/2, 13/2, 15/2).

Those four values are exactly what we got by transforming K to get J. The centered data and the orthogonalized data lead to apparently the same forecasts. Here’s that picture again. Looking at the commands, you see that the curve (f2) comes from the equation in powers of X, but the points come from data4, the fit using Z.

So.

We can eliminate the multicollinearity… but we didn’t change the fit in any significant way. We may, as a result, have more confidence in the validity of the fit to powers of X, but it’s a pain to use the orthogonalized data!

There are some odds and ends I want to show you about this… in what will probably be the next post. In particular, we have in fact used orthogonal polynomials… without the hassle of working them out or looking them up… but they would show us how to get from X to Z.

I also want to show you what happens if I include x^7 in my data matrices.

raw data

I have chosen to divide the years by 1000; in the next post I will do something else.

The output of the following command is the given y values… I typed integers and then divided by 100 once rather than type decimal points.

Fit a polynomial. With 8 data points, I should be able to fit a polynomial of degree 7 in principle, but I’m going to stop at 6. Get the powers 2 through 6 of x:

Run forward and backward selections:

We have considerably different choices between forward and backward regression. Still, let’s just blindly ask what the selection criteria would pick… but we have a problem, as in the previous post: the selection criteria need

n – k -2 > 0,

and with n = 8,

then 6 – k > 0,

i.e. 6 > k;

since k is an integer, we must have

k ≤ 5

– which translates to a constant plus 4 “variables”. We can only run “select” on the first four regressions; the last three have too many variables.

So, take the first four regressions.

OK, let’s look at regressions 1 and 4… (and I have looked at 5 too – it’s worse):

Oh, balderdash. Coefficients of 10^10 – we’re beyond multicollinearity and into linear dependence! At least, that’s my first reaction. But we will see. That 4-variable regression has R^2 = .993 and estimated variance 20% of the 1-variable regression.

Let’s check the inversion of X’X. It turns out that even regression #2 is not reliable: We get a warning message… and what should be an identity matrix is off by 10^-4.

Are the backward selections any better?

No, not really. The second regression doesn’t get the inverse of X’X very accurately.

Once I know X’X is being inverted somewhat inaccurately, there’s no reason to check the VIF R^2. We unquestionably have multicollinearity. What the heck? Do it anyway.

Under the circumstances, however, maybe I’d better check the first regressions, too. The inversion is good, for forward selection…

and for backward selection:

So, at least two regressions involving just one variable are reliable… I conjecture that all such regressions would be reliable; and at least two regressions involving two variables look unreliable… and I conjecture that all such are. But I’m not going to look at all of them.

While we’re exhibiting signs of multicollinearity, let’s look at the singular values of the data matrix. It’s dimensions are…

For an 8×6 matrix of full rank, we would get 6 nonzero singular values. We got only 4 – and two of them are under 10^-7, i.e. zero in single precision.

Three nonzero singular values, and the smallest is still around 10^-4, and the condition number for just the first three columns is

What about the backward selection? That is, the first four regressions – not because they’re the first four, but because they have 5 or fewer variables counting the constant.

Regressions 1 and 4…

The same kind of result as for forward selection: the 4-variable regression has R^2 > .993 and estimated variance 20% of the 1-variable regression.

We might as well literally take a look at the first forward selection, which used only X. Here is the fitted equation:

Here is a graph of the data and the fitted equation:

What about the 1-variable regression from backward selection?

Note that I am leaving the 1-variable fit in the graph.

Interesting. They seem to overlap over the interval of the data. Let’s forceast out to the year 3000 (i.e. x = 3.000):

We do see them diverge. Good.

Now let me try something reckless. Let’s look at forward regression #4. Here is the fitted equation – in two forms, one with X2 etc. and the other with X^2 etc.

Here’s a graph of the data and the fitted equation from the 4-variable regression:

OK, I’m impressed but also shocked that Mathematica gave me something so visually reasonable, although the coefficients of the fit are outrageously large.

I would not be surprised if other software packages were not so obliging.

What about the 4-variable fit from backward selection? Again, the fitted equation has X2 etc, and I need to turn that into X^2 etc.

Again, they appear to overlap almost perfectly. Let’s forecast out to the year 2500.

We should note the vertical scale: the predicted numbers are huge before we can see a difference between the functions.

orthogonalize

We have used orthogonalization successfully to eliminate multicollinearity. OK, let’s orthogonalize the data. Here’s the design matrix (from backward selection because it has the variables in order).

Let’s orthogonalize the transpose of that matrix and call the result Z. Let’s compute Z’Z to see if we get an identity matrix.

Sort of.

Wait a minute! The two rightmost diagonal entries are zero; Z’Z cannot be inverted. We better book at Z itself.

We only get 5 vectors out of the Gram-Schmidt process (counting the constant)! And I know that if I had not divided the years by 1000, we would only have gotten 4.

OK, let’s run with what we have, four orthogonal variables:

This time we got the same set of regressions from both forward and backward selection.

The 3-variable fit doesn’t seem all that much better than the previous one, which was

Let’s check the inversion of X’X for regression 3:

We really can gain by orthogonalizing. We have more reasonable coefficients, and a more accurate inverse of X’X.

On the other hand, it’s harder to look at things. Z2 etc are not simple powers of Z1: Z2 != Z1^2, etc.

(The multicollinearity is gone, as usual.)

Here are the predicted values for the 1- and 3-variable regressions:

Let’s look at the fitted values. Yes, I’m plotting them against X, not against Z1.

The red dots do lie on a straight line: Z1 is a linear function of X, and the fitted 1-variable equation is linear in Z1, hence linear in X.

The blue dots are nonlinear in Z1, hence nonlinear in X.

Finally, let me overlay the original 3-variable fit:

The orthogonalized 3-variable fit (shown only as blue points)

is very close to the original 4-variable fit…

over the region of the actual data.

In conclusion…

With x ranging from 1.986 to 1.993, and using powers of x, we encountered significant multicollinearity: R^2 from the VIFs were 1, the inversion of X’X gave warnings for any variable beyond x itself, and the coefficients of the fits were quite large for the 4-variable regression.

And yet, the fits themselves appeared to be valid. Furthermore, the 4-variable fit seemed to be a better fit than the 1-variable fit which used x alone.

We did, however, get different recommendations from forward and backward selection. And yet their fitted equations were visually indistinguishable.

When we orthogonalized the design matrix, we discovered that we had a new problem: we could only get five orthogonal columns, including the constant one.

Still, we were able to run regressions with just the created variables. Multicollinearity was eliminated, and we again got a good regression with multiple variables, this time with 3 instead of 4. And, this time we had nice coefficients.

I showed that the original 4-variable fit appeared to match the orthogonalized 3-variable fit.

In other words, orthogonalizing the data gives nicer equations, but the fitted values are effectively the same. We can eliminate multicollinearity, but we could not effectively improve on the original horrible-looking equations.

Next time, I hope to show you how a simple change to the x-values can also clean up our horrible-looking equations.

I’ve continued playing with fractional linear transformations, categories, circuit theory, and linear programming. In particular, I might be right on the edge of understanding one last detail in what I want to show you about linear programming.

And, in a blinding flash of insight, it is clear to me why barycentric coordinates are not unique unless we use triangles. I asked that question back in February 2009, toyed with it for a little while, put it down and moved on. And I really did move on, thinking about the question only when something reminded me of it.

Simply put… take a rectangle, cut it into 2 triangles with one of the diagonals, and then draw a point inside one of the triangles. Well, using barycentric coordinates based on the triangle which contains the point, all of its coordinates are between 0 and 1… but using barycentric coordinates based on the other triangle, one of its coordinates is greater than 1 because it’s outside the diagonal edge of that triangle.

In other words, if we want to use rectangles to cut up something (say for finite elements), and if we want to use barycentric coordinates, then within each rectangle we would have to define a triangle to select one specific set of coordinates. We might as well use triangles in the 1st place.

Mathematics, however, is not the only thing I do in this week between Christmas and New Year’s Day. This is when I write an end of the year letter to serve as a Christmas card. I’m still working on it… I don’t have much time for mathematics for the rest of this weekend… possibly not for much of this coming week. That’s one reason I have 2 posts already in stage V… so I don’t have to do very much other than my letter for a while.

And so, for a change, this post closes with: I’m off to do some non-mathematical writing.

Happy New Year.

This post is a day late because I was very lazy yesterday. I watched the Steelers shut out their opponent, instead of working on this post. Afterwards, I just puttered around the house.

I expect to be having dinner with friends this evening, but I’ve decided to make up for yesterday even though it’s Christmas, and do what I should have done yesterday.

Last weekend and during the week, mathematically…

I continued working on a 2nd example of polynomial regression. As I said before, it is a juicier example than I originally realized. That’s still true, even though I found a typo in my Mathematica commands and the computational results are nicer than I had thought. Nevertheless, I think this will be an illuminating example.

It’s also complicated enough that it will span three posts. I was having trouble keeping track of all the things I had done – which is a clear and powerful signal that I have done too much for one post.

I am amused to see that on Dec 3 I said I had “2 small examples of multicollinearity when we try to fit polynomials to data”. Well, they are small – the second one, yet to come, has only 8 observations – but I will not try to cram the discussion and computations into one post.

I’ve also tracked down three – not just one – quantitative illustrations of the Higgs mechanism in three of my books. My situation, however, is that while I can follow the mathematics… I cannot justify the mathematics. I still might decide to post one of these illustrations, even before I understand it, just because it is so darned interesting and timely. After all, the Higgs particle is what gives their mass to all the other particles that have mass. Way cool, huh?

My undergraduate alter ego hasn’t done any more circuit theory; my perpetual student has picked up linear programming again… reading instead of trying to compute… but I’m getting an itch to start computing, of course.

Now let me show you the generalized Stokes’ theorem. It subsumes more familiar equations – from undergraduate physics – involving the gradient, curl, and divergence of vector fields. They can all be replaced by one equation – and a geek should know that.

Let me begin by reminding you (you who have had electromagnetism in college) of the divergence theorem… also called Gauss’ theorem. In words, as applied to electromagnetism, it said that the flux of the electric field through the surface S of a volume V was proportional to the charge contained in the volume:

.

The mathematical statement, however, is that the integral over the surface of the (normal component of the) electric field is equal to the integral of the divergence of the electric field over the volume bounded by the surface:

.

The Maxwell’s equation

says that those two right-hand-sides are equal. That equality is what gets the usual electromagnetic statement from the general math statement.

Then there is Faraday’s Law of Induction: the rate of change of the magnetic flux through an area A is proportional to the induced electromotive force along the curve C bounding the area:

.

(A changing magnetic field generates a voltage drop, which generates a current.)

The mathematical equation (Stokes Theorem, Green’s Theorem) is that

.

and we get from the math to the physics using the Maxwell’s equation

.

Finally, there are two similar equations, one of which I hope we take utterly for granted after freshman calculus. That one is the Fundamental Theorem of Integral Calculus:

.

The other is a generalization of that from the interval [a,b] to a curve C:

.

(Some of you have been using that last equation since high school… you didn’t do the integral over the given path, you used a simpler path for which the potential difference was obvious: the change in gravitational potential energy is independent of the path taken; whether something travels down a playground slide or gets dropped straight down from the top of the slide, it hits the ground with the same speed.)

What do all these equations have in common?

One integral in n-dimensions, and the other integral in (n-1)-dimensions. We had a volume and the surface bounding it; an area and the curve bounding it; an interval and the endpoints bounding it; a curve and the endpoints bounding it. On the one side we had a function, and on the other some differential operator applied to it.

The generalized Stokes’ Theorem is written

.

is the boundary of the “manifold” M – where the term manifold encompasses intervals, paths, areas, volumes, and on to higher dimensions; and d is the exterior derivative of the form – where a “form” encompasses anything we can integrate over.

Depending on turns out to be the ordinary derivative, the gradient, the curl, or the divergence.

As with most of my growing list of things a geek should know about mathematics, it isn’t just an equation or four – it’s that little extra. In this case, I think a geek should know the generalized Stokes theorem – not just its special cases.

That is, the equations

are all special cases of

.

I haven’t said this in a while: I generally use “Dragon Dictate” to create text, rather than trying to type. Last week’s happenings post, however, was completely typed.

Or I might have been a little annoyed that I wasn’t able to talk about either Stokes’ theorem or the Axiom of Choice – and, at best, I will be lucky to write about Stokes theorem today. (It’s another piece of mathematics I think a geek should know.)

We’ll see how the draft develops.

For those of us even remotely interested in particle physics… it appears that the large hadron collider has seen something at 125 GeV. It could be the Higgs particle. Peter Woit’s blog, as usual, has something to say about it and provides a lot of links.

(I remember making a note that one of my particle physics books has a promising discussion of how the Higgs particle imparts mass to the other particles in The Standard Model. That is what it’s for. And “promising” only means I thought I might be able to follow the derivation.)

For those of us even remotely interested in being passengers on commercial airliners… someone has published the complete transcript of the black box recording of the crash of Air France flight 447. The Economist has a nice discussion. He’s catching a lot of flak… because a brutal summary of the crash would be: the autopilot turned itself off, and the pilots did not know how to fly the plane manually.

A less than brutal summary would add: the autopilot turned itself off because the instrument readings were too strange, so the pilots were faced with severe instrument failures, at night. I can certainly believe that “instrument flying” would be damned hard when the instrument readings are unreliable. Still, if the autopilot can turn itself off because things are bad, we should expect that a pilot might be handed the plane in conditions too bad for the autopilot.

My alter ego the kid has been looking at Lawvere and Schanuel’s “Conceptual Mathematics” – an iconoclastic undergraduate introduction to category theory, by one of the masters.

He’s also been looking at Ullrich’s “Complex Made Simple” – which might have given me the final clue I need for getting Mathematica to cope with fractional linear transformations. They are of the form

, with ad – bc ≠ 0, and z a complex number

… and the challenge is to figure out the 4 coefficients for a transformation which moves a specified set of points in a specified way. I haven’t tried it in a while… it might’ve been 8 years ago or so… but it didn’t work very well then. Maybe I’m ready to try again.

I’ve been feeling frustrated about how little mathematics I get done… and then I realized that I have a substantial amount of material which my undergraduate alter ego has worked out: group theory, strength of materials, lie algebra, and now circuit theory. In other words, my publishing can’t keep up with even the undergraduate material I’m going through, and yet I am feeling that I should be getting through even more material.

One of the framed quotations on my wall comes from “Life’s Little Instruction Book” by Brown. (It’s in my bibliography.)

If you’re going after Moby Dick, take along the tartar sauce.

What it means is, plan on success. (Don’t quibble that Moby Dick wasn’t a fish, so tartar sauce might be the wrong condiment; that misses the point.)

I finally realized why I like that quotation. Oh, I liked it enough to frame it, but I didn’t know why it mattered so much to me.

I have no hope, of course, of understanding all of mathematics. Hell, I have no hope of understanding all of the math books I already own.

The good side of that is that I am free to study whatever math I want to, because I can never learn all of it.

The bad side of that is that I am apparently chasing Moby Dick… my pursuit of mathematics is a lost cause.

I exaggerate. The key, however, is that although I cannot learn all of it, I’m not willing to rule out any part of mathematics or its applications. Despite knowing that I will never understand all my technical books, I will continue buying more, and having a go at whatever strikes my fancy.

(So I know what Moby Dick represents. Now if I could only figure out what the tartar sauce is….)

If you don’t like the answer, change the question.

(from an episode of Numb3rs.)

My doing mathematics isn’t about the destination – it’s about the journey. I try to take pleasure on a daily basis.

My alter ego the kid has a lot of fun just looking at things… and my alter ego the undergraduate derives a lot of satisfaction from solving (admittedly elementary) problems… and I’m still working with regression, and putting out posts… and soon regression will be replaced by another topic.

Well, it’s after noon, and although my draft of this post included the generalized Stokes’s theorem – required for geeks – I’ll not try polishing or publishing it today.

Now let me try working on Monday’s post… more about fitting polynomials to data… or maybe I’ll track down that “easy” derivation of how the Higgs field gives mass to other particles….