Approximating Intelligence

So I sometimes read a rather interesting blog by a statistician, William M. Briggs, who works on critiquing global warming science through statistical analysis. He's certainly interesting, and I have learned quite a bit from him, but I think he has a tendency to over-analyze things and relies a bit to much on quantitative analysis. Sort of can't see the forest but for the trees. Good science is rarely as exact or as statistically rigorous as people seem to think.

Anyway, in a response to one of his posts, I left the comment below, which I think is an important point to keep in mind.

The passage to which is was responding was:

I’ll write about these topics in more depth later, but briefly: people weight heavily the fact that many different climate models are in agreement in closely simulating past observations. There are two main, and very simple problems with this evidence, which I could have, at the time, done a better job pointing out. For example, I could have asked this question: why are there any differences between climate models? The point being that eight climate models agreeing is not eight independent pieces of evidence. All of these models, for instance, use the same equations of motion. We should be surprised that there are any differences between them.

The second problem I did point out, but I do not think I was convincing. So far, climate models over-predict independent data: that is, they all forecast higher temperatures than are actually observed. This is for data that was not used to fit the models. This means, this can only mean, that the climate models are wrong. They might not be very wrong, but they are wrong just the same. So we should be asking: why are they wrong?

And my response:

I am astrophysics PhD student who runs numerical simulations studying galaxy dynamics. While I have absolutely no direct experience working on climate models, nor do I have any but the faintest idea of the inputs or physics that goes into them, I am quite familiar with the similar numerical modeling techniques that are used in astrophysics calculations. I would like to very briefly address your question of “why are there any differences between climate models?”

The systems I study are actually much simpler than any climate system. I model the mass of galaxies as a simple N-body, collisionless, self-gravitating fluid of point masses- which means that the only relevant physics involved are Newton’s laws that we all learned in high school. Yet different gravitational codes yield slightly different answers. The reason is that there is no analytic solution to the gravitational N-body problem - there are only numerical approximations.

So each code has to make different assumptions on how to best approximate the result. There is the additional constraint that the time it takes to actually perform the calculations on current computers has to be reasonably small (hopefully, less than the time it takes to earn a PhD!). In order to decrease the computational expense, more approximations are used. Obviously, different approximation methods will yield slightly different answers, but all should yield approximately the same result. Since we can’t measure our approximations against the “true” answer (such an answer does not exist), comparing different techniques to make sure they give similar results is the best method we have for checking our codes.

Keep in mind that the system I have described is an extraordinarily simple system. Climate models have many more layers of complexity - and thus approximations - to deal with. Again, there is no “true” answer against which to compare the results of simulations.

So, let me be clear about the very narrow point I am trying to make. You are certainly correct that results from different codes are not necessarily independent pieces of evidence. It is also worrying that the codes are not fitting independent data. Asking why the models disagree with the data and trying to improve them is of paramount importance. However, simply asserting that “the climate models are wrong” seems to me to be missing the point. It does matter how wrong the models are because all we have are approximations and there is no a priori method of determining how good those approximations are. So, the best we can do is make lots of different models using different assumptions and hope they give reasonably similar answers. “Reasonably similar” may not seem quantitatively satisfactory, but it’s the only way we have to do science.