Open v. Closed Source-Climate Change Research

theidocles writes "The ongoing debate over the 'hockey stick' climate graph has an interesting side note. McKitrick & McIntyre (M&M), the critics, have published their complete source code and it's written using the well-known R statistics package (covered by the GPL). Mann, Bradley & Hughes, the defenders, described their algorithm but have only released part of their source code, and refuse to divulge the rest, which really makes it look like they have some errors/omissions to hide (they did publish the data they used). There's an issue of open source vs closed source as well as how much publicly-funded researchers should be required to disclose - should they be allowed to generate 'closed-source' solutions at the taxpayers' expense?"

Comments

real government research goals

[MyRuger]

Obviously you have never written an SBIR or BAA. You when you do research "At the tax payers expense", you need to show your plans to commercialize the results of the research. The government wants you to create a IP twoards a commerial project which will spur the economy, not to contribute to the scientific community as a whole. Take it as you will, but I think that most research would not get funded if your commertilization plan was to release it on sourceforge.

Re:The debate

[syphax]

The debate is well-documented (by the Mann team, at least) here, here, here.

CRAN is your friend

[Anonymous Coward]

http://cran.r-project.org/
However, the name still sucks.

taxpayers vs boffins

[dos_dude]

This is an extremely difficult issue, although it sounds pretty trivial.

For one thing, the taxpayer is rarely participating in discussions like this one. Moreover, the success of scientific institutions is often measured in terms of number of patents, successfully launched businesses by former students/researchers, etc. So not only is there little or no opposition to closed-source software (or scientific articles!), there are also good reasons for researchers to go the closed-source road.

Some researchers have a tendency towards secrecy. Some even seem a little paranoid when it comes to their data and methods. You could compare this to the tendency of the OSS zealot to suspect bugs, glitches, and omissions in any piece of closed-source software.

And as a German side-note: There are laws over here that require you to have the patentability of any piece of software you develop checked by university lawyers. GPLing something is technically illegal for a researcher. I have no idea how this is regulated in other countries.

Re:Just because it's code it should be open?

[climb_no_fear]

I work for a pharma company. When we publish, we have to publish the structure of the compound used. You or a skilled chemist could cook it up and reproduce my work. That makes it science. Even if it's patented, you can do this under the freedom to research clause.

Re:The debate

[syphax]

Looks like M&M have a blog too...

Re:But the Hockey Stick is True!

[grqb]

I think that the argument though is that the hockey stick has happened before in the past. I mean, the earth naturally warms and naturally cools, there has been global warming before the advent of fossil fuels.

This is the big problem for people trying to fight the critics. For me though it's easy. The CO2 levels in the atmosphere have never been as high as they are now (at about 370ppm) and they're expected to increase up to 700ppm if we finish off the oil (which may be in 70 years or longer). But the point is, even if global warming is/is not happening, having over 370ppm of CO2 in the atmosphere is just not good! Here's a pretty good summary of the global warming argumnts.

If you were wondering what real scientists think

[Uksi]

The blurb author attempts to paint one side as having something to hide, since they only released a part of their source code. Nevermind that both papers' data can be independently validated--no no, one side is bad for only describing the algorithm and not its source code!

So a team of real scientists (that is, by folks who work in climate science, not reporters or pundits) wrote a Dummies Guide to the latest controversy. Click on the link for a nice question-by-question breakdown, but I'll spoil the conclusion for you:

(MBH98 is the old paper with "closed" source, MM05 is the new "open source") paper)

7) Basically then the MM05 criticism is simply about whether selected N. American tree rings should have been included, not that there was a mathematical flaw?

Yes. Their argument since the beginning has essentially not been about methodological issues at all, but about 'source data' issues. Particular concerns with the "bristlecone pine" data were addressed in the followup paper MBH99 but the fact remains that including these data improves the statistical validation over the 19th Century period and they therefore should be included.

8) So does this all matter?

No. If you use the MM05 convention and include all the significant PCs, you get the same answer. If you don't use any PCA at all, you get the same answer. If you use a completely different methodology (i.e. Rutherford et al, 2005), you get basically the same answer. Only if you remove significant portions of the data do you get a different (and worse) answer.

9) Was MBH98 the final word on the climate of last millennium?

Not at all. There has been significant progress on many aspects of climate reconstructions since MBH98. Firstly, there are more and better quality proxy data available. There are new methodologies such as described in Rutherford et al (2005) or Moberg et al (2005) that address recognised problems with incomplete data series and the challenge of incorporating lower resolution data into the mix. Progress is likely to continue on all these fronts. As of now, all of the 'Hockey Team' reconstructions (shown left) agree that the late 20th century is anomalous in the context of last millennium, and possibly the last two millennia.

Read the rest for more explanation.

Re:The debate

[ifoxtrot]

The BBC also has an article that recounts the controversy here.

Replication

[mwvdlee]

Significant research data is generally replicated independantly of the original researchers for verification of the results. Without a description of the method of research used (in this case; the computer model), how can the data be replicated and thus verified? Indeed the very methods itself are commonly scrutinized in the scientific world and, IMHO, any scientist that does not approve of this is not looking for truth but for something else (personal agendas, fame, etc.).

Not detailing the methods used (in this case; giving the entire algorithms, either as source or as a 100% comlete and unambiguous description) basically limits the usefullnes of the resultant data as mere speculation, not proof nor even theory.

If I remember correctly, the computermodel in this case is known to include a rather lacking model of rainfall, which seems like a pretty big omision in a climate model to me.

in biology it happens too...

[operon]

Today biology heavily depends on specific software to analyse lab generated data. However, even academic, public funded software are not open-source. It's a sad situation, but there are efforts like Bioinformatics.Org trying to change the situation.

with open source, everyone can see you're dumb

[Nick Barnes]

See this debunking of McKitrick's work, showing, among other things, how he:

• denies that average temperature is meaningful,
• confuses degrees with radians,
• invents a whole new temperature scale,
• replaces missing data with zeroes

Re:But the Hockey Stick is True!

[Anonymous Coward]

"For me though it's easy. The CO2 levels in the atmosphere have never been as high as they are now (at about 370ppm) and they're expected to increase up to 700ppm"

http://www.geocraft.com/WVFossils/Carboniferous_ cl imate.html

Re:How much is enough?

[Anonymous Coward]

For astronomy such an archive exists: http://ascl.net/ascl_archive.html and
if anyone wants to pad their resume they
are looking for a new editor.

Re:Taxpayer funded whitewashes

[Travis Fisher]

If you want to read someone knowledgeable, levelheaded, and intelligent about the TWA flight 800 investigation, and the actual physical evidence in that crash, check out testimony of metallurgist William Tobin in the congressional hearing on the matter. Mr. Tobin was one of the lead scientific investigators of the recovered wreckage. A sample quote:

• Senator Grassley. What were some of the characteristics which negated the missile theory?

• Mr. Tobin. Well, probably the most prominent--actually, there were two main areas negating the missile theory. One, of course, again, is the absence of impulsive loading, or very high-speed fracture and failure mechanisms.

  But second was there were serious issues with every theory, or almost every theory, as to access of an external missile to the fuel, to the fuel tank. Even with, as I indicated earlier, if one would focus on an area where we did not recover all of the fuel tank, there were components nearby that would have blocked or at least recorded passage of any externally penetrating object. And if that were not the case, there were many layers, including the external underbelly of the aircraft, and that was recovered almost--a huge portion of that was recovered.

  So that, basically, the only plausible theory for some of the missiles to have occurred would have been if there were missiles such that could maybe get through a 1- or 2-inch opening, make an immediate left, go 90 degrees through a seam, and then maybe take another 90-degree right, and then maybe reverse itself and come back over. But those were some of the considerations.

This is the voice of reason in a case where reason is ignored...

Re:But the Hockey Stick is True!

[Glock27]

I don't know the details about this issue, but Kyoto is about CO2 budgets, not about air pollution. Burning wood for cooking may produce soot, but it doesn't produce extra CO2 as long as new trees take the place of the once that are burnt.

Yes, but that is why the Kyoto protocol is flawed. The authors of the cooking fire study estimated the warming effect of the soot was 30 times worse than that of the same mass of CO2.

Re:But the Hockey Stick is True!

[radtea]

The sphericity of the Earth was not generally accepted until much later than the 1st century C.E. More like the 8th. Even then, the way in which people thought about the Earth was radically different than the modern idea of a spherical Earth:

http://abob.libs.uga.edu/bobk/whiteb03.html

The "doctrine of the antipodes" asserted that even if the Earth was spherical, no humans lived on "the other side" because they'd have their feet in the air, wouldn't be able to observe the descent of Christ at the 2nd coming, etc.

Variants of this doctrine persisted long after the nominal sphericity debate had been settled, and I'd argue that until something like the modern view of a spherical Earth, antipodes and all, was generally accepted, it is not quite correct to claim that it was "generally accepted that the earth is round".

Ancient ideas are alien to our own, and it is easy to impose our modern understanding on the words the ancients used, creating great distortion. So I get to disagree with everyone: in the first millenium C.E. people neither believed that the Earth was flat, nor that the Earth was round in the modern sense. They believed the Earth had a special place in the universe, and their understanding of the shape and geography of the Earth grew out of Church doctrine as much as emprical observation.

--Tom

Re:If you were wondering what real scientists thin

[jnaujok]

You do know that Mann writes this website, right? You do realize that the source of your argument (http://www.realclimate.org/) is a shill for Mann and his cronies?

Second of all, there was a flaw in the original algorithm that was pointed out by McIntyre and McKitrick before they even got to the bad data being put into the equation.

And, to top it off, Mann's equation always produces hockey-stick graphs, even with randomly distributed data.

Don't point at Mann's own site as a defense of Mann.

Replication

[cvdwl]

As an academic computer programmer in ocean modeling, let me just say it HAS TO be open. Yes, my work is open source, though why anyone would WANT my code is beyond me. Most of what I do is quick, short-time, badly coded, inefficient data processing and vizualisation scripts. Still, feel free to email me and I'll send you a tarball of any code on my machine or a link to the developer's page.

1) Science functions only on open review. If you can't duplicate someone's results, they are useless (c.f. Ponds and Fleischman [sp?]). A scientific result is only of value if it describes a consistent replicatable process. This is why I consider the closed source work to be completely meaningless. It may be perfect, it may be bug-ridden garbage, we'll never know!

2) Every tax paying American has paid for my code and work. While I regularly feel they're not getting their money's worth, I definitely don't feel they're paying me to enrich me. They are, in a very real sense, my bosses, and I AM obligated to report to them, if they care. Think of it as a company requiring rights to your work.

3) As an academic working on a fairly limited budget, open source and free software have been a godsend for me and everyone else I know. We run linux because it's more efficient, secure and FREE; we use free or open-source compilers; and we cobble together high-perormance computers and beowulf clusters out of miscellaneous bare metal and lots of googling. The only piece of software I routinely have to pay for is MATLAB.

how about yes and no instead

[zogger]

People have been trying for years to get full information out of government for world war two era information, for example. It's only recently that a lot of the "Operation paperclip" information is coming out, some of the more detailed stuff anyway. The USS Liberty attack was another one I remember, they kept the real information quite hidden for a long time. How about obvious coup information like the JFK assassination? A lot of that is still sealed. And what is happening now is that they can just re-classify something as secret based on a pretty loose definition of allegedly protecting privacy and/or it's necessary for "law enforcement" secrecy, which is so loose as to make the FOIA almost useless. Or just mumble the word "terrorism" and that seems to cover most anything they want. Like people on this "no fly list", what's up with that? If people on this list are actual criminals, then charge them, don't have secret "lists", that's just bogus and dangerous.

Basically the government might give with one clenched fist, and take back with a team of mules pulling. The National Security Archives are one group of folks constantly struggling with the layers of governmental coverups. It's ongoing and pretty telling. They are having mixed results, some good finds, then a lot of what they are calling "over classification and pseudo classification" still existing. And then the problem becomes getting the information out to joey and janey citizen and voter, the "news" only mostly covers current, people have just been conditioned to accept todays fairy tales as "data and fact", over and over again. Then years later the real story comes out, by then it's too late to influence elections, etc. Look at the finally revealed data on the "Tonkin Gulf attack" that was the primary "lawful" reason for the Viet Nam war. They have (relatively historically recently)finally and quietly admitted it was an invention, but years too late to make it matter for most purposes.

So, in part I agree, some of what the government does needs to be kept secret, but it appears quite a bit is still overzealously kept hidden, primarily to protect the guilty-of-corruption-and-malfeasance aspects of government.

Re:Voodoo, not science

[climb_no_fear]

The following coordinates are from people at a university who used a small molecule from a company (Scios) to get their protein to crystallize. The structure of the small molecule doesn't appear anywhere in the paper (of course, a clever person could use the now-released electron density to calculate its structure).

http://www.rcsb.org/pdb/cgi/explore.cgi?pid=400211 11511253&page=0&pdbId=1IAS

You can use the status search link at PDB

http://www.rcsb.org/pdb/status.html

to find lots of things on hold (I found 211 when searching for Status "release on a certain date" AND "Release date" > 1 April 2005.

Also, I work at a pharma company, do publish and have seen lots of competitors do the sort of thing above.

The Bayh-Dole Act changed all that

[jmason]

how much publicly-funded researchers should be required to disclose - should they be allowed to generate 'closed-source' solutions at the taxpayers' expense?

It's worth noting that, while it makes sense that taxpayer-funded research should generate 'open-source' solutions, federal law dictates otherwise.

The Bayh-Dole Act was passed 25 years ago, which dictates:

Universities were encouraged to collaborate commercial concerns to promote the utilization of inventions arising from federal funding.

It was clearly stated that universities may elect to retain title to inventions developer through government funding.

Universities must file patents on inventions they elect to own.

So in other words the government has dictated since 1980 that government-funded research should not produce open-source solutions, necessarily, as the results of research are to be considered private-sector profit-generating centers for the host universities. (The implications for the 'next BSD4.3 TCP/IP stack', or similar advanced research, are obvious.)

Anyway, regarding the 'hockey stick' controversy, Tim Lambert's weblog is worth a read.

Re:Where?

[kippy]

Uhrm ... where? I haven't been able to find any code on any on of the pages mentioned. I agree it's essential to disclose all data and source code ...

Unless I'm mistaken...

Source Data

Re:If you were wondering what real scientists thin

[Lars T.]

Shill out.

Here, however, we choose to focus on some curious additional related assertions made by MM holding that (1) use of non-centered PCA (as by MBH98) is somehow not statistically valid, and (2) that "Hockey Stick" patterns arise naturally from application of non-centered PCA to purely random "red noise". Both claims, which are of course false, were made in a comment on MBH98 by MM that was rejected by Nature , and subsequently parroted by astronomer Richard Muller in a non peer-reviewed setting--see e.g. this nice discussion by science journalist David Appell of Muller's uncritical repetition of these false claims. These claims were discredited in the response provided by Mann and coworkers to the Nature editor and reviewers, which presumably formed the primary basis for the rejection of the MM comment.

...

Lets turn, now, to MM's claim that the "Hockey Stick" arises simply from the application of non-centered PCA to red noise. Given a large enough "fishing expedition" analysis, it is of course possible to find "Hockey-Stick like" PC series out of red noise. But this is a meaningless exercise. Given a large enough number of analyses, one can of course produce a series that is arbitrarily close to just about any chosen reference series via application of PCA to random red noise. The more meaningful statistical question, however is this one: Given the "null hypothesis" of red noise with the same statistical attributes (i.e., variance and lag-one autocorrelation coefficients) as the actual North American ITRDB series, and applying the MBH98 (non-centered) PCA convention, how likely is one to produce the "Hockey Stick" pattern from chance alone.