Slashdot Mirror
Open v. Closed Source-Climate Change Research
theidocles writes "The ongoing debate over the 'hockey stick' climate graph has an interesting side note.
McKitrick & McIntyre (M&M), the critics, have published their complete source code and it's written using the well-known R statistics package (covered by the GPL). Mann, Bradley & Hughes, the defenders, described their algorithm but have only released part of their source code, and refuse to divulge the rest, which really makes it look like they have some errors/omissions to hide (they did publish the data they used). There's an issue of open source vs closed source as well as how much publicly-funded researchers should be required to disclose - should they be allowed to generate 'closed-source' solutions at the taxpayers' expense?"
should they be allowed to generate 'closed-source' solutions at the taxpayers' expense?
No. I paid for it I want to see it. How else will we know if it works the way they say it works?
I'm a virgo and on Slashdot. Coincidence? Yes.
how much publicly-funded researchers should be required to disclose
All of it, baby. We're paying for it -- we should have the right to:
a) Know what you're spending our money on
b) Have the right to make it better ourselves
c) Learn of security flaws early so we can correct them
Especially when there is some doubt about the nature of the results in the closed source model from Mann et al.
The dangers of knowledge trigger emotional distress in human beings.
I hate projects with names like R. I used R a while back, and it's a great program, but try searching for "R" plugins on Google. Not fun.
is in the interest of the science in this case.
You might want to ask to see the model behind the CIA data which proved conclusively that a 747, deprived of its forward fuselage, can convince over 600 witnesses that said 747 was shot down by a SAM.
The Slashdot Paradox: "100% Overrated"
The problem with most of these studies is that they refuse to release the raw data.
... but no thanks !!!
A lot of times they select subsets of the data and then normalize or otherwise massage the data.
Thanks
For all we know, there could be a very valid reason why they haven't released all of it. I'm not sure what that reason could be, but given that we don't have anything to go on, we're stuck to just guessing.
Avantslash - View Slashdot cleanly on your mobile phone.
Science, like government, should be transparent. The public should be able to see and evaluate every part. Any science, or government, that hides it's implementation is inherently suspect to corruption.
Closed science is half a step from religion. You are expected to have faith in the researcher's methodologies, analysis, assumptions, and motives. Sorry, but good science does not rely on faith.
The pharmaceutical industry receives huge subsidies from us - they don't produce "open" drugs - why should this be any different? I know it's apples and oranges - but one should be really careful about the idea of withholding funds from -good- research just because of licensing issues. Lesser of two evils? Would we rather have -no- research?
complicated...
Arguments against open-source science:
I'm sure there are arguments on both sides.
Two wrongs don't make a right, but three lefts do.
I'm of the opinion that anything that gets published should be published in its entirety, at least at some point. For example, people who publish protein structures can put the coordinates "on hold" for up to 18 months.
And to say because the research is done with "taxpayer's money" is missing the point: If you can't reproduce every step, it's voodoo, not science. And we make policy decisions based on science, not voodoo (I hope).
Obviously you have never written an SBIR or BAA. You when you do research "At the tax payers expense", you need to show your plans to commercialize the results of the research. The government wants you to create a IP twoards a commerial project which will spur the economy, not to contribute to the scientific community as a whole. Take it as you will, but I think that most research would not get funded if your commertilization plan was to release it on sourceforge.
While I would like all works performed for the government that are not of National Security importance to be more open I don't think it is necessary.
A lot of work peformed for government agencies is contractual with businesses. These same businesses employ tricks of the trade and such to deliver what is required. To have them detail how the work is just suicidal. The same goes for software they develop for use by the government. Unless specifically addressed in the contract I do not believe there is a right to disclose the code, let alone make it available to the public.
That last part is key. Even if they disclose the source to the government there is no obligation on either party to make it public.
This argument that they have something to hide is childish. It is designed to provide no leeway. Simply put, once labeled as such what other option other than disclosure exist? You might as well say "You have to release it, its for the children" and then proceed to use whole "hates kids, wants kids to die" guilt trip that is far to common in politics today.
Summary. Release it if only its an upfront requirement of the project and agreed upon by both parties. In the future a requirement by law that all government projects must be fully disclosed to include the source of any software may be nice but I bet it would have so many exceptions written into it that it would result only in a "feel-good" law.
* Winners compare their achievements to their goals, losers compare theirs to that of others.
This is an extremely difficult issue, although it sounds pretty trivial.
For one thing, the taxpayer is rarely participating in discussions like this one. Moreover, the success of scientific institutions is often measured in terms of number of patents, successfully launched businesses by former students/researchers, etc. So not only is there little or no opposition to closed-source software (or scientific articles!), there are also good reasons for researchers to go the closed-source road.
Some researchers have a tendency towards secrecy. Some even seem a little paranoid when it comes to their data and methods. You could compare this to the tendency of the OSS zealot to suspect bugs, glitches, and omissions in any piece of closed-source software.
And as a German side-note: There are laws over here that require you to have the patentability of any piece of software you develop checked by university lawyers. GPLing something is technically illegal for a researcher. I have no idea how this is regulated in other countries.
That Global Warming is a manmade, real phenomona is accepted by 99.9% of scientists in the fields involved. To trot out the "only a theory", "some experts dispute" etc routine is like getting the Flat Earth Society involved every time someone talks about circumnavigation. "Heads in the sand" is going to be on our culture's gravestone when the next lot of intelligent life evolves here and starts wondering why parts of Nevada are 10,000 times the normal radiation level.
The Slashdot Paradox: "100% Overrated"
This is the big problem for people trying to fight the critics. For me though it's easy. The CO2 levels in the atmosphere have never been as high as they are now (at about 370ppm) and they're expected to increase up to 700ppm if we finish off the oil (which may be in 70 years or longer). But the point is, even if global warming is/is not happening, having over 370ppm of CO2 in the atmosphere is just not good! Here's a pretty good summary of the global warming argumnts.
In science you don't simply show the results of your research, you also describe you arrived at them. Science has always been like that. With more and more science becoming dependent on computers, it naturally means that one must describe the algorithms used to arrive at those results.
The easiest way to do that is to show the source code.
These "closed source" scientists need to remember their high school math teacher's admonitions to show their work.
So a team of real scientists (that is, by folks who work in climate science, not reporters or pundits) wrote a Dummies Guide to the latest controversy. Click on the link for a nice question-by-question breakdown, but I'll spoil the conclusion for you:
(MBH98 is the old paper with "closed" source, MM05 is the new "open source") paper)
Read the rest for more explanation.Actually, 99% of the well-educated people today incorrectly believe that 99.9% of the scientists in the middle ages believed in the concept of a flat earth.
The has been a generally accepted notion that the earth is round since the 1st century A.D.. Disputes have only been about (1) whether the sun revolves around the earth or the other way around, and (2) what the radius of the earth is.
Avantslash: low-bandwidth mobile slashdot.
Significant research data is generally replicated independantly of the original researchers for verification of the results. Without a description of the method of research used (in this case; the computer model), how can the data be replicated and thus verified? Indeed the very methods itself are commonly scrutinized in the scientific world and, IMHO, any scientist that does not approve of this is not looking for truth but for something else (personal agendas, fame, etc.).
Not detailing the methods used (in this case; giving the entire algorithms, either as source or as a 100% comlete and unambiguous description) basically limits the usefullnes of the resultant data as mere speculation, not proof nor even theory.
If I remember correctly, the computermodel in this case is known to include a rather lacking model of rainfall, which seems like a pretty big omision in a climate model to me.
Slashdot social media options: AIM, ICQ, Yahoo, Jabber and Mobile Text. Why no MySpace?
Today biology heavily depends on specific software to analyse lab generated data. However, even academic, public funded software are not open-source. It's a sad situation, but there are efforts like Bioinformatics.Org trying to change the situation.
---- Where is my mind?
Public funded means owned by the public. I am not talking about abvious things, like miletary secrets, but reasearch like this.
I asume this research has been done to widen our understanding and knowledge, not for profit. To achieve this goal the best thing is to check, check, recheck and then let others recheck as well. This can only be done if you give up all your findings and ways of how you found it.
This is about knowledge and not about being right or wrong (or at least it should be). The knowledge of proving that the theory is right is just as importand as proving it is wrong.
Don't fight for your country, if your country does not fight for you.
"For me though it's easy. The CO2 levels in the atmosphere have never been as high as they are now (at about 370ppm) and they're expected to increase up to 700ppm"
_ cl imate.html
http://www.geocraft.com/WVFossils/Carboniferous
I think there is no reason to demand that scientist should publish their source code, since scientist usually reuse their code and share their code with people they work with, but should not be obliged to help other scientist that they are competing for funding with to get their own simulation programs.
The demand on scientist are clear though, they should give enough information in their publications so anyone interested (or who want to refute their results) can reproduce what they have done. So any statistical or mathematical methods used should be mentioned. And if they use commercial packages (with closed source usually for all parties), mention which packages they use would be wise so that if there are found bugs in these programs, any influence on their results can be taken into consideration. If enough information is given, then any scientist who can program, can check out the literature how to implement the nummerical algorithms and write their own program. Often they can buy (fairly expensive) commercial packages or even find open source liberies that have already implemented these algorithms, and then reproduce the results.
If these two economist were able to reproduce the results of some major climate scientist, then these climate scientist have given enough information to their fellow scientist and the general public. So lets forget about these two guys, or buy their book if you want to believe they know better about climate changes than the general scientific community.
--- guns don't kill people, people with guns kill people ---
So the climate change critics' site has direct links to everything that supports their position, but when they mention "realclimate.org" they don't make it a link so you have to cut and paste into the URL to get the other side of the story. That pretty much sums up their intensions and intellectual honest right there.
IOW, it's just more FUD from the corporate lobbies. The "hockey stick" is real, it's too late, and we're all doomed to live in a bio-dome... sad but true.
Two different things. As a taxpayer you have a right to decide if there IS medicare and who gets it. Representative republic, you vote in your congresscritter and he/she/it does your bidding. Theoretically, of course.
Once you have decided that there shall be Medicare and that there shall be a bureacracy to take care of it, you have no particular rights to anything that happens inside it, unless its happening to you. Doctor/patient confidentiality applies irrespective of who's paying the doctor, both moraly and legaly.
The researcher taking public money is a completely different case. He's doing research for the government, which means for you. Theoretically, of course. Absent pressing matters of national security, there is no reason that the results of publicly funded research should not be available to taxpayers. You paid for it, you should get to look at it.
One caveat, if the researcher used a proprietary method or machine or software to either acquire the data or process the results, you are only entitled to the data and results, not the proprietary device. The government rented the use of it, they didn't buy the rights to it.
Yes, but that is why the Kyoto protocol is flawed. The authors of the cooking fire study estimated the warming effect of the soot was 30 times worse than that of the same mass of CO2.
Galileo: "The Earth revolves around the Sun!"
Score: -1 100% Flamebait
"Their argument since the beginning has essentially not been about methodological issues at all, but about 'source data' issues [...] Only if you remove significant portions of the data do you get a different (and worse) answer."
You're over-trivializing a DRAMATICALLY IMPORTANT POINT. The original study is focused on North American data almost exclusively for certain time periods. That data (from a single species of tree) skews the results in such a way as to make the current trend seem unique and drastic. On the other hand, if you treat that data source in such a way as to balance it with the other data that is available, you see a VERY DIFFERENT TREND!
The response has been to claim that weighting the data in this way reduces the number of data points unacceptably (I would agree, but that doesn't make MBH98 right).
That's the whole point here, and the other side continues to say, "you're throwing away data" when any competent researcher would have thrown it out in the first place (note: there's an exception. if you produced a report that was specific to N. America, MBH98 would be your model, and it seems to be a fine model for that... N. America is seeing record warming as compared with the last few centuries, and that's all you can extract from MBH98).
Also keep some perspective in mind here. We're in a period where temperatures could rise MORE than ANYONE is predicting and not make a dent in the graph over the last 10million years. If you graph out the last 10 million years, you see that temperatures over the last 10,000 years have been part of a huge, cyclical spike in temperatures. We're at what is likely the peak of a drastic temperature swing, and it WILL plumet again into a new ice age (unless we decide to and are capable of coming up with a way to prevent it). I'm not drawing any conclusions from that, just pointing out that there are natural forces at work here, capable of making temperature changes that we a) cannot yet conclusively explain and b) the likes of which no human has ever experienced.
It's important to keep a sense of perspective and to remember that we have very impressive climate models... all of which might be wrong.
Medicare, etc, are services. The intellectual property in that case is of little market worth and belongs to the patient as the interest in health privacy trumps the need to know.
However, in the case of research, federally funded research should have a complete disclosure. If you have a scientist doing work, and not disclosing the entire body of it, then in reality, the end product must not be regarded as science, but opinion. If Mann does not disclose his entire body of work used to comprise his conclusions, then how else can we assess whether his conclusions are accurate or not?
Science must be open source.
This is my sig.
Where's that wealth of information about the secret US wars in Central America in the 1980s? Or in Angola in the 1970s? Or in Chile in the 1970s? Or in Cambodia and Laos in the 1960s? Iran in the 1950s? These secret wars are secret largely for *political* purposes - the military secrecy benefits evaporate within months. But the political purposes - covering liability for abuse, war crimes, and just plain lying about the causes, effects, and benefits of the war - those last forever.
--
make install -not war
The sphericity of the Earth was not generally accepted until much later than the 1st century C.E. More like the 8th. Even then, the way in which people thought about the Earth was radically different than the modern idea of a spherical Earth:
http://abob.libs.uga.edu/bobk/whiteb03.html
The "doctrine of the antipodes" asserted that even if the Earth was spherical, no humans lived on "the other side" because they'd have their feet in the air, wouldn't be able to observe the descent of Christ at the 2nd coming, etc.
Variants of this doctrine persisted long after the nominal sphericity debate had been settled, and I'd argue that until something like the modern view of a spherical Earth, antipodes and all, was generally accepted, it is not quite correct to claim that it was "generally accepted that the earth is round".
Ancient ideas are alien to our own, and it is easy to impose our modern understanding on the words the ancients used, creating great distortion. So I get to disagree with everyone: in the first millenium C.E. people neither believed that the Earth was flat, nor that the Earth was round in the modern sense. They believed the Earth had a special place in the universe, and their understanding of the shape and geography of the Earth grew out of Church doctrine as much as emprical observation.
--Tom
Blasphemy is a human right. Blasphemophobia kills.
How many times have you asked someone, "What does your code do to solve this problem?" and got a description of an algorithm which, when you finally get to see the source, does not match the code?
In my case, the answer to that question is, "Lots." I have had it happen in pure science (neutrino physcis), applied science (medical physics) and software development (database programming, data analysis, etc.)
I am painfully aware that my own published descriptions of algorithms have often left out minor details that may be critical in some applications, but that page limits in peer-reviewed journals necessitate. It is not uncommon to get a call from someone doing similar work asking for details about what you've done, how you've done it, and in some cases, asking to look at source code.
In contentious areas of science such requests are not always met with full disclosure, which is a sign that the people involved are no longer doing science. They are doing politics. This happens a lot, and it brings the scientific process to a halt on the question at issue.
In the case at hand, the original authors have done a very poor job of describing what they have done, and an extremely poor job of defending their work. Their refusal to publish their source code for their analysis gives credibility to their critics.
There are certainly legitimate cases where code ought not be published. If a lab has spent many, many years developing a framework for solving a certain type of problem and wants to get the most advantage out of that framework before releasing it, they may reasonably want to limit it's disemination for a while. But those sorts of reason don't apply in this case, and the source should be made available to anyone who wants to reproduce their actual results. That would just be good science.
--Tom
Blasphemy is a human right. Blasphemophobia kills.
But if the consensus has arisen because the hypothesis has been 'proven' (ie hasn't failed any of the tests it has been subjected to so far) then you're golden. The problem for the guys on the skeptic side of this argument who keep repeating the 'science isn't about consensus' mantra is that the scientific consensus regarding GHG/CC is of this latter type.
Fact (1) Global warming via GHGs (including, but not restricted to, CO2) is most definitely real - observations of the conditions on Venus, Luna and Mars give clearcut demonstrations of the effect in action.
Fact (2) The rise in CO2 concentrations in the earth's atmosphere is most definitely real - we have ice core records going back 700k years and CO2 levels have never been as high as they are now. The second order delta is also extremely large, which is likely to prove significant as we go forward.
Fact (3) The anthropogenic origin of this CO2 is most definitely real - the isotopic measurements of atmospheric CO2 point to fossil sources for this rise over the past 250 years and we have estimates of the state of the significant carbon sinks that correlates well with our estimates of fossil fuel consumption since 1750.
The combination of facts (1) and (2) gives rise to the hypothesis that we should expect to be observe a measurable warming signal that is world-wide and across many different categories of instrumentation and temperature proxies. This signal has indeed been observed in many different places, across many different data series, using many different instruments and observational protocols - promoting the hypothesis to a theory and bringing us to fact (3) which strongly implies that doing something about this CO2 forcing is within our capability as a civilisation if we decide that the potential downsides of the forcing warrants an intervention. An assessment of the likely effects of the observed forcing and investigations of the practicality of various potential interventions would seem to be in order. Metaphorically sticking fingers in our ears and shouting 'Lalala. I can't hear you!' would be.... unwise.
The MBH98 and subsequent papers are a very small part of the supporting data for Climate Change Theory, so in many ways the MM critique is a bit of a sideshow - it would (if it proves well founded, which is very much in doubt) knock out one piece of a much larger observational corpus, the implications of which would still need to be addressed in the policy arena. The extent to which the MM critique is spun up into 'Climate Change Theory Is Bunk' by people who want to forestall any consideration of the policy implications of fact (3) and carry on with the finger sticking/lalala shouting is a matter for some concern however.
Regards
Luke
#include witty_one_liner.h
One's interests in keeping clients does not entitle you to make a scientific claim that cannot be peer reviewed. If a paper such as Mann is now regarded as fact, and indeed, makes policy, despite the obvious sloppiness regarding its data management process, then, what is the point of science anyway?
Science is supposed to be about peer review, rigor, that every assumption behind every assertion can be challenged. If, all we have is someone with a Phd can claim that they have a fact as our science, then, what is the point of even trusting them?
Without independent verification and an open process, there's nothing to separate scientists from creationists, and the people are going to pick whoever makes the most attractive sales pitch.
This is my sig.
You also funded Microsoft if you purchased anything from them. It does not mean you should be able to see the source for anything at all.
Why not?
There is a bill before congress right now that says basically that - in relation to automobiles. It says basically that people have a right to be able to fix thier own autos and manufactures do NOT have the right to make you go to a dealer for repairs because they hide the source for automotive computer systems.
Now living in a country where so many people can fix software, is it so hard to see that indeed ther should ALSO be a right for a consumer to fix his or her own software if it is not working? Why should you have to go back to the software "dealer" to fix a problem. There's not even anything like a software Lemon Law to protect you!
It's not that hard to see a bill like that passing someday - perhaps twenty years before the heads of government reach that degress of sophisitication is thinking, but it is not unlikley to see in our lifetimes.
"There is more worth loving than we have strength to love." - Brian Jay Stanley
You do know that Mann writes this website, right? You do realize that the source of your argument (http://www.realclimate.org/) is a shill for Mann and his cronies?
Second of all, there was a flaw in the original algorithm that was pointed out by McIntyre and McKitrick before they even got to the bad data being put into the equation.
And, to top it off, Mann's equation always produces hockey-stick graphs, even with randomly distributed data.
Don't point at Mann's own site as a defense of Mann.
Life, the Universe, and Everything... in my image.
So, now someone's reputation is all that's important to their scientific work? Better throw out relativity because that was written by a lousy patent clerk.
Life, the Universe, and Everything... in my image.
1) Science functions only on open review. If you can't duplicate someone's results, they are useless (c.f. Ponds and Fleischman [sp?]). A scientific result is only of value if it describes a consistent replicatable process. This is why I consider the closed source work to be completely meaningless. It may be perfect, it may be bug-ridden garbage, we'll never know!
2) Every tax paying American has paid for my code and work. While I regularly feel they're not getting their money's worth, I definitely don't feel they're paying me to enrich me. They are, in a very real sense, my bosses, and I AM obligated to report to them, if they care. Think of it as a company requiring rights to your work.
3) As an academic working on a fairly limited budget, open source and free software have been a godsend for me and everyone else I know. We run linux because it's more efficient, secure and FREE; we use free or open-source compilers; and we cobble together high-perormance computers and beowulf clusters out of miscellaneous bare metal and lots of googling. The only piece of software I routinely have to pay for is MATLAB.
... grumble, grumble, grumble, mutter, mutter, Millenium... Hand... Shrimp, I tol' 'em, I tol' 'em.
Basically the government might give with one clenched fist, and take back with a team of mules pulling. The National Security Archives are one group of folks constantly struggling with the layers of governmental coverups. It's ongoing and pretty telling. They are having mixed results, some good finds, then a lot of what they are calling "over classification and pseudo classification" still existing. And then the problem becomes getting the information out to joey and janey citizen and voter, the "news" only mostly covers current, people have just been conditioned to accept todays fairy tales as "data and fact", over and over again. Then years later the real story comes out, by then it's too late to influence elections, etc. Look at the finally revealed data on the "Tonkin Gulf attack" that was the primary "lawful" reason for the Viet Nam war. They have (relatively historically recently)finally and quietly admitted it was an invention, but years too late to make it matter for most purposes.
So, in part I agree, some of what the government does needs to be kept secret, but it appears quite a bit is still overzealously kept hidden, primarily to protect the guilty-of-corruption-and-malfeasance aspects of government.
However there was a link to McIntyre and McKitrick's website in the topic summary. Why was it relevant for Timothy to include that link, but not include a link to the matching item on RealClimate.org? Is it just non-scientists who are allowed to have weblogs about this stuff?
Regards
Luke
#include witty_one_liner.h
It's worth noting that, while it makes sense that taxpayer-funded research should generate 'open-source' solutions, federal law dictates otherwise.
The Bayh-Dole Act was passed 25 years ago, which dictates:
So in other words the government has dictated since 1980 that government-funded research should not produce open-source solutions, necessarily, as the results of research are to be considered private-sector profit-generating centers for the host universities. (The implications for the 'next BSD4.3 TCP/IP stack', or similar advanced research, are obvious.)
Anyway, regarding the 'hockey stick' controversy, Tim Lambert's weblog is worth a read.
One's interests in keeping clients does not entitle you to make a scientific claim that cannot be peer reviewed. If a paper such as Mann is now regarded as fact, and indeed, makes policy, despite the obvious sloppiness regarding its data management process, then, what is the point of science anyway?
As sad as it is to say, people will believe the Mann paper no matter what is published. Look, the source data are published already and people still believe. Numerous independent reviewers (which is to say _everyone_ else) have debunked Mann and people still believe.
They don't believe Mann because it's verifiable (which it isn't), they believe it because they want to. Or, they claim to believe Mann so that they can justify the self-serving actions they want to take.
The fight we're fighting isn't to convince people that global warming is happening. Really. It isn't. What we're really fighting is to get them to do something about it. It won't be until _after_ we've sold them on taking action that they'll admit that global warming exists.
There's no need to publish Mann's code to peer review its science. Peer review has already happened. The scientific community is already convinced. The only people still claiming to be unconvinced are those who ignore anything that doesn't suit their interest.
"We reject as false the choice between our safety and our ideals." --The American President (20.1.2009)
Uhrm ... where? I haven't been able to find any code on any on of the pages mentioned. I agree it's essential to disclose all data and source code ...
Unless I'm mistaken...
Source Data
Blaze a trail to the New World
The whole point of science is not so that we can trust the opinions of scientists, it is so that scientists can give us repeatable steps to demonstrate a new point.
This whole notion of "it makes the scientists happy so we should just trust them" goes against every single thing that we in the west have fought for since the renaissance.
Your whole argument illustrates this problem precisely. You argue that, "well, even though the key piece of statistical evidence in global warming is questionable, we should still believe in the conclusion."
This is so wrong.
Maybe if scientists published all of their data in a uniform format, to a uniform site, with exact steps to reproduce, all of their source data, and how they draw conclusions from them, then, you might have a field that is useful. But right now, you have got hyper expensive journals all over the place as a repository for articles that only sketch out a discovery and not actually do it, and that simply is not good enough to be taken credibly.
The scientific process is excellent. But today's scientific product sucks.
This is my sig.
Lars T.
To the guy who modded me down from perfect to terrible Karma - Apple haters still suck