Slashdot Mirror
Temperature Data Wants To Be Free
An anonymous reader writes "The UK's Met Office Hadley Centre and University of East Anglia have been refusing access to the data used for their global climate averages and scientific studies. A copy of the data has leaked, and attempts continue to accomplish the release of the data by whoever maintains it. Excuses have included confidentiality agreements which cannot be verified because no records were kept, mention of the source has been removed from the Met Office web site, and IPCC records were destroyed."
... refusing access to the data used for their global climate averages and scientific studies.
I realize governments are really in to wasting money and all, but this is ridiculous. The UK government has spent who knows how much money on a completely worthless study. Studies mean nothing without data.
We don't believe in it here at the Met Office.
There are lots of data sources which are perfectly reasonable to use. NOAA's data being probably the best and most comprehensive.
Yes, the UK is turning into a strange parody of itself with its attempts to close the government to the public on the one hand and monitor citizens very closely on the other. But it's not the only game in town. Despite my own country's recent 8 year slump towards the same type of fascist state as Britain, the US scientific community is still one of the best and most open in the world.
So come and get your data from us, ya'll.
This brings back a controversy from almost pre-internet times. The UK Government had a database of something which may have become damaging in the long term. It might have been data on cancer cases near nuclear power plants, or something along those lines. The Government announced that the database would be deleted because it was too expensive to store. It might have been a hundred meg or so. People were offering the relevant government agency free DAT tapes so save the life of the data. Of course, storage was never the issue.
http://michaelsmith.id.au
Radical deniers have destroyed the evidence of global climate change!!! Will the Vast Right Wing Conspiracy stop at nothing?
http://marriedmansexlife.com/
See, I tried to warn everyone about Gore's new world order , but no one would listen.
It was all oil , bush , climate change and look how you all ended up.
Cruise TT
That, or someone mixed up various censorship blacklists with the data,
signature is pants
This may be a very important story, but it references as evidence two websites which are used by conspiracy nuts, one of which appears to be broken - not /.ed, just broken - and no independent confirmation of the claims. Can anybody give any links to any mainstream news or science sites which are reporting this?
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
This is the first time I've seriously begun to question whether or not the global warming studies are in fact legitimate. If they won't allow free access to the data, so others can verify results or run it through alternative (or more refined) climate models, then the very obvious question becomes "why?"
What exactly is it they so keen on hiding that they'll remove all source citations from their publicatons?
NOTE: I am not about to buy into the fossile-fuel-funded arguments that global warming "isn't real"...it's very real, as anyone living in the northern lattitudes can trivially see. Even in London it's obvious that insects and plantlife that never used to thrive this far north now do...but anectdotal evidence, even as widespread and pervasive as this, is no substitute for rigorous scientific study, and I repeat the question: what the hell is it these people are trying to hide? There's no excuse for keeping data that is so fundamental to scientific inquiry, and has such a profound effects on public policy, secret.
The Future of Human Evolution: Autonomy
Opening Science is the way forward, the path through the darkness, the endarkenment of closed source science.
If's it's paid by the public purse it must be OPEN data that anyone can see and audit.
Science is based upon the notion of being able to validate or invalidate in whole or in part the "claims" made by various "hypotheses" put forward.
When you "BELIEVE" science you're just another religion.
When you can't audit the work of scientists whose work is the basis of public policy then you and the public are being endarkened and kept excluded. But why? For what or whose agenda?
As long as the data, the methods, the algorithms, the statical analysis, the step by step procedures are kept secret the work is suspect to scientific fraud.
Have the guts to open your science to the light of day, it will in the end be better for it once it's vetted by more eyes and brains and math nuts and others poking holes in it.
ANY AND ALL CLAIMS MADE BY PEOPLE WHO KEEP THEIR SCIENCE CLOSED AND SECRET is suspect of FRAUD. What are they hiding? Are they simply embarrassed to admit that they might be wrong? That they've made mistakes? That they are afraid that others might gain an edge in the grant process and shut them out of funding?
Open Source Science is the way forward through the darkness into the light that empower verification and falsification and thus progress EITHER way!!!
This site has some excellent quotes and articles on the topic: http://www.pathstoknowledge.com./
"The meaning of the world is the separation of wish and fact." - KURT GÖDEL
"According to Peirce's doctrine of fallibilism, the conclusions of science are always tentative. The rationality of the scientific method does not depend on the certainty of its conclusions, but on its self-corrective character: by continued application of the method science can detect and correct its own mistakes, and thus eventually lead to the discovery of truth".
A guiding principle for accepting claims of catastrophic global events, miracles, incredible healing, invisible friends, or fill in the blank is:
"extraordinary claims require extraordinary evidence." - Carl Sagan
"Two important characteristics of maps should be noticed. A map is not the territory it represents, but, if correct, it has a similar structure to the territory, which accounts for its usefulness." - Alfred Korzybski
"Science is a search for basic truths about the Universe, a search which develops statements that appear to describe how the Universe works, but which are subject to correction, revision, adjustment, or even outright rejection, upon the presentation of better or conflicting evidence." - James Randi
"Hypotheses are nets: only he who casts will catch." - Novalis
When you "BELIEVE" science you're just another religion.
In fact, open source science is the BEST and ONLY WAY to avoid science from becoming the new religion as it has, for example, in the climate debates.
The scientific method is the tool for vetting the works of science and if the work of science is closed and secret and kept close to the scientists chests by refusals to share their data, methods, source codes, procedures, etc... then their work can't be verified and might as well be works of fiction just like those of any religious cleric or priest or nutter.
If you can't take others vetting your scientific work then maybe you don't belong in science?
Open Source Science raises the bar and will in the long run improve the quality of the science that is done. Some progress is being made, much more needs to be done.
Let's see, we're supposed to spend literally trillions of dollars to fix global warming, yet we can't see the raw data the hysteria is based on?
WTF!?!?!
Along the same lines, when is the source code used for the climate models going to be published and thoroughly reviewed?
If AGW is in fact true, it can withstand the scrutiny.
The important question which I've never seen the math for is how much CO2 is output by random natural events during a certain time period versus how much we output currently.
We are taking a few hundred million years worth of biomass and burning it up in a about a hundred and fifty. Perhaps this has no effect on the environment, but I think it's prudent to make sure that we don't send the climate into a self-feedback loop that destroys our way of life. It's not as if riding around in traffic or having an iPod is worth giving up food and water.
What's missing here are some additional context facts; recognizing that the data are not UK data per se. Data from many countries has been collected and collated at the CRU (Univ. of East Anglia) and which feeds into some of the UK Met Office work. Some of this data were collected under the arrangement that the source data were not to be made public because of commercial or other interests. Outside of the USA this is quite common -- that national meteorological services (tasked with maintaining a national observing system and archive) treat their data as a commercial product -- and so they will not release it to just anyone. The fact that I and others think this is wrong and inhibits science is not the issue, the reality is that many countries are not willing to freely release their data. So the CRU and Met Office are between the rock and a hard place; publicize the data and risk ruining their relationships with the data sources, or hold onto the data so that they can keep the data stream flowing and be able to produce the valuable derivative products.
It is that submitter, or Slashdot itself, linked to it through nyud.net. Apparently the site doesn't allow that. Just take that out of the URL, it works fine. The site in question is run by Steve McIntyre. While certianly not a disinterested party (then again people who are involved in something are rarely disinterested) he does have some credibility. He was one of two people who worked on the whole "hockey stick controversy" in terms of showing that the model used to generate the graph was flawed (the model generated a similar shape graph with random inputs).
It is always a real red flag when data is withheld. The core of science is that "ideas are tested by experiment." Ok well that means that, for science to work, others have to be able to check your work. You have an idea and say "Here's my idea and here's my support." Ok well your support needs to include ALL your data, your methods and so on. Why? So that others can check your work. Only then, after they've repeated and independently verified your results, can we start to feel confident your idea might be correct. To me, hiding data says one of three things is going on:
1) You are dealing with something commercial, that is being held secret so you can market it. Ok well that shouldn't be the case here.
2) The data in fact does NOT support your conclusion, however you don't want to admit you are wrong and thus are trying to suppress it. Perhaps you are worried you'll lose grants.
3) You suck at the science. You think that science is a process where you, the scientist make a claim and the rest of the world just has to listen to you.
4) You are a charlatan, a con man, and you are trying to convince people of something that isn't real, you are trying to sell them snake oil as it were.
I just can't see any legit reason in a pure scientific study why all the data wouldn't be made available for all to see. That it isn't really sets off warning bells in my head. I've read papers like this in the behavioral sciences and always what I see happening is that their experiment was basically a bust, it falsified their hypothesis, or simply produced inconsistent results. However they don't want to admit it, so they find a way to tweak the numbers and then refuse to release full methodology and results.
So this worries me. If climate change is truly a threat to humans, then it should be in the interests of everyone that all the data is made available, unedited, unhindered, so that the theories can be checked and rechecked. Science should be allowed to proceed with as little barriers as possible so that it can proceed as rapidly as possible because the matter is of such importance.
My conclusion so far: it's very unlikely not to be co2 responsible for most of the warming we've observed since the 70s, it's likely to get much worse, and there don't seem to be any viable alternative explanations.
Denalists? So basically when you don't like someone's opinion, you make up a new, derogatory term to try and marginalize them? That isn't science, that is marketing. In particular, it is the kind of marketing con men do. When people question their products/methods, they shout down the critics, they deride them, they call them names. They basically try to make it look like you must be retarded if you don't agree with them.
You are also pulling another con man trick: The appeal to authority. That a site is run by "climate scientists" or is not, doesn't matter. Science isn't about who has the authority in a certain area, it is a process for finding out about the world. So trying to say "Well this site is run by climate scientists, this one isn't," doesn't strengthen your argument. That is along the same likes of "4 out of 5 dentists agree!" Ok well so what? Maybe 4 out of 5 dentists are mediocre, and the excellent 20% realize that it doesn't matter?
There is also the matter of what is a climate scientist? This isn't a degree listed at most universities, and didn't exist at all until recently. If you look at the people who run realclimate you find their PhDs are Applied Mathematics, Geology, Oceanography, and such. None of them have a degree in "climate science." So what a climate scientist is, is simply someone who studies the climate. Ok, fair enough, however that does mean it isn't an exclusive club that only certain people can be members of. For that matter, Watts is a meteorologist, which is also on the topic of climate studies.
None of that means a given person is right or wrong, but it is incorrect to appeal to authority and try and claim that "Oh realclimate is run by climate scientists so they are the only place you can trust." No, that's not the case. Science doesn't work like that.
When you pull shit like this, it really doesn't help your case. If you disagree with the theory someone is putting forth, or their criticisms of a theory, deal with that. Don't play salesman/con man tricks. To me, it makes it look as though you've something to hide.
See
http://www.sourcewatch.org/index.php?title=Anthony_Watts
"Watts was a speaker at the International Conference on Climate Change (2009) organized by the Heartland Institute think tank. Watts is also listed as a speaker for the Heartland Institute's June 2009 Third International Conference on Climate Change."
Nice gigs. Wonder whether he was given a nice hotel for that...
Or Lindzen:
http://www.sourcewatch.org/index.php?title=Richard_S._Lindzen
"He is one of the leading global warming skeptics and is a member of the Science, Health, and Economic Advisory Council, of the Annapolis Center, a Maryland-based think tank which has been funded by corporations including ExxonMobil."
Ah, the joys of being in a quango!
Roy Spencer?
http://www.sourcewatch.org/index.php?title=Roy_Spencer
"Since February 2004 he has been a columnist for TCS Daily writing over forty columns, almost entirely on the the topic of global warming. Until 2006, TCS Daily was run by DCI Group, a lobbying firm that works for ExxonMobil."
Plimer?
http://www.sourcewatch.org/index.php?title=Ian_Plimer
"He is a global warming sceptic and a non-executive director of three mining companies: Ivanhoe Australia, a subsidiary of Bob Friedland's Ivanhoe Mines, as well as CBH Resources and Kefi Minerals."
How about McIntyre:
http://www.sourcewatch.org/index.php?title=Stephen_McIntyre
"Stephen McIntyre has worked in mineral exploration for 30 years, much of that time as an officer or director of several public mineral exploration companies. McIntyre is also a headliner at the International Conference on Climate Change (2009), a gathering of climate change skeptics in New York from March 8th-10th. "
(remember that ICCC is funded by the Heartland Institute).
McKitrick:
http://www.sourcewatch.org/index.php?title=Ross_McKitrick
"Ross McKitrick is an Associate Professor in the Economics Department at the University of Guelph, Ontario, and, since October 2002, has been a Senior Fellow at the Fraser Institute, a libertarian think tank based in Vancouver, British Columbia"
"For example in late 1999 defended the Fraser Institute when it criticised proposals for an Endangered Species Act in Canada. "
All on the oil-based gravy train!
The thing I cannot understand is this. We have a bunch of scientists, lots of them. Starting with Michael Mann in front of Wegman, but including Jones, Thompson, lots of really well known and respected people. They have all done work which supposedly proves that the human race on Earth is facing catastrophe. They supposedly have decisive evidence for this, in the form of data and code.
We then have a lot of sceptics who allege that the data does not exist, is not as described, and the code used to process it does not do what it is said to do, and that there is no such threat as described, or at leas that there is no evidence for one.
You would expect the scientists to immediately produce their evidence and their code and to silence debate once and for all. It would be so simple, it would just be end of story, and now lets focus on what to do about it all. But they do not. Instead they refuse to reveal anything. Jones, for instance, refused to even reveal the names of the stations in China on which his study was based. Mann would not reveal the algorithm which generated the hockey stick to a Congressional Committee. Thompson is silent. Yet supposedly this secret evidence proves decisively, contrary to the claims of sceptics, that the future of the human race is under severe and imminent threat?
It makes absolutely no sense. They never give any reasons for refusing that make any sense either. Sometimes it is commercial considerations. What commercial considerations can there be that outweigh the possible extinction of humanity? Sometimes it is, as Jones once is reported to have said, that they do not want people trying to poke holes in it. WTF??? Sometimes, as with Thompson's ice core data, there is just silence.
It is very hard to believe that this wonderful evidence really exists, and really is as represented. Or maybe it is, and they really do not want to convince everyone of the threat? I don't know, but the story as told makes absolutely no sense. Something is not right here.
I'll quote Feynman, since he put it really well:
-----
It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty--a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid--not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked--to make sure the other fellow can tell they have been eliminated.
Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can--if you know anything at all wrong, or possibly wrong--to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.
-----
Remember: In science, we don't prove things true, we show them to be not false. Same thing? Not hardly. For a complete discussion on the topic, read the Logic of Scientific Discovery by Karl Popper. However what it comes down to is you do not do a test, and then prove a theory true. That can't be done. What you do is come up with a way to falsify your theory, that is to say you come up with a test that says "If things don't come out this way, we know this theory is wrong." You run the test, things come out that way. You have failed to falsify the theory, and we are now more certain it is true. The more than is done, the more certain we are a theory is correct. Each time we attempt to falsify the theory and fail, we are more sure it must be the truth.
If we do then falsify it, the theory has to be redone. That doesn't mean you toss the whole thing out, it may just mean some refinement is needed. For example you have a theory that predicts when X happens Y will results. In 400 tests, this is the case, however 3 new tests show it isn't. What you discover is that in all those tests, A was also present. You the refine your theory: Y will result from X, except in cases where A is present. Your theory is now a little more narrow in application, and fits with the evidence. Perhaps later you find out what A does, and incorporate that in to a more general theory.
The point of all this is that real science is all about trying to prove your theory wrong. You do everything you can to prove it wrong, then have other people do what they can to prove it wrong. When all of you fail at doing that, when the theory has been refined such that it fits all the evidence and you can't figure out how else to test it, then it is most likely the truth. THAT is what scientific rigor is about. It isn't about coming up with a theory, ignoring data you don't like, showing it to a few people who agree with you, and saying "Ok, we proved this true and nobody else can look at it."
The problem appears to be that the data itself was collected and supplied by various combinations of public and private entities throughout the world, and collectively released under a non-free license under which researchers aren't able to publically redistribute the data set. The British government also must respect the contract under which it obtained the data set. Now, you can argue that the data should be free in the first place, or that there should be no copyright law for data, or that there is a public interest in violating copyright law and the contractual obligations if you believe that this is a national security issue, etc.
Unfortunately the conspiracy theorists see this lack of public data as further evidence of the big conspiracy. Yes, it would be better if the complete data set were public domain. No, the data set being distributed under a less permissive license does not mean that global warming is not happening.
It should be noted that this is not a unique case - there are many instances where researchers at universities are given access to commercial or otherwise non-public-domain data sets which they use in their research and are unable to legally reproduce. Does this mean their research isn't following the scientific method? Not really - as long as other researchers are able to access the data set and reproduce the research, then it is science. The scientific method doesn't require that everyone in the world is able to reproduce your experiments, although it certainly does help.
More than 90% of those "ten thousand" scientists who publicly support global warming did nothing at all to prove or disprove the theory - they're researchers in related (and often unrelated) fields who took government money, wrote a paper, tacked "and was caused by Global Warming" onto whatever they were working on before, and got published. Tens of billions of dollars in government money over the last couple of decades have made sure that many scientists have a distinct financial advantage if they support global warming.
When that doesn't work, there's even private money available, like the several hundred thousand dollars in "awards" given to James Hansen of NASA for coming up with the "right" numbers that seem to support AGW - for example, temperatures which (over the last decade) disagree with pretty much all of the other temperature observations reported by other organizations. Apparently, NASA took their raw data, "corrected" it, and then released it to the world in heavily edited form. Another win for "private" science.
The actual, no-kidding, original "research" (simulations that are still pretty well obscured and/or disproven) leading to the theory of anthropogenic global warming was done by a very small number of scientists. Some of them were working in fields that gave them no practical expertise in the science involved, and much of the initial (and still obscured) results were created in simulations that have since been shown to be completely false, such as the "Hockey Stick" that STILL shows up in many "serious" AGW papers. A big problem is that much of the statistical work was done by people with a very weak background in actual statistics (or just enough of a background to know how to cherry-pick numbers and formulas to get the results they wanted).
Your note indicates you understand research has to be replicated to be science. This is very good step in the right direction. Do you also understand that you cannot just turn the data over to those who agree with you? People who disagree with your conclusions have to have access to the data, otherwise it is just pseudoscience.
From his blog: "For all of our UK readers, now is the time for all good citizens to come to the aid of their country (and science). The Met Office refuses to release data and methodology for their HadCRUT global temperature dataset after being asked repeatedly. Without the data and procedures there is no possibility of replication, and without replication the Hadley climate data is not scientifically valid. This isnâ(TM)t just a skeptic issue, mind you, others have just a keen an interest in proving the data. What is so bizarre is this. The FOI request by Steve McIntyre to the Met Office was for a copy of the data sent to Peter Webster. If the restrictions on the data hold for Steve McIntyre, why did they not prevent release of the data to Webster? When asked by Warwick Hughes for this data, Dr. Jones famously replied: Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it."
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.
There's no need to be condescending, Ron.
Do you understand that copyright law covers data sets, and that you can't just issue a Freedom of Information Act requesting that the government violate copyright law?
Here's a common situation: a pharmaceutical corporation sponsors graduate research on some drug, and as part of this they provide data sets from their own experiments and other research that they have sponsored, bought, or licensed. Now I discover that a government researcher has used this data and I disagree with their conclusions, so I put in a Freedom of Information Act request for the original data set. Do you really believe that at this point the government should just relicense the data set as public domain and hand it over to me? It would fundamentally alter the concept of copyright. Now, getting rid of copyright might not necessarily be a bad thing, but it is something that needs to be decided by society in a larger context, not in the context of one single FOI request for one data set.
Copyright is a complex issue, only a few hours ago Slashdot was discussing the Copyright Status of Thermodynamic Properties
I have called for this before.
Something similar to the open source movement, but science instead of software.
The two methodologies: Open Science and Peer Reviewed Publishing can coexist. If anything, this sort of thing might actualy make the peer reviewers do something.. like actualy review the work they are signing off on.
"His name was James Damore."
I discover that a government researcher has used this data and I disagree with their conclusions, so I put in a Freedom of Information Act request for the original data set. Do you really believe that at this point the government should just relicense the data set as public domain and hand it over to me?
Yes.
Give me Classic Slashdot or give me death!
Peter Sinclair (aka greenman3610 over at youtube.com) produced a video that did a nice job of dismantling some of Anthony Watts surfacestations claims. How did Watts reply? He filed a bogus DMCA complaint and as a result, youtube pulled Sinclair's latest video. Google up the phrase "Watts up with Watts" and follow an appropriate link to see for yourself.
Now what's all this about information wanting to be free????
If you need a citation, how about looking at: http://scienceandpublicpolicy.org/ [...]
The site and its owners may have it's own axes to grind (don't we all) but let's stop with the "oil-company" conspiracy theory. It's just another distraction tactic by those with something to hide. Publish your data and models please.
The Science and Public Policy Institute (SPPI) is a global warming skeptics group which appears to primarily be the work of Robert Ferguson, its President.
Prior to founding SPPI in approximately mid-2007, Ferguson was the Executive Director of the Center for Science and Public Policy (CSPP), a project of the corporate-funded group, the Frontiers of Freedom Institute.
Frontiers of Freedom receives money of tobacco and oil companies, including Philip Morris Cos, ExxonMobil and RJ Reynolds Tobacco.
According to a 2003 New York Times report, "Frontiers of Freedom, which has about a $700,000 annual budget, received $230,000 from Exxon in 2002, up from $40,000 in 2001, according to Exxon documents. George Landrith, President of FoF told the New York Times "They've determined that we are effective at what we do" and that Exxon essentially took the attitude, "We like to make it possible to do more of that".
You can't take the sky from me...
Personally, I'd love to see ALL corporate lobbying and donations to political figures banned outright, if just for the fact that corporations don't vote. Why should their interests be represented over ours?
{Yes, I KNOW it's not likely, but it's a pleasant thought, no?}
Don't tell me to get a life. I'm a gamer; I have LOTS of lives!