Slashdot Mirror
Temperature Data Wants To Be Free
An anonymous reader writes "The UK's Met Office Hadley Centre and University of East Anglia have been refusing access to the data used for their global climate averages and scientific studies. A copy of the data has leaked, and attempts continue to accomplish the release of the data by whoever maintains it. Excuses have included confidentiality agreements which cannot be verified because no records were kept, mention of the source has been removed from the Met Office web site, and IPCC records were destroyed."
There are lots of data sources which are perfectly reasonable to use. NOAA's data being probably the best and most comprehensive.
Yes, the UK is turning into a strange parody of itself with its attempts to close the government to the public on the one hand and monitor citizens very closely on the other. But it's not the only game in town. Despite my own country's recent 8 year slump towards the same type of fascist state as Britain, the US scientific community is still one of the best and most open in the world.
So come and get your data from us, ya'll.
This brings back a controversy from almost pre-internet times. The UK Government had a database of something which may have become damaging in the long term. It might have been data on cancer cases near nuclear power plants, or something along those lines. The Government announced that the database would be deleted because it was too expensive to store. It might have been a hundred meg or so. People were offering the relevant government agency free DAT tapes so save the life of the data. Of course, storage was never the issue.
http://michaelsmith.id.au
Radical deniers have destroyed the evidence of global climate change!!! Will the Vast Right Wing Conspiracy stop at nothing?
http://marriedmansexlife.com/
Data from a particular NERC (UK Research Council) project I'm involved with are allowed to be kept by the researchers for a certain amount of time (18 months, maybe?) but then have to be released to the BADC: http://badc.nerc.ac.uk/home/index.html - this gives the researchers time to do some analysis and get some papers out on all the hard work they've done, but obliges them to release the data to the community.
Some of the BADC data sets are restricted to non-commercial use only, so you need to flash your 'Academic Investigator' magic card at them to get it. These guys keep good metadata and license agreements and all that stuff. There's even some datasets from CRU, unrestricted (registration required).
This may be a very important story, but it references as evidence two websites which are used by conspiracy nuts, one of which appears to be broken - not /.ed, just broken - and no independent confirmation of the claims. Can anybody give any links to any mainstream news or science sites which are reporting this?
From scarped cliff or quarried stone she cries "A thousand types are gone, I care for nothing, no not one."
This is the first time I've seriously begun to question whether or not the global warming studies are in fact legitimate. If they won't allow free access to the data, so others can verify results or run it through alternative (or more refined) climate models, then the very obvious question becomes "why?"
What exactly is it they so keen on hiding that they'll remove all source citations from their publicatons?
NOTE: I am not about to buy into the fossile-fuel-funded arguments that global warming "isn't real"...it's very real, as anyone living in the northern lattitudes can trivially see. Even in London it's obvious that insects and plantlife that never used to thrive this far north now do...but anectdotal evidence, even as widespread and pervasive as this, is no substitute for rigorous scientific study, and I repeat the question: what the hell is it these people are trying to hide? There's no excuse for keeping data that is so fundamental to scientific inquiry, and has such a profound effects on public policy, secret.
The Future of Human Evolution: Autonomy
Because it's not science if you hide your data and methods and only release them to people you like. This isn't about misrepresentation or whatever dirt you want to throw - this is about access and transparency. Anything less is not science. It's as simple as that.
A key ingredient of the scientific method is exposing your methods to the cold hard light of day and making sure they withstand scrutiny. If you only show it to people you like, that is hardly serious scrutiny. It needs to stand up to scrutiny from people you don't like.
When you "BELIEVE" science you're just another religion.
In fact, open source science is the BEST and ONLY WAY to avoid science from becoming the new religion as it has, for example, in the climate debates.
The scientific method is the tool for vetting the works of science and if the work of science is closed and secret and kept close to the scientists chests by refusals to share their data, methods, source codes, procedures, etc... then their work can't be verified and might as well be works of fiction just like those of any religious cleric or priest or nutter.
If you can't take others vetting your scientific work then maybe you don't belong in science?
Open Source Science raises the bar and will in the long run improve the quality of the science that is done. Some progress is being made, much more needs to be done.
Let's see, we're supposed to spend literally trillions of dollars to fix global warming, yet we can't see the raw data the hysteria is based on?
WTF!?!?!
Along the same lines, when is the source code used for the climate models going to be published and thoroughly reviewed?
If AGW is in fact true, it can withstand the scrutiny.
The troll from ClimateAudit not being able to get free access to the source data isn't the same as there not being data. No one will give him data unless they have to because he dishonestly misrepresents it. And why the hell the UK government should spend ANY UK taxpayers money to even consider his request is beyond me. He's not British.
I would also note that the owner of one of the two sites mentioned in your story (Anthony Watts) has just employed the DMCA to have a video criticizing his Surface Stations project pulled from Youtube.
The important question which I've never seen the math for is how much CO2 is output by random natural events during a certain time period versus how much we output currently.
We are taking a few hundred million years worth of biomass and burning it up in a about a hundred and fifty. Perhaps this has no effect on the environment, but I think it's prudent to make sure that we don't send the climate into a self-feedback loop that destroys our way of life. It's not as if riding around in traffic or having an iPod is worth giving up food and water.
What's missing here are some additional context facts; recognizing that the data are not UK data per se. Data from many countries has been collected and collated at the CRU (Univ. of East Anglia) and which feeds into some of the UK Met Office work. Some of this data were collected under the arrangement that the source data were not to be made public because of commercial or other interests. Outside of the USA this is quite common -- that national meteorological services (tasked with maintaining a national observing system and archive) treat their data as a commercial product -- and so they will not release it to just anyone. The fact that I and others think this is wrong and inhibits science is not the issue, the reality is that many countries are not willing to freely release their data. So the CRU and Met Office are between the rock and a hard place; publicize the data and risk ruining their relationships with the data sources, or hold onto the data so that they can keep the data stream flowing and be able to produce the valuable derivative products.
It is that submitter, or Slashdot itself, linked to it through nyud.net. Apparently the site doesn't allow that. Just take that out of the URL, it works fine. The site in question is run by Steve McIntyre. While certianly not a disinterested party (then again people who are involved in something are rarely disinterested) he does have some credibility. He was one of two people who worked on the whole "hockey stick controversy" in terms of showing that the model used to generate the graph was flawed (the model generated a similar shape graph with random inputs).
It is always a real red flag when data is withheld. The core of science is that "ideas are tested by experiment." Ok well that means that, for science to work, others have to be able to check your work. You have an idea and say "Here's my idea and here's my support." Ok well your support needs to include ALL your data, your methods and so on. Why? So that others can check your work. Only then, after they've repeated and independently verified your results, can we start to feel confident your idea might be correct. To me, hiding data says one of three things is going on:
1) You are dealing with something commercial, that is being held secret so you can market it. Ok well that shouldn't be the case here.
2) The data in fact does NOT support your conclusion, however you don't want to admit you are wrong and thus are trying to suppress it. Perhaps you are worried you'll lose grants.
3) You suck at the science. You think that science is a process where you, the scientist make a claim and the rest of the world just has to listen to you.
4) You are a charlatan, a con man, and you are trying to convince people of something that isn't real, you are trying to sell them snake oil as it were.
I just can't see any legit reason in a pure scientific study why all the data wouldn't be made available for all to see. That it isn't really sets off warning bells in my head. I've read papers like this in the behavioral sciences and always what I see happening is that their experiment was basically a bust, it falsified their hypothesis, or simply produced inconsistent results. However they don't want to admit it, so they find a way to tweak the numbers and then refuse to release full methodology and results.
So this worries me. If climate change is truly a threat to humans, then it should be in the interests of everyone that all the data is made available, unedited, unhindered, so that the theories can be checked and rechecked. Science should be allowed to proceed with as little barriers as possible so that it can proceed as rapidly as possible because the matter is of such importance.
Why did you people moderate this as insightful? Steve McIntyre has shown in the past how various erroneous statistical analysis of climate related data has been dishonestly misrepresented by those in the "warmist" camp themselves. I'm thinking of the infamous Mann hockey stick, as well as various offerings from James Hansen and recently Steig et al. Unlike a lot of Climate Scientists (and the met office itself), McIntyre publishes in full his code, data and methods. I hardly think he can be reasonably described as a troll, unlike the poster above.
Denalists? So basically when you don't like someone's opinion, you make up a new, derogatory term to try and marginalize them? That isn't science, that is marketing. In particular, it is the kind of marketing con men do. When people question their products/methods, they shout down the critics, they deride them, they call them names. They basically try to make it look like you must be retarded if you don't agree with them.
You are also pulling another con man trick: The appeal to authority. That a site is run by "climate scientists" or is not, doesn't matter. Science isn't about who has the authority in a certain area, it is a process for finding out about the world. So trying to say "Well this site is run by climate scientists, this one isn't," doesn't strengthen your argument. That is along the same likes of "4 out of 5 dentists agree!" Ok well so what? Maybe 4 out of 5 dentists are mediocre, and the excellent 20% realize that it doesn't matter?
There is also the matter of what is a climate scientist? This isn't a degree listed at most universities, and didn't exist at all until recently. If you look at the people who run realclimate you find their PhDs are Applied Mathematics, Geology, Oceanography, and such. None of them have a degree in "climate science." So what a climate scientist is, is simply someone who studies the climate. Ok, fair enough, however that does mean it isn't an exclusive club that only certain people can be members of. For that matter, Watts is a meteorologist, which is also on the topic of climate studies.
None of that means a given person is right or wrong, but it is incorrect to appeal to authority and try and claim that "Oh realclimate is run by climate scientists so they are the only place you can trust." No, that's not the case. Science doesn't work like that.
When you pull shit like this, it really doesn't help your case. If you disagree with the theory someone is putting forth, or their criticisms of a theory, deal with that. Don't play salesman/con man tricks. To me, it makes it look as though you've something to hide.
The thing I cannot understand is this. We have a bunch of scientists, lots of them. Starting with Michael Mann in front of Wegman, but including Jones, Thompson, lots of really well known and respected people. They have all done work which supposedly proves that the human race on Earth is facing catastrophe. They supposedly have decisive evidence for this, in the form of data and code.
We then have a lot of sceptics who allege that the data does not exist, is not as described, and the code used to process it does not do what it is said to do, and that there is no such threat as described, or at leas that there is no evidence for one.
You would expect the scientists to immediately produce their evidence and their code and to silence debate once and for all. It would be so simple, it would just be end of story, and now lets focus on what to do about it all. But they do not. Instead they refuse to reveal anything. Jones, for instance, refused to even reveal the names of the stations in China on which his study was based. Mann would not reveal the algorithm which generated the hockey stick to a Congressional Committee. Thompson is silent. Yet supposedly this secret evidence proves decisively, contrary to the claims of sceptics, that the future of the human race is under severe and imminent threat?
It makes absolutely no sense. They never give any reasons for refusing that make any sense either. Sometimes it is commercial considerations. What commercial considerations can there be that outweigh the possible extinction of humanity? Sometimes it is, as Jones once is reported to have said, that they do not want people trying to poke holes in it. WTF??? Sometimes, as with Thompson's ice core data, there is just silence.
It is very hard to believe that this wonderful evidence really exists, and really is as represented. Or maybe it is, and they really do not want to convince everyone of the threat? I don't know, but the story as told makes absolutely no sense. Something is not right here.
Some of the BADC data sets are restricted to non-commercial use only, so you need to flash your 'Academic Investigator' magic card at them to get it. These guys keep good metadata and license agreements and all that stuff. There's even some datasets from CRU, unrestricted (registration required).
Where does that leave the hobbyist researchers then?
Today most household computers are potent enough to be able to sift through amounts of data that we only could dream about a few years ago.
Don't forget that the collection of the raw data has been done through the money of the tax payers. It is of course possible to have a reasonable fee for obtaining a copy in some cases, but it may as well be put on the web these days.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
I'll quote Feynman, since he put it really well:
-----
It's a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty--a kind of leaning over backwards. For example, if you're doing an experiment, you should report everything that you think might make it invalid--not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you've eliminated by some other experiment, and how they worked--to make sure the other fellow can tell they have been eliminated.
Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can--if you know anything at all wrong, or possibly wrong--to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.
-----
Remember: In science, we don't prove things true, we show them to be not false. Same thing? Not hardly. For a complete discussion on the topic, read the Logic of Scientific Discovery by Karl Popper. However what it comes down to is you do not do a test, and then prove a theory true. That can't be done. What you do is come up with a way to falsify your theory, that is to say you come up with a test that says "If things don't come out this way, we know this theory is wrong." You run the test, things come out that way. You have failed to falsify the theory, and we are now more certain it is true. The more than is done, the more certain we are a theory is correct. Each time we attempt to falsify the theory and fail, we are more sure it must be the truth.
If we do then falsify it, the theory has to be redone. That doesn't mean you toss the whole thing out, it may just mean some refinement is needed. For example you have a theory that predicts when X happens Y will results. In 400 tests, this is the case, however 3 new tests show it isn't. What you discover is that in all those tests, A was also present. You the refine your theory: Y will result from X, except in cases where A is present. Your theory is now a little more narrow in application, and fits with the evidence. Perhaps later you find out what A does, and incorporate that in to a more general theory.
The point of all this is that real science is all about trying to prove your theory wrong. You do everything you can to prove it wrong, then have other people do what they can to prove it wrong. When all of you fail at doing that, when the theory has been refined such that it fits all the evidence and you can't figure out how else to test it, then it is most likely the truth. THAT is what scientific rigor is about. It isn't about coming up with a theory, ignoring data you don't like, showing it to a few people who agree with you, and saying "Ok, we proved this true and nobody else can look at it."
The problem appears to be that the data itself was collected and supplied by various combinations of public and private entities throughout the world, and collectively released under a non-free license under which researchers aren't able to publically redistribute the data set. The British government also must respect the contract under which it obtained the data set. Now, you can argue that the data should be free in the first place, or that there should be no copyright law for data, or that there is a public interest in violating copyright law and the contractual obligations if you believe that this is a national security issue, etc.
Unfortunately the conspiracy theorists see this lack of public data as further evidence of the big conspiracy. Yes, it would be better if the complete data set were public domain. No, the data set being distributed under a less permissive license does not mean that global warming is not happening.
It should be noted that this is not a unique case - there are many instances where researchers at universities are given access to commercial or otherwise non-public-domain data sets which they use in their research and are unable to legally reproduce. Does this mean their research isn't following the scientific method? Not really - as long as other researchers are able to access the data set and reproduce the research, then it is science. The scientific method doesn't require that everyone in the world is able to reproduce your experiments, although it certainly does help.
That's an interesting point. It sounds a lot like something McIntyre would love his adversary to have said, if he could only get him to say it.
I've seen a fair amount of McIntyre's work and I get a hinky feeling from some of it. Personally, I'd like to believe what he says because I enjoy using fossil fuels and would love to think that they are completely harmless and will last forever and always be sixty cents a gallon if we could only get Al Gore to STFU. It seems to be more complicated than that, though.
You are welcome on my lawn.
Your note indicates you understand research has to be replicated to be science. This is very good step in the right direction. Do you also understand that you cannot just turn the data over to those who agree with you? People who disagree with your conclusions have to have access to the data, otherwise it is just pseudoscience.
"Where does that leave the hobbyist researchers then?"
You have to get out of the basement, go outside and talk to people. Same as all the professional researchers. If you show up at the appropriate guy's office at the nearest university, tell him you want to collaborate and do some research, for free, he'll likely be happy to get you your data in exchange for some input and his name on the paper.
Same with scientific journals. Free access equal to the level of any academic is as far away as the nearest university library. A little slower access is probably closer - the nearest local library.
From his blog: "For all of our UK readers, now is the time for all good citizens to come to the aid of their country (and science). The Met Office refuses to release data and methodology for their HadCRUT global temperature dataset after being asked repeatedly. Without the data and procedures there is no possibility of replication, and without replication the Hadley climate data is not scientifically valid. This isnâ(TM)t just a skeptic issue, mind you, others have just a keen an interest in proving the data. What is so bizarre is this. The FOI request by Steve McIntyre to the Met Office was for a copy of the data sent to Peter Webster. If the restrictions on the data hold for Steve McIntyre, why did they not prevent release of the data to Webster? When asked by Warwick Hughes for this data, Dr. Jones famously replied: Even if WMO agrees, I will still not pass on the data. We have 25 or so years invested in the work. Why should I make the data available to you, when your aim is to try and find something wrong with it."
Slashdot's rate-of-post filter: Preventing you from posting too many great ideas at once.