Hey! That's a good idea! Send one to the Liberal leader while you're at it, but word it so that he's likely to give it to the media as a shot at Harper.
They have tried this at least three times of the last few years, and each time the bill has been quietly forgotten about after public opposition and the sponsoring scapegoat, er, minister has been shuffled off to a different job.
The specs are available online. He may have just done it as an experiment. In North America a carrier will often replace your SIM card free if you tell them you lost it.
Sorry, it wasn't you. It's hard to keep track of all the different replies. You're the reasonable one so you don't stick out.
If the climatology field is trying to squelch Steve McIntyre they're doing a really crappy job of it. The guy has a pretty good publication record for a non-academic, including a paper in Nature! He's also been thanked by climate data publishing organisations for spotting problems in their data, following which they promptly incorporated his corrections, and, well, he's basically won - the hockey stick graph is discredited. As far as I can tell his position is not that the planet isn't warming (he says he doesn't know) but that he's critical of some of the methodology that's used and some of the results that have been obtained. In many cases his criticisms seem to have been successfully published in the scientific literature and incorporated by other organizations and researchers.
If you're talking about private e-mails from the CRU thing, or RealClimate.org vs. ClimateAudit.org, again, not science. The one is private communication, in which people say all sorts of stupid things, and the other is public debate which, as I said, is a disaster.
Yes, I see you're not the original poster. Still, the evidence that the scientific community is as you say is, as far as I've seen, poor. Your own justification ranged from invoking Al Gore (nothing to do with science) to claiming funding motivation. According to AGW proponents there's lots of funding available from industry for anyone willing to say global warming isn't happening, so that argument, besides being unsupported by evidence, rings hollow too.
I agree, the public debate is a disaster, but I don't see any actual evidence for scientific misconduct in the scientific community.
Do a few searches on Google Scholar. It turns up some PDFs you can read for free. Most of the papers are fairly balanced. There's none of the "OMG we're all gonna die!!" conclusions the media would have you believe. The few I read (I searched for "solar forcing, climate change" or something similar) generally arrived at the conclusion that the sun is likely responsible for some warming in the 20th century (usually quantified), but less than anthropomorphic sources.
Come on, the comparison is between Flash + some video codec (Adobe) or HTML5 + some video codec (Apple).
If you want to compare irrelevant parts of the given companies as some sort of background information why not include, say, Creative Studio (Adobe's "developer tools")? It costs more than XCode plus a computer to run it.
There are free, open, cross platform developer tools for Flash you say? There are free, cross platform developer tools for the iPhone too (GCC). In fact, XCode is just an IDE wrapped around GCC. You do have to sign your executables if you want to submit them to the app store. I'm not sure if someone has figured out a way to run codesign without actually having a Mac but I doubt such a thing is impossible.
Worse. He's comparing Flash to Apple. All of Apple. That's where I quit reading.
Even the usual comparison of Flash (proprietary) to H.264 (proprietary) is wrong. The proper comparison for video, which is the subject the article is harping on, is Flash (proprietary) to HTML5 (open standard). Most Flash video uses the encumbered H.264 codec, just as Apple has suggested H.264 is the ideal codec to use with HTML5. So the codecs are the same. What's the difference? The container, and one container is indeed proprietary while the other is an open standard.
I had a GPGPU example running on a web page in Safari (for giggles) about four years ago. If you can get full GPGPU access through a web page writing a little video decoder shouldn't be a big deal.
How is it a threat to the App Store? If you develop a Flash app you still have to sell it through the App Store.
Jobs doesn't want quick and dirty runs-(poorly)-on-everything ports in the App Store. Rather than reject them for being "crappy" he's decided to reject them because they're not written in C, C++ or ObjC.
The only real difference is that they use a script font for the title instead of Verdana. Okay, a script font might have it's uses.
They use a (italic) serif font for the headings instead of sans serif. All right, I think that's a bad choice, but there are serif fonts available in the standard web selection. Other than that, at the resolution provided I couldn't seen any real differences.
You're aware that Al Gore isn't a scientist, right?
Yes, the public discussion of global warming/climate change is very unscientific. You're aware that journalists, most Slashdot posters, lobby groups, congressmen, corporate spokespeople, most environmentalists, most bloggers and almost all members of the public are also not scientists, right?
The issues I referred to are not necessarily referred to in that particular readme. Your interpretation supports my point just as well though.
I've worked with a lot of crap code researchers (including myself) have written in all sorts of languages for all sorts of things, usually in a hurry. That's just fine for research purposes. Would I use any of it online in a hospital (I do medical research)? No way. It might kill someone.
Hard coded paths, code migration, lots of data, incorrect files, reprocessing of data, converters... all sorts of opportunities for mistakes. Many mistakes were found and recorded in that read me.
I don't think the CRU was committing fraud, nor do I suspect their conclusions are likely to be wrong, at least in the broad sweep, but now that their results (and the datasets they produce) are going to be used to help decide the future direction of civilization, don't you think it would be a good idea to go back and do things over, a bit more carefully? Just to check?
First, there is no proof there's a conspiracy to deny publication of dissenting papers. Several investigations have decided that there is no conspiracy. There is an outside chance that one little corner of science may have slipped into pseudoscience, but that's hardly justification for your statements about science in general.
The nondisclosure of data is a serious issue, but it's also not universal and even in this case it sounds like it's more due to the CRU not having the legal right to disclose the data in question, NOT to their unwillingness to do so. That's a problem with the law or with the commercial right-holders, not science. Again, even in the worst case scenario, it's not a justification for your statements about science in general.
Following are a few examples of large, publicly available scientific datasets that were assembled at considerable cost, entirely voluntarily (a small selection, several that I have personal experience with and others that I've included to try to give some breadth to the list):
There's hardly an overwhelming culture of closed and proprietary secret keeping in science as you suggest. Quite the opposite. Sure, some of the non-scientific appendages to science do have issues in that area (journals, for example) but scientists are usually all too willing to do end runs around such things. If you want to read a paper, e-mail the author and he's likely to send you a PDF despite that often being technically a violation of copyright. Failing that, go to a library and they'll let you read it, free.
"But still: at some point, it's a matter of what you, the fellow scientist, find sufficiently convincing."
I think that's the key. You give me the stats, all the information I need to make my own judgement call, and I decide whether to agree with you or not. More importantly, you can put statistically derived limits on the maximum magnitude of secondary effects so everyone can agree on just how strong your statement actually is.
Statements about climate have a pretty wide range. Journalists and some researchers who do interviews often state things like "climate change is caused by human activity" which is clearly not scientific. The IPCC report says something like "there is a 90% chance that more than 50% of the observed warming trend is due to human activity" which is much better.
One of the issues with the CRU seems to be that they're not exactly sure about what those statistical limits are on the third-party data they're using. Some of the data sound like it's delivered as a "here are the numbers" dataset without errors or adequate background on how it was processed. Some of the e-mails, and the findings of this investigation, suggest that disorganization within the CRU also led to some researchers being unsure what had already been done to their own data as well.
The CRU had to use what they had, but it's an issue that can be fixed with a bit of money and a little more work and, given how important this is, probably should be.
Yes, it is. Somewhat informal English, yes. To be completely correct I should have prepended "Have you" to the sentence, but it is common in informal English communication to omit such parts of speech when they are clearly implied.
Maybe you should avoid sarcastically calling other people "real intellectual giant[s]."
You made a sarcastic "analogy." I spit it back at you. Were you expecting a non-sarcastic response to your idiotic post?
According to other sources, BP estimates originally said 1000 barrels/day. The NOAA put it at 5 times that, 5000 barrels/day. Other independent sources, using satellite imagery, put it at 5-10k. There is one oceanographer who says it might be leaking at as much as 25,000 barrels/day. It sounds like the media like to "estimate" on the high side, with the Wall Street Journal coming in at 24k.
So the 5000 estimate seems reasonable, made by independent experts actually on the scene, and already a lot more pessimistic than BP's likely biased estimates.
50,000 seems to have come out of thin air. If you were going to engage in what-if scenarios it would be better to use BP's worst-case estimate of 163,000 barrels/day, so the 50k seems like a misunderstanding on the part of the journalist.
If they validated the use of simpler statistical methods then that validation should absolutely be part of their published research and would have certainly been available to the investigation. I've done exactly what you describe and, as I'm sure your friend does, the resulting paper always gets a paragraph saying something like:
"Analysis X and Y were performed on a randomly selected subset of the data with N=Z. X produced such and such result with estimated alpha=A, beta=B, etc. and Y produced the same result with estimated alpha=A', beta=B' etc. Although Y is known to be suboptimal in terms of statistical power, it is more conservative than X (possibly with an appropriate correction) and is much more computationally tractable. Therefore, analysis Y was used for the larger dataset."
Not sure if it's going to impact crude prices, but the article managed to both overestimate the leak rate by an order of magnitude and horribly exaggerate the size of the reservoir. It could certainly turn into a really bad disaster, but it's not the unprecedented catastrophe the article implies.
It has encouraged some needed scrutiny into drilling practices though. I like one explanation by an oil official (I think he was from BP) when asked why rig engineering drawings were frequently not signed by an engineer: oh, the regulations require that we have engineering drawings for everything, not that they're accurate.
For people who worry about running out of oil, there's enough recoverable in the tar sands in Canada and Venezuela at current prices that we're not going to run out anytime soon. As you point out, it does come at an even higher environmental cost though. On the bright side, we are making real progress on alternatives. I have a friend setting up an electronics collection and recycling shop who is seriously considering leasing a quarter section with no power service. Apparently they already has the windmills and solar cells to supply adequate power, and this is an organization operating on a shoestring budget at around 55 degrees north - not ideal territory for solar.
Present something to a scientific audience and they spend the entire presentation looking for weaknesses. At the end someone, or several someones grill you, and they usually turn out to be eminent experts in the field. The questions they ask are even tougher if you agree with them. Last time I did this was yesterday.
I've heard there are some branches of science that aren't quite like this, but I've never actually seen one first hand, and most such reports are from dubious sources.
As a neuroimager, I can pretty much guarantee you that the methods you were using probably weren't always beyond reproach.
However, I also suspect your research wasn't going to be used to set policy that has a large impact on human life. I can tell you that any work I've been involved in with potentially even moderate clinical impact has absolutely involved at least one statistician.
I'm a scientist working in medical research, occasionally involved with clinical trials. The investigatory "hey, here's a cool new technique that might be useful" stuff doesn't require me to carefully store my data, although it does require me to publish a good enough description of my methods that they can be duplicated by other people, on other data. I do keep the data around as well, actually, but I can't guarantee it will be available in twenty years.
Anything involved with a clinical trial absolutely requires that the data, code, descriptions of the code, descriptions of the methods, justification of the methods, validation of the code and methods and the infrastructure to run the code be archived in such a way that it will be available in the future. Want to make something faster with a fancy GPU algorithm? Sorry, the virtual machine that runs all the code (so that the "computer" can be archived as well) may not be able to run the GPU stuff the same way in ten years. And by the way, even the smallest trial collects orders of magnitude more data than those nine million data points.
Climate science started out more in the former situation, but it's certainly progressed into the latter. When your research has global importance to human life and well-being it's time to ramp up your standards. Should the CRU have been doing this all along? I don't know. Should climate researchers do so going forward? Absolutely.
Actually, a lot of scientific datasets are eagerly made widely available. When you collect a major dataset you write at least one paper about it. When you share that dataset with someone else and they publish papers based on it, they cite your paper describing the data's collection. That gets your paper lots of citations, which is very desirable.
My impression of the CRU climate data is that some of the data they were using was collected not by public science but for commercial purposes. Commercial entities care about how much money they make selling things, not how many citations their papers (if they have any) get. In this case I think the issue has become important enough that it would be well worth the relevant government(s) buying the required datasets and releasing them.
It's not quite as bad as you put it. There are ways to increase your certainty that errors are actually errors and not phenomenon, and when you state your results you include those numbers anyway.
Take Galileo. I duplicated that experiment way back in high school. You build a couple of ramps that you make as smooth as possible, then roll metal balls of different weight down the ramps (the denser the better, because that minimizes friction effects). You do many trials, and you do half of them with the ball-ramp combinations switched, to even out differences in the ramps. At the end you get many measurements on which you use established statistical techniques. When you're finished you don't have "objects accelerate due to gravity at the same rate regardless of their mass" you have "in our experiment we could not detect a difference in the acceleration due to gravity of objects of different mass to such and such limits with such and such confidence."
If you do your experiment carefully enough those limits will be small and you DO discover that objects of different mass in this experiment roll down the ramps at different rates, even accounting for friction effects. You've just discovered rotational momentum.
Looking at your other example, tree ring data, if you suspect that the tree ring thickness - temperature relationship changes over time, but that's all the data you have, you absolutely do not make up an "I think it's right" correction and apply it. You look for more data, such as an overlapping period where you have both reliable temperature data and tree ring measurements, that demonstrates the change. If you suspect the change is due to pollution, you look for evidence that this is true. Once you've got it all together you present your case, including all the uncertainties, so that everyone can see what you've done, what assumptions you've had to make, and what errors are likely still present.
Personally I think the scrutiny in the CRU case is misplaced. From all the information that's available, it seems that the CRU has done their best with complicated and diverse data, most of which they didn't collect. What they've done is fine for a "hey, this looks like something real" discovery. However, since they didn't collect most of the data, and since it sounds like a lot of that data was preprocessed in unknown ways before they got it, I think it would be wise to go back to the actual raw data and analyze it uniformly and openly. Yes, that will likely require some governments going around to dataset owners and saying "hey, this has turned into a matter of global importance: eminent domain."
Sure, it's unlikely there was fraud. Some of the comments are troubling... it sounds like they were working with data that was already heavily processed, in ways that they weren't actually sure about, then it was processed some more internally, in ways that weren't necessarily uniform, correct or properly recorded.
Either of those on it's own would make it really hard to do the stats correctly. Science can be very chaotic and your data is usually suboptimal, but it's probably a good idea, now that we're deciding the fate of civilization, to go back and do things carefully from the raw data.
Slashdot Mirror
Hey! That's a good idea! Send one to the Liberal leader while you're at it, but word it so that he's likely to give it to the media as a shot at Harper.
They have tried this at least three times of the last few years, and each time the bill has been quietly forgotten about after public opposition and the sponsoring scapegoat, er, minister has been shuffled off to a different job.
The specs are available online. He may have just done it as an experiment. In North America a carrier will often replace your SIM card free if you tell them you lost it.
Because Apple wants to use it in the new iPhone.
Sorry, it wasn't you. It's hard to keep track of all the different replies. You're the reasonable one so you don't stick out.
If the climatology field is trying to squelch Steve McIntyre they're doing a really crappy job of it. The guy has a pretty good publication record for a non-academic, including a paper in Nature! He's also been thanked by climate data publishing organisations for spotting problems in their data, following which they promptly incorporated his corrections, and, well, he's basically won - the hockey stick graph is discredited. As far as I can tell his position is not that the planet isn't warming (he says he doesn't know) but that he's critical of some of the methodology that's used and some of the results that have been obtained. In many cases his criticisms seem to have been successfully published in the scientific literature and incorporated by other organizations and researchers.
If you're talking about private e-mails from the CRU thing, or RealClimate.org vs. ClimateAudit.org, again, not science. The one is private communication, in which people say all sorts of stupid things, and the other is public debate which, as I said, is a disaster.
Yes, I see you're not the original poster. Still, the evidence that the scientific community is as you say is, as far as I've seen, poor. Your own justification ranged from invoking Al Gore (nothing to do with science) to claiming funding motivation. According to AGW proponents there's lots of funding available from industry for anyone willing to say global warming isn't happening, so that argument, besides being unsupported by evidence, rings hollow too.
I agree, the public debate is a disaster, but I don't see any actual evidence for scientific misconduct in the scientific community.
Do a few searches on Google Scholar. It turns up some PDFs you can read for free. Most of the papers are fairly balanced. There's none of the "OMG we're all gonna die!!" conclusions the media would have you believe. The few I read (I searched for "solar forcing, climate change" or something similar) generally arrived at the conclusion that the sun is likely responsible for some warming in the 20th century (usually quantified), but less than anthropomorphic sources.
Come on, the comparison is between Flash + some video codec (Adobe) or HTML5 + some video codec (Apple).
If you want to compare irrelevant parts of the given companies as some sort of background information why not include, say, Creative Studio (Adobe's "developer tools")? It costs more than XCode plus a computer to run it.
There are free, open, cross platform developer tools for Flash you say? There are free, cross platform developer tools for the iPhone too (GCC). In fact, XCode is just an IDE wrapped around GCC. You do have to sign your executables if you want to submit them to the app store. I'm not sure if someone has figured out a way to run codesign without actually having a Mac but I doubt such a thing is impossible.
Worse. He's comparing Flash to Apple. All of Apple. That's where I quit reading.
Even the usual comparison of Flash (proprietary) to H.264 (proprietary) is wrong. The proper comparison for video, which is the subject the article is harping on, is Flash (proprietary) to HTML5 (open standard). Most Flash video uses the encumbered H.264 codec, just as Apple has suggested H.264 is the ideal codec to use with HTML5. So the codecs are the same. What's the difference? The container, and one container is indeed proprietary while the other is an open standard.
I had a GPGPU example running on a web page in Safari (for giggles) about four years ago. If you can get full GPGPU access through a web page writing a little video decoder shouldn't be a big deal.
How is it a threat to the App Store? If you develop a Flash app you still have to sell it through the App Store.
Jobs doesn't want quick and dirty runs-(poorly)-on-everything ports in the App Store. Rather than reject them for being "crappy" he's decided to reject them because they're not written in C, C++ or ObjC.
The example in the article is illuminating.
The only real difference is that they use a script font for the title instead of Verdana. Okay, a script font might have it's uses.
They use a (italic) serif font for the headings instead of sans serif. All right, I think that's a bad choice, but there are serif fonts available in the standard web selection. Other than that, at the resolution provided I couldn't seen any real differences.
You're aware that Al Gore isn't a scientist, right?
Yes, the public discussion of global warming/climate change is very unscientific. You're aware that journalists, most Slashdot posters, lobby groups, congressmen, corporate spokespeople, most environmentalists, most bloggers and almost all members of the public are also not scientists, right?
The issues I referred to are not necessarily referred to in that particular readme. Your interpretation supports my point just as well though.
I've worked with a lot of crap code researchers (including myself) have written in all sorts of languages for all sorts of things, usually in a hurry. That's just fine for research purposes. Would I use any of it online in a hospital (I do medical research)? No way. It might kill someone.
Hard coded paths, code migration, lots of data, incorrect files, reprocessing of data, converters... all sorts of opportunities for mistakes. Many mistakes were found and recorded in that read me.
I don't think the CRU was committing fraud, nor do I suspect their conclusions are likely to be wrong, at least in the broad sweep, but now that their results (and the datasets they produce) are going to be used to help decide the future direction of civilization, don't you think it would be a good idea to go back and do things over, a bit more carefully? Just to check?
First, there is no proof there's a conspiracy to deny publication of dissenting papers. Several investigations have decided that there is no conspiracy. There is an outside chance that one little corner of science may have slipped into pseudoscience, but that's hardly justification for your statements about science in general.
The nondisclosure of data is a serious issue, but it's also not universal and even in this case it sounds like it's more due to the CRU not having the legal right to disclose the data in question, NOT to their unwillingness to do so. That's a problem with the law or with the commercial right-holders, not science. Again, even in the worst case scenario, it's not a justification for your statements about science in general.
Following are a few examples of large, publicly available scientific datasets that were assembled at considerable cost, entirely voluntarily (a small selection, several that I have personal experience with and others that I've included to try to give some breadth to the list):
http://physionet.org/
http://mouldy.bic.mni.mcgill.ca/brainweb/
http://www.med.harvard.edu/AANLIB/home.html
http://archive.eso.org/skycat/servers/usnoa
http://www.astrometry.net/data.html
http://www.ncbi.nlm.nih.gov/genbank/GenbankOverview.html
http://www.ncbi.nlm.nih.gov/guide/data-software/
And some publicly available code:
http://noodles.bic.mni.mcgill.ca/ServicesSoftware/HomePage (the MINC tools are apparently available from Debian as well)
http://www.bic.mni.mcgill.ca/~ilana/diffusion/diffusion_tools.html
http://www.vlfeat.org/~vedaldi/code/sift.html
http://www.itk.org/
http://www.cs.cmu.edu/~cil/v-source.html
http://iraf.noao.edu/
There's hardly an overwhelming culture of closed and proprietary secret keeping in science as you suggest. Quite the opposite. Sure, some of the non-scientific appendages to science do have issues in that area (journals, for example) but scientists are usually all too willing to do end runs around such things. If you want to read a paper, e-mail the author and he's likely to send you a PDF despite that often being technically a violation of copyright. Failing that, go to a library and they'll let you read it, free.
"But still: at some point, it's a matter of what you, the fellow scientist, find sufficiently convincing."
I think that's the key. You give me the stats, all the information I need to make my own judgement call, and I decide whether to agree with you or not. More importantly, you can put statistically derived limits on the maximum magnitude of secondary effects so everyone can agree on just how strong your statement actually is.
Statements about climate have a pretty wide range. Journalists and some researchers who do interviews often state things like "climate change is caused by human activity" which is clearly not scientific. The IPCC report says something like "there is a 90% chance that more than 50% of the observed warming trend is due to human activity" which is much better.
One of the issues with the CRU seems to be that they're not exactly sure about what those statistical limits are on the third-party data they're using. Some of the data sound like it's delivered as a "here are the numbers" dataset without errors or adequate background on how it was processed. Some of the e-mails, and the findings of this investigation, suggest that disorganization within the CRU also led to some researchers being unsure what had already been done to their own data as well.
The CRU had to use what they had, but it's an issue that can be fixed with a bit of money and a little more work and, given how important this is, probably should be.
"Is that even English?"
Yes, it is. Somewhat informal English, yes. To be completely correct I should have prepended "Have you" to the sentence, but it is common in informal English communication to omit such parts of speech when they are clearly implied.
Maybe you should avoid sarcastically calling other people "real intellectual giant[s]."
You made a sarcastic "analogy." I spit it back at you. Were you expecting a non-sarcastic response to your idiotic post?
According to other sources, BP estimates originally said 1000 barrels/day. The NOAA put it at 5 times that, 5000 barrels/day. Other independent sources, using satellite imagery, put it at 5-10k. There is one oceanographer who says it might be leaking at as much as 25,000 barrels/day. It sounds like the media like to "estimate" on the high side, with the Wall Street Journal coming in at 24k.
So the 5000 estimate seems reasonable, made by independent experts actually on the scene, and already a lot more pessimistic than BP's likely biased estimates.
50,000 seems to have come out of thin air. If you were going to engage in what-if scenarios it would be better to use BP's worst-case estimate of 163,000 barrels/day, so the 50k seems like a misunderstanding on the part of the journalist.
http://en.wikipedia.org/wiki/Deepwater_Horizon_drilling_rig_explosion
If they validated the use of simpler statistical methods then that validation should absolutely be part of their published research and would have certainly been available to the investigation. I've done exactly what you describe and, as I'm sure your friend does, the resulting paper always gets a paragraph saying something like:
"Analysis X and Y were performed on a randomly selected subset of the data with N=Z. X produced such and such result with estimated alpha=A, beta=B, etc. and Y produced the same result with estimated alpha=A', beta=B' etc. Although Y is known to be suboptimal in terms of statistical power, it is more conservative than X (possibly with an appropriate correction) and is much more computationally tractable. Therefore, analysis Y was used for the larger dataset."
Not sure if it's going to impact crude prices, but the article managed to both overestimate the leak rate by an order of magnitude and horribly exaggerate the size of the reservoir. It could certainly turn into a really bad disaster, but it's not the unprecedented catastrophe the article implies.
It has encouraged some needed scrutiny into drilling practices though. I like one explanation by an oil official (I think he was from BP) when asked why rig engineering drawings were frequently not signed by an engineer: oh, the regulations require that we have engineering drawings for everything, not that they're accurate.
For people who worry about running out of oil, there's enough recoverable in the tar sands in Canada and Venezuela at current prices that we're not going to run out anytime soon. As you point out, it does come at an even higher environmental cost though. On the bright side, we are making real progress on alternatives. I have a friend setting up an electronics collection and recycling shop who is seriously considering leasing a quarter section with no power service. Apparently they already has the windmills and solar cells to supply adequate power, and this is an organization operating on a shoestring budget at around 55 degrees north - not ideal territory for solar.
You've obviously never actually done any science.
Present something to a scientific audience and they spend the entire presentation looking for weaknesses. At the end someone, or several someones grill you, and they usually turn out to be eminent experts in the field. The questions they ask are even tougher if you agree with them. Last time I did this was yesterday.
I've heard there are some branches of science that aren't quite like this, but I've never actually seen one first hand, and most such reports are from dubious sources.
As a neuroimager, I can pretty much guarantee you that the methods you were using probably weren't always beyond reproach.
However, I also suspect your research wasn't going to be used to set policy that has a large impact on human life. I can tell you that any work I've been involved in with potentially even moderate clinical impact has absolutely involved at least one statistician.
I'm a scientist working in medical research, occasionally involved with clinical trials. The investigatory "hey, here's a cool new technique that might be useful" stuff doesn't require me to carefully store my data, although it does require me to publish a good enough description of my methods that they can be duplicated by other people, on other data. I do keep the data around as well, actually, but I can't guarantee it will be available in twenty years.
Anything involved with a clinical trial absolutely requires that the data, code, descriptions of the code, descriptions of the methods, justification of the methods, validation of the code and methods and the infrastructure to run the code be archived in such a way that it will be available in the future. Want to make something faster with a fancy GPU algorithm? Sorry, the virtual machine that runs all the code (so that the "computer" can be archived as well) may not be able to run the GPU stuff the same way in ten years. And by the way, even the smallest trial collects orders of magnitude more data than those nine million data points.
Climate science started out more in the former situation, but it's certainly progressed into the latter. When your research has global importance to human life and well-being it's time to ramp up your standards. Should the CRU have been doing this all along? I don't know. Should climate researchers do so going forward? Absolutely.
Actually, a lot of scientific datasets are eagerly made widely available. When you collect a major dataset you write at least one paper about it. When you share that dataset with someone else and they publish papers based on it, they cite your paper describing the data's collection. That gets your paper lots of citations, which is very desirable.
My impression of the CRU climate data is that some of the data they were using was collected not by public science but for commercial purposes. Commercial entities care about how much money they make selling things, not how many citations their papers (if they have any) get. In this case I think the issue has become important enough that it would be well worth the relevant government(s) buying the required datasets and releasing them.
It's not quite as bad as you put it. There are ways to increase your certainty that errors are actually errors and not phenomenon, and when you state your results you include those numbers anyway.
Take Galileo. I duplicated that experiment way back in high school. You build a couple of ramps that you make as smooth as possible, then roll metal balls of different weight down the ramps (the denser the better, because that minimizes friction effects). You do many trials, and you do half of them with the ball-ramp combinations switched, to even out differences in the ramps. At the end you get many measurements on which you use established statistical techniques. When you're finished you don't have "objects accelerate due to gravity at the same rate regardless of their mass" you have "in our experiment we could not detect a difference in the acceleration due to gravity of objects of different mass to such and such limits with such and such confidence."
If you do your experiment carefully enough those limits will be small and you DO discover that objects of different mass in this experiment roll down the ramps at different rates, even accounting for friction effects. You've just discovered rotational momentum.
Looking at your other example, tree ring data, if you suspect that the tree ring thickness - temperature relationship changes over time, but that's all the data you have, you absolutely do not make up an "I think it's right" correction and apply it. You look for more data, such as an overlapping period where you have both reliable temperature data and tree ring measurements, that demonstrates the change. If you suspect the change is due to pollution, you look for evidence that this is true. Once you've got it all together you present your case, including all the uncertainties, so that everyone can see what you've done, what assumptions you've had to make, and what errors are likely still present.
Personally I think the scrutiny in the CRU case is misplaced. From all the information that's available, it seems that the CRU has done their best with complicated and diverse data, most of which they didn't collect. What they've done is fine for a "hey, this looks like something real" discovery. However, since they didn't collect most of the data, and since it sounds like a lot of that data was preprocessed in unknown ways before they got it, I think it would be wise to go back to the actual raw data and analyze it uniformly and openly. Yes, that will likely require some governments going around to dataset owners and saying "hey, this has turned into a matter of global importance: eminent domain."
Sure, it's unlikely there was fraud. Some of the comments are troubling... it sounds like they were working with data that was already heavily processed, in ways that they weren't actually sure about, then it was processed some more internally, in ways that weren't necessarily uniform, correct or properly recorded.
Either of those on it's own would make it really hard to do the stats correctly. Science can be very chaotic and your data is usually suboptimal, but it's probably a good idea, now that we're deciding the fate of civilization, to go back and do things carefully from the raw data.