Slashdot Mirror

You only think that the data sets were merged/normalised in arbitrary ways because you don't seem to understand statistics.

No, it was because someone looked at the actual code (the linked example shows data being multiplied by a random looking array of numbers which happens to generate a "hockey stick" shape in the result). The comments are quite entertaining.

Also: if you don't think we should do anything you are insane.

[...]

but if you don't mitigate it at all it'll go way beyond the point where it's possible to adapt.

No evidence has ever been provided for this assertion. Actual predictions indicate relatively small temperature changes less than what the Earth experienced 50 or so million years ago (I see a graph that estimates peak temperature increase was 12 C over present day (the year 1999) for the region (present day arctic ocean). Solar activity hasn't changed that much over that period of time.

This is a standard chicken little threat. What's going to happen now that will be worse than what happened then?

Alternately, if you actually read those emails then you should be armed with precise quotes of the "deceptions and manipulations" which you found so powerfully convincing, or at least have some mental construct of their findings to provide us, rather than just a brief handwave in their direction preparatory to an ad hominem slur devoid of substantive content.

As, for instance, these quotes from 7 unrelated investigations which I find convincing:

"even if the data that CRU used were not publicly available—which they mostly are—or the methods not published—which they have been—its published results would still be credible: the results from CRU agree with those drawn from other international data sets; in other words, the analyses have been repeated and the conclusions have been verified."
-"The disclosure of climate data from the Climatic Research Unit at the University of East Anglia" http://www.publications.parliament.uk/pa/cm200910/cmselect/cmsctech/387/387i.pdf

"We saw no evidence of any deliberate scientific malpractice in any of the work of the Climatic Research Unit and had it been there we believe that it is likely that we would have detected it. Rather we found a small group of dedicated if slightly disorganised researchers who were ill-prepared for being the focus of public attention. As with many small research groups their internal procedures were rather informal. "
- "Report of the International Panel set up by the University of East Anglia to examine the research of the Climatic Research Unit." http://www.uea.ac.uk/mac/comm/media/press/CRUstatements/SAP

"Dr. Michael E. Mann did not engage in, nor did he participate in, directly or indirectly, any actions that seriously deviated from accepted practices within the academic community."
- "Final Investigation Report Involving Dr. Michael E. Mann" http://www.anenglishmanscastle.com/Final_Investigation_Report.pdf

"On the specific allegations made against the behaviour of CRU scientists, we find that their rigour and honesty as scientists are not in doubt. ... In addition, we do not find that their behaviour has prejudiced the balance of advice given to policy makers. In particular, we did not find any evidence of behaviour that might undermine the conclusions of the IPCC assessments."
- "The Independent Climate Change E-mails Review" http://www.cce-review.org/pdf/FINAL%20REPORT.pdf

"Petitioners say that emails disclosed from CRU provide evidence of a conspiracy to manipulate data. The media coverage after the emails were released was based on email statements quoted out of context and on unsubstantiated theories of conspiracy. The CRU emails do not show either that the science is flawed or that the scientific process has been compromised. EPA carefully reviewed the CRU emails and found no indication of improper data manipulation or misrepresentation of results."
- "Myths vs. Facts: Denial of Petitions for Reconsideration of the Endangerment and Cause or Contribute Findings for Greenhouse Gases under Section 202(a) of the Clean Air Act" http://epa.gov/climatechange/endangerment/myths-facts.html

"In our review of the CRU emails, we did not find any evidence that NOAA inappropriately manipulated data comprising the GHCN-M dataset or failed to adhere to appropriate peer review procedures. In addition, we found no evidence to suggest that NOAA was non-compliant with the IQA or the Shelby Amendment. "
- "Examination of issues related to internet posting of emails from Climatic Research Unit" http://www.oig.doc.g

http://www.anenglishmanscastle.com/HARRY_READ_ME.txt

Now, perhaps this fudging of the data wasn't malicious (in fact, I'll argue that it was done with the best of intentions), and perhaps some of the fudges actually have a reasonable rationale that we can agree upon - but let's not pretend that there is a magical thermometer we can stick in the air, and get the current Global Average Temperature (much less a magical thermometer we can read from 1000 years ago to do the same thing). At best, this is a field over-reliant on proxy data, and *everyone* should be skeptical of that sort of weak science.

It's not a conspiracy theory. It's an orthogonality problem. If you have a Medieval Warming Period (MWP) -- then temperatures *aren't* unprecedented and become mathematically decoupled from CO2. Mann's "Hockeystick" graph erased the MWP -- problem is, the approach is worthless, and while Mann may believe it (again not conspiracy theory), it isn't true. Thus we still have the MWP (and the RWP, the Minoan, and the Holocene optimum) -- all of which were warmer than today and none of which had AGW contributions.

Well, yeah! The Medieval Warm Period, which was probably local, and restricted to Europe, but with a lower global average temperature... You could at least try to read a bit before spouting talking points...

A challenge to the geeks at slashdot -- read "HARRY_README.txt". If you believe a single thing that comes out of CRU after that, I've got a bridge to sell.

Though you haven't actually linked it, I'll try to answer.

First, let RealClimate speak (scroll down a little)...

HARRY_read_me.txt. This is a 4 year-long work log of Ian (Harry) Harris who was working to upgrade the documentation, metadata and databases associated with the legacy CRU TS 2.1 product, which is not the same as the HadCRUT data (see Mitchell and Jones, 2003 for details). The CSU TS 3.0 is available now (via ClimateExplorer for instance), and so presumably the database problems got fixed. Anyone who has ever worked on constructing a database from dozens of individual, sometimes contradictory and inconsistently formatted datasets will share his evident frustration with how tedious that can be.

Second, how is this any different from major, even mission critical code in so many other domains? Even in places which could cost thousands of lives (nuclear reactor safety systems, for example... Ever done a code audit on the software for those safety systems)?

Keep in mind that these people aren't professional coders; they're scientists using IDL and Fortran (90, I presume), and probably other languages like Matlab. The code is an implementation of their hypothesis. It's usually ugly, and the first one that works the way they want it. Maintainability? Hah! (note: here, I speak as one who has had to translate "scientist" code into "real" code).

Spouting talking points is hardly critical thinking, which is why you people are called "deniers" and not "skeptics".

OK, so lets look at the quality of the CRU climate data as logged here http://www.anenglishmanscastle.com/HARRY_READ_ME.txt
I'll even quote some bits for you.

OH FUCK THIS. It's Sunday evening, I've worked all weekend, and just when I thought it was done I'm
hitting yet another problem that's based on the hopeless state of our databases. There is no uniform
data integrity, it's just a catalogue of issues that continues to grow as they're found.

I am seriously close to giving up, again. The history of this is so complex that I can't get far enough
into it before by head hurts and I have to stop. Each parameter has a tortuous history of manual and
semi-automated interventions that I simply cannot just go back to early versions and run the update prog.
I could be throwing away all kinds of corrections - to lat/lons, to WMOs (yes!), and more.

Now, this is a clear indication that the standard deviation limits are not being applied.
Which is extremely bad news. So I had a drains-up on anomauto.for.. and.. yup, my awful
programming strikes again. Because I copied the anomdtb.f90 process, I failed to notice
an extra section where the limit was applied to the whole station - I was only applying
it to the normals period (1961-90)!

Probably the worst story is temperature, particularly for MCDW. Over 1000 new stations! Highly
unlikely. I am tempted to blame the different lat/lon scale, but for now it will have to rest.

If I fix that, I get:...14 stations LESS than the previous exercise. That'll do, surely? It's not going to be easy to find 14 missing stations, is it? Since the anomalies aren't exactly the same. Should I be worried about 14 lost series? Less than 2%. Actually, I noticed something interesting.. look
at the anomalies. The anomdtb ones aren't *rounded* to 1dp, they're *truncated*! So, er - wrong!

The problem is that the synthetics are incorporated at 2.5-degrees, NO IDEA why, so saying they affect
particular 0.5-degree cells is harder than it should be. So we'll just gloss over that entirely ;0)

So, under /cru/cruts/version_3_0/fixing_tmp_and_pre/custom_anom_comparisons, we have a
'manual' directory and an 'automatic' directory, each with twelve 1990 anomaly files. And
how do they compare? NOT AT ALL!!!!!!!!!

This shows me the quality of data climate science works with is poor.
There needs to be rigorous peer reviewed climate data collection standards (which won't improve the historical data.. like for example, which stations had concrete jungles grow around them, how were they calibrated (including traceability) etc)
The raw data must be publically released (then it can't be fiddled with later)

Until then, I will assume GIGO.

That link, HARRY_READ_ME.txt, certainly is a trainwreck of bad organization, bad policy, bad procedure, bad communication, etc., but it's not an example of bad code. We'd have to see the code to say that. What I see from this document is not bad data or incorrect results - just horrific effort getting tools to work.

I see this with developers all the time who don't care enough about their code, its organization and its data - or are ignorant of why that's even important. I can totally see scientists being ignorant of IT and development best practice. Not excusing, just saying it's not as uncommon as many would think.

I agree with the CRU exoneration, however, that likely the results were correct but procedures were horrific. It looks like those guys were hacking around legitimate problems in their tools to produce their data - not manipulating their data.

There was "Das iTeam - Die Jungs an der Maus", which was a crappy shot-by-shot remake. It was really bad.

I don't know why, the Germans normally have an excellent sense of humour ;-)

He was referring to the climate model software written by CRU, I cou;dn't find an actual name for it but I did find the read me and along with it a great write up on why "open source science" would've helped avoid this scandal:

http://www.freesoftwaremagazine.com/columns/open_science_climategate_ipcc_cru_needs_take_leaf_out_cerns_book
http://www.anenglishmanscastle.com/HARRY_READ_ME.txt
http://news.bbc.co.uk/2/hi/programmes/newsnight/8395514.stm

Forget the emails. All they show is a few very prestigious climate scientists "hiding behind" intellectual property rights, refusing to adhere to FOIA rules (both of these normally anathema to /.ers), deleteing data and emails that might be incrimintory, revealing that they have manipulated peer review by keeping skeptical papers out, even to the point of changing the definition of peer review, refusing to release their data, caliming a peer reviewed article = 'settled science', exulting in the death of skeptics, attempting (successfully) to get editors they don't like fired. Just normal boys will beboys stuff. Nothing to see here> Move along.

But this is /. How about looking at the code? Like here: http://wattsupwiththat.com/2009/12/04/climategate-the-smoking-code/ or how about a little sympathy for a programmer, Harry. See what he has to say: http://www.anenglishmanscastle.com/HARRY_READ_ME.txt or look here: http://wattsupwiththat.com/2009/11/22/cru-emails-may-be-open-to-interpretation-but-commented-code-by-the-programmer-tells-the-real-story/

Or how about daling with teh mathematics of it all: http://www.americanthinker.com/2009/11/the_mathematics_of_global_warm.html

So forget the emails; look at the code. Then come back here and say, with a straight face, that the data has not been manipulated.

Some reading material... http://www.anenglishmanscastle.com/archives/003995.html
http://www.globalwarming.org/node/388
http://climatescience.blogspot.com/2008/06/historic-co2-level-data-deception.html
http://antigreen.blogspot.com/
http://www.freerepublic.com/focus/f-bloggers/1806245/posts
Once you see the REAL data you figure out that CO2 is not causing a crisis/