Yes, but the errors introduced by these choices should be included in your final, quoted experimental error. If you're trying to measure parameter A, your analysis result B +- C should be consistent with the true value of A. If an arbitrary choice of analysis parameters can make that not so, you're doing it wrong - either the calculation or the error analysis.
Yes, I realised that was not what I meant to say (too hasty), hence my correction above. You do indeed need to describe what you have done in sufficient detail.
Just as an example - I used to do particle physics analysis. As part of this I had a piece of code which introduced a fake signal of the thing I was trying to measure (and subsequently publish on) into a data set. It was even commented something like "introduce false signal with parameters (x,y,z,whatever)". If someone had stolen that piece of code and put it on the web, what would the average viewer have concluded?
In fact, it was part of a test routine to see how a known signal would emerge at the other end of the analysis chain, and certainly wasn't published as true data. But that would have not been evident at all from the code alone.
No it isn't, we have no background as to why it was done or what was done with the output - whether it was published as is, used for cross-checking, whether the data was faulty and needed correction (very common when the data was not collected with that purpose in mind). I've had loads of code fragments doing weird shit just to see what the results would be - how it would change the output if the input was different, etc.
Unsurprisingly, code comments were never meant to be a complete documentation of the analysis methods. Still, who'd ever have suspected ESR of jumping to a conclusion.
If their code was buggy and produced the wrong answer, nobody *would* be able to reproduce the results, except in the rather unlikely case of having introduced the exact same bugs into their own code. This would provide a nice indicator of a problem, and is why it's important to reproduce analyses fully independently, i.e. with different data sets, different code, different scientists.
Everyone using the same code is just asking for common errors in all the analyses. If you re-use parts of another analysis then you haven't really reproduced it.
In astronomy, say, if you're the only one with recorded data from a particular supernova, then nobody else can reproduce your analysis on it without data.
Is climate science like this? What's to stop someone else from going out and collecting more ice cores, for example, or using different temperature station data or different trees for tree ring analysis? Surely there can't just be one data set for each of the areas of interest.
Sorry, I realised that was worded badly. What I meant was that there were some aspects of analysis that would be known to any competent researcher. It should be spelled out in enough detail to be reproducible, bearing this in mind.
Any significant, non-obvious adjustment *should* be included in the published work. E.g. the discarding of tree ring data post-1960 was explained, rather than just silently replaced.
No, just what the algorithm was intended to do. That's enough to duplicate the analysis, if a competent researcher is doing it anyway, as he'll work out the necessary corrections for himself.
That's not the impression I got from reading the emails. No obvious "cooking" was there, at least to my eyes. There are always difficulties in comparing data sets from different sources, and lots of calibration and modification may be required. I don't know what data you worked with, but I've had to understand odd discrepancies between data sets when doing analyses before. There wasn't nearly enough background provided there (unsurprising as they were private communications and notes) to say whether what they were doing was reasonable or not.
As was pointed out in the article, other independent sets with independent analyses give the same results, so any "cooking" can't have had much of an effect.
That's not quite right. It's important that your results are reproducible. That requires a full description of how the data was gathered and how it was analysed. That way, someone can go and do their own experiments, collect their own data and conduct their own analysis. Giving out the raw data isn't a bad thing, but it's not necessary and actually doesn't happen that often.
You could make a case that it's in fact bad for people to all work off the same data set or code, as any mistakes (or even deliberate fraud) will then be common to all analyses.
Deliberate attempts to delete email and data even those pending FOI requests
Yes, possibly. Not scientifically relevant.
- Deliberate attempts to scuttle peer review
No, it was a response to someone else's hijacking of a peer-reviewed journal, which then published low quality papers. Their concerns were not unique, indeed there were several resignations from the journal as a result.
As for your other link, which says:
Without replication, science cannot move forwards.
And then goes on to suggest that providing data is necessary for this. It is not. It is necessary to provide a full description of what was done so that somebody can go off and reproduce the work.
The big fuss about providing data doesn't make much sense, as it's neither necessary to determine scientific validity (a full, published description of the analysis will do that) nor sufficient to detect fraud (the data could be falsified prior to its release).
In the UK, admittedly (it's where I'm from). There was a case where Simon Singh wrote an article describing chiropractic as "bogus" and got sued for it. Their argument was that bogus implied deliberate fraud - I don't see it myself but it's what they're claiming. Not sure what the outcome is yet but it's causing a lot of trouble for him.
It doesn't look nice, no. But it also looks like they're making an honest effort to sort it out, which would go against the "fraud" allegations.
Some of the data I've had to deal with has required a lot of head-scratching to understand what's going on, with seemingly conflicting results from different parts of it. At different stages I've thought similar things to this if not written them down. Since this "documentation" is someone's private notes, it's not going to paint a *complete* picture and so we can't say whether these issues were resolved. I had to deal with confusing and conflicting data and results before - the middle stages looked very nasty but I made sense of all the discrepancies and sources of error in the end. This looks like an unusually bad case, but who's to say they didn't manage it? I don't know. No matter what, there's no evidence of deliberate fudging in what you've quoted.
Anyway, according to RealClimate there's a new version of the database so presumably they made sense of it all. It's not as if it's the only one, either - other people have come to similar conclusions with different analyses of different data sets.
What you say is largely true. Scientists do have preconcieved ideas just like anyone else (well, rather less than the average person if they're any good, but there nonetheless). Group-think is also a possibility, and would be a problem if it was happening. But is it?
I don't know, but those leaked emails don't provide much if any evidence of it. On the other hand, I see it in vast quantities in most of the climate "sceptics", along with logical fallacies, superficial analysis (just enough to support their view but no more), personal attacks, lack of understanding of the issues, and many more failures. If they conducted honest, robust and high quality analysis then I have no problem with them contributing. But the vast majority that I've seen do not do this.
O'rly? Promoting policies, which are based on your research isn't going to influence you? But not promoting policies would? You're trolling, my friend, you can't be that blind to the obvious or dumb. If you are, you can't claim you understand the "science".
Perhaps I am dumb. But please explain in simple language for someone as stupid as me - It's clear how your science will influence what policies you advocate. What is *not* clear is how that advocacy will then feed back into affecting how you conduct your research. Why would presenting a view at a political debate suddenly change how I see the science?
While it may be *possible* that they're doing that, it isn't the only explanation at all. It's almost always necessary to apply manipulation to data to correct for various errors, especially if the data was collected for an entirely different reason and so won't have been done with your analysis in mind.
For details of this we really need to look at the published literature rather than an inherently incomplete document that wasn't even written by the person who first did the analysis anyway. But the fact that the explanation/justification isn't in the *code* is hardly surprising - not very well documented code seems to be standard practice in science.
I'm not agreeing that their claim that the papers were poor quality was valid - I don't know, I haven't seen them and it's not my field anyway. But if that was their opinion, so what? As long as they don't use any unethical methods to suppress publications (and I see no evidence that they have done so), what's the problem?
Publishing raw data has never been required for science. What is required is that what is published is enough to enable someone else to reproduce the analysis. So they'd need to explain where the data comes from, what the analysis code does with it, etc. Now, if their papers fail to do this I'll agree with you - but in that case the papers should have failed peer review.
It shouldn't be necessary for someone to have their data and code to reproduce the results, any more than it should be necessary for someone to have the exact same lab and equipment to reproduce some other scientific experiment. If it depends on a specific data set, then by definition it isn't reproducible and so isn't scientific.
Okay, how do they gain lots of money from this? And would this be more than the lots of money they could get from the fossil fuel industry by publishing valid anti-AGW papers?
I said that *I* couldn't trace it from the source based on the leaked files. Can they? I don't know, we only saw a very incomplete set of documents regarding this. No word on whether this code was exactly what was used for the final publications.
Yes, *if* the path from data to results isn't fully accounted for, that would be a problem. Is this the case? Can't tell from what we've seen.
In any case, if I'm not very much mistaken the results have been reproduced by other scientists at different institutes.
There's no such thing as a felony in Great Britain. And there's still no firm evidence one way or another regarding the FoI thing.
And where exactly are these underhand methods or backroom deals? All I saw was a frustrated statement that "I'll do by best to stop these papers getting into the IPCC report" (paraphrased). No mention of an underhand methods to be used in doing so.
Hiding from FOIA requests, conspiring to lock out a publication that wasn't swallowing their bate (how dare a peer review journal ask difficult questions of AGW!).
Lock out? They thought the publication was publishing poor quality papers. If that was their belief, why would they not refrain from publishing there or citing articles?
Then we have Phil working to keep these two papers [populartechnology.net] from being seen at the next IPCC meeting--
I can't see either of these papers being in the next IPCC report. Kevin and I will keep them out somehow - even if we have to redefine what the peer-review literature is !
So he doesn't think the papers are worthy of inclusion. No evidence of any underhand arrangements to stop it though, just that he'll make the case that they shouldn't be included.
Let's not forget that these are private emails and people are much looser with language. I once make a jocular reference to Schoning my data - how incriminating would *that* look if it was written down?
In my parents time it was global cooling, when I was younger it was a giant hole in the ozone above Australia caused by big evil America, a year ago it was Global Warming, and now it's become "Climate Change".
There was never a consensus in favour of global cooling (despite the media reports at the time), the ozone layer was real and an international treaty was made to deal with the problem, so I don't know what your point is there. Global warming is the same thing as Climate Change, it was just felt that the first term would perhaps confuse those who think that global warming implies local warming everywhere on Earth. And your "scheme to misuse taxpayer money" remark? How do the scientists benefit from that, exactly? That's verging on tinfoil-hattism. A global conspiracy amongst scientists.... to what end?
Perhaps you can supply some links. I'm not saying such things don't exist, just that I haven't seen them. However, one thing I did see was a list of signatures from people opposed to the climate change theory - almost all of whom had no science qualifications.
Yes, they get involved with IPCC or the media from time to time. But IPCC's role is not to set policy, but to present evidence and options.
Finally, I don't see any reason as to why any involvement in this way this would influence their research. The other way round, yes.
The first part of what you say is true, but again no surprise. However, the part about deliberately fudging it all to get the "right" answer? I don't see that. But I wouldn't expect to see a full picture in a few old notes and bits of code. Private notes are always incomplete.
Slashdot Mirror
You're rich, so you should be able to pollute more? Interesting philosophy.
Yes, but the errors introduced by these choices should be included in your final, quoted experimental error. If you're trying to measure parameter A, your analysis result B +- C should be consistent with the true value of A. If an arbitrary choice of analysis parameters can make that not so, you're doing it wrong - either the calculation or the error analysis.
Yes, I realised that was not what I meant to say (too hasty), hence my correction above. You do indeed need to describe what you have done in sufficient detail.
Tsk, replying to myself again...
Just as an example - I used to do particle physics analysis. As part of this I had a piece of code which introduced a fake signal of the thing I was trying to measure (and subsequently publish on) into a data set. It was even commented something like "introduce false signal with parameters (x,y,z,whatever)". If someone had stolen that piece of code and put it on the web, what would the average viewer have concluded?
In fact, it was part of a test routine to see how a known signal would emerge at the other end of the analysis chain, and certainly wasn't published as true data. But that would have not been evident at all from the code alone.
Naturally. But there would be other, valid and honest, reasons why two loosely similar algorithms may produce different results.
Not if they're trying to measure the same thing - then the outputs should agree within experimental errors.
No it isn't, we have no background as to why it was done or what was done with the output - whether it was published as is, used for cross-checking, whether the data was faulty and needed correction (very common when the data was not collected with that purpose in mind). I've had loads of code fragments doing weird shit just to see what the results would be - how it would change the output if the input was different, etc.
Unsurprisingly, code comments were never meant to be a complete documentation of the analysis methods. Still, who'd ever have suspected ESR of jumping to a conclusion.
If their code was buggy and produced the wrong answer, nobody *would* be able to reproduce the results, except in the rather unlikely case of having introduced the exact same bugs into their own code. This would provide a nice indicator of a problem, and is why it's important to reproduce analyses fully independently, i.e. with different data sets, different code, different scientists.
Everyone using the same code is just asking for common errors in all the analyses. If you re-use parts of another analysis then you haven't really reproduced it.
In astronomy, say, if you're the only one with recorded data from a particular supernova, then nobody else can reproduce your analysis on it without data.
Is climate science like this? What's to stop someone else from going out and collecting more ice cores, for example, or using different temperature station data or different trees for tree ring analysis? Surely there can't just be one data set for each of the areas of interest.
Sorry, I realised that was worded badly. What I meant was that there were some aspects of analysis that would be known to any competent researcher. It should be spelled out in enough detail to be reproducible, bearing this in mind.
Any significant, non-obvious adjustment *should* be included in the published work. E.g. the discarding of tree ring data post-1960 was explained, rather than just silently replaced.
No, just what the algorithm was intended to do. That's enough to duplicate the analysis, if a competent researcher is doing it anyway, as he'll work out the necessary corrections for himself.
That's not the impression I got from reading the emails. No obvious "cooking" was there, at least to my eyes. There are always difficulties in comparing data sets from different sources, and lots of calibration and modification may be required. I don't know what data you worked with, but I've had to understand odd discrepancies between data sets when doing analyses before. There wasn't nearly enough background provided there (unsurprising as they were private communications and notes) to say whether what they were doing was reasonable or not.
As was pointed out in the article, other independent sets with independent analyses give the same results, so any "cooking" can't have had much of an effect.
That's not quite right. It's important that your results are reproducible. That requires a full description of how the data was gathered and how it was analysed. That way, someone can go and do their own experiments, collect their own data and conduct their own analysis. Giving out the raw data isn't a bad thing, but it's not necessary and actually doesn't happen that often.
You could make a case that it's in fact bad for people to all work off the same data set or code, as any mistakes (or even deliberate fraud) will then be common to all analyses.
Deliberate attempts to delete email and data even those pending FOI requests
Yes, possibly. Not scientifically relevant.
- Deliberate attempts to scuttle peer review
No, it was a response to someone else's hijacking of a peer-reviewed journal, which then published low quality papers. Their concerns were not unique, indeed there were several resignations from the journal as a result.
As for your other link, which says:
Without replication, science cannot move forwards.
And then goes on to suggest that providing data is necessary for this. It is not. It is necessary to provide a full description of what was done so that somebody can go off and reproduce the work.
The big fuss about providing data doesn't make much sense, as it's neither necessary to determine scientific validity (a full, published description of the analysis will do that) nor sufficient to detect fraud (the data could be falsified prior to its release).
In the UK, admittedly (it's where I'm from). There was a case where Simon Singh wrote an article describing chiropractic as "bogus" and got sued for it. Their argument was that bogus implied deliberate fraud - I don't see it myself but it's what they're claiming. Not sure what the outcome is yet but it's causing a lot of trouble for him.
It doesn't look nice, no. But it also looks like they're making an honest effort to sort it out, which would go against the "fraud" allegations.
Some of the data I've had to deal with has required a lot of head-scratching to understand what's going on, with seemingly conflicting results from different parts of it. At different stages I've thought similar things to this if not written them down. Since this "documentation" is someone's private notes, it's not going to paint a *complete* picture and so we can't say whether these issues were resolved. I had to deal with confusing and conflicting data and results before - the middle stages looked very nasty but I made sense of all the discrepancies and sources of error in the end. This looks like an unusually bad case, but who's to say they didn't manage it? I don't know. No matter what, there's no evidence of deliberate fudging in what you've quoted.
Anyway, according to RealClimate there's a new version of the database so presumably they made sense of it all. It's not as if it's the only one, either - other people have come to similar conclusions with different analyses of different data sets.
What you say is largely true. Scientists do have preconcieved ideas just like anyone else (well, rather less than the average person if they're any good, but there nonetheless). Group-think is also a possibility, and would be a problem if it was happening. But is it?
I don't know, but those leaked emails don't provide much if any evidence of it. On the other hand, I see it in vast quantities in most of the climate "sceptics", along with logical fallacies, superficial analysis (just enough to support their view but no more), personal attacks, lack of understanding of the issues, and many more failures. If they conducted honest, robust and high quality analysis then I have no problem with them contributing. But the vast majority that I've seen do not do this.
O'rly? Promoting policies, which are based on your research isn't going to influence you? But not promoting policies would? You're trolling, my friend, you can't be that blind to the obvious or dumb. If you are, you can't claim you understand the "science".
Perhaps I am dumb. But please explain in simple language for someone as stupid as me - It's clear how your science will influence what policies you advocate. What is *not* clear is how that advocacy will then feed back into affecting how you conduct your research. Why would presenting a view at a political debate suddenly change how I see the science?
While it may be *possible* that they're doing that, it isn't the only explanation at all. It's almost always necessary to apply manipulation to data to correct for various errors, especially if the data was collected for an entirely different reason and so won't have been done with your analysis in mind.
For details of this we really need to look at the published literature rather than an inherently incomplete document that wasn't even written by the person who first did the analysis anyway. But the fact that the explanation/justification isn't in the *code* is hardly surprising - not very well documented code seems to be standard practice in science.
I'm not agreeing that their claim that the papers were poor quality was valid - I don't know, I haven't seen them and it's not my field anyway. But if that was their opinion, so what? As long as they don't use any unethical methods to suppress publications (and I see no evidence that they have done so), what's the problem?
Publishing raw data has never been required for science. What is required is that what is published is enough to enable someone else to reproduce the analysis. So they'd need to explain where the data comes from, what the analysis code does with it, etc. Now, if their papers fail to do this I'll agree with you - but in that case the papers should have failed peer review.
It shouldn't be necessary for someone to have their data and code to reproduce the results, any more than it should be necessary for someone to have the exact same lab and equipment to reproduce some other scientific experiment. If it depends on a specific data set, then by definition it isn't reproducible and so isn't scientific.
Okay, how do they gain lots of money from this? And would this be more than the lots of money they could get from the fossil fuel industry by publishing valid anti-AGW papers?
I said that *I* couldn't trace it from the source based on the leaked files. Can they? I don't know, we only saw a very incomplete set of documents regarding this. No word on whether this code was exactly what was used for the final publications.
Yes, *if* the path from data to results isn't fully accounted for, that would be a problem. Is this the case? Can't tell from what we've seen.
In any case, if I'm not very much mistaken the results have been reproduced by other scientists at different institutes.
There's no such thing as a felony in Great Britain. And there's still no firm evidence one way or another regarding the FoI thing.
And where exactly are these underhand methods or backroom deals? All I saw was a frustrated statement that "I'll do by best to stop these papers getting into the IPCC report" (paraphrased). No mention of an underhand methods to be used in doing so.
Hiding from FOIA requests, conspiring to lock out a publication that wasn't swallowing their bate (how dare a peer review journal ask difficult questions of AGW!).
Lock out? They thought the publication was publishing poor quality papers. If that was their belief, why would they not refrain from publishing there or citing articles?
Then we have Phil working to keep these two papers [populartechnology.net] from being seen at the next IPCC meeting--
I can't see either of these papers being in the next IPCC report. Kevin and I will keep them out somehow - even if we have to redefine what the peer-review literature is !
So he doesn't think the papers are worthy of inclusion. No evidence of any underhand arrangements to stop it though, just that he'll make the case that they shouldn't be included.
Let's not forget that these are private emails and people are much looser with language. I once make a jocular reference to Schoning my data - how incriminating would *that* look if it was written down?
In my parents time it was global cooling, when I was younger it was a giant hole in the ozone above Australia caused by big evil America, a year ago it was Global Warming, and now it's become "Climate Change".
There was never a consensus in favour of global cooling (despite the media reports at the time), the ozone layer was real and an international treaty was made to deal with the problem, so I don't know what your point is there. Global warming is the same thing as Climate Change, it was just felt that the first term would perhaps confuse those who think that global warming implies local warming everywhere on Earth. And your "scheme to misuse taxpayer money" remark? How do the scientists benefit from that, exactly? That's verging on tinfoil-hattism. A global conspiracy amongst scientists.... to what end?
Perhaps you can supply some links. I'm not saying such things don't exist, just that I haven't seen them. However, one thing I did see was a list of signatures from people opposed to the climate change theory - almost all of whom had no science qualifications.
Yes, they get involved with IPCC or the media from time to time. But IPCC's role is not to set policy, but to present evidence and options.
Finally, I don't see any reason as to why any involvement in this way this would influence their research. The other way round, yes.
The first part of what you say is true, but again no surprise. However, the part about deliberately fudging it all to get the "right" answer? I don't see that. But I wouldn't expect to see a full picture in a few old notes and bits of code. Private notes are always incomplete.