Slashdot Mirror
Call For Scientific Research Code To Be Released
Pentagram writes "Professor Ince, writing in the Guardian, has issued a call for scientists to make the code they use in the course of their research publicly available. He focuses specifically on the topical controversies in climate science, and concludes with the view that researchers who are able but unwilling to release programs they use should not be regarded as scientists. Quoting: 'There is enough evidence for us to regard a lot of scientific software with worry. For example Professor Les Hatton, an international expert in software testing resident in the Universities of Kent and Kingston, carried out an extensive analysis of several million lines of scientific code. He showed that the software had an unacceptably high level of detectable inconsistencies. For example, interface inconsistencies between software modules which pass data from one part of a program to another occurred at the rate of one in every seven interfaces on average in the programming language Fortran, and one in every 37 interfaces in the language C. This is hugely worrying when you realise that just one error — just one — will usually invalidate a computer program. What he also discovered, even more worryingly, is that the accuracy of results declined from six significant figures to one significant figure during the running of programs.'"
Particularly if the research is publicly funded.
Great!
I'm getting somewhat tired from reading articles, where there is little or no information regarding program accuracy, total running time, memory used, etc.
And in some cases, i'm actually questioning whether the proposed algorithms actually work in practical situations...
If Pandora's box is destined to be opened, *I* want to be the one to open it.
seem to understand the very idea of scientific methods or processes, or the reasoning behind empiricism and careful management of precision.
It's a failure of education, no so much in science education, I think, as in philosophy. Formal and informal logic, epistemology and ontology, etc. People appear increasingly unable to understand why any of this matters and they essentialize the "answer" as always "true" for any given process that can be described, so science becomes an act of creativity by which one tries to create a cohesive narrative of process that arrives at the desired result. If it has no intrinsic breaks or obvious discontinuities, it must be true.
If another study that contradicts it also suffers from no breaks or discontinuities, they're both true! After all, everyone gets to decide what's true in their own heart!
STOP . AMERICA . NOW
Much quantitative academic and scientific work could benefit from the use of tools like Sweave, which allows you to embed the code used to produce statistical analyses within your LaTeX document. This makes your research easier to reproduce, both for yourself (when you've forgotten what you've done six months from now) and others.
What other kinds of tools like this are /.ers familiar with?
"Anyone who [rips a CD] is probably engaging in copyright infringement." - David O. Carson
I've always been a big fan of releasing my academic work under a BSD licence. My work is funded by the taxpayers, so I think the taxpayers should be able to do what they like with my software. So I fully agree that all software should be released. It is not always enough to just publish a paper, but you should release your code so others can fully review the accuracy of your work.
The scientific community needs to get as far as we can from the policies of companies like Gaussian Inc., who will ban you and your institution for simply publishing any sort of comparative statistics on calculation time, accuracy, etc. from their computational chemistry software.
I can't imagine what they'd do to you if you started sorting through their code...
Hey mate, spare a sig?
Please apply Hanlon's razor before leaping to conspiracy theories. Or Occam's razor might inform you that a conspiracy among thousands of scientists is a highly improbable occurrence; look for a solution that doesn't involve a perfect lid of secrecy among a group of (frequently) socially inept people.
$_ = "wftedskaebjgdpjgidbsmnjgcdwatb"; tr/a-z/oh, turtleneck Phrase Jar!/; print
One significant figure?
What? Scientists showing their work for peer-review? It's MADNESS I tell you. MADNESS !
I'd like to see actual examples of the code failures mentioned in the T experiments paper.
Or at least Figure 9.
"Why should I make the data available to you, when your aim is to try and find something wrong with it"
-Prof. Jones CRU
My bet is there is a simple explanation...namely that scientists outside of computer science are too busy in their respective fields to know anything about code, or even care. The egocentric Slashdot-worldview strikes at the heart of logic yet again.
Nothing seems to change ...
30 years ago it was a standard joke that most "fundamental particles" were bugs in the Fortran programs of the day.
I wouldn't be surprised to discover that some of the programs inestigated are just the result of 30 years of further modification of the ones we knew ... and that nobody understands them now!
And it's situations like this which make the general public distrust scientists, or even science in general.
The media plays a major role, as well -- it oversimplifies and dramatizes scientific research as if it comes to conclusions that it usually doesn't -- but when it comes to light that a scientist has made a mistake, or that a research paper has had false premises or inaccurate results, then the average Joe Public thinks to himself, "Can't trust those scientists. Shoulda known."
Liberal? Conservative? Compare perspectives at Left-Right
You are correct that it's not a conspiracy: it's more of an organic phenomenon. Politicians do not generally gain renown and reelection for doing nothing. So research dollars go to people who can show politicians that they should be doing something. Eventually, the profession selects for people who already have a certain lens for looking at the world. And here we are.
I got my PhD in fluid mechanics funded by NASA, and as such my findings are easily publishable and shared with others. My analysis code (such as it was) was and is available for those would would like to use it. More importantly my experimental data is available as well.
This represents the classical pure research side of research where we all get together and talk about our findings and there really aren't any secrets. But even with this open example, there are still secrets when it comes to ideas for future funding. You only tip your cards when it comes to things you've already done, not future plans.
But more importantly, there are whole areas of research that are very closed off. Pharma is a good example. Sure there are lots of peer reviewed articles published and methods discussed, but you'll never really get into their shorts like this guy wants. There's a lot that goes on behind that curtain. And even if you are a grad student with high ideals and a desire to share all your findings, you may find that the rules of your funding prevent you from sharing.
Sheldon
Will ExxonMobil release all code that their scientists use?
Aside from logistics, there is no excuse for not doing this. In my experience, software innovations are notoriously sensitive to subtleties in input data (e.g. data mining, AI, image processing). Posting both code & data (and a test driver, of course) should be mandatory for all publications that claim to have found a signal in data, better or faster.
The question is, how to maintain code & data long after the publication publishes? IMHO, any peer-reviewed publication should be required to maintain such a repository for perhaps 20-30 years, ideally under a GPL (or its kin) so access to it would be free in perpetuity.
Maybe such a service would finally justify peer reviewed pubs' exorbitant fees for non-subscriber access.
The scientific process is to invalidate a study if the results cannot be reproduced by anyone else. That way you can eliminate all potential problems like coding errors, invalid assumptions, faulty equipment, mistakes in procedures, and 100 of the other things that can produce dodgy results.
It can be misleading to search through the code for mistakes when you don't know which code was eventually used in the final results (or in which order). I have accumulated quite a lot of snipits of code that I used to fix a particular need at the time. I am sure that many of these hacks were ultimately unused because I decided to go down a different path in data processing. Or the temporary tables used during processing is no longer around (or in a changed format since the code was written). There is also the problem of some data processing being done by commercial products.
It's just too hard. The best solution is to let science work the way it has found to be the best. Sure you will get some bad studies, but these will eventually be fixed over time. The system does work, whether vested interests like it or not.
Nobody said conspiracy, just plain crappy code. You don't need a conspiracy if you are "trying to prove" something, your crappy code spits out what you want to see and you run with it. You just need plain old incompetence.
Many scientists get their code from companies or individuals that license it to them, much like most other software. They're not in the position to release the code for many experiments...!
As it is written, the editorial is saying that if there is any error at all in a scientific computer program, the science is usually invalid. What a lot of bull hunky! If this were true, then scientific computing would be impossible, especially with regards to programs that run on Windows.
Scientists have been doing great science with software for decades. The editorial is full of it.
Not that it would be bad for scientists to make their software open source. And not that it would be bad for scientists to benefit from some extra QA.
and by that I mean climate science.
Science suffers from methodological "flaws", which are really just the rational interest of the people involved. One of them is that scientists do not tend to disclose data, defended and explained several times by scientists under the pro-climate banner, as effectively that if they DID publish, then a) people would misuse the data, but more commonly and much more importantly (after all, most science isn't controlversial) b) that other scientists would use their data and programs and publish papers on the basis of them. A complex data set is hence the same as JOB INSURANCE. I kid you not, look up the statements yourself.
Another is that advanced science with multiple obscure data sets needs advanced statistical knowledge, which by its very nature requires significant professional judgement. This is also obvious from having read the debates, e.g. the 700-page hacked analysis document.
Now, in most science, none of these problems are really a problem. Firstly, the science is rarely _very urgent_. The scientists can therefore sit in an ivory tower and debate for a decade or so before the research "leaks" into the outside world, or is even required or wanted by the outside world. People who disagree can say "is that really the case though? look at this little piece her over in the corner which looks strange" and everyone can gather around for tea. Life-critical science is always checked by multiple independent people and in small scales as much as possible - like drugs research, for example. If data and models diffuses slowly, and models are subject to judgement, this doesn't have much of an impact in the long run, and can be worked out amicably and in preservation of everyone's dignity and efforts wholly within the "scientist sphere". Getting personally attached to a cause is also meaningless, because few causes provide enough job security or money to even risk allegations of misconduct.
I feel it does represent a problem in the climate case though. One reason is that the science will impact _everyone_ before the data is fully comprehensive. There is no 'scientist sphere' or 'trial runs', all conclusions are implemented as soon as they are out. Secondly, the conclusions couldn't even be checked by nonscientists whom it would affect. Thirdly, the extreme passion and personal stakes in a strong climate movement makes me very skeptical that the professional judgement of statistical analysis has been exercised dispassionately and objectively (if that is even possible). Scientists who say "is this really the case?" are pillored. Pro-Climate people will say that "that's noe true, if anyone could disprove climate science they would be heroes!". Anti-climate people would respond that "much like proving takes a ton of work and hundreds of articles, so would disproving, and in the meantime their lives would be hell'.
As a result, I am convinced that the scientific community and method we have is totally unsuited to research something as complex as climate science and make a conclusion within a few years. I don't want to change my life and reduce my consumption on the basis of what might well be bullshit - so it's either very painful enforcement against the will and good conscience of a lot of people, or, the 'data' for a catastrophe would only be conclusively found when it happens.
People understand the theory behind science just fine. The problem, however, is that it is nothing but theory. And theory, like most things in academia, only works properly under a highly controlled "reality".
In the real world, people need to eat. People need vehicles. People need clothing. People feel the need for luxury items. To get these things, people need money. To get money, the vast majority of people need to do something of value for somebody who already has money.
When it comes to scientists, they need to get their funding. Much of that funding these days comes from corporations. Corporations are often in the business of fucking over other people. Poor science often helps them achieve this. Who provides this lousy science? Scientists, of course.
But wait, you'll say that some scientists get funding from the government. That is true. But keep in mind that in most Western nations, the governments are selected by the "people" from a slate of candidates funded by a small number of corporations or industry groups. So even the scientists who get their funding from politicians end up having to create "research" that fulfills the needs of the corporations funding the politicians.
The reason many researchers, especially climate scientists, are not so happy about divulging their models and data is that they can be sued by crackpots, as it has already happened. Even if they are proven right, a lawsuit is an expensive business. I can already imagine hordes of Exxon sockpuppets suing any random climate scientist they don't like.
Granting immunity from lawsuit should make them more willing to share data. Anyway, if something really bad is found in the research, the researcher will have their reputation tarnished, which in the environment is bad enough to ruin a career.
By the way: I am a researcher, and I attached the source code of my models in the PDF version of my PhD dissertation.
Victims of 9/11: <3000. Traffic in the US: >30,000/y
IT's the price of a society that doesn't actually value the liberal arts and the technology. Studying the greeks and romans matters and you need to be a well rounded thinker.
This is my sig.
As an example: Releasing automotive code falls clearly in the interest of public safety. Do you really think any company will release source code? If WOZ had the source for his Prius, the current problem might have had a happy ending.
When people have confidence in their code, they release sources. When they are afraid of what might be hiding in their code, they lock it up.
Very few have confidence, whether in industry or science.
I'm perfectly OK with the fact that their job is science and not coding, but would they go to the satellite assembly guys and start gluing parts at random ?
Non-Linux Penguins ?
The reason many researchers, especially climate scientists, are not so happy about divulging their models and data is that they can be sued by crackpots, as it has already happene
On what basis of damages can a researcher be sued?
This is my sig.
As it happens, my students and I are about to release a fairly specialized code - we discussed license terms, and eventually settled on the BSD (and explicitly avoided the GPL), which requires "citation" but otherwise leaves anyone free to use it.
That said, writing a scientific code can involve a good deal of work, but the "payoff" usually comes in the form of results and conclusions, rather than the code itself. In those circumstances, there is a sound argument for delaying any code release until you have published the results you hoped to obtain when you initiated the project, even if these form a sequence of papers (rather than insisting on code release with the first published results)
Thirdly, in many cases scientists will share code with colleagues when asked politely, even if they are not in the public domain.
Fourthly, I fairly regularly spot minor errors in numerical calculations performed by other groups (either because I do have access to the source, or because I can't reproduce their results) -- in almost all cases these do not have an impact on their conclusions, so while the "error count" can be fairly high, the number of "wrong" results coming from bad code is overestimated by this accounting.
Back in college, I did some computer vision research. Most people provided open source code for anyone to use. However, aside from the code being of questionable quality, it was mostly written in Matlab with C handlers for optimization.
In order to properly test all of the software out there you would need:
1. A license for every version of Matlab.
2. Windows
3. Linux
4. Octave
I had our school's Matlab, but none of the code we found was written on that version. Some was Linux, some Windows, (the machine I had was a Windows box with Matlab) consequently we had to play with Cygwin...
I mean, basically, you need to distribute a straight-up VM if you want your results to be reproducible. (which naturally rules out Windows or Matlab or anything else proprietary being at the core.)
So if scientists use MS Excel for part of their data analysis, MS should release the source code of Excel to prove that there's no bugs in it (that may favour one conclusion over another)
Soumds fair to me.
And if MS doesnt comply then all scientists have to switch to OO.org ?
Sure it's nice for peers to review, but to make it a mandatory thing? I thought science was based on rules with the assumption there are no rules.
I wonder what Newton, Einstein, Kepler, or Goddard would have thought if they were demanded to follow this type of review. Really. Peer review has become the Java of the research world if you know what I mean (the solution for everything)--there should be multiple forums for informal discussion/verification, formal proposals/verification.
You're right, Occam's Razor. Conspiracy is generally too hard, even if you know what you're doing. Who needs conspiracy? Group-think, socio-political cliques, popular public funding streams, fashion, peer pressure, yearning for acceptance by an in-crowd. Know what really brought the US to its knees in Viet Nam? Hippy Chicks. Wanted to get laid? You were anti-war.
I'm working on my dissertation proposal, and I'd like to be able to re-run the benchmarks that are shown in some of the papers I'm referencing. But must of the source code for those papers has disappeared into the aether. Without their code, it's impossible for me to rerun the old benchmark programs on modern computers so that I and others can determine whether or not my research has uncovered a better way of doing things. This is very far from the idealized notion of the scientific method, and significantly calls into question many of the things that we think we know based on published research.
Humans are hardwired for fear and have to learn to think factually. Like most scientific issues that become political, fear and misinformation dominate over political fact. There will always be a certain segment of the population that believes vaccines cause autism and global warming is a trick to tax us with cap and trade. With vaccines you wait for the kids of autism avoiders to die of measles and polio. With global warming change the message from tax to disincentive to tax credit to incentive. Energy independence (make it a defense issue), tax credits for solar and wind that make the payback for a home owner less than a decade (I suspect a five year payback will get homeowners and home builders forking over for energy improvements).
If all scientists run their results through the same analytical software, using the same code as the first researcher, they are not providing confirmation, they are merely cloning the results. That doesn't give the original results either the confidence that they've been independently validated, or that they have been refuted.
What you end up with is no-one having any confidence in the results - as they have only ever been produced in one way and arguments thatt descend into a slanging match between individuals and groups of vested interests who try to "prove" that the same results show they are right and everyone else is wrong.
politicians are like babies' nappies: they should both be changed regularly and for the same reasons
If you ever get a chance take a look at some of Baggerly's (MD Anderson / bioinformatics/stats) analysis of the number of rather embarrassing mistakes were used in developing genomic biomarkers used for a clinical trial at Duke. He has been giving talks around at stats conferences (and pharma's about this), its one of the best talks i've heard in recent years. But what it boils down to is the analysis (and input) programs used by Duke had a series of fundamental mistakes in it causes the results to be incorrect leading to an incorrect conclusions which unfortunately lead to a series of clinical trials which certainly should not have happened. After Baggerly attempted to respond negatively to the original series of articles being posted he reposted in a stats journal and basically got the clinical trial shut down. For slashdot readers, one of the rather many egregious mistakes here was the analysis program used has in its instructions the need for a header line, the input the Duke researchers used did not include a header line causing a shift in the results with regards to their input. My understanding is nature medicine refused to publish baggerlies initial correspondence with full details as it was "too negative" so he published in a stats journal which then got the critical coverage to shut everything down..
Here are some random links
Here is the original Potti genomics article:
http://www.nature.com/nm/journal/v12/n11/abs/nm1491.html
Here is one of the baggerly nature medicine letters describing what is wrong in summarized form:
http://www.nature.com/nm/journal/v13/n11/full/nm1107-1276b.html
here is the halt of the trials :
http://cancerletter.com/tcl-blog/copy108_of_whats-going-on-with-nih
http://cancerletter.com/tcl-blog/copy111_of_whats-going-on-with-nih
Since you brought up the socially inept idea, I might also suggest taking a look at Bic's razor or Gillette's razor.
Please don't use "umm" or "err" or "erm".
...so science becomes an act of creativity by which one tries to create a cohesive narrative of process that arrives at the desired result.
As someone who listens to Talk Radio on occasion, that sounds like you're creating a work of fiction. Rush and Hannity would have a whole week of shows based on that statement.
I would put it more like "piecing the narrative from the evidence" or "from facts" or something like that.
Scientists need to realize that if they're going to get public support, they really need to be very careful with their choice of wording. Like it or not, the scare mongers, and I mean scare mongers in the sense that there are people who are trying to scare folks into believing that Global Warming is some sort of wealth redistribution scheme by the socialists, are going to use any hint, real or not, that scientists are making up their findings.
For example, interface inconsistencies between software modules which pass data from one part of a program to another occurred at the rate of one in every seven interfaces on average in the programming language Fortran, and one in every 37 interfaces in the language C.
Ah, classic example of the difference between s/w engineering and s/w development. Problem is it's hard to tell and engineer to thinking like a scientist and vice versa.
The code quality of many well-known scientific software packages is abysmal.
In chemistry, you should at least expect that the outcome of descriptor computations on a set of molecules is independent of the order of atoms and bonds in a molecule, and the order of file records.
Well, this is disturbingly often not the case, as we discovered in a recent study.
In an attempt to raise awareness of this problem, we have launched a public Web-accessible computational result verification service (http://www.xemistry.com/cv). A poster explaining this app and some background, including sample test results, can be found at http://www.xemistry.com/Presentations/verifier_panel_2009.pdf.
Unfortunately, the worst application we have encountered so far appears to be a standard tool for adding Wikipedia data for chemicals, systematically poisoning it with incorrect data.
That statement was kind of breathless, but the study he was citing focused on bugs that specifically affected the accuracy of the output, and found that they were a common occurrence. I agree with the author, if you are going to use a computer program to get results, you need to publish code otherwise your methods are packaged in a black box. A lot of people don't want to do this because scientific code is not usually done by people we can say are knowledgeable in how to write reliable, verifiable code. It's usually a pieced together means to an end. Not that there's anything wrong with that, IF it can be available for verification. I HATE reading studies that for example constantly refer to a dataset and then never give you the dataset. I guess unlike many people I don't naturally trust the authors to be perfect.
What about McIntyre's faulty data?
Ah, no FOIA there, because he's toeing the party line.
Note: He's not the only denial ditto who refuses to release his code:
http://www.realclimate.org/index.php/archives/2009/12/please-show-us-your-code/
Oh, the meeja is quiet about that, isn't it...
Who said it was a conspiracy of a thousand people? Isn't the claim that the ipcc reports are written by thousand people? That's not exactly true. The actual report was written by something like 50 people the thousand number comes from the supporting data. There were a lot of groups of people who did independent research that showed "whatever", and the "whatever" supports the "conclusion" so it is used, and things that don't support the "conclusion" are suppressed.
Now we have a fancy report, we pass that around, and it is solid and everyone agrees with what they see
I am suspect of the interface reference. Are they counting things where an enumeration got used as an int, or there was an implicit cast from a 32bit float to a 64bit one? From a recent TV show "A difference that makes no difference is no difference." Stepping back a bit there will be howls from OO/Functional/FSM zealots that look at a program and declare its inferior architecture, lack of maintainability etc. indicate its results are wrong. These are programs written to be run once to turn one set of data into a more understandable and concise one. A truth test set run through it is good enough, they don't need iso compliant, triply refactored, perfectly architectured code to get the right answer. I don't think any of my CS proffs would have cared about such inane drivel they barely paid attention to what language we each picked to solve the assignment in. My software engineering proff would have yelled about comment density and coding standards compliance, but I consider that a different discipline primarily applicable to widely used and/or safety critical code.
*However*
Keeping track of digit precision through a calculation isn't CS, its fundamental grade school science. That is only one step from forgetting to do unit analysis for a sanity check. If they are forgetting that, they are probably also not looking at numerical conditioning, or trying to get by with doubles when they need bignums. None of this is CS egocentrism, its stuff we learn in math and science courses.
refactor the law, its bloated, confusing and unmaintainable.
After the flawed pseudorandom number algorithm whose use may have invalidated quite a few statistical simulations.
start their posts in the title? it make it seems as if they do not think what they are going to write.
On topic with the article, I completely agree with this "release the scientific code" position. I am currently working within a EU project in which we are developing ABM*. In my project it was made clear from the beginning that the code license will be GPL.
However the place where I work has some program they have used to conduct simulations, these programs are complete closed (only a handful of people have access to the code, everyone within the institute). Nevertheless, simulations performed with such program has been used for several publications (journals, congresses, symposiums and even PhD thesis!).
And people in my line of work wonder why simulations are not taken more seriously (e.g. accepting papers) by people in more "classical" research fields.
Ubuntu is an African word meaning 'I can't configure Debian'
I'm finishing a program that inverts GRACE data to reveal fluctuations in gravity such as those caused by melting glaciers. This program will eventually be released as open source software under the GPLv3. It's largely built on open source libraries like the GNU Scientific Library, but snippets of proprietary code from JPL found their way into the program years ago, and I'm currently trying to untangle them. The program can't be made open source until I succeed because of an NDA that I had to sign in order to work at JPL.
It's impossible to say how long it will take to banish the proprietary code. While working on this project, my research is at a standstill. There's very little academic incentive to waste time on this idealistic goal when I could be increasing my publication count.
Annoyingly, the data itself doesn't belong to me. Again, I had to sign an NDA to receive it. So I can't release the data. This situation is common to scientists in many different fields.
Incidentally, Harry's README file is typical of my experiences with scientific software. Fragile, unportable, uncommented spaghetti code is common because scientists aren't professional programmers. Of course, this doesn't invalidate the results of that code because it's tested primarily through independent verification, not unit tests. Scientists describe their algorithms in peer-reviewed papers, which are then re-implemented (often from scratch) by other scientists. Open source code practices would certainly improve science, but he's wrong to imply that a single bug could have a significant impact on our understanding of the greenhouse effect.
All the software I and my team write is version controlled (CVS and SVN) and releasable (some of it with export restriction.) I do admit that our software engineering is not the best, but we also have a rule that code cannot be committed before an extensive suite of tests is run on the code (for our main scientific application--'helper' tools are not so tightly controlled.)
Our scientific colleagues, provided they can satisfy the export control restrictions, can get source code and poke around all they want. Some have even contributed back valuable changes, and most have contributed back valuable feedback.
In fact, we look down upon colleagues who do not release source code. Are you really doing serious science if others cannot delve into your methods?
--PeterM
NIH funding standards promote commercialization of publicly funded software. This appears to have been implemented before the modern internet, and the idea may have been that a commercial product would make the code more available, and perhaps fix some of the quality issues with code cobbled together by "non-programmers". The result is that companies like Accelrys own a huge amount of software developed under public funding. Now, the public has to pay to use software trhat they paid to develop, and it is impossible for other scientific research to extend that publicly funded effort.
I want to see an NIH version of SourceForge, and mandate all government funded software development to be stored there. Unlike SourceForge, there could be delayed release to the public so that researchers have time to publish their work.
As part of publication and peer review all data and providence of the data as well as any additional formula's, algorithms, and the exact code that was used to process the data should be placed online in a neutral holding area.
Neutral area needs to be independent and needs to show any updates and changes, preserving the original content in the process.
If your data and code (readable and compilable by other researchers) isn't available then peer review and reproduction of results is foolish. If you can't look in the black box then you can't trust it.
Ward
. Silence! Be thankful thy species is unpalatable! .
Besides code, it would be nice to have mechanically verifyable proofs too!
But code would be a nice first bit of progression.
If Pandora's box is destined to be opened, *I* want to be the one to open it.
What is that supposed to mean?
Swedish plasma phys. PhD student; MSc EE; knows maths, programming, electronics; finance interest; seeks opportunities
...so they can also release the proofs.
Code should be release, but this should not be confused with replicating scientific results. ] If you want to replicate research, you need to write your own code according to the methods described in the research. your answer then needs to match the original to test the code.
Not sure on any "lucrative" fields exclusion as basically all science eventually has engineering then lucrative aspects to it, even if it might be some time down the road and is obscured now.
Climate science and this huge acrimonious debate has trillions associated with it, such as the artificial carbon credits market,(among others) which had no previous business market or demand that existed, else there would have been a big demand for carbon credits previously. If anything, this lucrative "science" should therefore be as open as possible, precisely because trillions is a huge inducement for outright fraud and obfuscation of same.
Personally,if you want a cleaner environment with the possible long range benefit of possibly keeping the climate more moderate, the carbon tax then on to the wall street gamblers idea is quite a stretch as to real effectiveness.
I will contend there is a much cheaper way, a way wherein those same trillions would go *directly* to deployment of the new technologies, and that is the tax credit instead of the tax itself. If governments were serious, they could offer 100% tax credits for deployment of new cleaner tech, from a personal level to any size corporate level, then that would be that, the tech would get used, quite willingly, and all that money go to it, rather than getting most of it shuffled off to the already bloated artificial financial products "industry", which already takes an inordinate amount of wealth out of the system and places it into the international casino.
They hire statisticians to argue that the errors average out ...
Code should be release, but this should not be confused with replicating scientific results. ] If you want to replicate research, you need to write your own code according to the methods described in the research. your answer then needs to match the original to test the code.
Just in case there is some confusion, code release is part, but not the whole of replicating scientific results.
People are taking the article the wrong way if you actually read it you will see that most of the errors are "Undeclared function prototypes". I hardly see how that can scientifically effect the outcome of any result. Next, the article is awful in that it hardly defines what it is saying at any moment, it doesn't mention actual programs except Motif and X11??? Are you serious? Since when is X11 a scientific program?
I switched from MS Excel to OOcalc for some analysis I'm doing at the moment. Excel was slow, made huge files and the bar charts were hideously shaded by default.
OMG there is bugzorz in teh codezor'z? Oh Noooooesonehundredandeleventyoneoneone!!!!!
Confucius say, "Find worm in apple - bad. Find half a worm - worse."
This brings up an interesting point: what about all the different closed source packages which are used in the hard sciences? On a regular basis I am required to use code written using closed source framework al la IDL or MATLAB. These frameworks are invaluable in the sense that we do not have to write the interface between the detectors and a standard PC, but they are often essentially black boxes. Do people really want to require scientists to write (inevitably bad code) to interface their scope to a PC just so they can release that code to the public?
Finally, I waste enough time fixing bugs in code that will likely never be used by anyone else that I would loath to have to clean it all up for others to use and understand. Scientists are writing tools to get things done, if they had the resources most would love to hire a programmer to write and document the program from a requirements document but very few if any do have such resources. There should be a balance between the publics desire to look over the shoulder of scientists are scientists desire to do science and not waste their time getting 60k emails regarding a bug which has no effect on their result.
yes but its the Guardian and their grasp of technology is a little weak at the best of times.
I would go further than just publishing the code used in scientific research. I would build the code by running it a real open source project. In fact, I've done exactly that, and it worked out incredibly well. I believe our open source approach lead to better science, and also better software.
I worked with researchers from MIT and Columbia on a research project that involved gathering and analyzing a large amount of publication data. The results of the study are about to be published (you can read the working paper at the lead researcher's website).
We intended the code for this project to be released from the beginning, so we ran it as an open source project. I followed the basic formula from Karl Fogel's excellent (and free to download) book, Producing Open Source Software: set up a website for the project, created lots of documentation, tried to make it as easy as possible for someone to get up and running, made the source available via Subversion, and made it easy to contact us.
Quality was really important for us, so we put a lot of effort into testing. I definitely believe that the fact that we intended the project to be open source from the beginning helped with that. We weren't treating the code as some piece of throwaway or replaceable lab equipment. I'm convinced that treating it as a real product of the research caused us to take the development and the quality much more seriously than a lot of researchers. I've since heard from other researchers who are starting to use the software as well, and everyone who sees it feels that it came out really well.
There was another scientific benefit that should definitely appeal to anyone who lives in the publish-or-perish world of science research. We published a paper specifically on the project (Azoulay P, Stellman A, Zivin JG. PublicationHarvester. An open-source software tool for science policy research. Research Policy 35 (2006) 970-974. -- there's a link to the PDF on the lead researcher's website.)
It's funny -- I wrote an article a few years ago with Jennifer Greene for O'Reilly ONLamp called What Corporate Projects Should Learn from Open Source. I'm now convinced that science research projects can also learn a great deal from open source as well.
Building Better Software
You mean scientists who don't toe the party line, that AGW is just a communist plot to let Obama take all USians into a Maxist state and kill their grannies.
The more important the research, the larger the item under study, the more rigorous the investigation should be, the more carefully the data should be checked. This isn't just for public policy reasons but for general scientific understanding reasons. If your theory is one that would change the way we understand particle physics, well then it needs to be very thoroughly verified before we say "Yes, indeed this is how particles probably work, we now need to reevaluate tons of other theories."
So something like this, both because of the public policy/economic implications and the general understanding of our climate, should be subject to extreme scrutiny. Now please note that doesn't mean saying "Look this one thing is wrong so it all goes away and you can't ever propose a similar theory again!" However it means carefully examining all the data, all the assumptions, all the models and finding all the problem with them. It means verifying everything multiple times, looking at any errors any deviations and figuring out why they are there and if they impact the result and so on.
Really, that is how science should be done period. The idea of strong empiricism is more or less trying to prove your theory wrong over and over again, and through that process becoming convinced it is the correct one. You look at your data and say "Well ok, maybe THIS could explain it instead," and test that. Or you say "Well my theory predicts if X happens Y will happen, so let's try X and if Y doesn't happen, it's wrong." You show your theory is bulletproof not by making sure it is never shot at, but by shooting at it yourself over and over and showing that nothing damages it.
However that this process is done right becomes more important the bigger the issue is. If you aren't right on a theory that relates to migratory habits of a sub species of bird in a single state, ok well that probably doesn't have a whole lot of wider implications for scientific understanding, or for the way the world is run. However if you are wrong on your theory of how the climate works, well that has a much wider impact.
Scrutiny is critical to science, it is why science works. Science is all about rejecting the ideas that because someone in authority said it, it must be true, or that a single rigged demonstration is enough to hang your hat on. It is all about testing things carefully and figuring out what works, and what doesn't.
Only if the real programmers out there promise to be nice to us scientists.
Most scientists will know a lot about, well, science... but not much about writing code or optimizing code.
Like my scripts. All correct, all working... lots of formulas... but probably a horribly inefficient way to calculate what I need. :-)
the last thing I need is someone to come to me and tell me that the outcome is correct but that my code sucks.
(And no, I am not interested in a course to learn coding - unless it's a 1-week crash course).
Not all code needs to be released; sometimes, the code isn't needed for reproducing a result or the formulas in the paper are sufficient.
However, climate modeling is such a complex process, and its implications so serious, that it is outrageous if climate modeling code isn't fully and completely available. Climate modeling code needs to be extensively peer reviewed and until people can reproduce the results and show that the models behave reasonably under different conditions, the results cannot be trusted.
I don't subscribe to Hanlon's razor; I have my own. Mine says "never ascribe to incompetence or stupidity that which can be explained by greedy self-interest".
Although in this case, Occam rules. The "mcgrew's razor" almost never applies to scientists and almost always applies to corporate types.
Free Martian Whores!
Comment removed based on user account deletion
This is hugely worrying when you realise that just one error -- just one -- will usually invalidate a computer program.
Back in the 1970s, a bunch of CompSci guys at the university where I was a grad student did a software study with interesting results. Much of the research computing was done on the university's mainframe, and the dominant language of course was Fortran. They instrumented the Fortran compiler so that for a couple of months, it collected data on numeric overflows, including which overflows were or weren't detected by the code. They published the results: slightly over half the Fortran jobs had undetected overflows that affected their output.
The response to this was interesting. The CS folks, as you might expect, were appalled. But among the scientific researchers, the general response was that enabling overflow checking slowed down the code measurably, so it shouldn't be done. I personally knew a lot of researchers (as one of the managers of an inter-departmental microcomputer lab that was independent of the central mainframe computer center). I asked a lot of them about this, and I was appalled to find that almost every one of them agreed that overflow checking should be turned off if it slowed down the code. The mainframe's managers reported that almost all Fortran compiles had overflow checking turned off. Pointing out that this meant that fully half of the computed results in their published papers were wrong (if they used the mainframe) didn't have any effect.
Our small cabal that ran the microprocessor lab reacted to this by silently enabling all error checking in our Fortran compiler. We even checked with the vendor to make sure that we'd set it up so that a user couldn't disable the checking. We didn't announce that we had done this; we just did it on our own authority. It was also done in a couple of other similar department-level labs that had their own computers (which was rare at the time). But the major research computer on campus was the central mainframe, and the folks running it weren't interested in dealing with the problem.
It taught us a lot about how such things are done. And it gave us a healthy level of skepticism about published research data. It was a good lesson on why we have an ongoing need to duplicate research results independently before believing them.
It might be interesting to read about studies similar to this done more recently. I haven't seen any, but maybe they're out there.
Those who do study history are doomed to stand helplessly by while everyone else repeats it.
The best way to keep this from happening is to avoid passing formatted, human-readable data between programs. That's what FORTRAN's unformatted I/O was meant for. Same thing for C. Don't convert to a convenient human-readable form until the very end.
CUR ALLOC 20195.....5804M
While I am fan of open source and this idea in general, for climatology, this is a non-issue. Look there: http://www.realclimate.org/index.php/data-sources/
It's more code out there than one amateur can eat for life. And you know what? From the experience of people who wrote these programs, there isn't actually much people looking at it. I doubt that any scientific code will get many eyeballs. This is more a PR exercise.
The problem isn't so much one of not releasing source code, but not releasing any code at all, not releasing the methods used. If someone uses an excel sheet to do calculations and releases that sheet, that's good enough. You can then test it and see how it works. You don't need to audit the excel code, you only need to verify that the results you are getting are valid.
What is important is full disclosure of methods. Someone else should be able to replicate your findings independently. That means you need to disclose all your data (and how you obtained it) as well as all your methods. Well, of your wrote custom code, that code would be part of your methods. If you used an off the shelf program, say which one.
In the case of the Excel situation so long as they released the excel sheet everything would be fine. You could of course load it up in excel yourself and see what happened. You could also test excel if you though there was a bug with your own data. You could take the calculations they used in excel and put them in to OO or SPSS or whatever other calculations package you liked and see if the result was the same. You could even do it by hand, if you had the time.
The problem starts when methods and/or data are kept hidden away. When there is a "We analyzed this data using our special program and got this result but no you can't see it," situation. The experiment can't be replicated at that point, and there's all sorts of room for error. On point with the climate thing would be the "hockey stick" program. When it was analyzed, by running it not by looking at the source, it turned out the thing just liked to generate graphs that curved up like a hockey stick, regardless of the data input. Ok well clearly there was a problem with the method, the code. Doesn't really matter what that is, you don't need to audit the code and find it, just to find out that the method was flawed, so the conclusion doesn't follow the data.
A good example of how disastrous one error in data analysis code can be comes from the field of biochemistry, specifically analyzing the x-ray diffraction patterns from crystallized proteins to determine the three dimensional structure of the proteins. Geoff Chang, faculty at the prestigious Scripps Research Institute in San Diego, had published a series of landmark papers describing the structures of various important membrane proteins, whose structures had never been solved before. Although these results did not mesh other researchers' models for how these proteins worked, many researchers used Chang's structures as starting points to develop and test new models for how these proteins might work. However, in 2006, things came crashing down:
In September, Swiss researchers published a paper in Nature that cast serious doubt on a protein structure Chang's group had described in a 2001 Science paper. When he investigated, Chang was horrified to discover that a homemade data-analysis program had flipped two columns of data, inverting the electron-density map from which his team had derived the final protein structure. Unfortunately, his group had used the program to analyze data for other proteins. As a result, on page 1875, Chang and his colleagues retract three Science papers and report that two papers in other journals also contain erroneous structures.
(from a Science news article (subscription required to view full article):
Of course, there were other factors that caused this situation (because the proteins are notoriously difficult to work with, the data were fairly poor quality. Had the data quality been better, Chang would have likely realized the mistakes prior to publishing the papers). However, it is sobering to note the resources wasted on research following up on these incorrect structures produced by a simple coding error (I know a few people whose entire theses were invalidated by these retractions).
It is, however, unclear whether releasing the data analysis code would have fixed this situation. The software for analyzing x-ray diffraction data is fairly standard (I don't know why Chang was using his own homemade software) and various open-source software are available. Furthermore, in his field, it is common to release the raw diffraction data (I'm not sure if it was released in the case of these five structures), so it may have been possible for others to double check his work with their own analysis software. Perhaps the greater error here is in Chang, the peer reviewers of his publications, and the scientific community for believing Chang's conclusions (based on relatively poor quality diffraction data) over the conclusions of many other researches whose techniques may not have been as sophisticated, but who had generated data of much higher quality.
So what if there are bugs, isn't what matters that the answer is correct? This is done, at least in my organization, as follows: 1) with a test problem. you can run a calculation that is easily solved analyitically and then compare results. You can also challenge yourself a little more by running a problem with ready experimental results for, and then again, comparing results. Only after this first step is done (many, many, many times)does confidence in the code build up and it can begin to be used in new areas. This is called qualification, verification, or validation. 2) When attacking problems not previously solved (which, after all, is the reason for writing the code), the scientist/engineer/end-user must have an expectation of what the results will be like. They may not know values, but they should expect what has changed since the last model they ran (i.e., "if I increase temperature by 10 degrees in my model, then this should happen..."). 3) While this may not be possible in all fields, either an experiment, or manufactured product should be tested to ensure that you got out what you predicted with your code. Engineering organizations can do this. Climatologists probably can't, I'd imagine. While I don't care about the debate of opening up the source code, I take issue with the fact that a comp sci guy looked at some scientist/engineer's code and said 'omg bugs!' Sure, it may not be how the comp sci expert would program, but it doesn't matter in the end (provided qualification is done adequately). Personally, I don't want to open up my code beyond who is necessary to see it - the code is not the end, its just a means. Its like a Doctor allowing everyone to see his personal diary on patients - its only work to support the diagnosis, and not the diagnosis itself.
It seems to me that what's important is the theory being modeled, the algorithms used to model it, and of course the data. The code itself isn't really useful for replicating an experiment, because it's just a particular - and possibly faulty - implementation of the model and as such is akin to the particular lab bench practices that might implement a standard protocol. Replicating a modeling experiment should involve using - and writing, if necessary - code that implements the model the original investigators intended to implement, but distinct from that which they actually used.
Running the same code on the same data demonstrates very little, and finding bugs in the original code tells you nothing about what results would/should have been achieved had the model been implemented correctly. But of course it's great for throwing stones and "discrediting" a result without actually adding anything constructive to the issue at hand.
I recommend the "reproducible research" methodology described at http://reproducibleresearch.net/index.php/Main_Page . The idea is that for each paper you publish, you make available an archive with the software, data, scripts, etc., so the user need only type "make" to reproduce every figure in the paper.
I hate to break it to you but all programming is highly specialized. Climatology is in no way special in this regard.
Neither do programmers have to understand the abstract model of the program to write it or evaluate it. The vast majority of professional programmers do not understand the abstract model of the code they create. You do not have to be a high-level accountant to write corporate accounting software and you don't have to be a doctor to write medical software. Most programmers spend most of their time implementing models created by non-programmers from fields of which the programmers have no detailed knowledge.
Does that mean that programmers can't spot crappy code just because they don't understand the details of the model? No, it does not. Most software errors don't arise from the model but from sloppy practices in the management of the software project itself. An experienced programmer doesn't even have to know the language of project to see that it's creation and maintenance was incompetently handled.
You don't have to be a climatologist to know that the CRU software was utter crap that would produce sound outputs only by divine intervention. For any experienced programmer, it was immediately obvious that it was a great reeking gob of amateur coding with no structure, no plan and no standards. In my experience, most scientific software is like the CRU software. It evolves in an ad hoc manner over many years with no governing organizational structure.
Commercial software developers have created a wide range of tools and procedures to manage large, vital projects. In the main, scientist use none of these tools and most of them appear unaware they even exist much less why and when they are needed. As a result most scientific software project management is completely amateurish. If most scientific software were written for commercial applications, the developers would be sued or imprisoned for fraud.
Scientist tend to be arrogant and dismissive of the work of others especially those who work in the commercial sector. You believe that because you understand climatology that you therefore understand all the tools you are using. Well you don't. You think that because no one can understand your abstract model that therefore they cannot find significant errors in your code. Well, they can. You think we should reengineer our entire civilization based on your unquestioned and unexamined computerized ivory tower auguries.
Well, we won't.
Your just going to have to suck it up and withstand the at least the same scrutiny we give important commercial software.
I'm old school but we were drilled to death on knowing the degree of accuracy of any mathematical results. I can't imagine scientists who do not know the degree of inaccuracy in any work product.
Does this include the "science" of Economics?
If so - I think that would be a VERY good thing.
Particularly in the area of derivatives trading.
It's estimated that 20% of the global economy now relies on trading derivatives, whose valuation is based on formulae, that are considered "proprietary" - and therefore, you're supposed to just trust the seller on the valuation. (by the way, I've got this bridge I'm selling, in Alaska . . . )
Ironic that Climate science is regarded with such skepticism, yet people in the high institutions of banking or the University of Chicago, Economics department so feverishly cling to the belief in some "invisible hand".
These are my friends, See how they glisten. See this one shine, how he smiles in the light.
If you do your data analysis in R and write your paper in latex, then you can use sweave to create a single file with the R code embedded in the latex. When you process the latex file, the R code is run, generating your stats and figures on the fly for the resulting document. If this file (plus any data files it reads, and any non-standard code it calls) is posted as "supplementary material" along with the PDF journal article, then there is no question about the software or analysis that led to the figures in the paper. Of course, the data itself is still open to question.
No one likes their And_Then_A_Miracle_Occurs() subroutines aired out in public.
Verifying that the code used by the researchers is consistent with the methods described, without accidental errors, is basically what we're talking about here, and is essentially equivalent to writing your own code. Eliminate researcher errors in their coding, and you're left with one less possible flaw in an experiment.
-mrxak
Onions Will Kill You
If a proposal like this is to succeed -- and I hope so hard that it does -- we need a central repository to store code and meaningfully link it to the papers that use it.
This repository should have the same amount of peer review and, therefore, authority that scientific journals have now. Maybe that can happen by existing journals adding the ability to link code to a paper (and enforce that any code used to generate results is included), maybe a new organization has to rise up to the challenge (I would love to see a code.arxiv.org).
Already I can hear the outcry of scientists claiming that their code is "sloppy" and "not ready to be released," but those concerns are simply irrelevant: all that matters is that the code produces the output cited in the paper given the input cited in the paper. That's it. If another researcher finds your result interesting, then let it be up to them to parse out your code -- it's probably still way better than trying to reproduce your result based on the prose that describes your algorithm.
My thesis advisor rewrote his textbooks every five years or so. But bemoaned losing the original copies of figures and being able to regenerate them in ever-improving computer/print media. So he started the requirement of reproducible documents : the computer programs, both scientific & graphic along with raw data much me assembled together for every figure in your thesis and scientific paper. These would be assembled into makefile-like system to create whatever portion one needed. Then they were archived for posterity. The ultimate test was to "burn" your figures, i.e. erase them from the upcoming document. Then the programs would be run to regenerate them. In practice there are "degrees" of regeneration. Sometimes the figure data is the output of a multi-month supercomputer run. So just the document formating programs would actually be run on the raw output data.
This system was just mentioned in Science magazine Jan 22 2010 p 415. (no free link)
"Philosophy of science is about as important to scientists as ornithology is to birds." - Richard Feynman
You know how you guys usually cringe when some clueless Economist or Law professor wants to introduce some new law for how the internet should work? Like demanding that all TCP connections be logged and the data kept for 10 years ( or something even more stupid ).
Well a fair share of you should now know exactly why they do this, because you keep doing the exact same thing every time a Slashdot article about science is published. To put it simply, if you're one of the idiots who thinks its a good idea to demand that every science paper to be published should document everything right down to what color of toilet paper the researchers used to blow their noses, then you never again get to complain when management wants all decisions to be accompanied with a cost-benefit analysis.
Its been 15 years since I worked in university labs, but I clearly recall that the code written by most Physics professors was horrid.
It was only the people like myself who had studied Computer Science who knew now bad most of that software was. Nightmares to debug and maintain.
But it doesn't stop there. Consider the data handling techniques used in the social sciences. I've seen first hand how the data from survey's were manipulated over and over, replacing the original data sets on each step. The original data was completely lost and the steps to get from the original to the published data were completely untraceable.
My conclusion is that (in general) academic scientists are no better than the general public at data handling and data verification techniques.
There are benefits and drawbacks to releasing code. First, let me say I am a scientist, and I do release my code, primarily under the GPL unless I am contractually obligated by license agreements to do otherwise. The exception would be trivial single use code or limited use code. Despite that, I don't get a lot of eyes looking at my code.
Some people here have claimed that journal policies are a part of the problem. Not as far as code releases go. Journals don't publish code. I think there are real impediments to code release, even when most scientists would claim that the release is a good idea.
In the case of climate modeling code, I'm guessing that #2 is probably the biggest hurdle. In my code, #3 is probably the biggest hurdle, and it does take significant effort to work around it.
Please read up on the history of behind measuring the charge of an electron. You are wrong in essence, the conspiracy theorists are wrong about motivation.
But either way, you cannot trust scientists data as truth. Especially early science, when the experiment has not been copied by millions.
Scientists are human, just like anyone else. To misquote, "the scientific method is the least terrible method of finding truth that we have yet discovered."
while (sig==sig) sig=!sig;
Damn slashdot and its awful posting interface.
There are benefits and drawbacks to releasing code. First, let me say I am a scientist, and I do release my code, primarily under the GPL unless I am contractually obligated by license agreements to do otherwise. The exception would be trivial single use code or limited use code. Despite that, I don't get a lot of eyes looking at my code.
Some people here have claimed that journal policies are a part of the problem. Not as far as code releases go. Journals don't publish code. I think there are real impediments to code release, even when most scientists would claim that the release is a good idea.
In the case of climate modeling code, I'm guessing that #2 is probably the biggest hurdle. In my code, #3 is probably the biggest hurdle, and it does take significant effort to work around it.
Support SETI@home
Seems to me that there are several issues here:
1. Making code available. What do we mean by "code"? Do we mean 100K lines of f77 that's been hacked for 30 years? Or do we mean R, SAS, or Mathematica code? The former is probably not so useful to others, though the latter may well be incredibly illuminating, even for non-experts in the specific field.
2. Making DATA available. Yes, in the leaked climate emails, there was some amateurish code. But the big issue there was actually sharing DATA, not code. In fact, emails indicated that there were threats of deleting data, and data that was never actually confirmed, hence a scandal.
3. Unqualified people using your data/code to attack your position. Seems to me that unqualified people are already attacking positions without data/code access, so it's not really a winning position to refuse to share. In fact, it's quite legitimate to suspect someone who is unwilling to share their data/method for arriving at a conclusion. The 1950's "according to scientists" simply doesn't fly anymore.
4. Unqualified wasting your time answering questions about your data/code. The open source movement has had to deal with this for quite some time. Not only that, the presentation of a question or objection: a) can often be judged by its own statement, and b) may be a common question that needs to be answered. In the first case, someone writing to a climatologist and saying "I downloaded your data and put it in excel and used that curve fitting thing and my curve shows temperatures peaking and going down" needs nothing beyond a canned reply, whether in public or private. (And a public discussion of why this is NOT a scientific objection would actually advance the overall state of education and science in the world.)
I'm not sure that 300K of crufty f77 code would be very useful to anyone to see. Though I'd also say that knowing someone's conclusions were based on 300K lines of crufty f77 code would be a point in the "not so sure" column. Which I think is much of the objection of releasing code: it takes guts to put not only your conclusions on the line, but also your assumptions and reasoning and most people are simply not willing to do this. Scientists (capital "S") should be willing to do this, but it would be pretty embarrassing to say, "My conclusions are based on a model that involves 400K lines of fortran code that has been tweaked for the last 30 years and which no one living actually understands. It seems to interpolate data very well, and we have reasons X, Y, and Z to believe that it extrapolates well also."
You may be right, but how many people are willing to say this? Easier to say, "Our proprietary model, developed by [Cue authoritative music] the most EMINENT SCIENTISTS on the PLANET, says A, B, and C," and then Appeal To Authority (tm).
It is already common knowledge among researches that simulated results can not be trusted. A program is only as good as it is tested. Only for scientific programs, testing is not usually as trivial as with application software. For example, you may have to analyze a system statistically and verify that the simulation results match the expected statistics. The scientific value is in that analysis and simulation results verified against such an analysis. While I welcome the idea of making the code used in scientific work public, I believe a much better reason to do so is that other researchers can improve on it. Of course, research is not always meant to be for the common good...
I believe that scientists should publish their algorithms and methods, but publishing code may be counterproductive for several reasons. Firstly, another group trying to replicate and/or verify the method should start from scratch to ensure their work doesn't simply import flaws from the original. Secondy, I don't believe it is possible to debug science code to the point where it is defect free - people keep debugging only until the results agree with their intuition. Thirdly, scientists should not waste their lives sifting through thousands of line of Fortran written by long-gone grad students hoping to find errors, they should be creating and investigating new stuff. Namgge
This is hugely worrying when you realise that just one error -- just one -- will usually invalidate a computer program.
If you believe that, you are forced to concluded that most likely no valid program has ever been written in the history of the human race. All programs have bugs in them (yes, I truly mean all), and in most cases those bugs do not invalidate all their results.
Don't misunderstand me. I definitely support releasing the source code for scientific programs, and I believe that finding bugs in them will ultimately lead to better science. But nothing useful is achieved by absurd hyperbole like the quote above.
"I'm too busy to research this and form an educated opinion, but I do have time to tell everyone my uninformed opinion."
From reading the actual paper, the analysis of source code is based on automatic static analysis, NOT on humans actually reading the code. Static analysis software can generate a high amount of false positives, so knowing that lots of software triggers the static analysis criteria for a fault doesn't necessarily mean anything at all about number of bugs or quality of the software. It does mildly suggest that there might be a problem, though.
The second part of the paper is about comparing different commercial software for doing calculations on seismic data. The results of this is that even though the operations that these software systems are supposed to carry out are completely mathematically stringently defined algorithms, the answers you get in the end differ wildly, and in some cases the different packages agree on only 1 figure! There is no one package that seems to generate better output than the others. As the paper states, this could result in drilling a 20 million dollar oil well in the wrong place, since up to three significant figures can be needed for that purpose.
To sum up, this is absolutely scandalous, and there is no reason to suspect that software for seismic calculations is any worse than for lots of other areas.
Who's gonna wager their hard-earned that this is a tactic by the climate denial industry to open up another avenue to discredit researchers?
And how long until idiots like Monckton miraculously become expert software engineers overnight, then start ripping climate researcher new assholes because their computer models contained logic errors?
The article cited for the error rate in scientific programs is from 1997, on research done in 1990 & 1994. Programming has changed since then, for example the "Fortran" examined was Fortran 77, rather than Fortran 95/2003 which provides interface checking. This isn't to say that all scientists are good programmers.
I did always wonder what would happen if we feed random data into these proggies they use to calculate temperature anomalies. Would they still produce the hockey stick?
Not so anonymous: I am Darrel Ince who wrote the article. It has generated a lot of reaction. It is a really tricky subject which 800 words in an article does no justice to. A major point is that there are a number of valid reasons why someone will refuse to divulge their code which seems fine by me: commercial agreements, the issue of intelelctual property rights (something which clouded the whole issue of the hockey stick code). Also there is the problem that developing computer code is tough and a scientist should be allowed time to develop publications from the work; Steve Scneider of MIT has suggested a two year moratorium.
There is also the problem of a vexatious demands. As far as I am concerned if you have released the code and it is complete then that is it. The recipient is on their own. The computer is chaning so much about science that one of the good things that has come out of the CRU incident is that it has highlighted it and the work that fourth paradigm people are doing.
Darrel Ince
What brought the US to its knees in Vietnam was the fact that the Vietnamese knew eventually we would have to leave and they would still be there. The cost (not just monetary costs either) were too great for US to stay. The same thing is happening in Afghanistan.
Having looked at some of these "research" codes I was astounded to see "PI=3.14" in one. I got access to this bit code because a parallel version would differ in the 19th digit of an IEEE float as the number of processors changed and I was supposed to fix the machine.
Truth is stranger than fiction, but it is because Fiction is obliged to stick to possibilities; Truth isn't. Mark Twain.