Slashdot Mirror
Ask Slashdot: How To Encourage Better Research Software?
An anonymous reader writes "There is a huge amount of largely overlapping but often incompatible medical imaging research software — funded by the US taxpayer (i.e. NITRC or I Do Imaging). I imagine the situation may be similar in other fields, but it is pronounced here because of the glut of NIH funding. One reason is historical: most of the well-funded, big, software-producing labs/centers have been running for 20 or more years, since long before the advent of git, hg, and related sites promoting efficient code review and exchange; so they have established codebases. Another reason is probably territorialism and politics. As a taxpayer, this situation seems wasteful. It's great that the software is being released at all, but the duplication of effort means quality is much lower than it could be given the large number of people involved (easily in the thousands, just counting a few developer mailing list subscriptions). No one seems to ask: why are we funding X different packages that do 80% of the same things, but none of them well?"
How To Encourage Better Research Software?
Use serious grant money to pay for it - even if it's paying students at a university.
"No one seems to ask: why are we funding X different packages that do 80% of the same things, but none of them well?"
When I think about this, I'd rather have that than one single package, if only for the reason that without competition, I'd not be able to know if it was doing anything well or not.
Pragmatism here says plurality is probably better than some kind of Stalinist central control.
"And the meaning of words; when they cease to function; when will it start worrying you?"
Not only that most researchers are not proficient in programming language, they shape their codes more like prototypes so that they can modify the codes easily as the science progress. Conventional programmers will be frustrated with this approach since they want every single spec set in stone, which will never happen in research setting since research progresses very rapidly and specs can change dramatically in most cases. If you can set the spec in stone, it is usually a sign that the field has matured and is getting transitioned to engineering-type problems. Once the transition happens, it's no longer research, it's engineering. Then you can "make the code better".
--
Error 500: Internal sig error
Git and Mercurial (hg) are not sites, they're programs. They have nothing to do with code review. You could say that they do promote "efficient code exchange", but so does any other VCS. Are you seriously trying to tell us that these big labs are not using version control while developing their systems?
It's probably a good thing that there's at least two different groups working on the same thing. Competition creates incentives for those within it to write better code so that it's more widely adopted and they get more funding. Why do we have Chrome and Firefox?
This happens in private companies too. I heard a story about a private company that hired two different offshore contractors to write the same software independently of one another -- they were on a tight deadline and had actually read the Mythical Man Month. So when one of them was terribly buggy by deadline, they had something that worked.
I've written software for too long to believe that any one approach works for everyone, and that by not putting your eggs in one basket, the investor (in this case, the American people) comes out ahead.
-- Political fascism requires a Fuhrer.
This problem is widespread in almost every discipline which uses any form of computation. I think the best way is for major funding sources like the NIH, NSF etc to build in to the grant terms which coding language, existing libraries be used. Or how/what/ software will be developed should be used an additional metric for deciding which proposals to accept. Proposals which are strong otherwise but do not state in clear terms how software will be built should be asked to modify their proposals to include such information. Pre-existing, well-designed, modular software architectures should be extended rather than building architectures from scratch. This is a waste of funds and time. Funding organizations must also recognize that developing good software takes time and money and set aside budgets in the grant for hiring dedicated programmers. (Scientists are very often not good software engineers and they are interested rather in trying things out quickly to see if it works at all) Such programmers can then take hacky research code from the scientists and turn it around into great reusable code.
If you really were in a position where to make a positive impact in this area you would surely be in the position to also direct one of the existing projects so that they would do better and force the other guy(s) to step up their game.
As a Ph.D. candidate who writes scientific software at a large research university in the US under NIH grant funds, I can say that simply adding more developers to a scientific software project is an unrealistic solution to the problem. Having been to a conference of developers for a well-known chemistry software package (which will remain nameless) I have seen firsthand how seemingly good intentions can quickly turn into an epic battle of conquest and control of the software. Add in the huge egos and arrogance of these scientists (some very famous in their field), and you wind up with software that no one wants to develop due to problems that have nothing to do with funding or lack of qualified developers. This is probably one of the main reasons new scientific software is created in the first place.
A surprising number of researchers work around this restriction and keep the software proprietary (or at least secret) by contracting the software out and purchasing outside services.
Even when the software is public domain, there is no uniform requirement to make is openly available. Often you have to write to the principle investigator and after delay and obfuscation you may get some undocumented compiled code but no source or incomplete source.
The government should adopt an open source software policy which stated that software created by the government must be put in a public repository (along with documentation) and this should be verified and enforced.
This would help build a government open source ecosystem where researchers can build on the ideas of other researchers.
I don't read your sig. Why are you reading mine?
The trouble is that you don't get grants for software development. You get them for original research, i.e. novelty. All you need to publish a paper is a hackish implementation that works once. After that's done, there's no reward for improving your code and iterating further. If you're trying to stay competitive, you move on to the next thing.
Developing good software is for industry to do. Unlike academia, industry can get massive rewards for making a well-implemented toolkit. No academically-developed software will be able to compete with that in the long run.
The other guy's stuff simply won't do for serious, cutting edge science.
I'm a computer scientist in the middle of getting my BA, but for research experience or in the process of taking an elective, I've spent time with grad students in other departments- mostly biology and linguistics- and the software they write. Smart people? Absolutely- they're experts in their field. But they can't write code to save their lives. I've seen things that make me want to run screaming to TheDailyWTF and the quality software engineering on display there ;)
I don't think this is a bad thing, myself. Most of this code is single-use only, being written for a specific purpose (or a specific thesis paper), and will never be used again. Not to mention they're taking enough time to get their degrees as it is- I don't think it's reasonable to ask them to become expert software engineers as well. OP claims that taxpayer dollars are being wasted, but think how much waste there'd be if every researcher had to get a CS degree before they started in their own field, too.
Dislike the Electoral College? Lobby your state to join the National Popular Vote Interstate Compact.
Requirements are the hardest part of software.
Imagine two rooms, right next door to one another. Both have quite large whiteboards in them. In the left room, a group has drawn an urban scene with buildings, cars, people walking on the street, etc. Now, imagine that a group in the right room is attempting to duplicate the design on their whiteboard, but the only method of communication between the two is plain old text messages that don't contain any graphics or URLs.
The task we've described above is easier than getting the concept of software from one brain to many others, even with all the diagrams, charts, code, and tools we can bring to bear.
Now you should understand the answer to your question.
If you're not happy with what's out there, you need to roll your own. If what's out there is open source, you can pick the best of each of them and build the solid system you're looking for. With research projects, once the stated goal has been reached they are done - until a follow-up grant for further work is awarded. That seems to be what research is about - showing that things can be done or done a different way - not producing a useful software product. Once they show what and how, it's up to someone else to take that and make something great from all the pieces. Unfortunately that means sifting through all the duplicate stuff and finding the best approach and possibly reimplementing it to fit in with everything else you're doing.
For example, you may find Kalman filters, genetic algorithms, neural networks, GPU implementations, etc. all able to solve a particular problem. For real-world software you really don't care about all that, you just want the ONE that works best in your application. Of course then there will be papers on "extensible frameworks" with "plugins" that can handle any of those implementations... Again, for real software you pick the one that works "best" for your definition of best and go with that. To make this happen, you need to get an ego-less (read non-PhD) software team to pull it all together.
The article description sounds like a perfect description of the state of all the linux distro's, all the linux desktop managers, and all the linux word processors. That is, there is a proliferation of not quite compatible products that do 80% of the job well.
So I guess the article is saying we should take this shining example from computer engineering and use it to refor how scientific packages are developed.
Wow. glass houses much?
Some drink at the fountain of knowledge. Others just gargle.
... wasted effort, but the allocation of the money and the people involved.
It is not all that hard to create very good, powerful and even big applications, but it becomes a hell of a lot harder if you throw tons money and people at it.
And yes I have worked in university physiology and they have horrendous software as well, but there is simply no other alternatives and very few real programmers working in the field, so no one who could fill the hole knows about it.
Troll is not a replacement for I disagree.
Competing implementations and competition leads to better code, and a status quo that constantly moves forward.
You are arguing FOR government-gifted monopolies in health care?
You astroturfing shill. I hope your little "medical records" startup fails, you jackass.
There is a legend that this is what happens at Intel and Microsoft. It used to be said that every odd numbered Intel was not much of an improvement. It's still true since Windows 1.0 that every other release of windows has sucked. It was perfectly predictable that Vista would tank. (No I don't hate microsoft. Even people that love microsoft can see this has become a "law".)
In both cases the supposed explanation is that there are two difffenent teams working at the same time. The better one gets the first release and second one patches their changes into it for the sucky intervening release.
No idea if that is true in practice.
Some drink at the fountain of knowledge. Others just gargle.
Having worked on a few of the imaging packages you refer to, I can tell you from first hand experience that the NIH does not give any incentive to write good code, nor to share with other researchers. The NIH awards money based on research publications. Most of the software produced by these labs was written by grad students who's only incentive was to publish, and then graduate. Their code is usually not of the highest quality, and they have no incentive to make it better, nor share with anyone else.
There are a few exceptions, people who 'get it'. Have a look at the ITK effort (http://www.itk.org), Slicer (http://www.slicer.org) and the CTK (http://www.commontk.org/index.php/Main_Page) effort. It is rare to find a research lab willing to use other's software...
Cheers
I track a lot of scientific software on Freshmeat. You'd be amazed at the redundancy. Medical stuff isn't as bad as some areas.
It's a small world and it smells funny; I'd buy another if it wasn't for the money; Take back what I paid (SoM)
Although I admit, some of it's the 'not invented here' problem, one of the big reasons there isn't better collaboration is that most scientists don't know that someone's working on something similar.
I deal with software that works on FITS files. The two main fields that use FITS -- medical imaging and astronomy. Do you think the two collaborate? Hell no.
And even if you do find out about some great new tool ... it's after it's been released. Which might be two or more years after they started development, so you've been duplicating work for all of that time.
I'm of the opinion that we need some sort of a cross-discipline registry for science software. List all of the stuff that can make plots ... what formats they read, what file formats they can export to, what special features they have, etc. Let me search for all of the software that can read NetCDF files ... or rice-compressed FITS, etc.
And then we need to get people to list their software (and most would, because if people use their software, and give acknowledgements in peer-reviewed papers, they can use that to justify continued funding), and in the future, we need people to list projects striving to make new software, so we know about it, and rather than expend effort to make something new, we can contribute to a project working on something similar.
Build it, and they will come^Hplain.
1. A center grant. These are understandably difficult to get, and typically require some venerable central Dumbledore (preferably with nobel), but they get around the insightful: "Science wants novelty, not quality" comment. That is, a center grant is designed to allow money and publications to flow for development without overt novelty.
2. Keep the software modular, well-documented, and open. Publish everything about every interface, file format, and provide a dazzling array of examples and tutorials. As an excellent example of this see the CCP4 suite of software http://en.wikipedia.org/wiki/Collaborative_Computational_Project_Number_4/
As other commenter have sensibly pointed out the way good software is typically produced these days requires a large commercial effort; but this represents the current path of least resistance, and not historical impossibility. Good academic software, albeit rare, is often the most amazing in scope and design.
As a PhD student in bioinformatics/computational biology I can tell you first hand that the grant system we work under is what causes the problem.
The vast majority of this type of software is written by grad students and our goal is not to make software, it is to get results so we can write papers. Our PI's encourage this approach. I have been told point-blank "...we are not working on tools, we are working on methods. Tools are the result of that." Which, of course, anyone who has worked in software engineering knows is complete crap. So anyone looking for a tool for a particular purpose in their field is likely to find crap that is difficult to use, has outdated dependencies, no documentation, etc. To the point where it is easier for them to write their own.
Until the effort to develop usable tools, document them, and publish them, is recognized and funded by the grants that pay for the work to begin with, the current situation will continue to hold and probably get worse.
It's a simple problem, research software are either written by scientists that don't program or programmers who don't understand the science. So either you end up with a powerful and technically correct software that has an interface that is completely cryptic, confusing and generally unusable or you get a nice glossy looking software that doesn't do what it's supposed to do.
It's really hard to find people who can do both.
Don't fund the authors. If you must fund something, fund buyers. Let the market create the product; purchasers will quickly and accurately weed out the crap.
The high cost of scientific software and its lack of accommodation to what scientists really need were the reason we came up with Sparklix electronic lab notebook and its business model.
Apparently a lot of today's scientific software is developed by engineers who know nothing about the scientific areas they are targeting, trying to create yet another CRUD or CRM-like application with scientific flavor. What we attempted to do with Sparklix is to bring the researchers an experience which would be as close as possible to their paper notebook, because this is the way they are thinking and this is how they expect their software would work. To get to good results our team is composed of both developers and scientists, and this makes the software fit the actual need and not remain in "thoughtland".
Regarding cost we made it simple - the product is free, and additional services (such as on-site installations) cost money. This is the direction the internet is going to, we see no reason in charging money for the basic software in today's world of Google, Facebook and Twitter.
Regarding FITS (Flexible Image Transport System), if this is used in significant ways in medical imaging, the astronomical FITS user community would love to know about it and collaborate. Regarding rice-compressed FITS, I (and undoubtedly my coauthors) would be beyond fascinated to learn of either medical imaging use cases or compression tools for this purpose. Alternately, any FITS-based medical imaging applications should be aware of the astronomical data compression work accessible through http://heasarc.nasa.gov/fitsio/fpack (hopefully I'm not slashdotting myself :-)
Another field planning to use FITS is digital manuscript archiving per the Vatican ( http://bit.ly/aagZxN ).
Regarding the topic of this thread, the comments here emphasize that the real issue is standardizing on data formats. The richer the community (and none are richer than health and medicine), the richer the software ecosystem.
Medical imaging is a special case.
The FDA has a "in-house" exemption for things like software-based medical solutions (for example, the software that calculates the best way to deliver a radiation blast to your tumor, or the software that identifies tumors in MRI results, or whatever). In essence, to share software you've developed, you have to go through a lengthy and expensive approval process. Once you've written something, no matter how nice it is, there's a huge threshold of liability, expense, and hassle you have to overcome in order to be legally allowed to give it to other institutions.
Eliminating duplication of effort sounds good but it would be a total disaster.
The great advantage of the American research establishment is that there is no central authority deciding what to research. There are many different funding sources and many different establishments. Central authorities which decide on the "right" approach to pursue have no place in science --- it cannot be done by consensus.
Perhaps NIH itself knows how to produce good open source imaging software with tons of plugins. Try ImageJ, from http://rsbweb.nih.gov/ij/
It's kind of useless to post this on Slashdot honestly. What good is it going to do? If you have an idea for how to solve this and are a researcher then talk to the NIH. Otherwise this is just a lot of hot air.
If the software is mature enough to be widely useful, then a company should try to commercialize it under and SBIR or an STTR or something. If it's not mature then it should stay what is is - a research prototype. Most research prototypes end up being useless in my experience and it's not worth the effort at all the try to pull them all together. If there are a bunch of useful ones like you seem to be claiming, then submit a grant and do something about it.
Drop whatever you are doing right now and start writing a grant proposal. Here is what you will propose to do:
(While common revision management applications and schemes allow only a hierarchical, "branching," structure for the repositories, this repository will likely require the ability to track a very complex "graph" of the evolution of software and all the ways that researchers mix, combine, and improve the software.)
If you play your cards right you could end up with a lifetime career managing this clearinghouse. Why am I not writing this proposal? Because I am working on something else which I consider to be even more important.
I would like to see the statement "why are we funding X different packages that do 80% of the same things, but none of them well". We have X different packages because they do what they were written to do very well. That 80% overlap is applying mostly the same standard processing tools. One tool doesn't fit all.
In fact there are several well-designed user-extensible medical image processing frameworks available already. ImageJ, MIPAV, and ITK were funded by the NIH and fill the very void suggested by the OP. Many more mature medical imaging tools that serve a variety of niches are freely available, many of which include free source code.
Frankly, I think the OP's main thesis is fundamentally wrong. Medical imaging research is about inventing or improving IP techniques and algorithms, not implementing and distributing software tools. Asking researchers to deliver more than a design or perhaps benchmark results would be counterproductive and a poor use of research funds. If better software tools are the goal, then some more constructive questions might be: Who best should manage such an effort? Who should fund it? And how could we fund and coordinate such endeavors better?
Personally, I'd like to see, as part of any publication, the software, data, and runtime parameters be part of the submission. "Unreproducible research considered harmful", should be the new maxim. But I digress.
IMHO, the current state of gov't funded medical imaging research tools is doing quite nicely, thank you. If the OP really does in fact know a better way, then he should write up his grand plan and submit a grant proposal of his own.
this is the same crap you get on quora where OO freaks question why Fortran is used rather than C++ for a lot of technical programming.
yay, a few thousand medstudents / medical engineers with a interest in programming contributing to a common code base. The thing would become a monolithic mess inside of a week, and every new developer could wast 12 months learning how the x million lines of code is cobbled together. Every new addition will then destroy 5 things that used to work....