Slashdot Mirror
Free Software, a Matter of Life and Death
ChiefMonkeyGrinder writes "Software on medical implants is not open to scrutiny by regulatory bodies. Glyn Moody writes: 'Software with the ability to harm as well as help us in the physical world needs to be open to scrutiny to minimise safety issues. Medical devices may be the most extreme manifestation of this, but with the move of embedded software into planes, cars and other large and not-so-large devices with potentially lethal side-effects, the need to inspect software there too becomes increasingly urgent.' A new report 'Killed by Code: Software Transparency in Implantable Medical Devices' from the Software Freedom Law Center points out that, as patients grow more reliant on computerized devices, the dependability of software is a life-or-death issue. 'The need to address software vulnerability is especially pressing for Implantable Medical Devices, which are commonly used by millions of patients to treat chronic heart conditions, epilepsy, diabetes, obesity, and even depression.' Will making the source code free to scrutiny address the issue of faulty devices?"
That the Pacemaker Genuine Advantage warning I got last week was a bit of a shock...
Dupe! This was covered a couple of days ago.
Custom electronics and digital signage for your business: www.evcircuits.com
To me, this is just common sense. This code doesn't necessarily have to be FL/OSS in my mind - let them keep the copyright, but it most definitely should have code available for public review. Would you be willing to take a new wonderdrug where the drug company won't tell anyone what's actually in it, but assures you that it'll work? If they must disclose the formula to their drugs, then they ought to be required to disclose the code to their software. Let existing laws like copyright ensure that no one else uses it.
"People who think they know everything are very annoying to those of us who do."-Mark Twain
I'm a big fan of FOSS, but I've got my QA methodology hat on right now. Opening up the source code isn't providing better Quality Assurance coverage, it's just getting more eyes on the code, and at best a bunch of User Acceptance Testing. (Though clearly not with pace makers, I doubt people are going to line up for that Beta test. Unless maybe Blizzard claims it's part of their next big MMO.) This could be as much an argument for higher standards of quality assurance as for open source software. In face, hell, I can see companies opening up the source to reduce their liability and cut the costs of their own QA.
I don't think that the notion is that all medical code is going to be written by happy-go-lucky FOSS volunteers, the notion is that people ought to be able to inspect the code that is going to becoming a part of their life-critical systems...
Even the best software can go completely wrong with the wrong person operating it.
Taxation is legalized theft, no more, no less.
Making the code available for scrutiny is a double-edged sword. Sure, it might help to catch critical bugs in the software faster. But with some devices, it really raises a host of new issues. Some pacemakers out there are configurable wirelessly once they are in the patient's body. This is actually a very critical feature. But do you want to risk everyone being able to reverse-engineer the protocol used for adjusting the settings for such a device? A wrongly configured pacemaker can be deadly. Right now these things are fairly secure because they are rather rare and not interesting enough as targets for hacking.
Besides, proving that some piece of code works as intended (or at least fails gracefully in all circumstances) in an essentially uncontrolled environment is quite a feat. Embedded equipment is hard to service, has to have a longer hardware lifespan than normal hardware, is often custom designed for a single application and thus may have subtle hardware defects not reproducible on similar test systems... oh, and who says that the compiler or even the CPU is bug free? This is all common knowledge around here, but I just want to give the full list. What this means is that just opening the source code might not cut it. The validation would have to be performed on the hardware it was designed to run. So, where's the call to open up the hardware design and documentation to public scrutiny as well?
http://www.moonlight3d.eu/
Really. Finding security holes in software that runs on a plain vanilla PC is one thing, finding the cause of glitches in the nanosecond range on an embedded system is another thing entirely.
The Therac-20 radiation therapy device worked reasonably well. Despite the software flaws, the hardware safeties in place prevented any deadly accidents. Problem is, because of the hardware safeties, nobody knew just how bad the software was. It had never been formally verified.
Then some numbskull decided, "Hey, let's let the software handle the safety interlocking, and we can cut down on hardware manufacturing costs!" The result was the Therac-25, which maimed and killed people.
After the machine was recalled, someone finally sat down and did a real analysis of the code, and found a whole raft of problems and bad assumptions. Nancy Leveson wrote the definitive report (PDF) on the failures in the R&D processes that made the Therac-25 so deadly.
Yet, armed with this warning (among many others), both manufacturers and purchasers keep human lives as transactions on a double-entry ledger. It simply comes down to, how many deaths per thousand uses are "acceptable"? Manufacturers and medical facilities already have so many costs. Is it worth it to add on the cost of formal code analysis?
But nobody will ask the Therac-25 victims and their families.
I decided early on in my I.T. career, that I didn't want the stress of people's lives depending on my correct code. I hadn't had any training in formal verification. In hindsight, I see my worries would have come from incompetent management, more than from myself.
Some mechanical devices and most bridges and buildings require licensed engineers or architects to put their stamp of approval on the designs. They do not require publication of the engineering or architectural drawings though.
I for one would welcome professional licensing for certain "it can kill you if it goes wrong" software, particularly in isolated devices whose software can't be tampered with undetectably.
If a licensed Professional Software Engineer puts his seal on a pacemaker or airplane, and the software kills someone, he's just as responsible as the civil engineer would be if a faulty bridge design kills someone. In both cases, the licensed professional's responsibility would come back to "was the engineer acting in accordance with professional standards at the time" and "was the device built and maintained in accordance with the design."
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Formal methods on their own are not enough -- at least, not with the current state of formal methods. Formal methods and testing tend to expose different bugs. But the principle is right: maybe an independent safety assessor evaluates the process and products, and the manufacturer submits their argument as to why the system is acceptably safe to a regulator.
We need to be careful about what is "sufficiently safe" though. If somebody would die for sure without the implant then "the implant probably won't kill them" is a big improvement, whereas achieving "the implant almost certainly won't kill them" might price the implant out of reach of most people who need it so it goes back to the situation in which they die. As a rule of thumb, moving up one IEC61508 SIL increases costs by about an order of magnitude. Formal proofs mean that you're talking about SIL 4, so you're talking of the order of 10 000 times the cost of normal commercial standard software (treating that as SIL 0). Increase the development cost of a life-saving implant by a factor of 10 000 and unless you have massive economies of scale you're going to end up indirectly killing people by pricing it out of the market.
Quidnam Latine loqui modo coepi?
^THIS
Implantable pacing devices, cardioverters, and pumps (life-sustaining devices) depend on complex custom hardware designs as their platform, and that hardware is *highly* interactive with the software. Many of these devices can only achieve their miraculous longevities on a primary cell by deferring functions to hardware. If you don't have access to the information re: the hardware, the code itself might as well be inscriptions in Atlantean glyphs. You'd have to bust trade-secret protection to get a public viewing of everything needed to review the code, because you'd have to see, *everything*.
"Well, do you want your pacemaker to have intuitive manageability through Group Policies, or not?"
No, just a good SNMP MIB and traps.
I was joking with an ex-sys-admin friend of mine who just underwent open heart valve replacement, and commenting on the idea that the wireless ECG gear needed SNMP to alert when it detected a loose lead. She laughed - not good when you were doing a good impression of an Aztec sacrifice just a couple of days previously.
www.eFax.com are spammers
Look, if you don't like the idea of implanted medical devices controlled by software, just say so.