Slashdot Mirror
Software Quality In a Non-Software Company?
Nicros writes "I work for a publicly traded biotech company that happens to write software that is, in fact, kind of critical for the business — without it no data would ever be read from our instruments, and no analyses would be performed on that data. The problem is that as a 'biotech' company, we are not taking software quality seriously. We have no senior management with any history of commercial software development — our C level has really no clue whatsoever what software really is, much less what is going on in software development. All of our quality processes are related to manufacturing our system (not software), so we are constantly forced into ad-hoc development since there is no real process for our development. Repeated requests to hire someone with some real commercial software development experience have gone unanswered. I have been to the CEO directly one-on-one and although he agreed this was an issue, thanked me, and said he would look into it, that was the end of it. He has bigger things to worry about. So the question: Is this just a fact of life and I need to deal the best I can? What else can I do to get some attention on software quality in the company?"
We have no senior management with any history of commercial software development
That reminds me of Arianespace. It took the crash of a 150M$ rocket to make them change that.
Management are possibly right, the important thing is getting the product to market. If the R&D people write bad code, but code that works, and it gets the instrument to market then ship it. If it's instrument based, the software isn't the critical problem (if it works better than the other guys you win, doesn't matter if the primary data analysis software sucks so long as it more of less works).
However, you should try and convince you're management to open source the software. In this industry the probability is that if you don't open source it someone else will write an open source replacement (see Phred/Phrap, and the open source replacements of the primary data analysis software on next-gen sequencers which are starting to appear). That means your company losses control of the primary data analysis and possibly device control software, and that's bad for your company.
Open source has the added benefit that your development costs will fall (you can start using GPL code), it'll help you engage with the scientific community and you'll get people outside the company doing free work for you (seriously people want to get this stuff working, they'll help). You'll also get free peer review on your code which will drive standards up.
Scared of showing your crap code? Don't be, in this industry I've seen enough to know that most of it sucks (a lot of it's written by Biologists with no formal training). The clincher? "ABI are doing it, why can't we!" http://solidsoftwaretools.com/gf/
If you are in the bio tech field then all of your processes need to conform to ISO13485. There is a section specifically about software. Your company won't be in an FDA/CE regulated environment long unless you comply with those quality standards. I suggest you research the guidelines and point them out to your quality manager.
The reason we subjugate ourselves to law is to better procure justice. If law does not accomplish this purpose then it m
Excellent point, if you present yourself as the man with the solutions, and especially if they promote you as such, you're going to be taking the heat when things do not go perfectly.
The thing about rigid development and testing processes is that they delay releases. You can do featureful but buggy from-the-hip releases rapidly, or you can do rock solid heavily tested releases very slowly.
It's the old speed, quality, cost diumverate. Sounds like today they're choosing speed and cost, and quality is there only because your product isn't yet so mature that regression bugs aren't common.
Companies starting a development methodology need time to adjust to the reduced agility that these require. It's best to work your way into it.
Start with introducing source control and some formal testing, along with build releases. It's not very likely that they're going to reject you on these things as their benefits are fairly clear and they shouldn't have that much impact on your bottom line. Create a branch per release, and suggest that the software not be released to manufacturing until that release passes formal testing.
Later you can introduce things like regression testing and a proper software development lifecycle.
You have to be careful how you present it. Make sure they're aware that you're easing the company into a larger methodology, and until fully implemented there's still going to be gaps for bugs to fit through. Explain that during this process you'll be performing gap analysis on each bug that does appear, identifying how it got through testing, and adjusting the process as necessary to ensure that such a thing is less likely in the future.
You don't want to present it as a silver bullet out the gate, you want to be sure to explain that it's iterative and the objective isn't immediate perfection but continual quality increase. Even if you have 30 years of software development lifecycle managerial experience, you can't convert a company to a full-blown lifecycle overnight, and depending on how a company works and what its needs are, lifecycles will in fact differ a bit from company to company. Even if a perfect lifecycle existed, you still need to give people time to get accustomed to it, so you can't just throw the switch over night or everything comes to a halt.
Slay a dragon... over lunch!
I've been a research scientist for, well, long enough and a computer nerd for a lot longer. I've always wondered how it is that a company can make a state-of-the-art piece of research equipment and then bundle it with a PC running Windows 95 with a serial interface and the worst, least intuitive, and most expensive software I have ever seen.
Now that I'm in charge of picking what we buy I make it a point to find companies that support Linux because it usually means they have a real software development team and that they don't outsource their development. (Jovin Yvon who bought up all their competitors were the worst at this, charging $5,000 to re-install their fluorimeter software, the installation CD for which they refused to sell based on "licensing" issues.). And I'm usually right.
Most recently I purchased a $70,000 high-speed InGaAs camera that came with Windows-only software (that wouldn't run in virtual machine either because it required low-level NIC access). They charged another $2,500 for the "intermediate" software package (which I have no problem with in itself) that had a bug in it. The bug? It wouldn't export movies longer than 10,000 frames, which at 1,000 fps is 10 seconds of video. It took them three months to get me a beta version of a different piece of software that would allow me to export longer movies.
Another company, which I love doing business with so I'll mention them here, EPIX Inc., makes less expensive high speed cameras, but develops their software in Java and releases Linux versions. The software is buggy--as is all instrument software--but I can actually call a real software developer on the phone and tell him/her what the problem is--imagine that!
Sorry for the rant, but this story makes it clear to me why instrument software is so terrible. The first major company that figures out how valuable intuitive, functional, cross-platform, (and for the love of god, not hardware-keyed) software that doesn't store data in some unholy proprietary format that physically ties people to the machine that the instrument is attached to just to process the data is... Ok, well the few companies (I'm looking at you Bruker) that have figured that out have staying power and brand loyalty in university research where as the others are used as pejoratives.
Actually, I wrote my thesis on life experience.
Let me tell you a little anecdote which was relayed to me by the IT manager at a previous employer.
When he came to the company, software quality had never been taken particularly seriously. They'd insourced IT where previously everything was handled by an outside company, presumably in the hope of getting better quality services for their money, but were seeing little benefit - mainly because the IT department was so busy implementing new features the business wanted they never had time to debug existing issues.
Helpdesk call levels were very high, the IT department wasn't particularly highly respected in the rest of the business and while the business probably did want less buggy software, they were always too busy chasing after the Next Big Feature to allow the IT department to concentrate on bugfixing.
So he went to the business (ie. the directors/senior management) and said "OK, here's a suggestion. We'll spend the next three months working on nothing but bugfixes. No new features. What glitches with the system are impacting your staff?". The business wasn't hugely keen on the idea of no new features for three months, but he was able to persuade them that the benefits of having more stable software outweighed this.
Three months later, the business was so impressed with the improvements that they asked if the IT department could spend another three months doing nothing but bugfixes.
Sometimes, the business needs a little poke from IT to understand how to get the best benefit from the IT department. Being able to recognise this and make a suitably diplomatic poke is what IT management is there for. If there isn't clear IT management in place to make such a poke - well volunteered.
In most companies where software is not the main focus, the deadlines are for the main product and the software had better be done by that deadline. The software guys have nothing to say about the schedule.
I saw this effect first-hand at Digital Equipment Corporation. Although we did a lot of software, the corporate culture focused on hardware, so the software had to be ready when the hardware shipped. Fortunately, the software guys were generally able to get a prototype far enough in advance of "first customer ship" that it was possible to get the drivers written. From a software point of view the hardware documentation was sometimes lacking: once I recruited the assistance of an in-house field service guy to help me read the engineering drawings so I could figure out how to program a new device.
Rarely does the statement "You've got a problem and you need to solve it" ever get a good response.
If you say "We have a problem, this is how other people solve it, and this is what I will need to solve it. Give me the budget and I'll solve the problem." then you are vastly more likely to get what you want. Then you'll have to deliver, but if you are like me (not that I am the best way to be), you'll find the responsibility gratifying and the deadline invigorating.
FATMOUSE + YOU = FATMOUSE
Your explanation of your product/service sounds like a medical device. Assuming that is true, your company is surely registered with the FDA and is audited by them every two years or so.
The comments in this list about federal law requiring a quality system *including software quality procedures* are correct. There is no way out of this and the company has a tiger asleep in engineering. The reason the omission has not surfaced is that FDA's budget has prevented them from auditing deeply enough - yet. They haven't been able to send auditors with enough software background to be able to detect the absence of the expected levels of software QA. They definitely have the qualified people, just not enough of them.
An additional reason could be that the product/service has not hurt anyone, or if so, the incident(s) have not been reported - which is another federal law incumbent on the manufacturer AND the hospital/clinic/doctor. FDA audits and warnings can come any time if enough of these reports stack up. Or if the docs send them in and the company does not.
Even if the code is really good and no medical problems have come up, that will not stop FDA from pulling your product off the market if they find you non-compliant with their regulations.
So the company has 'enjoyed' a prototyping phase. Once the management has read the FDA regulations on the personal liability of the company officers, they will probably want to get started with the formal software QA system right away. It doesn't have to be completed overnight. But when/if FDA look deeply enough into your company, you would want them to find records of your diligent work in building up the software QA procedures and practice in an ongoing and steady way. And doing the right things first.
SLIDE A:
1) We create software.
2) Software is used in medical devices.
3) We forego QA and industry best practices for software development.
4) Something goes wrong.
5) We get sued AND we lose (to the five-nines sure).
6) Change #1 to "Update resume".
SLIDE B
1) We create software.
2) Software is used in medical devices.
3) We follow industry best practices for software development and have a solid QA program.
4) Something goes wrong (yes, it still happens).
5) We get sued.
6) Our controls and best practices are a reasonable defense.
I use irony whenever I can, but my shirts are still wrinkled...
One route to making people serious about IT processes is to relate it to relevant federal regulations.
For example, we've been doing work that will eventually involve us as a partner in upcoming clinical trials. There's a bunch of federal regulations about IT processes connected to clinical trials, and it has been easy to get management to accept that while our current processes can be as ad-hoc as we like, at some point having compliant processes will be essential to continuing the work we do, so we may as well get it right the first time rather than have to reimplement years worth of ad-hoc development somewhere down the track.
In my case, one trivial example has been being able to implement gpg signing of documents as a consequence of setting up the infrastructure to be able to be quickly compliant with 21 CFR 11, which we'd need to do if we're part of a clinical trial.