Slashdot Mirror
Software Quality In a Non-Software Company?
Nicros writes "I work for a publicly traded biotech company that happens to write software that is, in fact, kind of critical for the business — without it no data would ever be read from our instruments, and no analyses would be performed on that data. The problem is that as a 'biotech' company, we are not taking software quality seriously. We have no senior management with any history of commercial software development — our C level has really no clue whatsoever what software really is, much less what is going on in software development. All of our quality processes are related to manufacturing our system (not software), so we are constantly forced into ad-hoc development since there is no real process for our development. Repeated requests to hire someone with some real commercial software development experience have gone unanswered. I have been to the CEO directly one-on-one and although he agreed this was an issue, thanked me, and said he would look into it, that was the end of it. He has bigger things to worry about. So the question: Is this just a fact of life and I need to deal the best I can? What else can I do to get some attention on software quality in the company?"
You're obviously fighting an up-hill struggle. Going straight to the CEO is both a good and bad idea - if it works you'll get immediate affect, but it's likely to be ignored.
You need to argue this case as much as possible. If you're the developer, or in charge of development, enforce decent developmental practices and ensure your estimates include them. Err on the side of caution. Take an estimate and double it. Managers talk money, not standards, so you'll have to hit them where it hurts.
Otherwise, is there anything off-the-shelf that could alleviate some development?
ilovegeorgebush
It sounds like in your company there is no one doing this job. You've talked to the CEO. Get him to make you VP of software and tell him you'll solve the problem if he gives you responsibility.
Anarchy is an opportunity for the ambitious and unprincipled. Take it and make yourself software Czar.
echo -e 'global _start\n _start:\n mov eax, 2\n int 80h\n jmp _start' > a.asm; nasm a.asm -f elf; ld a.o -o a;
This is a common theme from many companies, especially the one I'm with right now. Companies just don't get it sometimes and they let things slip by them. The biggest issue is, should you care more about it then the Management above you? Should you burn yourself out? Let them worry about the botttom line, u should do whats best for yourself. And above all else, don't stress yourself out over it. Once you've learned to let go, you'll realize how many other things are more important in your life. Plus you can always switch jobs so no biggie!
Could you propose to hire a software test consultant for a day or two and let him point out serious quality issues (data integrity, security, correctness..)?
A serious, alarming report by an external software test professional could help reinforcing your requests?
I have the same problem where I work, the problem is I am the dev with real commercial experience; I just can't convince them that we need to do things in a manner that I would consider correct - it's all ad-hoc development and it's all driving me nuts.
The problem is, if our software doesn't work correctly, then the data we collect and process using it becomes screwed up, which is a major issue for the core business - data is our crown jewels.
My current solution to the problem is looking for a new job in a company that actually takes software development seriously. I just can't see any way of getting things here working the way I want. There wasn't even any revision control in place on the source code when I started.
The problem I'm finding is that the lack of structured development and design here is actually beginning to hurt me professionally: prospective employers, who have software development as a core aspect of their business, actually ask about this kind of thing. If you're looking to hire someone who takes their profession serious, for god's sake make sure they're actually going to be able to do their job - otherwise your company is just going to turn in to a blot on their CV.
Yeah, I had a sig once; I got bored of it.
Hi OP,
I'm a developer/Engineer for two biotech companies: one a small startup, with me being the only part-time employee. The other is a large DOD-backed institution. I can tell you that in the short time that I've been there, it has been a frustrating uphill battle to instill an Engineering/Developer mindset. While I firmly believe Scientists and Engineers seems to have a similar approach to work, it's interesting to see how passive the science-minded folks are towards hardware/software advancements. They are only concerned about how many protein cells it can accurately count, or whatever. There is no interest in what goes on 'behind the scenes', and consequently, what goes on to get there.
There are absolutely no Engineering controls in place at either Employer, and software development is as you said: made for the moment. Personally since I am the one-and-only, I find that I just have to do the best with what I have. I comment and doccument well, keep a code revision repository, and do my best(within reason) to make sure someone else can pick up where I left off.
It won't be my problem if/when the day comes I leave, but at least I'll be able to sleep at night.
*-PGP Please!-*
I'm in a quite similar situtation, and perhaps I can provide a few hints from what we're currently doing.
I'm working for a relatively well-known institute in academia (biotech field), with a group that among other research projects, also provides web-based services to the research community. Funding is partially tied to the operation of the services, so there is actually enough pressure to make sure that they work and work correctly at all times.
Still, until about a year ago, development was very ad-hoc, in a mix of languages, and with many "islands of knowledge", where some parts of the system were only known to one post-doc, and other parts could only be fixed by the group head (who, as they are, was usually busy with many other things...). After some hard times and near-misses, we started looking around for ways to improve our development.
I was quite attracted by the ideas of Agile, and I believe that they're a good fit to the kind of processes you find in science, as well as in software engineering. We initially had a professional Scrum coach come in and talk with us about software development practices, and then decided to apply Scrum to our processes.
It's now a bit more than 1 year since then, we're still using Scrum with a few adaptations to fit the academic environment (we're also using Scrum for projects that are really science and research, not software development). In a recent secret poll among the team, Scrum got high marks for making the team more productive, and for creating an environment where code and knowledge is shared. People are happy with the structure that Scrum provides, and we always know where the project stands. Incidentally, we also write better software faster.
But we're still improving the way we work. The transition is slow and painful, and we're only slowly adopting things such as test-driven development, automated builds and pair programming. In my experience, there's a lot of resistance against these "newfangled" methods in the academic culture, especially that of people who weren't trained as software engineers, but rather as physicists, chemists, biologists, but now find themselves producing software.
Some hints on what I've found useful in re-shaping our work environment:
- You can't change the whole structure in one day. Get permission to run a small, isolated project in "the new way", and use this to demonstrate the advantages. Remember, there are many metrics for success: Code quality, timely delivery, not having single points (persons) of failure, as well as team velocity and personal satisfaction. Try to make a case from this small project (and gain experience while doing so), and then grow it out slowly.
- I would not advise to do some clever "breaking the build, and thus showing everybody how fragile the system is" exercise. This may not be seen as constructive.
- Instead, provide convincing evidence by example that your way is more productive and more certain. Bugs that are fixed stay fixed, and don't creep in later again. Timelines are better kept. That sort of thing...
- If you can get someone in to talk about the current best thinking in software development, do so (someone else mentioned this already). It's good to hear an outside opinion, and to understand that these practices are not theoretical but used by large companies world-wide.
- I found Joel Spolsky's 12-point assessment very useful to find out where your organization stands: http://www.joelonsoftware.com/articles/fog0000000043.html ... These are also good points to whisper into management's ears.
I have worked in enough software companies to know that they are not necessarily better.
This sounds like the perfect scenario for open sourcing your software with you as the main developers maintaining it.
For the regular users, nothing much will change.
For the power users, those most likely to complain, this will be a tremendous benefit. If they don't like it, they have the possibility to improve it. This often reduce the number of problem reports and increase the good problem reports from your knowledgeable customers. Sometime you might even get useful patches, that save you some work. If you're really lucky, you might get a few users who start to code enhancements.
It also might generate some good-will towards your company and ease the integration of your bricks with other solutions.
What has this all to do with software quality? With your software out in the open, quality problems tend to be treated more like bugs that will be fixed as fast as necessary and possible and you get a better feedback where work is important. Making the software and drivers open source won't save your company any money, it won't cost more either, but it will improve what you get for your effort.
No they are not as evil as the Democrats says. Most of them want to do a good job so you will call them back in the future.
Why contract except for hire.
1. They can be paid for out of your department budget not the general budget. So it requires less steps up to get approval.
2. They work outside of HR. So you can hire them and fire them if you feel they are not doing the job correctly
3. Use a Multi-level support system. Get some Jr. Consultants to do the grudge work for cheap for 3 days on 2 days with a Sr. Consultant to insure the Jr.s are on track and solve larger issues deal with your department give any bad news and estimates etc...
4. More experience for less years. Especially if you get a good mix. You can get a specialist in X and Y and Z because you can use a consultat for their strong points.
5. If you give them motivation that there may be more work down the line they will focus on getting the project done. Yes they want to stay there but if there is a chance of a new project further down the chances of them milking you is a lot less.
If something is so important that you feel the need to post it on the internet... It probably isn't that important.
Make it fail. Make it fail spectacularly, to the tune of millions of dollars. That will certainly get the CEO's attention, and he will be sure to take measures that will stop such failure in the future. Of course, I can pretty much guarantee that you will not like his solution, but software development will be much more professional afterward.
If this is not what you want, ask yourself what you actually want to change. You do know what you want to change, right? Discuss those things with colleagues and managers, then formally propose doing them.
I'm guessing you probably want a more structured development process, with better-organized change requests, and at least some semblance of formal testing. That is very, very hard to set up, because it also requires the help of your users, and they don't care about software, they just want to have their problem solved. If this is the case, always remember that you are there to solve their problems, but they are not there to solve your problems. In other words, don't force them into a process they don't like. You might do better if you can show an advantage other than "it makes my life a little easier".
If all you want is a bugtracker and a version control tool, just request a budget of about $2000, then buy a Dell PC with Linux and install Bugzilla and SVN on it. That will set you back $400 or so, the rest of the $2000 is to show that you are a business thinker and did not forget to include installation time ;-)
If you want to institute Methodologies (like extreme programming or similar), good luck with that. It will probably end in your colleagues defenestrating you...
Go read up on the Capability Maturity Model (CMM) or ISO 9000 and come back when you have a clue.
You don't even need to formalize the process to that extent to make leaps and bound improvements on the hack-it-together and test it approach you are suggesting... At a minimal a decent software development process should have:
Requirements specifications & reviews
Design specifications & reviews
Test specificiations & reviews
Codng standards
Code reviews
Source control
Regression tests
Functional tests
Load tests
This argument is also known as "The Enron Gambit": those wildly successful guys who are raking it in hand-over-fist must know better than those of us who think that their business model makes no sense. They sure showed us.
Censorship is telling a man he can't have a steak just because a baby can't chew it. --Mark Twain
I'm a big fan of open source.
Most of the hardware at home has had its OS/firmware replaced with open-source variants.
BUT
Keep in mind that the software in certain types of devices is part of the 'competitive advantage' over other suppliers.
If you open-source the firmware/software of your instrumentation, a competitor can very easily build a similar device cheaper (because you already did most of the development).
I'm not saying you shouldn't do it, but you should show management that you are not 'giving away the keys'.
Show them you can provide a better quality of software to clients than competitors.
New features will make it into your software first, then the competitor will still have to factor it into their code.
"I was in love with a beautiful blonde once, dear. She drove me to drink. It's the one thing I am indebted to her for."
...and that language is money.
Find out how much it will cost the business if the software stops working. Estimate the risk (number between 0 an 1) of this happening. Multiply these two numbers. The result in dollars is the amount of money your company will lose with certainty. Not maybe, with certainty.
My opinion? See above.
Either start working for a company that do care about software quality, or start your own. I know it is a pretty drastic thing to say, and one I haven't (yet) followed myself, but I do believe that it is the only way.
My hopes would be that the more good engineers that start software companies that care about how things are done (and thus the end result) the less new companies needs to be started because the old ones suck. There are reasons why one sticks around a crappy company that treats you badly in one way or another (which I think is the case in this case), and the best hope for the people who can't leave that easily to form their own company is others doing so and they following when things have stabilized.
Keep in good contact with each and every good engineer you get to know :)
Are you a programmer? A SysAdmin? PS? QA?
Are you a worker, a lead, a manger, a PHB?
Each of these generates a different answer.
There is no "-1 offended" or "-1 you don't agree with me" mod options for a reason.
Unfortunately, software quality isn't even on most companies' radar. Until it exposes them to major losses like a balsa-wood skyscraper built next to the airport on the shores of the Petrol Sea.
Software has the disadvantage of being intangible and we all know that "Any kid can write software".
Any kid can bandage a cut, but that doesn't mean you want that kid doing a colonoscopy on you.
At some point the software industry is going to need to establish itself as a rigorous practice with rigorous standards. Not some silly cert that says you know Language-of-the-Week, but something along the lines of GAAP for accountants. I'm not holding my breath, though.
IF you happen to have - or be able to cultivate - the right social skills, take an active role. However, despite what the "don't like it, get-entreprenurial" crowd asserts, there are those of us who'll never be able to tolerate forcing their introverted personalities to assume an extroverted task on a long-term basis even with the best of counseling, self-help and medications. It can be wearying and it steals time energy from what we can do that the extroverts can't.
So if you aren't socially adept and don't see yourself swimming through office politics like Nemo, the best advice I can give is keep your resume up to date and network to whatever degree your social skills allow so you can bail before the tower collapses.
Then again, you can be Monty Hall and still come out of the losing side in the office, so keep the resume up to date anyway.
Do not bring up agile in this situation. You want to push them into a paradigm that is structured towards responsibility, not something that allows development to wash their hands from anything and just blame business. In the situation described in this story the best way to go is by setting up a structure that forces a couple of things: documentation of requirements up front, system and design specs, phased iterative development, unit testing of course, QA department, responsible management. I know it sounds difficult, but you have to work towards it, nothing is easy.
You can't handle the truth.
I write software for clinical researchers. From the summary description, sounds like the company has software that is part of their clinical quality system which is not being tested/validated. If the description is correct and the software is actually part of the clinical therapy they're selling, they should have an external auditing agency take a look at it before the FDA does.
The first time one of their products is audited by the FDA, the warning letter they receive will communicate to management exactly what is lacking in their compliance with FDA cGMP. Unfortunately, everyone from regulatory compliance down to the lowliest coder who had something to do with the products in question will share in the group spanking (been there).
I'd be shocked if they didn't already have a relationship with an auditing company--unless they're the tiniest of startups. If they do, the submitter should look through their last audit summary and see if anything has gone unaddressed, and if the scope of the audit matches the submitter's idea of the actual quality system for the product(s) in question.
Here's a possible solution, but you may not like it, because it will involve quite a bit of work for you. Whether or not you do this will depend on how much you really care, and whether or not you view this as an opportunity to make an impact (which it is). I would write a point paper that discusses a comparison of doing nothing vs doing it the "right" way. Make sure you include information about best practices, increased efficiency, improved productivity, and most importantly, how it will save money. Executives really only think about one thing: money. You start throwing dollar signs around and people will pay attention. Show them how much money (real money, based on the company's actual revenue and operating costs) is spent on backtracing due to bad software development management. Then show them how much they need to invest to do it "right", along with how much more money they can save, etc, etc. Remember, everything becomes a "cost", whether its time spent by employees redoing work, faulty products returned due to bad software, along with lost opportunities because the software isn't good enough for your products to enter new markets. It's a lot of work, but armed with carefully crafted and accurate information, with a sharp recommendation, could get you noticed, might actually get the company to make the change, and you could end up a hero. If they refuse to listen, you have at least learned a ton in the process, and you have something tangible you can show to a new prospective employer about much you care about making a contribution.
There are better ways of doing that, such as persevering, practicing what you preach (good development standards and approach), being firm with management about the issue.
ilovegeorgebush
Put together a business case showing how much money is spent on issues relating to software quality, and how much capital investment (e.g. new software and equipment) and operational expense (e.g. new staff) would be required to solve the problems. Calculate the cost savings over time, and show a positive 3-year NPV.
You might not be able to get the C-level execs to speak your language, but you can learn to speak theirs.
There is a ton of advice here, and almost all of it is good. But it all comes down to two realities -- (1) the company feels like it does not have a problem and (2) you do. So fuck 'em. If you are right, you are working on the Titanic, and they will tie you to the bow as they approach the iceberg. If you are wrong, then they will eventually fire you for it.
You may like the people and the work environment, but if either of the two points above are true, then that will change. So, use your initiative to go work for a competitor and show them how to exploit weaknesses like those at your current job. Believe me, your current employer has absolutely NO loyalty to you. They deserve no more from you. It is easier to find a job when you have a job.
Good advice. Mod parent up. From a technology perspective, a Version Control System, Test Driven Development, and Continuous Integration can go a long way towards improving quality. If the OP is in a MSFT shop, then you are most probably stuck with VSS or TFS. VSS is file based so it is not very good for distributed development. You will need to enhance VSS with SoS if you have remote developers. TFS doesn't have that problem and also has support for TDD's unit testing. If the OP is willing to use OSS, then there are plenty of good options available. There is plenty of good advice here as to OSS VCS. There are various unit testing frameworks for Java, .NET, Ruby, PHP, C++, you name it. Also, check out Cruise Control for Continuous Integration.
Technology alone cannot solve quality issues, however. Changes in methodology, process, and even corporate culture may also be needed. Take a look at my site for more advice on that.
I agree with the previous commenters that it's important to have a solution or plan before raising the alarm. Having said that, once you raise the alarm and you're not being heard to your satisfication, there are several options available: * First, clear your mind of what you think you know about software development and what SHOULD be and try to see the situation from an open-minded perspective. Are the issues you're seeing really an indicator of poor quality or are they an indicator of a system that's different from what you know/like? As a quality/regulatory person myself, I've seen many unnecessary projects and alarm bells simply because of a lack of understanding/perspective on a given topic. I'm not saying that's the case with you, just that this is the kind of issue that's good to be absolutely clear with yourself about. * Once you're clear that there is in fact an issue, go to QA and request an internal audit on your software development/quality systems. If your QA team doesn't have a procedure by which you can request an audit, then you should find a QA partner who can work with you on this. It's good to have a QA partner anyway so building the bridge on this project won't be a waste of time regardless of the outcome. * If auditing isn't an option, or if you need ammo to sell the audit idea, another approach is to analyze deviation and/or CAPA trends on your software development process, as well as your validation process. For example, try to find out how many validation deviations are being generated when new/updated software is released from your development team. Working with QA, you could develop an estimated cost-per-deviation, which would be a huge pile of ammo for your management presentation. Also, pretty charts and graphs will help too. * Find other tangible evidence of the issue. Without specific examples it will be difficult to be clear about the problem and/or the solution(s). * If you find evidence and QA and/or management still won't listen, it's time to consider your options. You can either stay, knowing that a ticking time-bomb exists, or you should carefully plan and execute your exit from the company. My litmus test for working at a company is to regularly ask myself whether I'd give my company's medicine to a family-member [a family member I love :P ]. If the answer is no, I don't stick around. So far, I've only had to do that once in 10 years and it was absolutely the right choice. Good luck!