Software Quality In a Non-Software Company?

Nicros writes "I work for a publicly traded biotech company that happens to write software that is, in fact, kind of critical for the business — without it no data would ever be read from our instruments, and no analyses would be performed on that data. The problem is that as a 'biotech' company, we are not taking software quality seriously. We have no senior management with any history of commercial software development — our C level has really no clue whatsoever what software really is, much less what is going on in software development. All of our quality processes are related to manufacturing our system (not software), so we are constantly forced into ad-hoc development since there is no real process for our development. Repeated requests to hire someone with some real commercial software development experience have gone unanswered. I have been to the CEO directly one-on-one and although he agreed this was an issue, thanked me, and said he would look into it, that was the end of it. He has bigger things to worry about. So the question: Is this just a fact of life and I need to deal the best I can? What else can I do to get some attention on software quality in the company?"

Comments

Practice What You Preach

[ilovegeorgebush]

You're obviously fighting an up-hill struggle. Going straight to the CEO is both a good and bad idea - if it works you'll get immediate affect, but it's likely to be ignored.

You need to argue this case as much as possible. If you're the developer, or in charge of development, enforce decent developmental practices and ensure your estimates include them. Err on the side of caution. Take an estimate and double it. Managers talk money, not standards, so you'll have to hit them where it hurts.

Otherwise, is there anything off-the-shelf that could alleviate some development?

Re:Practice What You Preach

[b4upoo]

Between being right and being popular with bosses being popular wins out every time. Pestering them about better ways to do things is not a great idea. Play golf with them and never mention a darned thing relating to work and if something does crop up then make them think it was all their idea all along. Money shall flow to you as well as job stability. Make sense and be logical and you might as well start a job search.
          Thia may not apply in other nations but in America it works every time.

Re:Practice What You Preach

[Hotawa Hawk-eye]

Your statement "We have a problem, this is how other people solve it, and this is what I will need to solve it. Give me the budget and I'll solve the problem." is close to complete, but you're missing one piece.

"We have a problem, this is how other people solve it, and this is what I will need to solve it. This is what it costs us _not_ to solve it. Give me the budget and I'll solve the problem."

If you can show that the software development 'process' currently in place is costing the company $N a month and you will need to spend $X to improve the process, if you're going to be developing software for more than (X/N) months, it'll be more cost-effective to fix the process.

Re:Practice What You Preach

[electroniceric]

I also work for a biotech (... this my other brother Darrell), and we're facing the prospect of FDA regulation as a device. So we're presently working our way through ramping up a formal (21CFR820) Quality System now. My boss happens to have been through this before and to be a pretty effective evangelist for the FDA's Quality System methodology, which is required for all medical devices and drugs. As he says, a Quality System is just putting to paper all the things you should be doing anyway. So one place you may want to start is by discussing the utility of complying with FDA regs with regard to software.

My boss also notes (on any occasion where there's an opening) that when the FDA introduced design controls, most companies complained they were going bankrupt (as companies are wont to do when regulation, merited or unmerited, is proposed). But when the FDA went around doing their roadshow to show that they weren't just making rules without listening to industry, people from the device companies gradually started to get up and explain how using a Quality System actually lowered their costs and decreased their time to market for revisions and product upgrades.

So as an evangelization tactic I'd look on the FDA's site for guidances relating to the introduction of the Quality System Regulation. For example, this guidance on general principles of software validation is pretty good. If you mentally translate into software industry language you can really see that they're trying to get you to do better engineering by thinking and documenting early, really getting straight what the software is trying to do, and being structured about showing that it does what it needs to do. The truth is that despite the startup effort of introducing documentation and procedures, controlled engineering methodologies work way better - they reduce requirements failure, increase code quality (and more importantly, design quality), and - though developers start out hating paperwork - even make the developers happier because more code works and sees real use. If the company plans to be in the software business for the foreseeable future, it's almost certain that the effort invested in good software practices will pay huge dividends down the road. The key is point out that quality is not an esoteric consideration, it's a driving cost and business risk consideration. Sooner or later the cost of low quality software shows itself.

One thing I will note is that the QSR is pretty waterfall oriented, both because it predates the formalization of iterative/agile methodologies, and because it's written for engineering of physical boxes that have to be released to manufacturing (which implies a fair bit of waterfallism). Part of our effort is to practice iterative development methodology while documenting to the FDA's standards.

Anyway, take a look at all that.

Anarchy is an opportunity

[Hal_Porter]

It sounds like in your company there is no one doing this job. You've talked to the CEO. Get him to make you VP of software and tell him you'll solve the problem if he gives you responsibility.

Anarchy is an opportunity for the ambitious and unprincipled. Take it and make yourself software Czar.

Re:Anarchy is an opportunity

[pieterh]

Agreed. Don't raise issues for other people to solve, you are just labeling yourself a trouble maker. Raise issues, attach costs to them, and then present yourself as the person with solutions, and ask for budget to solve them. Make a proposal with figures, planning, clear savings, and get approval. Then hire and build a competent team and/or find a good subcontractor. Use open source where possible to save costs. Report your progress and ensure you get budget every year.

Think of ways to turn a profit from the software. Maybe it can be licensed to other firms? If you can earn revenue you will suddenly become much more valuable.

Problem is: you will stop coding and become a manager. But if you do a good job, you can get power in the firm.

If you present a good plan that will solve real problems for the company, and you are not given the green light, then look for another job. If/when things go bad, they won't thank you for it.

Re:Anarchy is an opportunity

[southpolesammy]

Careful what you wish for though....

The flip side of becoming a point of authority in an environment as this is that if/when code defects bubble to the surface on your watch that result in a major hit to the company's bottom line, you may need to have a thick layer of asbestos underwear on in order to prevent the blame game from claiming you as a victim.

Right now, you've identified the problem for your mgmt and have suggested solutions, but you're not yet responsible for the implementation of those solutions. Becoming the VP of such an org not only makes you responsible for the fixes, but also directly accountable, possibly including from a legal standpoint. In other words, you'd better hope that the bugs in your software don't have the potential to cause medical or financial harm to your customers.

Re:Anarchy is an opportunity

[kitgerrits]

If you want to keep yourself safe, document and report all 'blind spots' in your method.
Make sure you present an overview of what you can control and what you cannot control.

If management does not agree with a certain blind spot, show them the resources required to cover that blind spot.

You cannot have bug-free code without strict rules and a literally astronomical budget. (and even NASA has had a few bugs)
What you can do is prevent embarassing/dangerous bug from making it into 'production software'.

Opinion from the outside?

[DerCed]

Could you propose to hire a software test consultant for a day or two and let him point out serious quality issues (data integrity, security, correctness..)?
A serious, alarming report by an external software test professional could help reinforcing your requests?

I have the same problem

[_Shad0w_]

I have the same problem where I work, the problem is I am the dev with real commercial experience; I just can't convince them that we need to do things in a manner that I would consider correct - it's all ad-hoc development and it's all driving me nuts.

The problem is, if our software doesn't work correctly, then the data we collect and process using it becomes screwed up, which is a major issue for the core business - data is our crown jewels.

My current solution to the problem is looking for a new job in a company that actually takes software development seriously. I just can't see any way of getting things here working the way I want. There wasn't even any revision control in place on the source code when I started.

The problem I'm finding is that the lack of structured development and design here is actually beginning to hurt me professionally: prospective employers, who have software development as a core aspect of their business, actually ask about this kind of thing. If you're looking to hire someone who takes their profession serious, for god's sake make sure they're actually going to be able to do their job - otherwise your company is just going to turn in to a blot on their CV.

Some hints for your situation

[Apogee]

I'm in a quite similar situtation, and perhaps I can provide a few hints from what we're currently doing.

I'm working for a relatively well-known institute in academia (biotech field), with a group that among other research projects, also provides web-based services to the research community. Funding is partially tied to the operation of the services, so there is actually enough pressure to make sure that they work and work correctly at all times.

Still, until about a year ago, development was very ad-hoc, in a mix of languages, and with many "islands of knowledge", where some parts of the system were only known to one post-doc, and other parts could only be fixed by the group head (who, as they are, was usually busy with many other things...). After some hard times and near-misses, we started looking around for ways to improve our development.

I was quite attracted by the ideas of Agile, and I believe that they're a good fit to the kind of processes you find in science, as well as in software engineering. We initially had a professional Scrum coach come in and talk with us about software development practices, and then decided to apply Scrum to our processes.

It's now a bit more than 1 year since then, we're still using Scrum with a few adaptations to fit the academic environment (we're also using Scrum for projects that are really science and research, not software development). In a recent secret poll among the team, Scrum got high marks for making the team more productive, and for creating an environment where code and knowledge is shared. People are happy with the structure that Scrum provides, and we always know where the project stands. Incidentally, we also write better software faster.

But we're still improving the way we work. The transition is slow and painful, and we're only slowly adopting things such as test-driven development, automated builds and pair programming. In my experience, there's a lot of resistance against these "newfangled" methods in the academic culture, especially that of people who weren't trained as software engineers, but rather as physicists, chemists, biologists, but now find themselves producing software.

Some hints on what I've found useful in re-shaping our work environment:

- You can't change the whole structure in one day. Get permission to run a small, isolated project in "the new way", and use this to demonstrate the advantages. Remember, there are many metrics for success: Code quality, timely delivery, not having single points (persons) of failure, as well as team velocity and personal satisfaction. Try to make a case from this small project (and gain experience while doing so), and then grow it out slowly.

- I would not advise to do some clever "breaking the build, and thus showing everybody how fragile the system is" exercise. This may not be seen as constructive.

- Instead, provide convincing evidence by example that your way is more productive and more certain. Bugs that are fixed stay fixed, and don't creep in later again. Timelines are better kept. That sort of thing...

- If you can get someone in to talk about the current best thinking in software development, do so (someone else mentioned this already). It's good to hear an outside opinion, and to understand that these practices are not theoretical but used by large companies world-wide.

- I found Joel Spolsky's 12-point assessment very useful to find out where your organization stands: http://www.joelonsoftware.com/articles/fog0000000043.html ... These are also good points to whisper into management's ears.

What exactly means "Non-Software Company"?

[Tanuki64]

I have worked in enough software companies to know that they are not necessarily better.

Re:What the Hell are you talking about?

[SpinyNorman]

Go read up on the Capability Maturity Model (CMM) or ISO 9000 and come back when you have a clue.

You don't even need to formalize the process to that extent to make leaps and bound improvements on the hack-it-together and test it approach you are suggesting... At a minimal a decent software development process should have:

Requirements specifications & reviews
Design specifications & reviews
Test specificiations & reviews
Codng standards
Code reviews
Source control
Regression tests
Functional tests
Load tests

Re:Here's a revolutionary idea

[indifferent children]

How much combined experience does the management team and board of this company have?

This argument is also known as "The Enron Gambit": those wildly successful guys who are raking it in hand-over-fist must know better than those of us who think that their business model makes no sense. They sure showed us.

Use the right language

[ebbe11]

...and that language is money.

Find out how much it will cost the business if the software stops working. Estimate the risk (number between 0 an 1) of this happening. Multiply these two numbers. The result in dollars is the amount of money your company will lose with certainty. Not maybe, with certainty.

Garbage in, Gospel Out?

[Anonymous Coward]

Unfortunately, software quality isn't even on most companies' radar. Until it exposes them to major losses like a balsa-wood skyscraper built next to the airport on the shores of the Petrol Sea.

Software has the disadvantage of being intangible and we all know that "Any kid can write software".

Any kid can bandage a cut, but that doesn't mean you want that kid doing a colonoscopy on you.

At some point the software industry is going to need to establish itself as a rigorous practice with rigorous standards. Not some silly cert that says you know Language-of-the-Week, but something along the lines of GAAP for accountants. I'm not holding my breath, though.

IF you happen to have - or be able to cultivate - the right social skills, take an active role. However, despite what the "don't like it, get-entreprenurial" crowd asserts, there are those of us who'll never be able to tolerate forcing their introverted personalities to assume an extroverted task on a long-term basis even with the best of counseling, self-help and medications. It can be wearying and it steals time energy from what we can do that the extroverts can't.

So if you aren't socially adept and don't see yourself swimming through office politics like Nemo, the best advice I can give is keep your resume up to date and network to whatever degree your social skills allow so you can bail before the tower collapses.

Then again, you can be Monty Hall and still come out of the losing side in the office, so keep the resume up to date anyway.