Slashdot Mirror
How Healthcare.gov Changed the Software Testing Conversation
An anonymous reader notes an article about how the tribulations of Healthcare.gov brought the idea of software testing into the public consciousness in a more detailed way than ever before. Quoting:
"Suddenly, Americans are sitting at their kitchen tables – in suburbs, in cities, on farms – and talking about quality issues with a website. The average American was given nightly tutorials on load testing and performance bottlenecks when the site first launched, then crumbled moments later. We talked about whether the requirements were well-defined and the project schedule reasonably laid out. We talked about who owns the decision to launch and whether they were keeping appropriate track of milestones and iterations. ... When the media went from talking about the issues in the website to the process used to build the website was when things really got interesting. This is when software testers stepped out of the cube farm behind the coffee station and into the public limelight. Who were these people – and were they incompetent or mistreated? Did the project leaders not allocate enough time for testing? Did they allocate time for testing but not time to react to the testing outcome? Did the testers run inadequate tests? Were there not enough testers? Did they not speak up about the issues? If they did, were they not forceful enough?"
load testing and performance bottlenecks"
That's great but how about we teach the average American how to spot Europe on a map first.
Well you either blame the help, or else someone important is going to have to take the blame.
May the Maths Be with you!
Having been brought on late to QA a few death marches/trainwrecks in my time I have found that many projects don't get QA involved until way too late in the game.
.com boom days when everything was developer-centric and testing was seen as an unneeded cost that could be covered by the coders themselves -I don't need to tell anyone here why that is a bad idea.
This was very common in the
Ideally QA gets to help validate that the functional requirements are adequately addressed in the design. In many cases, lacking a spec of any kind I would have to create one of my own based on what the product was able to do or close to being able to do at that time in order to make a test plan.
When you are brought on board a sinking ship there is no point in blaming the crew for the state of the ship -all you can do is damage control to validate whatever is working and then lower the bar as to what constitutes 'working' or 'functional' -particularly if some major components or functionality are missing.
You're going to be seen by management as the people who are going to point out what idiots and incompetents the developers were and be seen as the enemy by the developers who were probably led down the rabbit hole by changing or nebulous requirements and unrealistic schedules...
So it is important to try to walk the middle line -making observations about the current situation without casting blame or making guesses about how the project got to that state (although it may be obvious when you look at the principals and the agenda). Gap analysis of both testing and in the product functionality and features is another thing that needs to be done more often in order to present a realistic picture of the current state of the product or project.
As a consultant it is nice to be able to come into these things knowing that you didn't help cause the trainwreck -you are just there doing triage and trying to save the patient....and sometimes management will listen to you about project and requirements that they ignored when brought up by their own people. Even if it is 20/20 hindsight perhaps they will heed the techies the next time they embark upon this path -Nah!
-I'm just sayin'
It does not sound to me as though known management tools were used. Did they sit down with the government personnel in charge, and present their approach, and what the site would look like (menus, flow, etc) when finished? Were there testable milestones, and a final presentation of working software? It sure doesn't sound like it.
They might well have done all these things and still failed to catch the problems before the site's launch.
Performance, like security (ack! scary!) is a non-functional requirement -- that is to say it's not the kind of requirement where you can sit down with a checklist and say, 'yep, this it works,' or 'no, it doesn't.' You have to develop a more sophisticated test.
Load testing is a step in the right direction, but you also have to look at system architecture. Remember the days before people figured out that you had to load web ads asynchronously, after the page content was loaded? Sometimes the page load would be slow, not because the page's server was loaded, or because of the user's browser or internet connection were slow. Often it would be the ad server that was overwhelmed, which if you think about it is bound to be more common than the content server being overwhelmed. You could functional test and even load test the heck out of a page with synchronous ad loads, but until it went into production chances are you wouldn't catch the fatal performance flaw. That kind of problem is architectural; some of the data being delivered is coming from servers outside your control.
Ordinary tests are about ensuring reproducible results, but when the architecture leaves you vulnerable to things happening on servers and networks outside your control your problems *aren't reliably reproducible*. You have to design around *possibilities*.
Some of the problems with Healthcare.gov were of this nature, although with not so simple a solution as "use window.onload()." The site is supposed to orchestrate a complex, *distributed* process *synchronously*. You have to go out the Homeland Security's database to confirm citizenship, then to the IRS databases to confirm claims about income, then get quotes from the private insurers that meet the customer's needs. There is, in my opinion, no way to be 100% sure, or even 80% sure that a system like that will work under real work load, unless you present it with real work load.
Were I architecting such a site, I'd plan to do a lot of that work batch; that is I'd build the healthcare application offline on the user's browser, with internal consistency checks of course. Then I'd send the user's application through a batch verification system, emailing him when a result was ready. This is a clunky and old-fashioned approach, but it wouldn't force the user to chain himself to his browser . And it would have more predictable performance. Predictability is a vastly under-rated quality in a system. A system which is fast most of the time may not be as desirable as one which provides the answer consistently.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
The real world does not work like this and I find the best work comes when the guy who will be writing the software actually collects the requirements and does a good job of that including a plan of how to test the code.
The worst code I see is written when the programmer is given some narrow requirements on some ticket and they code directly to those requirements with little to no knowledge of the overall system they are working on. In some magical world where all of your requirements are perfect this could work very well. But part of being a solid developer is knowing how to spot the "smell" of bad requirements. While sometimes you can do this without any knowledge of the overall system, you are much more likely to have the right insights if you have some relevant domain knowledge.
I have a current project that I am maintaining where I wrote most of the code in a vacuum without understanding the customer's real needs. It is a horrible mess and I would have done things completely differently if I was involved in meetings with the clients early on. I'm not saying my code would have been perfect, but there were some massive disconnects between the assumptions I drew from the written requirements and the explanations I got from the clients once I was given more access to them. Now there is no time or money for massive rewrites or refactorings, so it continues to be a thorn in my side.
-- All that is necessary for the triumph of evil is that good men do nothing. -- Edmund Burke