Slashdot Mirror
How Healthcare.gov Changed the Software Testing Conversation
An anonymous reader notes an article about how the tribulations of Healthcare.gov brought the idea of software testing into the public consciousness in a more detailed way than ever before. Quoting:
"Suddenly, Americans are sitting at their kitchen tables – in suburbs, in cities, on farms – and talking about quality issues with a website. The average American was given nightly tutorials on load testing and performance bottlenecks when the site first launched, then crumbled moments later. We talked about whether the requirements were well-defined and the project schedule reasonably laid out. We talked about who owns the decision to launch and whether they were keeping appropriate track of milestones and iterations. ... When the media went from talking about the issues in the website to the process used to build the website was when things really got interesting. This is when software testers stepped out of the cube farm behind the coffee station and into the public limelight. Who were these people – and were they incompetent or mistreated? Did the project leaders not allocate enough time for testing? Did they allocate time for testing but not time to react to the testing outcome? Did the testers run inadequate tests? Were there not enough testers? Did they not speak up about the issues? If they did, were they not forceful enough?"
Ya those damn testers, they just can't communicate the issues to management. Like that NASA engineer and the O-rings. Stop blaming the testers.
It is not really a question of testing. Parts of the software were missing or incomplete. You can't test what isn't there.
I cannot imagine a worse job than to have worked on that project.. The ratio of "status update" meetings and management pud-pulls to useful work accomplished must have been damn close to infinity..
You haven't worked on my project.
Faster! Faster! Faster would be better!
load testing and performance bottlenecks"
That's great but how about we teach the average American how to spot Europe on a map first.
Most of the Affordable Care Act has nothing to do with the web site. The site didn't have to implement those "2.8 million words of Obamacare regulations" as code: it only had to match patients up with insurance plans, which means interacting with dozens (hundreds?) of government and industry databases.
Some states, like California, managed to implement their sites without any of the problems of the federal exchange. The federal exchange mainly suffered from (1) being rushed, and (2) having to deal with a larger number of external systems than any single state exchange.
Visual IRC: Fast. Powerful. Free.
Imagine being the QA inspector on a 1985 Jugo car. No matter what you say, the entire thing is a POS. The only question is whether you need your paycheck that badly. Politics and unrestrained corruption simply don't mix well with code.
One of the most insightful truths ever told to me:
It is always management's fault.
This goes right to the root of the tree, because by definition if the people further out couldn't get the job done or didn't have the right resources to do it, it was management's responsibility to fix those problems. The buck stops with the most senior managers on a project, whose only two choices are to explain what is needed to succeed and then do so if given those things, or to fail.
If you disagree, post your argument. (-1, Overrated) isn't your personal censorship tool for views you don't like.
Why? Is Europe's location somehow significant to average Americans?
And there is the proof of the OP's implied statement
It does not sound to me as though known management tools were used. Did they sit down with the government personnel in charge, and present their approach, and what the site would look like (menus, flow, etc) when finished? Were there testable milestones, and a final presentation of working software? It sure doesn't sound like it.
They might well have done all these things and still failed to catch the problems before the site's launch.
Performance, like security (ack! scary!) is a non-functional requirement -- that is to say it's not the kind of requirement where you can sit down with a checklist and say, 'yep, this it works,' or 'no, it doesn't.' You have to develop a more sophisticated test.
Load testing is a step in the right direction, but you also have to look at system architecture. Remember the days before people figured out that you had to load web ads asynchronously, after the page content was loaded? Sometimes the page load would be slow, not because the page's server was loaded, or because of the user's browser or internet connection were slow. Often it would be the ad server that was overwhelmed, which if you think about it is bound to be more common than the content server being overwhelmed. You could functional test and even load test the heck out of a page with synchronous ad loads, but until it went into production chances are you wouldn't catch the fatal performance flaw. That kind of problem is architectural; some of the data being delivered is coming from servers outside your control.
Ordinary tests are about ensuring reproducible results, but when the architecture leaves you vulnerable to things happening on servers and networks outside your control your problems *aren't reliably reproducible*. You have to design around *possibilities*.
Some of the problems with Healthcare.gov were of this nature, although with not so simple a solution as "use window.onload()." The site is supposed to orchestrate a complex, *distributed* process *synchronously*. You have to go out the Homeland Security's database to confirm citizenship, then to the IRS databases to confirm claims about income, then get quotes from the private insurers that meet the customer's needs. There is, in my opinion, no way to be 100% sure, or even 80% sure that a system like that will work under real work load, unless you present it with real work load.
Were I architecting such a site, I'd plan to do a lot of that work batch; that is I'd build the healthcare application offline on the user's browser, with internal consistency checks of course. Then I'd send the user's application through a batch verification system, emailing him when a result was ready. This is a clunky and old-fashioned approach, but it wouldn't force the user to chain himself to his browser . And it would have more predictable performance. Predictability is a vastly under-rated quality in a system. A system which is fast most of the time may not be as desirable as one which provides the answer consistently.
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
Yes, yes, because every bridge the government builds falls down three or four times a day in the first couple of weeks after it's opened.
Slashdot - News for Nerds, Stuff that Matters, in ISO-8859-1 Has just realised that beta makes this signature redundant
If you find out in testing that your architecture or design does not cut it, you are screwed. The only thing you can usually do is scrap the project and start again. Testing does only work for simple things like simple busiess logic and the like, where you know the characteristics very well beforehand. For anything that is a new design, the only thing that helps is very capable and experienced architects and designers that have a good change to get it right by intuition. This will be people that can do architecture, design and implementation and can do all three well. Not many of those exist, but there is no replacement for them. Those that think they can do things on the cheap without not only having this type of expert but also listening to them closely will fail. This can be observerd time and again and can alost be called a "well established industrial practice", because quite a few "managers" do not actually know that it can be done better. Funny thing, in other fields, you have chief enineers, architects and the like and the critical work is not given to people that are likely to fail. Only IT messess it up regularly, because talent and exerence is not respected.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted by this and ignored otherwise.
And not to mention, Battlestar Galactica too. A lot of the needless drama on that show could have been avoided if they had just licensed their testing framework from the Cylons.