Slashdot Mirror
How Would You Handle a $1,000,000 Coding Error?
theodp writes "The Chicago Tribune's efforts to upgrade its computer system over the weekend turned into a fiasco when the system crashed, halting all printing operations and leaving about half of the Trib's subscribers without papers. The software contained 'a coding error,' according to a spokesman who estimated the cost to resolve the problem at 'under $1 million.' Any advice for the poor schmuck who's going to get the blame?"
Well, ok so that might not fly, but hey, it works when its true if you work for a modestly forgiving employer...
;-)
Now if the cause was insufficient testing, well then QA has to answer for it.
And if there's no QA, well that's managements fault...
Now if it all comes down to dumb circumstances, it's poor planning on the papers fault for not testing themselves
That said, fess up, worse comes to worse, you now have national infamy, and any fame is good fame, right??
-- (appended to the end of comments you post, 120 chars)
Where was the pre-install testing?
A good test should have identified some errors, especially if it blew up IMMEDIATELY.
Management frequently makes mistakes which cost much more. The difference is that their mistakes are not as easily identified or attributed to a single person.
The culprit should just admit it. Shit happens, it's unavoidable even if you take all precautions. Don't make the same mistake again, though.
In my experience being honest about your mistakes and having the willingness to learn from them always pays off.
With any large roll out, if only one person is at fault for a fiasco like this, then the project mas mismanaged. They should have had a plan in place to backout the change.
Where was the phased or parallel deployment?
You don't just change a system like in a weekend. There WILL be problems, so you have to have ways of dealing with it. Maybe that means flicking the switch back to the old system if it fails, or maybe it means running with degraded capacity a while, but whatever it is, it's dead-in-the-water is not your Plan B.
Forget thrust, drag, lift and weight. Airplanes fly because of money.
One of the benefits of working for a big company is a QA/UAT department. You have an entire department of people lined up just to test your shit. And, usually this type of job makes a person very anal. They log defects for just about everything.
The person writing the code can unit test to his or her best ability, but it is really the job of someone else to put it through the wringer testing thousands of simulated real-world scenerios. Sure, a coder could do this testing. But a QA guy or gal is doing really well if he makes 3/4 the salary of the guy who wrote the code- so a divison of labor only makes sense.
Not to mention the person writing the code makes the worst tester in the world. You only test it the way you THOUGHT people would use it. So, while a coder is perhaps the one who created the original problem, the real fault is in whoever let this slip through to production. Assuming, of course, that it wasn't some kind of time-bomb easter egg that would have been impossible to test. Although, good QA testers should alter their system date/time when testing date sensitive routines.
Good planning would have had an abort procedure, so the show would go on. Everything changed should be undone if it did not work. They could figure it out after the paper was printed.
Errors are inevitable. Good planning and implementation keep you from falling on your face even when you publish seven days a week. It's not the coder's fault.
Friends don't help friends install M$ junk.
Noone [in their right mind] orders a brand new paper publishing system from a single consultant. The software probably was priced in several million dollars. Somewhere between the components something broke. For example, the file format that the publisher produced was rev. 2.1, but the software at the presses side was only aware of rev. 1.7 and below... If the coder only tested his code with the "other" piece of latest revision, he would never see any problem; and it is not his guilt that in real life the real customer uses some obsolete stuff that isn't compatible...
This kind of problem is clearly of administrative nature, of a system design and of checking which pieces work with which other pieces. Clearly, blame should be assigned to non-existent QA procedures, insufficient unit testing and [obviously] inadequate integration of components. The coder is nowhere here, it's all system design and QA stuff, realm of managers.
So the paper can deliver every day for 158 yrs using mechanical printing presses ~ except where natural disasters occur ....
The printing problems at the Chicago Tribune were related to efforts to upgrade computer equipment used to produce the newspaper, Malone said. The Tribune acquired customized software for the upgrade from an outside provider, and it contained a "coding error," he said.but as soon as computers are involved their printing press has morphed into a computer system. I wonder what provisions to *test* the upgrade before use where made?
fail to recognise newspaper as computer system?it would be easy to blame the developers and company and there should be some recognition of responsibility for technical accuracy. but what about the newspaper. they have made a fundamental mistake in not recognising that printing press + computer = computer and let their newspaper system fail at the mercy of coding mistake.
It seems while the paper can handle *mechanial* failure (158 yrs, 1 non delivery) it has yet to grasp *software* failure.
peterrenshaw ~ Another Scrappy Startup
Mmm, yeah, perl, obviously the best language to debug, especially if it's someone else's code. *shudder*
I think I'd rather debug someone else's assembly language than someone else's perl.
Software testing is boring boring boring. You have to try things out again and again after each change. Modules that haven't changed gain confidence in the face of changes and might not be tested, but omitting tests can end up being the Achilles heel. There can be an overwhelming desire when a project nears completion to just get things done and over with. After all the hard problems may well be solved and it's all down to seemingly inconsequential details.
These days programmers have a Sword of Damocles hanging over them. Once they finish a major piece of code they may have a hard time finding new work. The economy has not lived up to forecasts of more jobs. Outsourcing has reduced computer opportunities. Management of many companies do not see new uses for computers. Off-the-shelf programs abound for almost every aspect of computerized work.
Stress may distract software engineers enough that someone will make a major mistake.
Know your pads. One time pad: good for cryptography. Two timing pad: where to take your mistress.
If they had a clue, they would grow 10000 acres of canabis which;
A) grows 10000x faster than trees
B) makes 10x more pulp per acre
C) uses 100x less water.
D) stick it to the govt.
But would they ever do that? NOOOO coz there are no patents in the process to expoit and oh the trouble of the govt wackos like bush n old guys being so anti-canabis (to protect their buddies profits)
I guess they wouldnt want 100s of pot heads heading up to the 100000s acres of weed to take a few home, but what is so wrong with that OTH?
Liberty freedom are no1, not dicks in suits.
Grab all the eMails where someone in management told you to cut a corner, or replied that they didn't want to spend too nuch time designing, or authorized fewer QA hours then should have been done, and print it all out, with headers, and forward it all to another account.
When they come after you, present it as if it you were trying to do it right, but somebody wouldn't let you.
If they fire you, sue.
Unless:
a) you work for one of the few companies that actually supports a real team atmosphere, or
b) Everything was done by the book, and you still screwed up.
When someone in an industrial field is forced to work 16 hour a day, 7 day a week, and has a mistake the company suffers the ramaifications, not the worker(or the workers faimly).
The Kruger Dunning explains most post on
My experiences tell me cannabis is a much more desirable drug than alcohol, both from the users and society's point of view.
Use both drugs with some sense and nothing bad will happen. Overdo alcohol and it will make you loud and often aggressive. Overdo cannabis and you will fall asleep (which can be loud but seldon aggressive). Neither are very suitable for driving. (Although I prefer people who smoked over people who drank: they drive more relaxed.)
It is when talking addiction that the large difference arises. Alcohol is a hard drug, you get physically addicted, cannabis is not. Alcohol demolishes you while it degrades you. Cannabis use over large timeperiods is claimed to deteriorate memory. (So, don't drink to forget, smoke! ;) If you smoke the cannabis (instead of eating it) you get the same risks as with tabacco use.
Here in Belgium, cannabis is more or less legal now (we are allowed to carry upto 3.3 grams on the street and use it in private places and such). It is a good thing, because we did that anyway (I live about 40 kilometer from the closest cannabis shop in the Netherlands where I can buy as much as I like legally).
There were no sudden changes in behaviour. No millions extra addicts, no stepping stones, nothing. The people who are inclined to (ab)use drugs usually do not care about legality.