Slashdot Mirror
How Would You Handle a $1,000,000 Coding Error?
theodp writes "The Chicago Tribune's efforts to upgrade its computer system over the weekend turned into a fiasco when the system crashed, halting all printing operations and leaving about half of the Trib's subscribers without papers. The software contained 'a coding error,' according to a spokesman who estimated the cost to resolve the problem at 'under $1 million.' Any advice for the poor schmuck who's going to get the blame?"
Check out this link. Sorry, dude. Any of us could have done it.
> How Would You Handle a $1,000,000 Coding Error?
I would have to follow Dogbert's Top Secret Management Handbook, and take full responsibility for the bungle. That way when the next job comes up two or three rungs above me, I'll be at the top of the list of people with actual experience with massive projects, and it won't matter that it was a colossal screw-up because I will have jumped two or three pay-grades. Corporate fall-guys, if they take it right, always end up better off than quiet behind the scenes types.
So my advice is that you should take full responsiblity and sharpen that resume, but be sure to make it known that you have learned from your mistakes and you worked hard to correct them. Nobody gets anywhere without making big blunders along the way. Be a good sport and you'll jump at least two pay grades for this blunder.
The dangers of knowledge trigger emotional distress in human beings.
(It pays to use Splint)
Sigs cause cancer.
As for the QA department...
Change your name? Use that guy that cost the Cubs their game last year as a shield?
run, cause you can't hide.....
... and blame it on Microsoft.
[SIG] It's like putting a moose in the blender -- a recipe for disaster!
Time for plan B
They seem to not have problem dealing with the multibillon dolar programming error called Windows, maybe being there will help to put the facts in perspective.
Go to Mexico
Debuggers and Virtual Machines are your friends!
Just have each of their coders chip in a dollar, problem solved.
*ducks*
Introducing the new Occam Fusion! Now with sqrt(-1) fewer blades!
Blame it on the company not supplying enough caffine?
Anyone else think it was poor 'theodp' ??!
duck
Well, ok so that might not fly, but hey, it works when its true if you work for a modestly forgiving employer...
;-)
Now if the cause was insufficient testing, well then QA has to answer for it.
And if there's no QA, well that's managements fault...
Now if it all comes down to dumb circumstances, it's poor planning on the papers fault for not testing themselves
That said, fess up, worse comes to worse, you now have national infamy, and any fame is good fame, right??
-- (appended to the end of comments you post, 120 chars)
I would go out, and get so absofreakinlutely drunk that I wouldn't be able to remember my middle name, let alone that I made a $1M error. And then when the lawsuits are about to go to court and I started showing signs of severe alcoholism, I would put my head inbetween my legs and kiss my ass goodbye. 'Cause man, that would really suck.
Well, you asked.
Any advice for the poor schmuck who's going to get the blame?
;)
Well my first advice is to come clean, yes I mean you theodp, I think we all know who this poor schmuck is
I stole this Sig
He should blame the requirements.
There's always a mistake in the requirements.
Don't know; Don't care; Don't ask
I hear Newsweek is firing their publishing chief because he published their news too early. Perhaps Mr Chicago Tribune man could offer Newsweek some help by delaying publication until Steve Jobs holds his press conference.
Just my two pence.
Sincerely,
Seth Finklestein
Media Watchdog Gadfly
Not affiliated with Seth Finkelstein
Where was the pre-install testing?
A good test should have identified some errors, especially if it blew up IMMEDIATELY.
That isn't a bug - its a feature!
Even heroes have the right to dream
If they didn't test the system well enough before throwing it in there, the people who installed it and the Paper's employess share the blame.
It's so easy! Code a bunch of nonsense, and when it screws up, just shrug and keep doing it!
Become an electrical engineer, and one single, tiny, almost undetectable mistake, your career is over!
23:44:03 up 48545 days, 6:15, 1 user, load average: 0.00, 0.00, 0.00 Blink. up 0 days, 1:00, 1 user, load average: 0.00, 0.00, 0.00 I hope they got a SS of that massive uptime.
You could refund their subscription for the day, or you could add an additional week to their current subscription. Which one would generate better goodwill and thus better publicity? (perhaps a slashback)
Tell the truth, fess up. People will accept the error as long as you give them free stuff.
-NOAH
Who would have guessed "start" didn't mean the same in India?
I like muppets.
It always works. ;)
there's no place like ~
They'll just blame the contract coder that left six months ago because the guy that really f*#@ed it up was the boss...
I hear that McDonald's in India is hiring.
SD
âoeWho knew something as harmless as willful ignorance could end up having real consequences?â
I didn't get my paper this morning and was angry until I read this.
I'm not angry anymore, I'm sympathetic for the poor schmuck as well as all the customer service people who probably got yelled at this morning.
-- Kevin J. Rice
Unitarian Church: Freethinkers Congregate!
People, it's called QA.
Toss his newspaper subscription and egg his car. Other than that, leave the poor geek alone.
How many people here have fucked LILO into the ground the night before a java assignment on a laptop with no floppy? anyone?
yeah. i thought as much.
ZERO
Management frequently makes mistakes which cost much more. The difference is that their mistakes are not as easily identified or attributed to a single person.
The culprit should just admit it. Shit happens, it's unavoidable even if you take all precautions. Don't make the same mistake again, though.
Start looking for a new job.
The darkness... controls the music. The music... controls the soul.
tell the moon to tell to tell the march tell the moon to tell to tell the march.. he is here tell the moon to tell to tell the march he is here, tell the moon to tell to tell the march
You dickwads just paid a megabuck to train me.
Faith: n. -- That human impulse that drives them to steal appliances when the power goes out
LIMITED LIABILITY
Software provided as-is. Softare developer/company is not liable for any physical, financial, or any other loss or damage arising from use of software.
Doesn't all software come with things like this? (nevertheless, thank-goodness I'm not a software developer)
$cat
" .. I must have missed a zero somewhere ... damn I always do that!"
If I ever was on the specifying end of software (instead of the coding end, where I do these things reflexively anyway), I would demand the following of the team I hired:
1) 100% unit test coverage, verified by a second outside team using whatever tools are appropriate for whatever language they are using (i.e., something like Jester for Java).
2) Reduction of existing code at each iteration. After the project implements the basic features, I would demand reduction of logical code lines (i.e., actual code statements, not comments or multiple statements on each line) at each iteration. In other words, existing code must *shrink* before you can add new code.
3) full source code and copyrights
If they couldn't meet these requirements as well as the actual project requirements, I would fire them, not pay them, and find someone else. Cost and deadline would be secondary concerns to these.
And on the flip side, if I were the programmer and I couldn't get clear requirements and enthusiasm from the customer, I'd drop the project.
Ahh.. a man can dream can't he?
You see, Windows is all teh bugs!
Mod parent +2 so dang funny!!!
"Any advice for the poor schmuck who's going to get the blame?"
My advice: Prepare three envelopes
PR huff and puff
They misplaced a decimal... it was actually a $1 billion error.
Change your name, and switch to a "skills" based resume rather than an experience based one...
Karma: 0 (But I wield a mean +10 Vorpal Apathy)
I knew we shouldn't have switched our system to MS Office + a bunch of VBA scripts...
And this is why you don't use an Access database for a job like this.
-- I could tell right away that she was impressed with my HUGE Slashdot Karma.
Down, not across. (motto of alt.sysadmin.recovery referring to best method of slashing one's wrists).
Blame the users, of course.
Or The journalists that work at the outfit the link went to. Did you notice it took 3 of them to write that article? Talk about overstaffed.
the software that caused the crash was open source and there are currently 6000 geeks working on the problem as we speak... -- no news is good news! --
With any large roll out, if only one person is at fault for a fiasco like this, then the project mas mismanaged. They should have had a plan in place to backout the change.
if they just called this MyNewspaper.NET this would never have been news. Hell, they could have blamed it on a "routine update."
seppuku.
I was pretty sure when I read the thread title that this was going to be chock full of posts containing bad jokes. And what horrible jokes they are. Truely awful. Bad, I say.
Well, if I was in management.. I would find the programmer responsible, and have him snipped!
Do daemons dream of electric sleep()?
Simple enough.
Take responsibility and ownership of the problem. Don't make excuses, but give real reasons.
Fix it..do whatever it takes, even if it means working over a weekend.
Write a good post mortem, explaining how th e fix is different from the original problem.
And hope to god that your management is understanding enough to keep you on.
This is comong from a guy, who in 1997 blew a $100,000 test weekend by kicking off the systems tests by loading the wrong generation of tapes.
I took the blame, and expected to lose my job. But I knew that the right thing to do was to try to recover from the problem. I stayed in the office from 1:00AM Sunday to 10:00AM Monday morning rerunning every job and report and proving out the results.
Not only did I keep my job, but I got promoted a year later. I made a name for myself that weekend....sure I could f*k up, but I work hard to keep things right for the company.
wbs.
Huh?
Where was the phased or parallel deployment?
You don't just change a system like in a weekend. There WILL be problems, so you have to have ways of dealing with it. Maybe that means flicking the switch back to the old system if it fails, or maybe it means running with degraded capacity a while, but whatever it is, it's dead-in-the-water is not your Plan B.
Forget thrust, drag, lift and weight. Airplanes fly because of money.
The foreman responsible for the error wasn't fired, to the surprise of almost everyone. The owner was asked why the guy wasn't fired. He answered something along the lines of,
I've had coworkers who made major bugs that crashed servers and workstations and caused a lot of downtime. This is because they wrote sloppy code in a hurry and never bothered to check it. Management usually wants faster turnaround time on projects.
So your choices:
Plan A: Blame managers for forcing you to work under stressful conditions that lead to a workplace hazard (stress) that caused you to make the error. Cite that you had to work a lot of overtime and the lack of breaks and sleep caused you to miss a major bug.
Plan B: find someone like me who takes their time coding and have them look over the code and fix the problem for you. Sometimes another pair of eyes helps to find things you've missed.
Plan C:
Go to work in flip-flops, a Hawaiian shirt, sunlasses and tell everyone you are on vacation. Make Pacman noises, and talk to your invisible friends. Claim insanity and see if that works.
Plan D:
Start looking for another job ASAP.
Remember, Slashdot does not have a -1 disagree moderation, and no, troll, flamebait, and overrated are not substitutes.
Even after an editor assured her that she wasn't alone, a Stickney woman had to ask one more question to ease her mind.
She wanted to be sure that her missing Monday paper had nothing to do with the fact that her Sunday Tribune had been labeled "Final" edition.
Yep... sounds about like the end users I know.
Heh, that's why I didn't get the paper until about three in the afternoon.
"The condemned man received a jeweled dagger from the emperor. He selected as his second a faithful friend, received official witnesses, and plunged the dagger into the left side of his abdomen, drew it across to the right, and made a slight cut upward; his second then beheaded him with one stroke of a sword, and the dagger was returned to the emperor."
http://www.bartleby.com/65/ha/harakiri.html
If you aren't a perfect human being, you shouldn't be coding.
ogg
Black cat, searing pain, flames...? I must be in Heaven! - Homer Simpson
One of the benefits of working for a big company is a QA/UAT department. You have an entire department of people lined up just to test your shit. And, usually this type of job makes a person very anal. They log defects for just about everything.
The person writing the code can unit test to his or her best ability, but it is really the job of someone else to put it through the wringer testing thousands of simulated real-world scenerios. Sure, a coder could do this testing. But a QA guy or gal is doing really well if he makes 3/4 the salary of the guy who wrote the code- so a divison of labor only makes sense.
Not to mention the person writing the code makes the worst tester in the world. You only test it the way you THOUGHT people would use it. So, while a coder is perhaps the one who created the original problem, the real fault is in whoever let this slip through to production. Assuming, of course, that it wasn't some kind of time-bomb easter egg that would have been impossible to test. Although, good QA testers should alter their system date/time when testing date sensitive routines.
Good planning would have had an abort procedure, so the show would go on. Everything changed should be undone if it did not work. They could figure it out after the paper was printed.
Errors are inevitable. Good planning and implementation keep you from falling on your face even when you publish seven days a week. It's not the coder's fault.
Friends don't help friends install M$ junk.
...featured the tagline:
;-)
"To hell with shoes that shit to sprint..."
F7 could've saved this guy his job.
IronChefMorimoto
Send the coder to the Open Source world because no one is going to pay him to code anymore.
And send his supervisor too for not testing the system properly before trying to roll it out.
George Bush + Linux = "I will not let information get in the way of the fight against Windows"
off-by-one-million error.
"The crows seemed to be calling his name, thought Caw."
If your theory is different from practice, then your theory is wrong.
Mind you, here in Perth we only have one daily newspaper and it sucks, so I can't imagine getting worked up about a failed delivery.
How Would You Handle a $1,000,000 Coding Error?
Frankly, I can't believe anyone would pay $1M for a coding error. Hell, the guys I work with make coding errors all the time, and practically for free!
(That's free, as in beer.)
Noone [in their right mind] orders a brand new paper publishing system from a single consultant. The software probably was priced in several million dollars. Somewhere between the components something broke. For example, the file format that the publisher produced was rev. 2.1, but the software at the presses side was only aware of rev. 1.7 and below... If the coder only tested his code with the "other" piece of latest revision, he would never see any problem; and it is not his guilt that in real life the real customer uses some obsolete stuff that isn't compatible...
This kind of problem is clearly of administrative nature, of a system design and of checking which pieces work with which other pieces. Clearly, blame should be assigned to non-existent QA procedures, insufficient unit testing and [obviously] inadequate integration of components. The coder is nowhere here, it's all system design and QA stuff, realm of managers.
I'm a programmer for a large, (US) national newspaper chain and screwing up the publication cycle is somewhat more common that you might think.
/dev/null was deleted and the backup systme had been down for 6 mos. and take out $50,000 - $100,000 in advertising.
Most daily newspapers produce various editions, between 2 and four, and I've seen a couple of times, where only one edition is printed due to "codeing errors" (like the 1 billion seconds from the epoc thing - my personal favorite).
Of course the vendor had to be called at the $500/hour emergency rate to fix their own error.
Once I saw a print pre-processor go off line because
The call daily newspapers "the daily miracle" and when you look at some of the computer band-aids they have producing them, you can see why.
Try before you buy.
REnew there indemnity insurance now, before they make a claima nd there premius go thru the roof :D
...he'll be fine.
Does anyone know what software package they were using? Adobe's integrated publishing system? Quark's system?
I know their online operation is run with an in-house written J2EE application called "Oxygen," programmed by their technology arm, Tribune Interactive.
I had heard that their print stuff was closely tied and integrated with the online system. Could it be that it was one of their internal programmers who screwed up, or was it an outside company?
Well almost every upgrade of any size these days comes with a presigned agreement that the company making the software isn't responsible for anything that takes place. So talk with the lawyers then if they say it's not a problem go ahead and take the responsibility. The customer would have nothing to come back at you anyways.
Once that is done bust ass to fix it and make them happy. And then spend a few weeks creating new processes on how to do major installs that would catch this problem.
If you're company is responsible for it then they should have made so much in profit that they can eat a $1,000,000 payout if thats what the courts decide. You still have to fix it.
Mistakes happen and anyone working in deals of this size should be very well prepaired for the onslaught to follow
Blame QA... they should have tested it better. :)
Mucky Source Code Errors certified I bet :D
AC comments get piped to
Find a new line of work.
DISCLAIMER:
I don't believe what I write, and neither should you.
Like homer once said it
"Blame it on the one who can't speak English. Ah Zutroy how you've saved us all"
Where is all the *testing and validation* that should have caught it?
Where is the *fallback plan* if things went wrong?
If all it takes is one line of code to bring an operation to it's knees, then that operation has one hell of a lot more problems than one coder who makes a mistake.
There's not really one single point of blame for things like this.
There's the coder, the QA team, the code-reviewers, and I'm sure management failed in many many ways.
If something goes wrong at the plant, blame the guy who can't speak English. Ah, Tibor, how many times have you saved my butt? -- Homer's rules for success in the workplace, ``Marge Gets a Job''
I smell a Slashdotting.... Some poor IT bastard is fired.
I'm not normally an irrational zealous dickhead, but I figure "When in Rome..."
Which operating system was being used?
Go not unto/. for advice, for you will be told both yea and nay (but have nothing to do with the question)
True story. I was working an assignment as a tester for Microsoft. I apologize for the use of variables, rather than names, but I don't want to get sued for breaking NDL. There was a deadline on the release, and if we missed it, there was a penalty of $1 per copy shipped. 20 million copies were due to be shipped on date X. The day of date "X", we realize there's a fatal bug that causes Product "Y" to crash after running any segment that lasts longer than "Z" minutes. Somehow, I'd completely missed this bug. I have no idea how, don't ask, but I completely missed it. We even checked back 3 months worth of revs...the bug was sill there in each one. Of course, the product was late, costing Microsoft a whopping $20 million. What did I do?
I was "allowed" to resigned gracefully, quietly, and have learned a valuable lesson about software testing: It's not whether you miss something, it's whether or not someone else will find it in time to cost you your job. (nods sagely)
-The Libra
"Please be patient--The future will begin momentarily."
I only recieved the business and classified sections this morning and I was wondering why my neighbors would be interested in half a paper... =) And for the Tribune Company, a million-dollar mistake isn't *that* costly in relation to its billions of dollars in yearly revenue...
Change your name to Mud, move to Iceland, and buy a lemming farm. The property next door to mine is for sale. It's nice and quiet here, and you can't possibly make another mistake that'll leave a big company hunting you down for their million dollar losses.
Serious? Seriousness is well above my pay grade.
Remember how exciting it was when eBay went down? When Amazon and Yahoo went down? And when the freakin' root servers started going down?
-----------------------
Freedom or Evil: Freevil.net
G. W. Bush says, "You decide!"
PLEASE let this project be something that was outsourced and coded in India!
LK
"Hi. This is my friend, Jack Shit, and you don't know him." - Lord Kano
If I where him, I would leave the USA and not look back, because I don't think that the guy has $ 1,000,000
-Joey
Microsoft.
Just blame the CIA. Seems to be working for other folks.
This is my post. There are many others like it. If you don't like what you read here, go try one of the others.
Just like Colin Powell, always have an exit strategy. For me, its as simple as keeping good backups, and using Visual Source Safe to keep versions. Rolling back is easy, and if shit happens, I can get everything back to how it was within an hour.
The Doormat
If you're not outraged, then you're not paying attention.
They're always hiring. And if you screw up a burger, it only costs the company about $0.17.
Be sure to lube up before bending over....
Run. Run very, very fast.
Yes, I am female. No, I do not want to date you.
I'm involved with "coding errors" like this everyday. I work for the Air Force. We call them something different.
"Who are in control, they are not in control of anything - they don't even control themselves!" - Glen Beck
Need reliable software? Use GNU Ada.
That situations such as these seem to produce massive amounts of retribution on employees. For me, and I suspect many others, the larger outages are rarly the deal breakers with who we deal with. An example is my phone company. It took about a week for them to fix a problem with my line after a large thunderstorm - I don't have any ill-feelings about them because of this. I hate them not because of this, but because they have an obnoxious policy of calling up their users to try hocking upgrades and equipment on them. It's things like this, or their half-assed customer service that took forever to get past the phone menus and to a human being that turned me off them.
Here in France we had the ticket reservcation
n ce .train.glitch.ap/
system crash last thursday July 16 on Friday morning
when everyone wanted to go on vacations, the tickect agents
couldn't sell tickets as the computer system was broken. They
use Microsoft and it seems they were updating the software and
it all fell down like a house of cards, so they sent all the kings
men to fix it and by Monday, they had suceeded in getting
previous version.
http://edition.cnn.com/2004/TRAVEL/07/17/bt.fra
I wonder how much this cost the company and who pays.
You know they are a state owned company and will soon be
going up for sale, or at least part of it.
From the guy in Paris
This is a test!
Nick Leeson managed to lose 1.3 billion dollars mis-investing funds. A $1million dollar bug is pretty candy-ass next to that.
If I were a multi-billion dollar software giant, with FUD and spin.
If I were a solitary pleb software engineer, with plenty of alcohol.
Seems to me that the problem is not with the programmer but more with the QA team or QA procedures. I have worked at many large companies in the utilities marketspace and have observed that shoddy QA practices lead to problems of this nature. If someone points a finger in the direction of a programmer regarding a coding 'mistake' that programmer should hurl them down the corridor to see the idiots who tested the code and certified it for promotion to the productive environment. Pass the buck - its easy and it works!! Seriously - can we really believe that a company of this size does not have a QA team / procedure in place to safeguard a seemingly critical process?? Weak...
Some lowly programmer gets fired, and management gets a bonus for "seeing the company through a difficult time."
That's how it usually ends up, anyway.
Programmer: "How many parachutes would you strap on and check before you jump from a plane?"
That said, I *always* point out when people make these bad decisions. I attempt to get them to acknowledge that they know the risks, and in many cases they do something (usually not enough to be safe) to lessen the potential dammage.
Hmmmm. THere was a place I used to work where we had a mirror of our production servers. Now, so you don't go and say, "well that's all good and well, but whart about 100 servers..." we had 30 production servers and 15 for testing... If it does not work on the test system, don't bring it up on production...
"Who are in control, they are not in control of anything - they don't even control themselves!" - Glen Beck
You insufferable ass -- you just slashdotted Illinois.
May we never see th
Set bios to boot from CD and restart.
Seriously, as others have said, it's a lot more important how you handle recovering from the error than the fact you made an error. Everyone makes errors, not everyone is able to follow up appropriately and with grace.
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
"Do the Right Thing. It will gratify some people and astound the rest." - Mark Twain
it's never the fault of anyone in the executive club.
It's a well documented fact that only poor people make mistakes. Rich people are needlessly persecuted by the government.
As long as I keep checking in my code as someone else, I won't have to.
Show me on the doll where his noodly appendage touched you.
Welcome to the world of delivering the papers you used to help produce.
Many of us are just one misstep from being out of a job. Swallow your hubris for a second and consider what would happen if that last backup you did didn't restore, or perhaps that last "minor mod" to the code you wrote caused a kernel panic. It's sobering.
Everyone makes mistakes, good teams try to find those before deployment.
Where was the testing?
Who decided that there would be no testing?
Who decided that they would simply deploy the thing with no plan B?
Obviously the PHB, you just need to point that out to the VP and you can have the PHBs seat.
Less look fast, more go fast.
So the paper can deliver every day for 158 yrs using mechanical printing presses ~ except where natural disasters occur ....
The printing problems at the Chicago Tribune were related to efforts to upgrade computer equipment used to produce the newspaper, Malone said. The Tribune acquired customized software for the upgrade from an outside provider, and it contained a "coding error," he said.but as soon as computers are involved their printing press has morphed into a computer system. I wonder what provisions to *test* the upgrade before use where made?
fail to recognise newspaper as computer system?it would be easy to blame the developers and company and there should be some recognition of responsibility for technical accuracy. but what about the newspaper. they have made a fundamental mistake in not recognising that printing press + computer = computer and let their newspaper system fail at the mercy of coding mistake.
It seems while the paper can handle *mechanial* failure (158 yrs, 1 non delivery) it has yet to grasp *software* failure.
peterrenshaw ~ Another Scrappy Startup
Bugs are the user's problem.
Any advice for the poor schmuck who's going to get the blame?
:p
Find the tallest building you can find and jump...
All kidding aside, I would say the scapegoat should write this off as an unforseen flaw which is now resolved... It's worked for a certain Redmond company for quite some time...
Business \Busi"ness\, n.;
A scam in which all people involved perceive as beneficial...
Get a faster machine then...
God I feel for them but the first question that popped up in my mind was "How did the backout procedure fail?"
I don't want knowledge. I want certainty. - Law, David Bowie
EVERY out of work QA geek should be polishing their resume and hounding those stupid nimnos to hire a proper QA team.
"A million bucks pissed away from lack of testing? Cool: give me $500k and I will guarantee that it won't escape the test server with a single showstopping bug. sure: you just blew $500k, but think of it as SAVING $500k over having a catastrophic faliure...."
If they turn you down, they're dumbasses.
RS
Shoes for Industry. Shoes for the Dead.
Show how nice a company they are by letting him keep his job. And next year quietly outsource his job.
Bonus points for replacing his co-workers too....
It doesn't make any difference if it's a broken punch or a whole set of dies cracked down the middle ($4000 for a 6 inch section, over 60 inches you do the math)...
"If you say 'oops', it's OK."
Did he say Oops?
Seriously though...shit happens. That's why you don't bill employees directly for the mistakes they do. Suck it up, learn, and move on.
--
BMO
Bah, this is absolutely nothing compared to the coding error that brought down Canada's Royal Bank last month, leaving millions of customers without paychecks, access to their accounts, etc.... And this too was attributed to human error, but had far more drastic repurcusions than not getting your morning paper, and cost RBC a heck of a lot more than a million dollars.
It's better to burn out than to fade away
Don't mention you were responsible during the interviews for your new job.
i hotdog.
Either you have it or you don't. How does this work?
"Who are in control, they are not in control of anything - they don't even control themselves!" - Glen Beck
Advice?
"Watch the corn-hole."
The old system was working! But, they didn't follow the rule, and look what happened!
a $1,000,000 coding error... well, must be a farking big error considering i'm only a university student. So i wrote something that somehow destroyed entire computer networks through my university? I'd prolly run very fast!
Giving IE users a taste of their own medicine since 2005 - http://pods.-is-a-geek.net/
"Thank fsck for EULA"
it is only after a long journey that you know the strength of the horse.
the management got to understand this simple way of truth of life... SHIT HAPPENS !!
I just hope and pray this was a job they outsourced to India.
Everyone knows Visicalc does not run on the C64. Maybe they should have looked that up before popping in a new tape.
Need Free Juniper/NetScreen Support? JuniperForum
Next time your PHB dismisses testing as an "unnecessary waste of time and money, just write your code carefully and you won't need to test," resign.
My new
A book lying on its side is stable and a nail standing on its end is unstable.
Even though the loss is high the software might not be as unstable as it appears. It may be a very peculiar set of inputs that causes a failure. This is hard to detect in testing. Imagine a chair with 4 legs but one leg is telescoping with some friction. Someone can sit perched over any of the legs but the chair falls only when one leg is heavily loaded.
Know your pads. One time pad: good for cryptography. Two timing pad: where to take your mistress.
That gives new meaning to /.
Did Steve Bartman become a programmer?
newton62 (56617) Karma: Bad
use it to sell Software assurance.
MS does it all the time.
old gnomes joke
1. sell buggy software
2. it breaks
3. sell software assurance
4. profit!
----
http://www.hellection.com
... so it won't hurt that much. Oh, and demand a condom for your bosses (if they care to listen): you never know how do PHBs waste their money :P
Boy, there are two things you need to know in life:
Good idea boss
It was like that when I got here!
If I've built a $100M system that then later is shown to have a $1M bug. I could pound myself that I cost the company $1M or I could be a bit more positive and say that I still made the company $99M.
Engineering is the art of compromise.
"Always blame it on the guy who doesn't speak English"
Homer Simpson
It wouldn't have happened if they had run GPL'ed software....gotta be a microsoft bug.
How about AT&T's siebel upgrade debacle? 100 million in lost revenue, and the issue took over 3 months to get resolved.
Unfortunately, I can't find the orignal source, so here's my versions:
:)
A high level minister of the USSR is on his way out and comes to his replacement to offer advice. He hands him two letters and tells the man "If you ever get in a situation that you cannot figure out how to get out of, open the first letter. If you ever get in another, open the second letter."
Well time passes and the new minister discoveres himself in a position from which there is no escape, so he opens the first letter. It says: "Blame everything on me." He does as it says and blames everything on his predicessor, and all is well. Some time later, he is again stuck with no means out so he opens the second letter. It says: "Get a pen, sit down, and write two letters."
So I guess it just depends on which letter applies to you
In all seriousness I'm not sure what to do in a situation like that. My level of responsibility doesn't afford me the ability to make mistakes of that magnitude.
We don't have no steeking bugs ... they're called features.
"The printing problems at the Chicago Tribune were related to efforts to upgrade computer equipment used to produce the newspaper, Malone said. The Tribune acquired customized software for the upgrade from an outside provider, and it contained a "coding error," he said."
See what outsourcing to asia does?
He should just toss in some comments near the bug that have some one else's name in them. [evil grin]
"...spokesman who estimated the cost to resolve the problem at 'under $1 million.' "
All of my software problems cost under one million to resolve. With open source, it's often under one million by quite a lot.
That's a colossal amount of shit. Man, I wouldn't want to be responsible for that much shit.. Where would it all go??
Besides, how can you even ascribe monetary value to shit? I suppose you could say it takes $1 mill to remove the shit, but... that's a hell of a lot of money.
I think the shit would need to be the size of a small city.
Ugh, ok. Need to stop thinking about this.
Works everytime:
http://www.sanecomputers.com/articles/humor4.htm
slashDOTTING one's wrists?
First, buy one of Kent Beck's book and start testing. Secondly, you need to get management's buy-in to the philosophy of test-driven development. You need to get them to realize that the cost of a single failure of this magnitude (both technical costs and costs to the company's reputation) is far, far greater than the cost of taking the time to test properly. Human based testing is not adequate; we make far too many assumptions and subjective decisions. If you're unsure, then wait. Don't deploy and pray for the best. As an added benefit, test-driven development actually builds developer confidence and allows for much more rapid development.
I haven't seen all of the details of the problem, but it's possible this was an issue with business and/or technical requirements, failure to review a third-party's work or perhaps worrying about a deadline and the implications to one's invoice. I've personally seen far too many cases of management bringing in consultants to do the work and then not reviewing the product that's been delivered. Mistakes that have cost companies millions of dollars, forced them to sell to new owners and most importantly, disrupted families.
In any case, I really hope the Tribune does a thorough root cause analysis, identifies issues with their process and implements real change. Don't look to place blame (I know, it's a public corp), identify assumptions, locate weaknesses and come up with solutions.
1. Make a dopey "error" that costs a million. Get mentioned on Slashdot. 2. Make heroic effort to get them back up and running. Get recognized for brilliant skills. 3. Write book about the whole affair. Get book mentioned on Slashdot. 4. Profit!
The difference between spam and poop is that you don't have to dig through septic tanks looking for real food. -- Me
In the aerospace industry we deal with very expensive space equipment. As a result there are procedures that must be followed so nothing terrible happens. You can see where this story is going...
Imagine a satellite, nearing completion, bolted down , and ready for final inspection. Joe Blow forgets to write in the change log that he took the un-bolted the satellite from the base. Workers come in the next day, do some work after checking the logs, and... the satellite tips over. OOPS... a billion dollars well spent. That is 1,000,000,000.
"Uhm, boss, the good news is we finshed the satellite yesterday and... I don't know how to say this, but our last two years of work... well, I sort of... well, I tipped it over and it's destroyed.... "
Or how about the contracts guy who forgets one Zero on a contract. Instead of ten million, the contracts reads one million. Of course everyone misses the zero... except the people PAYING. Contracts are signed and oops... "we want to start a new contract, we sort of forgot to add a zero." To which they reply, "Fuck off, you signed it..." and prompty save the company 9 million dollars.
It truly is a site to see. The speed at which they print is fantastic. A minimum run on many of them is 20,000 copies, in the time it takes to spin up and spin down, that many will have come off.
This is necessary too, if we wish to efficently print the massive quantity we desire. There are a lot of daily newspapers. Even in my small city there is at least 8 I know of. An old mechanical pres simply wouldn't be able to keep up. Never mind printing speed or anything else, setup time was a bitch. You had to have plates made to stamp your text on the page. These then had to be loaded and calibrated for each run that was to be done.
Now it's all electronic. At the minimum, you place the reference prints under a camera, and normally the layout files themselves are loaded in to the press. It then can go to work right away.
I know it's kind of retro-geek cool to bag on how much harder technology makes everything and how much better it was in "The good ol' days" but that's not usually the case. Old nechanical presses simply cannot compete with the speed of computerised presses, which are necessary to operate with the speed and efficency that is demanded today.
Bad news: We missed printing half of our papers.
Good news: Rainforest saved.
paintball
Seriously. This was a greal learning experience.
MSBPodcast.com The opinions expressed here are my own. If you don't like 'em... Think up your own stuff.
"Next time, watch that divide by zero."
...hide!
Oh, I hear they are auctioning those Y2K Bunkers on eBay. I bet they are wonderful this time of year.
Career Limiting Event!
Either the rest of the world ridicules us as a bunch of ignorant louts or we get "worked up too easy" if we bitch when our morning paper is late.
"Eve of Destruction", it's not just for old hippies anymore...
Since "the programmers should be able to test their own code". There is an interesting trend toward this being marketed to middle managements. QA is *expensive* and produces no income and only *slows things down*. Obviously, if you simply increase the Quality of the code the programmers are producing, you don't...need...QA!
Because Java is for woosies.
paintball
The Tribune also owns the Cubs baseball team. They've been failing for years and nobody seems to get any blame. The same Tribune owns the Cubs broadcast outlets, TV and radio, and have had some of the worst announcers ever working for them. Nobody seems to care about that either. What makes you think they're going to even consider holding someone accountable?
Software testing is boring boring boring. You have to try things out again and again after each change. Modules that haven't changed gain confidence in the face of changes and might not be tested, but omitting tests can end up being the Achilles heel. There can be an overwhelming desire when a project nears completion to just get things done and over with. After all the hard problems may well be solved and it's all down to seemingly inconsequential details.
These days programmers have a Sword of Damocles hanging over them. Once they finish a major piece of code they may have a hard time finding new work. The economy has not lived up to forecasts of more jobs. Outsourcing has reduced computer opportunities. Management of many companies do not see new uses for computers. Off-the-shelf programs abound for almost every aspect of computerized work.
Stress may distract software engineers enough that someone will make a major mistake.
Know your pads. One time pad: good for cryptography. Two timing pad: where to take your mistress.
God bless Mac OS X, loads PDFs in a second. (I sure wish my fucking double-the-gigahertz PC would)
mogorific carpentry experiments
Most project managers (especially ones with no technical experience... who shouldn't be let near a technical project) plan their projects with timelines with rose colour glasses. They assume there will be no coding issues discoverered in testing. Or worse, they do, but then let scope creap come into it, and borrow time from testing for the new items introduced in the scope creep. Bye bye testing time.
Mind you, I have also seen QA managers who believe that the testers only need to understand the software, and not the business where the software is to be used. This has sometimes leads to problems in end use. In any case, I tend to blame poor management before I blame the little guy. Projects like this are big enough that the process should have been able to catch things like this... unless the process was flawed.
My opinion... ready, set, slag away!
-- I ignore anonymous replies to my comments and postings.
"Never ascribe to malice, that which can be explained by incompetence." Roedy Green
I have two million of errors and omissions, and a five million umbrella policy. If I make a million dollar error... it's covered.
These levels of coverage are pretty standard, so I'd be a bit surprised if the person responsible actually had a problem.
I'm willing to bet there will be an opening for IT manager soon.
Well.. maybe. Or Maybe not. But Definitely not sort of.
How would you handle a $1,000,000 coding error?
Just tell people you also wrote the code for the damage cost estimation...it was really a $100 coding error.
God bless Mac OS X, loads PDFs in a second.
Well that isnt the case for my Beige powermac upgraded to a G4 500MHz. Perhaps if I turned quartz on. btw are you using acrobat reader or Quartz?
If you want a real challenge for a PDF reader, www.mta.info. Download the NYC subway map.
--- Justin Dearing http://www.justaprogrammer.net/ We're just programmers.
The solutions to these types of problems is simple coding practices and a *FULL* backup of the software system and all data before the new system is put into use. If the code was painfully simple and obvious, the bug might have been found in time.
However, what is 20 million dollars to a company 'worth' maybe 10,000 times that or so....
Update résumé, maybe???
Do as the BOFh would do! :-p
It's embarassing, but not such a big deal. If you work in a big company, you group probably lost a deal worth more than $1M because it couldn't deliver some feature on time. And companies as a whole - oh boy. Microsoft stranded a Navy carrier at see before. If I had a dollar for every million lost because of their security/stability problems, I would drive a porsche by now.
I hear Southwest Airlines is good at getting you to a new location when something goes wrong...
With a shotgun.
But there is another kind of evil that we must fear most... and that is the indifference of good men.
As you pointed out, QA should have caught something this basic. There had to be a lot of careless decisions made here, and none of them are necessarily any one coder's fault. Blaming a "coding error" is simple, and makes people forget that a manager didn't do their job correctly. I've seen this particular scenario played out a dozen times before:
Last Monday Suzy Manager shouted at her team, "The schedule says we install on July 18th, so this damned product damned well better be installed on July 18th, you all got that?!"
But the vendor's ship dates slipped, and testing dates got pushed back, even though there was nothing particularily important about July 18th; except for Suzy Manager's promise to the CIO that she'd get WhizBang 2.0 installed by July 18th. And she would, too -- she had 25 points on her review riding on that very promise.
By the 14th, when a new patched version arrived that fixed the bug they discovered on the 10th, Suzy was visibly distressed. "They damn well better have that transmit bug fixed, they've been dragging their feet long enough."
Perhaps the testers just kept testing the version from the 10th instead of upgrading to the version of the 14th. It was beautiful on Saturday, so maybe the tester called in with a bad case of 'weekend flu.' Perhaps they got the patch late Friday afternoon, and the vendor swore up and down that it was just one little bug, our guy knows it's fixed, don't worry, it's better now. Whatever -- Suzy was under the gun, so she simply said "ship it."
Regardless, some nameless coder is flapping in the breeze today. Suzy is probably running around the IT department at the Tribune screaming, "we'll never buy code from those bastards again, I swear!" in a vain attempt to deflect criticism from her department.
But the CIO usually knows better, and Suzy knows the CIO knows better, and she's already sent out her interview suit to the cleaners. Even so, she'll feign total surprise to her department as she boxes up the little wooden carving she picked up during a drinking cruise to Mazatlan a couple years ago. A couple of tears later, she's interviewing over at Microsoft Consulting Services.
Or, maybe I'm completely off the mark. Perhaps they've been testing the code for a month and it's worked fine, but they installed the new code with the old libraries, or the new libraries with the old code, or the destinations were SP2 with some new security turned on. Of course, the QA department should be testing the installation packages as well, but we all know that in hindsight, right? As Yogi Berra might once have said (were he an IT manager,) "In theory, there's no difference between the lab and production, but in production there is."
John
And "Errors and Omissions" policy.
- Tjp
I am in wallow with my inner money grubbing capitalistic pig. ... Oink!
There are no $1,000,000 coding errors. If it cost that much, then either it was an error case that was enforceable, or the proper quality control and risk management steps weren't taken.
However, what is 20 million dollars to a company 'worth' maybe 10,000 times that or so....
It's still 20 million dollars. Ask any investor what he'd think about his company losing 20 million dollars for not catching a bug like what happened in that post.
In Soviet Russia, Corn Saves You!
1. Save The Corn.
2. ???
3. Profit!!!
All Your Crop Circle Are Belong To Us!!
OMG, think of the Children Of The Corn!!
et cetera et cetera
Free as in mason.
Erers and Omisions Insurance :)
$700 a year for me and my staff. I wouldn't code without it since we are a "Microsoft Shop."
After downloading the subway map PDF file (about 9 seconds on a T1), I opened the PDF file using Preview on Mac OS X 10.3 (Panther), by double-clicking on the PDF file icon.
Slightly over 2 seconds to fully rendered on an iBook (G3 processor, and relatively slow, nowhere near a G4 let alone a G5).
Gotta see what the G5 at work will do!
In theory, practice and theory are the same. In practice, they rarely are.
The coding error, if it existed, was minor.
The serious error was in switching to a new system with such clearly inadequate testing.
I write software for a company that handles $45,000,000+ of client cash every week.
A mistake I made in May (discovered this very day, by yours truly) had backed up about $400,000 per week.
Did I get stomped?
No.
A bottleneck had been identified, repaired, and eliminated!
Behold the power of positive thinking.
Writers imply. Readers infer.
rookie mistake
i used an itoa when i should have atoid...
Lack of skill is what got him in this mess.
Learning to mitigate the damage of a colossal coding screwup is the experience he acquired as a result.
paintball
...they will next time.
You only make that mistake once.
Or the guy who signed off on rolling the thing out without extremely thorough testing beforehand. The IT / Software end of companies is a lot like animals in this respect- cut the head off and you prevent the body from doing anything grossly stupid.
:P You get what you pay for! :|
Start with the CTO and work your way down. If it's a software problem, why wasn't it discovered sooner? Who was in charge of QA? Who was in charge of making sure QA did their jobs? Who said YES WE CAN DO IT!, lying out their ass?
The fun thing about capitalism is greed and/or the desire for profit leads to systems like this being built by the lowest bidder.
Nice map, but that's got nothing on the VTA System Map.
It sounds like bad planning and testing to me. Why did they not do a trial run on small number of copies while still having the old system doing production? After the switch, why did they not have a plan to do a fast back-out if there were problems?
...as Communications Minister in the Australian government any day now
insecurity asks the wrong question irritation gives the wrong answer
It's a joke, it's funny (if anything) not interesting. Hence the :).
Haven't they heard of cdrom.sneaker.net?
"Under a $1,000,000" is a mere bagatelle, the failure to check for arithmetic overflow cost the European Space Agency two Communications Satellites at well over 600,000,000 Euros. For more such fun see Forum On Risks To The Public In Computers And Related SystemsForum On Risks To The Public In Computers And Related Systems
$1,000,000 coding error? Pshaw. Windows is a multi-billion dollar coding error.
I was in charge of a very similar upgrade albeit at a much smaller paper. We had plans in place to build the pages manually if the system didn't come up in time. It seems that the people at the tribune have gotten so used to automation they forgot that they didn't use to have these systems. I am surprised that the second things didn't start to go right they didn't have people instantly creating the paper the old fashioned way. The real cost isn't in the code but in the advertising dollars they won't collect because it didn't get out on time. That hurts.
The Queens borough bus map has that beat. Well the VTA map wins the beauty contest, but in terms of complexity based on number of lines Queens county has it beat.
--- Justin Dearing http://www.justaprogrammer.net/ We're just programmers.
Especially the black box testers
If you think
It's never an individual's fault. It's a breakdown in the QA/FVT/review structure. Is it the person who coded it's fault? Is it the team that reviewed the code? Is it the author of the FVT tests? Is it the person in charge of QA?
What's that you say, this is all the same person? No wonder you had the bug to begin with...
int func(int a);
func((b += 3, b));
...then "look over there!" pointing out the window, then run out the door ;-)
These posts express my own personal views, not those of my employer
According to Steve Jobs, programmers are 100x more productive in Objective-C (the world's best programming language) and are 100x less likely to make coding errors!
Blame the guy that doesn't speak English (or, in the case of outsourcing to India, the guy the *does* speak English)
==========
Intelligence should not be rewarded; ignorance should be punished
==========
I must have left out a period or something, shit I always do that! /Michael Bolton
Here is the full text of the article in the Tribune:
A story we never thought we'd print
By James Coates
Tribune computer columnist
Published July 19, 2004, 6:40 PM CDT
Nothing built by humans can go wrong in as many ways or with as nasty an outcome as a computer system.
The people who create the Chicago Tribune started relearning that fact about 4 p.m. Sunday when they noticed that nothing was getting through as they attempted to beam the stories, artwork and ads from Tribune Tower to the Freedom Center printing plant.
About 13 hours later, they finally started printing a 24-page version of Monday's Tribune that should have already been landing on their readers' porches.
It was a misfortune that most people in the news business don't ever expect to experience. Newspapers do not miss days -- and Monday was close.
The only time the Tribune failed to print was during the Great Chicago Fire of 1871. That time, the lesson was that nature can be fickle and dangerous.
Now, the paper has learned that the same goes for the computer technology that has graced the industry with unparalleled productivity since the 1990s.
Business computer systems are cobbled together as row upon row of workstations, each running an operating system based on an estimated 50 million lines of instructions. In turn, the worker bee desktop computers connect to the queen machines with their own millions of lines of code in a different language.
An endless nest of wires, cables and even radio signals move instructions at light speed between the central computer and the workstations. The main computer also talks to all the peripheral devices needed to accomplish the mission.
The peripherals can be banks of hard drives, storage bays, printers, scanners, cameras and specialty devices as diverse as a pager or a printing press several stories tall.
The certainty that each and every one of these massively complex systems will crash haunts the people charged with keeping this thoroughly digital world up and running.
Those people are engineers, and so they often reduce it to numbers.
An often quoted study by Carnegie Mellon University computer scientists studied 30,000 software programs and found five to six defects per 1,000 lines of code.
And this is for finished software sent to customers.
When writing new programs, there is typically a defect in every 10 lines of code. About a half dozen defects per 1,000 lines remain after a process of checking, rechecking, cross checking, testing, retesting and finger crossing.
The hubris of computing becomes clear as one realizes that each of these errors in code branch out with instructions to millions of other lines of code. Quite often, they find pathways never before taken by that particular program.
Collisions occur on these pathways and trouble is spotted. Maybe it can be fixed or maybe technicians can only perform a "workaround" that can't be guaranteed.
Dick Malone, the Tribune's senior vice president and general manager, said that around 9:30 a.m. on Sunday technology crews started a planned upgrade to increase the newspaper's Sun Microsystems servers from so-called 10K models to 15K machines.
To do this, experts from the company that makes the newspaper's core Windows-based publishing software, Denmark-based CCI Europe A/S, needed to install upgrades of its Newsdesk brand software that the Tribune and other clients use.
Malone noted that they checked and rechecked, tested and retested all day. Everything seemed to be working without a hitch. Then, they punched the button that was supposed to send all of the content for the newspaper to the printing plant.
Nothing arrived.
Frantic hours went by as deadline after deadline slipped while crews struggled to find a fix. Malone said he went so far as to start setting up the newspaper's pages on the art department's Macintosh desktops, hoping to get at least something printed.
I worked for an Internet company that had over 100000 products for sale in a database for over 24000 customers. One day, someone accidentally through in a DELETE FROM products WHERE nID > 00000 instead of 100000.
Noone fessed up, but the guy who was red as a damn chili with sweat beading down his face *might* have been the guy.
P.S.
Backup was 1.5 months old.
Only victims make excuses
It didn't take long on my 2 GHz Compaq/512 MB RAM. I think it has shitty Intel shared video, too...
Only took a few seconds at most for all the layers to build up.
When they came for the communists, I said "He's next door. Take him away. Goddam commies."
... that he didn't say it to the Vikings. It would probably read something like
Det er min første uge... (Danish)
Does anyone know if SCS is the vendor? They make custom newspaper software and mention having the Chicago Tribune as a customer.
... as many places, you're not allowed to fire someone for alcoholism or mistakes made as a result thereof, without first offering a rehab program...
"Why the hell didn't you see this bug?!"
"You smell funny!" *puke*
start looking in the help-wanted ads right away. Go get a newspaper and... Oh wait.
Ctrl+Alt+Del
(B) + (D) + (B) + (D) = (K) + (&)
Johnny Cochran anyone?
...it's a *feature*."
Step One: invent time machine
Step Two: travel into the past
Step Three: shoot yourself before you adopt the testing mentality demonstrated by a certain branch of VA Systems. Err, VA Linux. Uh... whatever the fuck they are today.
If you don't get the joke, try to use features late Wednesday evenings (or, for the bonus special, almost all day today).
Jesus was all right but his disciples were thick and ordinary. -John Lennon
While an individual may have keyed the erroneous code, they are not soley responsible for the error.
In a properly managed project there should have been significant testing.
There should have been a disaster recovery plan with multiple abort points and plans on how to rollback to a known good version if it was only a software upgrade.
If it was a full system upgrade, the existing system should have been available to cut across immediately should the new system fail.
The person at fault is the person who approved the implementation project plan without a real disaster recovery plan to back it up.
Sara
Designer, Gamer, Macgrrl in an XP World
" Any advice for the poor schmuck who's going to get the blame?"
Lube up.
Chicago2600.net more than a lifestyle, its a survival trait.
I don't know what could have gone wrong. I was using Microsoft programming tools and deploying to a Microsoft operating system. The chances of anything going wrong are 1 in a ....click.......zip.......>>> division by zero error.
If they had a clue, they would grow 10000 acres of canabis which;
A) grows 10000x faster than trees
B) makes 10x more pulp per acre
C) uses 100x less water.
D) stick it to the govt.
But would they ever do that? NOOOO coz there are no patents in the process to expoit and oh the trouble of the govt wackos like bush n old guys being so anti-canabis (to protect their buddies profits)
I guess they wouldnt want 100s of pot heads heading up to the 100000s acres of weed to take a few home, but what is so wrong with that OTH?
Liberty freedom are no1, not dicks in suits.
Hopefully the programmer has learned from his mistake and won't do it again.
Screw up once and it only shows you're human. Screw up again in the same way and it shows you're careless, lazy, or incapable of learning.
File under 'M' for 'Manic ranting'
QA's an attitude, not a department.
If more people would figure that out, the world would be a better place.
+++OK ATH
Especially in CS classes, that verybody's code was different.
It wasn't until Perl that it became apparent how different someone else's code could be.
paintball
One of the people who hired me long ago once said something that I hold true to this day. "The job of QA is not to find bugs. The job of QA is to verify that there are no bugs."
Laugh while you can, monkey-boy!
Competent QA people are definitely worth whatever you pay them. In fact, as long as the testers in question are not built like pre-Subway Jared, they are worth their weight in gold.
You cannot apply a technological solution to a sociological problem. (Edwards' Law)
The key issue here is that Chicago Tribune has a very limited IT group, really it's just a "systems" group. All of the other traditional IT functions have been outsourced to Corporate.
That was fine before the Times-Mirror merger, because Chicago Tribune was the top dog, and could bully all the services they wanted out of Corporate IT. But now the LA Times has that spot.
The good news is, the FUBAR situation on Monday morning was strictly a "systems" issue within Chicago Tribune IT, and not just a "systems" issue, but one where "systems" could direct all the blame to their vendor, which might help them keep their jobs a little longer...
The only possible advice here:
Work on your resume. Right now.
Cuz if you're not outta there RIGHT NOW, you will be very very soon.
"Champagne for my real friends - and real pain for my sham friends!" http://ericblade.postalboard.com/
Just standard Slashdot procedure.
I'd rather debug perl that most other stuff.
There's a lot of different ways of doing stuff, but the end result is generally quite compact.
If the person codes in a generally sane manner (i.e. not using gotos, or using for loops to do everything, or something strange like that), it's easier, I think, to debug perl than many languages simply because you have less lines you have to look at to figure out whats going on.
This is especially true when you're talking about parsing text, since you get to look at regex patterns to figure out what's going on (looks to hard to manage? get over it. Regex is a small, orthogonal set of commands).
That said, it's still hard if they don't name the variables very well or comment the complicated parts, or if they do either of these in a language you don't understand.
Mod me down and I will become more powerful than you can possibly imagine!
And everyone knows it. Why make such a clown out of yourself?
ITIL-conformant change management would have minimized the exposure of this problem.
Grab all the eMails where someone in management told you to cut a corner, or replied that they didn't want to spend too nuch time designing, or authorized fewer QA hours then should have been done, and print it all out, with headers, and forward it all to another account.
When they come after you, present it as if it you were trying to do it right, but somebody wouldn't let you.
If they fire you, sue.
Unless:
a) you work for one of the few companies that actually supports a real team atmosphere, or
b) Everything was done by the book, and you still screwed up.
When someone in an industrial field is forced to work 16 hour a day, 7 day a week, and has a mistake the company suffers the ramaifications, not the worker(or the workers faimly).
The Kruger Dunning explains most post on
:P
G4 550 powerbook. Loaded in about 10 seconds, the NYC subway map. I think your main problem is the fact that your system bus runs at somthing like 33 or 66 mhz, while modern systems run at more than double that, 133mhz. not to mention much faster ram.
moox. for a new generation.
Where I work (you'd never guess in a million years) we have this PERL code that someone wrote with subs... inside other subs... it's not an attempt to make a class or anything, it's just bad fucking code. The "programmer" has stuff like # This is important! and # This is garbage! at random places in his code, almost like comments but far too vague and never visibly in reference to anything. It's a fucking nightmare.
I _love_ PERL, but whoever wrote that code should be taken out and fucking shot.
Who uses printf, strlen, gets, and other "pound me in the ass with buffer overrun" functions.
you made a mistake!
you're famous!
woohoo!
some of us have made mistakes that lost companies far more than a million in a few hours. at least people are noticing yours, and probably least notably the fix.
surely someone will appreciate it.
this really just highlights something that happens all the time.
Well, although can't say the guy did a great job... if the DB was so important, why was there not a regular backup?
You are pointing out two problems taking place simulataneously.
One is a minor human error, but it is obviously an unintended act.
NOT having a recent-enough backup IS a serious issue. This issue has been pending for, as you say, 6 weeks, and it is a critical issue (if the data is valuable as you seem to imply).
You do not go around deleting all entries in your DB for fun, but you know some software is going to go bananas on you one day and start messing up with your DB, whether it is in such an obvious form as deleting all the records or simply altering them all in a subtle way that takes a while to notice... (change all prices from euros to dollars?).
A succesful project or business is much more than the sum of little individual acts. There is such thing as planning for things going wrong. And in this day and age, a database backup is no longer a problem.
It's still 20 million dollars. Ask any investor what he'd think about his company losing 20 million dollars for not catching a bug like what happened in that post.
When the stock market crashed hard back on October 19, 1987, Sam Walton said that he lost billions...on paper. Didn't seem to worry him a bit.
He didn't jump out of a window like they did on October 29, 1929.
As such, I found his remark notworthy because of his attitude toward the volatility of the stock market...nowadays just one small step above a casino....
E) Pulp does not require hardly any bleaching or even a tiny fraction of the toxic chemicals wood-pulp requires to process.
E.1) No toxic chemicals to expensively dispose of (less pollution).
F) Pulp requires a fraction of the processing compared to wood-pulp.
G) Same (non-THC-producing) hemp grown for rope and clothing can be used... existing/established farming methods.
H) Requires _much_ less fertile ground (no fertilizer) for growing... technically it _is_ a weed (not just a nickname).
H.1)
I) Requires much less expensive processing equipment to farm (ground requires drastically less/no tilling, collection can be done with hay-baling equipment instead of heavy trucks and tree-cutting machinery, etc.).
I'm sure I'm forgetting some.
Note the reference to a non-THC-producing strain... I'm not into pot, but I certainly can see a phenomenal idea when I see one (seen this one many years ago).
- Preferences: Solaris 10 (servers), Ubuntu (desktops), Solaris 11 (personal servers) -
This shows why truly redundant systems should be build using a mixture of different hardware and software developed by independend teams. This would reduce the risk of all devices being hit by the same problem.
The person who let it thru testing should be shot
don't blame the coder, blame the moron manager who didn't schedule enough testing. bugs are a known fact, and not planning for them is stupid. i think some of you can relate.
All good and sound advice, I saw something similar. The reason for not having a backup? They didn't want to spend the money on a backup strategy and they tried to fix it themselves.
It's a shareholder value thing, you know.
my 2 cents
Telcos often user different vendors for their transit and signalling layer for that exact reason.
You should have specified that you quoted Traffic.
perception is reality
Cool, this means there will be IT openings at the Chicago Tribune now!!!
"The poor schmuck" will, in my experience, have spent the last 18 months hearing phrases like:
"Time / Quality / Functionality: Choose Two"
"You can't test quality into a system"
"Measure twice, cut once"
"We need to parallel run the UT system"
"Engineers shouldn't be testing their own code!"
"I wouldn't be using NT for that, mate"
and so on.
These are the words technical people use to warn management of impending doom. Managers on the other hand have other things to worry about like delivery dates, sales, penalty ratchets and so on. When the "go" decision was made it will have been made by senior managers who get paid the big bucks to take the big decisions and the big sh*t when it all goes pear shaped.
The question is how the management handled mitigation by way of backups to manual processing, rollbacks to the old system or risk analysis during project planning.
Automation of an entire printing plant is a big job and it is probable they planned for a failure as a worst case scenario and will just put the 1M loss down to experience.
I wish at was Friday, but I dont want to wish my life away. So I wish it was last Friday.
Hmm, and there's a reason why you dot-commers got shelved. You know, you could have done a "rollback" and recovered those 100,000 records, i.e. if you were using any real database.
- mritunjai
emad? I haven't heard of that dick in years, I thought he was dead as BSD. Or at least naked and petrified.
On my iBook 1GHz (Quartz/Quartz Extreme is always on, as far as I know) Preview loads PDFs ridiculously fast. Acrobat Reader.. well, no.. but that's the sort of crap I have to use on my Windows box.
I loaded up that full NYC subway map PDF and it took four seconds until it had finished putting the last writing up. Zooms take just under a second to redraw. Scrolling around freeform (using the 'hand' tool) probably gets me just over a frame a second I'd guess.
"The job of QA is to verify that there are no bugs."
ROFL. How? Use a nice big rubber stamp that says "Certified Bug Free?", and hope for the best?
The job of QA is to find bugs that 90+% of people would find 90+% of the time, over 90+% of the product's lifespan. Pick the curves suitable for your company and product.
The job of IT security auditors is to find bugs/issues that only the savvy 1% to 5% of the people would find etc etc.
Give him a job at any of the big UK IT companies ;-)
Suttree, a weblog about casual games development
As Yogi Berra might once have said (were he an IT manager,) "In theory, there's no difference between the lab and production, but in production there is."
This is the core of the problem, lazy shits too lame to duplicate production, coupled by the saying that you put 2nd rate staff in testing - old hacks and has-beens.
I know test environments that are 3, 6 months or even 2 years behind that of production. No one will 'rebuild them', because it would be a miracle if they got it right. Disaster Management in production consisted of a bunch of mobile numbers...
Fire the dude who green-lighted/signed off on this testing. He/She got caught with their trousers down.
He'd have included an EULA with a "I'm not responsible for anything yadda yadda" at the end. Yes, even for inhouse software. It's not like anyone has to read it, you only have to include in it "by installing the software you agree with this".
---- Take the Space Quiz!
seppuku
Take a look at some K code (there are examples in the user manual) and then come back and say that. If K is too exotic, then try looking at some macro-heavy LISP code -- it has the same problem just slightly less so.
Code density can be good when you're trying to see the big picture (fewer screenfulls of code is a good thing in this case), but it can work against you when you're trying to understand the little details.
Regular expressions are nothing more than a hack to make up for the fact that generalized LR parsers were quite inefficient up until a few years ago. Just compare a reasonably complex regular expression to the BNF form of a grammar for parsing the same input to see how much easier GLR is to use -- you can see some examples of just how easy GLR parsing is to use here. And it can actually handle more general patterns with nesting, etc. I really think regexes are really just a question of premature optimization -- with GLR you just start out with an incredibly readable and simple grammar, and if it proves to be slow (i.e. if there are lots of points of ambiguity along certain parse trees) you can optimize it towards a purely LR(k) grammar.
HAND.
The newspaper I work for recently purchased a production system (server, archive, Workstations, etc.) the problems we saw came about because management went about looking for a new system the same way parents go about looking for their kids first car, ie. "how much is this going to cost us?" verses "Will this system be easy to migrate to both from an IT standpoint as well as end users and will it cover all our needs?"
The consequence of the managements decision is that my newspaper now has a system that parts of it are still being beta tested (at the expense of my work not the companies we bought it from), a system that before hours of hard work by our IT staff just kind of randomly lost files, and still does on occasion, and a system that has an archive thats database is limited by the number of entries and not how big its hard drives are, what this means is that our archive will be full after a year and a half, two years if we're careful.
all of this came after our IT department and all of their advisors let our management know that the wise decision would be to go for the next least expensive system that has show itself to be a good relyable system through use at multiple large news papers. Did Management listen, no they just saw a pricetag. what it got us is a system that after all the extra work that had to be done by our IT staff is mostly usable, and cost us more than managements second choice system. On top of all that the new system almost kept us from putting out our paper. The final effects of putting in this system is that to cover the extra costs jobs had to be cut.
So like theshowmecanuck says yea the code may have been flawed, but that should not have been a problem had management gone about impimenting the code properly having done proper testing before implimentation this problem may have been avoided, the same goes for the system I'm forced to use, sure there are bugs in the code, but we got what we paid for when my company purchased a system that still has parts of it in beta.
"Napalm is nature's toothpaste" - Chef Brian
Ah, yes, but now you are a step higher on the corporate ladder, and while in conversation with colleages the finger of blame always points up, in conversation with the boss however the finger always points down that ladder. Management is never to blame for bugs.
I think it's time you folks learn about the virtues of eXtreme Programming: Short production cycles, testcases and frameworks, constant feedback loops..
It realy works. The only problem is getting your management to leave you this 'freedom' to choose your way of managing software production down to the smallest details.
With great power comes great electricity bills.
All I know is, if the systems I look after are down for any period of time, someone is liable for $1.7M per hour.
Pat the box.... pat the box....
404 Not Found The requested signature was not found on this server.
WE HAVE BUSSES HERE IN SAN JOSE!?!?
Perhaps the bigger the PDF, the worse the bus system? MTA pwns VTA. Of course, MTA pwns almost all other *TA's. Still, our busses come like once an hour, and they don't go anywhere useful.
(Light Rail is fun though)
...Had this been an actual emergency, we would have fled in terror, and you would not have been informed.
Also, live half way around the world, and if pressed say "Go ahead and sue me, fat American imperialist, I've already spent the money."
If you were blocking sigs, you wouldn't have to read this.
Yes, QA is magic and any peice of software that passes through a QA process will be garaunteed 100% Bug Free!! or your money back. Not.
Testing is almost always an afterthought. QA is the last department to be staffed at a new company, the first to be downsized at an old one. QA is usually seen as a roadblock by managment and a pain in the ass by developers. Project Managers never allow for proper build/test/fix cycles. If you have a QA department it probably does not have proper component, system and integration test teams because of lack of resource. If it does, not a single team has been given enough time to do a full test.
Working in QA sucks, and I got out of that gig a while ago because of it.
Oh and by the way, why didn't the developer catch this bug during Unit Testing? You do Unit Test, don't you? Oh look, a developer who doesn't Unit Test yet expect miracles from QA, and it quick to lay the blame at their door. Your attitude is not uncommon.
You must end all the lines of your Perl scripts with Semi-Colons!
;
; ; ; ; ; ; ; ; ;
Use these to salt & pepper in your code.
This
If a fault was going to cost that much, surley they could have run the two systems side by side for a while.
thank God the internet isn't a human right.
Which time? I'm the guy who (unintentionally) wrecked the first Saturn ever wrecked (job #65). Since then I've wrecked one other (job 2 million and something), so my track record isn't that bad :)
:)
:)
Most of the time you don't actually break something (be it product or be it equipment), but fixing the bug and getting everything rolling again takes time.
And since the "value" of the product that is running on the line is about $5000 a minute, time is indeed money.
I've probably had a couple 1+ hour breakdowns, but this doesn't even compare to the time my buddies plant went down for three days x 2 shifts per day ($14M).
They were Lear-jetting parts in on a daily basis (they kept blowing up the new stuff and didn't seem to have the sense to order spares). Ron would show up at the service entrance at the airport to pick them up and it got to the point where the guys would just open the gates when he drove up
My most recent one was when we changed the line speed of the skillet line and the thumbwheel switch messed up and opened up the 8's bit in the ten's digit (faulty thumbwheel switch) so that instead of running at 42 jobs an hour it was trying to run at 80 JPH (it would have tried to run at 122 but it's limited in the software to 80 JPH)
Zoom zoom.
Oh wait, that's the other guys
John
I dream in binary.
Sorry, but you've got it completely backwards.
If QA find no bugs then they have failed. Sack them.
To use an engineering phrase - "A successuful test is one that fails".
The programmer won't find the bugs because he's too close to the code - he doesn't have the same usage patterns as the end users and should *not* be relied upon to find such errors.
(As an example, I'm on a project that's on it's final release candidate, been through about a month of testing so was rolled out internally for actual use. Within 3 or 4 hours we found an "Oh sh*t" bug that is going to have to be fixed. It's hard to replicate as it requires specific usage patterns and there's nobody to blame really, but when it happens it totally screws up the system).
using Visual Source Safe to keep versions
Whats your backup plan for VSS?
Honestly, it's the biggest peice of shit ever to escape from Redmond (Which is saying a lot). I should know, I admin the little fucker.
$10,000 is "under $1 million." For that matter, so is $1.
Also note that its all pre-sold advertizing at this point. They run another paper with an extra couple pages of advertizing, which they don't have to sell, and viola.
Meh.
.... that there are still people out there willing to throw away their money.
You know, they paid for a service, the newspaper is not for free.
IANAL but write like a drunk one.
Microsoft seem to have no problem in losing buisness billions of $ a year through their mistakes
just do nothing, it works for them
I cost the Times $1,000,000 and all I got was this lousy T-Shirt
Some co-workers not too far from my cube have sometimes used the acronym URLT (Update Resume, Leave Town) when describing what to do in such a situation. None of them ever have and I wonder if their management during those times was more to blame than their supposed mistakes.
I write this as I wait for fucking Acrobat to load
Downgrade to version 5. It doesn't have the bloat that's the problem with version 6.
You never begin a software upgrade without backout plans. At multiple checkpoints during the upgrade, backout plans are fully documented and if the upgrade has gone well, you move onto the next checkpoint.
Of course, none of that matters if you haven't placed a "production-like" loan on the system before attempting to upgrade production. It is about risk management. You can't remove all risks in an upgrade, but you should be able to manage them assuming the risks are all documented and provided to stakeholders. If you've told the stakeholders all the risks clearly and in writing and pointed out that there was no test system, no DR system or no good backout plan that wouldn't impact half the customers **AND** they still decided to go forward, oh well. Their decision.
Most of the world uses A4
I work as a system administrator for a newspaper since 7 years back. 5 Years ago we were out-sourced to another company, my job stayed the same (save for extra work needed) but the decision paths and cost terms has changed a lot. -- More management, less money, cutting corners, less contact with customers has actually led to an increase in costs by 25% for the newspaper.
:)
For 5 years we have worked on cutting costs instead of doing what we originally did; produce a newspaper. This has led to a lot of cut corners, patchy systems and above all stupid decisions. Now we have to spend most of our time with our hands tied behind our backs because there's no way to prove a _direct_ profit we can put on the price-tag we show to a (non-technical) customer when we are suggesting a change. It's always cost > functionality.
Companies that only sell services to customers has no goal, does not work. There has to be something you produce, something to live for instead of just being a money making machine.
Management cannot be just management to be management. A good manager is someone involved working with something they have a passion for. My boss didn't create this newspaper, nor did the boss of the actual newspaper and they probably don't have a special interest in media, it's just a career pushing money making machine for them.
Oh, I guess this turned into a rant
Well thought out and possibly true. I've seen it too many times, but for a system with $1M going thru it, the customer usually takes on additional responsibilities and performs 6 weeks or more of in house testing. When this kind of money is involved (I work on systems daily that are estimated at $1M per hour of downtime), the decision is a team decision, not a single person.
If the app support team had any problems, or the SA or the tester, or anyone else, they should have spoken up. On my systems they do - we beg them to tell us since nothing is worse than pulling the trigger and having unplanned downtime. Management must foster a team environment and no shoot the messenger.
If this were the 5th install attempt due to prior issues you have to wonder, but what can you do? By that time, everyone just wants it over.
From the OS X man page for "sticky (8)":
/usr/include/sys/stat.h for an explana-
/tmp which must
NAME
sticky - sticky text and append-only directories
DESCRIPTION
A special file mode, called the sticky bit (mode S_ISVTX), is used to
indicate special treatment for shareable executable files and directo-
ries. See chmod(2) or the file
tion of file modes.
STICKY DIRECTORIES
A directory whose `sticky bit' is set becomes an append-only directory,
or, more accurately, a directory in which the deletion of files is
restricted. A file in a sticky directory may only be removed or renamed
by a user if the user has write permission for the directory and the user
is the owner of the file, the owner of the directory, or the super-user.
This feature is usefully applied to directories such as
be publicly writable but should deny users the license to arbitrarily
delete or rename each others' files.
Any user may create a sticky directory. See chmod(1) for details about
modifying file modes.
At the source of every error which is blamed
on the computer you will find at least two
human errors, including the error of blaming
it on the computer.
- Tom Gibb
1) It is a small icon if you have a lot in the Dock
2) It is in the same place, so if you are under pressure and unmounting a lot of stuff you probably wont even look at it
3) If enough users make this mistake than this is evidence of bad UI design.
That said, engineers see this sort of thing all the time. It breaks down like this.
Bad Performance = Deaths of thousands (ford pinto, challenger shuttle, etc)
Good Performance = Plaque of Appreciation
So what the responsible party should do is resign before they're fired, and find another job before the news gets out(Pun never intended). And next time try not to be in the line of fire when things go to hell...
"I am the Black Mage! I casts the spells that makes the peoples fall down!" ~8BT
The problem isn't the coding error. Such errors will inevitably happen. They also happened when we designed mechanical printing presses, which might break down unexpectedly because of built-in design errors. The problem here is poor management.
In the past, creating single points of failure was hard: you had lots of men working on lots of printing presses. You couldn't do something as stupid as replacing them all in a single night--it just wasn't physically possible. Computers have just given greedy management the freedom to make more serious mistakes in a shorter amount of time. In this case, the mistake was upgrading a whole infrastructure at once and believing, naively, that that would necessarily go smoothly.
Oops!
shit happens, this story is not newsworthy.
"Man in Utah spills milk, news at 11"
please
So, the printing press and the "tribune tower" are in the same city, right? The problem was not with the actual printing but the electronic transfer...
Now why in the name of three supernatural ducklings didn't they just burn it on a dvd-r or a couple of cds and send it by courier or in person?
I'll burn down the building...
I believe you have my stapler?
It says "under a million". The point is not about it costing a million, but less than a million.
I'd say that a statement like that is targetted more at the financial guys to say "hey, it's a screw up, but don't get wound up, it's not even a $1 million screw up".
Any advice for the poor schmuck who's going to get the blame?
Cuba is nice at this time of the year...
You look like a million dollars. All green and wrinkled.
You obviously don't work for a fortunte-500 company with a standardize purchasing plan.
Hey - I bet AT&T could have saved a few hundred thousand dollars by standardizing on a single vendor!
This is just like IT security - cut and cut until you finally get burned, then up the budget a little until the problem goes away, and then go right back to hacking away at it...
Somebody may get fired, but not the person who caused the error. Everybody will point their finger at everybody else. And everybody may actually believe that they are not to blame.
can you buy stock in the newspaper? If so, have all the IT department purchased stock? That is a little used avenue to fight back with embarrasingly cheap and inefficient/clueless management. As a stockholder, you have a few goodies to throw into the mix, notably, if management can be demonstrated to be making incorrect decisions, technology decisions that can be absolutely proven to be incorrect, and especially after they were warned against it, you *might* have some interesting legal redress. And if they retaliate against you for going that route, in some states you have other interesting legal redress.
Just something to think about
Then it becomes two mistakes. Bad code and bad test.
Yeah, I've got one: Don't Quit Your Day Job
They didn't make sure their "user story" was implemented correctly.
Ask me about my vow of silence!
...that I've brought my passport to work on the day this appears on Slashdot?
My boss is wondering what I've done. No, really.
:)
and good morning to you too.
Stick your head between your knees and kiss your ass goodbye.
YOU'VE BEEN FIRED.
-- Game Developers: Stop porting badly-textured games from crappy console systems!
With impeding deadlines and cost overruns due to poor management forecasts, it is no wonder a lot of software these days have bugs in it. Not because code writers today are any worse (or better) than those of yesteryear, but because projects are cutting more-and-more 'fat' from their projects.
Translation: not enough testing.
Testers are often looked upon as the bottom rung of the overall software life cycle. Their duties are perceived by many to be hum-drum and easy to take out of the cycle. Unfortunately, cases like this show exactly why testing is one of the most importants facets of the software life cycle.
Remove, or severely limit, testing in your product, and you have only yourself to blame when problems arise out in the field. For this particular mele, if testing was removed from the project, I would blame the project manager and whoever made the decision to remove it. If testing was to blame, I would instil better procedures, beefed-up test cases, and possibly hire test engineers who ARE test engineers and not some developer who has a few cycles to burn.
...to protect them from just such a thing.
Right?
That originally pushed pot onto the restricted list in the 30's. They were trying to promote newly-invented nylon rope, and did not want competition from hemp rope, which was dominant at the time. Purchased congressmen got on the floor of the House and spouted nonsense about "pot makes black men violent and makes them desire white women". Then, in the 50's, when passing further restrictions, same purcharsed congressmen argued that "pot makes people into pacifist communists". Never let facts get in the way of your dogma (see Partnership for the Truth-Free America).
you research the new techniques and tools on the Internet, and then they get you for using too much Internet time. You have your innovative solutions; however, you are fired for using the Internet to research your problem.
Then your job is outsourced to another country where they can work 80 hours a week for 1/10th your pay. They may not come up with pragmatic solutions, have communication issues, and hardly ever give the managers what they wanted, but they can get by because they work for less pay.
Remember, Slashdot does not have a -1 disagree moderation, and no, troll, flamebait, and overrated are not substitutes.
Last Monday Suzy Manager shouted at her team, "The schedule says we install on July 18th, so this damned product damned well better be installed on July 18th, you all got that?!"
The funny part is that we just got a new Director here that fires managers that do exactly that. The man has business sense and refuses to treat management and other employees like children and cater to their pissyness. In fact he recently told all management that the ymust part in the out-lot and the closer parking lot is for the employees that show up early. Management can park in the closer lot if they show up before 8:00am. It pissed off a huge chunk of the prima-donna self serving PHB's, but it's working.
This man was hired to shake the place up and increase profitability by the end of 3Q 2005. and the man saw that the problems were in management.
problem is that corperate culture is to coddle these wastes of office space called PHB's so I'm betting this guy will be leaving in disgust soon.
It's rare that you get to meet a real leader in business, but when you do it is near impossible to not look up to them and want to do your best for them.
worker productivity has increased significantly as well as morale. People know that when they have a concern that it will be heard. (he encourages that you end-run the entire management ladder and email him directly if it is important. and calls people into his office regularly just to shoot the crap for 5 minutes.)
Anyways, managers that make insane promises like the one I used for an example need to be fired. but alas, that is not the corperate way.
Do not look at laser with remaining good eye.
Yeah, the guy/girl (No Fem-Nazi's need reply) that wrote the code screwed up, but what about the implementation plan. Did they not prepare for failures/problems? Did they not do any test runs? It sounds to me like they didn't evaluate the situation very well before they walked in, threw everything in place and expected it to work. Yeah, the person that wrote the code should be in hot water, but the person that implemented the code should be right there next to the coder, that is, if that person isn't already picking themselves up out of the parking lot.
When implementing new systems/software, you should never expect them to work right out of the box. If they do, then you have reason to celebrate but NEVER expect it and always try to have a backup plan.
'And all the monkeys aren't in the zoo Every day you meet quite a few...'
Remember, the greatest joy of coding is that you are not legal responsible for your errors. c.f. lawyers, doctors, business executives (post-Enron)
Yeah he may get fired, but he can't be held liable for the damages.
Buy this book, you'll need it to, er, tweak you're employment history.
Make sure they spell your name right in the next edition.
Blame Vinay.
Well, anybody except the all the people trying to do things like report newhires online, file tax and wage informaiton online, apply for unemployment online, or look for work via the job seeker system.
Thanks,
Anonymous IL Dept of Labor employee.
Ticketing system crashed after its update. You can read about it here : http://edition.cnn.com/2004/TRAVEL/07/17/bt.france .train.glitch.ap/
Admit your mistake. Identify the problem. Explain how this problem won't plague you again, and finally remind them of how many millions you make for them otherwise.
This has happened to me, before. I write code for a rather high-profile eCommerce site, (posting anonymously to avoid 'bragging') and have goofed by seven figures. I'm still employed, presumably by the advice I've listed above. For every million I may have lost my company, I've made them 1500 times more in a single year.
You think I'm exaggerating, but I'm not. Companies will go where the money goes, and when your company pulls $1.5 billion from the web, they're willing to put up with the occasional hiccough.
If newbies wouldn't make mistakes... us real developers wouldn't have to lart them for software bugs, which save Federal forrests. /wait a minute /removes larting tool from noobies skull
Buy this book, you'll need it to, er, tweak YOU ARE employment history?
...now I have proof that it was a mistake and not a bottleneck...you're fired!
After all, a $1,000,000 error is nothing compared to some of the mistakes SCO's made ...
Y|
I am Bennett Haselton! I am Bennett Haselton!
<me> I haven't touched the old server. If the new one hiccups one whit, we fire up the old box and produce product.
I've never worked in news, but I've had variations on this same conversation. I usually try to arrange the first cut-over attempt for a week or two before the deadline. My version has me telling my boss' boss something like:
<me> We're doing a test cut-over tonight. Something is going to go wrong. We've got two hours scheduled to test it and diagnose what the problem is. Whether or not anything goes wrong, the old server comes back at the end of those two hours and it's business as usual. We get the problems fixed this week and do another drill next weekend on the same kind of schedule. If that second drill works, the real cut-over is two weeks from tonight. Even then, the old server will be available to take over if anything goes wrong.
When you have a plan like that, managers are much more willing to listen to explanations about what went wrong. It is especially good when the conversation two weeks and a day later consists of explaining why you are still running on the old server.
The pressure to log the national forests would be for the purpose of getting precious wood from old hardwood trees that are getting scarce, not to get pulp wood. You don't cut down a healthy 100-year-old oak tree to make paper.
While it may be true that trees cut for timber are killed off faster than they are replanted, pulp wood is much more easily replenished. My father owned a small logging operation when I was a teenager (no clear-cutting involved). Pulp often comes from scruffy trees that wouldn't even be suitable for making furniture or much else. Poplar trees, which grow like weeds, and ash trees are commonly used.
Unlike trees harvested for timber (to be used in furniture, etc.) which need to grow for 40-100 years before they are much use, pulp can be acquired from trees only 15-25 years old. The tree can even have a little rot in the middle and still be useful for pulp. That's still a very long time, but certainly a renewable resource.
Religion is the opium of the people. Evolution is the opium of scientists.
I've personally caused outages (not intentionally that had at least his big an impact at the company). I was working for a financial services company, and our process to transfer transactions from the phone transaction system to the processing system failed one day.
Because these transactions are tied closely to market activity, the act of adjusting accounts over the next week to manually fix the lost transactions, cost the company over $1M in making up for what customers would have earned in the market.
The result for me personally was a stern conversation, and lots of explanations of what we will do to prevent this from happening again.
Another time at the same company we had a similar failure, for different reasons, and due to market conditions at the time the company came out about $10M ahead due to the market and the effect on correcting transactions.
I think this applies to managing software projects as well. Do you think anyone saw this coming?
If you're Micro$oft of course.
Hmmm, I had a deadline for July 18th and our last few days on the project were just as frantic.
All our process was thrown out the window and we just were patching code.
A senior engineer once told me that "Test doesn't have time to go looking for new bugs, they just make sure the old ones are fixed." You and I must work at the same place! We've just started referring to our customers as 'external QA'
Have you ever seen used corn?
Bugs happen, that's why there should be very tight QA control on releases... they are the ones that let it through.
This sig is the express property of someone.
What are you talking about? It sounds like you're either talking about rockets, or cars, or maybe even Lear jets, but I'm not sure. Oh, and you make cookware too? (skillets?) Either way, I think you're in manufacturing, but maybe you're in the software end of manufacturing...though it sounds as though you're managing an entire plant too, I'm not sure. And is that a mechanical thumbwheel, or a software-equivalent thumbwheel? Anyways, if you're any of the above, I'd like to know more. I don't see too many other manufacturing guys posting to slashdot.
....we are constantly reminded that we HAVE to get a paper out. So, I keep a big roll of paper and a box of crayons around just in case we have a problem with one of my systems. Heck, we even managed to get a paper out here in the northeast when the power went BOOM.....
course, I have renamed the business. We are not in the news biz, we are in the deforestation biz.
take the blame for the mistake. If you are the programmer and it was a programming error, the fault clearly lies with the QA people who didn't catch it. If you are the sysadmin or the QA guy, whatever happened was clearly a problem with management settting unrealistic timelines or expectations. If you are a middle manager the problem is definitely your inadequate budget.
Now before I get modded down, I be to remind whoever might read this that what I am saying is FACT. - bogaboga
one bad malloc can destroy the world
Is it 5:30 yet?
...and announce to the world how great a job they'd been doing. After all, that's what they did in NASA after Columbia, and that's what Bush did with the FBI after Agent Christina Rowley testified before Congress, and thats what Bush did for Tenet of the CIA and Rummy....
If it's good enough for those at the top, to not be fired and go to jail, it's good enough for the rest of us, too.
mark
You mean The president of the Newspaper? I sure hope they don't try to blame the coder on this one. because if I do my mental math correctly, I would think that the coder probably *makes the newspaper* more than the million that was lost.. Not only that. I'd blame the testing department, or moreso, the Programming Manager before the coder, but then, really. with that big of a loss. I'd blame the president of the company. - happened on his watch.
It was an accident. I said that in the original post. It was a dot com, and they are actually still around and (believe it or not) profitable. There were no VCs, share-holders, etc, and the only backup we had was a mirroring RAID for the file system.
It was MSSQL, and our backup system for this was actually a second server with MSSQL where we ran an ASP script to simply delete/re-insert records from one DB to the other. To do this it tool almost all night which is why it had happened over 6 weeks prior and not regularly. At the time I joined the company there were only a few fresh-out-of-college programmers there. No DB Admins, no networking smarts at all.
When you are in a company like this, sometimes money is scarce, so you stick to what you can do with what you have. Heck, what we did is similar to backing up an MySQL server with table dumps, so we felt like it was an OK solution. The logistics to how we would do it more often was the problem. Plus our solution made it pretty easy to recover small portions of data, or to easily lookup a persons old account who had changed something in 1 record in 1 table.
I left the company shortly after this happened because I didn't like the way things were run. (Not just because of the backup system). But I did hear that I was replaced with a networking guy instead of a programmer.
Only victims make excuses
BACKUPS!
Dawn of the Dead
...for a paltry half million bucks. Canadian even!
Chaeron Corporation
IMHO the blame should fall on management. I work in an office that supports 400 users, small potatoes compared to the Chicago Tribune, but even we have a back-up server. Even if that fails, we can reload everything in a few hours from tape.
---
Lousy rotten karmic retribution.
Money aside, sounds like they do a good job. Everyone is complain because they are used to getting the paper on time. When the don't complain becasue it's allways late is when you should start worrying.
1. Start rumors that you caused error
2. Deny everything
3. Hold out for book deal
4. Write your story
5. Make lots of money for screwing up.
Its all just smoke and mirrors.
"...blame it on the guy who cant speak english oh Tebor, how many times have u saved me?"
Well, the Mac based systems continued to work, I see.
Any manager who signed off on a software upgrade plan without the possibility of an abort and fallback procedure made about the most basic mistake possible.
So maybe Tribune management decided it was just cheaper to inconvenience their subscribers and eat crow.
There are not too many advantages to being sane but knowing what is funny is one of them. - Kingsley Amis
Blame your co-worker. Seems to work at the companies I work at. Here are people to blame:
I was in a 'special educational' environment where one factoid stood out - they tested some volunteers' abilities at motor skills, then got them drunk/stoned (where do I sign up for this stuff?). Then they tested the volunteers motor skills again, and also asked them how they felt. Surprise, surprise, they said they felt drunk/stoned and stunk up the tests.
Then, days later, they repeated the tests/questions. The alcohol recipients said they felt normal and tested normal. But the pot group said they felt normal, but still tested as impaired.
This is why the teacher said MJ will never be legalized in the US - too difficult to set a legal limit on a DUI level the way BAC% works for ethanol consumption.
There are some signs that the Tribune, at least, knows there was a publishing process involved here. They mention the number of errors and lines of codes one can expect in published software in the article. Our slashdot poster, however, goes right to the "Who's the poor rube to blame" model.
If the paper fires whoever was supposedly to blame, this is only going to happen again. What they need is for management to "get" that there's a publishing process with software that involves a certain set of stuff. QA, backups, fallback, and so on are all necessary.
This was a failing in upper management, and the specifics don't matter nearly as much as we think down in the engine room. They gave someone a deadline and didn't understand what was necessary. The IT people were working on no resources -- or maybe they were "can-do" hackers who gave management a bogus estimate that they then had to follow through on. Upper management needs to be able to assess that situation and deal with it.
The process is busted. Scapegoating the gearheads is no solution at all.
"Fundamentalism" isn't about divine morality. It's about human authority.
"I told you I needed more time to work on it !"
A million monkeys and this is the best sig they could come up with...
oops.
Insert funny smart-ass comment here.
This reminds me of someone I knew in 1996. He was a high-end developer/DBA type. He kept his resume on his Apple Newton. He joked (but was serious) that if he got let go, by the time he hit the front door he would have faxed/emailed his resume out to numerous headhunters. So, I suggest the scapegoat had better have their resume up-to-date and ready to go.
For firing the QA staff and just hiring a few people to test the code. You can't test quality into a product, and if they HAD a QA program, this would never have happened.
I doubt that mistake actually cost $60 million in lost revenue. Most likely, people just made their calls once the system was back up.
plenty of jobs where people on the ground are working with kit worth more than that. Easy for a forklift or truck driver to cause a lot of damage when moving stuff around.
It happened where I worked once.
Auto company using a "Sel 32" computer as the central tool for automating distributor testing-calibration. Systems Engineering Labs warned them that they were going to discontinue the line so they needed to stock any spares while production was still happening, so the company they were leasing it from jacked up the price to pay for stocking a bunch of spares.
Company decided to buy their own to save a few bucks and be sure the plant kept running and wasn't hostage to future price rises on an irreplacable, mission-critical machine. Cost was a few hundred short of a million. (The 98 cents pricing phenomenon, no doubt.)
Box showed up on the loading dock. One rack, floor to ceiling. Forklift operator picked it up, took it down the asile, took a corner too fast, and it fell off the forklift. Hit so hard it not only set off the tip/shock detectors but BENT THE RACK.
SEL, of course, wouldn't warranty it. The auto company was self-insured. So they buoght ANOTHER one (and kept the "clunker" for spare boards if anything failed in the future.)
Forklift driver was NOT fired. (Union, hadn't been notified he was toting a megabuck this time, and lift drivers are allowed a quota of oopsies.)
Bantam Dominique roosters crow a four-note song. Once you've heard it as "Happy BIRTHday" you can't NOT hear it that way
Yup, you get what you pay for. Cutting corners rarely saves money in the long run. But wait... in the USA we don't worry about the long run, we gotta pump that stock price up for the next quarter, so the no-goodnick CEO can execute his automatic sale of options. Does he own any actual shares in the company???? Not many.
As far as QA goes, I'm glad our development team doesn't have the attitude of "testers as enemies". I'm personally relieved at every bug they find of mine, because that's one less bug in front of the customers. And QA doesn't get the blame for new bugs, either -- it's not their job to discover every error. They are there to make sure that all the ordinary stuff still works, and that any other discovered bugs get fixed. They update their regression tests to make sure that new situations that lead to new bugs get covered by testing in future release cycles. But they aren't the ones who wrote the bad code.
Actually, our company (well, our team anyway) is usually pretty cool about bugs. We recognize honest errors happen, and as long as everyone who can help drops everything to help recover from the problem, life goes on. After the fact, we'll analyze the root cause to determine what process or procedure slipped up and figure out how to make sure that particular issue doesn't happen again.
We're an expensive team -- I can think of a couple of hundred thousand dollar errors we've had. But we also perform million dollar savings, and a lot more often. And we never, ever forget the lessons the expensive errors teach us. Ever.
John
Shouldn't the Testers and QA get the shaft here?
... can they?
:P
the coder can't be held responsible for a bug that apparently wasn't tested for
I think that someone should slap the QA group
To my old buddies that are still at the chicago office:
You have my sympathies.
or, rather, NEE NER NEE NER
heh heh....
since you won't die for a day or so from the disembowelment, unless you really lift the tip of the blade as you're coming up your ribcage and hit your heart.
the challenge is whether you can successfully gut yourself without showing any emotion, then calmly set the tanto down on the table and bend forward, presenting the back of your neck to the second for the sword blow; if you're going to start screaming like a little girl when your intestenes spill out, the second pins you down and stabs you to death. either way, i'd rather use a shotgun.
Facts do not cease to exist because they are ignored. - Aldous Huxley
Notice the first line in my post "Unfortunately, I can't find the orignal source". I remembered the line, I didn't remember the source. That's why I didn't specify, and also why I didn't quote the orignal version verbatim.
It's not the programmer's fault; code errors are a normal part of programming. It's up to the managers to plan for them and deploy new applications carefully. Even then, assume a coding error will take you down and generate a plan for isolating the problem and fixing it.
This and more on this page (no, I am not affiliated
A coding error is merely a programmer's mistake. Any mistake that leads to fiasco when it's put into operation is of course due to and the responsibility of the overall head of the project.
Quite possibly, responsibility lies higher up the ladder, at whomever did not give the project head sufficient cability to have detected the problem or deal with it appropriately.
I've done research myself... not in some lab, but by watching numerous friends and family members as I grew up (I used to hang out with potheads, though I didn't smoke it myself - they were cool with that = more for them).
Pot definitely DOES impair your judgement, reaction-time, and coordination... no matter how "with it" you feel when you're on it.
I gave my brother's best friend the nickname "SlowBob" when I was only 12 years old... over 2 decades ago, and this nickname sticks to this day (as he still smokes pot all the time). You practically have to hit the guy in the head with a 2x4 to get his attention (but only when he's smoking pot).
I've seen how people drive after smoking pot... and it's scary. They're reduced to the level of a tired, worn out 90 year old retiree in south Florida! Oh well, they let them drive, so why not, eh?
People CAN and DO get violent on pot... I've seen people get physical over who gets the next hit, really pathetic.
Pot definitely IS addictive.
The same guys who were smoking at least a joint a day nearly 2 decades ago are STILL smoking at least a joint a day.
They're also still living in their parents garages or basements.
They're still losing every job they get due to severe lack of drive that is induced by smoking the stuff (this is VERY easily observed).
They get cleaned up for a couple weeks, get a great job... their life starts turning around, promotions, etc. Then one day they light one up... within two weeks they lose their new job due to repeatedly showing up late or not at all, and the cycle starts all over again.
.
If you want to talk about a drug that is illegal that shouldn't be... look at ecstacy!
It was developed over a hundred years ago, and was used for much of the time by psychologists to help with their patients' therapy (with _much_ success).
The LD50 (Lethal Dose 50% of the time) level is many thousands of hits... so it's much safer than most OTC drugs.
The way the drug works (by saturating the seratonin receptors in your brain), taking more than one hit at a time does nothing extra (you can't get more than 100% saturated), so no-one would ever take the LD50 level anyhow.
You want to talk about government lies about drugs?
One report I saw said a girl died because she took too many and thought water was an antidote... so she drank too much water and her brain swelled up because she was waterlogged and she died.
I know many in the medical field (including many family members) and they all say that is absolutely ridiculous... your brain does NOT swell up from drinking water, only from SEVERE physical trauma (or a disease like encephalitis).
Granted, taking this drug does heat up your system, but if you have half a brain you drink water and cool down when you're hot anyhow.
Most of the time when people are taking this, they're at a dance party or rave, and the people there are making sure they have lots of water, etc.
The worst thing that a person on ecstacy is likely to do to you is hug you longer than you find comfortable... it is a shockingly non-violent drug.
It is considered a class 1 drug (or something like that, forget the term) which places it at the same level as heroine, in the US.
Any trace amount is a felony, even though it is far less dangrous, addictive, etc. than either pot or alcohol.
Other countries are starting to ignore the US-instigated ban on this drug, and have begun testing it again to see if it does have medical uses (which was already proven for a century).
I think Sweden or France... can't remember who (because it was at least 3 years ago that I read about it), but they really pissed off the US when they started the program as it questioned the US government's "all powerful" authority.
I'm sure he'll probably just blame the QA team.
cache
compare it to how it looked year ago
My other UID is 1337
...by going to the bank and making a "withdrawal" for "1 MILL-ion Dollars" (in my best Dr. Evil voice)
+1 Insightful, -1 Troll. What can I say, I'm an Insightful Troll.
Your string may not be zero terminated. In this case strlen will return an arbitrary string length which may fuck up some of your code down the road (especially if you use other "pound me in the ass" functions). Your printf can easily screw things up if its format string is user defined. The worst offenders are gets and sprintf, though. These two functions need to be removed from C library. :0)
Go work for the RIAA or MPAA, and keep doing things the way you were doing them. At least that way, you can claim some good from your mistakes
Dammit, you posted that before I could :-)
So I'll quote his other bit of wisdom that is impressed on my brain, aka "Crowder's Law of Optimization": "If the function result doesn't have to be correct, I can make the function as fast as you like."
You're missing the point.
Let me fill in the context, as I was the unfortunate recipient of this statement.
Our team was working on a project that was behind schedule and buggy besides. I was a fresh hire, just a few months out of college. In a status meeting, I attempted to be helpful, and suggested that if QA was finding more bugs, we'd be able to fix more bugs, thus rendering it less buggy. (Our QA department was in another city, so we had no day-to-day contact with them.)
At this point, the head of development gave me a stern look and said, "The purpose of QA is not to *find* bugs; the purpose of QA is to *verify* that *there are no bugs*."
I, of course, remained silent for the rest of the meeting...
But I learned a valuable lesson, namely that you don't code something, then "throw it over the wall" to QA to flush out the bugs; you code with the intent that there are no bugs, and when QA finds one, you take it seriously, and as a defect that you should strive to avoid by careful design and coding practices.
cat /etc/passwd |awk -F: '{print $1}' > /etc/ftpusers
What do you mean it's not an allow list?
Two words: "staging environment". Or perhaps one word: "testing".
Do you really blame the coder when something this important gets rolled out and all hell breaks loose? I'm sure the manager in charge of running that system did, but that doesn't mean he's right.
I wonder if there's a written specification that says that the system was supposed to work in a certain way, and the "coding error" resulted in a different implementation. Maybe the "coding error" was really a bit of undocumented functionality that nobody spec'ed out, nobody tested, and which (shockingly) failed when put into production.
When this sort of thing happens, it means that a coder didn't write the code that the customer needed, but that doesn't automatically make it the coder's fault. Did the customer (or "customer" in an all-internal situation) require it in writing? Did the development manager make sure it was in a spec? Did the analyst writing the specs capture that requirement correctly? Did the QA folks test for that feature? Did the docs say that the feature worked in the way the customer thought it would? Did the paper thoroughly test the software, accept it in writing, and train the users before rolling it out?
Only if all of this happened and there was *still* an obscure scenario in which the app behaved in a way that didn't adhere to the spec is it the coder's fault. Maybe. Or maybe it's the QA folks' fault for failing to test for that case, etc. etc.
It's much more likely to be a failure in pre-rollout testing, which isn't the coder's job. It's the IT manager's job at the paper.
Much of Sun's support has sucked since, oh, early March or April. I'm told they outsourced Silver-level calls then to...God knows where. India? Australia? Way out west of London? It's always different when I call.
While I know that an E10 will have been on a Gold or Platinum contract, I can't be the only person whose Silver support was so badly hosed that a near-by SSE was called in to pour a little oil on the water. (In my case, the guy stood up from the guts of someone's else's E15k to fix my crappy ol' 420R, the guts of which had been smashed by an earlier FE. And it still doesn't work right, five months and seven cases later. Grrr...)
I think there's still ripples from this going out.
...although the paper's editors were able to edit and design pages for Monday's planned 48-page edition, the computerized pages couldn't be transmitted to the paper's Freedom Center printing plant on the Near North Side.
Sounds like a proprietary format/OS/software might have doomed them from a quick recovery. If they were saving the files in a standard flat file format, or were able to extract the files from the database to such a format, transportation to another computer at the printing plant should have been tivial at best:
tar cvf largetarball.tar *
ftp ppmachine
(login process)
put largetarball.tar
quit
(login to printing plant machine)
tar xvf largetarball.tar
Voila - your files have been transfered.
Poor system design - specifically the inclusion of 'get out of jail free' cards in the form of open standards and tools for extracting data to different formats and moving the data around between machines - is the number one problem associated with proprietary systems design. I would require all new systems have more flexible capabilities for several reasons:
1. Disaster recovery.
2. Protection from unscrupulous vendors, or the vissisitudes of the business world (vendor goes out of business due to downturn of no fault of their own).
3. Protection from software glitches.
4. Migration of data from one software platform to another as needs change.
Lodragan Draoidh
The more you explain it, the more I don't understand it. - Mark Twain
It was in my employer's interests!
If I had actively tested, I'd have missed bugs! Costing them money!
I should ask for a bonus.
(former contract testor on SMS 2.0)
Blame it on hackers...or better yet, hacker terrorists. This phrase is going to be the "the dog ate my homework" of the new millenium.
When I was working at Citibank I transferred 40M twice (major airline payment for one of the largest ...
airports in the country). Luckily the airport agreed
to give it back
To call an event an accident is to assume your contribution to the event was beyond your reasonable ability to control. If a 4000lbs bomb goes off behind you and you push the car ahead into traffic, that's probably an accident.
If your tires are getting bald, if you don't check your brakes regularly, your windshield is dirty, adjusting the radio, thinking about mating, etc, then there's a strong possibility that a rear-ender is in fact a fuck-up, not an accident. He is responsible for making the kids orphans.
How Wes deals emotionally with his actions is personal. How he deals with it financially is a legal and business decision. How society deals with people not giving due consideration to what they're doing is by guilty-tripping them, en mass. As a result, other people are made better aware of Wes' fuck-up, and adding up the personal, financial, and social costs, may be less likely to repeat any number of automotive fuck-ups.
Hey, it worked for me.
Luke, help me take this mask off
I feel your pain. I work in one of your competitor's plants (won't mention the name). Our cost for downtime is a little higher then yours. We recently switch VIN plate emobossers and I had to add some new code to handle the machines. When I turned the things on, all VINS were misprinted. I had a two day fight with the vendor before they finally concluded I was right--it was there fault. Two 8 hours shifts times two and a half days times $24000=a whole PILE of money.
Those wouldn't result in the execution of arbitrary code, just as the grandparent said. Sure, it could screw up your program, but many things can do that. So long as you always use static string buffers (setting the last byte to '\0') and use snprintf, etc. using the length of the buffer -1 you shouldn't have any of these problems.
I would never allow a user to define an arbitrary format string however (at least not without pre-processing the string first).
I certainly would NOT fire someone for using printf or strlen for crying out loud. Those are very useful, safe functions. You may as well outlaw malloc. Many C functions can be given arguments which will totally screw up the program after all, just 99% of them won't result in buffer overruns.
Rule number 1 when writing software connected to a piece of hardware:
Hardware Lies - verify as much as possible what it is saying.
An ominous cow herd.
I once delivered some code to Time magazines's printers which caused some small portion of the halftone photographs in the mag to come out glitched(An off-by-one error when the number of rows was 3 mod 4, or some such weirdness). Luckily, the same release of software had compression code in it which allowed them to delay their photo deadline by a day, so I kept my job.(They were using 4800 bps lease lines back then, so transmitting pictures took a long time.
Cuz $DEITY forbid anyone should ever want to get high on anything.
"A great democracy must be progressive or it will soon cease to be a great democracy." --Theodore Roosevelt
Anyway, from everything I have seen, the mindset in the US is quite a bit different than in Europe. I think in the US there might be a lot problems with:
- Lots more people spending all of their time high. Not sure "addict" is the right label here, but certainly the millions on welfare that aren't interested in working would have something new to occupy their time.
- Stepping stones for sure. If we saw pot go from "bad, real bad" to "good, real good" what would this say to the 12 year olds that are watching friends smoking now? Why would anyone stop them? And, this would certainly bring them into more contact with the dealers that used to be dealing pot.
In the US high prices, limited availability and legal problems are the only things keeping a lot of folks off the stuff.I'm glad Belgium has more responsible people, but that doesn't help us over here.
Never keep more than a 1 year contract with AT&T. Never. A 3 year contract would turn into hundreds of thousands of dollars in overbilling that they will refuse - straight out refuse - to refund without a lawyer getting involved, and even when we did - they wanted to go to court. Renewal time, ho ho ho, well, I guess we were wrong after all, here, have a $75,000 check, and sorry for all the trouble.
One guy alone does not make a mistake like that. I'm a big believer in: If it's possible to make a mistake (particularly one of this scale) then preventive procedures and quality assurance are not up to scratch. This was a team effort, not a 1 man screw-up. If one guy loses his job or suffers as a consequence of this event, then I urge you all to action. Action similar scale to our support of Kevin Mitnick. How would you like to write code and (say) forget a comma, have it pass cleanly through all peer reviews, unit testing, system testing etc etc, only to find that it caused a $1m problem and your career and livelihood was now on the line ? One of the biggest reasons you are constantly subjected to security audits and process audits and finance audits is not because you have been a naughty boy and someone from up on high wants to catch you out, it's to find holes in processes and procedures that might at some point in the future cause a problem (like this one) (I'm tipping that the auditors don't get their arse kicked over this.) If one guy loses his job, you'd all better stand up and make with the noise, otherwise no one will come to your rescue when you need it.
In my experience being honest about your mistakes and having the willingness to learn from them always pays off. actualy, mine was for $6.8 million. My first year programming I reversed the flag on a particular discount on a billing program. I got to talk on the phone with several vice presidents and then the CEO of a large multinational corporation. That wasn't as much fun as it would have otherwise been. There is really nothing else to do but admit that you screwed up and do your best to make things better. I wrote the script to pull out the list of people who'd been mischarged as fast as anything I've ever written. I actualy offered to resign. I think my candor was what kept me on the job. I know I would have immediately fired anyone I caught trying to cover that up. Sometime later, I got to watch a co-worker even less comfortable than I was explain to similar bigwigs why the phrase "This is bullshit!" whas showing up on their screens. I was happy to learn that lesson secondhand. I don't even put curse words in comments or test data.
thats why I said:
;)
"or the workers faimly"
Now, I know that's confusing, but replace whatever the hell 'faimly' is with family, and it'll make sense.
The Kruger Dunning explains most post on
I used to carry $ 1,000,000 libality insurance
Diplomacy is the art of saying "Nice doggie" until you can find a rock. Will Rogers
"Any advice for the poor schmuck who's going to get the blame?"
In the words of the great Dave Chappelle, "RUN BITCH, RUUUUNNNNN!!!!!!"
"Instant gratification takes too long." - Carrie Fisher
Why should we provide drug users with new needles?
If you get a nasty infection from using heroin, maybe the problem isn't that you didn't have a clean needle...
Maybe the problem is that you're using heroin?
I say no free needles. If you come to the emergency room and you have heroin in your system, the emergency room should be allowed to throw you out and let you die.
If this bothers you, don't use heroin.
paintball
It's easy to arm-chair quater-back where your news paper comes from, but I for one don't subscribe to anything but online sources. You should too....
Well, I for one welcome our new pulp-consuming overlords.
paintball
Regex is logically equivalent (i.e. there is a polynomial time mapping between the regular expression language and the BNF) to a LR grammar, though it is more compact. I don't think the authors intended it as a substitute until LR got faster. I believe they intended it to be an LR-equivalent language.
I will admit, though, that sometimes I need the full thing, so I used a LR parser to make this.
It is a bit inefficient, actually, but I think it's worth it.
You may also notice if you look at my code that occasionally I use a regex in there every now and then when I know it won't be too expensive.
Regex isn't (always) fast, it isn't pretty, but it works, it's compact, and it has already been heavily tested.
One last thought: the concept of LR grammars where invented as a mathematical way to express regular expressions as a series of states, because states are a lot easier to deal with mathematically than regular expressions.
That's the only real reason. Regular expression came first, though. They're easier for people to come up with - more natural.
Mod me down and I will become more powerful than you can possibly imagine!
And he was factually incorrect. It is a term that describes the entire incident, the term itself does not infer nor assign who the victim is or the level of blame for the person who caused "the accident".
I'm aware of the dictionary definition, and it's my contention that the perpetrators of most "accidents" are hiding behind it. God knows, it's a pain in the ass to regularly check the safety features on a car every now and then when firing it up in the morning. It's a pain to maintain distance from the vehicle in front when moving or stopped. Most of us elect to ease off on the anal retentiveness - me included - based on our previous experience that we'll successfully pull off a day's driving, or activities in general. However, sometimes we don't, and as much as we wish it weren't, it's a fuck-up, not a God-reached-down-and-blinded-me accident.
I'm not saying that Wes needed to go Japanese on us, but neither should some pop-psych jump on the ass of someone who utters the popular assertion that for Wes to make things right, he oughta make sure the freshly minted orphans get a good shot at making it to adulthood without getting lost in the shuffle, ie: "for the rest of their lives".
Luke, help me take this mask off
> I'm not saying that Wes needed to go Japanese on us, but neither should some pop-psych jump on the ass of someone who utters the popular assertion that for Wes to make things right, he oughta make sure the freshly minted orphans get a good shot at making it to adulthood without getting lost in the shuffle, ie: "for the rest of their lives".
That depends entirely what you mean by making sure..
Making sure as in, putting in attention and time in order to get them the best position possible? yes. As in, carry the financial consequences for bringing them up? no.
Orphans exist due to many reasons, and are the victim of some situation beyond their control in all cases. There is no reason why they should all of a sudden get a better chance because they became orphan as the consequence of the actions of a rich person while those who become orphans due to their parents being killed by some homeless junkie out for money get no chance. Rather, society should make sure orphans get a fair chance in life, regardless of why they became orphan.
You're either trolling or misinformed & pushing the luddite position "Anything Nukular is EEEVVIIILLL".
The only plutonium powered powersource Nasa has ever used have been RTGs. RTGs were designed to survive reentry from orbit + landing on a hard granite surface without rupturing their containment package and exposing the plutonium. RTG's used in a launch that exploded & crashed into the ocean have been recovered _and_reused_!!!
The plutonium in RTGs is an alpha emitter. Alpha particles are very nocive if inside the body, but are easily shielded by anything as flimsy as a piece of paper. Thus the only way for the plutonium to poison anyone would be for it to enter the foodchain or as an extremely fine (breathable) powder. However, the plutonium used is in bricks of ceramic plutonium oxide in which form it is almost impossible to do either.
Your Nasa/Chernobyl/Promethieus claims, while colorful, have no basis in fact either, otherwise you would have responded to Yunzil's post.
Heating up fissile elements does not alter the rate at which fission occurs, so your claim that reentry of a reactor would be "heating up enough to get fission going" is another pile of steaming crap in a post already full of it.
Anyone interested in the facts instead of Melantha's BS can start off here: http://www.seds.org/spaceviews/cassini/rtg.html
Democracy is a sheep and two wolves deciding what to have for lunch. Freedom is a well armed sheep contesting the issue
but I usually do a real parser for even trivial lexical data since trivial hacks have a way of becoming notrivial with time. But, hey, to each his own. :)
HAND.
Yikes! Holy crap!
:)
I'm currently in the throes of adding pneumatic couplings and upgrading drives to one of our lifts. Up until about 9pm last night I didn't think the damn thing would ever move.
Then I noticed that I was telling it to go up and down at the same time...
Today is a much better day
John
I dream in binary.
Well, I work for GM at Saturn. And we build cars :)
:)
And yeah, I'm a PLC jockey, so most of the code I write is ladder logic (check out www.plcs.net if you are interested). I also serve as one of the troubleshooters so I fool with hardware as well (tonight it's a couple of 20hp 480 drives).
As for the skillet, that's a cross between a skid and a pallet.
And yup, we still give tours, so if you're ever down this way stop in
John
I dream in binary.