Slashdot Mirror


Database Glitch Grounds American/US Airways

An anonymous reader writes "According to numerous news sources, all American Airlines and US Airways flights were grounded for two or three hours this morning. Both problems were caused by a computer glitch in the systems hosted by EDS. Quote: The operating system that drives the airline's flight plans went down."

274 comments

  1. Re:Windows by kd5ujz · · Score: 0

    Windows was a given, now for bonus points, tell us which version.

    --
    -William
    God is everything science has yet to explain.
  2. Great News! by MikeDawg · · Score: 2

    Well isn't that some great news, that makes me feel 20x better about taking my gf to the airport this morning. Fortunately she wasn't flying U.S. Airways or American Airlines.

    She is absolutely frightened of flying, and somewhat of a computer nerd, I can't wait to talk to her, and tell her the scary news.

    --

    YOU'RE WINNER !
    Another lame blog

    1. Re:Great News! by Anonymous Coward · · Score: 0

      I don't get it, why is this scary? It's the equivilent of a gas station being all lined up because the pumps are down. You wait till it's fixed, and you move on.

      I think you're overreacting.

    2. Re:Great News! by Vlad_the_Inhaler · · Score: 4, Informative

      Typical Airline applications are Reservations, Check-In, Weight-and-Balance, Flight Planning (which route to take and how much fuel to carry) and Ticketing. Once you have left the terminal and are heading for the runway, software crashes cease to be relevant.

      Once you head for the runway, you care about Air Traffic Control's software. The only exception I can think of is for flights to the US where the authorities want passenger lists.

      I work for an airline and we host for other airlines. I feel sorry for whoever carries the can for this mess. As to the OS, those who said it will be MVS are almost certainly correct. AA and US Airways are/were IBM customers.

      --
      Mielipiteet omiani - Opinions personal, facts suspect.
    3. Re:Great News! by Chess_the_cat · · Score: 4, Insightful

      What scary news? The airplanes are piloted by people, not computers. And certainly not the computers that control flight plans. Do you think that airplanes will start falling from the skies because a computer went down somewhere? I guess you packed your basement with cans of beans for Y2K too.

      --
      Support the First Amendment. Read at -1
    4. Re:Great News! by Trurl's+Machine · · Score: 2

      She is absolutely frightened of flying, and somewhat of a computer nerd, I can't wait to talk to her, and tell her the scary news.

      One of my worst flights ever was on a business trip to Edinburgh, Scotland. I was accompanied by a genuine RAF pilot, who flies the tanker Boeings for NATO warplanes. There was a rainstorm and strong wind over the whole UK and my friend was busy explaining me that Boeings are very vulnerable to strong winds and wind is the scariest threat for Boeing pilots and so on. It wasn't a nice thing to say on board, especially when the pilot announced a delay in approaching due to strong wind.

      On return flight I discovered that this time we'll be flying on an Airbus. I told the (supposedly) good news to my friend, hoping that this time he will spare me his horror stories. Unfortunately, his reply was:
      - Do you know why the pilots call it "Scarebus"?

    5. Re:Great News! by eyegone · · Score: 1

      As to the OS, those who said it will be MVS are almost certainly correct.

      Actually, it's probably TPF.

      --
      "They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety."
    6. Re:Great News! by Vlad_the_Inhaler · · Score: 2, Interesting

      Ho hum.

      After I submitted the grandfather post, I saw something I'd missed first time around:

      The operating system that drives the airline's flight plans went down. It might even be a Windows problem. A 'Flight Planning' application is a low volume application where you work out the optimum route for a plane based on the weather. That bit about the weather involves serious number crunching and the PC world has more of that kind of power to spare than the mainframe world. I helped write one of these apps 20-18 years ago and the central part has since been converted to run on PCs.

      Sorry about that :-)

      --
      Mielipiteet omiani - Opinions personal, facts suspect.
    7. Re:Great News! by luke923 · · Score: 1

      Typical Airline applications are Reservations, Check-In, Weight-and-Balance, Flight Planning (which route to take and how much fuel to carry) and Ticketing. Once you have left the terminal and are heading for the runway, software crashes cease to be relevant.

      No, AA and USAir use Sabre for rather mundane tasks such as plane maintenance as well. So, if Sabre was down, planes shouldn't be able to take off b/c things such as engine maintenance would be skipped. This, more than anything, would cause the halt of flights I would figure.

      Now, as for the OS, I can tell you that - without a doubt - that for services such as plane maintenance and baggage were affected, both AA and USAir use an OS/390 system, whereas other things, such as the things you mentioned, Vlad, are taken care of inside TPF.

      But, I do have to agree with you on one thing, though - it has to be a difficult time for the techs who have to work this mess. I can tell you from personal experience that Peter Gibbons had fewer bosses than these people. Most of them are underpaid, and living inside asluT (and, all this time I wondered why Tulsa Univ's mascot was the Golden Shower) - where job prospects in IT are slim - they hold onto what slim hope of holding their jobs while being berated by the many Hydra they call boss. This just makes it worse for them.

      --
      "Good, Fast, Cheap: Pick any two" -- RFC 1925
    8. Re:Great News! by CDWalton · · Score: 1

      I knew a pilot who flew for one of the mjor airlines, and from what I was told A pilot now days is only there to take the plane off the ground. After that it is all computer including landing. the pilot NOT the computer is the redundent part....

      --
      When the going gets tough, the tough get drunk
    9. Re:Great News! by Anonymous Coward · · Score: 0
      What scary news? The airplanes are piloted by people, not computers. And certainly not the computers that control flight plans.


      Not the Boeing 7E7.
    10. Re:Great News! by Anonymous Coward · · Score: 0

      They crashed OS/390? Out of interest can anyone think of how exactly they might have done that? Did the newbie IPR the production system or something?

    11. Re:Great News! by keithcstone · · Score: 2, Informative

      Unlikely the OS (which was TPF BTW) crashed at all. Title says "database curruption" not OS or hardware crash.

    12. Re:Great News! by Anonymous Coward · · Score: 0

      Apparently I type with a Japanese accent. Initial Program Road eh? Damn.

  3. US Air down for half of Sunday! by Anonymous Coward · · Score: 0

    In other news, US Air announces highest level of on-time flight arrivals in their corporate history.

    1. Re:US Air down for half of Sunday! by Anonymous Coward · · Score: 0


      Useless Air, The Lawn Dart of The Airline Industry

  4. Re:Windows by Anonymous Coward · · Score: 0

    my bet is on XPsp1

  5. Operating System (singular) by Hypharse · · Score: 5, Insightful
    "The operating system that drives the airline's flight plans went down."

    How in the world can they state that as singular. Surely they have a backup of some sort. Especially with all the supposed "increased security" around air flight, you are telling me that one system crash can knock out half of the major airlines? That's ridiculous. Have they not learned about redundancy?

    1. Re:Operating System (singular) by Anonymous Coward · · Score: 5, Funny

      Yeah, Have they not learned about redundancy?

    2. Re:Operating System (singular) by k4_pacific · · Score: 2, Insightful

      Well, one operating system can run on multiple machines. Google, f'rinstance, has thousands of machines running Linux, and that's one operating system. Probably there was some service running on all the systems that choked simultanously on some piece of bad data or they distributed a bad upgrade.

      Either way, somebody fucked up somewhere.

      --
      Unknown host pong.
    3. Re:Operating System (singular) by Anonymous Coward · · Score: 1, Funny

      Yeah, Have they not learned about redundancy?

    4. Re:Operating System (singular) by Anonymous Coward · · Score: 1, Informative

      use the 'yes' command in linux to take care of that.

    5. Re:Operating System (singular) by njcoder · · Score: 5, Funny
      "Have they not learned about redundancy?"

      Yep, their so good, even the failure was replicated!

    6. Re:Operating System (singular) by waynelorentz · · Score: 2, Insightful

      ... you are telling me that one system crash can knock out half of the major airlines?

      That's not what it says at all. American and U.S. Airways certainly don't count as half the major airlines in the United States. There are hundreds of airlines in the U.S. of A., and maybe a dozen qualify as "major." And by some measures, U.S. Airways doesn't count as a "major." So, no, you're completely wrong. Don't read things into the article that aren't there (assuming you RTFAs.)

    7. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      Yeah , Have they not learned about redundancy?

    8. Re:Operating System (singular) by Anonymous Coward · · Score: 0
      Yeah, Have they not learned about redundancy?

      If I had mod points, I'd mod your comment redundant.

    9. Re:Operating System (singular) by Vlad_the_Inhaler · · Score: 4, Informative

      This is partially a question of cost, redundancy costs money and those airlines are rather short of the readies (although this crash will cost serious money).

      For any *normal* 'extreme situation', a reboot should help.

      Having just read that The operating system that drives the airline's flight plans went down, it might even be a Windows problem. A 'Flight Planning' application is a low volume application where you work out the optimum route for a plane based on the weather. That bit about the weather involves serious number crunching and the PC world has more of that kind of power to spare than the mainframe world. I helped write one of these apps 20-18 years ago and the central part has since been converted to run on PCs.

      --
      Mielipiteet omiani - Opinions personal, facts suspect.
    10. Re:Operating System (singular) by Anonymous Coward · · Score: 4, Funny

      I'm an MBA. Would you please explain the joke?

    11. Re:Operating System (singular) by Osrin · · Score: 2, Insightful

      You are thinking like a computer techie... I suspect this guy does not even know what a Computer Operating System is, he is more likely refering directly to the underpinning infrastructure that runs his airline.

    12. Re:Operating System (singular) by gabbarbhai · · Score: 1

      if i had mod points left, i'd have mod you up as insightful.. :)

    13. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      Yeah, Have they not learned about redundancy??

    14. Re:Operating System (singular) by afd8856 · · Score: 0, Redundant

      WTF don't you get from the repeating of the joke? HINT: it's about redundancy.

      --
      I'll do the stupid thing first and then you shy people follow...
    15. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      And if you knew how to actually spell "they're", it would be even funnier.

    16. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      Yeah, Have they not learned about redundancy?

    17. Re:Operating System (singular) by britneys+9th+husband · · Score: 0, Redundant

      They knew about redundancy, but since they read Slashdot, they also know that Redundant is a -1 mod, and therefore redundancy is a bad thing.

      --
      Hear recorded Slashdot headlines on your phone! New service beta testing. Just call (248) 434-5508
    18. Re:Operating System (singular) by Profane+MuthaFucka · · Score: 1

      It wouldn't be funnier than a spelling troll who begins sentences with a conjunction.

      --
      Fascism trolls keeping me up every night. When I starts a preachin', he HITS ME WITH HIS REICH!
    19. Re:Operating System (singular) by mindriot · · Score: 2, Funny

      Can somebody with mod points please mod this (-1, Redundant)?

    20. Re:Operating System (singular) by WindBourne · · Score: 2, Interesting

      Actually, my father is a retired captain from American, and I used to go into crew schedule. It used to be that flight plans were done on a mainframe. I doubt that it is still that way, but it was one computer back then.

      --
      I prefer the "u" in honour as it seems to be missing these days.
    21. Re:Operating System (singular) by B747SP · · Score: 2, Insightful
      Yeah, Have they not learned about redundancy?

      Well duh! Your average "aviation writer" is flat out telling the difference between an aeroplane and a hole in the ground. Asking them to write a story that involves aeroplanes *and* computers is just asking for trouble. Why, that's like asking a six year old to pat his head and rub his stomach at the same time!

      Basically the way these morons operate is that they latch on to any half-valid snippet of information - if their editor believes it, it must be true - and rush it out the door without so much as a damn about checking facts. This is 'journalism' facts have no place here!

      It is routine here in Oz for the journos to get as far as identifying that an aeroplane was involved, then completely lose the plot. Nighttime TV news, for example, might run a story about somebody's grandmother smuggling a pair of nail clippers onto a Boeing 767, and show stock footage of a BAe146 landing somewhere. There's a disturbing number of six seater Cessna 210s and twin engined Piper Warriors here in Australia too, not to mention the Beechcraft KingAir jets that everyone seems to be having trouble with lately.

      To put it simply, 'journalist' is a polite euphamism for 'clueless'. You're doing yourself a disservice if you think any other way! :-)

      --
      I find your ideas intriguing and I wish to subscribe to your newsletter.
    22. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      The problem with redundancy is being too redundant.

    23. Re:Operating System (singular) by Fulcrum+of+Evil · · Score: 2, Funny

      I'm an MBA. Would you please explain the joke?

      Well, it's like when you have two people doing the same job in case one gets sick, except that, instead of firing one, you keep both around.

      --
      "We returned the General to El Salvador, or maybe Guatemala, it's difficult to tell from 10,000 feet"
    24. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      Last recorded message at EDS Central:

      "...if our calculations are correct, then our backup will experience a similar failure in...who turned out the lights?"

    25. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      The flight tracking (not planning) system at a very large airline I used to work at consisted of HP Unix workstations connecting to IBM mainframes. The "flight planning" came from our headquarters but I have no idea where it came from. I'd assume a similar system.

    26. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      it's still a sentence; nothing wrong w/ starting it off w/ a conjunction.

    27. Re:Operating System (singular) by Anonymous Coward · · Score: 1, Insightful
    28. Re:Operating System (singular) by blastedtokyo · · Score: 2, Insightful

      Yeah, but if Google loses a website in their results, nobody notices. If an air traffic controller loses a plane with a few hundred people on board and lets another plane fly close to or into it, you can be damn sure that people will notice.

    29. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      Yeah, have you not heard of EDS? No, of course not, this is slashdot, where people are even more clueless than journalists.

      Anyway, I business I've worked at has run into reliability problems with EDS and were assured it was a freak accident that exposed an exception in our code combined with a freak failure of hardware systems which exposed a NEVER BEFORE SEEN bug in EDS software and it will never happen again. It is totally a coincidence that the super high priced system that is designed especially for data protection (redundancy) that has never happened before to anyone else and will never happen again.

      When it comes down to it, it is my opinion that cheap RAID and backup tape systems do a better job than these high priced systems that are supposed to never fail. The recovery takes loads longer, but recovery is needed less often, and at very worst, you can always get your information EVENTUALLY.

      I'm scared about things like banks and airlines and payroll and customer information systems depending so wholly on these proprietary six-nine systems that are really nothing more than a buggy bill of goods with a six digit price tag.

    30. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      Yeah, have they not learned about redundancy?

    31. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      WTF don't you get from the joke about modding redundant? HINT: it's about redundancy.

    32. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      Or someone who end sentences with "a conjunction".

    33. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      Either way, somebody fucked up somewhere.

      Oooo, "somebody" fucked up "somewhere"! A post deserving of +1 Insightful.

    34. Re:Operating System (singular) by Anonymous Coward · · Score: 0

      Can't RTFA if it requires that I register first, which I refuse to do for reasons of personal hygine.

      (There was a link to continue to the article without registering, but when I clicked it, it returned me to the same page (probably because I had "referer" turned off).)

    35. Re:Operating System (singular) by Profane+MuthaFucka · · Score: 1

      And milk came out my nose on that one.

      --
      Fascism trolls keeping me up every night. When I starts a preachin', he HITS ME WITH HIS REICH!
    36. Re:Operating System (singular) by NateTech · · Score: 1

      Thank you for mentioning this. The press is just as clueless here in the States about anything Aeronautical. Most "Aviation Experts" on CNN, etc... are generally sensationalists, not professionals. Commercial aviation is boring because it's DESIGNED to be... system failures with multiple redundant systems and training on how to use all of them and understand the systems is... a non-event in almost all cases. (The computer industry could learn a lot from the Aviation industry, but the drama queen project managers and their staff who like drama would never stand for it.)

      About the only good source of Aviation news (although highly biased toward supporting Aviation and not ashamed to admit their bias unlike the "Fair and Balanced" reporting of some...) in the U.S. is AvWeb.

      The Aircraft Owners and Pilot's Association magazine and website aren't bad either, but they have to be careful not to step on toes... if they want to accomplish anything. Phil Boyer is an excellent statesman, but sometimes it takes an outside editorial from "the press" before he can really say anything bad about any particular sub-set of aviation.

      --
      +++OK ATH
    37. Re:Operating System (singular) by Vlad_the_Inhaler · · Score: 1

      Why would you need to track a flight?
      You know where it is coming from and you know where it wants to go so what is the point of spending resources tracking it?
      Leave that to Air Traffic Control, they have to do it anyway.

      --
      Mielipiteet omiani - Opinions personal, facts suspect.
  6. I wonder if they could get any more vague.. by Anonymous Coward · · Score: 0

    A glitch could mean absolutely anything. What reason would they have for not giving even the smallest of hints as to the nature of this glitch?
    Yes I can think of a few. Just trying to get you thinking of them as well.

    1. Re:I wonder if they could get any more vague.. by Rosco+P.+Coltrane · · Score: 3, Funny

      What reason would they have for not giving even the smallest of hints as to the nature of this glitch?

      The PATRIOT Act?

      --
      "A door is what a dog is perpetually on the wrong side of" - Ogden Nash
  7. anyone care to venture a guess? by Rockenreno · · Score: 1

    as to what Operating System they were running?

    --

    Forecast for tomorrow: A few sprinklings of genius with a chance of DOOM!
  8. EDS works with a variety of systems by Rex+Code · · Score: 4, Insightful

    EDS is by no means a Windows shop. They work extensively with "big iron" mainframes. In fact, they recently got the contract to handle the database of terrorist information that'll be used at airports. Likely this will be hosted on a 390 or something... Windows can't handle that kind of I/O.

    1. Re:EDS works with a variety of systems by jhines · · Score: 1

      American Airlines has been around for long time, I'd guess there is a lot of legacy code running on mainframes.

      Stuff like scheduling and cargo loading was mentioned.

    2. Re:EDS works with a variety of systems by Anonymous Coward · · Score: 0

      I spent a few months at AA just before Y2K as a consultant. There's tons of legacy code still running on mainframes from the '70s. Pretty unbelievable when you think of all the changes in the airline industry since then.

      --
      perl -e '$??s:;s:s;;$?::s;;=]=>%-{<-|}<&|`{;; y; -/:-@[-`{-};`-{/" -;;s;;$_;see'

    3. Re:EDS works with a variety of systems by johnnyb · · Score: 4, Informative

      Their reservation system is on Vax/VMS, if I remember correctly. I used to work in their midrange department, but I knew some of the VMS coverage guys. They have quite a diverse setup. The only operating system I _didn't_ see there was HPUX.

      They actually have a data center that is underground, and has a retinal scanner to get in (for some reason, our group got in with keycards - I'm not really sure why). Their tape library is about three times the size of my house. It's a pretty massive operation. Travelocity, hosted in the same location (but on the ground floor, not downstairs), is a bunch of huge SGI machines (8 processors and more each - probably about 30 of them).

      They run pretty much everything under the sun. I enjoyed being around the cool equipment while I was there, but absolutely hated the "big company" mentality, so I left after a year.

    4. Re:EDS works with a variety of systems by Anonymous Coward · · Score: 0

      Cool quine in your sig... how does it work?

    5. Re:EDS works with a variety of systems by Anonymous Coward · · Score: 0

      You are such a cunt. That fucking perl script trashed my system!!

    6. Re:EDS works with a variety of systems by Anonymous Coward · · Score: 0

      Are you working for American/US Airways?

    7. Re:EDS works with a variety of systems by sloveless · · Score: 1

      Don't log in as root next time you run strange perl scripts. You might want to install something other than Lindows this afternoon.

    8. Re:EDS works with a variety of systems by Anonymous Coward · · Score: 0

      Hehe, travelo means transsexual in french :)

    9. Re:EDS works with a variety of systems by colinrichardday · · Score: 1

      My Harrap's has it transsexuel or transsexuelle

    10. Re:EDS works with a variety of systems by Anonymous Coward · · Score: 0

      In fact, they recently got the contract to handle the database of terrorist information that'll be used at airports. Likely this will be hosted on a 390 or something... Windows can't handle that kind of I/O.

      Maybe that is why it crashed?

  9. I'm sure we can all guess what which OS tanked by Anonymous Coward · · Score: 0

    MVS

    1. Re:I'm sure we can all guess what which OS tanked by keithcstone · · Score: 1

      Nope. FOS is on TPF and odds are TPF didn't tank, the database got currupted. YOu can duplicate the hardware, make the software non-stop, but if some dork loads a bad module that currupts the data you're down.

  10. Gee, gods, ... by Anonymous Coward · · Score: 0

    ... am I glad I'm flying Delta next Saturday :-) :-) :-)

    Yeah, that helps,

    Toon Moene

    1. Re:Gee, gods, ... by Rosco+P.+Coltrane · · Score: 2, Funny

      ... am I glad I'm flying Delta next Saturday :-) :-) :-)

      Don't be so sure...

      --
      "A door is what a dog is perpetually on the wrong side of" - Ogden Nash
  11. Settle down, we've all seen this before... by the_seal · · Score: 4, Funny
    1. Re:Settle down, we've all seen this before... by Anonymous Coward · · Score: 0

      Not the same.

    2. Re:Settle down, we've all seen this before... by Anonymous Coward · · Score: 0

      It is amazing how often that happens at airports. I think I have a shot of a bsod at London Gatwick.

  12. Last thing you want to hear by xIcemanx · · Score: 5, Funny

    I'm guessing the last thing you want to hear on a plane now is the pilot saying, "What do you mean, fatal exception error?"

    >_ Why don't they swtich to Linux?

    1. Re:Last thing you want to hear by nairb774 · · Score: 1

      Then they put linux on the machines and if a core dump comes around I can her th pilot say, "What is a core dump? ... Who crapped on the computer?"

    2. Re:Last thing you want to hear by Brandybuck · · Score: 5, Funny

      Q: How far can the plane fly after a fatal exception error?

      A: All the way to the scene of the crash. Hell, it will probably beat the paramedics there by half an hour!

      --
      Don't blame me, I didn't vote for either of them!
    3. Re:Last thing you want to hear by afd8856 · · Score: 1

      by Ron White (White drunk in public)

      --
      I'll do the stupid thing first and then you shy people follow...
    4. Re:Last thing you want to hear by Col.+Klink+(retired) · · Score: 1

      Q: How long does a pilot have to fix a fatal exception error?

      A: The rest of his life.

      --

      -- Don't Tase me, bro!

    5. Re:Last thing you want to hear by Brandybuck · · Score: 1

      That's "publik"

      --
      Don't blame me, I didn't vote for either of them!
    6. Re:Last thing you want to hear by Anonymous Coward · · Score: 0

      Actually, that is irrelevant as far as flying the plane. If all the computerized devices on an aircraft fails, every plane that is FAA approved still works in manual mode. Yes even a 767 can be flown without any aid of a computer or any of the computerized readouts and controllers. I bet even I could fly one. Now landing that would be a different story.

  13. BSOL by Anonymous Coward · · Score: 4, Funny

    Blue screen of life. Because US Air cancelled the flight and we were forced to fly on a competent airline.

  14. Re:Stable Operating Systems by beebware · · Score: 1

    Yep, they just don't make computers as they used to. I've seen computers ran over by cars still working (only needed a couple of replace plastic case supports replacing), laptops dropped down 12 stories ...

  15. EDS by sql*kitten · · Score: 2, Interesting

    My experience with EDS is that problem is most likely to have actually been operator error. These people, and CSC are the absolute bottom of the barrel as far as outsourced data centres go. Yes IBM GS costs more, but there's a good reason for that! I'd sooner use Accenture than EDS, and that's saying something.

    1. Re:EDS by Anonymous Coward · · Score: 0
      I'd sooner use Accenture than EDS, and that's saying something.
      I'd sooner use a yellow legal pad and a pencil than Accenture, and that's saying even more.
    2. Re:EDS by Anonymous Coward · · Score: 0
      Something I noticed a while ago is that data center personnel in general are not the people I want with life and death power over my company. Like any job you have a few bright people and the rest are either borderline useful, useless or an actual liability. You have to count on the company itself having good procedures to prevent errors from getting out of control.

      I've only worked with EDS as a vendor so I've never seen their operations or support people, but the EDS people I dealt with did not seem to be worse than any other such company I've dealt with. Have you had multiple, personal experience of institutional rot at EDS that would cause idiots to be hired and a lack of proper operational procedures that could cause such an outage?

    3. Re:EDS by Anonymous Coward · · Score: 0

      I was going to take exception to that, but I just cant be bothered right now.

    4. Re:EDS by Detritus · · Score: 2, Insightful

      Staffing quality is rather strongly correlated to what the customer is willing to pay for labor. If the customer only cares about the bottom line, they get what they deserve.

      --
      Mea navis aericumbens anguillis abundat
    5. Re:EDS by dcocos · · Score: 1

      Actually Perot, sold of EDS several years ago, wait a couple more years and then started Perot systems.

    6. Re:EDS by sql*kitten · · Score: 2, Insightful

      If the customer only cares about the bottom line, they get what they deserve.

      Penny wise, pound foolish. Always the way, these days.

    7. Re:EDS by Anonymous Coward · · Score: 0

      surely you mean Andersen Consulting
      always knew they were crooked, never had them down as genuinely incompetent..

    8. Re:EDS by Anonymous Coward · · Score: 0

      If you are going to outsource your Data Processing to another company, you better expect at best, mediocre service. In a lot of cases, this will work, especially for companies that didn't have good service to begin with. But if you are talking about hosting airlines, then it's not going to work. Duh, why, because companies like EDS make their money by paying as little as possible for the tech personnel.

    9. Re:EDS by keithcstone · · Score: 1

      Duh, why, because companies like EDS make their money by paying as little as possible for the tech personnel. Duh, because the companies that hire them want to save money on a support function that is primarily wage based. To solve the problem companies "outsource" to save the trouble of laying off the people themselves. The outsources keep the "transitioned" people around long enough to move the work to some third world rat hole and still make the SLA's. You get what you pay for and in outsourcing the only people that come out on top are CEO's.

  16. EDS? Quelle surprise. by leathered · · Score: 5, Interesting

    Sorry, have to rant where I see EDS mentioned.

    EDS, in cahoots with the UK govenment, have wasted millions of pounds of taxpayers money on failed IT projects. Notable ones include the Inland Revenue (UK IRS), Child Support Agency (£50M over budget and still not working) and an email and directory service for the NHS (withdrew at last minute allowing C&W to steal at a much inflated price).

    Though the blame cannot completely be laid at the door of EDS, the government has been guilty of sloppy auditing and the worst being the willingness to hand over extra money when EDS has come around with the begging bowl.

    --
    For all intensive porpoises your a bunch of rediculous loosers
    1. Re:EDS? Quelle surprise. by Devar · · Score: 1

      EDS what now?
      EDS is the acronymn for Evil Day Star. Those of you with less knowledge of things may be inadverantly calling it the sun.

      Either way, it's evil, just like the EDS in the parent. :)

      --
      It's a Bagel.
    2. Re:EDS? Quelle surprise. by oddRaisin · · Score: 4, Insightful

      If we're going to lay blame, let's make sure we're spreading it evenly. A lot of contracts, and especially government ones, suffer from extreme scope creep. I have seen projects that started out with a 20 page description grow to over 150 pages by the end of the project.

      EDS and other large IT vendors try their best to discourage scope creep by making changes-after-the-fact billable for time and materials, instead of a negotiated cost. This makes the project go over budget. If the clients knew what they wanted at the begining, instead of wasting time and money doing engineering on the fly during the project, then the costs wouldn't be so high.

      Don't be so quick to slag EDS about the outage either. There are lots of factors out there that could have contributed. I have worked on projects where the clients say the servers are mission critical, yet can't be bothered to shell out money to upgrade from ultra-1 and ultra-5s, let alone pay for an HA solution. The technical people keep providing the justification and making the requests, but it's the project managers and accountants that really determine what kind of solution is feasible.

    3. Re:EDS? Quelle surprise. by jimicus · · Score: 2, Insightful

      If the clients knew what they wanted at the begining...

      Isn't this what Systems Analysts are supposed to do?

    4. Re:EDS? Quelle surprise. by Anonymous Coward · · Score: 1, Insightful
      I have seen projects that started out with a 20 page description grow to over 150 pages by the end of the project.

      If you have a 20 page project that is not 150 pages by the end of it, you're doing something wrong. The 20 pages tell you nothing. Every single requirement has to be clarified, expanded, all assumptions listed, etc. Most projects I've been on begun with a couple page description, and the final requirements documents ended up being well over 200 pages. The devil IS in the details, and you as the customer do not want the vendor weaseling out because something was not spelled out exactly, and you as the vendor do not want to be stuck with vague requirements that you don't know the exact scope and impact of.

    5. Re:EDS? Quelle surprise. by Anonymous Coward · · Score: 0
      Can you say bribery?

      I knew you could!

    6. Re:EDS? Quelle surprise. by tlk+nnr · · Score: 1

      I have seen projects that started out with a 20 page description grow to over 150 pages by the end of the project.

      What's the record for government contracts/descriptions?

      German Toll Collect (a country-wide, per-kilometer road use toll from trucks) had an 17000 page contract with the goverment.

      The system should have been running since mid 2003, it still doesn't work.
    7. Re:EDS? Quelle surprise. by demachina · · Score: 3, Insightful

      Don't think you can say much better or anything difference about CSC or the rest of the small cadre of IT companies who specialize in winning government contracts around the globe. They've had their share of multibillion dollar fiascos too.

      The problem with these companies is they specialize in winning big contracts. They put their best people on the proposals. They don't specialize in delivering great systems. Their best people probably move to the next RFP and they mostly fill the contracts with warm bodies.

      They can get away with it because its pretty rare for them to actually be punished for poor performance. If they get blacklisted by the agency that awarded the contract the agency ends up just replacing EDS with CSC or vice versa and the results don't get any better. I'd be interested if someone could cite a huge government IT contract that actually worked well. At some point governments need to figure out this methodology doesn't work and try something new.

      --
      @de_machina
    8. Re:EDS? Quelle surprise. by TheMysteriousFuture · · Score: 1

      what's your source? a quick google doesn't show it.

      --
      .sig
    9. Re:EDS? Quelle surprise. by Saeed+al-Sahaf · · Score: 1
      what's your source? a quick google doesn't show it.

      That's odd, "a quick google" produced about 43,000 hits for me.

      --
      "Who are in control, they are not in control of anything - they don't even control themselves!" - Glen Beck
    10. Re:EDS? Quelle surprise. by Kent+Recal · · Score: 1

      Isn't this what Systems Analysts are supposed to do?

      Most clients aren't willing to pay for a systems analyst.
      Or even have an idea why they really should...

    11. Re:EDS? Quelle surprise. by Anonymous Coward · · Score: 0

      ..an email and directory service for the NHS.

      Hey! We'll have none of that talk! Frank managed to get OpenLDAP running fine. It wasn't his fault Sendmail for Dummies hasn't arrived yet!

    12. Re:EDS? Quelle surprise. by jhpaes · · Score: 1

      My *personal* experience with EDS so far (on two completelly different occasions) is that they are:

      1) Incompetent
      2) Dishonest
      3) Arrogant

      This happened in two completelly different projects lasting many months each, in two different parts of the country, with two completelly different EDS teams, and in two different problem domains (both in the IT area, tho), spaced by 3 years. So it's hard not to generalize and say that EDS as a company does terrible service.

      Apart from my personal experience: here in Brazil at least, they have the worst possible reputation; just going around and asking people, couldn't find anyone who would put in a good word for them.

      One of the people I talked to told me that the only ones really good in EDS are their sales people and their lawyers: the sales people excel in selling overpriced contracts to customers, and the lawyers contrive such 'fantastic' contracts that when the client realizes how bad they are being had, they can't even quit, and if they try they are sued by EDS, just like Xerox was some years ago (Can you imagine that? A services company suing their customers!).

      So, it really looks like EDS is something to be avoided at all costs...

    13. Re:EDS? Quelle surprise. by Johnny+Mnemonic · · Score: 1


      At some point governments need to figure out this methodology doesn't work and try something new.

      They either will, or they'll be beaten by an enemy that does internal management better/more efficiently. How many terrorist attacks can we suffer that result in 200 Billion dollar wars, before our economy collapses for good? I'll bet Al Queda knows. In fact, I think it's their goal.

      Think it can't happen? What happened to the Roman, British, Spanish, French "world-wide" powers? Answer: they ran out of money to protect their worldwide interests. If we continue to fuck away money on imperial adventures, we will run out of money to support Israel. And then the terrorists win.

      --

      --
      $tar -xvf .sig.tar
    14. Re:EDS? Quelle surprise. by Anonymous Coward · · Score: 0

      That's nothing, I'm working in a privately owned company on a project that started out with no description apart from some ideas in my boss's head, still has no description apart from a lot more ideas in my boss's head and 70000 code lines. When I think it will be finished? Maybe around the year 3000. When it will be stable/reliable? Never. When it will be ready for a total rewrite (for the second time): About two years ago. But we don't have time for that.

    15. Re:EDS? Quelle surprise. by tswann01 · · Score: 1

      Cringely had an interesting column about EDS et al and the contracting process. Read it here.

  17. Steve Balmer by Stevyn · · Score: 2, Funny

    You know he's going to convince them not to switch to linux. First he's going to get on a plane...oh wait.

  18. Operating system? by SlamMan · · Score: 2, Informative

    Only one of the articles mention said anything about an "operating system." The rest called it a system problem. That does not necessarily mean an OS, or anything related to it. I think katu's reporter jumped to a conclusion.

    --
    Mod point free since 2001
    1. Re:Operating system? by Saeed+al-Sahaf · · Score: 1
      ...jumped to a conclusion...

      Here at /.? No!

      --
      "Who are in control, they are not in control of anything - they don't even control themselves!" - Glen Beck
    2. Re:Operating system? by Anonymous Coward · · Score: 0

      I agree, I can't believe no one else read this and picked it up.

      I don't think the writer jumped to a conclusion though, I think the writer doesn't know that the phrase "operating system" doesn't just refer to a computer system in general.

      I'd guess that the "operating system" is actually nothing of the sort, but the reporter just didn't know how to describe it.

    3. Re:Operating system? by SlamMan · · Score: 1

      I expect is from us /.ers, but I try to at least pretend that reporters try to keep it to a minimum.

      --
      Mod point free since 2001
  19. doesn't it say by Viceice · · Score: 1

    Isn't it stated somewhere that a cetain OS, which is forever fair game in this community, should not be used for 'Mission Critical' situations?

    --
    Sometimes I wish I was a plumber, then I'd know how to deal with other people's shit.
    1. Re:doesn't it say by Anonymous Coward · · Score: 0

      It wasn't. EDS runs UNIX and Linux systems as well as IBM mainframes.

      By the way, when's the last time you saw Microsoft.com go down? Surely that website is hit harder than most anything on the next. Don't believe me? Everyone click here. Just saying, don't treat operating systems like religions.

    2. Re:doesn't it say by CaptainTux · · Score: 2, Insightful
      By the way, when's the last time you saw Microsoft.com go down? Surely that website is hit harder than most anything on the next. Don't believe me? Everyone click here [microsoft.com]. Just saying, don't treat operating systems like religions.

      I'm sure the MS website experiences problems all the time. The difference is that, because it's such a large scale website and the company took the time to plan for failure, there is a lot of redundancy.

      Microsoft has *at least* three levels of redundancy on the corporate web site and possible more on the MSDN side of things. It doesn't *appear* to go down but, believe me, it does.

      That said, realize that while it might be popular to try to blame the OS for this failure it really has nothing to do with the OS. Be it Linux, Windows, Solaris, Unix, or even Mac, one CAN build a stable and reliable system with nearly 100% uptime if you take the time to plan, plan, plan. This isn't a software failure as much as it is a human one.

      --
      Anthony Papillion
      Advanced Data Concepts, Inc.
      "Quality Custom Software and IT Services"
    3. Re:doesn't it say by moby · · Score: 2, Informative

      By the way, when's the last time you saw Microsoft.com go down?

      Uhh ... like 2 days ago.

      http://www.informationweek.com/story/showArticle.j html?articleID=12808118

    4. Re:doesn't it say by Kent+Recal · · Score: 1

      In fact, the MS approach to availability has been (and i suppose still is) clustering. They know their software can't run straight for months or (gasp) years so they deploy n machines for a given task and shedule regular reboots in a roundrobin fashion.

    5. Re:doesn't it say by Anonymous Coward · · Score: 0

      did you even read it?

      Microsoft's Web Site Brought Down By Attack Aug. 1, 2003

  20. Re:Windows by Anonymous Coward · · Score: 2, Funny
    Windows was a given, now for bonus points, tell us which version.

    Microsoft Bob. Now, where do I go to collect my bonus air miles?

  21. I thought everyone knew by Xerp · · Score: 5, Funny

    NEVER open Windows in an airplane!

  22. My guess ... by Tim+Ward · · Score: 2, Interesting

    ... would be a hand-crafted real time kernel, written in assembler, running on an IBM 360 mainframe - isn't that still what drives critical aviation systems?

    1. Re:My guess ... by sysjkb · · Score: 4, Informative
      Sounds very unlikely to me. You will find weird custom S/360 derivatives in places like the Space Shuttle, but coordination and route planning doesn't sound a likely place for one.

      Of the 360-based operating systems, IBM's TPF has a major presence in the airline industry, but this probably isn't the system in question. TPF tends to handle ticketing and reservations. TPF stands for the Transaction Processing Facility; it's the descendant of the old Airline Control Program (ACP) developed for Sabre. Sabre in fact is still running TPF, although I believe they're busy transitioning away from the mainframe to Tandem's er I mean Compaq's er I mean HP's NonStop/UX.

      Of course, it might not be an IBM mainframe at all; Unisys has a niche in the airline industry. But heck; given that this is route planning, just about anything from AIX to z/OS is a possibility. Even *shudder* Windows.

    2. Re:My guess ... by Tore+S+B · · Score: 1

      No, it's on a 1970s Tandem mainframe, with PDP-11/34 terminals. No, I kid not :)

      --
      toresbe
    3. Re:My guess ... by Anonymous Coward · · Score: 0

      So what you're saying is that the teletype ran out of paper?

    4. Re:My guess ... by keithcstone · · Score: 1

      No it's not, never has been. FOS (Flight Operations System) is on TPF. Since TPF doesn't go down for more than a couple minutes a year, and the hardware IS redundant, the only plausable explanation is the one given then mangeled by the press. The database was currupted. That's the achilles heel of all real-time systems, since while you can duplex it and do all sorts of stuff to keep it up, the data itself is shared between the primary and back system (and has to be to support fallover).

  23. That's what you get when you use EDS by mysterious_mark · · Score: 1

    Again another example of EDS shoddiness, why anyone would give money to EDS for anything is beyond me, they deliver inferior service at outrageous prices. M

    1. Re:That's what you get when you use EDS by Anonymous Coward · · Score: 0

      Doesnt the word "backhander" mean anything to you?

    2. Re:That's what you get when you use EDS by luke923 · · Score: 1

      Actually, I saw the contract EDS signed with AMR to support AA's infrastructure. EDS was so eager to get the deal, I wouldn't be surprised if EDS loses alot more than they get from the deal. Then, there's ways for AMR to save even more money, such as EDS not filling out trouble tickets properly.

      Then again, EDS - like most outsourcing companies - is nothing more than a dog-and-pony show, where if you make the 'client' believe that the problem is solved, you'd be better off than actually fixing the problem. You get what you pay for.

      --
      "Good, Fast, Cheap: Pick any two" -- RFC 1925
  24. Wild speculation by MisterP · · Score: 3, Insightful

    They aren't telling the whole story.

    I come from Solaris/Veritas/Oracle and Redhat/Oracle RAC environments. One single system going down cannot take out the service. Database HA is somewhat complicated and expensive, but it's not rocket science, regardless of platform.

    I find it very difficult to believe that they would have any single points of failure in a system of that importance. Blaming MS is the easy way out.

    1. Re:Wild speculation by Anonymous Coward · · Score: 0

      "Blaming MS is the easy way out."

      And I'm easy. It's all Microsoft's fault!

    2. Re:Wild speculation by treat · · Score: 2, Insightful

      Standard database clusters use shared or replicated database storage. If you trash the database - a human error - the cluster is useless.

    3. Re:Wild speculation by Archibald+Buttle · · Score: 4, Interesting

      Blaming MS is the easy way out.

      I just read all the stories that were linked to this article.

      None of them blamed Microsoft. In fact the only blame pushed in their direction was your comment...

      The articles did say that there was a problem with the operating system. Now we don't know who exactly said this, or what they said precisely, so it is quite possible that this isn't entirely accurate reporting.

      I find it very difficult to believe that they would have any single points of failure in a system of that importance.

      I agree it's unlikely, but it is possible that there is a single point of failure in their system. There are a great deal of shoddily engineered systems in use today.

    4. Re:Wild speculation by Anonymous Coward · · Score: 0

      Blaming MS is the easy way out.

      But... this is slashdot. Anything for a giggle!

    5. Re:Wild speculation by christophersaul · · Score: 1

      Which is why you deploy an infrastructure which can roll back errors that have been made.

    6. Re:Wild speculation by Antitorgo · · Score: 1

      You're right... American's Operations Center runs on Solaris, Oracle and Macintosh (Last I know of it was MacOS 9 (ugh)). If they were running on Windows, I think the media would be shouting it from the rooftops.

      So, I guess that means that all those Windows jokes will go right out the window?

      If I had to guess, since it apparently was affecting flight ops as well as the loading ops, it was most likely something in the DB or maybe a physical networking problem. The last I knew, those were two seperate programs.

    7. Re:Wild speculation by uucp · · Score: 1

      They aren't telling the whole story.

      1. Multiple airlines experience technical issues downing air traffic for several hours.
      2. New York's Terror Alert Level rises as do concerns about terrorists attacking financial institutions.
      3. ???
      4. Profit!

      Sadly I believe that this is how our leaders actually reason.

      I am *so* stoned right now.

      --
      Sig (appended to the end of comments you post, 120 chars)
    8. Re:Wild speculation by underworld · · Score: 1

      Why does it have to be a single failure that caused the outage? I have seen my fair share of system outages that were caused by multiple concurrent issues. In fact, I believe that last year's massive power outages was a good example of an outage caused by multiple failures in a system designed to handle only single points of failure.

      The idea is that the odds of a single failure are like 100,000 to 1 making the odds of 2 concurrent failures 100,000 * 100,000 (10,000,000,000) to 1. And while that is highly unlikely, it is still incredibly possible and occassionally will still happen.

    9. Re:Wild speculation by sql*kitten · · Score: 1

      If you trash the database - a human error - the cluster is useless.

      One word for you my friend: Flashback. You say to Oracle "restore this table to the state it was in 5 minutes ago" and it's done, all transactionally consistent. No messing around with backup tapes and the like. It's enormously powerful, takes the concept of "rollback" to a whole 'nother level.

      As if DBA egos weren't out of control already, now we have control over TIME ITSELF! BWAHAHA! Back, sea! Back!

  25. Re:Windows by Anonymous Coward · · Score: 5, Informative

    The following entities were NOT mentioned in the article you're linking to:

    (1) American Airlines,
    (2) US Airways,
    (3) EDS.

    So, what the hell are you talking about?
    Why did you link to this article?
    (I know, I know, because nobody will read it anyway)

  26. Re:Windows by ChatHuant · · Score: 4, Informative

    Sounds like a troll. The article quoted by the parent is about a small regional airline (Atlantic Coast Airlines) that's doing its IT work internally. The article doesn't mention EDS at all. Moreover, browsing EDS's site, you can see that the solution they implemented for Continental Airlines is UNIX-based.

  27. Not Windows, Unix by JohnQPublic · · Score: 5, Informative

    This is undoubtedly a problem with Sabre, which EDS runs on behalf of Sabre Holdings. Both American Airlines and US Airways use Sabre for much of their operations.

    Sabre started it's life as an American Airlines internal system (SABER, slight spelling difference), running on a rare operating system (PARS, later called ACP and currently TPF) on IBM mainframes. In the last few years Sabre completed a lengthy migration to HP Unix on Non-Stop (i.e. ex-Tandem) hardware. The mainframe systems were rock solid, but software talent was hard to come by, so they decided the time had come to switch.

    Sorry, no Microsoft to blame here!

    1. Re:Not Windows, Unix by justins · · Score: 2, Interesting
      This is undoubtedly a problem with Sabre, which EDS runs on behalf of Sabre Holdings. Both American Airlines and US Airways use Sabre for much of their operations.

      They use the same system for flight operations and for reservations? I've seen Sabre in use at the travel agent's office, somehow I would have thought this problem involved a different system...
      --
      Now before I get modded down, I be to remind whoever might read this that what I am saying is FACT. - bogaboga
    2. Re:Not Windows, Unix by Stinking+Pig · · Score: 2, Interesting

      I concur, though there are a number of middleware apps around Sabre that could have conceivably had an issue. However, those apps that I've been made aware of are almost universally Java on Solaris.

      --
      "Nothing was broken, and it's been fixed." -- Jon Carroll
  28. Guess IT Does Matter by Anonymous Coward · · Score: 2, Funny

    well, i guess IT Does Matter after all....

  29. You don't understand how databases work! by Safety+Cap · · Score: 1

    Any attempt to retrieve information from them (flight data, schedules, FOIA requests) will result in total, immediate, and irreversable loss of data!

    --
    Yeah, right.
    1. Re:You don't understand how databases work! by Anonymous Coward · · Score: 0

      The system's document handling software, itself an antique, operates on Microsoft Windows 95.

      Wholey Marie, Muther of Gawd.

      Yeah, it might be dangerous to try a copy. Sometimes you just have to start the whole thing over, the right way.

  30. Re:Windows by macdaddy357 · · Score: 2, Funny

    Not only that! It's Windows 98.

    --
    How ya like dat?
  31. Re:Windows by Anonymous Coward · · Score: 2, Funny
    The following entities were NOT mentioned in the article you're linking to:

    (1) American Airlines,
    (2) US Airways,
    (3) EDS.

    So, what the hell are you talking about?
    Why did you link to this article?
    (I know, I know, because nobody will read it anyway)

    You are such a nitpicker.

  32. here's what happened by Scythr0x0rs · · Score: 3, Funny

    "It looks like you are flying an aeroplane, Would you like help?" YES!

  33. EDS by sinnfeiner1916 · · Score: 0

    That's H. Ross Perot's company right? I bet that probably explaines it.

    --
    The More Laws, the less Justice --Marcus Tullius Cicero
  34. What *REALLY* happened... by catdevnull · · Score: 5, Funny

    At about 4:30 a.m., the outsourced SysAdmin was setting up to do routine patches to Windows 2003 server nodes. But just before, he decided to check his e-mail with Outlook and he opened an important message from his system administrator advising him that his e-mail would be de-activated if he didn't open the important attachment. I think we all know what happened after that...

    --

    I might know what I'm talkin' about, but then again, this is Slashdot...
  35. So they had a system crash ... by mec · · Score: 1

    ... were any human beings killed or injured? ... were any human beings in danger of being killed or injured?

    It's hard to tell from the sketchy news stories, but it looks like AA and UA *do* have a backup plan and *are* executing it. The backup plan is a ground stop for 2-3 hours while they sort things out.

    If you want them to have a backup plan which involves providing full service with no interruptions, then you would have a ticket price to fund that.

  36. Not Smart Enough by PingPongBoy · · Score: 3, Interesting

    It seems that computer failures are not very graceful. In a large business if an employee or even the chairman of the board is sick, the business still runs. However, failure of the central computer means no one knows how to make anything run.

    Perhaps the efficiencies of a computerized business offset the cost of short downtimes, and the business is able to grow to the complexity that it isn't worth running without the computer. A 2 or 3 hour stoppage once in a blue moon (that was last month, and it looked big) might not be worth working around.

    All the same I'm hesitant to let computer failures stand in the way of normality. Major infrastructure may be interrupted by nature but it can be scary for it to be stopped by computer problems. Who knows how long the system will be down? Who knows how much damage to information went unnoticed? Who knows what errors still exist?

    Increasing computerization causes increasing paranoia. Guard yourself prophylactically? Ask hard questions before entering relationships with big business? Insist on financial compensation against computer delays?

    Computer systems need to be built with more safeguards (redundancy, logging, checkpoints, backups), isolation of failure, data accessibility during failure (example: Windows safe mode) even for end users, etc.

    --
    Know your pads. One time pad: good for cryptography. Two timing pad: where to take your mistress.
    1. Re:Not Smart Enough by Anonymous Coward · · Score: 0

      I think you are missing a big point with most poeple and thus most businesses. GREED. They will buy a somewhat or even very inferrer product are lower price. I have seen poeple take bids for redoing the driveway. Some Pilipinos got the job. They were not too bad it looked good to me. The problems that came along later were not pretty. The contractors were of course no where to be found. If you want a system that is truely robust you are going to have to spend big bucks. The trick is that some people who charge a lot can also never get the job done. I think that in designing software one has to looked at the software is needed and even if not prefect can start saving the compiany money right away. If the old system has like 70% up time and your new system which could us some polishing is about 95% up time. It will seem like a good idea to just start using the new system right away. The problem is that once it is running who wants mess with a good thing and try your new 99.9% uptime system. My grandfather used to say that one could build a car that would not get damaged at 70mph and run for 500,000 miles without service. The problem is it would be to expainsive and a big waste of resources.

  37. Re:Windows by Anonymous Coward · · Score: 0

    I agree -- trolling. How is it that the original anonymous author gets 4 points for trolling, yet ChatHaunt is left with a mere 3?
    Would be funny if American contracted to EDS to use Windows, but that doesn't appear to be the case.

  38. Linux? by bogaboga · · Score: 1

    Anyone know whether these airlines are using Linux in any [mission critical] way? BTW, what would you consider as the most important mission critical system? Could it be in Banking, Hospitals, Airline systems, Educational, Nuclear....? What is a mission critical system anyway?

    1. Re:Linux? by HR · · Score: 1

      Mission critical just means that in order for the business to operate, the system must be functional.

    2. Re:Linux? by t_allardyce · · Score: 1

      Depends what the backup is - Hospitals could survive on paper records for a week, banks often have their local computers crash (windows terminals) and im sure their big systems have some sort of fail-safe (some paper receipts and a guy taking phone calls but at a slower rate?) educational? who cares, air traffic control has paper backup - all the planes are recorded on slips and planes can be flown by hand. nuclear plants will just drop the control rods in at the first sign of trouble. any company thats sole purpose is on the web (online store etc) will be in trouble. But the real mission critical system of them all is solitare imagine the chaos!

      Also coffee, if that coffee machine goes down you've had it!

      --
      This comment does not represent the views or opinions of the user.
    3. Re:Linux? by Mister+Liberty · · Score: 1

      My Linux box with xine or mplayer running
      my favorite p*rn would be an example of a
      Mission critical system. I mean, imagine
      me sitting there all set and ready to go,
      and happens a system crash. You bet I'd
      describe that at best as a big letdown, and
      at worst as a situation where I'd really be
      f*cked.

    4. Re:Linux? by bgbarcus · · Score: 1

      > planes can be flown by hand
      >
      Not always. A complete computer failure (about as close to impossible as is possible) in a fly-by-wire plane could really ruin your day.

    5. Re:Linux? by mrshadow76 · · Score: 1

      The most important is the telecommunications sector. They provide the communications between business. This runs the economy. No internet, no long distance equals no business. No business equals no money.

  39. TERROR WARNING LEVEL ORANGE!!!! by Anonymous Coward · · Score: 0

    We're all gonna die!

  40. Re:Windows by Anonymous Coward · · Score: 1, Interesting


    The system in question is most likely "AirFlite",
    a Unix based system hosted by a joint
    venture between Sabre and EDS.

  41. Oh well... by Anonymous Coward · · Score: 0

    In Soviet Russia, US Airways grounds database glitches anyway. :-D

  42. Do you know the cost of redundency ? by aepervius · · Score: 4, Interesting

    Here around we studied it, for one major airline in EU. We wanted a "backup system" in case the main system went down. Total Cost, without maintenance, about *3 whole day* of traffic "benefits"... Yes, that much. Right now the project is still discussed but most of us thinks it is dead in the egg. Instead the "older" and "less powerfull" developpement system will be used in case of break down.

    Redundancy is OK, as long as it is not bleeding you dry.

    --
    C. Sagan : A demon haunted world:
    http://www.amazon.com/gp/product/0345409469/
    visit randi.org
  43. Re: And in related news by Black+Parrot · · Score: 0, Troll


    > the terror level has just been raised to "Orange" high at the same time that Tom Ridge announced that after the election he will reteire due to financial difficulties. I guess $175,000 a year of tax payer money just doesn't buy a lot now a days

    Probably has more to do with the latest election polls.

    --
    Sheesh, evil *and* a jerk. -- Jade
  44. You probably won't hear it by Anonymous Coward · · Score: 5, Informative

    The systems that run the aircraft and the navigational and communication systems really are redundant. It's the law. It also means that usually there are two different ways to do something not just the same thing repeated twice.

    Example 1 - The pilot and co-pilot can't eat the same meal. That way, only one of them can get food poisoning.

    Example 2 - The hydraulic system fails and the wheels won't go down. There's a hand crank.

    Example 3 - The communication systems at every tower I have worked at have two separate backbones. There are two of absolutely everything. If that fails, there are emergency radios under the desk. If the emergency radios don't work ... We used to joke that the controllers would climb to the top of the tower and wave fire extinguishers to warn the planes away. (I think it was a joke.)

    Example 4 - You can't fly very far over open water in a single engine aircraft.

    It used to be frustrating working on systems older than I was but we never had to worry about surprises.

    Of course all of this redundancy is very expensive. You spend the money where people's lives are at stake. On the other hand, if the worst problem is that some planes will be late, perhaps you don't spend the big bucks.

    1. Re:You probably won't hear it by Zeinfeld · · Score: 1
      Example 1 - The pilot and co-pilot can't eat the same meal. That way, only one of them can get food poisoning.

      Yeah just as well they have that rule, just imagine the arguments if the co-pilot was allowed to eat the captain's lunch. Do they have rules that stop them both sitting in the same seat as well?

      Example 2 - The hydraulic system fails and the wheels won't go down. There's a hand crank.

      Its the same thing with software, no shortage of cranks.

      --
      Looking for an Information Security student project suggestion?
      Try http://dotcrimeManifesto.com/
    2. Re:You probably won't hear it by Anonymous Coward · · Score: 0

      Do they have rules that stop them both sitting in the same seat as well?

      That's covered by sexual harrassment law.

    3. Re:You probably won't hear it by the+pickle · · Score: 1

      Example 3 - The communication systems at every tower I have worked at have two separate backbones. There are two of absolutely everything. If that fails, there are emergency radios under the desk. If the emergency radios don't work ... We used to joke that the controllers would climb to the top of the tower and wave fire extinguishers to warn the planes away. (I think it was a joke.)

      It's not a joke. They have light guns (think God's Own Spotlight here) instead of fire extinguishers, and they use them from inside the tower. The light guns are typically used for communication with airplanes that have lost their radios. The tower systems are significantly more reliable than the airborne systems are.

      p

    4. Re:You probably won't hear it by RadioTV · · Score: 2, Insightful

      If the emergency radios don't work ... We used to joke that the controllers would climb to the top of the tower and wave fire extinguishers to warn the planes away.

      I though that was what hand lights were for.

      --
      I have great faith in fools - self confidence my friends call it. - Edgar Allan Poe
    5. Re:You probably won't hear it by lnxpilot · · Score: 1

      Example 3: if the radios fail, tower will use the standard light signals:

      http://www.risingup.com/fars/info/part91-125-FAR .s html

    6. Re:You probably won't hear it by Anonymous Coward · · Score: 0

      At a large airline I used to work at, the backend ticketing and checkin process is pretty redudant also.

      Here is some typical setup for a mid sized airline terminal that was installed about 3 years ago.

      FDDI ring connecting all of the communications rooms together, multiple switches in each communications room and all computer home run to the switch (no small unmanaged hubs/switches or rig jobs). At any gate or ticket counter, every other PC connects to different switch. There are multiple servers (if a server is even needed as the ticketing and checking is all handled by the mothership mainframe not located on site) and each different gate connects to different one. That may be a muddy explanation but it results in worst case scenario of 3 out of 4 computers going down). Each communications room can have no more then 50% of the switch ports populated so they could move everything over to another chassis/rack/blade if needed. Each switch has 2 power supplies and one is attached to a rack mount UPS, the other straight into the wall (I've seen more UPS failures then power failures) and each on a different circuit.
      Two of the communications rooms have routers and a CSU/DSU to get out (of course I've seen both routes knocked out when a car hit a single pole nearby, I believe someone high up at AT&T got involved in that one). Each router and pipe must be able to carry the entire traffic load if needed. All equipment can be remotely power cycled through a seperate dialin system. Users are well trained on how to reroute printed tickets and general preflight printouts to other printers if needed.
      Very very litle network traffic leaves the airport that is not directly related to the ticketing and gate operations. Meaning, very little internet access for employees and it is tightly controlled, no video streaming to headquarters, no voip, software updates are sent to a local server and then distributed etc..
      You can easliy run a small/mid sized operation on a single 128kbit connection, a medium/large hub can breeze by on a single T1 (of course they have two anyway).

      Keep in mind, the airline creates and maintains 100% of their own network in the airports. The airports may "own" or be responsible for some of the physical fiber, long cable runs, or the long term power backup but they do not get involved in anything else.

      Things do go down but preplanning and everyone aware on how to work around what they need to do causes very few delays from a computer malfunction.

    7. Re:You probably won't hear it by Anonymous Coward · · Score: 0

      You can't fly very far over open water in a single engine aircraft.

      That's just not true. In fact, I've been in a single-engine plane that took off from and landed on open water.

    8. Re:You probably won't hear it by Anonymous Coward · · Score: 0

      Forgte a cloding tag? "Preveiw is you're freind.

    9. Re:You probably won't hear it by Anonymous Coward · · Score: 0
      Example 4 - You can't fly very far over open water in a single engine aircraft.

      Don't tell the US Navy. They're headed back toward single engine carrier based aircraft.

    10. Re:You probably won't hear it by sharkey · · Score: 1
      Example 1 - The pilot and co-pilot can't eat the same meal. That way, only one of them can get food poisoning.

      Now, that's a very good idea.

      --

      --
      "Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
  45. Re:And in related news by Gzip+Christ · · Score: 2, Insightful
    the terror level has just been raised to "Orange" high at the same time that Tom Ridge announced that after the election he will reteire due to financial difficulties. I guess $175,000 a year of tax payer money just doesn't buy a lot now a days.
    That's only for DC. The terror level is still "Bert" for the rest of the country.

    Regarding his pay, Ridge has got to have one of the most stressful, time consuming, and important jobs in the country, and as such I for one do not think that $175K is nearly enough. Corporate officers frequently make more than that, so why would anybody willingly subject himself to a much more stressful and dangerous job for less money? I've got to side with Ridge.

  46. But EDS has such wonderful commercials... by Anonymous Coward · · Score: 0
    With commercials such as the wonderful 2001 super bowl ad, Herding Cats, how could they be a shoddy company?

    Now if you'll excuse me, I've got to go order some generic via..., er, drugs, from this flashy animated gif popup e-mail I just got.

    [But seriously, the commercial is my all time favorite.]

  47. Air Canada at the same time too by acidrain · · Score: 2, Interesting

    Sorry, I think this is happening to a number of Airlines:
    href=http://www.ctv.ca/servlet/ArticleNews/story/C TVNews/1091237095342_4/?hub=TopStories

    Probably just the CIA moving them all onto some big CIA super-computer.

    --
    -- http://thegirlorthecar.com funny dating game for guys
  48. Re:Windows by 88NoSoup4U88 · · Score: 1
    So how much Saudi Arabians have been flewn back when all the others airplanes were stranded ? :P

    On a more serious note ; a few weeks ago (two i think) : We also had a software problem in the Netherlands (Schiphol Airport) : Causing the monitors which display the departures/arrivals, to go blank : Only to get working 2 hours later.
    This also caused great delays, as people had to ask to the airport personnel where to go.

  49. computer error by moartea · · Score: 1

    was it something like this ?
    http://www.cs.utk.edu/~shuford/terminal/airport_bs od.jpg

  50. EDS by nuxx · · Score: 0, Offtopic

    Interestingly, last night while at a party I burned my copy of an EDS book which was ghost written for the old CEO. Yep, the same CEO that got EDS investigated by the SEC, caused a bunch of employees to lose jobs, etc.

  51. Re:Windows by NineNine · · Score: 2

    Kid, you don't know what in the hell you're talking about. EDS is a mssive, global company that does hundreds of millions in sales a year. They handle systems for thousands of large companies. To say that EDS uses Windows is fucking moronic. A company as large as EDS, and doing as many diverse things as they do probably has every OS ever invented running somewhere in the organization. Sheesh. Slashdot needs some age requirements for posting.

  52. Campaign of fear by Anonymous Coward · · Score: 0
    What a surprise.

    Ridge's reporting about specific threats and the threat levels are increased. Just in time for Bush's comeback after DNC.

    The bastard even gloated that this specific information (which will not be shown to us, of course) was obtained because of GWB's "solid leadership".

    Bush's campaign is back on track and fuelled by FUD.

  53. Re:Windows by Anonymous Coward · · Score: 0

    It's not the OS (Bad) but EDS's habit of billing multiples of what they pay and their moronic managment style. Fuck EDS(Every day is shit)

  54. This isn't what you think by xenophrak · · Score: 3, Interesting


    Even though this sounds dire, I have a feeling that this does nothing to compromise airline safety.

    From the sounds of it, the flight planning system went down. This is a ground-system only, often a terminal next to the ticket checking counter. The purpose is to file flight plans, check weather airport conditions, etc. It is not an onboard system. This would not have likely decreased passenger safety.

    The reason that the FAA got involved was because AA decided to ground the planes because the pilots most likely couldn't file flight plans electronically. If left to the filing flight plans the old way, it would have delayed things more and caused more headaches to just wait out the system outage.

    However, when any business runs and depends on a particular piece of software to generate revenue and to provide a service, I would be more inclined to host such a system on something like a mainframe or at least a big Unix server.

    --
    Contrary to popular belief, life is not a bitch. It is far far worse.
    1. Re:This isn't what you think by Anonymous Coward · · Score: 1, Funny

      Even though this sounds dire, I have a feeling that this does nothing to compromise airline safety.

      Actually, it does:

      1. Airline flight scheduler mainframe goes down for three hours. Three hours of revenue are lost.
      2. Customers affected by poor service migrate to other airlines. Additional revenue is lost.
      3. Airline attempts to recover revenue by cutting costs.
        1. Jet maintencance schedules are lengthened.
        2. Frequency of mechanical failure increases.
      4. Pilots' wages are decreased or frozen.
        1. Lower-skilled pilots are hired.
        2. Frequency of pilot error increases.

      Safety is comprimised.

  55. "Just technological" by Greenspan · · Score: 1

    "Hotard said the problem was purely technological."

    Oh, what a relief!

    That reminds me of the Seinfeld episode "The Betrayal" where the gang goes to India and George finds out Jerry slept with Nina:

    JERRY: Alright, I admit it. I slept with Nina, but that's all.

    GEORGE: (Outraged) "That's all"?! That's everything! I don't know what all the rest of it is for anyway!

  56. Yeah Linux has 100% uptime with no bugs by Anonymous Coward · · Score: 0

    I'd say the fact that nothing like this has happened or been heard of in years, I'd say its pretty reliable considering the small margin of error.

    Your comment is typical of the attitude that hinders the oss movement itself. Get a clue.

  57. Acronym homophones? by Mr+Guy · · Score: 1

    BSOL? Used in a sentence: If you wanted to fly USAIR you'll BSOL?

  58. UNIX by rd_syringe · · Score: 1

    According to this, EDS runs UNIX-based systems.

    Nice try, though.

    1. Re:UNIX by WindBourne · · Score: 1

      EDS pushes Windows. I would not be surprised if this was not an attempt to switch to Windows and it hiccuped. Think in terms of when Hotmail hiccups. Every time, it is MS trying to switch out the back-end that is doing the real processing.

      --
      I prefer the "u" in honour as it seems to be missing these days.
    2. Re:UNIX by Erik+Hollensbe · · Score: 1

      If you haven't crashed a linux box, you obviously aren't pushing it hard enough.

      Unix is not a panacea. Shit happens.

    3. Re:UNIX by deaddrunk · · Score: 2, Insightful

      In this case the OS is irrelevant. EDS are very good at screwing things up by making promises they can't deliver on and then rushing things in.

      --
      Does a Christian soccer team even need a goalkeeper?
    4. Re:UNIX by Anonymous Coward · · Score: 0

      But fi you are crashing Windows, then you are asking it do to much, so quit moving the mouse, or keyboard.

    5. Re:UNIX by Anonymous Coward · · Score: 0

      Yeah but look at their alliancies at the bottom of this page

      Case Studies

      they do Microsoft too..

    6. Re:UNIX by Codifex+Maximus · · Score: 1

      > If you haven't crashed a linux box, you obviously
      > aren't pushing it hard enough.

      I crashed a Linux box a couple times... I just can't remember back that far.

      > Unix is not a panacea. Shit happens.
      Unix/Linux/BSD are the closest things to a panacea as far as open operating systems go. IMNSHO

      --
      Codifex Maximus ~ In search of... a shorter sig.
    7. Re:UNIX by Anonymous Coward · · Score: 0

      Then the pilots are supposed to make up the lost time in the air so as not to make everyone miss their connecting flights.

    8. Re:UNIX by Anonymous Coward · · Score: 0

      Unix/Linux/BSD are the closest things to a panacea as far as open operating systems go.

      Lord fucking help us all in that case.

      Sorry to break the news to you, but they're really not. In fact, a lot of things about Unix, Linux and BSD are downright shit.

  59. I found the root of the problem by Anonymous Coward · · Score: 5, Funny

    There is a line of code that raised the problem but is commented in Punjabi, I think it says "fuck this $3/hour job".

  60. MOD PARENT DOWn by WindBourne · · Score: 2, Informative

    Folks, Not only does the link say nothing about windows, but AMR's flight res system (SABRE) is located in Tulsa in a silo and is absolutely not Windows(IIRC, IBM mainframes). Now, it is more likely not their flight system but some immeadiary system. While it is likely to be Windows (based on past history), there is so far no comfirmation of that. In addition, historically, AMR did not run windows as it was too expensive and too prone to crashes. Of course, that was when R. Crandel was there, which was a while ago.

    --
    I prefer the "u" in honour as it seems to be missing these days.
  61. not windows by Anonymous Coward · · Score: 0

    Don't be silly, it's not windows. It has to be something else because big corporations don't run mission-critical applications on Windows.

  62. Re:Windows by Anonymous Coward · · Score: 0

    Slashdot needs some age requirements for posting.
    No, it needs some age requirements for moderating. YHBT YHL HAND

  63. Offtopic reply to sig by gaijin99 · · Score: 1, Offtopic
    Nice sig, but shouldn't it be "support diversity", or "support weirdness", or something similar? The first ammendment just guarantees people a right to speak, it doesn't mean I have to listen.

    For my money, I see so many "Fuck Whoever", and "GNAA" posts when I read at -1 that I only bother when I'm moderating. Their first ammendment rights a) don't apply to a privately owned board, and b) don't mean I have to wade through the crap they spew to see the good stuff.

    I'm completely offtopic here, but it bugs me to see the whole "first ammendment guarantees an audience" argument.

    --
    "Mission Accomplished" -- George W. Bush May 1, 2003
    1. Re:Offtopic reply to sig by back_pages · · Score: 1
      Ah, but now you're infringing on the fabled "Everybody is really quite special" clause in the, um, Bill of Rights or something, which guarantees that each individual (and his or her or its speech) gets preferential treatment. You know what I'm talking about. "Sure, everybody knows that nobody gives a rat's ass about morons, but when I'm the one being a moron, then I'm exercising my Right to Free Speech and the whole world must listen!"

      It's really only a hop, skip, and a jump to a socialist utopia once we get a functional implementation of the "Everybody is really quite special" clause. And then, brother, all your problems will vanish and it will be nothing be food and drink for the rest of your days.

      That or the grandparent post's sig is full of crap. Six of one, half dozen of the other.

  64. Image Incorrectness by moby · · Score: 1

    That is not a BSOD.
    That is simply a Fatal Exception Error.
    With a FEE, there is still a chance that the OS may recover ... after you hit the key to continue about 20-30 times.
    A BSOD does not offer the ability to hit any key to continue but rather dumps your RAM to disk and then stares at you !

    1. Re:Image Incorrectness by Anonymous Coward · · Score: 0

      See now Microsoft need to make it clearer to the end user that there is a difference between a FEE and a STOP. Both need to stand out more. Maybe they could color the FEE with yellow text and border, and STOP's in red text and border, and make them flash..

    2. Re:Image Incorrectness by Anonymous Coward · · Score: 0

      You sound like BSOD's only happen on NT/2k/XP... Because those are the systems that dump the RAM to disk on a fatal error.

      Win95/98/Me gives this blue screen, and allows you to press a key to get back to (usually) a black screen, where nothing happens at all.

    3. Re:Image Incorrectness by Anonymous Coward · · Score: 0

      BSOD dumps RAM to disk?
      Since when?
      How to I access the dumped memory?
      What file is it put in?

    4. Re:Image Incorrectness by sharkey · · Score: 1

      memory.dmp

      The default location is either the Windows/WINNT or System32 directory.

      --

      --
      "Outlook not so good." That magic 8-ball knows everything! I'll ask about Exchange Server next.
  65. Re:Not Unix, HP-UX by Anonymous Coward · · Score: 0

    If indeed it is HP-UX to blame for downtimes YET AGAIN, then a rant about outsourcing is way overdue for this non-Unix branded pile of bugs.

    HP-UX is the ONE *nix which is nearly as bad as MS-Windows when it comes to mission-critical bugs not getting fixed for years on end, and for the vendor's patches being ineffective and sometimes far worse than the underlying bugs. HP-UX may be better than Windows, but not by much.

    If you want an HP-UX outsourcing rant, simply reply to this and I'll share my personal experience with that overkludged bundle of patchware known as HP-UX which nearly caused the failure of a $1 Billion dollar company I once worked for.

    But here's the summary: HP's own JetDirect-equipped printers were internally DOCUMENTED to crash the print spooler in HP-UX. This bug went outstanding for years, despite multiple integer OS releases. Large distributor installation with ~150 print queues can't do shipping for 30 days. So-called "patch" received only after threat of lawsuit, and has the effect of crashing HP-UX even faster than print spooler can hang. Customer learns that HP provided the patch secretly knowing that it would most likely be ineffective. Desperate customer performs a trouble-free migration to Digital Unix on an AlphaServer 8400 within a single week, then enjoys a full year without any crashes.

  66. Sabre by Anonymous Coward · · Score: 1, Informative

    This is Sabre. EDS was contracted to run the Data Center. The shop has been mostly TPF/VM/MVS(ZOS) for some time, but they announced a couple of years back that they were going to shift most of the work onto servers (not sure what operating system. I do know that they were from around 20? mainframes(these are now the size of large refridgerators and IBM likes calling them servers) to about 4,000 servers. I am not sure how far they have gotten on this. As for this knocking out a couple of major carriers, I'm not surprised. Most of the domestic carriers are handled in 3 data processing centers, and lately those centers have gone down hill. I know at Galileo, they outsourced to IBM who are doing a very lackluster job of running it. So stand by for more of these type of outages.

  67. Solid Edge by Anonymous Coward · · Score: 0

    Is EDS the same company that makes the CAD software Solid Edge?

  68. uh oh, was that me? by bob_jenkins · · Score: 1

    Every time I see one of these articles, I read the fine print just to be sure it wasn't one of my bugs that brought the system down. Looks like I'm probably safe this time.

  69. Haha, brilliant. by Anonymous Coward · · Score: 0

    That is the best ASCII art I've ever seen. It's so simple, yet elegant, and it gets the point across. Bravo!

  70. Probably Sabre Holdings, rest probably wrong by Markus+Registrada · · Score: 5, Interesting
    First, they didn't "complete a migration". They're still deep in the middle of it, and will be for years to come.

    Second, this failure isn't in the Sabre reservations system, it's in some ancillary product, so who knows? Maybe they have no intention of switching it to Unix.

    Third, he didn't say so, but the migration isn't just to Unix. It's also migration to MySQL! (Hahahahahahahaha. Then again, coming from TPF, coded in assembly language for 4Kword pages, and a hierarchical database, that might seem pretty advanced.) Sabre had to fund a MySQL port to 64 bits, and a new "stored procedures" feature.

    1. Re:Probably Sabre Holdings, rest probably wrong by Qrlx · · Score: 2, Funny

      and a new "stored procedures" feature

      I guess the should have heeded the cabin crew's warning: Procedures stored in the overhead compartments may have shifted during flight.

    2. Re:Probably Sabre Holdings, rest probably wrong by bunyip · · Score: 1

      Sabre had to fund a MySQL port to 64 bits, and a new "stored procedures" feature

      Actually, that would be a "no" on both counts. The systems don't use stored procedures and MySQL was already 64 bit when we started, we just downloaded it and went. We now have commercial support, but not to get specific features. Check out some of our presentations fromthe MySQL user conference and other forums.

    3. Re:Probably Sabre Holdings, rest probably wrong by leandrod · · Score: 1
      > the migration isn't just to Unix. It's also migration to MySQL

      That pretty much explains it.

      > coming from TPF, coded in assembly language for 4Kword pages, and a hierarchical database, that might seem pretty advanced

      Yes, but TPF was rock-solid, while MySQL is a disaster.

      Yes, it is good enough if you don't value your data and availability. But this is not the way to run a business - nor any kinda organisation.

      Ultimately, the problem may not even have been MySQL itself, even if MySQL has been behind quite some site failures - Slashdot itself, Fastmail and so on -; but it probably have something to do that the kinda person that chooses MySQL over, say, PostgreSQL is also bound to make many other stupid tech decisions.

      --
      Leandro Guimarães Faria Corcete DUTRA
      DA, DBA, SysAdmin, Data Modeller
      GNU Project, Debian GNU/Lin
    4. Re:Probably Sabre Holdings, rest probably wrong by Euphonious+Coward · · Score: 1
      [We] don't use stored procedures and MySQL was already 64 bit when we started

      Evidently you don't know what the Sabre CIO is saying in Sabre press releases. He specifically reported commissioning special work on both counts. If you never heard about this, maybe it's because there's more than one project at Sabre (imagine that!) and the press releases were about one of those that you're not on. Be glad that some other group within Sabre commissioned and took delivery on the 64-bit work before you got to it.

    5. Re:Probably Sabre Holdings, rest probably wrong by bunyip · · Score: 1

      I've read the press releases, but you probably haven't - Sabre's CIO is a "she" not a "he", maybe you're thinking "CTO". Anyway, I'm the architect of the project using 64-bit MySQL and the press releases were about this project, I helped write them (imagine that!). If you want to know more about it, you can probably downlod the presentation from the MySQL user conference (I wrote that too). Please don't believe everything you read in the press, journalists don't get all the facts right.

  71. Re:Not Unix, HP-UX by nbvb · · Score: 1

    ... only to have DIGITAL purchased by Compaq, which was later melded with HP. ... to have Tru64 (nee DIGITAL Unix) cannibalized into a new version of HPUX.

    Lovely. :(

  72. Not HP Classic, Compaq/Tandem by JohnQPublic · · Score: 1

    Sorry, we can't blame HP-UX either. The Sabre system was converted to Tandem Non-Stop hardware running Unix. This is only and HP issue because Compaq bought Tandem before HP bought Compaq. Doncha love the computer business?

  73. What's going on here? by iantri · · Score: 2, Interesting
    Air Canada experienced numerous delays yesterday, too...

    Hmm... what's going on here?

    1. Re:What's going on here? by T-Ranger · · Score: 1

      Air Canada and US Airways are both members of the Star Alliance. But American is not. I think some level of scheduling coordination is the primary reason for Star to exist, but that doesnt explain why American was smashed as well.

    2. Re:What's going on here? by micron · · Score: 1

      .... but the Air Canada delays cost us 40% less!

      sorry, had to take that shot! (it is a currency joke for the uninitiated)

  74. Drain the Swamp by Anonymous Coward · · Score: 0



    It's the only way we'll ever get back to Oscar.

  75. I thought you meant... by Anonymous Coward · · Score: 0

    made an ISO, not combustion.

  76. Re: And in related news by Anonymous Coward · · Score: 0

    Which ones? These?

  77. Bug by Anonymous Coward · · Score: 0

    Everybody seems to be missing the most important question:

    The bug that brought down the database server, was it in a stored procedure?

  78. Mainframe issues... by mekkab · · Score: 1

    Mainframes will be the death of us (Due to a lack of talent)! The last major delay in the UK air traffic system was their NAS mainframe going down after a failed patch, and taking three hours (on a Friday morning!) to get back in service. (you can google for NATS being grilled by parliament)

    --
    In the future, I would want to not be isolated from my friends in the Space Station.
  79. (nit picking) not vague - just wrong by Anonymous Coward · · Score: 0

    The word "glitch" has a defined meaning (look it up). In news reports of computer failures it is almost always used wrongly.

  80. I like how spokesfolk try to .. by BigGerman · · Score: 1
    ..turn anything in a positive way.

    "Customers won't necessarily miss their connections," he said, "because everything was stopped."

  81. You're not gonna DIE by Old+Telco+Guy · · Score: 2, Informative

    This, ladies and gentlemen, is a flight plan. Now how the hell you gonna die because some FAA form can't get filled out right? All it was was a paperwork requirement. Planes still fly, pilots still know how to land them rubber side down.

  82. Re:Windows by Anonymous Coward · · Score: 0

    It could use sense-of-humor requirements too.

  83. Sunday Morning?? by Anonymous Coward · · Score: 0

    Could it be that they installed a new version of the application and it failed under load? Naw, that never happens.

  84. Re:And in related news by iroll · · Score: 1

    First, he ain't starving.
    Second, he'll get a cushy retirement package.
    Third, hello, this is public service, isn't a higher calling more important than $?
    Fourth, he'll get $millions to write a book.
    and Fifth, of course, I'm sure Halliburton will come up with a place for him when everything is said and done!

    (OK, that was a low blow--but seriously, he'll be fielding all kinds of high dollar job offers when he gets out; former cabinet members don't drive Hyundais)

    --
    Repetition does not transform a lie into the truth. - FDR
  85. Can't be windows... by Anonymous Coward · · Score: 0

    Since this is the first time the OS died, it
    can't be windows unless the install was done last
    Friday. The OS would have died plenty otherwise.

  86. Not "OS" by Master+of+Transhuman · · Score: 5, Informative

    When they said "operating system", they meant "operations system" - not the OS.

    See this quote from one of the articles:

    Wagner said a database malfunctioned that "basically runs every aspect of our client operations -- aircraft dispatch, crew scheduling (and) reporting weight, passenger load, balance."

    This system is hosted by EDS, who only said it was a "systems issue".

    So there's no evidence it was an OS problem. It could have been anything - OS, Oracle/DB2/SQL Server database, application code, upgrade, whatever.

    Nothing to conclude here except that somebody screwed up - and even that isn't certain - could have been a bad memory board someplace, who knows.

    Not having a backup is even irrelevant, since the "backup" might have taken three hours to bring up, when you're dealing with a production system like this. "Failover" is what you want, and they should have had, but if something got screwed there, it could still have been three hours.

    Shouldn't have happened, but crap like this happens all the time because nobody can do their damn jobs.

    --
    Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
  87. Same Thing Happened to NorthWest by Salis · · Score: 2, Interesting

    Their computer system went down for at least *3* hours in Minneapolis, shutting down the entire terminal. They couldn't check flight plans, ticket information, scheduling, logistics, etc. No planes in, no planes out.

    You'd think they'd have redundancy and backups, but they probably don't. That requires some planning beyond the immediate need of the company and, even if it's more profitable to invest in backups, long term planning simply isn't considered as much.

    This happens to my University all the time. The power goes out in one building for a few hours and services across the entire University are disrupted completely. This building happens to house most of the license servers for important software, but no one would _ever_ think of putting a backup license server in another building _just in case_. No, that'd be thinking ahead.

    --
    Favorite /. tagline: "On the eighth day, God created FORTRAN." And it was good.
    1. Re:Same Thing Happened to NorthWest by Anonymous Coward · · Score: 0

      If so many different airlines experienced a system failure at the same time, could this really be TIA or some other government and airline monkey business that needed system downtime to impliment and have up and running?

      We all know we can trust the airlines to maintain our privacy now can't we. This may sound a little paranoid, but it wouldn't surprise me one bit!

  88. More info about Sabre than you ever wanted... by airbatica · · Score: 4, Informative

    Sabre is a multitude of software products, for lack of a better definition. They include RES, DECS, TIM, BMAS and a couple of others that I can't remember.

    All Sabre applications are text mode, no GUI whatsoever... think CLI from hell, with no command history if you fat finger an entry.

    The system that went down was probably DECS (Dispatch Environment Control System), which is the system used by both American and USAir for generating flight plans, load planning, weight and balance, and various other flight operations functions.

    RES is the Reservations system, which covers the spectrum from building reservations and selling tickets, to customer checkin, boarding and god knows what else. IIRC, it will even do car rentals and hotels.

    TIM is also called Timatic. Its used for accessing information from the US State Department regarding internation travel to any country, from any country in the world. It covers entry and exit requirements, documentation, and pretty much anything you could want to know.

    I don't remember what BMAS stands for, but it is a lost bag tracking and reporting system. When AA or US looses your luggage, this is what they use to find it.

    Sabre is used by a whole variety of airlines and travel agencies, and is customised in modules to each particular user's needs.

    Now you are probably wondering how I know all this... I work for a major airline that uses a majority of the systems listed above, with the exception of the Dispatch system. We were not affected by whatever snafu took down that portion of Sabre :)

  89. Unfortunately, no by DesScorp · · Score: 2, Informative

    The airline's backend systems will continue to run on either old Tandem mainframes or port to new IBM mainframes (not running Linux, as of yet). Most of the airline's new IT investments are at the airport end.

    Unfortunately, the Windows-everywhere trend seems to be winning here. My airport is going to a common-use terminal system, and it's Win2K based. All but one of the big common use vendors are selling Windows-based equipment. Northwest's CUSS (common use self service) terminals are Win2K based as well.

    When I asked our vendor, who specializes in smaller airports, whether his company was doing any Linux development, he replied that nope, since most of the systems will never be on a public internet, it was easier and cheaper to get windows developers. No security concerns without the Internet, and 2K/XP/2K3 have become much more stable than older Windows platforms (his company still has older installations overseas that run NT 4 based systems, all due for an upgrade).

    --
    Life is hard, and the world is cruel
  90. Re:More info about Sabre than you ever wanted... by keithcstone · · Score: 1

    From what I saw from the reports is was Flight Operations (FOS). TPF based. Unlikely it was a TPF failure or hardware. One report mentioned database curruption, which means some programmer's going to be in deep dodo in the am. I'll know in the morning, one of my guys used to work on FOS.

  91. Two engine planes are twice as likely to fail by AnEmbodiedMind · · Score: 1
    Example 4 - You can't fly very far over open water in a single engine aircraft.

    I'm not sure how true this is for modern two engine planes, but I have heard that some of those little two engine prop planes are twice as dangerous as one engine planes because they need both engines for safe flight, and they are therefore twice as likely to have an engine failure and crash.

    I don't really care if it is true, it makes a good story anyway :P

    1. Re:Two engine planes are twice as likely to fail by AJWM · · Score: 1

      That's true for some light twins, especially if they're flying at or near full load. That class of aircraft is sometimes called "doctor killers" because doctors tend to be people with enough money to own one but not enough time to spend much of it practising emergency procedures. (A twin is much harder to control if one engine goes out, especially in the first few seconds after engine failure.)

      My father-in-law, who is a doctor, used to own just such (a Cessna 310). Mind, he's also an Air Force officer (retired).

      Any time I (I'm also a pilot) or any pilot I know has flown a single over open water it's after calculating the altitude needed to glide to shore if the engine quits at the halfway point. Well', except of course for float planes or other amphibs.

      --
      -- Alastair
  92. Re: And in related news by Anonymous Coward · · Score: 0

    What an awesome piece of crap you've linked to. That is the most believable thing I have ever seen. Get a fucking clue, you worthless pile of dog feces.

  93. bu-bu-but we want to blame it on Windows! by Anonymous Coward · · Score: 0

    It's a Slashdot requirement. Windows has to be at fault. We won't have it any other way! [/poke fun]

    First of all, shout outs go to the poster about 'no disclosure / PATRIOT ACT'.

    Despite that, /. posts weigh heavy on the "find a way to blame M$". I agree that a daily dose of M$ distain keeps you sane. However, read todays posts, and ask yourself, is this really about the news, or just another excuse for hopeless irc/net addicts to entertain themselves? It turns it, it may not be a Windows issue at all, or even a UNIX one, but an operating systems (yes, plural, a heterogeneous system overall) It's no wonder I only occasionally visit /. - Most of the postings do not add to the current events. Almost excessively, posts are made in self gratification. (Not the motive for my post - call me a hypocrite, and you'll believe anything you tell yourself)

    In closing, I'm trying to say, we haven't learned much for the amount of material that's on this page. It's ever learning but never knowing. What was the underlying issue? I'm sure there are hundreds of possible fault causes. I reviewed the story to find out the "why", but if no one knows, time will tell. (if we're patient?)

    What ever it is, I hope they remember what caused the issue. And by remembering, I mean sharing it again and again so it will not be repeated. (internally alone is ok to).

  94. Not just in the UK, sport ... by quarkscat · · Score: 1

    EDS has a major US government contract to
    upgrade many/most of the US Marine Corp.
    computers (desktops AND servers). The
    contract is in very big trouble, being
    plagued with huge cost overruns and failure
    to supply the equipment in a timely manner.

    EDS keeps popping up in the news here, and
    the news is rarely good. I think the company
    really went downhill after H. Ross Perot
    sold it off. Too bad, really ...

  95. Please don't call it the "PATRIOT" act by some+guy+I+know · · Score: 1
    The PATRIOT Act?
    It's not the "PATRIOT" act; it's the "USAPATRIOT" Act.
    Please use the full acronym, or its full name: "Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Trrorism".
    The "USAPATRIOT" Act has nothing to do with patriotism, so calling it the "PATRIOT Act" is misleading.
    (Considering how the Act is being misused these days, even using its full name is somewhat misleading (How is copyright infringement "terrorism"?).)
    Personally, I pronounce it "the you sap at riot act" to avoid confusion.
    Other pronunciations are "the US ap uh TRY ot act" and (as Jar-Jar) "the YOUsa pah TR-R-RE-E-E at act".
    --
    Those who sacrifice security to condemn liberty deserve to repeat history or something. - Benjamin Santayana
  96. Snapshots by Anonymous Coward · · Score: 0

    Putting Alan Cox on a Chip has produced a snapshot related to this story.