Slashdot Mirror
Database Glitch Grounds American/US Airways
An anonymous reader writes "According to numerous news sources, all American Airlines and US Airways flights were grounded for two or three hours this morning. Both problems were caused by a computer glitch in the systems hosted by EDS. Quote: The operating system that drives the airline's flight plans went down."
How in the world can they state that as singular. Surely they have a backup of some sort. Especially with all the supposed "increased security" around air flight, you are telling me that one system crash can knock out half of the major airlines? That's ridiculous. Have they not learned about redundancy?
EDS is by no means a Windows shop. They work extensively with "big iron" mainframes. In fact, they recently got the contract to handle the database of terrorist information that'll be used at airports. Likely this will be hosted on a 390 or something... Windows can't handle that kind of I/O.
Airport BSOD
I'm guessing the last thing you want to hear on a plane now is the pilot saying, "What do you mean, fatal exception error?"
>_ Why don't they swtich to Linux?
Blue screen of life. Because US Air cancelled the flight and we were forced to fly on a competent airline.
Sorry, have to rant where I see EDS mentioned.
EDS, in cahoots with the UK govenment, have wasted millions of pounds of taxpayers money on failed IT projects. Notable ones include the Inland Revenue (UK IRS), Child Support Agency (£50M over budget and still not working) and an email and directory service for the NHS (withdrew at last minute allowing C&W to steal at a much inflated price).
Though the blame cannot completely be laid at the door of EDS, the government has been guilty of sloppy auditing and the worst being the willingness to hand over extra money when EDS has come around with the begging bowl.
For all intensive porpoises your a bunch of rediculous loosers
NEVER open Windows in an airplane!
Typical Airline applications are Reservations, Check-In, Weight-and-Balance, Flight Planning (which route to take and how much fuel to carry) and Ticketing. Once you have left the terminal and are heading for the runway, software crashes cease to be relevant.
Once you head for the runway, you care about Air Traffic Control's software. The only exception I can think of is for flights to the US where the authorities want passenger lists.
I work for an airline and we host for other airlines. I feel sorry for whoever carries the can for this mess. As to the OS, those who said it will be MVS are almost certainly correct. AA and US Airways are/were IBM customers.
Mielipiteet omiani - Opinions personal, facts suspect.
What scary news? The airplanes are piloted by people, not computers. And certainly not the computers that control flight plans. Do you think that airplanes will start falling from the skies because a computer went down somewhere? I guess you packed your basement with cans of beans for Y2K too.
Support the First Amendment. Read at -1
The following entities were NOT mentioned in the article you're linking to:
(1) American Airlines,
(2) US Airways,
(3) EDS.
So, what the hell are you talking about?
Why did you link to this article?
(I know, I know, because nobody will read it anyway)
Sounds like a troll. The article quoted by the parent is about a small regional airline (Atlantic Coast Airlines) that's doing its IT work internally. The article doesn't mention EDS at all. Moreover, browsing EDS's site, you can see that the solution they implemented for Continental Airlines is UNIX-based.
This is undoubtedly a problem with Sabre, which EDS runs on behalf of Sabre Holdings. Both American Airlines and US Airways use Sabre for much of their operations.
Sabre started it's life as an American Airlines internal system (SABER, slight spelling difference), running on a rare operating system (PARS, later called ACP and currently TPF) on IBM mainframes. In the last few years Sabre completed a lengthy migration to HP Unix on Non-Stop (i.e. ex-Tandem) hardware. The mainframe systems were rock solid, but software talent was hard to come by, so they decided the time had come to switch.
Sorry, no Microsoft to blame here!
At about 4:30 a.m., the outsourced SysAdmin was setting up to do routine patches to Windows 2003 server nodes. But just before, he decided to check his e-mail with Outlook and he opened an important message from his system administrator advising him that his e-mail would be de-activated if he didn't open the important attachment. I think we all know what happened after that...
I might know what I'm talkin' about, but then again, this is Slashdot...
Here around we studied it, for one major airline in EU. We wanted a "backup system" in case the main system went down. Total Cost, without maintenance, about *3 whole day* of traffic "benefits"... Yes, that much. Right now the project is still discussed but most of us thinks it is dead in the egg. Instead the "older" and "less powerfull" developpement system will be used in case of break down.
Redundancy is OK, as long as it is not bleeding you dry.
C. Sagan : A demon haunted world:
http://www.amazon.com/gp/product/0345409469/
visit randi.org
The systems that run the aircraft and the navigational and communication systems really are redundant. It's the law. It also means that usually there are two different ways to do something not just the same thing repeated twice.
... We used to joke that the controllers would climb to the top of the tower and wave fire extinguishers to warn the planes away. (I think it was a joke.)
Example 1 - The pilot and co-pilot can't eat the same meal. That way, only one of them can get food poisoning.
Example 2 - The hydraulic system fails and the wheels won't go down. There's a hand crank.
Example 3 - The communication systems at every tower I have worked at have two separate backbones. There are two of absolutely everything. If that fails, there are emergency radios under the desk. If the emergency radios don't work
Example 4 - You can't fly very far over open water in a single engine aircraft.
It used to be frustrating working on systems older than I was but we never had to worry about surprises.
Of course all of this redundancy is very expensive. You spend the money where people's lives are at stake. On the other hand, if the worst problem is that some planes will be late, perhaps you don't spend the big bucks.
Of the 360-based operating systems, IBM's TPF has a major presence in the airline industry, but this probably isn't the system in question. TPF tends to handle ticketing and reservations. TPF stands for the Transaction Processing Facility; it's the descendant of the old Airline Control Program (ACP) developed for Sabre. Sabre in fact is still running TPF, although I believe they're busy transitioning away from the mainframe to Tandem's er I mean Compaq's er I mean HP's NonStop/UX.
Of course, it might not be an IBM mainframe at all; Unisys has a niche in the airline industry. But heck; given that this is route planning, just about anything from AIX to z/OS is a possibility. Even *shudder* Windows.
There is a line of code that raised the problem but is commented in Punjabi, I think it says "fuck this $3/hour job".
Second, this failure isn't in the Sabre reservations system, it's in some ancillary product, so who knows? Maybe they have no intention of switching it to Unix.
Third, he didn't say so, but the migration isn't just to Unix. It's also migration to MySQL! (Hahahahahahahaha. Then again, coming from TPF, coded in assembly language for 4Kword pages, and a hierarchical database, that might seem pretty advanced.) Sabre had to fund a MySQL port to 64 bits, and a new "stored procedures" feature.
Blaming MS is the easy way out.
I just read all the stories that were linked to this article.
None of them blamed Microsoft. In fact the only blame pushed in their direction was your comment...
The articles did say that there was a problem with the operating system. Now we don't know who exactly said this, or what they said precisely, so it is quite possible that this isn't entirely accurate reporting.
I find it very difficult to believe that they would have any single points of failure in a system of that importance.
I agree it's unlikely, but it is possible that there is a single point of failure in their system. There are a great deal of shoddily engineered systems in use today.
When they said "operating system", they meant "operations system" - not the OS.
See this quote from one of the articles:
Wagner said a database malfunctioned that "basically runs every aspect of our client operations -- aircraft dispatch, crew scheduling (and) reporting weight, passenger load, balance."
This system is hosted by EDS, who only said it was a "systems issue".
So there's no evidence it was an OS problem. It could have been anything - OS, Oracle/DB2/SQL Server database, application code, upgrade, whatever.
Nothing to conclude here except that somebody screwed up - and even that isn't certain - could have been a bad memory board someplace, who knows.
Not having a backup is even irrelevant, since the "backup" might have taken three hours to bring up, when you're dealing with a production system like this. "Failover" is what you want, and they should have had, but if something got screwed there, it could still have been three hours.
Shouldn't have happened, but crap like this happens all the time because nobody can do their damn jobs.
Richard Steven Hack - This sig is TOO GODDAMN SHORT TO DO ANYTHING USEFUL WITH! MORONS!
Sabre is a multitude of software products, for lack of a better definition. They include RES, DECS, TIM, BMAS and a couple of others that I can't remember.
:)
All Sabre applications are text mode, no GUI whatsoever... think CLI from hell, with no command history if you fat finger an entry.
The system that went down was probably DECS (Dispatch Environment Control System), which is the system used by both American and USAir for generating flight plans, load planning, weight and balance, and various other flight operations functions.
RES is the Reservations system, which covers the spectrum from building reservations and selling tickets, to customer checkin, boarding and god knows what else. IIRC, it will even do car rentals and hotels.
TIM is also called Timatic. Its used for accessing information from the US State Department regarding internation travel to any country, from any country in the world. It covers entry and exit requirements, documentation, and pretty much anything you could want to know.
I don't remember what BMAS stands for, but it is a lost bag tracking and reporting system. When AA or US looses your luggage, this is what they use to find it.
Sabre is used by a whole variety of airlines and travel agencies, and is customised in modules to each particular user's needs.
Now you are probably wondering how I know all this... I work for a major airline that uses a majority of the systems listed above, with the exception of the Dispatch system. We were not affected by whatever snafu took down that portion of Sabre