Comair System Crashes; Passengers Stranded

Broerman writes "30,000 people have had their flights cancelled by Comair this weekend thanks to a computer system shutdown. It appears that due to weather and other problems that flights began to be cancelled on Thursday and the backlog choked the system. 1,100 flights have been cancelled so far, including all flights through 12/26. Does anyone know what platform their system was based on? What kind of system just totally crashes? The official statement is that 'There was a cumulative effect with the canceled flights and trying to get crew assigned that caused the system to be overwhelmed.' It seems highly improbable that a system would crash because it had too many reservations. The system should only be able to hold as many reservations as it has flights/seats. It would seem that it's more likely that the system was overloaded with use and that caused a meltdown. When you add in the problems experienced by US Airways, this hasn't been a Merry Christmas for many."

Comments

Official my arse...

[Omicron32]

Sounds like my Mother wrote the official statement. A techy would never report something in that way.

Besides, it's pretty obvious their OS wasn't digitally signed. :p

Someone's gotta say it...

[mOoZik]

Yep, it was Windows XP. ;)

I don't know. Frankly, it has less to do with the platform than the custom software that runs on it.

Re:Someone's gotta say it...

[Pharmboy]

You would think so. The IT Director is respsonsible for making sure everything IT works. Not to do it himself, but to make sure it is done and done right. I can't see how someone can argue with that. Even if it IS the janitor unplugging the UPS to plug in a floor buffer.

Whether it is the cooling system for the computers, the operating system, the applications or simple hardware issues, it HAS to be the IT Director's responsibility. I mean, who the hell else?

Re:Someone's gotta say it...

[Antique Geekmeister]

Occasionally, however, the head IT guy gets over-ridden by management or by available finances. I've been there, saying "we need to spend money on this" and having to make do with much less money, or even with a cut in funding. You need to document the problem in advance to cover your ass, and get it in print and saved offsite to protect yourself from that kind of mistake. I've done that, too. It helped protect me from a nasty lawsuit because I demonstrated where I had told a consulting client, in print, when the systems would start failing and the resulting legal liabilities, and gotten it signed by the company notary.

Re:Fire away!

[mirko]

There recently was a big card problem here, in Europe.
It did not come from a peculiar OS but just because a partition got filled by index tablespace extents.
So, it could just be that they ran out of place and it froze the whole application.

stating the obvious

[Anonymous Coward]

"Does anyone know what platform their system was based on? What kind of system just totally crashes?"

A stab in the dark here but I'm assuming a system without foresight and redundancy?

blaming the system can backfire

[ext42fs]

It's not the OS, it's the people behind who's to blame. Yes, stupidity and MSW often go together but in a few years one will probably occasionally see a massive linux outage due to... similarly stupid people.

Re:Scalability and Twelve Step TrustABLE IT

[hughk]

No, its more difficult in the airline industry. The system by default tries to keep as many planes in the air earning money as possible. If you have an outage which disrupts this choreography, there is a tremendous knock-on effect as passengers/urgent cargo must be rebooked.

I have seen the major hub for an airline closed because of snow for just a couple of hours in the early morning, but the resulting chaos of rescheduling/rebooking caused the reservations system to crash after just a few minutes of uptime. The same would keep happening after restarts.

It is normal to test system up to several times normal load, but they were seeing peaks at over 100x. The old, 3270 emulator based system would have slowly got through it but the newer system died.

Re:I'm surprised

[HangingChad]

It seems that after Sep. 11th, the news wants to try to connect everything even remotely bad with terrorism

What else do they have to do? They've got this huge ass budget, all those people watching a lot of honest citizens. It was 10 years between the first attempt on the world trade center and the second. We've built and paid for this entire monster agency for an event that might be 10 or 15 years away. What are they going to do in the meantime? Grope women at the airport. They have to do something to justify their existence, Otherwise we'd have admit we over-reacted to 9-11.

Re:Southwest refuses to drink the Kool-aid

[PPGMD]

It's isn't that easy, for the longest time Southwest was the hardest to book a flight for because they had no web system that could figure out it's route system (only 5 years later they just released one). Up about about July of this year to book a web flight you needed a route map and schedule to figure out what cities you had to go throuh if there was no direct flight option.

The hub-spoke system is easier to manage, and can be profitable if the airlines relize that they aren't unlimited resources, and decentralize the hubs on a limited basis.

Anyways Southwest doesn't drink anyone's koolaid, they run all their own in house designed systems (I am not sure they are even on Sabre anymore), including web apps. It's an intresting concept, but it probably causes their IT managers to pull their hair out.

Re:Simple Solution

[the pickle]

Sure, that's eminently practical. I can take 48 hours to get from Detroit to LA, or I can take six (including travel time and check-in time at both airports).

p