Slashdot Mirror
Comair System Crashes; Passengers Stranded
Broerman writes "30,000 people have had their flights cancelled by Comair this weekend thanks to a computer system shutdown. It appears that due to weather and other problems that flights began to be cancelled on Thursday and the backlog choked the system. 1,100 flights have been cancelled so far, including all flights through 12/26. Does anyone know what platform their system was based on? What kind of system just totally crashes? The official statement is that 'There was a cumulative effect with the canceled flights and trying to get crew assigned that caused the system to be overwhelmed.' It seems highly improbable that a system would crash because it had too many reservations. The system should only be able to hold as many reservations as it has flights/seats. It would seem that it's more likely that the system was overloaded with use and that caused a meltdown. When you add in the problems experienced by US Airways, this hasn't been a Merry Christmas for many."
You know I think it was. btw the system being used by Comair?
Its one of SCO's last large scale deployments. You know who to blame now.
Online Anime Gallery's
Yes, but it is mostly recoverable. The heavy iron handles things like backend reservations, checkin and cargo. Smaller systems handle things like weight/balance and fuel and PCs are typically used for the front-ends.
Weight/balance calcs can be done more or less by hand if necessary, however a larger fuel margin is needed. Checkin can be done by hand (you have seen those sticky label systems). However to lose reservations is a major problem.
See my journal, I write things there
'There was a cumulative effect with the canceled flights and trying to get crew assigned that caused the system to be overwhelmed.'
.
:-)
I am only trying to make sense out of the above comment from the official statement above.
Crew assigment is a hard problem, it is usually an MILP (Mixed Interger Linear Programming)
Such problems may be very hard to solve in reasonable time. Maybe (I'm shooting in the dark here) the first delays made the crew assigment problems grow too large for being solved in reasonable time.This would generate a snow ball effect as the assimgment problems would keep on growing maing the system "crash".
We may never know what really happened but this would be a nice example for my classes
Of course, a techie didn't write the PR release. Who in their right mind would let a techie anywhere near a PR release?
BTW, Comair, a Delta feeder headquartered outside Cincinnati, says the system that crashed was used to monitor crew locations and track working hours to ensure no one went over the legal maximum. Comair says the system crashed as a result of massive crew rescheduling following a record snow in their service area on Wednesday. There is no backup.
-- Slashdot: When Public Access TV Says "No"
From: http://www.fly.faa.gov/FAQ/faq.html
The term "Rule 240" refers to a rule that existed before airline deregulation. There is no longer an actual Rule 240. The term, as it is now used, refers to each airlines "conditions of carriage" policy. You would need to contact the airlines to obtain this.
My sister flew Delta on Dec 23rd from Detriot to Atlanta. Plane was 2 hours late, but no big thing. Waited 5 hours for her luggage, with no dice. By the time we got in line for luggage services, there were at least 600 people in the line already.
Talking to other passengers from 10+ different flights from different cities, no one got their luggage that night. Apparently, it wasn't just Atlanta - the local news in Tampa and Detroit had segments on how the airports had taken over parts of taxiways to sort through seas of bags that didn't make it on to planes.
It's been 2 days, and Delta has no idea where the stuff from that flight is. I'm guessing it isn't just Comair that got hit by some computer problems.
Jerry
http://www.syslog.org/
This article outlines how this joint venture re-vamped Delta's IT systems (again remember, this is 1995):
The trail runs dry here, job postings stopped around 2001.
Which really raises suspicions that all the code is written and maintained offshore. The question now becomes who is handling this for Delta.
One of Tata's spinoffs, Airline Financial Support Services, is described as
Wipro handles some of Delta's inbound reservation calls in India and the Phillipines.
In conclusion, it would appear that either Tata's AFS arm or Wipro do the IT for Delta airlines.
Judging by www.comair.com and their job ops, it's probably HP-UX or Windows. More than likely the Unix flavor rather than Windows. Why down for a couple of days, probably a database restore. Never happens in TPF. Those mainframe systems crash and are back up with very little database degredation. By the way, in the job ops, if you want to be a crew scheduler, only need HS diploma!
I've done contract programming for Comair. They use HP/UX for the servers that handle most things except HR which is mostly Windows. The system that went down was a COTS app that handles crew scheduling. It is a bid system that allows crew to bid for flights based on seniority w/ constraints to match FAA rules.
Their IT director is really sharp, but he faces some real problems. First, IIRC, they only created a dedicated IT shop about three years ago. Second, their budget is small compared to the task they have to perform. Comair is an airline and airlines have been in real trouble since 9/11.
More than likely it is TPF as Delta is a TPF shop.
m l) has been around since the '60's and is used by all the major airlines, most of the large hotels and most bizarrely NYC 911.
TPF (http://www-306.ibm.com/software/htp/tpf/index.ht
Someone from Comair (who shall remain anonymous) provided me with some details whch people here would be interested in:
CVS is teh suck. Use Vesta instead.
Just to be absolutely clear: I've only ever communicated with this person on-line, and I can't verify who they are in real life or that they actually work for Comair. It seemed credible though, and it seemed worth posting to de-bunk the slashdot knee-jerk reaction of blaming Microsoft. To me, an application using a 16-bit integer for something seems like a very likely explanation.
CVS is teh suck. Use Vesta instead.
What happened to Comair here could happen to just about any airline. There is no comprehensive suite of software that handles crew scheduling, aircraft scheduling, reservations, and the myriad of other functions that are needed to run an airline.
Reservations, for other than tiny airlines, are still managed by large TPF mainframes. TPF is a very "bare bones" operating system that runs on IBM mainframes, and was written specifically to deal with high volume / high transaction rate systems. Personally, I've seen 5 attempts at 3 different airlines to replace it with something modern. ( like Unix with an RDBMS ). Each attempt failed miserably, and the airline went back to TPF. Note that TPF is not MVS, OS/390, or any other more mainstream Mainframe OS. It's purpose built.
Unfortunately, this means that all of the other applications have to interface with TPF via screen scraping. To further compound the problem, no "suites" exist to handle the following functions, so most airlines have to "sew together" best of breed solutions for these basic functions:
- Crew Scheduling - F/A's and pilots bid on
slots to fly, this system takes those bids and
turns it into a schedule.
- Aircraft Scheduling - Tracks which tail numbers are flying which flights for the dispatchers
- Optimization - Different optimizers to do
things like:
- Fuel Tankering - Use the jets as "tankers" so that you buy fuel where it's cheapest for flights later in the day
- Crew Optimization - "Traveling Salesman" type solver to incur lowest labor cost, get crews
back to home base, etc
- Schedule Optimization - Use the aircraft
in the most cost efficient way to cover all of
your scheduled flights.
- Maintenence Optimization - Pull aircraft in
for Scheduled Maintenance at the optimum time.
- Reacommodation - When things go wrong ( weather, mechanicals, whatever, pull in all of the above variables to crank out a new schedule,
crewing, mx schedule, etc )
- Booking Engines, for the internet and
reservations agents
- Point of Sale and Boarding functions for
agents, skycaps, and kiosks
- Interline functions where other airlines
sell your tickets, and transfers for bagggage, etc
Anyhow, this list isn't comprehensive, but shows enough of the disparate pieces that you can imagine why these "glitches" happen. Very few of the items from the list above come from the same vendor, or even run on the same platforms.Your statements are accurate.
I was a unix sys admin there, but left for greener pastures during the dot-com craze. The non-redundant hardware at the time ran AIX, and had a great support contract from IBM. The SBS application however, always had monthly issues, at least at that airline. They were looking for a replacement then, and I'm not suprised they still haven't replaced it.
Take Amtrak!
Amtrak receives around $500 million for a total budget, while the airtravel receives around $15 billion in subsidies. Take the train and save everyone money!
_____________
Huh?
Maestro is delivered on AIX, uses a rather old version of Informix for it's database, and is tied together using the TUXEDO TP monitor from BEA.
The business logic is written in C, and abstracted away using Tuxedo.
In the case of a major schedule disruption, this program isn't responsible for "solving" the problem, but is responsible as being the system of record for holding the new crew schedule.
My guess is that the changes to the crew schedule were large enough that some piece of the system was overwhelmed. ( For example, a transaction that was too large and overran the rollback buffers in Informix ).
Without the system of record in place, a manual process would be very difficult. You would have to figure out:
- Which crews where in which locations
- What aircraft each crew member was qualified
on.
- How long they had flown already that day. ( Legalities about how much time you can fly before
you need mandatory rest )
- Which routes to send those crews on
- How to get the crews back to a specific city
to run the next day's schedule
Of course, any mistakes you made doing this manually would overflow into other systems. For example, you might send an aircraft that's due maintenance to a city with no maintenance facilities.Also, for those that were critical of the system not being highly availble...this doesn't sound like the kind of problem that HACMP and replicated databases would have helped. The hot standby would have choked at the exact same point.
just to give you an idea, here is the applicable FAA reg for crew scheduling, and the pilots contract may have additional terms that must be met.
121.471 Flight time limitations and rest requirements: All flight crewmembers.
top
(a) No certificate holder conducting domestic operations may schedule any flight crewmember and no flight crewmember may accept an assignment for flight time in scheduled air transportation or in other commercial flying if that crewmember's total flight time in all commercial flying will exceed--
(1) 1,000 hours in any calendar year;
(2) 100 hours in any calendar month;
(3) 30 hours in any 7 consecutive days;
(4) 8 hours between required rest periods.
(b) Except as provided in paragraph (c) of this section, no certificate holder conducting domestic operations may schedule a flight crewmember and no flight crewmember may accept an assignment for flight time during the 24 consecutive hours preceding the scheduled completion of any flight segment without a scheduled rest period during that 24 hours of at least the following:
(1) 9 consecutive hours of rest for less than 8 hours of scheduled flight time.
(2) 10 consecutive hours of rest for 8 or more but less than 9 hours of scheduled flight time.
(3) 11 consecutive hours of rest for 9 or more hours of scheduled flight time.
(c) A certificate holder may schedule a flight crewmember for less than the rest required in paragraph (b) of this section or may reduce a scheduled rest under the following conditions:
(1) A rest required under paragraph (b)(1) of this section may be scheduled for or reduced to a minimum of 8 hours if the flight crewmember is given a rest period of at least 10 hours that must begin no later than 24 hours after the commencement of the reduced rest period.
(2) A rest required under paragraph (b)(2) of this section may be scheduled for or reduced to a minimum of 8 hours if the flight crewmember is given a rest period of at least 11 hours that must begin no later than 24 hours after the commencement of the reduced rest period.
(3) A rest required under paragraph (b)(3) of this section may be scheduled for or reduced to a minimum of 9 hours if the flight crewmember is given a rest period of at least 12 hours that must begin no later than 24 hours after the commencement of the reduced rest period.
(4) No certificate holder may assign, nor may any flight crewmember perform any flight time with the certificate holder unless the flight crewmember has had at least the minimum rest required under this paragraph.
(d) Each certificate holder conducting domestic operations shall relieve each flight crewmember engaged in scheduled air transportation from all further duty for at least 24 consecutive hours during any 7 consecutive days.
(e) No certificate holder conducting domestic operations may assign any flight crewmember and no flight crewmember may accept assignment to any duty with the air carrier during any required rest period.
(f) Time spent in transportation, not local in character, that a certificate holder requires of a flight crewmember and provides to transport the crewmember to an airport at which he is to serve on a flight as a crewmember, or from an airport at which he was relieved from duty to return to his home station, is not considered part of a rest period.
(g) A flight crewmember is not considered to be scheduled for flight time in excess of flight time limitations if the flights to which he is assigned are scheduled and normally terminate within the limitations, but due to circumstances beyond the control of the certificate holder (such as adverse weather conditions), are not at the time of departure expected to reach their destination within the scheduled time.
As such it used some very interesting data representations. For example, it tracked time using julian minutes. There are 44640 minutes in a 31 day month. That's small enough to fit in a 16-bit unsigned variable. This approach, nearly taboo by modern standards, was a God-send during Y2K. The system never needed to know what year it was. It became the running wisecrack, "You can't have a Y2K problem if you don't have a 'Y'".
The Aircraft to Flight assignments is another system, but the two share information.
Well... to try and provide a little clarification here, as I work for Comair. Here's the skinny:
:)
Crew and aircraft scheduling is done through a software package called SBS Track. This very same software package is used by many other airlines, including the two I worked for before coming to Comair. I don't know if their systems have the same hard-coded limit that ours does or not. This software package has _nothing_ to do with reservations, or anything concerning passengers whatsoever. It is simply the software we use to schedule our aircraft and crews to fly the list of flights that Delta wants us to fly.
Crew scheduling is done by creating "pairings". A pairing is a sequence of flights that comprise a crewmember's trip. Anytime a change is made, a new pairing is generated, with the new sequence of flights. The system has a hard-coded limit of 32k pairings ("transactions" is the what the IT folks call it) in a calendar month. As of 10:00 pm on 12/24, that limit was reached. Crew Scheduling was unable to create any new pairings, unable to track who would be flying what airplane to where, and basically unable to keep the airline flying at that point.
It was not any kind of a hardware failure, there are backups for that. It is simply a software limitation, that when it was coded many years ago, nobody realistically thought it would ever be reached. Why they hardcoded a limit into it in the first place is beyond my knowledge.
A major part of the problem is Comair's concentration in Cincinnati. CVG is our only crew base, and it is the largest single crew base of any airline in the world. Over 1800 pilots and 1100 flight attendants in one base. Not even any of the majors have a single base that large. Several of our software packages are woefully inadequate, and replacements have been sought for some time.
As for getting things up and running on paper, this is a monumental task. Scheduling for 160+ aircraft and 2900+ crewmembers, and compliance with all FAA regulations, maintenance requirements, crew rest requirements, and contractual requirements is incredibly complex. In addition, we have crews and aircraft stranded across the country due to the weather that moved through that caused this whole mess in the first place. Add to that the very limited number of people who actually have the knowledge of all the requirements for scheduling, and coming up with a full schedule for the next day would be nearly impossible.
Jan. 1 starts a new month, and the system will return to full functionality then. Until that date, however, our operations will be very limited.
Some of you have no clue.
The BS&T quotient on your average travel application is on the relatively nuts scale. Expedia, Travelocity, hotwire, priceline, whatever -- I'd ask that some of you with simple solutions go and speak to the lead travel-server dev for the product.
You'll probably have to change pants after the conversation. Travel is stable, reliable, and generally rock-solid. The algo's for selecting airline flight prices or hotel room block-reservations are known and well-tested. The methods and protocols of communication are well-documented and generally straightforward.
Until recently, it was all on hardware (And i'm speaking generally about the large travel providers -- Worldspan and Sabre come to mind) that was considered arcane. Ancient versions of Netware on an X.25 pad; screen-scrapers on top of it. Have Fun trying to modernize!
This does not suprise me in the slightest. We are stressing our ancient systems more than ever these days, and it should not be a suprise when the occasional ancient application (ctime, folks) gets floor'd and dies a bloody death.
It'll be patched in a month.