Slashdot Mirror
Comair System Crashes; Passengers Stranded
Broerman writes "30,000 people have had their flights cancelled by Comair this weekend thanks to a computer system shutdown. It appears that due to weather and other problems that flights began to be cancelled on Thursday and the backlog choked the system. 1,100 flights have been cancelled so far, including all flights through 12/26. Does anyone know what platform their system was based on? What kind of system just totally crashes? The official statement is that 'There was a cumulative effect with the canceled flights and trying to get crew assigned that caused the system to be overwhelmed.' It seems highly improbable that a system would crash because it had too many reservations. The system should only be able to hold as many reservations as it has flights/seats. It would seem that it's more likely that the system was overloaded with use and that caused a meltdown. When you add in the problems experienced by US Airways, this hasn't been a Merry Christmas for many."
Anybody know what they were running? I'd like to see this flamewar get started as soon as possible.
When I lived in Chicago, they would lose their radar system on what seemed like a strong wind. And I got stuck in Denver overnight once because the computer system they use to calculate the weight of departing flights crashed. I have a feeling these kinds of crashes are much more common than most people think.
Sounds like my Mother wrote the official statement. A techy would never report something in that way.
:p
Besides, it's pretty obvious their OS wasn't digitally signed.
Yep, it was Windows XP. ;)
I don't know. Frankly, it has less to do with the platform than the custom software that runs on it.
A blog like any other.
They're a bunch of incompetent boobs. The news keeps reporting on a "computer glitch" or a "computer malfunction". That's bullshit. This happened because some human(s) fucked up.
Linking to their home page will surely help the situation..
The janitor pulled out the plug for the mainframe and used it to drive is floor polisher..
Simon.
"Does anyone know what platform their system was based on? What kind of system just totally crashes?"
A stab in the dark here but I'm assuming a system without foresight and redundancy?
That doesn't need answering.
# cat
Damn, my RAM is full of llamas.
I blame the software. Sounds like a more likely culprit than the OS, even if it is Windows.
They obviously didn't take mcbride's "license or we will have you shut down" threats seriously enough.
It's not the OS, it's the people behind who's to blame. Yes, stupidity and MSW often go together but in a few years one will probably occasionally see a massive linux outage due to... similarly stupid people.
Sounds like Comair could have used a little virtualized scalability and third party audited builds.
See Twelve Step TrustABLE IT : VLSBs in VDNZs From TBAs.
and also The ActiveGrid(TM) Grid Application Server and Grid Computing in general.
Back on May 1st of this year Delta's internal traffic monitoring system grounded them worldwide when it was hit by a worm (forget which one). Yours truly was flying that day. I spent 7 hours on a runway in Cleveland. (Talk about adding insult to injury.) Comair is a regional carrier of Detla's. I wonder who handles Delta's IT needs?
FAA's Rule 240 says that if your flight gets canceled for any reason other than weather, the airline has to get you on the next available flight to your destination, regardless of carrier. So if you're stuck in an airport bar reading this article go talk to your airline!
No, it means they didn't make a big enough swap partition.
# cat
Damn, my RAM is full of llamas.
It is not easy to do real world extreme situation testing on large systems, but I wish people would at least try.
It is fun to say Windows, blah, blah but given the number of buffer overflow problems found in programs/packages on all platforms, I would say that many programmers of every stripe severely underestimate the real world range/type/size of data their programs will encounter when in non-typical situations.
To whoever wrote/maintains/admins this software:Global "climate change" means weather "events" will be more frequent and more exteme in coming years, another terrorist event on US soil may cause days of air travel disruption. Please "refactor" your shit with those things in mind. You're on the East Coast and Midwest for god's sake you're going to get storms that will shut down regions for days at a time. What happens when the FAA finds some issue with an aircraft part or maint. procedure and grounds your whole damn fleet to have it fixed.
Now, I could be wrong, but I heard from a lot of people that it is the year 2005 bug... How did we not see this come?
This and this might offer a clue.
Blearf. Blearf, I say.
If you choose a standardized virtualized platform then you need not be limited to using in house clusters. Check out ActiveGrid(TM) info page, it includes support for third party distributed hosting provider such as Akamai, . Other providers in the future, will provide massively scaleable systems such as Cray's Red Storm Cluster. All running Linux.
Was it the design of the software or the limitations of the hardware? See my post on Scalability on demand and third party servers.
Too bad the airline will go bust because of this. But then all airlines lose are loosing billions except for Southwest.
'There was a cumulative effect with the canceled flights and trying to get crew assigned that caused the system to be overwhelmed.'
.
:-)
I am only trying to make sense out of the above comment from the official statement above.
Crew assigment is a hard problem, it is usually an MILP (Mixed Interger Linear Programming)
Such problems may be very hard to solve in reasonable time. Maybe (I'm shooting in the dark here) the first delays made the crew assigment problems grow too large for being solved in reasonable time.This would generate a snow ball effect as the assimgment problems would keep on growing maing the system "crash".
We may never know what really happened but this would be a nice example for my classes
One more advantage of a Virtualised Standard platform, would be the ability to do development and stress testing on third party servers. Full on stress testing is something that most organizations cannot afford to do on the currently deployed hardware.
...slashdotted reservations?
Find a job you like and you will never work a day in your life.
Yeah, but the real cause for the crash was an Access backend. So there! :)
Find a job you like and you will never work a day in your life.
30,000 passengers? Getting dangerously close to an integer overflow there.
Of course, a techie didn't write the PR release. Who in their right mind would let a techie anywhere near a PR release?
BTW, Comair, a Delta feeder headquartered outside Cincinnati, says the system that crashed was used to monitor crew locations and track working hours to ensure no one went over the legal maximum. Comair says the system crashed as a result of massive crew rescheduling following a record snow in their service area on Wednesday. There is no backup.
-- Slashdot: When Public Access TV Says "No"
With the amount of Y2K 'fixes' I have seen around, (and some of them very dubious), I wouldn't be at all suprised if it was a Y2K problem. Looks like someone didn't have all their test cases written down properly and/or didn't test properly and/or tried to 'fix' the problem.
-- main(s){printf(s="main(s){printf(s=%c%s%c,34,s,34
As a preliminary finding that may or may not give us a clue as to what the internet system was running, Netcraft reports that www.comair.com is running Apache on HP-UX.
So don't assume that the internal system was Windows just yet. Then again, don't assume that it wasn't.
Hopefully I didn't put any [] around my words.
My sister flew Delta on Dec 23rd from Detriot to Atlanta. Plane was 2 hours late, but no big thing. Waited 5 hours for her luggage, with no dice. By the time we got in line for luggage services, there were at least 600 people in the line already.
Talking to other passengers from 10+ different flights from different cities, no one got their luggage that night. Apparently, it wasn't just Atlanta - the local news in Tampa and Detroit had segments on how the airports had taken over parts of taxiways to sort through seas of bags that didn't make it on to planes.
It's been 2 days, and Delta has no idea where the stuff from that flight is. I'm guessing it isn't just Comair that got hit by some computer problems.
Jerry
http://www.syslog.org/
Don't smirk; it'll be the next topic on Ask Slashdot.
It would be cool if it didn't suck.
Some of my co-workers are on contract developing Java software for Comair.
Comair are very tied to particular systems, and don't want to change even when the developers have pointed out problems. Case in point: a J2EE-based employee portal, based on Novell exteNd (Novell Portal Service) and a one-way HPUX server. NPS runs in Tomcat, which is servicing requests (via mod_jk) through Apache. No other application shares the machine, and Comair will only consider vertical scaling, not horizontal.
The application creates at least two threads per connection, and when the thread count goes beyond a relatively low threshold (between 300 and 400), Tomcat deadlocks. It's not because they're running out of space in the allocated JVM heap, and they've tuned mod_jk to allow for heavy load. The current solution is to restart Tomcat when the system locks up.
Novell's support has been less than stellar, so the Java contracting group was informally asked what to do. We had all kinds of useful suggestions, from dumping NPS for another portal implementation, to creating custom thread-pools, to using JDK 1.4 new I/O and a minimally-threaded design, and even using round-robin DNS and a group of independent portal servers to share the load. Comair are wedded to particular minimal cost solutions, however, and it shows.
At least when the portal crashes, it only impacts employees and not passengers.
Somewhere deep in the code is a comment that says:
// I don't need to check for this condition because
// my asshole manager Steve Johnson says it'll
// never happen
{friggin' slash - When I say plain old text, I mean plain old text!}
that in the name of sensationalism reporters haven't said, "terrorism is probably not to blame but the Dept. of Homeland Security is looking into it." It seems that after Sep. 11th, the news wants to try to connect everything even remotely bad with terrorism, and of course the Dept. of Homeland Security encourages them by using as vague of language as possible. Are people that easily frightened?
Monstar L
http://home.hccnet.nl/jaap.kranenburg/fun/xx/image s/fun20020415.jpg
Computers don't freak out or get depressed when work piles up. Backlogs mean nothing; they just keep processing one piece at a time until the pieces run out. I think someone was speaking imprecisely.
I suppose that the system *could* have been built with a rule to detect that the results are becoming more and more untimely, and at some point just say "TILT!" and deliberately exit. I can't imagine why, though; getting there late is better than sitting in the terminal forever.
This article outlines how this joint venture re-vamped Delta's IT systems (again remember, this is 1995):
The trail runs dry here, job postings stopped around 2001.
Which really raises suspicions that all the code is written and maintained offshore. The question now becomes who is handling this for Delta.
One of Tata's spinoffs, Airline Financial Support Services, is described as
Wipro handles some of Delta's inbound reservation calls in India and the Phillipines.
In conclusion, it would appear that either Tata's AFS arm or Wipro do the IT for Delta airlines.
Nah, under Linux you can trivially create new files as swap space when needed. It may mean they overflowed available partition space on critical systems, or were unable to administer a heavily loaded fast enough to add swap before it overflowed.
Knowingn nothing else, I'd guess they overflowed a key database partition. A lot of old programmers very foolishly over-partition available disk, trying to outguess the OS about what partition will need how much space and instead of protecting themselves from disastrous overflows, actually causing disastrous overflows. It's an old programmer's habit that's hard to train people out of.
WTF can't they do it manually? It's just keeping track of seats on planes for fsck's sake. Sure, they may not be able to accomodate everyone right away, but they could certainly do better than "nobody can fly at all because our computer system crashed". If a restaurant loses their computer, they don't stop admitting people. They just go back to paper orders/receipts.
Sounds like Diebold may have been contracted for the job.
- First they ignore you, then they laugh at you, then ???, then profit.
This is a worst case scenario for a system of that nature because of so many dependent calculations and calls to other systems. It takes more than just having a plane and a crew...which is a lot of work all by itself. It has to have a gate and connecting flights. Then multiply all that by 30,000 people, roughly 120 plane loads, and complicate it by some airports being closed. I bet you could actually watch the lights get dimmer in the server room. Still when you know the potential peak demand you have reserve capacity. Slow is okay, stop is unacceptable.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
Well, the janitor did not pull the plug, but at a major airlines, the carpenter did. He wanted to plug in his electric drill and pulled the plug on the server and its power back up and every other wire plugged into a power source in sight!!! This came to light after it was reported to my company (we provide the software) that the system had crashed totally. We sent 2 people over and they came back laughing so hard that it took them 30 minutes to tell us what had happened.
After 9/11, pretty much all of the domestic airlines were bailed out by the government to keep them from going poof (except for Southwest and a couple of others, who didn't have their heads up their asses). So I just want to know how long it will take this Delta affiliate to plead for money. That not only has it screwed over all of those passengers, the taxpayers will collectively pay for it.
"You're never ready, just less unprepared."
At least, if he is speaking the truth about this ;-)
Some company refuse to do a complete system over haul. They just keep patching and upgrading. Their decade old software.
1 599.htm
What i am saying is the software could be a couple decade old and be a rats nest of code.
Kobold anyone?
I did a little researdh Conair was made Y2k compliant with only mirror changes.
http://budgettravel.about.com/library/weekly/aa10
actually disaster management strategy IS optional.
this story as an evidence.
though seriously, there's quite a lot of companies out there that instead of hiring incompetent people could be better off buying the services from outside. outsourcing doesn't necessarely mean it's crap, there's a lot of domestic in-house crap and idiots everywhere.
world was created 5 seconds before this post as it is.
I have watched the operation at Atlanta for over 21 years, and I've seen how cutthroat the competition for a major hub is, but it feels like watching two dogs fight over two bones--you can't tell if they're fighting out of greed or stupidity. Southwest doesn't even fly into Atlanta--they know that only a pyrrhic victory would be possible under those circumstances. Management at the other airlines has been criminally incompetent ever since airline deregulation, but it's the passengers, employees and shareholders who pay the penalty time and again.
Judging by www.comair.com and their job ops, it's probably HP-UX or Windows. More than likely the Unix flavor rather than Windows. Why down for a couple of days, probably a database restore. Never happens in TPF. Those mainframe systems crash and are back up with very little database degredation. By the way, in the job ops, if you want to be a crew scheduler, only need HS diploma!
The system should only be able to hold as many reservations as it has flights/seats.
Useless commentary. Airlines overbook to fill planes and the system has to accomodate this.
Is that a trick question?
quidquid latine dictum sit altum videtur.
The problem with your analysis is that point-to-point flying doesn't work when you start talking about international travel. It's just not possible to fly passengers to, say, Germany or Japan from every domestic airport. The way you do it is to accumulate passengers at a major hub on the coast and then fly from there.
The hub-spoke system is easier to manage, and can be profitable if the airlines relize that they aren't unlimited resources, and decentralize the hubs on a limited basis.
Anyways Southwest doesn't drink anyone's koolaid, they run all their own in house designed systems (I am not sure they are even on Sabre anymore), including web apps. It's an intresting concept, but it probably causes their IT managers to pull their hair out.
I worked on a car dealers' wide area network for a short time. Their entire network, all connections to other dealerships, internet connectivity, not to mention their Novell network, dealership inventory, parts, and tie-in to the manufacturer(s) was tied to a single router. They had problems, and I finally drove out there, and found the router "installed" in the drop ceiling above the mechanics' bathroom. The opposite side of that wall was the backer board for the telephone lines, located in a broom closet. I pulled the router down, and the inside had green mildew on the board. Routinely, the housekeeping service would unplug the 25 foot ORANGE extension cord plugged into the single-socket bathroom outlet! I advised the general manager about these problems, told them that they'd best extend their demarc, move the router to a better location, but they never bothered to fix it.
From Yahoo Jobs:
Software Engineer Cincinnati, OH $40K -$50K
More than likely it is TPF as Delta is a TPF shop.
m l) has been around since the '60's and is used by all the major airlines, most of the large hotels and most bizarrely NYC 911.
TPF (http://www-306.ibm.com/software/htp/tpf/index.ht
My wife says things just snowballed.
Crew assignment is a hard problem...
Records keeping, very tricky. You would not want to try that with any old database, no sir, it might pop a window. Just thinking about how every other airline has managed this tricky problem since before computers makes my head hurt.
We may never know what really happened but this would be a nice example for my classes :-)
Yeah, it's a real class act for those 30,000 people sitting around in airports for Christmas, employees doing the same and those who have to recover from this disaster. Management is going to be happy about the publicity they just earned while their huge capital investment in AIRPLANES sits idle during a time of year that's supposed to be their most profitable because their far to expensive M$ "soloution" "melted". A chain is only as strong as it's weakest link. Employees, I'm sure, are also stranded for Christmas. For the New Year they get to ponder layoffs. What a happy company for you to dissect at your leisure next semester. Season's Best!
Here's what I'll bet you might learn: WHEN SOMETHING MELTS, YOU LOSE YOUR ASS IF YOU DEPEND ON IT. MICROSOFT MELTS AND HAS POOR OR NO FAIL OVER CAPABILITY, SO YOU BETTER NOT DEPEND ON IT.
Friends don't help friends install M$ junk.
What happened to Comair here could happen to just about any airline. There is no comprehensive suite of software that handles crew scheduling, aircraft scheduling, reservations, and the myriad of other functions that are needed to run an airline.
Reservations, for other than tiny airlines, are still managed by large TPF mainframes. TPF is a very "bare bones" operating system that runs on IBM mainframes, and was written specifically to deal with high volume / high transaction rate systems. Personally, I've seen 5 attempts at 3 different airlines to replace it with something modern. ( like Unix with an RDBMS ). Each attempt failed miserably, and the airline went back to TPF. Note that TPF is not MVS, OS/390, or any other more mainstream Mainframe OS. It's purpose built.
Unfortunately, this means that all of the other applications have to interface with TPF via screen scraping. To further compound the problem, no "suites" exist to handle the following functions, so most airlines have to "sew together" best of breed solutions for these basic functions:
- Crew Scheduling - F/A's and pilots bid on
slots to fly, this system takes those bids and
turns it into a schedule.
- Aircraft Scheduling - Tracks which tail numbers are flying which flights for the dispatchers
- Optimization - Different optimizers to do
things like:
- Fuel Tankering - Use the jets as "tankers" so that you buy fuel where it's cheapest for flights later in the day
- Crew Optimization - "Traveling Salesman" type solver to incur lowest labor cost, get crews
back to home base, etc
- Schedule Optimization - Use the aircraft
in the most cost efficient way to cover all of
your scheduled flights.
- Maintenence Optimization - Pull aircraft in
for Scheduled Maintenance at the optimum time.
- Reacommodation - When things go wrong ( weather, mechanicals, whatever, pull in all of the above variables to crank out a new schedule,
crewing, mx schedule, etc )
- Booking Engines, for the internet and
reservations agents
- Point of Sale and Boarding functions for
agents, skycaps, and kiosks
- Interline functions where other airlines
sell your tickets, and transfers for bagggage, etc
Anyhow, this list isn't comprehensive, but shows enough of the disparate pieces that you can imagine why these "glitches" happen. Very few of the items from the list above come from the same vendor, or even run on the same platforms.Hub and spoke isn't the problem. You NEED it to get anywhere that's not a nonpopular destination. By saying that hub and spoke is a flawed concept, you effectively resign smaller cities to death.
Say I want to go to Butte, Montana and I live in Boston. How is a direct route method of Southwest airlines going to get me there? Isn't the most efficient and cost effective way for an airline to transport me on a larger jet, to say Denver, and then, use a smaller less than 100 passenger plane to fly me to my destination of Butte? How do you say to medium sized cities without rail lines, we're not going to use the hub and spoke method and we're going to destroy any sort of business you have (not just tourism, but meetings/conferences)? How do you tell that to the people who will have to drive hours to get to a major airport?
I agree that the upper management is corrupt. It really is. But because management is corrupt, you can't go saying that hub and spoke is flawed. In the future, I forsee a multitude of direct route airlines, and one big airline that still employs hub and spoke (either government subsidized, or just large enough and efficient to turn a profit). After all, how are airlines like Southwest going to get us overseas? Suddenly, the Jet Blue/Southwest system doesn't seem to efficient.
So, what you're saying is this industry is operated by a disorganized hodge-podge of cross-connected hacked-up computer systems, such that it is a minor technological miracle that planes a. get off the ground, or b. ever make it to where they're going. Thanks ... I feel much better now.
The higher the technology, the sharper that two-edged sword.
Your statements are accurate.
I was a unix sys admin there, but left for greener pastures during the dot-com craze. The non-redundant hardware at the time ran AIX, and had a great support contract from IBM. The SBS application however, always had monthly issues, at least at that airline. They were looking for a replacement then, and I'm not suprised they still haven't replaced it.
Take Amtrak!
Amtrak receives around $500 million for a total budget, while the airtravel receives around $15 billion in subsidies. Take the train and save everyone money!
_____________
Huh?
Oh come off it. What would you like him to do, sit there feeling bad for all the people who got screwed over? At least he's being proactive about it studying and analyzing it to try to figure out ways to prevent it in the future, as opposed to sitting on Slashdot and bitching how someone is being analytical when the troll (sorry, poster) thinks they should be sympathetic instead.
Of course, he might be sympathetic as well, but who cares when we have blanket assumptions like the ones you've made in your flamebait post.
Buy Steampunk Clothing Online!
It's an intresting concept, but it probably causes their IT managers to pull their hair out.
Interesting question: Would you rather have an easy(ier) job with a company that loses billions of dollars a year, or a hard(er) job with a company that actually makes money and is going to be around for the next five years?
Me, I'd choose the later. I'd rather be bald than working for a company with no future.
What part of "shall not be infringed" is so hard to understand?
Yep...SWA is still on SABRE or more precisely SAS, Braniff's old rez system in Tulsa(still TPF based).
Everything else is done in house in Dallas.
Which leads one to a question - why does the most profitable airline in USA not outsource its IT??
Skip ------ See the latest from http://www.anArchyFortWorth.com
I got into Boston on the evening of the 24th for the final leg of my HNL-YFC flight. The small gate area for Comair passengers was overflowing with-not-so-happy looking folks. Checking with a few of them I discovered many had been waiting since early afternoon for their flights, with nothing more from the Delta folks but glib, non-informative announcements now and then. The aircraft were on the ground, but there were no crews available to fly them. I waited patiently while my flight got delayed and delayed again. Finally, after many passengers took the only available flights remaining to alternate destinations, did Delta/Comair finally admit our flight was cancelled. They gave us hotel/meal vouchers (well, I got a hotel voucher anyway) and we went our way disappointed but expecting to get to our destinations a day late.
The next morning when we arrived back at Logan, our flight had already been cancelled. The lineup stretched from one end of Terminal C to the doors leading to Terminal B, and then some. While we waited, an airport rep was walking the lines asking where people were going. When our Comair destinations came up, he said "Go home. There'll be no flights out until maybe Monday or Tuesday." I thought people were going to cry right there. Our little band decided to try renting a car and driving. No luck, with no cars available at Logan that day. We tried buses, but no buses were running to our destinations that day. I seized upon the idea of using my Air Canada Aeroplan miles to get a free ticket to my destination (plus booking and taxes, $60).
When I was waiting in the Air Canada line, I saw one of the people I was with earlier. She told me the Delta rep told her to go to other airlines and ask if they would give carriage based on the Delta tickets she had. Basically, Delta was telling people they were on their own. One guy told me he watched a Delta agent tell a lady to fuck off. Anyway, I got to the counter and asked the Air Canada agent if they were doing anything for Delta passengers. She told me unless Delta signed the ticket over to us they were not doing anything special for us. I didnt know what that meant, but I knew Delta was doing fuck all for us, so I went ahead and used my Aeroplan mileage ticket.
The Delta fuck-up was basically all the news on Christmas and Boxing Days. Even the Canadian Immigration guys who never talk or joke around were feeling sorry for us. I dont know what I can get from Delta for the HUGE PITA they caused me, but I dont think Ill ever fly them again.
He said he fell back and by accident hit the power switch.
You said Write-permit rings or write rings are light and don't actually fly well, so I doubt they were playing with that as it is unlikely to mass enough to flip the switch.
His body certainly had enough mass.
Infuriate left and right
"From Yahoo Jobs:
Software Engineer Cincinnati, OH $40K -$50K"
That's more than I make at McDonalds.
A few years ago, a fellow was running an antique steam engine at the county fair. The steam tank ruptured, and one person was killed.
This is obviously a tragic event for the person who was killed and his family, but it's also an interesting and important engineering problem. So much so, in fact, that the following year's PhD qualifier in our mechanical engineering department consisted of an analysis and redesign of the steam engine, so as to prevent such an explosion from happening again.
It's a *good* thing to study problems like these, even in an academic sense, because academia is the very first step in the production of usable goods. If people aren't learning from these mistakes, then problems like the stranding of thousands of passengers in airports or the accidental death of a poor fellow at the county fair will happen again.
because their far to expensive M$ "soloution" "melted"
In case you forgot to read the first half of the comments to this thread, it's already been revealed that ComAir's system runs on AIX, a product of IBM, and that the software was developed by a subsidiary of Boeing. When you flame Microsoft for something, at least make sure they're *involved* first.
I sent a summary of these Slashdot comments to my cousin who works at American Airlines hq in Dallas. Here's his response!
---
"ugh... I worked 9pm-1am yesterday (xmas day). I spent the first two
hours of my shift calling people to tell them their flight was
cancelled and reschedule them. Most of them were taking flights out to
Miami and the Caribbean to spend New Years Eve partying on the beach.
Honestly, I had little pity telling them they were going to miss out on
one day of tanning especially since they seem to 'blame' the weather on
us.
"One hour into my shift our reference system went down. No IT people
were willing to come in and fix it. I had the system up for booking
flights and making reservations, but I could not look up any of our
rules and regulations. Ah well, enjoy your xmas off IT guys!! Enjoy
the weather in Cabo San Lucas!! Cheers!!
"Fortunately, we have a backup of all our html files saved as text
files. However each text file can only hold serval hundred text
characters. So, when I want to look up our baggage policies the normal
html file is called BAG INFO. In the backup system BAG INFO is
separated into 10 or 20 text files and I have to 'page' through them by
typing BAG INFO P2, BAG INFO P3, BAG INFO P4. The text files are not
indexed and are not searchable. It took me 10 minutes to find and
advise someone how big a bag they can take to Puerto Rico.
"After I started taking incoming calls again, there were people calling
in on Christmas day to book their trips for Spring Break. There were
over 100 calls on hold to talk to us, and there were people sitting on
hold for half an hour to ask me how much it would cost to book a trip
to Fort Lauderdale in March. Couldn't that wait until the day after
Christmas?
"Yes, the airline industry does not prepare for emergencies as well as
it could for the holidays when people want to travel in record numbers.
However, I think the general public could try to have their own backup
plans in place as well and realize that the travel industry in general
does not have the equipment or the staff to handle everyone in the
country wanting to travel all at once in one week. Do people stock
their refrigerators year round with enough food to feed everyone in
their families at one meal like they do at Christmas?
"Even though we try to accommodate everyone as best as we can on the
holidays, we want to to have a holiday just as bad as the rest of
everyone else. Working in the travel industry should not indenture us
to be your slaves over holidays. The public needs to have a little bit
of compassion and realize how much we give up in our own personal lives
just to help you get where you are going. Frankly, the way most people
treat me on the phones I don't think they deserve our help and
compassion. And don't call on Christmas day to book flights in March.
That phone call is making someone work on a day they shouldn't have to.
"anyways.... heh..... guess i had a bad night at work last night, huh
"MERRY XMAS!"
They believe in doing things differently from all the rest of the airlines, but IMO they are throwing the baby out with the bath water in somes cases, developing your own software has left them behind when it comes to web orders, which is possible revenue, and less seats empty.
Don't get me wrong I respect Southwest, but there are areas where they could have done better.
As has been pointed out by others, while hub-and-spoke has it's flaws, it's also the best solution (from an economic standpoint) to get from say Billings, MT to Jacksonville, FL.
There is no way any airline could make money flying that route directly.
And Southwest does, in fact, operate on a hub-and-spoke system. They have major hubs in Dallas (DAL), Houston (HOU),Phoenix (PHX) and Baltimore (BWI) among others.
However, they also supplement that system with regular point-to-point service, but only on profitable routes (from big metro area "a" to big metro area "b"). Again, they'd lose their shirt on the "Billings-Tallahassee" route.
On the other hand, Delta can get you from Billings to Tallahassee....
Scott
©20014 angrykeyboarder & Elmer Fudd. All Wights Wesewved
A freak weather event did more damage to a computerized reservation system in one night, than all of the hackers, viruses, trojans, spyware and idiot lusers combined over all of 2004.
Unless you take the overinflated guesstimates of the likes of mi2g at face value, anyway.
Toronto faced a blizard last week. Some two hundred flights cancelled because of bad weather. Air Canada, West Jet, Jetsgo, etc didn't go down even if their planes did. That too, caused more damage to YYZ's fiscal health than all of its computer security woes combined through 2004.
And then I read the letter posted here about the IT guys sunning themselves during all of this.
Merry xmas. I hope you can justify your jobs in 2005.
Use Evolution instead of Outlook? Bewa
Still "Rule 240" went went out with Airline Deregulation years ago. It's a generic term these days and every airline can pretty much do what they want as far as re-accommodating passengers.
For the most part they are fairly consistent and I'm sure that's mostly only for customer service/competition reasons.
Scott
©20014 angrykeyboarder & Elmer Fudd. All Wights Wesewved
"Hub and spoke isn't the problem."
Yes and no. Southwest does have hubs. I would say that Salt Lake City is a Southwest hub.
"You NEED it to get anywhere that's not a nonpopular destination."
Well, you have a few choices.
Accept that you can't fly everywhere. (Olympia, WA, the capital of the state, has had great trouble keeping air service of ANY kind-they seem to survive).
Accept that it will more inconvenient. Instead of flying to a hub, you will have to fly to a city from which the flight that goes to "nowhere, USA" originates. Many Southwest flights have multiple stops....
Accept that you will have only a few large profitable "hub and spoke airlines" (fewer than now because they aren't profitable....) And they will routinely suffer huge delays....
"By saying that hub and spoke is a flawed concept, you effectively resign smaller cities to death."
No, they just don't get convenient/cheap air service. If I want to go to Olympia, Wa., I have to fly into SeaTac (an hour drive). Always been that way, probably always will be.
Remember the old joke (paraphrased):
How do you become a millionare? Buy an airline when you are a billionare....
I think the key algorithms we studied in the mid-90s were developed in the late 80s. If my poor memory is right, the complexity dropped from O(n^3) to something like O(n^lg(n)) or O(n^lg(lg(n))).
There's even more stunning improvement in the algorithms for solving multidimensional partial differential equations. (e.g., weather). Put a modern vector processor supercomputer running the algorithms from the 70s in one room, and a TRS-80 running the latest algorithms in another room, and the TRS-80 will easily beat the supercomputer. (Assuming it has sufficient memory to hold all of the model data, of course!)
Implement the modern algorithm on a 1024-node cluster....
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
Time to spare?
Go by Air.
that all the women had. inductive loads are a real bitch.
maybe one day i'll be smart enough to come up with a cool sig, too.
Yep, They will do anything to "cut costs"
in my way of thinking this was a management failure. technical, finiancial, and operational management share this joyful problem.
my understanding is that the software wasn't designed for the volume of txn pushed down it's throat. slowly this information "trickled up" the food chain and right back down. rinse, repeat.
the folks that supported the system would say it's overloaded. the technically-responsible management would tell the next layer up that they needed money to fix the problem. the financially-responsible management layer would tell the operationaly-responsible layer that someoneone wanted more money than the fiscal budgeting planned for and it would be painful to fix. the operational-management would squeal like stuck pigs and never, ever, tell shareholders a story like that one. the financially- responsible people would convey this unhappiness downwards. technically-responsible management would "find a way make it work" and divert the unhappiness to the support people. the support people would piss/moan/bitch and make-it-so as best as possible.
repeat this cycle 'til flames shoot out of the mission critical system(s).
then "suddenly" the problem is handled properly but in a somewhat hurried manner.
i am snow. fear me.
The trick is to have a crewcut. That way you can't get a good enough hold to pull the hair out. ;)
What if I wanna fly to Atlanta?
The proactive people said not to use M$ crap. Studying the mess afterwards is the very definition of reactive. The lesson was obvious before it happened, and the rest of us are entitled to a happy, "I told you so".
we have blanket assumptions... in your flamebait post.
Microsoft sucks, yeah, yeah, yeah. All your apologies are worthless.
Friends don't help friends install M$ junk.
It's interesting how you dissmiss an OS with a track record of failure in order to blame anyone other programmer. This assumes Microsoft has better programmers than anyone else, an assumption Microsoft marketing loves you for but is unsupported by any objective review of of performance. The same "stupid people" have been and still are writing applications for Linux, Unix and Mac right now but they have better tools and make fewer mistakes there than they have with M$'s crappy SDK's and pathetic OS.
Two examples of software that just works are Apache and Sendmail. People write all sorts of applications for both of them without this kind of meltdown and both dominate their "markets". Microsoft's efforts at both, IIS and exchange have been a total dissaster.
Wanna bet what crappy OS is behind this? Blaming you developers is sorry stuff.
Friends don't help friends install M$ junk.
Well... to try and provide a little clarification here, as I work for Comair. Here's the skinny:
:)
Crew and aircraft scheduling is done through a software package called SBS Track. This very same software package is used by many other airlines, including the two I worked for before coming to Comair. I don't know if their systems have the same hard-coded limit that ours does or not. This software package has _nothing_ to do with reservations, or anything concerning passengers whatsoever. It is simply the software we use to schedule our aircraft and crews to fly the list of flights that Delta wants us to fly.
Crew scheduling is done by creating "pairings". A pairing is a sequence of flights that comprise a crewmember's trip. Anytime a change is made, a new pairing is generated, with the new sequence of flights. The system has a hard-coded limit of 32k pairings ("transactions" is the what the IT folks call it) in a calendar month. As of 10:00 pm on 12/24, that limit was reached. Crew Scheduling was unable to create any new pairings, unable to track who would be flying what airplane to where, and basically unable to keep the airline flying at that point.
It was not any kind of a hardware failure, there are backups for that. It is simply a software limitation, that when it was coded many years ago, nobody realistically thought it would ever be reached. Why they hardcoded a limit into it in the first place is beyond my knowledge.
A major part of the problem is Comair's concentration in Cincinnati. CVG is our only crew base, and it is the largest single crew base of any airline in the world. Over 1800 pilots and 1100 flight attendants in one base. Not even any of the majors have a single base that large. Several of our software packages are woefully inadequate, and replacements have been sought for some time.
As for getting things up and running on paper, this is a monumental task. Scheduling for 160+ aircraft and 2900+ crewmembers, and compliance with all FAA regulations, maintenance requirements, crew rest requirements, and contractual requirements is incredibly complex. In addition, we have crews and aircraft stranded across the country due to the weather that moved through that caused this whole mess in the first place. Add to that the very limited number of people who actually have the knowledge of all the requirements for scheduling, and coming up with a full schedule for the next day would be nearly impossible.
Jan. 1 starts a new month, and the system will return to full functionality then. Until that date, however, our operations will be very limited.
No, I don't feel hurt by some highly moderated anonymous "inside information" that clearly contradicts
Friends don't help friends install M$ junk.
Some of you have no clue.
The BS&T quotient on your average travel application is on the relatively nuts scale. Expedia, Travelocity, hotwire, priceline, whatever -- I'd ask that some of you with simple solutions go and speak to the lead travel-server dev for the product.
You'll probably have to change pants after the conversation. Travel is stable, reliable, and generally rock-solid. The algo's for selecting airline flight prices or hotel room block-reservations are known and well-tested. The methods and protocols of communication are well-documented and generally straightforward.
Until recently, it was all on hardware (And i'm speaking generally about the large travel providers -- Worldspan and Sabre come to mind) that was considered arcane. Ancient versions of Netware on an X.25 pad; screen-scrapers on top of it. Have Fun trying to modernize!
This does not suprise me in the slightest. We are stressing our ancient systems more than ever these days, and it should not be a suprise when the occasional ancient application (ctime, folks) gets floor'd and dies a bloody death.
It'll be patched in a month.
Okay, nobody's going to read this, because it was yesterday's story and it's been modded into oblivion.
But can somebody please explain to me how this is off-topic? Somebody posted something that wasn't, admittedly, perfectly on topic (but worth a `Flamebait' more than `Offtopic') and I replied to that exact post.
I could cope with being modded Flamebait. Having a read, it probably does qualify. I'd even be happy with a Troll. But I made a perfectly relevant, and thus ontopic, reply to a post.
I could go on, but there's no real point. Metamods, a bit of justice please?
Cogito, ergo sig.
Learned that long ago from a military man, I understood why later, he did a tour at the Pentagon.
I have to wonder about the Healthcare area being worse.
I worked, off and on, at a major VA Hospital. The VA, of course, like all parts of the Military system not related to procurement, is seriously underfunded. But, the only localised system in the entire facility I worked in, was the CT-scanners. They needed super hi-def imaging, so they used SGIs, and something that 'looked like' an X-window-type OS. I never ran those boxes, so I don't have better, more accurate info on that, sorry. (It might have been some version of solaris with a KDE or Gnome lookalike desktop, don't know. They were all leased from GE, anyway, and had really intelligent people running them. Every other department was tied-in to the same over-all 'system', as far as boxes and software, but separated from other departments (for obvious reasons, "Admissions" didn't need to be looking at/accessing Food Services, etc).
As for the rest of the hospital, all of it was on a unified system. With the exception of paper-based Medical Records. You cannot imagine the enormity of the paper-based issue. It defies simple 'scanning' and conversion to electronic docs, due to the wide range of forms, crazy handwriting, etc. All that aside, maybe the 'normal' hospital systems are different. Or your post was referring to the Healthcare industry at the governmental level. Not sure.
But (this is the 'plug'): If you, or anyone, care(s) about the kids and fellows that have already 'done their duty' (whether you're pro- or anti- political war is irrelevant, IMHO) be aware that the Administration (White House, power structure, whatever) is contemplating further severe cutbacks in health care, hazard duty pay, death benefits for families, etc, and write to your representatives, expressing your opinions.
As far as the attitude, expressed by some, that the airlines (or any company, for that matter) should go bankrupt, to be 'taught a lesson', for management stupidity, or bean-counting decisions, may seem reasonable, but the only people hurt (and they are numerous) are the ones with the least responsibility for the failures, as a whole. Think of it this way: In a country that has a tax and Congress (whose primary purpose is to divy up tax dollars between competing corporations (aka the real 'Special Interests') working as a system designed to facillitate Corporate Welfare for the rich and powerful, the last thing we need are more 'little people' needing money for food, shelter, their families, etc.
Vendor A, B, C, ...; package Z, Y, X, ...; ...; contractor ....
integrator 1, 2, 3
Maybe I'm too old, but it sure was nice when
at least the applications were written in house,
yea we would bitch when the writer of the code
was long gone, but still, we could fix it.
Oh well, I just wish I chould get use to saying
"its the vendors fault" when ever something goes
wrong.
Conair made the descision not to replace/upgrade
before January. End of Year, make the profits look better ? The roll of the dice ?
-pete
which i didn't do. i clipped out the bit referring to the floor buffers which also nailed us. sorry, thanks.
maybe one day i'll be smart enough to come up with a cool sig, too.
Make in the early 80's I used to work making applications for and servicing very early microscomputers from a company called IMS. They basically consisted of dishwasher sized box with a S100 bus into which a number of single board z80 computers plugged, sharing a single hard drive (that's "Winchester" to you sonny) controlled by a master computer, and comunicating using a hacked up CP/M compatible system called TurboDos. The result was one of the first multiuser computer systems suitable and affordable enough for routine office use; you could equip a half dozen people for paltry twenty or thirty thousand dollars.
Anyway, the manufacturer discovered that the office environment was a bit more, uh, challenging than they anticipated. Static was turning out to be an issue. So one day they issued a tech bulletin saying that secretaries should stop wearing pantyhose. In the formal (in those days) frozen Northeast, women did not wear slacks to work, any more than men went to work without a tie. So the resourceful ladies began to bring spray bottles of fabric softener to work; they'd run into the ladies room to "freshen up" every hour or two.
Of course aside from the inexperience of the manufacturers, we still had the ususal load of PEBKAC malarkey. We had one customer who was complaining about hardware crashes. Now you can imagine these weren't altogether uncommon given the primitive nature of these boxes and the operating system they ran, but he seemed to really having a bad time of it. We sent a tech, who on entering the customer's newly built and rather swanky computer room, noticed a dimmer switch on the wall next to the door.
"What's this?" he asked.
"Oh, that runs the computer."
Post may contain irony: discontinue use if experiencing mood swings, nausea or elevated blood pressure.
So lets think this one through for a second. The people who work there say the system that failled runs on AIX and that its the application thats gone whoopsie. So they obviously must be lying since everyone knows that the minute an application is ported to AIX all the bugs fall out of it.
Of course with this type of thinking there is no way that reputations are ever going to change since every computer error is attributed to Windows even if it has nothing to do with the issue.
I suspect that the HR advert is for a completely unrelated job.
I also would hazzard a guess that the real problem at the place now is not the system anymore. The system is probably back up but they are now having to deal with planes that are in the wrong places and crews that have no flying hours left because of decisions that were taken manually while the system was down.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
its a 24+ year old application, it should have been repalces years ago with a more mordren system.
Not much a OS can do about a varible overflow in an application program, windows or AIX.
High on the list of things Lisp offers that most other languages botch is the idea that (+ x 1) for any integer x should return a number bigger than x in all cases. It seems like such a small point, but it's often quite useful. -- Kent M. Pitman
Dyslexics have more fnu.
Because SWA knows that IT is a critical part of their business not an "expense" to be minimized. Anybody with the proper qualifications can build an airline. To run one efficiently one needs to be in control of all the variables. To do this one needs efficient and effective IT.
At the time, you did not know that people who worked there said that. All you had to go on was a post by another Slashdotter claiming an anonymous person told them that. It was pure hearsay, but it seems to have turned out correct as Comair has come out and said the same thing.
with this type of thinking there is no way that reputations are ever going to change since every computer error is attributed to Windows even if it has nothing to do with the issue.
Actually, the reputation will not change because Microsoft will not change. This one woopsie just happened to not be Windows, that does not make the platform any more stable. Microsoft has been warned about the dangers of their system designs but has chosen to blunder forth.
Given a choice, which one would you rather be responsible for? Which one would you use for a mission critical application? Unbelievably, many airlines use Windows as a terminal for ticketing and other very important functions.
Friends don't help friends install M$ junk.