Comair Done In by 16-Bit Counter

Gogo Dodo writes "According to the Cincinnati Post, the Comair system crash was caused by an overflowed 16-bit counter. Perhaps Comair should have paid for the software upgrade to MaestroCrew." You heard it here first...

Comments

Forget Y2k...

[Cytlid]

This was Y32k!

Re:Forget Y2k...

[Bigby]

you mean Y65K?

2^16 (or unsigned short in C++ on x86), stores values from 0 to 65535

Re:Forget Y2k...

[stupidfoo]

RTFA

It was a signed integer. The problem occured at 2^15 (32768) (although the article reported it as 32,000)

Re:Forget Y2k...

[stupidfoo]

I replied to a guy who said "you mean Y65K?"

Not the guy who correctly stated "Y32K"

Re:Forget Y2k...

[FormerComposer]

So it was actually a 15-bit counter. And that was necessary because we all know that a negative number of changes was likely to happen so it had to be implemented with a SIGNED counter rather than an unsigned counter. (Would doubling the number of changes allowed by using an unsigned counter have eliminated ... errrrr, postponed .... the problem to another day?)

Well...

[Tuxedo Jack]

It seems that 16 bits and 640K wasn't enough for them after all.

Re:Well...

[HiThere]

Actually, it *MIGHT* have. No guarantees.

But it would have been much easier to fix.

The problem here is that even though "with enough eyes, all bugs are shallow", this application is specialized enough that there might well not have been enough eyes. Still, if it were open, then the people who work there might have spent some time looking through it. And *MIGHT* have found the problem.

OTOH, with open source, when the problem manifested, it could have been debugged and recompiled starting immediately. This might well have saved they several days, certainly several hours.

OTOH, if it were FOSS, then it would be their job to maintain it. This would increase the chance of the problem being fixed, but decrease their ability to point fingers at someone else and say "It's all their fault!", which is the capability that they seem to most desire.

actually...

[erroneus]

...I heard it on BugTraq first...

Re:actually...

[Chris Mattern]

I heard it on comp.risks.

Re:actually...

[nuclearspike]

I heard it from the ComAir desk at the airport when I was trying to get home. :(

Re:actually...

[mobiGeek]

I heard it from the ComAir desk at the airport

Attention passengers awaiting ComAir flight 5884, destination Cinncinati Ohio. The pilot and co-pilot regret to inform you that due to a 16-bit numerical overflow in our scheduling system, departure of this flight is being further delayed from our regular list of excuses.

Please continue waiting. We will misinform you regularly as we make up various status reports. If you have any questions or comments, please feel free to keep them to yourselves.

Again, we apologize for the delay. I will be leaving for the day now, going home to have dinner with my family, but please do make yourselves comfortable here in these plastic chairs we kindly provide you at no extra charge.

Oh, and one last announcement: If you are a computer programmer, please identify yourself to any of our staff members so that we may have security keep an eye on you.

Have a great evening!

Re:actually...

[nuclearspike]

They got on the PA and read off only 16 bits of information, then halted. It was a hidden message, meant only for the programmers. Much like the story of the telegraph station who put out an ad for an operator. In the room full of applicants, there was a tapping sound that only one applicant heard. In Morse code, they were tapping "If you understand this, you are hired, come to the back room. STOP" But I'm not going to work for ComAir. :P

Re:Signed or unsigned

[Vengeance]

I believe this will answer your question:

Tom Carter, a computer consultant with Clover Link Systems of Los Angeles, said the application has a hard limit of 32,000 changes in a single month.

"This probably seemed like plenty to the designers, but when the storms hit last week, they caused many, many crew reassignments, and the value of 32,000 was exceeded," he said.

So it sounds like a signed int.

Common problem

[confusion]

Well, not this specific problem, but businesses have a common problem of outgrowing the systems that run their business. OTOH, this was an outsourced solution, so this case is pretty hard to explain away, other than sheer incompetence.

Re:Common problem

[Anonymous Coward]

That's not true.

Even if a system is outsourced it doesn't provide a company with 100% stable system. Frequently businesses define the type of system they want hardware/software and the amount they're willing to pay for it.

I work in a company that provides outsourced solutions. Monthly we provide info to businesses about their system. Also, we frequently make recommendations to augment the systems to improve performance. Businesses often choose to ignore our reports and recommendations.

Nothing's more frustrating then a meeting with a business having them tell us we mucked it up and in return we drop off the last 6 months of recommendations on upgrades to provide them additional hardware for their growing requirements and question why they choose to ignore it.

Now I'm not saying the provider didn't muck up. But, what I am saying is your statement that it's all the provider's fault may not be the case as the airlines probably choose to stay on that system as it 'met' their needs as they saw them.

Re:Common problem

[plover]

My point was more that it is much much harder to upgrade a system when it's managed internally.

Only if it's done wrong.

The "value" of outsourcing in that particular example is that it forces the company to completely spec out the system requirements. No changes without documentation. With poorly controlled internal development, changes happens in the hallways or the cafeteria: "Hey, Rick, did you add the code to handle the offline situation?" "Oh, right, I'll just put in a return value for you." This leads to code that doesn't match its spec, making it harder to maintain. Outsourcing tends to enforce a good interface between spec and code, (which is what your claim seems to be.)

Internally developed programs don't necessarily receive the same amount of attention to detail because the programmers typically have an idea about the business domain of the problem, and can work more with less documentation. In some organizations, this leads to "fast and loose" -- great for response time, not so great for maintainability.

I think the "value" of outsourcing in a case like ComAir's is one of liability: ComAir will probably try to play "let's blame the vendor." Or, maybe they'll offer up for sacrifice only the one guy who signed the contract with the vendor, and not an entire division. But, when a failure reaches this magnitude, I don't think they'll get off that easy.

Re:Common problem

[cayenne8]

"I think the "value" of outsourcing in a case like ComAir's is one of liability: ComAir will probably try to play "let's blame the vendor."

And...maybe it lends good reason NOT to outsource a company's mission critical system. Regarless of what you think of the quality of overseas IT....it is generally preferrable to have someone within reach to point the finger at, wring neck or...and obtain financial satisfaction...

Hard to do that in a foreign land on the other side of the world.

Re:Common problem

[plover]

I don't know, I think it's easier to blame a faceless "offshore vendor" than it is to blame "Jane and Ted and the guys over at QalityTech".

Not that any director I know has a hard time aiming the blamethrower at whoever's convenient ...

Re:Common problem

[SpaceLifeForm]

The "value" of outsourcing in that particular example is that it forces the company to completely spec out the system requirements.

Does it? I've seen such events occur because the internal IT staff did not have time to spend on the project. Guess what happens? Many non-IT people get involved in the specs, and as far as they can discern, all is well. The specs are then handed off to the outsourced company, and they run with it. GIGO.

Re:Common problem

[localman]

My point was more that it is much much harder to upgrade a system when it's managed internally.

Wow, in my (admittedly limited) experience, this is the exact opposite of what I would say. At the company I work for nearly everything is written in house. We even wrote our own warehouse management software (in Perl!?!). On the occasions that we outsource development, it has always proven more difficult and time-consuming to get the contractors and outsource companies to make the changes we need. We almost always end up writing an in-house version after tiring of the external software management cycle.

Just my $0.02.

Cheers.

Re:Common problem

[plover]

Then there's a strong argument to find a decent outsourcing firm. The ones we deal with won't touch a project with incomplete specs, and they spend a fair amount of time with our business analysts getting clarifications, etc.

If your outsourcing firm is that incompetent that they can't recognize crap on the way in, you're getting precisely what you pay for. Pretty much the meaning of the word "value".

Re:Signed or unsigned

[Thowllly]

Signed. Had the developers used unsigned then it might never have overflowed at all (They were going to replace the system in a few months anyway.)

Re:Signed or unsigned

[jejones]

Since TFA quoted someone who cited a limit of 32,000 reassignments, I would guess signed (and that they were just truncating--if it's really a 16-bit counter, the upper bound would be 32,767).

Bugtraq covered this as well..

[EvilStein]

Here's the original post:

Hi,

On Christmas Day last Saturday, Comair Airlines had to completely stop
flying
all of its planes due to computer problems. Comair blamed the computer
problems on their pilot scheduling software being overloaded after bad
weather earlier in the week forced many flights to be rescheduled. Comair
now hopes to have all of its 1,100 daily flights restored by tomorrow.

An article which was published today at the Cincinnati Post Web site
provides some interesting details of a software failure in Comair's pilot
scheduling software:

How it happened
http://www.cincypost.com/2004/12/28/comp12-28-2004 .html

According to the article, Comair is running a 15-year old scheduling
software package from SBS International (www.sbsint.com). The software has
a hard limit of 32,000 schedule changes per month. With all of the bad
weather last week, Comair apparently hit this limit and then was unable to
assign pilots to planes.

It sounds like 16-bit integers are being used in the SBS International
scheduling software to identify transactions. Given that the software is 15
years old, this design decision perhaps was made to save on memory usage.
In retrospect, 16-bit integers were probably not a good choice.

An anonymous message posted to Slashdot the day after Christmas first
described the software failure at Comair:

http://slashdot.org/comments.pl?sid=134005&cid=111 85556

Earlier this year, an overflow of a 32-bit counter in Windows shut down air
traffic control over southern California for 3 hours:

Microsoft server crash nearly causes 800-plane pile-up
http://www.techworld.com/opsys/news/index.cfm?News ID=2275

This problem occurred because of a known design flaw in older versions of
Windows:

http://tinyurl.com/5n9gc

Richard M. Smith
http://www.ComputerBytesMan.com

Re:Bugtraq covered this as well..

[dmccarty]

It sounds like 16-bit integers are being used in the SBS International scheduling software to identify transactions. Given that the software is 15 years old, this design decision perhaps was made to save on memory usage. In retrospect, 16-bit integers were probably not a good choice.

Rubbish. Don't judge yesteryear's programs by today's standards. Back then 4MB RAM cost more than $200. That's how important memory conservation was. In 1989 using an int was a perfectly acceptable choice. If you were programming back then you'd know how loathe programmers were to use longs when they didn't have to. (Granted an unsigned int would've worked better here, but that 64K limit could've also been reached.)

The software spec probably says something to the effect of "Don't attempt to schedule more than 32,767 crew changes." If you're running software that's more than a decade old you need to know what the limits of your software are.

Re:Bugtraq covered this as well..

[imsabbel]

200$ for 4MB? Thas more 1994 than 1989...

Re:Bugtraq covered this as well..

[Anonymous Coward]

In 1989 I was running GNU Emacs on a 386/20 with 16MB of memory running Unix, and I shared with one other person. I think this was a few years after, I think, Richard Stallman said he'd be surprised if Emacs would ever run on an x86 platform.

Sure, we had serial terminals instead of graphical displays, but still, a 16-bit signed int for memory conservation in 1989? That's a load of BS.

I wouldn't be surprised if they did it because they never gave it a second thought to use an int, or they needed to save on-disk storage space. Maybe their database backup wouldn't fit onto a 1.2MB 5.25" floppy otherwise.

Re:Bugtraq covered this as well..

[GarrettZilla]

I disagree. Which takes more memory - adding a couple more bytes to this counter, or putting in the code to check for maximum value exceeded and emit a message saying "Cannot perform more than 32767 crew reassignments in a single month."? Or did they just press on into unspecified behavior after an integer overflow?

Hell, just putting the word "unsigned" in front of "int" (do you really need to tally a negative number of crew reassignments?) would have prevented this particular problem and given double the capacity, all else being equal. If you're worried about memory usage, it's certainly not a good idea to waste a bunch of bits on unneeded signs.

By 1989, we certainly knew that the world was in the habit of using software for ten years or more. Software was being modified all the time as larger memory spaces and requirements came along. It was practice long before that to be explicit about memory-related design decisions because you knew it would be your problem in five years to update the software. Unless you just ran away and quit before the problem came up, and it was somebody else's worry.

Re:Bugtraq covered this as well..

[graikor]

Well, in late '89, I paid $250 for 2MB, so Moore's law would put the figure he quoted at '90 or '91. Not too far off.

Re:Bugtraq covered this as well..

[plover]

In 1988 I was constantly having this argument with one of our other developers. He insisted on using a char when enumerating 80 or 90 status codes, or a short when conditions were "unlikely" that we'd need a long. We both grew up programming in the '70s (at which point I'd have agreed with you -- back then we only had 16Kwords to play in.) Yes, our 2MB boxes were pretty tight on memory, but even in the 1980s it was obvious that saving a single byte in the executable was a false economy, if it risked stability.

The only place where shaving bits made sense for us was on data records: we had a hash file with 2.1 million records, each 29 bytes long and it they all had to fit on a single 80MB hard drive. We squeezed every single bit out of those records (including developing a 3-byte integer to handle amounts that we told them could never exceed $99,999.99 (among other things, larger amounts would not have printed correctly.) But they were read-only records to us: we never wrote more than a few thousand rows of data, and we had plenty of space for the day's processing. And when they did have the odd line item that exceeded $100,000.00, they figured out to break it up into multiple smaller items.

And we got bit more than once by overflows. It took like three separate f-ups to get this guy to acknowledge that he needed to stop being stingy with the bytes. Even then, he'd still try to sneak in some memory "savings", but at least he stopped arguing when we called him on them.

Re:Bugtraq covered this as well..

[imsabbel]

Well, i remember painy 400DM (at that time 250$) for 4 MB in late 94 when i bought my 486... But in the 90s there were long times of stagnant memory prices (or even price increases... anybody remember those sad "plastic factory producing chip resources burned down" stories that were used to fake a supply drop?)

Re:Bugtraq covered this as well..

[multipartmixed]

I remember the day I started as a tech in a computer store in Canada; early september 1993. 486DX/2-66s cost $660 (about $500 USD then) and 1MB SIMMs cost $85 (about $70 USD).

4MB of RAM would then have been worth about $280 US in early '93, so definately more than $200 in '90 or '91.

Re:Bugtraq covered this as well..

[StarCat76]

So if memory truly were $200 for 4MB, those extra 16 bits would have cost them 9.54 x 10^-5 dollars. How much do you suppose canceling 1,100 flights cost them? Certainly in the millions of dollars. Let's say two million to be conservative. In order for the decision to use 16-bit ints to be cost-effective, the program needed to have 20.97 billion integers, which is a bit higher than usual. The point is that buying hardware is almost always cheaper than coping with a software error.

If you're running software that's more than a decade old you need to know what the limits of your software are.
What should they have done, decided to stop rescheduling flights? There was no good solution.

Re:Bugtraq covered this as well..

[fm6]

If you're running software that's more than a decade old you need to know what the limits of your software are.

Indeed. I get the impression that Boeing is very unmotivated when it comes to keeping its IS technology up to date. Until recently, they were still using slips of paper to track the process of assembling their airplanes!

What's particularly disturbing is that nothing was done about this during the big Y2K push 5 years ago. Of course, the official goal of Y2K efforts was to make sure your computers didn't crash on 1/1/2000. But it's pretty hard to separate Y2K bugs from other clock bugs, and I think most places didn't even try. Easier to fix or document the bugs than to classify them. I was involved with the Y2K effort at SGI, and we looked at everything from leap year bugs to the Unix 16-bit clock overflow -- which won't occur until 2038!

Re:Bugtraq covered this as well..

[SpaceLifeForm]

I believe the problem was that they couldn't handle any additional crew re-assignments. Hence, no crew assigned to flight, no flight.

Apparently, doing crew assignments via paper and pencil was beyond the reach of management.

Re:Bugtraq covered this as well..

[sconeu]

to the Unix 16-bit clock overflow

I assume you mean the Unix 31-bit clock overflow? (time_t generally a signed 32-bit value).

Re:Bugtraq covered this as well..

[justins]

The software spec probably says something to the effect of "Don't attempt to schedule more than 32,767 crew changes." If you're running software that's more than a decade old you need to know what the limits of your software are.

Now that is rubbish. If you're going to build a completely arbitrary limitation into the software you need to have the software hit that limitation gracefully, rather than cough up blood. The computer can keep track of how close it is to hitting the 32,767 change limit much better than an end user can. Duh.

This was just poorly written software. That's a "standard" that doesn't change from decade to decade.

Re:Bugtraq covered this as well..

[PCM2]

And we got bit more than once by overflows. It took like three separate f-ups to get this guy to acknowledge that he needed to stop being stingy with the bytes. Even then, he'd still try to sneak in some memory "savings", but at least he stopped arguing when we called him on them.

Wait a sec -- the root problem that leads to overflows isn't a lack of bytes. It's a lack of bounds checking. Right?

Re:Bugtraq covered this as well..

[budgenator]

Don't judge yesteryear's programs by today's standards. Back then 4MB RAM cost more than $200
Might have been more than that considering the back-end could very easily have been developed for a legacy big-iron system that had a wire-wrapped CPU and ferrite core memory.

Re:Bugtraq covered this as well..

[StarsAreAlsoFire]

Not buffer/array overflow. When you try to add just ONE MORE to a variable that is at its max value. Thus causing the value to wrap around to either some exceedingly large, negative value, or 0, depending on if it is signed or not.

Re:Bugtraq covered this as well..

[llefler]

If you're going to build a completely arbitrary limitation into the software you need to have the software hit that limitation gracefully, rather than cough up blood.

That's not an arbitrary limitation. It's a design decision. Arbitrary would have been something like 25,000, because it's not based on anything other than pulling a number out of your ass. It might have been an uninformed or arbitrary design decision to use a signed vs unsigned variable though.

Granted, I agree there should have been a check in the software to make sure it wasn't approaching a limit without notifying someone. And there should have been a documented process to fix the problem short of shutting down an airline.

Re:Bugtraq covered this as well..

[justins]

That's not an arbitrary limitation. It's a design decision. Arbitrary would have been something like 25,000

They are equally arbitrary to the end user, I'm afraid.

Re:Bugtraq covered this as well..

[randomblast]

>> (Granted an unsigned int would've worked better here, but that 64K limit could've also been reached.)

It should definitely have been an unsigned int. It's not even possible to have a negative number of crew changes...

From Another article...

[bje2]

from information week

"The computer failure that grounded an airline's entire fleet over the Christmas weekend and stranded thousands of travelers was due to creaky software that couldn't count higher than 32,768." ...

According to the Post, the software -- which tracks all details of crew scheduling, including how long they have flown (an FAA regulation restricts airtime), and logs every change -- has a 16-bit counter that limits the number of changes to 32,768 in any given month. ...

to be fair (although it's not an excuse), but 32K crew changes in a month? that's like 1,000 a day? that's crazy!...

Re:From Another article...

[Weissmohr]

Well, if it logs all changes to the "I've flown this long" counter, supposedly to make sure not to overstep an FAA restriction, I can imagine that many changes would happen, yes. :-P

Re:From Another article...

[Eudaemonic Pie]

Yes --

Comair has roughly 1,100 flights, with an average of 3 crew members per flight.

That's 3,300 changes in a day if everyone's affected by the weather which is close to 10% of your counter's limit.

Re:From Another article...

[Anonymous Coward]

>... 32K crew changes in a month? that's like 1,000 a day? that's crazy!

You arent by any chance the original developer of this software?

Re:From Another article...

[mikesmind]

Legacy systems will often contain such hard limits. Usually, they are buried deep in the code and sometimes no one knows that they exist. Any point where such hard limits exist must be discovered. A solution needs to then be designed for each situation. If you are a manager or a maintainer of such a system, it is your responsibility to do this. When you are questioned, just point out the Comair computer disaster.

Re:From Another article...

[tsangc]

to be fair (although it's not an excuse), but 32K crew changes in a month? that's like 1,000 a day? that's crazy!...

I would suspect the attitude of debating a limit without knowing the business context your design choice exists in is probably what created this error to begin with.

Re:From Another article...

[MegaFur]

bingo!

1000 crew changes a day might be crazy or it might not--it depends at least partially on the size of the operation.

Re:From Another article...

[jnhtx]

to be fair (although it's not an excuse), but 32K crew changes in a month? that's like 1,000 a day? that's crazy!...

Well, figure 3 crewmembers per flight, something like 1500 flights per day, cancel most flights for one snow and ice day near the end of the month.

Maybe not so crazy.

Re:From Another article...

[SWTP_OS9]

Agree. Why use a sign bit on it unless it was for overflow?

Re:From Another article...

[mpe]

1000 crew changes a day might be crazy or it might not--it depends at least partially on the size of the operation.

What was the size of the operation 15 years ago? Maybe in 1989 it would have been impossible to get that many crew changes in a day.

Re:From Another article...

[drawfour]

Shouldn't it be _at least 3_ crewmembers per flight? Pilot, copilot, and at least one steward/stewardess. Unless stewards/stewardesses are counted differently for some reason... Or unless I'm missing something, which is all too likely.

Re:From Another article...

[ArtStone]

The purpose of the software is FAA compliance for pilot time. Requirements for flight attendents are much less restrictive than pilots (# hrs/day, #/month, sleep breaks, qualified to fly on specific aircraft) The prior thread (dup?) on this topic had some posts with more detail from someone who knows the rules and complications and what it takes to implement in software.

The only real complication for flight attendents is they sleep overnight in a different city than they planned - but the airlines have large blocks of hotel rooms on permanent availability for just that reason.

Let's not be too hard..

[Staplerh]

This was a horrible chain of events that severely inconvenienced a lot of people for Christmas, and I would be hoppin' mad if I was in any of their places. However, let's not jump on ComAir too hard, IMHO. From TFA:

"This probably seemed like plenty to the designers, but when the storms hit last week, they caused many, many crew reassignments, and the value of 32,000 was exceeded," he said.

It's true, it was an extreme connection of circumstances... horrid weather (heck, there was snow in some Texas town for the first time in like 80 years or something, read it in some glurge article) coupled with the winter holidays. They should redesign their system and admit that they've grown to a level where their system is unable to hand extreme circumstances, and this should serve as a great wake-up call for them.

In the past I've always chuckled at the thought of 'upgrading for the sake of upgrading', but I suppose this is one case where an earlier upgrade could have saved them millions and made a lot of people's holidays better.

Re:Let's not be too hard..

[j0217995]

I agree, but also didn't the stewardess/stewards(?) and bagage handlers for Delta, which ComAir is a part of go on strike the same day as well? I think I read that somewhere

Re:Let's not be too hard..

[danheskett]

It was an unoffical job action (aka not a strike) - about 1/3 of the flight crew personell called in sick, or did not show for work.

It was a very, very selfish thing to do - stranding thousands of people on Christmas to complain about pay cuts. Will it be effective? Time will tell...

Re:Let's not be too hard..

[afidel]

They HAD outgrown their current system, and they knew it. That's why the new system was scheduled to go online in the next couple months. Unfortunatly they met with a perfect storm of problems just at the wrong time. If you've ever worked with retail you know that NOTHING gets changed from mid November to early January unless god and the CEO both say it has to be so, I imagine airlines are pretty much the same. Heck airlines probably have an even larger freeze window since few people book flights at the last minute for holiday travel.

Re:Let's not be too hard..

[SpaceLifeForm]

In defense of the crew members, the snow was very heavy, roads were jammed. Some (many) would never have even gotten to the airport in time for the flight. Some people were stuck in traffic for 24 hours (on highways), barely moving.

Re:Let's not be too hard..

[Money for Nothin']

It was a very, very selfish thing to do

Oh boo hoo, the airline workers were selfish -- as if that's a bad thing.

As a businessman, if you want people to work when you demand them to work, then pay them a better rate than they can get elsewhere. If you don't agree to the employee's demands for time off, then fire him and hire another, more cooperative person.

Of course the employees are selfish, just as the businessman who hired them is selfish. All people -- even the most community-minded people -- are selfish, whether they care to admit it or not, and that is a *good* thing. Welcome to the human race.

How old was this software?

[wiredog]

I stopped using 16 bit ints for anything 10 (or more) years ago when I had the joy of migrating systems from a 16 bit OS to a 32 bit OS.

Re:How old was this software?

[adlaiff6]

The article said it was 15 years old. I guess 16-bit systems are really named for their expiration date.

Re:How old was this software?

[Bigby]

I still use shorts and unsigned shorts (and for measure, chars and unsigned chars) for 'for' loops that are guaranteed to be small. There's no reason to take up 2 extra bytes of memory.

Re:How old was this software?

[imsabbel]

of course your compiler will just say "fuck you" and padd them to 32 or 64 bit... So nothing saved and potential errors added...

Re:How old was this software?

[tsmithnj]

Well, if that's true, then you don't use Acrobat to read PDFs. Adobe uses a 16 bit counter to name temporary work files. I know this b/c I ended up with 32K files in "My Documents\User Data" and Acrobat stopped working. My point is, I suspect this impacts our lives in more ways than we know.

Re:How old was this software?

[Detritus]

That depends on the compiler. There are still plenty of programmers who use compilers that support 16-bit integers. Sometimes that's all that's available, or there are time/space reasons to select 16-bit code generation.

Re:How old was this software?

[imsabbel]

Im not a compiler programmer, but IIRC even if the compiler supports 16bit integers, every memory object will be padded to a 32bit offset just to provent page breaks with any kind of optimisation enabled, so unless you create arrays of those shorts, you wont safe space.

Re:How old was this software?

[Detritus]

That depends on the compiler, the target and its options. I've programmed systems that ran faster with 16-bit code because of slow busses and memory. You have to approach it on a case-by-case basis sometimes. Know the requirements, run some test cases and pick the best options. Sometimes, due to external factors, you don't have a choice.

So after Y2K is this ...

[adzoox]

what Initech handles?

Yeahhhhhh! Mmmmmmkay!

Did you get that memo?

Re:So after Y2K is this ...

[bje2]

wouldn't "Initrode" handle that stuff now?

Re:So after Y2K is this ...

[justkarl]

You mean, "Penetrode"?

Re:So after Y2K is this ...

[bje2]

actually, both Initrode and Penetrode were competitors to Initech...i believe Micahel & Samir went to work for Initrode at the end, right?

Re:So after Y2K is this ...

[Carbonite]

I believe "Penetrode" was Michael's derogatory name for Initrode. I don't think they were separate companies. This is a good excuse for me to watch the movie again and confirm.

Re:So after Y2K is this ...

[justkarl]

You're correct, but it wasn't a derragatory remark, just a mistake. But it is a dirty joke.
Get it? Penetrate-Penetrode-Initrode? Ah, nevermind.

Re:So after Y2K is this ...

[shyster]

I always thought Penitrode was a reference to a Initrode being like a cubicle penitentiary.

Re:Comair?

[bje2]

just RTFA linked in the summary ("conair system crash")...

Re:Maybe it had "worked just fine" for them?

[kirun]

It's interesting because it provides a lesson in software design - arbitary limits will trip you up eventually. It's not as if nobody knew to avoid them before, though.

Re:Comair?

[slapout]

The human slashdot editors where replaced long ago. I think it's some google news beta program that currently posts the stories.

Re:Maybe it had "worked just fine" for them?

[jedidiah]

This assumes that they had the resources. Given the current competitive environment in terms of consumer price and fuel costs, it would not be surprising if IT got the short end of things.

Re:Signed or unsigned

[Evangelion]

Since 2^16 = 65536, I'm guessing signed.

Hilarious

[October_30th]

"Perhaps Comair should have paid for the software upgrade to MaestroCrew." (in the Simpsons Comic Book Guy voice)

Wasn't it Nic Cage?

[AtariAmarok]

I thought the Comair crashed when Nicholas Cage steered it into the window of a Las Vegas casino.

Re:Wasn't it Nic Cage?

[Zorilla]

I knew Delta should have left the bunny alone!

Let's try to remember

[CodeWanker]

That when you are talking about an airline, a COMPUTER crash is by far the least traumatic kind you can have.

Re:Let's try to remember

[thetroll123]

COMPUTER crash is by far the least traumatic kind

What about two of those little baggage carts crashing in the arrivals area? Surely that's even less traumatic?

Re:Let's try to remember

[joemc91]

I'd say the computer crash is worse for the airline. It gives the passengers the feeling that Comair doesn't know what they're doing. Passengers will avoid flying on them (Delta) because of the inconvenience they have caused. Besides this, a physical crash may cost less money than grounding all flights for a day. Since the cancelations were due to something within the airline's control, the Federal Aviation Regs say that the airline must compensate each and every passenger accordingly, such as a free flight somewhere, money, and at least room and board for the night.

Just consider the much money American Airlines loses when they crash an airplane and then consider how much they lost during the flight attendant strike. The strike almost killed the airline but the crash merely lost a plane.

Re:Let's try to remember

[HaloZero]

Not if my laptop (bagged or unbagged; ever seen The Italian Job?) is in one of them. Then we're going to have words, and not a single one is going to be happy.

Re:Let's try to remember

[CodeWanker]

Uh, no. Let's stay with your approach and pretend human life has no value when we approach this calculation. The airline ValuJet was destroyed by a single plane crash in the Florida Everglades. And it was a crash caused by cargo improperly marked my a third party... The airline vanished (vanished into AirTran.)

And I don't think Comair's computer crash has a body count associated with it, let alone 110 people killed and turned into alligator poop. But in your universe, that apparently is trivial. Except for the $30,000,000 million dollar lawsuit outcome against the airline. So maybe that shows up on your radar.

Re:Let's try to remember

[joemc91]

True, the cost of a crash is also associated with what the cause was. Accidents in which it looks like negligence cause a lot more damage to the airline than a crash caused by some crazy circumstance (like the United Souix City crash). Some airlines also manage to weather crashes better than others. Somehow American has managed to not get hurt by their accidents whereas other airlines such as ValuJet are destroyed because of just one accident that wasn't even their fault. I'm just trying to think about it in the way the airline bean counters will about this.

Re:Let's try to remember

[DrEldarion]

Yes, it's good to make this distinction. I was a little disappointed that the highest-rated comments were people making jokes until I realized that.

Re:Let's try to remember

[Detritus]

It wasn't ValuJet's fault?

I'd argue that it was their fault. ValuJet had the final and ultimate responsibility for the safe operation of their aircraft. If they outsource part of their business, they are still responsible for selecting a qualified vendor and making sure that the job was done right. Too many companies attempt to evade accountability for their actions by blaming their contractors.

Re:Let's try to remember

[joemc91]

SabreTech was considered qualified before the accident. As a maintenance shop, they had to be certified the FAA as well as approved to do the maintenance for ValuJet by ValuJet's POI (an FAA employee in charge of making sure the airline complies with regulation). While ValuJet is responsible for making sure SabreTech does it's job, they can not check every maintenance item, nor are they supposed to. This is like the FAA making sure the airline complies with it's operating specifications but the FAA doesn't watch every single thing going on. It's impossible. The crash was caused by oxygen generators placed onboard without notice to the crew, which was an illegal action by the single employee of SabreTech who packed the container. There was also some fault placed with the airplane manufacturer for not having any way to detect a fire in that part of the cargo hold. In any case, according to the NTSB all parties involved, SabreTech, ValuJet, and the FAA were faulted. I worded my statement badly and you are right about it also being ValuJet's fault. This accident is why AirTran and many other airlines now do all their maintenance in house. NTSB report

Re:Let's try to remember

[lachlan76]

Modern planes are mostly controlled by a computer nowdays...there would be a backup system (I hope, anyway), but most of the work is done by a computer.

Re:Comair?

[buckeyeguy]

Potential trolls aside, Comair is a regional air carrier, based at the Greater Cinci airport, that was bought up by Delta, and turned into their secondary route provider. They handle both short and medium-range non-stop flights (i.e. Ohio to Atlanta or Orlando). So it's more closely-related than the code-sharing arrangement that some carriers have.

Now my question would be, since they're owned by Delta, why wouldn't Comair flights be handled within Delta's own reservation/flight tracking system?

p.s. I've traveled through CVG, on Delta, during the holidays. Not anymore... One weather-delayed flight and the whole system falls apart.

Heard it here first?

[Eudaemonic Pie]

Heard it here first? Sorry, I heard it first on the web/digest version of comp.risks... http://catless.ncl.ac.uk/Risks Excellent place to read about the risks of modern computing equipment and the risks to society by using same usually from mistakes like the 16 bit counter.

65535+2 post

[bjb]

It looks like it was a signed 16-bit value from the detail in the article (32,767 maximum schedule changes a month). Why they would possibly need the sign is beyond me. Regardless, having 32767 schedule changes in a month? Must track every flight in the world.

Re:65535+2 post

[arkanes]

Hypothetical: There's some function that accepts a crew change and returns either the number of schedule changes to date or an error code. The error code is a negative value. This is a really common paradigm in C code.

Re:65535+2 post

[R.Caley]

Regardless, having 32767 schedule changes in a month? Must track every flight in the world.

The article says they had 1100 flights on one day. That's 34100 per month. So basicly they seem to have had on average about one crew schedule change per flight.

Now, that is a very bad failure rate, but given that any one crew change probably causes a mass of knock-on changes (Fred misses this flight, so you have to substitute John, and then someone has to take over what John should be doing for the rest of the day, and Fred won't be on the right place to do what he was supposed to do this afternoon, and John has reached his flight-hours limit so you have to get Harry in early and...), a good flu epedemic going around the fleet plus some bad weather delays and technical faults all coming in the same month would probably do it.

Re:65535+2 post

[YrWrstNtmr]

Regardless, having 32767 schedule changes in a month? Must track every flight in the world.

Each weather affected change could trigger possibly dozens of schedule changes in the system

The aircraft itself is out of place. That also affects the a/c it replaced, and the one it is replacing. The crew is now also out of position for their next flight(s). Maybe 10 potential changes after each flight. Pilot goes one way, copilot goes another, the stews disperse to other a/c altogether.

Everything is tracked on the schedule. A/C, aircrew, cargo, fuel. A couple of dozen changes in their system for each weather affected flight would not be out of the realm of possibility.

Re:65535+2 post

[tdemark]

If you are dealing with a counter, why would you waste HALF of value space for a few error codes? Instead, make (INT_MAX - x) to INT_MAX your error space.

With a few #defines, you wouldn't have to make any major changes to the code logic:

if (rv = ERROR_MIN)

- Tony

Re:65535+2 post

[jacksonj04]

I think this should have a mod as insightful - it's the only post I've seen which vaguely explains why relatively few problems can quickly escalate - and bad weather/illness across lots of flights can play hell with crew scheduling.

Re:65535+2 post

[arkanes]

Because it's more complicated. People do that too, but the negative error code is more common.

Re:65535+2 post

[Moofie]

Three crewmembers/flight, 1500 flights/day...

That's not a lot of overhead when you have a huge weather event.

Re:Maybe it had "worked just fine" for them?

[barzok]

Not surprising? I've come to expect it. If a department isn't actively making money, they're considered an expense. Management is usually too shortsighted to notice that without IT, the rest of the departments can't make money (or can't make as much).

Damn you 2s Complement!

[jellomizer]

It could have worked if it wern't for the 2s complement they would be good for twice what they had. I think programming languages should make numbers unsigned unless asked that way we can take advantage of that extra bit. For things like counters where negitive numbers just wont happen is like having a 15bit number taking 16bits of space.

Re:Damn you 2s Complement!

[ThogScully]

You could also suggest that changing the default behaviour would be a confusing matter. And programmers worth their salt will probably specify signed or unsigned if it's going to matter or if there's an obviously better choice.
-N

Re:Damn you 2s Complement!

[rat_love_cat]

Having fixed point numbers default to unsigned is not a good idea because, at least with C's unsigned rules, it's easy to end up with a huge number if a negative number is ever generated. This has bitten me enough times in C that I avoid unsigned unless I'm damn sure that it can never go negative (and even then I check all subtractions real carefully).

Re:Damn you 2s Complement!

[jridley]

This really bugs me too. Heck, why is "char" default signed, even when used in a string? How many times do you want SIGNED when you say char* ?

Yet if you try to use unsigned char*, you have trouble because so much code uses the default signed that you wind up having to do a ton of useless typecasting. It's easier just to live with the useless signage.

Re:Damn you 2s Complement!

[multipartmixed]

> This has bitten me enough times in C that I avoid unsigned unless I'm damn
> sure that it can never go negative (and even then I check all subtractions real carefully).

Yeah, tell me about it. Ever tried to malloc -1 bytes?

Re:Damn you 2s Complement!

[multipartmixed]

> Heck, why is "char" default signed, even when used in a string?

Char is just a really short int. A char * is no different than an int *, except for the dereferenced datatype and the multiplicand for pointer arithmetic.

Why would one integer type have different rules than another? Should the rules change based on CONTEXT? How is the compiler supposed to know that you're considering that hunk of memory to be a string?

Hint: If you want that type of behaviour, you should be using a STRING CLASS, not an array of eight bit integers!

If you want really messed-up, notice the type of the argument to the ctype functions -- it's int!

Re:Damn you 2s Complement!

[pclminion]

char is not always default signed. The C language does not specify. On some compilers it defaults to signed, on others to unsigned. On GCC, you can switch between the two with the -fsigned-char and -funsigned-char flags.

Re:Damn you 2s Complement!

[spitzak]

char defaults to signed mostly because the PDP11 did sign extension when reading 8-bit data into a 16-bit register.

Later machines acted differently, which led to the decision to say the default for char is compiler-dependent. Certainly true of the 8080, perhaps true of the 8086?

Eventually machines became powerful enough that they provided two instructions to load 8-bit data so the compiler could choose to act either way with no speed loss. Although initially the majority seemed to be unsigned due to the 8080, in this case official standards prevailed over popularity. The fact that char was signed in the first C compilers, plus the consistency of having all integer types default to signed, pushed the C standards to define signed as the default.

Personally I have never wanted a char to be signed and would be perfectly happy with unsigned.

I suspect i18n and UTF-8 would be much further along if char defaulted to unsigned. Because of stupid code that did things like c=' ' to test for whitespace, it was easier in many cases to switch programs to "wide characters". Wide characters have been the biggest impediment to i18n because they prevented solutions that are truly compatable with communication and system API's, resulting in a horrible mess that we will probably be still cleaning up 30 years from now. The proper solution is UTF-8.

Re:Damn you 2s Complement!

[corngrower]

I too have been bitten by this bug too often to casually use unsigned. The problem is that one tends to forget about needing to be extremely careful when doing calculations with unsigned. I almost always use signed numbers, even when their value is always supposed to be a nonnegative number.

A good rule of thumb: don't use unsigned numbers unless you know you need that extra bit and you've double checked to make sure there are no negative intermediate results in your calculations, and have adequately commented code to warn others of the use of unsigned variables so they are aware of the potential for this problem if they need to make changes to the code.

It's usually better just to use a long instead of an unsigned int or an int instead of an unsigned short.

Re:Damn you 2s Complement!

[42forty-two42]

Yes, it retuned NULL. Usually malloc won't be able to obtain the allocation from the OS and will fail gracefully.

Re:Damn you 2s Complement!

[ArtStone]

Why are people assuming that software written in 1989 was written in C? Or that memory prices for a PC in 1989 are releveant? (mainframe memory was orders of magnitude more expensive)

The computers I worked on in that era (Tandem) used a language (TAL) that did not have an unsigned data type... that's because the underlying hardware instruction set in earlier chips did not support unsigned integers. The only way to actually support a 16 bit unsigned integer would have been to either add instructions in software around every arithemetic opcode to check for potential overflow conditions... cpu cycles were also much more finite and expensive in 1989...

Even if you solved that problem, you could not use that unsigned integer in a key in the database, because it would consider 32769 to be smaller than 32767, and all your database searches would fail...

I"m not implying that the software ran on Tandem (which was based on HP-3000 technology), but rather that people open up their minds to realize that there is a lot of software out there not written in C.

Don't mod me up. Mod up the Zucker brothers.

[AtariAmarok]

Can't help but remember the scene in one of the "Airplane" movies where a kid sneaks into the hi-tech air traffic control room. He sees one of the airline's shuttle-like planes on a screen, and grabs the nearest joystick and begins to (he thinks) play a videogame.

When the shuttle on the screen blows up, and is accompanied by a very loud explosion sound outside the building, the kid looks sheepish and sneaks away.

Re:Don't mod me up. Mod up the Zucker brothers.

[Skater]

Actually, that was Airplane II, and the Zucker brothers weren't involved - they felt they'd done all the airplane jokes they could in the first one.

Great scene though. IIRC, they used the Atari 2600's Yars Revenge for the sound effects.

--RJ

Re:Don't mod me up. Mod up the Zucker brothers.

[AtariAmarok]

"and the Zucker brothers weren't involved "

Mod 'em up anyway. Just because. "Airplane II" had so many great gags that anyone should be proud to have their name on it:

the automatic doors that go SHH when you go SHH at them.

talking to Shatner on the viewscreen which turns out to be a window in a door

keeping with current trends, elevator music in the near future will be deafening.

the aforementioned kid videogame scene.

Re:Don't mod me up. Mod up the Zucker brothers.

[Skater]

"Danger: Vacuum"

"I've pulled Striker's record." "And?" "You're not going to like it, sir." [Shows copy of "Ted Striker's 400 Polka Favorites"] "My God, it's worse than I thought."

The scene where the guard shot Scraps the dog: "Just kidding! Blanks! See, Scraps is fine!"

I wonder if the movie wasn't better because someone else did it - sort of a fresh take on the gags.

--RJ

Error checking is the real culprit

[Anonymous Coward]

So it turned out to be problematic to use a signed 16-bit integer.

But the real problem is a lack of error checking. It sounds like the code had something like:

int num_crew_changes; ...
crew_change_list[++num_crew_changes] = blah;

And the counter wrapped and the system crashed.

The code should have said:

if (num_crew_changes == MAXINT)
{
ERROR(E1234, "too many crew changes");
}

The system is still degraded after 32767 crew changes. It might be so degraded as to be unusable. But at least the company would know the extent of the degradation and could pull out the appropriate "Plan B". It's much safer and better to work around a known problem of known scope than to work around a system crash when you don't know the exact problem.

Re:Error checking is the real culprit

[MadHungarian1917]

Although the application crashed that is not why Comair needed to cancel all their flights.

In aviation there is a concept of "Legality" which basically states that the FAA in cooperation with the ICAO www.icao.int has set the allowable hours a pilot or F/A can work in a day. Since this is a crew scheduling application the airline was no longer able to determine the flight status of their crews and by FAA regulations needed to stop flying until they could once again determine their crew(s) flight status.

The only plan B the FAA alows is flght cancellation unless the airline has an approved greaseboard process.

The int overflow was just the nail in Ben Franklin's
famous poem

For want of a nail a shoe was lost
for want of a shoe a horse was lost
for want of a horse a rider was lost
for want of a rider the battle was lost
for want of a battle the kingdom was lost
and all for the want of a horseshoe nail.
(or long counter)

Re:Error checking is the real culprit

[upsidedown_duck]

Your code example should go into next year's textbooks.

Re:Error checking is the real culprit

[RogerWilco]

I would prefer something like this

if(Num_Crew_Changes > (MAXINT - MAXINT/10))
{ //If we exceed MAXINT the behaviour of this counter is outside spec.
if(Num_Crew_Changes >= (MAXINT-2))
{ //We might exceed MAXINT this is outside spec.
Logbook->WriteError(Error_Large_Number_Of_Crew_Cha nges);
Error(Error_Large_Number_Of_Crew_Changes);
}
else //Too warn users that we are approaching limits
Logbook->WriteWarning(Warning_Large_Number_Of_Crew _Changes);
}

It's times like this...

[AtariAmarok]

It's times like this when you begin to realize that the Vic-20 (duct-taped to the bulletin board and surrounded by haywires) might not be the best choice anymore as mission-critical hub of your operations.

Re:It's times like this...

[tomstdenis]

Oh but didn't you hear? The airline industry is bankrupt. ... Bankrupt to the CEO's million dollar tropical mansion that is.

If my seat literally costs $300 and they sell it to me for $225 that's just stupid. Even if I only fly that airline it still COSTS THEM MONEY to fly me. ...

So why don't airlines just charge what it actually costs to fly. The others who don't will die anyways because well they're losing money with each "heart, mind and body" they "win" with lower prices.

As for this particular bug it's a matter of being cheap. They could spend the time and money to fix it [say like a decade ago] but instead they wait until it causes trouble and then spend 10x to fix it in a shorter amount of time.

Go people with MBAs!!!

Teh Winner is you!

Tom

Re:It's times like this...

[tomstdenis]

Ok dude do the math.

A sells tickets for $0 loss.
B sells tickets for $75 loss.

B gains many customers. However, the more customers the more loss they incur. Recall EVERY SEAT costs them $75. Eventually B just runs out of money and ups the costs.

Now A and B sell at the same cost. Customers notice the price hike and get upset [because for some reason people think air travel is a god given right so they get insanely upset at everything].

Sure some won-over customers will stay with B but many will spread out [many are also not particularly loyal they just use whatever cheaptickets.com tells them to].

Tell me I'm wrong. Tell me that most airlines haven't been filing for protection. Come on, tell me ;-)

Tom

Re:It's times like this...

[afidel]

Because if they DON'T offer some discounted fairs to fill the seats then they have a HUGE fixed cost being spread over even fewer full fare customers. Basically your discount butt pays for the interest on the money it costs to lease the plane, and the only chance of making money they have is business class customers paying full fare. Unfortunatly for the airlines there aren't as many businesses willing to pay full fare, they would rather allow flexibility in their employees schedule than pay the full fare last minute price most of the time. Add to that the short haul carriers with reduced overhead and lower cost due to less senior pilots and reduced (until this season) overall passenger flights and you've got a hurting industry. Of course the goverment is really, really hurting the industry by proping up the old failed airlines rather than allowing business darwanism to get rid of the failed models. Government subsidised competitiors which can afford to lose money because they know that the taxpayers will most likely bail them out are NOT a good thing for the industry.

Re:It's times like this...

[tomstdenis]

A loss is still a loss though. I mean if you're net income is -2% of your remaining assets that may not seem "huge" but the implications are. You're not making any money.

The simple math of the situation is if it costs them money for me to sit in the seat then they're not on the road to recovery.

How can you honestly tell your investors "you're doing all you can to not go for broke" when you sell seats at a loss?

Tom

Re:It's times like this...

[tomstdenis]

I agree with your darwinism issue.

The only reason why the airlines are so popular is because they made it relatively cheap. It's just not the reality of the situation.

It's a business that truly fills a niche [e.g. travel by air as a last resort] that it should be at least for now. I never shed a tear for businesses that assume it's their right to make billions in profits. This is the same thing as the movie/music industry bitching when people stop shilling over the dough for their latest "product".

If a seat costs $300 sell it for at least $300. Anything less is just asking for trouble. If you can't afford then to have 1200 planes in your fleet maybe the "demand" for your service isn't as big as you think.

Imagine if Hersey stocked local corner stores with 2000Kg of product a day because they had a wet dream of selling that much product. Then because they weren't selling they lowered the price from the $1 per bar to below their costs [say it costs $0.30 per bar] at say $0.15 per bar.

Would that sound smart?

Tom

Re:It's times like this...

[RollingThunder]

They sell it to you for $225 because that's $225 LESS that they will lose on that flight.

Airplanes don't expand and contract to fit the number of people on them. That seat will fly empty and cost $300 to fly from one end to the other no matter if you're in it or not (yes, I know there's some cost savings by not processing you or your baggage, but the biggest cost is flying the plane's mass).

It's the next-to-last minute seats and business class travellers that make the real money. All the rest is just trying to squeeze the most money out of the flight.

Re:It's times like this...

[phurley]

But if the airplane has a (more or less) fixed cost to fly, then even if they sell your ticket at a loss, unless they were going to cancel the flight -- they are still loosing less money than they would have.

Using totally fictious numbers:

Cost of flying empty plane: $10000
Each passenger incurrs $10 additional cost (fuel, food, etc)
Seats 100 -- true cost per seat if full: $110
Estimated utilization 50% -- cost per seat $210

Now if the at some point selling seats (even for less then $110, as long as it is greater than $10) will earn the airline additional revenues (or at at least lesser losses) -- __as_long_as__ they do not steal away higher paying customers.

Re:It's times like this...

[Bastian]

I think what the parent is suggesting is that if you have a choice between having your net income be -2% of your remaining assets and having it be -1% of your remaining assets, you damn well better take the -1% option.

How can you honestly tell your investors, "We've hit some hard times so we're just going to give the fuck up. So long, and thanks for all the cash" when you could have sold your seats at a loss and tried to bring the company back?

Re:It's times like this...

[ch-chuck]

Bankrupt to the CEO's million dollar tropical mansion that is.

It happens all the time - some group buys a business w/ leveraged funding, extracts a fortune for themselves while running the business into the ground.

Re:It's times like this...

[DavidTC]

Sure, they lose money on each sale, but they'll make up for it with volume.

Wait a minute....

Re:It's times like this...

[RollingThunder]

That's very true. They want the extra capacity and frequent routes to be a "better choice" for the traveller.

Re:It's times like this...

[tomstdenis]

The real problem is they're trying to supply for an artificial demand. People only fly [over train or drive] because it's artificially cheap [e.g. not true cost].

If they kept plane tickets at the true cost people would opt for other means of travel. They would then "service less people" but also "not lose money with every flight".

But where's the fun in that. I mean americana! gotta just build big, not sufficient.

Tom

Re:It's times like this...

[recursiv]

Hello? Anyone there?

The airline does not save money by refusing to sell under cost tickets. Do you think they might conserve a little fuel because of less weight being on the plane?

Genius, go play lemonade stand until you figure out what's really going on.

Re:It's times like this...

[tomstdenis]

They would stop losing money by trying to supply an artificial demand.

Sure I'd love to fly to England for $50. Doesn't mean the airliners should offer it. It's irresponsible to their share holders.

The only reason people fly more than they really need is because it's really cheap. To them though. The airliners are losing money to gain "more clients".

What if Sony sold the PSP for $7 just because they wanted to "sell a lot". Are we then to cry for them when they post net revenue losses of billions?

Tom

Re:It's times like this...

[Teancum]

The problem with this sort of thinking is that you are deliberately trying to gain market share on the presumption that once that market share is established, you can recover your money afterward with higher prices.

John D. Rockefeller did this very effectively with the Standard Oil Corporation, by eliminating all of his competition, and through reducing costs by scales of operation and improved manufacturing techniques, he was able keep the prices low afterward, and even make some money. Quite a bit of it.

Every entrepreneur since Rockefeller has tried to duplicate his success, and it was people who cited him that created the internet bubble. Note the similarities between the wildcat days of the oil industry and internet companies during the 1990's.

A problem with trying that strategy with the airline industry is that profit margins are paper thin, and it appears as though barriers to entry are not as tough as it would appear at first glance. Indeed, most of the reason why traditional major airlines are getting into trouble is because there are new airlines that are pushing prices down (JetBlue, Southwest, and more), and everybody has to buy the airplanes themsevles from the same suppliers (Boeing and Airbus), so there really isn't that much differentiation between each airline as there may have been in the past. That is why people like baggage handlers are getting attention right now, because it is with doing that sort of activity that can make the difference between one airline vs. another one.

Re:It's times like this...

[drew]

If a seat costs $300 sell it for at least $300

the problem the parent tried to point out is that there is not a fixed per seat cost- the fixed cost is per-flight. look at it this way: it doesn't cost $300 per person to fly a 300 seat jet from new york to chicago, rather it costs $75000 to fly the jet, whether it's full or empty. (or, if you want to be picky, it's probably more like $68000 + $30 per person (yes, i am completely pulling these numbers out my a**, but you should be able to get the general idea...))

now, if they can only fill up 25 seats, by your logic each seat would cost $3000. of course if they charge $3000 per seat, then they probably wouldn't even get 25 passengers, which means the price would be even higher. on the other hand, if they only charge $300 for the seat, then they only need to be able to get 250 people on the plane to cover their costs. any thing beyond that is extra profit. and if they fall short- say they only get 200 passengers due to a slow weekend or higher than expected competition- then they are still better off than if they charged $3000 per seat and only got 15 passengers.

basically, the airlines are playing a tricky game- they know how much it will cost to fly the airplane, and they have to guess at what price they can sell their tickets to maximize the revenue generated by ticket sales. your logic only works if the airlines can assume they will sell the same number of tickets no matter what they set the price at, something that is obviously not true. (if you've ever taken an economics class, go refresh your money on elastic vs. inelastic goods.)

all that said, i also wish the government would stop subsidizing the failed airlines to the extent that they are. if there are airlines out there that can operate and make a profit, there is no need to artificially prop up those that can't. and if some of the airlines go out of business because they are no longer subsidized, maybe that would give a little more business to amtrak for shorter distance travelling, and the government could stop subsidizing amtrak to the extent that they are as well....

Re:It's times like this...

[bstone]

B gains many customers. However, the more customers the more loss they incur. Recall EVERY SEAT costs them $75. Eventually B just runs out of money and ups the costs.

Remember, this is an airline we're talking about. Try the math this way:

A and B have 100 seats per plane and the trip costs $30,000 to fly (that's $300/seat).

A sells tickets at $300 for a $0 loss on a full plane.
B sells tickets at $225 for a $75 loss per passenger on a full plane.

A sells zero tickets because all 100 passengers that day buy B.
B sells 100 tickets.

A loses $30,000.
B loses $7,500.

A runs out of cash quicker than B and goes bankrupt.
B ups their fares to $500.

B now makes $20,000 on each flight.

Re:It's times like this...

[tomstdenis]

So you're saying the solution is to lower the price so they can recoup costs.

My point is the **real** solution is to

get this, this is a doozy

====> **** NOT FLY THE FUCKING PLANE IN THE FIRST PLACE **** <====

If you're over supplying the true demand then you're always going to waste money. Don't make 90 million gizmos when there is only demand for 1 million gizmos.

The demand for air travel only surged when discount rates appeared. Discount rates only appeared to fill seats [re: artificial demand].

Essentially if you're flight is an emergency or longer than say 3-4 hours than it's a "need". People used to go by bus and train remember? You could take a one hour flight to the next city or spend a day on a bus. When flights were 400-500 dollars a seat people used to take the bus. Now that the same flight may cost 150 or so it's more attractive to fly.

But just because you CAN do something doesn't mean you SHOULD.

I mean they COULD sell the tickets for a buck each. That would fill up planes quick too. By your logic is that a good idea?

Tom

Re:It's times like this...

[tomstdenis]

Um I don't know where you're from but local transpo [e.g. busses] GO UP every quarter not down. ... This is even with the fact that local transpo is government sponsored.

Used to be $0.65 for a bus ticket [few years back] now it's $0.90 [two tickets to ride the non-express busses]. Which makes sense. Cost of living goes up, cost of bus goes up.

Similarly long-haul buses aren't that cheap. From Ottawa to Windsor is about 160$ [round trip]. How much does it really cost to load up a bus of 45 people and drive for 20 hours? The bus driver is making say 20$/hr which is 400$ then you have say 200$ in gas and 200$ in repairs [on average per trip].

That's a total cost of $800-$1000 per round trip. They rake in about $7200 even if half the people are on discount rates they're still making decent profits. In fact they'd have to sell the seats for less than $22 [or so] to "lose money" on the trip.

I've seen greyhound trips specials for $50 [roundtrip] which is still twice the minimum required to break even.

Tom

Re:It's times like this...

[sjames]

However, the more customers the more loss they incur. Recall EVERY SEAT costs them $75. Eventually B just runs out of money and ups the costs.

That'snot how it works. All of the seats on a flight are effectively a sunk cost once the first ticket sells since they can't cut the plane in half. At that point, $200 is better than $0 for the seat. The challenge is to make sure the AVERAGE ticket is profitable.

The real problem is that passenger air travel apparently costs more to provide than it provides in value to the consumer. In part, the very cost cutting measures over the years have contributed to it's devaluation. They have crammed the planes so full of seats that air travel is quite uncomfortable for many, so people look a little harder at alternatives. Other factors are out of the airline's control such as the way airport security and terrorism paranoia turns people off.

People go with whatever's cheap because for the most part, no matter who they fly with, they'll be poked, prodded, and generally treated like criminals reporting to prison, then be stacked up like cordwood for a few hours. If you really want to charge more than the rock bottom price, you have to provide something your competition doesn't or do whatever you do better than the competition.

Re:It's times like this...

[Dun Malg]

A runs out of cash quicker than B and goes bankrupt.
B ups their fares to $500.

B now makes $20,000 on each flight.

Except that it's more like:

A runs out of cash first, but gets bailed out at taxpayer expense B never gets monopoly position and must keep fares low

A & B both run out of money, but manage to subsist on gov't subsidies

we're gonna end up with another Amtrak situation

Re:It's times like this...

[Sloppy]

A sells tickets for $0 loss.
B sells tickets for $75 loss.

B's choice is:
B sells tickets for $75 loss.
B's plane flies with an empty seat, at $300 loss.

So instead of flying with a totally empty plane at a huge loss ($300 * number of seats), they fill a plane with only-$75-loss-per-passenger seats, thereby making it up in volume. This is the elusive stage 2. Then comes stage 3: Profit!!!

Re:It's times like this...

[rsidd]

I mean they COULD sell the tickets for a buck each.

Like Ryanair, you mean? One of the more profitable airlines in existence.

Why do I get the feeling you don't have a clue about how airlines work, what the costs are, where the revenues come from?

Re:It's times like this...

[tomstdenis]

Even assuming the trip cost the company $2000 [which is possible but I doubt it]. At ~45 people [and most buses are packed from what I see] paying say $50-$75 each [promo prices, I paid more cuz I didn't fit in one their promos which actually I expect many wouldn't] That's still roughly $800 profit per trip. Given that 800 > 0 I'd say it's a good business model.

Though a quick google seems to indicate they're doing fairly sour recently. Could be that everyone is using planes instead of buses.

So congrats airlines CEOs, not only are your businesses going south but you're pulling others with you!

Yeah humanity!

Re:It's times like this...

[drew]

actually, i agree with everything you say in this post. (note when i commented in my previous post that if the government stopped supporting airlines to the extent that they did, they could probably stop subsidizing amtrak and other rail passenger services to the extent that they do as well.)

my only objection was to your original statement that the airlines could simplify things by just charging what it costs to fill the seat. there are a lot of variables in airline flight scheduling, requiring airlines to make estimates about what flights will be profitable or sustainable months or even years in advance. likewise, setting the prices requires making a lot of estimates far in advance, and most of those estimates are unable to be verified until after the plan actually flies (at which point it is too late to change the price)

airlines that estimate well will usually be profitable. airlines that do poorly will not, and should go out of business or be taken over by somebody else. (unfortunately, they do not, as the government has them solidly propped up, as you and i have both pointed out) many airlines are able to be profitable, even offering discount fares. so the problem is not that all airlines are discounting their prices too much, it's just that some airlines are not doing a good job of estimating demand and providing supply accordingly. whether they are unable to do this, or just don't care because they know the friendly government will keep them going is a discusion for another time.

New CIO?

[Mr. BS]

I wonder how fast this CIO is going to be on his butt.

"Well... we were holistically mitigating our financial stance outside the box of current processes while try to forcast our future technological stability within the transport industry."

"Well... you're fired! NEXT?!

Re:New CIO?

[mslinux]

I wonder how fast this CIO is going to be on his butt.

Probably never. Our CIO is an idiot when it comes to technology. He has a law degree from a big college. He earns six figures for sitting in his office and trading stocks (his stocks) all day. I'll never forget the day he picked up a WordPerfect Office11 box looked at it and then said, "So Tom... who is it that makes Word?"

These guys are dumber than dirt, but they're well-connected in the "good ole boy club"

Re:New CIO?

[ncmathsadist]

Nah, they'll ship this guy out onto the street with millions in his pocket. Remember, companies are run for the care and feeding of their managements.

Re:New CIO?

[prisoner]

His name wan't Steve is it? The trading stocks part sounds just like him. He sat in one of those glass-walled offices like on "24" and essentially never came out. When we were called in it was pretty clear he was day-trading non-stop as you could see the Datek streamer on his screen and about 10 browser windows minimized.

Best war-story of my career: He came back from a Java convention and informed me that instead of continuing to use our Windows NT server (this was awhile ago) to run our small business that we were going to buy a Sun Ultra 400 or something. It was about the size of a trash compactor and cost like $75k. In addition, we were going to use Oracle.

It was here that I pointed out that we only had 12 people working per shift and wouldn't spending over $150k (hardware/software, etc) be a bit over the top? He didn't respond favorably. I got torn a new ass for not seeing the big picture and understanding the strategic plan. I was "thinking above my pay grade."

8 months later, the first paychecks started to bounce. At about the same time it was revealed that the President was banging this hot blonde sales manager (for whom he had purchased a new Jag). I heard he got coked up in his car in a downtown parking lot and the cops caught him in there with her. His wife wasn't happy.

Re:98

[Anonymous Coward]

Ah, Slashdot. Where even after the true culprit is unveiled, the problem somehow continues to be on Microsoft's end.

I'm not sure how the open source movement can pride itself with the quality of imbecile fanboys it attracts.

Once did IT support for Comair

[Anonymous Coward]

Having once done tech support for the Maestro program used by Comair (and other scheduling software for other airlines as well), I think the software is junk. The employees undoubtedly said "I told you so!" when it broke, because they hated it as much as the support team did. IMO the airline didn't bother upgrading because they didn't think the old version was broken enough or outdated enough to warrant it.

Re:Once did IT support for Comair

[dbacher]

Ran into this a lot with ATM industry as well, people wouldn't upgrade as long as there was the perception of the software working, even if the upgrade was for free.

unsigned

[lophophore]

Hmmm.

why would anybody make ae event counter a signed value?

short numberScheduleChanges;

hello?

unsigned short numberScheduleChanges;

fixes the problem.

Re:unsigned

[rjstanford]

unsigned short numberScheduleChanges;

fixes the problem.

You do realize that you've just fallen into the same trap, right? That doesn't fix the problem worth a damn. I mean, sure it doubles the amount of changes. And yes, 64,000 should be enough. But, hey, 32,000 should have been enough too, right?

Programs have internal limits. That's kosher. What's not appropriate is allowing the user base to exceed them or - for something like this - come close to exceeding them, without giving some kind of warning that notifies people of an impending problem and provides possible solutions (purge data, etc). Now you may point out that adding that kind of security increases the cost and complexity of software. Yup. That's why true enterprise software is expensive. Because that's what you're paying for.

Another alternative would have been for the software wrap and start purging existing records to make room for new ones. Either way, there should have been some defined strategy for the boundary condition, and there wasn't.

The other thing that the software vendor should have done when pushing their upgrade is point out that the previous version wouldn't allow flights to continue in that situation, but the new version expanded it to (some large number). Instead, they probably said, "We're 32 bit!" or something totally meaningless to the people evaluating the business case for the upgrade.

Re:unsigned

[hugesmile]

unsigned short numberScheduleChanges;

fixes the problem.


No, that delays the problem until it's a bigger storm.

Re:unsigned

[kryptkpr]

Programs have internal limits. That's kosher.

remalloc() .. wonderful function.

But seriously, I completely agree with your points. I don't know if going around overwriting old data is the correct way to go about it, but there should have at least been a check.. if you assume a max of 32k transactions, you'd sure as hell better check that assertion is true, and tell the user if it fails. Bonus points for telling the user if it ever comes close to failing.

But hey.. it's 15 year old highly-specialized software, I bet the code was a result of some late-night hacking session.

Re:unsigned

[lophophore]

Yeah. I know. There is still a limit. OTOH, it uses the same size (16 bits) so the size of the data remains the same....

Re:unsigned

[42forty-two42]

No, it makes it worse. Instead of writing to fooArray[-32768], screwing up a pointer somewhere, and crashing, you overwrite fooArray[0] and maybe lose some data. The *correct* solution is to use:

unsigned long long numberScheduleChanges;

And not have to worry about it for a few billion years.

Re:Maybe it had "worked just fine" for them?

[TopShelf]

According to the article, the system was on track to be replaced in the coming months...

That said, it's very true that many businesses get by "just fine" with existing, antiquated systems. Justifying system upgrades can be difficult from a conventional cost-benefit standpoint, when a large part of the benefit is based on preventing theoretical problems like this one.

Playing with fire

[ravingidiot]

Why was conair using signed shorts to track their scheduling changes anyway? It seems to me that a company of that magnitude should expect to run into more than 32000 schedule changes within one month more than once. I mean, I can understand that the counter was probably designed with space constraints in mind, but for christs sake, it would've only only been two extra bytes to fix this. That brings the total up to some 4 billion unsigned if I'm not mistaken. Technically, they could've used just three bytes, but then again, I wouldn't expect them to because how many languages have 24bit integers built in as primitives? Of course like someone else said, I guess we can't blame this all on the programmers either. I wouldn't just consider it very comforting that such a system could become crippled just because the programmers didn't think to allocate enough memory to allow for enough flexibility in scheduling.

Re:Playing with fire

[wiit_rabit]

As it has been said before, memory was much more expensive than now and those 2-3 bits per entry had a real cost (although using signed ints for a counter is puzzling). During normal operations the number of scheduling changes probably never approched 16K. The real question is how the number of schedule changes not be checked for overflow. Major Mistake. But, what was the software QA like 15 years ago?

Re:Playing with fire

[puhuri]

Probably the running number was used in records as on identifier too. With two bytes for 32k records that makes 64kB of extra memory. In 80s that was lot a memory.

Just finished optimizing two bytes off each record as I noticed that a struct had 2 extra bytes because of aligment. It does matter even today if you have more than 300 million records daily and need to store data indefinite amount of time.

Re:Playing with fire

[DavidTC]

It probably wasn't a signed int. It was probably unsigned...

...everywhere but one place.

Re:Playing with fire

[Duhavid]

Unsigned would give a range of 0 to 65535.

The sign was probably ( as indicated by other posters ) used to indicate an error. I.E. negative return value indicates error. But then, the other programmers probably cheerfully ignored the return value anyway.

Re:Playing with fire

[DavidTC]

Yeah. It's amazing how often everywhere but one place is correct.

Re:Playing with fire

[morcheeba]

Two things:
- They were probably not using short ints, but regular ints. In 1989, most compilers were 16 bits. It sounds like they just picked the default, especially because the better choice (unsigned int) wouldn't take any more room.
- C has always had 24 bit integers.

Re:Maybe it had "worked just fine" for them?

[Remlik]

bet *now* they'll upgrade, but until this particularly hairy situation arose, they didn't really see a need to upgrade a computer scheduling system that had been working great for them.

RTFA RTFA RTFA - The new system goes live in January. Good god its like herding cats around here.

Gotta love /. when you can get moded +5 insightful without RTFA AND posting verbal vomit....

Maestro sucks.

[Anonymous Coward]

Maybe Maestro should just die. My friend is a flight attendant for Southwest and has to use Maestro to plan her schedule. To use it she has to citrix into their main server and wait for an open client (I assume they have either a license or horrible programming restriction on concurrent users). On the very day that the new schedules are posted, it can take hours to log in. It's a joke.

This stuff could be handled by a team of a dozen web based programmers (Java? C? ASP? LAMP? You pick.) in a few months. It's not difficult.

Re:Maestro sucks.

[upsidedown_duck]

It's not difficult.

Famous last words. So many projects live and ultimately die by this philosophy that the collective trauma causes a mass amnesia that causes it all to happen again. The small software projects universe is absolutely amazing. Usually the downfall occurs when the developers realize that there are a thousand details in "scheduling software must accomodate partial work days" or "ad hoc scheduling to assist elderly and children must be supported." It's not difficult..."oh crap...Sally Bob quit last week but his code assumes...oh crap...I have to re-do this whole source file...oh double crap...".

Re:Hmmmm.... Maybe I should edit my software

[Jamesie]

I haven't used 16 bit integers since the early 90's, on a modern pc with the number of transactions you are talking about the change to 32 bit would do no harm and If you reuse the code to record greater numbers it will already be safer (unless your numbers are greater than a few billion).

Re:98

[Anonymous Coward]

The paperclip popped up and said, "It looks like you're writing a letter... you should use a 16 bit variable here!"

The other Maestro

[AtariAmarok]

When Mike Oldfield recently came out with his Maestro game, I did wonder that the "Maestro" name was not used for software before. Now I know better.

Not only that, a lot of Oldfield's game involves piloting glider- or airplane-like avatars.

/. acquired by The Sun (UK) ?

[rokzy]

seems possible with the ever-decreasing quality of journalism seen here...

"yeah what 'appened right was this computer-me-thingy went all pear-shaped and these silly buggers Comair went and got themselves 'Done In'..."

Re:/. acquired by The Sun (UK) ?

[conteXXt]

fings got a lil wobbly did they?

Re:/. acquired by The Sun (UK) ?

[mekkab]

"Top Brass has gone all slopey shoulders saying quote 'every'fing was all tickey-boo last week...'"

Re:Maybe it had "worked just fine" for them?

[GoofyBoy]

>I bet *now* they'll upgrade, but until this particularly hairy situation arose, they

Thank you Mr. Monday-Morning-Quarterback.

Its all different once you are on the field.

ComAir Now Hiring IT People

[JavaDev04]

Hey everybody! Comair is hiring Unix System Administrators and IT Software Engineers! http://www.comair.com/hr/other/

Re:ComAir Now Hiring IT People

[hugesmile]

At least they aren't requiring 5 years of Office 2003 experience, or 10 years of .net web development, or some other nonsense ads.

Re:ComAir Now Hiring IT People

[dgb2n]

Hey everybody! Comair is hiring Unix System Administrators and IT Software Engineers! http://www.comair.com/hr/other/

Read to the bottom. They're also hiring a "Staff Scheduler". Only a high school diploma required and 1 year of experience. Maybe they should raise their qualification requirements for this one given recent difficulties ....

Re:ComAir Now Hiring IT People

[Dun Malg]

Read to the bottom. They're also hiring a "Staff Scheduler". Only a high school diploma required and 1 year of experience. Maybe they should raise their qualification requirements for this one given recent difficulties ....

It wasn't the monkey filling in the scedule that decided to use a 16-bit signed integer. Furthermore, getting several hundred schedule-fillers with higher education wouldn't have kept that integer from overflowing.

Re:ComAir Now Hiring IT People

[dbIII]

It wasn't the monkey filling in the scedule that decided to use a 16-bit signed integer.

No, but a couple of dozen people with a clue, a big patch of floor and a lot of chalk and a pile of telephones could cobble something together and get some planes moving.

Re:ComAir Now Hiring IT People

[Dun Malg]

It wasn't the monkey filling in the scedule that decided to use a 16-bit signed integer.

No, but a couple of dozen people with a clue, a big patch of floor and a lot of chalk and a pile of telephones could cobble something together and get some planes moving.

As a previous poster noted, any system used to keep track of crew scheduling has to be FAA approved, be it a computerized system or a bunch of people in a hangar with chalkboards and phones. They had no pre-approved "greaseboard" procedure, so nothing those couple dozen people came up with would be allowable anyway. The FAA rules are very clear, and they don't cut any slack or make exceptions. Crew scheduling procedures must be submitted, carefully examined, and approved before use and that's simply not going to happen on short notice.

Time is valuable.

[oneiros27]

As much a troll as the parent might've been, he does make a valid point in that people's time is valuable.

I can't stand it when someone posts a URL to some mailing list, telling everyone to go and look at it, without telling us why we should care about it.

When taken without neighboring information, the only clues that Slashdot gave about the article was that it was in the 'IT' section, and had a 'bug' picture next to it, so we know it was a technology problem, which most computer geeks would have known from 'overflowed 16-bit counter'.

It doesn't take that much extra effort to add a little more detail so that people can make a decision if it's of sufficient interest to spend our time reading the article:

According to the Cincinnati Post, the Comair system crash that resulted in 1,100 airline flight cancelations on Christmas day was caused by an overflowed 16-bit counter.

Re:Time is valuable.

[babbage]

As much a troll as the parent might've been, he does make a valid point in that people's time is valuable.

I can't stand it when someone posts a URL to some mailing list, telling everyone to go and look at it, without telling us why we should care about it.

Thank you -- that's all I'm trying to say. I'm frustrated at how amateur most of the Slashdot editors are, but I honestly wasn't trying to be a troll about it.

A simple amendment to the statement they used would have been enough to clarify everything for everyone, and would put the link in the proper context for deciding if we want to read more about it.

Why has Slashdot never managed to get this right? They've certainly had enough time and enough complaints about it by now; I can only assume that they're being willfully sloppy. That's again not meant as a troll; it's disappointment in how Slashdot is less useful and credible than it should be simply because such journalistic basics aren't taken seriously here.

Re:Comair?

[bje2]

The "Comair system crash" link was just a link to a previous slashdot story (no change for /.'ing that), and already contained a summary...it would have been (-1) redundant to provide a new summary for what originally happened, when you can just link to the original summary...

Re:Comair?

[babbage]

Potential trolls aside, Comair is a regional air carrier, based at the Greater Cinci airport, that was bought up by Delta, and turned into their secondary route provider. They handle both short and medium-range non-stop flights (i.e. Ohio to Atlanta or Orlando). So it's more closely-related than the code-sharing arrangement that some carriers have.

Thank you. So if the article just said something like...

Gogo Dodo writes "According to the Cincinnati Post, the Comair airline shutdown in the midwest last weekend was caused by an overflowed 16-bit counter. Perhaps Comair should have paid for the software upgrade to MaestroCrew." You heard it here first...

...then it would have been enough. That one amendment -- "airline shutdown in the midwest last weekend" -- would have been enough to make the article perfectly clear to anyone who wasn't up to date with the story so far.

Is this really so much to expect? I wasn't trolling, honest, I'm just increasingly frustrated with how addled the editorial review of articles is around here. They've had years to figure this stuff out, but it's just as bad now as it was when Slashdot got started. Are they ever going to start taking their jobs as editors seriously & professionally? (And this isn't meant to be a blanket complaint -- Pudge for one seems to be aware of these basics and tries to do right with his articles, but other editors consistently muck this up...)

I did RTFA

[EvilStein]

"The computer software that crashed and grounded Comair's entire fleet on Christmas Day was an antiquated system due to be replaced in the coming months."

First paragraph. I had just forgotten about it by the time I got to the *end* of the article. 6am + ADD - caffeine = me missing that bit. My bad. :P

Re:Maybe it had "worked just fine" for them?

[jcr]

Maybe the existing system was working just fine?

Apparently not.

-jcr

Re:Maybe it had "worked just fine" for them?

[Saven Marek]

Well the good thing now is they have to respond quickly. They have problems in december which FORCED them to upgrade and they expect to have it done by january.

So what would have happened if they hadn't been forced to? things would just get worse and quickly

Online Anime Gallery's

Re:98

[kg_o.O]

Ah, Slashdot. Where even after the true culprit is unveiled, the problem somehow continues to be on Microsoft's end. What I was reffering to was the California case, similar to this one (described in a bugtraq post). Mixing them up was of course my bad. Anyhow, the main problem was not the fact, the bug exists, but the fact, that they're using OS with known bug where people's lives depend on it, and that's what my comment was reffering to.

Re:Maybe it had "worked just fine" for them?

[EvilStein]

Oh, you're quite welcome. Be sure to stay tuned for my next opinion piece regarding "World Peace in 6 Easy Steps."

The "Real" culprit...

[Nick Driver]

...is the management of software companies who ceased using real computer scientists to design and write their apps because disposable code monkeys work for so much cheaper. And outsourcing is even cheaper... in the short term (which seems to be all that matters in this industry anymore)

Re:The "Real" culprit...

[sray]

But isn't management cutting costs in response the contemporary customer who has eschewed the travel agent in favor of web sites that compare prices and allow you to book through the lowest price? This rewards airlines that can cut their costs the most. So airlines cut costs right down to the bone and the end result is that everyone (including the parsimonious customer) suffers.

Talk about a self-feeding loop.

Re:The "Real" culprit...

[Anonymous Coward]

who ceased using real computer scientists to design and write their apps

Dude, take a chill pill. This software is 15 years old. What code have you written that's lasted that long (and is nearly as complex)?

Hindsight is 20/20 and you can't check all bugs. I'm sure a "computer scientist" would of brought up the 16 bit int issue ... and would of still been overdesigning that project to this very day.

There was a high profile example of this problem

[hey!]

back in the early 80's. There was a big financial company that had an automated system that watched the prices of certain commodities and issued automated trade orders. The transactions where stored in arrays addressed by 16 bit signed integers, with the (now) highly predictable result on the first day that trading volume exceeded 16384 transactions. Since in C arrays are just syntactic sugar for pointer arithmetic, the system started executing trades based on "data" from random bits of heap memory. This apprently went on for some time before a human being figured out something had gone wrong, and (reportedly) the company lost billions in a single day. This might be somewhat exaggerated, since the event now has passed into folklore.

In any case, this is one of those incidents like the Therac-25 accidents that experienced programmers should always have in mind.

Re:Maybe it had "worked just fine" for them?

[EvilStein]

You're right, it's supposed to be upgraded in the near future. I missed that bit, sorry 'bout that.

It's also true that many organizations don't upgrade things because they continue to run worry-free for ages, therefore upgrades aren't seen as necessary.

Upgrades & security patches often get overlooked because of the old saying - "If it isn't broken, don't fix it!"

Re:Comair?

[Alioth]

Probably so the crews from both parts of the airline don't get mixed-n-matched - there is strict seniority in airline heirachies, and the system Delta has probably can't handle two seniority lines at once for the same job. Your seniority affects what schedules you get - the more senior you are, the more likely you're going to get the schedule you want to fly. New pilots/cabin crew tend to get all the shitty routes no one else wants to fly.

Re:Maybe it had "worked just fine" for them?

[freeze128]

I bet *now* they'll upgrade

Don't count on it. They'll probably just store the information in a text field under 'notes'.

Hmm.. Delta owns Comair...

[EvilStein]

Delta Airlines has also been gutting its IT (and other) staff in recent years, while offering expensive protections to executives and many other employees making over $200,000/yr.

I'm wondering when Delta & Comair identified the troublesome software package as something that they should upgrade? The article says that it's going to be upgraded in upcoming weeks, so when did the upgrade get approved?

Is Comair going to try to go after SBS International for damages?

Has this issue ever come up before or did the powers-that-be know about the possibility of major issues beforehand?

It'd be interesting to see what happens as a result of this.

Re:Hmm.. Delta owns Comair...

[drew]

Is Comair going to try to go after SBS International for damages?

Most likely there's a small print clause in the software documentation to the effect of:

NOTE: This software can handle a limit of 32,700 schedule changes in one month. System behavior in the event of more than 32,700 schedule changes per month is undefined.

But 15 years ago when Comair was reviewing the software, 300 schedule changes per month probably seemed like a lot, so they promptly forgot the number and have acted ever since as if the number was effectively infinite.

So the lessons here are:

If you are a programmer, and you have to impose an arbitrary limit on something, always document that limit, no matter how high it is (even better if the limit is already specified in the contract or functional spec). even if the client completely ignores the documentation, your ass is covered.

If you deploy mission critical software, always find out all of the limits of your software, no matter how ridiculous they might be, and put into place a system or policy that will watch for when you approach those limits (and ideally prevent you from exceeding them). then if you hit a limit that was not documented, you can have somebody's ass served to you on a plate.

Is he among us?

[cbelle13013]

I wonder if the poor sap that wrote the software is reading through all these comments and just smacking his heaed. How do you go home at night and tell your wife that you signed an int, and it ruined Christmas for a whole lotta people?

Re:Is he among us?

[Opie812]

How do you go home at night and tell your wife that you signed an int, and it ruined Christmas for a whole lotta people?

Here's how:
"Hi honey, remember when I was complaining to you about how my boss wanted this project done double-time and they didn't have enough programmers because the pay is sh*t, and the testers had their hands tied because of internal politics, and the Q/A department couldn't do their job properly because we had an artifically tight deadline? Remember that? Well, that just came and bit us in the ass."

ahhh, I love my job.

Re:Maybe it had "worked just fine" for them?

[EpsCylonB]

IT should serve the bussiness, the bussiness should not be serving IT. If the lastest stuff is too expensive then your job is to make what you have work. Life (and work) are not easy.

Re:Maybe it had "worked just fine" for them?

[aoty]

My wife works for Comair here in Cincinnati. The computer system under discussion was in the process of being upgraded prior to the crash. Comair's IT recognized weaknesses in the current system some time ago. The upgrade just happened to be taking a little longer than anticipated. Timing is a bitch, isn't it?

Re:98

[Anonymous Coward]

I'm not sure how the open source movement can pride itself with the quality of imbecile fanboys it attracts.

We're working on a solution. Right now we've just started to dangle shiny things in front of the imbecile fanboys to distract them; we've called them "Gentoo" and "Unbuntu". That way most of the uninformed and downright wrong fanboy opinions are becoming contained in small, easily avoidable groups where they can't do much harm to the rest of us.

The next stage in the plan is working out some way of getting them to fight each other without causing any fallout which might damage the rest of us.

martian rovers two week computer crash

[peter303]

Remember last February the Martian rovers were sidelined by a computer crash. The reputed problem was the huge flash memory appeared to fill due to not properly building the free inode list in the Unix OS. This was the realtime VxWorks UNIX from Charles River used by NASA on space probes for 20 years. The Spirit rover went into a perpetual reboot cycle for a day (rebooting its default response to a severe crash). Then it took two weeks to diagnose, repair and test the software patch.

So the point is, that bugs can appear even in highly tested OSes.

Re:martian rovers two week computer crash

[CyberDave]

Uh, that's VxWorks by Wind River. (My DSL modem runs it, and I've got a buddy who works on embedded systems with that OS--I hear a lot of complaints about it from him, but that's another story.)

And as I recall, the problem with the rover(s? -- I don't remember if both rovers had this issue or just one) was that there was a scheduling conflict where a high priority task spawned a low priority task that blocked waiting for some event, and subsequently prevented a higher priority task from starting (that might have been a task related to filesystem stuff on the flash memory, now that I think about it).

I'm trying to find a link now. This link says the problem was with removing files in the DOS (do they mean VFAT?) filesystem in Spirit's 256MB flash memory (no mention of the same problem with Opportunity). Nothing about a priority problem. Huh. Where did I hear that?

Re:martian rovers two week computer crash

[trailerparkcassanova]

vxWorks is from WindRiver and is not UNIX. Charles River Data Systems had a real-time OS called UNOS back in the '80s. It was UNIX-like bit still not UNIX.

Re:martian rovers two week computer crash

[multipartmixed]

> Nothing about a priority problem. Huh. Where did I hear that?

I don't know, but I read it too.. Possibly on a NASA web site. Long story short, the solution was to change the default mutex type on the system so that low-pri processes inherited the priority of any process they were blocking... so they could finish, and the high-pri process could get back to work.

Re:Maybe it had "worked just fine" for them?

[nolife]

I worked for a major airline a few years ago. They were and still are using old IBM servers (35xx?) running OS/2 as the backend for the ticket/gate computer systems which run Win3.1 on token ring. Those backends speak to the mainframes with modem sharing devices in some protocols and timings that I can not recall at this time (I do remember something about SNA/SDLC and configuring the polling intervals though). It worked then, works now and they still use it. I was used in my time there to convert the smaller non hub airports over to a TCP/IP system to connect to the mainframes (basically still W3.11 but with TCP/IP using terminal emulation software) via ethernet, this basically provided the same functionality but removed the need for any location specific backend servers. They converted what they wanted but still left quite a few of the older systems in place.

Re:Hmmmm.... Maybe I should edit my software

[aldoman]

Am I the only one which reads POS as 'piece of shit' regardless of the fact I know it means point of sale?

It always fits perfectly in the context as well, as this example proves.

Re:It shows what unions are all about

[Anonymous Coward]

Properly organised industrial action would have been the right thing to do, but what the US Airways staff did was just the wrong way to go about it.

As for the US Airways workers being greedy, maybe you should read up on what US Airways has done to their pensions, benefits packages and wages over the last couple of years. Some US Airways workers are surviving below the poverty line, relying on welfare and food stamps to feed their families. So which would you prefer; that the workers continue to live on Government handouts (Paid for by you!) or that they try and get their employer to pay a fair wage which they can use to support themselves and their families?

Happened to me too

[wandazulu]

I worked at a bank in the early 90s that had a trading system based on SQL Server and the client was written in Visual Basic 3. Apart from every other bad design choice in this system (I inherited it when the designers got promoted and started working on another, even bigger system), the all important record counter was an integer, so when trade 32768 was posted, the application crashed, and simply could not be started again, because the first thing it did was try to show the current total (it was written for operators to use, not traders). Worse was that the counter variable wasn't a global, and it was often times a stack variable, and always with a different name (sometimes iCounter, sometimes iCount, sometimes x).

The upshot was that I was able to convince management to totally scrap it and allow me to write a new one. The downside was that the idiot who designed the original system went on to spend 100 million dollars on this new, grandious system that too was eventually scrapped, but he knew long before that his turkey wasn't going to fly, so he quit and became a lead architect at some other company.

*Sigh*...okay, back to coding.

Re:Happened to me too

[jtwine]

Yet another reason why VBers should never be in a design or architecture position... :)

C++ developers have been ragging on VBers for years... Ever wonder why? Read above.

Peace!

Re:Maybe it had "worked just fine" for them?

[Anonymous Coward]

Arbitrary being the key word. The limits probably weren't arbitrary when they were put in. The system probably had an expected life, and instead of maintaining their infrastructure the people tasked with running the company probably gave themselves pay raises while postponing payments into the employee pension fund. What stories like this are really about, are the complete worthlessness of MBAs. They exist for the sole purpose of diffusing responsability and obstructing accountability.

Very rarely does anyone have the luxury of designing for something with a hundred year life expectancy and a budget to match.

Re:Hmmmm.... Maybe I should edit my software

[mbaciarello]

See comment before yours... Check the number of transactions against MAX_INT and fire an alarm if equal...

Or, better yet, set it to fire an alarm at [MAX_INT - (AVERAGE_NUM_OF_TRANSACTIONS_IN_ONE_MONTH)]...

That is, if it would be too costly to switch to larger variables...

Re:Maybe it had "worked just fine" for them?

[Meostro]

Is one step ??? and another Profit!?

overdue

[PureCreditor]

now THAT'S an upgrade longgggggggggggggggg overdue due to corporate bureaucy failing to commit a budget

Arbitrary limits are unavoidable

[tepples]

arbitary limits will trip you up eventually. It's not as if nobody knew to avoid them before

For one thing, there is an "arbitrary limit" on the number of atoms in the visible universe. At any given time, there is an "arbitrary limit" on the amount of memory available for a given price in dollars, and there is also an "arbitrary limit" on the amount of arithmetic that can be done in one second for a given price in dollars. Arithmetic in variable-length integers (called bignums) allocated on the heap often carries significant performance overhead compared to arithmetic in machine word-sized integers (called ints) in fixed-size data structures. On a given computer system, it may have been an engineering decision to use ints rather than bignums.

Re:Arbitrary limits are unavoidable

[Nasarius]

Unnecessary. 64-bit integers, with a capacity on the order of 10^19, are effectively unlimited for all but a few uses (science, math, etc.). Businesses never have to deal with such large numbers, and probably never will. But in this case, Ye Olde 32-bit Integere would have worked too. 10^8 is a lot more than 65,536.

Re:Arbitrary limits are unavoidable

[SnakeJG]

But, at the time the system was setup, 16-bit might have been the normal integer size, and working with 32-bit or higher might have, at the time, fallen under bignums...

Re:Arbitrary limits are unavoidable

[Kjella]

For future software, I agree... but remember that there is a lot of legacy code, and legacy data structures. You say 64-bit is more than they'll ever need. 32-bit as well. But years ago, some programmer probably thought that 16-bit was more than they'll ever need too. Probably not as many planes, as many crew, as many routes.

And there are always times when you'd need more... oh, 128-bit encryption key? 160-bit hash? 1024-bit public key? Personally, I deal in C++/Qt, and that is one of the few things I don't like about C++ (Qt has fixed pretty much all the rest). Just allocate as much space as needed. I know dynamic numbers have a penalty, but isn't is possible to create some dynamic fallback? I.e.

x = 54
Try 8 bit (char/uchar) ok.

x = 544
Try 8 bit (char/uchar) op -> overflow.
Try 16 bit op -> ok.

x = 70,000
Try 16 bit op -> overflow.
Try 32 bit op -> ok.

x = 5,000,000,000
Try 32 bit op -> overflow.
Try 64 bit op -> ok.

x = 17,000,000,000,000,000,000
Try 64 bit op -> overflow.
Switch to heap -> ok.

This really should be somewhere in the programming language. It'd stop a whole lot of annoying problems. It'd still have a penalty if x -> huge -> small (as tests for smaller size ints would cause too much penalty) but normally that'd go x -> huge -> crash instead, so... That way it would *never* run into this kind of issues.

Kjella

Re:Arbitrary limits are unavoidable

[LWATCDR]

The limit was a signed 16bit int counter of changes made per month. It took a huge number of things going wrong all at once to overflow it. Moving to a 32bit would fix it. No need to go for bignums yet.

Re:Arbitrary limits are unavoidable

[roadrunnerro]

it's the Variant type - most languages have them in one form or another. A good example of how this isn't a solution (and what it promotes) is Visual Basic..

Re:Arbitrary limits are unavoidable

[CableModemSniper]

No he's not talking about a variant type, he's talking about a numeric type that stores numbers it can natively, and automatically promotes numbers that it can't to a heap representation. Such as in ruby for instance:

irb(main):001:0> x = 1234
=> 1234
irb(main):002:0> x.class
=> Fixnum
irb(main):003:0> x = x ** 30
=> 54885141842922207462706050754135963848157659549253 1140166924984667909193218216496312222744576
irb(m ain):004:0> x.class
=> Bignum

Re:Arbitrary limits are unavoidable

[CableModemSniper]

I wasn't trying to start a python vs. ruby war, I swear.

MOD THIS GUY UP

[EvilStein]

You kind of answered a question I posed elsewhere - "How long has the airline been aware of the fact that this software has potentially serious issues?"

"some time ago.."

Re:Maybe it had "worked just fine" for them?

[Tassach]

IT should serve the bussiness, the bussiness should not be serving IT

In general terms, that is correct. However, it should be obvious that for many businesses, IT *is* the business.

When an information system is such an integral part of the business that you cannot conduct routine operations without that system, you have to regard it as a piece of capital equipment. In the case of Comair, the computer crash shut down their business just as completely as if an airplane had crashed.

Re:Maybe it had "worked just fine" for them?

[barzok]

When business won't give IT the money needed to keep business's systems operational (be it for manpower, software upgrades, or electricity) and makes the final decision in purchases, something's going to have to give.

Business decides to buy a software package. After a while, upgrades come out, and the old version keeps getting pushed to the limits. IT adivses business of this, and says that an upgrade/replacement will resolve the problem, but business refuses to authorize said upgrade/replacement.

How do you propose IT "make it work" when their hands are tied? Even worse, IT will take the blame when it wasn't even their decision to make.

I don't know if it's the case

[P2PtheBeginning]

But my experience in the IT industry is like 'Does it work ? then it's ready'. And also 'Can you do it faster? then do it faster and patch the program later'. There's a wrong misconception that everything should be done quickly in the IT industry when actualy all should be done to the perfection because it takes even more time otherwise, the programs will have to be fix soon or later.

Re:Comair?

[amabbi]

Now my question would be, since they're owned by Delta, why wouldn't Comair flights be handled within Delta's own reservation/flight tracking system?

There probably isn't any reason to. Comair, as a regional jet carrier, has separate crew contracts and crew rules than Delta, a mainline carrier. Thus they operate completely different types of jets, with different crew staffing requirements. The FAA crew rules might even be different. While it might make sense from a consolidation standpoint to merge the two systems of Comair and Delta, since in reality there would be no interaction and no overlap between the two systems (an RJ pilot isn't suddenly going to jump over to fly a 757) the expense isn't worth it.

p.s. I've traveled through CVG, on Delta, during the holidays. Not anymore... One weather-delayed flight and the whole system falls apart.

Then I hope you also avoid United/United Express/Ted at O'Hare/Denver, Continental at Newark/Houston, Northwest in Detriot, USAirways in Cincinnati, American at O'Hare/Dallas... etc. etc. Every airline, not just Delta, uses hubs, and ground stops at any of these airports will cause significant delays. That's just the reality of air travel these days; if you're really worried, book non-stop travel (and pay up to 10x more).

Re:It shows what unions are all about

[duffbeer703]

Try taking a 30% pay cut and work 20-50 hours of mandatory overtime with a demanding job with an undermanned crew.

Airlines are going bust thanks to incompetent management -- they cannot even keep afloat with the huge government subsides and loan guarantees that they've received for decades.

Re:Comair?

[babbage]

Then I hope you also avoid United/United Express/Ted at O'Hare/Denver, Continental at Newark/Houston, Northwest in Detriot, USAirways in Cincinnati, American at O'Hare/Dallas... etc. etc. Every airline, not just Delta, uses hubs, and ground stops at any of these airports will cause significant delays. That's just the reality of air travel these days; if you're really worried, book non-stop travel (and pay up to 10x more).

Or fly Southwest, which doesn't use the hub-and-spoke system, and get a direct flight to anywhere they fly for 10x less. Or maybe not 10x less, but in any case cheaper than any of the bigger airlines they compete against. There are alternatives, provided you are travelling between cities Southwest or an airline like them (JetBlue, Song) goes to...

The far bigger snafu...

[tjgrant]

...was over at US Air, where a large number of selfish and uncaring union employees decided to mess up a huge number of their customer's Christmas plans.

While stupid stuff like this software problem is embarrassing, and the SBS and the people who wrote the software should hang their heads in shame it was still unintentional and is nowhere as shameful as the deliberate sabotage done to US Air's customers by their union employees.

Re:The far bigger snafu...

[Anonymous Coward]

' The only way for employees to fight back is to hit the company where it hurts the most, by striking at a busy time. '

If they hate their jobs so much, why don't they just quit and go elsewhere? This "strike" is like pouring water in a sinking ship. They'll be the first to suffer the negative effects. Not only does it damage the company so that it has less money to pay these employees: it also tells the company to do whatever it can to get rid of these employees.

Re:The far bigger snafu...

[Thundersnatch]

You're spreading a lot of misinformation.

95% employment is considered "full employment" laregely because on average in this country several precent of the workforce chooses to be unemployed at any given time. They quit for something better, move towns, go to school, have kids, etc. It's called "frictional unemployment". There are jobs available, but people are busy doing other things. When employement gets higher than 95%, wages take of and inflation spirals out of control, as it did in the 80s. Look up "unemployment" on Wilkipedia for more information.

Also, the prison populations have at most a 1.5% effect on unemployment numbers (less than 0.7% of total population), so you're full of shit there too.

Basically, you need to stop using Spin Magazine as your primary news source. Buy a subscription to The Economist or something.

Re:The far bigger snafu...

[/dev/trash]

While I'm not the biggest fan of the unions, USAir corporate has been asking for and getting TONS of paycuts and benefit cost sharing deals for awhile now. And they want more, and when they didn't get it as easily as they had before they pulled the "We'll just declare bankruptcy." card. THAT's the crime here.

Re:The far bigger snafu...

[tjgrant]

Oh I agree that US Air Management aren't angels. However, that does not mitigate the behaviour by those employees.

Sick out some other time. Doing this to innocent Christmas travellers was completely inappropriate.

Re:It shows what unions are all about

[DavidTC]

And nothing stopping US Air from doing that, except then they'd lose all their emoployees thanks to the unions and that no one would want to work for such a shitty company except you, so they'd immediately fail. Meanwhile, all the strikers would end up working for the other companies that appeared to take up the slack.

You really are a moron, aren't you? Absolutely nothing is legally or contractually stopping US Airways from doing what you said.

How soon before the first passenger lawsuit?

[davidwr]

This is one time I'm glad airlines have limits on their liability to passengers.

Can you imagine what would happen if they didn't? Everyone would sue them for a ruined Christmas and Comair would be in Chapter 11 by tomorrow.

If Comair can afford it, they should give refunds to all affected passengers. I hope they've already comp'd the hotels and incidentals. Every executive and the highest-level IT people should surrender their 2004 bonuses as a gesture of contrition.

Re:Comair?

[buckeyeguy]

Actually, I avoid it all by not flying anymore (out of disgust with the big airlines), but your point about avoiding hubs is well-taken, especially in the case of larger airports with divided terminals. Short layovers at divided airports make changing planes an aerobic workout I'd prefer to avoid.

If and when I absolutely need to fly somewhere, I most likely will pay extra, for the non-stop and possibly for the first class/business section, or just chuck it all and book a flight on a charter airline instead. I hate the commercial airline 'experience' just that much.

Re:Maybe it had "worked just fine" for them?

[jcostantino]

Well, I have #5 as "???" and #6 as "Profit!" - what's step 1 through 4? :)

Re:There was a high profile example of this proble

[t_allardyce]

Makes you wonder - perhaps important systems like that should be written in a higher-level fully memory managed language, even if its slower, buying extra fast hardware to run it means nothing to a decent sized company, but loosing a million because of over-complex code with potentially 100's of little bugs means allot...

Re:Maybe it had "worked just fine" for them?

[Kalak]

RTFA - the upgrade was planned. It's probably tested by now, and they're not likely to upgrade during such a busy time of the year. No heads will probably roll because of this, but upgrades may be rushed in the future.

Re:Hmmmm.... Maybe I should edit my software

[Duhavid]

Do the number of cashiers factor into this?
Or is it per terminal ( person )?

As a fellow programmer, you are *probably* safe. Were it my decision to make, I would bump to 32 bits the next time you need to open the code to fix other bugs. And look for other instances of this problem in other areas. Caveat, I am the kind of programmer that checks return codes, the return from malloc/new, etc, etc, in addition to try catch blocks. I use the length limited versions of the string copying functions, etc, etc...

Computer crash is worse for airline

[AtariAmarok]

"I'd say the computer crash is worse for the airline"

Obviously, computer crash victims complain about the airline long and loud. Plane crash victims tend to be silent.

Re:Maybe it had "worked just fine" for them?

[EvilStein]

I don't know, but isn't there a way that we can blame Microsoft or SCO for this whole Comair mess? :)

Re:Hmmmm.... Maybe I should edit my software

[prshaw]

>> Check the number of transactions against MAX_INT and fire an alarm if equal...

And maybe shutdown the system so it doesn't cause any problems?

Gee.... sounds like just what they did.

Re:Hmmmm.... Maybe I should edit my software

[prshaw]

>> I haven't used 16 bit integers since the early 90's,

And didn't I read that this system was about 15 years old?

Little bit of 16 bit integer math and ....

So you are the one that used the 16 bit integers in this!

Actually...

[caveat]

...my aunt and uncle had their flight from Denver cancelled when a baggage truck blew across the tarmac and into their plane.

Could just have easily been a 32-bit counter

[Junks Jerzey]

In this case, a 32-bit counter would have prevented the crash, but really, how many coders check for overflow in their 32-bit counters? In long running systems, you *have* to. And it's easier to overflow a 32-bit counter than you think; it depends how often it gets incremented (that is, what is being counted).

In languages with graceful promotion from integer to bignum, this is a non-issue. Not for most languages, however. (And lest anyone think I'm oversimplifying here, such promotion is not so simple, in that bignums take a variable amount of memory. To handle this truly transparently you need some form of automatic memory management, which is a no-no for many embedded systems.)

Re:Could just have easily been a 32-bit counter

[praxis]

I check all my arithmetic variables--regardless of size--for overflow and underflow. Doesn't everyone?

Re:Could just have easily been a 32-bit counter

[nsxdavid]

It's not about checking for an overflow, its about not having one or having some way of dealing with it other than crashing/malfunctioning. An assert() tells the programmer (at least) where things went wrong, but hardly gets planes in the air again.

-- David

Re:Could just have easily been a 32-bit counter

[praxis]

I did not say I just assert, I have a plan in place for dealing with overflow and underflow errors. Having a large enough counter is of course necessary as well. I do both, and when there is something that can be done to mitigate the overflow, it is.

Comair and...

[trailerparkcassanova]

16 bits will get you a cup of coffee...

Re:Comair and...

[corngrower]

6 bits will get you a cup of coffee...

Not if it's Starbuck's

Eventual cost

[eobanb]

I think their downtime now is probably costing them MUCH more than $200.

Not quite,

[955301]

Somewhere in his awkward choice of words is a point. Speeding an application to market by skimping on the quality of the work causes this problem. Real computer scientists balk at leaving problems half done, monkey's don't. The outsourcing thing at the end, yeah, I don't agree with that as a mantra.

Re:It shows what unions are all about

[cayenne8]

"Airlines are going bust thanks to incompetent management -- they cannot even keep afloat with the huge government subsides and loan guarantees that they've received for decades."

Hmm...maybe they should take a look at ones that are doing fairly well, like Southwestern. I pretty much fly them exclusively. They fly to most places I go...have great specials, are generally on time...and the people that work for them are all friendly and courteous.

What are the other airlines doing so wrongly?

Re:Signed or unsigned

[Jim Hall]

Yes, they had to have used a signed 16-bit integer counter. A little background for those who may not know how to count in binary:

1000000000000000 ("1" and fifteen zeroes) = 32768

If you know your binary, you know you can actually store a number of about twice that if you fill all the zeroes with ones:

1111111111111111 (sixteen ones) = 65535

But, if that's a signed integer, you'll need to reserve a position for +/-, leaving only fifteen binary places for the integer.

0111111111111111 ("0" and fifteen ones) = +32767

So as soon as the Comair system needed to count 32768 flight crew changes for that month, it crashed. It counted -0, or zero.

-jh

Accountability seems rare at that level...

[Richard Steiner]

I've seen CIOs and VPs of IS do extremely stupid and/or damaging things to the departments they were in charge of and then leave the company with more laurels danging from their CVs.

It doesn't seem to matter that the real-world impact of their actions and policy decisions actually hurt the company.

Very frustrating to see this at multiple companies and (in some cases) multiple times at one company.

Re:Hmmmm.... Maybe I should edit my software

[pclminion]

Uh, there's no problem with taking advantage of real-world constraints. Using an unsigned int would imply a minimum transaction time of 86400/65535 = 1.31 seconds per transaction.

I get the sense that you are objecting that the system has no room for expansion. Well, yeah, but you're probably complaining outside of the requirements. If the stated software requirement was to run on a single POS system managing a single register, then the limitation is perfectly appropriate. There is no human clerk, and for that matter no customer, who could complete one transaction every 1.31 seconds on a single register, constantly. It's simply impossible.

Re:There was a high profile example of this proble

[sharkey]

Seems like there was another example of this sort of thing on November 2nd, 2004 as well. IIRC, some North Carolina machines dumped 3000+ votes due to a similar problem.

Re:Comair?

[AceyMan]

mod parent down.

One poster already noted; The wholly-owned carriers fly different equipment and are staffed by pilots who are members of a different seniority force.

Moreover, typically the crew tracking system is integrated with the flight operations/dispatch system, and the maintenance control system, and the route planning system, and the trip optimization system. You wouldn't want to try to integrate all those functions into the parent carriers system unless you *had* to.

Finally, CFR 14 Part 121 says that each certificated carrier has to have their own dispatchers on staff. Comair, et al, are technically independant carriers -- they have their own certificate (DOT license to run an airline), and therefore have to staff their own flight operations (dispatch) office.

Therefore, Comair cannot integrate their staff with Delta's, even if they wanted to. Of course, that doensn't mean they couldn't still use Delta's operations software, but it just shows how separate the airlines actually must operate -- making the advantage of merging systems specious at best.

\FAA licensed aircraft dispatcher

You've missed one dimension

[cdrguru]

The dimension of time. An airline seat changes in value to the consumer as time goes on. Currently, we're told that the optimal price point changes at 14 days, but I believe it to be a little different than that in reality.

You want to fly to Los Angeles in a month and purchase the ticket then. The price you pay reflects the value of being able to make that choice then and assuring the airline of a seat being filled in a month. The value of the seat changes as time goes on such that 1 day before the plane leaves the seat is now worth a lot more to someone that has to get to Los Angeles the next day, no matter what the cost. Of course there is the other aspect as well - the seat has no value once the plane leaves.

Managing this changing value is what makes airline ticket prices incredibly complicated.

Re:Maybe it had "worked just fine" for them?

[xilet]

Wow that was an impressive twisting of the article. Not that I disagree with your point of worthless MBA's but the fact that you somehow turned a software design flaw into the current CEO's fault.

Re:Maybe it had "worked just fine" for them?

[SWTP_OS9]

Odd are someone or some thing will. In all major blame games. You must sarcifice someone or some thing to the god of disaster and oops.

Else will loos your head. Esp when the Fed's will be hunting for the problem so thay can create a usless soloution that cost billions and makes the inital problem worse.

Some poor little software package is crying bits right now! Wishing he was 32 and not 16 bit!

Re:98

[SWTP_OS9]

Just think if they were running XP and its problems with WiFI and Firewire breakdowns! Wow what a train wreck! Or in this case a plane wreck!

Re:Maybe it had "worked just fine" for them?

[greenhide]

Programmers become obsessed with special cases, as in "What would happen if there was a ginormous snow storm during the most busy travel season of the year?"

Interesting, but it's much more important to have a system that deals with mosts cases efficiently and correctly.

It's possible when this program was first written the hardware it was written for would have had difficulty with 64 bit integers. Maybe the additional memory space would have been a problem. I don't know. In any case, the system worked pretty well for a long time. Special cases can trip up even the best software.

Also, 32,000 changes in a single month is a *lot* of changes. The fact that this problem only happened once is an indication that the existing system was working fine.

I'll never forget a definition of legacy systems once -- "Systems that *work*"

It's possible that the system that will be used to replace the old one will be less stable and crash more on the order of once per year or month rather than once per software lifetime.

Re:Maybe it had "worked just fine" for them?

[spectre_240sx]

The trouble there is that the user doesn't realize that it is, in fact, broken. People have this idea that once a piece of software is written, that's it. They don't realize the nature of programming and the fact that even the best programmer will have bugs in his / her software.

Re:It shows what unions are all about

[WhiplashII]

I work in the industry, so I might be able to provide an alternate viewpoint. Esentially what happened to the airline industry is that the market changed from a luxury market (high profit margins, low competition) to a commodity market (low profit margins, intense competition). Unfortunately, the airlines had all made long term deals with the trade unions that presumed stagnate market conditions - so when the market changed, they could not change with it.

The smaller carriers all have one thing in common - no unions. They do not pay their pilots as much, and their pilots do not get paid if they don't fly. The number one expense for an airline is fuel, but the number two expense are the pilots, stewardesses, mechanics, and baggage handlers. There was no way for older airlines to meet the new market conditions (fly more for less profit per flight) without paying people less. The problem is that no one wants to be paid less, so instead they get rid of the least powerful people (who also happen to be the least paid). This is also specified in the union contract... all laying off actions must be FILO.

Essentially, the major carriers are hamstrung by the unions, and they will not survive long term. Unions work by artificially limiting labor supply - but that doesn't work if there is not enough work.

The unions say how evil it is that they are getting pay cuts, but where exactly do they expect the money to come from? The government really should not prop up certain providers when others are eager to take there place. Competition works for the most part. Air travel is becoming a commodity market, like cars. Market transitions cause upheavals, and change the market leaders - especially if the current leaders cannot change their bussiness structure.

I have to say that I totally disagree about management being incompetant - the current management (at least the upper level ones I deal with) are extremely good. They may even get the airline to survive and change to the current market conditions. But what has really destroyed the airlines is the changing markets, and the unions preventing the old airlines to change with the times. The only thing management could have done would be to have rejected the union contracts earlier. But I doubt if that was possible.

Unions seem to believe that society owes them a living. The problem is that society (except in the form of government) is not a person, and so recognizes no debts. Fighting that is totally ineffective because there is no one to fight.

Re:It shows what unions are all about

[WhiplashII]

should read up on what US Airways has done to their pensions, benefits...

That is a common mis-conception. US Airways did not do these things - US Airways is struggling for life, very near death. The market change caused by Southwest and the consumers caused this. There is a simple choice, lose these things via the company cutting them, or lose these things via the company going chapter 7.

The company has only one choice (it cannot kill itself) - it cuts everything that can be cut in an attempt to survive. This means that people would make more doing something else, so they should leave. If they do leave, the company fails and southwest takes up the slack. Jobs move, but the total number of jobs doesn't really change. If they don't leave then that means that for them working at a lower wage is better than being out of work, because they made the choice.

No one is forcing these people to continue working for below minimum wage. There are other places to work, if you are good at what you do. If you are not good at what you do, find something you are good at. That is simply how the world works! (And fighting against that will not change it, and will doom you to failure!)

SIGNED 16-bit!!

[adam31]

what-- were they expecting negatives also?

Re:SIGNED 16-bit!!

[pclminion]

Using a signed integer allows you to distinguish between error and non-error conditions. In UNIX for example, system calls return a negative value on error, so these calls often are declared to return a signed int even though the number they return in a non-error condition will always be positive.

This might be viewed as laziness depending on the cirucmstances. Obviously, it seems weird to waste half of the integer space just so you can return -1 on error, but if you need to report many various error conditions, using negative numbers to do so makes things a little easier because you can just check if the return value was negative in order to detect an error (instead of comparing the return value one by one against each possible error code).

However, the entire problem is mitigated if you just switch to a slightly different calling convention. If you need a function to return some value which is always positive, but you still want to indicate possible error conditions, forget about using the return value to return the result. Instead pass a variable by pointer or reference, stick the result in there, and return 0 on success or -1 on failure.

Unfortunately many programmers regard this as ugly, so we're stuck with silly crap like wasting half the integer space just in order to report errors.

Re:SIGNED 16-bit!!

[The Master Control P]

Er... why not just reserve the last 4096 of 65535 possible return values for errors? if(returned > 61439) ohShit(there_was_an_error);

Coding practice

[hey!]

Well, there's a curious thing about this story.

Back in the 80s when I was C programmer (K&R, thank you, the one true C), C integer types were not standardized. "Integers" were defind to be the most natural size for a machine (typically a data word), "shorts" were defined to be no larger than ints, but possibly smaller (and thus possibly more space efficient). This reflected the philosophy of C-as-portable-assembler: if you were indexing an array of character representations of digits, for example, there was no reason not to use a short. It was conceivable that, since arrays were essentially immutable pointers and array indices were merely offsets against those pointers, you might want to refernce a negative offset from an array base in some kind of clever trick.

Various C implementations used short/integer sizes like 8/16 (for microprocessors like the 8080), 16/16, 16/32. These days, there are some mininal assumptions we can count on. Ansi-C specifies the following as minimal data sizes for char/short/int/long: 8/16/16/32. In practices IIRC, most modern compilers use 8/16/32/32, in other words a 32 bit int. GCC, I think uses 8/16/32/64.
The problem with this airline scenario I would expect is a kind primitive cousin of cut-and-paste coding. This is where the the programmer is pasting something like this from his mental scrapbook:

int i; // index into transaction array
TRANSACTION trArray[]; ...

It's very easy to do something like this. A really conscientious programmer asks himself whether the index value is indexing something that doesn't have a prescribed limit in the specifications (in this case I'm guessing it was probably indexing a file position). If there is no prescribed limit he uses an unsigned long. If there is a prescribed limit that would allow an integer index, he still uses an unsigned long unless he indexing something which logically can't grow larger, or until the profiler forces him.

Which brings me to what I find curious about this. Either: the programmer chose to index the value by an signed short (which would be almost inconceivably stupid as opposed to unforgiveably negligent), or he was using a C compiler with a 16 bit integer, which while possible under ANSI IIRC, seems terribly archaic.

Java, of course, uses 32 bit ints. But you aren't completely safe from this sort of thing. For example FileInputStream has two methods of interest here:

long skip(long n)

this is very safe, since it uses a long, which in java is 64 bits; even unsigned, there is little chance of overflow.

However consider this:

int read(byte[] b, int off, int len)

What happens when a programmer decides to skip around in a LARGE file using this API? If he decides to skip forward by more than 2,147,483,647 bytes the signed int will silently be converted to a negative offset, at least as of java 1.4. Granted the possibility is slim in most applications.

Re:Coding practice

[t_allardyce]

I always wondered why types were machine specific (originally?) it doesnt seem like a good idea even if you do want to be portable - a type is a type, no fooling around, no-one thinks "do i want a quite short number here or a quite long number" they say "i want an 8/16/32/64 etc sized variable here and nothing else" - if you want to compile on a different machine, you're still going to want that size even if the bus/cpu is a different width, IMHO they should have just defined types as int1, int2, int3....int64 etc and let the compiler damn well figure it out - if you want to do 32 bit addition on a 16 bit machine the compiler would warn you, but still hack in 32 bit addition even if it took more instructions.

Lol I wont even get started on index overflowing in arrays, C is great for those hardcore bits of ricing, but memory managed high-level no-hacks programming is what most businesses need for major systems.

Re:Coding practice

[hey!]

What you're describing is more or less the situation we have today with ANSI C, only an int8 is called a char, and int16 is called a short etc. Compilers are free of course to promote the type to a larger type.

One thing to remember that the world hadn't settled on the idea of having even powers of two for arithmetic units, until well after C was developed. The PDP-8 had 12 bit arithmetic, and IIRC the PDP-10 (I could tell you stories about that one) had 36 bit integer arithmetic. The PDP-11 had provisions for 18bit virtual addresses. Things got even stranger if you considered some of the schemes used to addresss large memory spaces.

Even if you are porting a program between different machines that have power of 2 word sizes, hardware was a lot more diverse back then; it had to be. You didn't waste the chip real estate a 32 bit hardware multiply would imply on a machine somebody was going to word process on. If you were porting something Unix to a variety of machines, it is very useful to be able to say "give me an integer index big enough for any purpose I'm likely to need", and for the most part the compiler implementation would be consistent with the kinds of problems that machine was built to solve. Machines that had to handle lots of data had large address spaces and registers, and large integer units to go along with them. Machines meant for affordability weren't expected to handle lots of data.

In any case, since memory sizes were measured in K not Gigs, your perception about what is a reasonably commodious integer index would have been different back then. If you have 24K of memory, you certainly don't worry about overflowing an unsigned 16 bit int. However, the world changed, and software that didn't keep up hit the wall.

Wow. 0 Troll?

[EvilStein]

If any post on here is a troll, this ain't it. heh.

Over-engineer, then optimize!

[Kyont]

You should over-engineer that counter to a ridiculous degree. It's not a bridge made out of expensive steel, it's just bits and bytes. Memory is cheap, disk space is cheap, and bandwith is becoming cheap. If something calls for the number of meters in an Olympic pool (50), size it to accommodate the number of millimeters across the known universe. Over-engineer first, then slim it down only if it causes performance problems.

Picture this: In five years, the stores have RFID tags or faster machines that reduce the POS transaction time to 1 second. The stores are popular and quadruple in number. To match their new inventory system, they artifically track each item bought as a separate sale (average 5 items per customer). Voila, you are up to 1,728,000 possible transactions in a day already.

Customers will always find new and unpredictable ways to use and abuse their software, rather than go through the pain and expense of an upgrade. By 2009, the client will have forgotten your name, but they'll (probably) never have to hunt you down if that counter goes to 100 million or so.

Re:Over-engineer, then optimize!

[Anonymous Coward]

size it to accommodate the number of millimeters across the known universe ... they'll (probably) never have to hunt you down if that counter goes to 100 million or so

Note to self:

The universe is about 100 million millimeters across.

Re:There was a high profile example of this proble

[VeriTea]

Therac-25, for those of you who don't know, was a radiation treatment machine. It was an upgrade to the previous model, and included all-new electronic controls. The controls had one small bug that wasn't discovered until the machine had been on the market for a while.

There were two modes of operation, a very low power mode - intended for long, slow radiation treatment, and a very high power mode - intended for very short bursts of radiation treatment. If a technician accidentally selected the high power mode, then switched back to the low power mode and hit the radiate button immediately, the device would deliver the high power beam, but the display would show it as a low power beam.

Patients would complain about a burning sensation, and then die. Happened quite a few times before anyone figured out what was going wrong.

Re:Maybe it had "worked just fine" for them?

[budgenator]

I bet *now* they'll upgrade
Probably not unless the software is purchased by Delta and they run an instance of the parent companies' software because the christmas rush was critical for the airlines who had to taken in max revenues and min expenses to save their asses this year. My magic eight-ball says this is the beginning of a death spiral for Comair, and the PHB's will see little benifit in purcahsing software to cover a once-in-a-decade event in a company thats unlikely to live 3 more years. If the upgrade is pretty much contractualy bound, the articles

The computer software that crashed and grounded Comair's entire fleet on Christmas Day was an antiquated system due to be replaced in the coming months.

will be in the too little, too late catagory rather than the Knight in shining armour catagory.

Re:There was a high profile example of this proble

[hey!]

The really interesting thing about the incident was that the bug did not show up under deliberate testing. You could try the exact sequence of operations that produced a fatality and not be able to reproduce the effect. The key was that experienced users entered the setup very quickly. I don't recall whether the issue was that the safety mechanism didn't kick in, or whether the input was not properly recorded. IIRC the Therac 25 had its own custom rolled hardware monitor where today we would use an embedded operating system, and the fault was ultimately tied to that, not to incorrect specifications for the user input routine, or incorrect programming of the user input routine.

Re:Maybe it had "worked just fine" for them?

[jedidiah]

As they like to say in my particular part of the retail world: You build for Easter Sunday.

THIS INCLUDES COMPUTING.

Part of serving the business is being built to capacity. This is no different than a correctly tooled factory or having enough warm bodies.

Your mentality simply falls from the illusion that IT isn't an integral part of the business. You can always choose to be "penny wise". However, that always comes with inherent risk.

They question you need to ask your CIO is: Do you feel lucky?

Ultimate solution:

[Atario]

Henceforth, all numeric variables shall be of type BigNum.

unreleased tidbit of information

[Anonymous Coward]

I don't think many people know, but 1/3 of Pittsburgh flight attendants called in sick on Christmas Day. That alone is a cataclysmic problem since that is a main USAir hub.

Maybe USAir doens't want anyone to know how much they suck as an employer aside from how they suck as a corporation. Their employees don't seem to ever smile - they probably have no reason to.

As a side note, the JetBlue CEO has been know to frequent flights handing out cookies, talking to passengers, perform baggage handling, and cleaning up trash after the flight. Every single employee looks happy...

Re:unreleased tidbit of information

[ArtStone]

Not to worry - US Air and Delta are about to reap the benefits of this activity - unless of course, the Federal Govt steps in again to reward failure.

Re:Signed or unsigned

[Vengeance]

In my day, our ints had 16 bits and we LIKED it!

Re:Maybe it had "worked just fine" for them?

[iamacat]

Maybe the existing system was working just fine? Upgrades too expensive?

That's fine to say when you are running a restaurant. If your system goes down, just take orders with a pencil and calculator until you get a new one. If you are running a critical infrastructure like an airline, its not enough that your computers work so far. You have to maintain a system based on modern software design, retain enough programmers who understand current code and do preventive maintanance on both software and hardware.

Oh, modern systems will still crash, but they are far less likely to run into unrecoverable errors and hard limits. There will be a log of all the submitted requests stored in redundant locations. If the database is totally corrupted, it can be still restored from a backup and the work since then re-done without losing any data.

US government should consider covering part of the expense of re-writting software in infrastructure companies, because the impact on the society - people not being able to get home for Christmas, businesses losing customers in whatever places they were going to - exceeds financial damage to the airline.

Re:There was a high profile example of this proble

[Politburo]

IIRC, what happened is that the operators would go back up on the display to change values that were inputted erroneously. However, the new values that were displayed on the screen were never passed along to the underlying code. So the operator thought "Whoops, lemme change that power setting", hit the up arrow and changed the value and then went along like normal. The machine didn't know the power was supposed to change, and blasted the patients with radiation.

I'm sure someone else will chime in with a link to the correct version of the story.

Re:Sure-fire way to keep terrorists off the airpla

[gelfling]

It's pure bullshit. The effects to the US from neglected to micromanage the companies themselves the way they believe they can micromanage their own customers to the edge of outright criminal behavior is the same. On what's the busiest travel day of the year the entire system comes to a screaching halt no different than the US declaring the grounding of all aircraft on 9-11. But hey, I got modded down by dumbass hillbilly redneck who thinks failure is a good idea.

Re:It shows what unions are all about

[2old2rockNroll]

Airlines are going bust thanks to incompetent management -- they cannot even keep afloat with the huge government subsides and loan guarantees that they've received for decades.

And yet Delta CEO Leo Mullin made $13 million last year while holding his hand out to Congress for money. The system is broken.

"Simple" Solutions

[Detritus]

It's possible that the capacity limits in the software were more complicated than what type was selected for a few variables. How many other things were dependent on the maximum number of changes per month? Increasing the limit may have resulted in a program that could no longer meet other requirements for memory usage and run-time. Crew scheduling sounds like one of those problems that doesn't scale linearly. The worst-case scenarios when the software was designed may not match present conditions.

Re:It shows what unions are all about

[cayenne8]

"Essentially, the major carriers are hamstrung by the unions, and they will not survive long term. Unions work by artificially limiting labor supply - but that doesn't work if there is not enough work."

That sounds like a very reasonable explanation. And please excuse my ignorance...as that I've never been exposed really to unions before. Why can't the carrier just drop all the union employees...and hire a new crowd. I know they sign contracts..but, those contracts have to expire don't they? Why don't they just refuse to deal with them on expiration of the contract and try to save their business?

It seems to me that the day of unions, which were needed and useful back in their day...need to vanish. They seem, as in this example, to be causing more harm than good, causing companies to fail...and removing jobs from the populus..

Re:There was a high profile example of this proble

[slamb]

We talked about the Therac-25 in Engineering I. What you said is likely true, but there were two additional faults we discussed:

• UI fault: it regularly put up a lot of error dialogs which looked the same. They could range from "invalid data; reenter the field" to "if you hit radiate now, your patient will die". Technicians used to the first kind would learn to ignore the dialogs...so they'd miss the second also.
• There was a metal plate that was supposed to be in place when using one kind of radiation but not the other. It depended on software to ensure that this plate was in place. There should have been a hardware interlock for a precaution so simple and so important.

Re:Maybe it had "worked just fine" for them?

[theshowmecanuck]

I don't know why the moderator knob said this was a troll. It is an interesting point. It points out that upgrading from old software technology sometimes has hidden advantages.

Re:It shows what unions are all about

[duffbeer703]

What was missing from the parent's post was that 80's deregulation created the market for the "discount" airlines.

And the reason that Southwest & JetBlue have lower costs is that they pay their employees dramatically less. Ground handlers and baggage handlers are often paid minimum wage.

The side-effect of low pay is high turnover... so the non-union airlines don't have to fund such high-cost frills such as pension and benefit programs... and things like disability insurance rates stay low because their workforce is young and healthy.

The wise men in government & airline executives are merely converting a "professional" industry into a fast-food like assembly line.

The same process has already taken place in other industries that once paid a living wage... legacy industries like textiles, meat packing and restaurants.

Re:98

[Marxist Hacker 42]

As if Linux or XP or Unix never overflows a 16 bit signed integer.....this is a QA problem with the outsourcing vendor, nothing more. Pretty damned common problem with CMM shops, near as I can tell.

Re:It shows what unions are all about

[duffbeer703]

Unions seem to believe that society owes them a living. The problem is that society (except in the form of government) is not a person, and so recognizes no debts. Fighting that is totally ineffective because there is no one to fight.

Why is so wrong for workers to associate and bargain collectively? And why is it ok for airline industry associations, music industry assocations, etc to spend billions lobbying congress and demanding government concessions?

The visionary management at the major airlines have had twenty years to react to deregulation... but chose instead to seek short-term profits.

Who knew the meltdown would happen?

[DenvilleSteve]

What I am not reading here and in the press is who at Comair knew the software had this drop dead limitation and why were they not prepared for it? My suspicion is that the FAA has such tight controls on software change management at the airlines that the Comair IT managers knew the problem could occur, but were powerless to modify the package to prevent the meltdown. -Steve

Re:Maybe it had "worked just fine" for them?

[Fallen_Knight]

If software is designed and built to last X years with Y employees, but some MBA or CEO or biz guy up the chain o command decieds that theres no need to upgrade, its all working fine... for now, it us that guys fault.

It is only a software design flaw if the system was buit to last long enough for that coutner to ever possibley overflow.

I'd be willing to bet it was a MBA's choice to keep the system up without updateing it/replaceing it to save money and it should have been replaced long ago. MBAs typically have no idea how software works and just go with whatever buzzword makes they giddy or the cheapest option.

from another post: "It is the version of SBS that pre-dated Maestro. It was brought into Comair in the early 1980's. It's written in FORTRAN and uses whatever record managment system that came with the compiler."

Just a tad old on a much larger airline... seems like a accident waiting to happen.

Re:It shows what unions are all about

[skink1100]

> Why is so wrong for workers to associate and bargain collectively?

The problem occurs when unions get greedy (which they do), and force the consumer to suffer as a result. Ask US Airways customers about this. My business suffered during the UPS strike a few years ago so that truck drivers making $60k a year could get even more.

S

Re:It shows what unions are all about

[llefler]

Why can't the carrier just drop all the union employees...and hire a new crowd.

Imagine the company you work for firing EVERY employee that is not a member of management at the same time. You can't interview/hire/train replacements ahead of time because the unions would walk out if they heard about it. And, assuming you managed to pull this off without killing your company, you'd spend the next 5 years in court fighting the lawsuits. You couldn't, for instance, lockout the pilots union and replace them because the baggage handlers and mechanics unions would walk out in sympathy (self preservation). It's an all or nothing plan, and extremely risky.

BTW, the original purpose of unions still exist, but not to the degree that it once did. Know anyone working in IT required to work 50hrs a week as a salary requirement as well as any other OT required to meet a deadline, on call 24x7 and required to carry a cell and/or pager? Ask the developers at Electronic Arts if a union wouldn't have it's uses.

Re:Comair?

[Gogo Dodo]

I've written longer, descriptive submissions before that have either been rejected and some shorter submission was put in it's place, or had the original article chopped up like what happened here. My original article said that Comair outsourced the system to SBS. The /. editors seem to like short, concise submissions, which unfortunately mean you have to follow a bunch of links to get the whole story.

Re:It shows what unions are all about

[WhiplashII]

My only comment on this is about the line:

...wise men in government & airline executives...

Perhaps you could blame the government (although I believe deregulation - which leads to lower cost airlines - is a good thing in this case), the airline executives fought against deregulation tooth and nail. It essentially made them give up their monopoly, and made them compete in an environment in which they knew they would lose. See Pan American airlines for comparisons.

Re:Comair?

[babbage]

Ahh, I see. So it's not your fault then, and we have another datum point to suggest that the editors are indeed being willfully obtuse with the house editorial style. How nice...

Re:It shows what unions are all about

[WhiplashII]

In addition to the comments above, what is wrong with a union is that it is a (typically government enforced) monopoly, without the necessary government regulation. Laws have been passed requiring management to respect unions, but no laws have been passed to restrict unions in any way. This is the same problem as Microsoft controlling the software industry - the little guy is squashed, inovation suffers (using computers to improve efficiency is illegal, for example), and reaction time to changing market conditions suffers (how long did it take for MS to make security #1, how long did it take airlines to lower cost structures).

Back when the economy was strong, I made several proposals for IT projects that could have saved millions of dollars. These proposals were shot down by the unions, because higher efficiencies would have lead to fewer jobs.

Why on Earth did we have to get the union's permission for those projects? No wonder Southwest is kicking everyone's butts.

Re:It shows what unions are all about

[WhiplashII]

I agree that unions have some positive aspects (just like Microsoft's monopoly helped defragment the computer industry), however I think that the problems outweigh the benefits. Anything that removes the basic link between productivity and pay is a problem.

I think that many of the problems that led to unions can now be solved, because of the change in how companies are formed. In the old days (TM), you couldn't form a new company unless you had the right parents, etc. Today, if you think you are underpaid, you realistically can start a new company doing your current job. If you can't do that and make your current salary, you are not underpaid.

Re:Comair?

[Gogo Dodo]

So it's not your fault

Some of it is, but I'm not taking all of the blame. :-) I will readily admit that I didn't put in something like your suggest amendment. However, I do know that the /. editors have a habit of changing submissions. At least mine they do, but I imagine they do it to others.

Your code sucks

[furball]

if (MAXINT - num_crew_changes < whatever_you_wanted_to_add_to_num_crew_changes) {
ERROR(E1234, "too many crew changes");
}

Think about it.

Re:Signed or unsigned

[cyberkreiger]

No, incrementing 0111 1111 1111 1111 (32767) would yield 1000 0000 0000 0000 which is -32768 in decimal. Assuming two's complement, which has no negative zero, of course.

And how much does a 747 cost?

[cheekyboy]

Compare the price of one 747 refuel, that would pay for 1gig of ram in 1989.

The ram limitation is in the server database, not client software. Surely they could afford to spend alittle more dont you, its not like they run out of fuel over the pacific do they.

Its just called CHEAP-ASS[tm] management. If they could, they would water down the fuel with anything, but its illegal.

And surely they could have recompiled the server software to support unsigned 16bit back in 1999.

This is all a symptom of Mr Manager doing cost cutting and getting great bonuss'

why 32k/month, not per week?

[cheekyboy]

Why per month? couldnt they just purge when it hit 32000 and continue fine, who cares if they loose the previous changes, back it up to a massive text file report.

They should have had a gracefull recovery/continue, unless the software tech support was on holiday in Phuket

australia does well... and makes profits

[cheekyboy]

Why cant thses big fatcat silly airlines in usa do what qantas does? if unions are the issue, can the big airline just start a small-subsidery or small seperate partnered airline? Then the new one can easily 'compete' with the parent and take non union labor and reuse the parents facilities.

Or are there FAA regulations preventing this? if so.. change the laws.

Blaming the unions for a bad management decision?

[dbIII]

Essentially, the major carriers are hamstrung by the unions,

It's often easier to blame the unions than fix bad management practices, and in a lot of cases competitors that have workers in the same union operate well and get the job done instead of throwing their hands in the air, blaming the unions, and refusing to move into this century.

Unions are also used to seeing promises that pay will go back to normal in good times broken. The reality of capitalism is that if you don't have your act together enough to meet a known wages bill your company is probably going to expire.

While unions are not the problem - some unreasonable bastard in a paticular union may be - but you get that in all kinds or organisations. It sounds like there is an "us or them" attitude going on, where each group hates the other, which can lead to all kinds of problems and the end of the company if it isn't sorted out.

The USA has all kinds of protections to stop better run airlines coming in from overseas to create even more intense competition. The land that gave us Valuejet and the mess that was United in its final years really needs to get its act together, stop blaming the unions, and see if they can do as well as any of a score of airlines that would be happy to come in as soon as deregulation happens and show how airlines work in the rest of the world.

When I go to the USA I'd better catch a bus, I bet the bus companys scheduling software is less than fifteen years old and has been updated if the company has grown - that's what most places do for business critical applications, and it has nothing at all to do with unions.

Self correcting - sort of

[dbIII]

The article says they had 1100 flights on one day.

I'm surprised this didn't happen sooner then. Not making sure that a business critical application keeps pace with expansion (for fifteen years?) is going to cost them at lot - but at least it won't have as many as 1100 flights a day to keep track of for now. I wonder how long it will be before they can fill that many planes again? This is the sort of stuff-up which will make a lot of people avoid the airline entirely.

Re:Self correcting - sort of

[R.Caley]

Not making sure that a business critical application keeps pace with expansion (for fifteen years?) is going to cost them at lot

Well, it seems they were about to replace it. If it took a year to get the replacement ready, plus a year before that doing the management dances to make the decision, they may well have been rather more comfortably under the limit when they realised they had a looming problem. It depends on the shape of their growth curve. Eg, 2 years ago they might have been on 500 flights per day, when someone realised that the just proposed expansion plans had IT implications and started running to deal with them.

The low-cost airlines are low cost because they do exactly this kind of tap-dancing on the edge of collapse. E.g. they run the minimum number of aircraft and crew they need to have a very very slim contingency, which is why they so often end up with delays when they have weather or mechanical problems. From here in Scotland to south east England I'd have the choice of BA and EasyJET. The price difference is significant, but if I absolutely had to be down there on time I'd go BA and pay the difference. For a non critical meeting or family visit, I'd take EasyJET and gamble the delay against the saving.

Deja Vu All Over Again

[SiberianPgmr]

It has happened before and it will happen again. The biggest case of a transaction counter overflowing a 16 bit signed integer was in 1985 at the Bank of New York - they came up 32 billion dollars short and nearly caused a collapse of the financial system. A description of the problem - Washington Post, 13 December 1985, p. D7 - Computer Snarled N.Y. Bank - $32 Billion Overdraft Resulted From Snafu - can be found in a message at: http://www.mirrors.wiretapped.net/security/info/te xtfiles/risks-digest/1/risks-1.31

Re:Signed or unsigned

[Jim Hall]

Whoops, my bad. Got my binary math wrong.

Re:It shows what unions are all about

[duffbeer703]

Until government enacts and reliably enforces laws to prevent employers from intimidating and abusing their workers, unions are needed.

Society has an obligation to provide more than just a capital environment to improve business. We have a responsibility to improve the lot of our fellow citizen, and the corporations stand opposed to that. IBP, Wal-Mart, McDonalds and Electronic Arts all demonstrate that.

Re:New CIO?-Specialization.

[cheekyboy]

At least us programmers could learn these dumbasses jobs quicker than they could learn programming, ie 10 years of coding, vs 6 months crash course in "bullshiting and running things" Central Idiot Officer more like it. Well at lease once there is a global depression or something as bad a tsunami, then we can all camp out at the CxOs mansions.

Re:It is not a trivial system

[cheekyboy]

All this should already be inside some massive spec document with UMLs etc..

Re:It shows what unions are all about

[OldManAndTheC]

Society has no obligation to me, and I have to obligation to it.

No man is an island.

This counter...

[Scooter]

..it wasn't labelled something like "TrialMinutesLeft" or something was it? :P

Re:There was a high profile example of this proble

[Lehk228]

one (of many) Big problem there is a safety critical system using input rather than configuration state to control the Status display, hell the status display should not even be aware of most input commands, except by way of detecting the resulting configuration changes, and parhapse having the input system notify the status display that "something has changed, check status"

government propping up failing businesses

[jeif1k]

Unions seem to believe that society owes them a living.

So do corporations, apparently.

The only thing management could have done would be to have rejected the union contracts earlier.

If management made contracts that they couldn't keep, they screwed up and their companies should go out of business. That's the way the free market works.

In fact, in reality, big airlines are screwing up not only on their labor contracts, they are poorly run, heavily government subsidized businesses offering mostly uncompetitive transportation options. The sooner the big airlines go out of business, the better off we all are.

Unions work by artificially limiting labor supply - but that doesn't work if there is not enough work.

Unions are a free market mechanism by which employees get together and bargain collectively. That's not very different from shareholders forming corporations in order to make collective contracts. Unions "artificially limits the labor supply" about as much as car makers "artificially limits the car supply" when they set a price for their new SUVs.

What it comes down to is that you want intrusive government: you want government to prop up failing, uncompetitive businesses, you want government to protect management from the consequences of their bad decisions, and you want government to prevent employees from associating freely and making a free-market choice to bargain collectively. Come on, admit it to yourself: you are just another one of those typical intrusive-government Republicans.

No, I didn't...

[carpus]

Actually my friend, I heard it on Bugtraq first.