Computer Error Grounds Japanese Flights

zephiros writes "Mainichi Daily News reports that a "computer glitch" in Tokyo air traffic control systems resulted in the cancellation of 203 flights this weekend. At 7am Saturday, the error "caused the names of airlines and flight numbers to disappear from radar screens." A Japan Times article suggests the problem may be related to upgrades on a system which exchanges flight plans with the Defense Agency. Makes one wonder about the integration and maintenance risks of systems like CAPPS II."

Comments

Can we say change control.

[hsidhu]

Ummm,

Don't they have change control.

whatever.

Risk Maintentance 101

[gostats]

I've work quite a bit with risk maintenance. Most often situations like these increase the budget for disaster prevetion and other related expenses. This failure *should* make fewer failures in the future and generally a safer airport. But then again that all depends on how much passion they have for their job.

Maybe I should take a trip to Japan in a few months.

Re:Risk Maintentance 101

[kryonD]

Actually, the damage was almost minimal to the Japanese air system. The delay only lasted 50 minutes. Unlike American travellers, Japanese people will quietly and orderly board a fully booked 747 in under 20 minutes. If asked to hurry, they will board it even faster. That combined with Narita and Haneda's ability to handle traffic far above their average had most flights back on time before noon. Only a small handful of international passengers may have had to rebook a connecting flight. Domestic flights are almost always direct.

As far as risk management, had there actually been a perceived emergency due to the malfunctioning radar display system, the airports would default to an agreement with Yokota and Atsugi US airbases to provide fallback flight control facilities.

This is really a non news item. The system administrator correctly applied upgrades during non-critical operation time. (i.e. not during the main business week) The problem was identified early on and corrected pretty damned quick. This happens hundreds of times a week all over the world. Had the glitch actually halted the entire Japanese air system for a long period of time, then it would make more sense.

Re:Windows?

[SILIZIUMM]

Sure look at the photo : http://web.ukonline.co.uk/eric.price/humour2/AIRPO RT.jpg

Why?

[Cyno01]

Why even bring up CAPPSII, is has nothing to do with air control, only with passenger data.

Re:Why?

[abh]

Because this is Slashdot, where every article must somehow involve a violation of rights by the big bad government.

Re:Why?

[56]

Because the program that caused the error was similar to CAPPSII.

To quote the article:

A Japan Times article suggests the problem may be related to upgrades on a system which exchanges flight plans with the Defense Agency.

Re:Why?

[Anonymous Coward]

Pffft!!, CAPPSII exchanges flight manifest information,
which has nothing to do with air trafic control. The
Japanese system exchanges flight plans which has
everything to do with air traffic control. The situation
is not even remotely similar.

Re:Why?

[zephiros]

Because it's a large system that will have to integrate with numerous airline systems from god knows how many vendors. And it will need to be maintained and patched. And it's a potential single point of failure (from a software standpoint; obviously they could stripe it across as much hardware as needed).

Even if CAPPS is only connected to ticketing and passenger information, a bug could result in a pretty nasty transportation snarl. Suppose airlines are unable to issue boarding passes for an hour, or an unusually large number of people were flagged for screening.

For any of these total-information-awareness type systems, one has to ask "what happens when some part of the patchwork breaks?" Even the most diehard "I have nothing to hide from my government" type understands that multi-hour flight delays are bad.

Re:Why?

[Anonymous Coward]

Ok, fine, so this *specific* problem will not accur again. The point is that these are tricky systems and adding levels of complexity increases the odds of a fuckup.

Re:Why?

[Anonymous Coward]

That means absolutly nothing. No relationship whatso ever. BTW, I am a Nav Aids specialist. Civilian air traffic information is ALWAYS provided to local defence organizations, AND vis-versa. How in the hell do you think Military Air Traffic control insures that exersises don't impact civilian flights? Ideally the civilian and military "towers" are connected by a dedicated mWave link. The error happend with an upgrade of the existing system, NOT the addition of some new Orwellian evil.

Re:Why?

[Anonymous Coward]

Why am I suprised that the nitpickers came out of the woodwork over this?

I was simply pointing out why there was a comparison made between the error grounding Japanese flights and CAPPS II.

I made no statement about whether or not this was feasible, or even possible, in reality.

But in serious response to your post, fine, so the actual system for sharing information with the Defense department wasn't the cause, the implimentation of that system was the cause. Big whoop, an error is an error is an error.

Re:Windows?

[SILIZIUMM]

Oh, forgot to add : *hides*

What does CAPPS II have to do with this?

[revmoo]

"Makes one wonder about the integration and maintenance risks of systems like CAPPS II."

Does that seem like flaimbait to anyone else? Computers crash all the time, granted steps can be taken to ensure redundancy, but this is nothing new. This problem has nothing to do with the CAPPS II system other than the fact that they are both computerized systems, I'm not trying to defend CAPPS II, I just don't think that it is any way related to this this tokyo airlines problem. Computers crash, it's a fact of life, the real question here, is why weren't there multiple redundancies in place for such a mission critical application.

Re:What does CAPPS II have to do with this?

[Phroggy]

Computers crash, it's a fact of life...

Been listening to Microsoft too much lately, eh? It shouldn't be something we take for granted.

Re:What does CAPPS II have to do with this?

[revmoo]

I take it you've never had a kernel panic then...

What about hardware failures? Even the best code still has bugs in it, and the potential to fail

Re:What does CAPPS II have to do with this?

[Anonymous Coward]

Even the best code still has bugs in it,

I've never been able to crash helloworld.c.

Re:What does CAPPS II have to do with this?

[frdmfghtr]

Looks like flamebait to me...I don't see how anybody in their right (and even not-so-right) mind would interface CAPPS II with the ATC network. Thus, as such a lunatic concept, I discount the Japan Times comment that the two are somehow related as the words of somebody who "knows not of what they speak."

Do YOU want the ticket agent behind the counter to have access to a network that is interconnected with air traffic control? Neither do I.

Re:What does CAPPS II have to do with this?

[frdmfghtr]

Allow me to clarify...when I said "interface to CAPPS II" that really should be read as "interface passenger flight tracking"...I made the mistake of using CAPPS II as a generic term for travel agency database stuff.

Re:What does CAPPS II have to do with this?

[Stween]

But hardware really shouldn't fail as much as we're used to, and we really shouldn't have to put up with the number of bugs even the `best' code has.

Seeing as we're still in a young industry, it's nice to think that over the coming years things *should* get better, on the hardware side at least.

The problem with software development is that it's so damn easy to ship a product, let hundreds (thousands? tens of thousands?) of users test the product for you, then release appropriate bug fixes. People don't design a new bridge and let thousands of people try it out before declaring it safe, nor when building new buildings, and usually not even when designing a new microprocessor, so why are we still getting away with it in software engineering?

Presumably because Microsoft has let the general public get used to this as "the way it is" ;)

Re:What does CAPPS II have to do with this?

[AlephNot]

"'Even the best code still has bugs in it' I've never been able to crash helloworld.c."

Uh, I wouldn't refer to helloworld.c as "best"... I think the grandparent was refering to the fact that larger (that is, more complex) programs are more likely to have more bugs in them.

Re:What does CAPPS II have to do with this?

[56]

Because there is a possibility that the error was caused by a problem in a program similar to CAPPS II.

To quote the article:

A Japan Times article suggests the problem may be related to upgrades on a system which exchanges flight plans with the Defense Agency.

Re:What does CAPPS II have to do with this?

[gilesjuk]

"It's a computer glitch", "the computer made a mistake" etc.. I hate such lines, computers don't design themselves. Humans design processors, humans write software.

Such news articles should say "hardware failure" if it is a hardware failure, if a computer crashes due to software it should say a bug in the software caused the problem.

Re:What does CAPPS II have to do with this?

[gilesjuk]

Exactly, the term "fail safe" doesn't seem to be known by some people.

Brake systems in cars don't fail safe, yet they do on some trains. This isn't a fault of braking systems, just the application of the system in one vehicle.

Re:What does CAPPS II have to do with this?

[The_dev0]

Yeah, but that is just a cover for this conspiracy. I was on one of these flights, and the captain announced:

"Uh, folks, we're experiencing some moderate Godzilla-related turbulence at this time, so I'm going to go ahead and ask you to put your seatbelts back on. When we get to 35 thousand feet, he usually does let go, so from there on out, all we have to worry about is Mothra, and, uh, we do have reports he's tied up with Gamera and Rodan at the present time. Thank you very much."

Don't believe the lies!

Re:What does CAPPS II have to do with this?

[Anonymous Coward]

WEll Skippies mom only lets him use the computer between 7 and 8pm. SO thats not really long enough for him to get it screwed up.

Re:What does CAPPS II have to do with this?

[HerbieStone]

Computers crash all the time, granted steps can be taken to ensure redundancy, but this is nothing new.

Well for one, who said some computer crashed? From what I understood from the story. The system had a glitch after a software upgrade. So it seems the new software is at fault. That's not the same as some hardware failure. How do you fight a sysematic error in software with a redundancy?
The trick with software upgrades is to do some testing in as-real-as-possible environment before going live.

Re:What does CAPPS II have to do with this?

[mpe]

The problem with software development is that it's so damn easy to ship a product, let hundreds (thousands? tens of thousands?) of users test the product for you, then release appropriate bug fixes.

In the case of some proprietary software either charging for the bug fixes or intermingling them with new "features" (which may very well contain their own bugs). That's assuming they don't insist on playing "it's a feature not a bug".

People don't design a new bridge and let thousands of people try it out before declaring it safe,

There is a story that in the USSR once a bridge was completed the designers and archiects had to stand underneath whilst the Red Army drove a column of tanks across.

Re:What does CAPPS II have to do with this?

[HiThere]

...Computers crash, it's a fact of life, the real question here, is why weren't there multiple redundancies in place for such a mission critical application.

I'm guessing that the computers didn't crash. The description sounds more like software bugs. Still... one would think that for a system like this they would have done the upgrades on a duplicate server, and only switched over to them after all checks had been passed.

(Perhaps the only thing CAPPS II has to do with this is that it takes a currenly extrememly complex system and makes it more complex without sufficient testing. That would be enough similarity, but perhaps the comparison should have been more explicit.)

Computers are just too fragile?

[caluml]

"Computers are just no good," said one 51-year-old company manager leaving for Sapporo. "I'm sure they're helpful, but they're just too fragile."

Lol. Depends how they're set up. I'd say you can get them fairly robust. Clustered, load balanced, hot-swap, failover, etc.

Re:Computers are just too fragile?

[WegianWarrior]

Computers are way to fragile - I just broke my third mice in two years...

Re:Computers are just too fragile?

[caluml]

Your hot-standby mouse should have kicked in and taken over without you noticing.. ;)

Re:Computers are just too fragile?

[Anonymous Coward]

And I've been using the same one for 4 years...

Re:Computers are just too fragile?

[delta407]

Clustered, load balanced, hot-swap, failover, etc.

Yes, but you have to remember: the second most common cause of computer failure -- after hardware -- is operator error.

From the article:

The system has a backup, but both systems went down at the same time, according to the ministry.

Simultaneous fialure of two independent systems that are designed to tolerate failure doesn't sound like a hardware issue to me. I could be wrong, but I'm more than willing to bet it's an "oops!" situation, albiet somewhat more serious than most.

2 things I want to know...

[Spazholio]

1) How the hell did the flights get DOWN once the radar died? It said they disappeared from radar, and you don't keep radar on the planes that are on the ground, so....?

2) Whose bright idea was it to do a "systems upgrade" while there were large, flying metal objects carrying many people still in the air?!?! Wouldn't you do a test run, install it on a backup system, or one that's not systems-critical?

This just makes no sense....someone explain it to me?

Re:2 things I want to know...

[Geaty]

1: Because the planes themselves still showed up on radar, air traffic control authorities were able to direct flights after contacting aircraft to determine their flight numbers and other details, but all departing flights were grounded as a precaution.

Re:2 things I want to know...

[Anonymous Coward]

> 1) How the hell did the flights get DOWN once the radar died? It said they disappeared from radar, and you don't keep radar on the planes that are on the ground, so....?

Read the article. It says that just the airline name and flightnumber tags printed beside the radar blips vanished. The radar worked just fine.

> 2) Whose bright idea was it to do a "systems upgrade" while there were large, flying metal objects carrying many people still in the air?!?!

Read the article. The change was made early in the morning on a weekend. When would you suggest?

> Wouldn't you do a test run, install it on a backup system, or one that's not systems-critical?

The article (did you read it?) hints that might have been a networking problem when they integrated the military database with the civilian database. A backup system is a good first start, but isn't always the same as the production system. Network problems can't always be perfectly tested or simulated.

Re:2 things I want to know...

[Spazholio]

re: 1) Ok, the airline names and tag numbers are important when bringing them down, so although not impossible, it would have been a bitch to do.

re: 2) Let's see, early morning on a weekend. Weekends are known to be busy times for airports and morning is when they tend to fly. When would I recommend? Maybe a Thursday? Middle of the night?

re: 3) Bullshit. The backup system should be IDENTICAL to the production system, otherwise it's a worthless backup. Especially in something as critical as ATC. But I will concede the problems with testing network problems being hard to simulate.

Re:2 things I want to know...

[tang]

"Bullshit. The backup system should be IDENTICAL to the production system, otherwise it's a worthless backup"

Well, it depends what the purpose of the backup is. If the backup is perfectly identical to the main system, that means when something kills the mainsystem,(say a bad string of data is being passwed around the network) you switch to the backup, and it has the same flaw,so it dies too! Thats why its sometimes nice to have a backup system that uses slightly different code/hardware.

Re:2 things I want to know...

[mickwd]

"Whose bright idea was it to do a "systems upgrade" while there were large, flying metal objects carrying many people still in the air?!?!"

Actually, there are planes in the air most hours of the day. There is no time when planes aren't flying.

The best time (when there are fewest planes) may be at night. But that's just the time when the people actually doing the upgrade are going to be half asleep.

"Wouldn't you do a test run, install it on a backup system, or one that's not systems-critical?"

I'm sure they did. But the live system is bound to be different in some small way
- maybe a different (more powerful) system, which might cause different timing issues;
- maybe a different disk configuration, perhaps with a file system running out of space (e.g. more online logs);
- maybe the live database (if any) is different to that on the backup system.

These things can easily go wrong. In my experience, it's vital to ensure you have a way of getting some sort of system operational if you do screw up. Maybe ensure a backup system is capable of running live first, then attempt the update of the live system, and if it goes wrong, you have a backup system capable of operating until you can correct the screw-up.

Re:2 things I want to know...

[Rantank]

1) How the hell did the flights get DOWN once the radar died? It said they disappeared from radar, and you don't keep radar on the planes that are on the ground, so....?

Big airports DO have ground radar... they like to keep track of everything either in the air or on the ground. I would imagine they use different displays for each radar given the differences of scale, but the data is probably merged. A plane attempted to take off in Taiwan in a storm in low visibility, ended up on the wrong runway and people died. The lack of ASDA radar meant the tower couldn't track their ground path and spot the pilot error.

2) Whose bright idea was it to do a "systems upgrade" while there were large, flying metal objects carrying many people still in the air?!?! Wouldn't you do a test run, install it on a backup system, or one that's not systems-critical?

I would expect a control tower's computer systems to have the tightest change control procedures possible. Unforeseen situations can occur no matter how careful you are. What matters is what procedures you have in place to deal with it when everything goes pear-shaped. Given no planes fall-down-go-boom I think their procedures worked out quite nicely. There's nothing in the article to say the "glitch" occurred at the time of upgrade or whether the upgrade was done earlier and the computer waited until 7am to stick it's finger in it's ear....

A Control Tower's ability to identify planes isn't magically connected to their ability to fly... and I think they're fairly good at teaching pilots how not to bump into things...

Re:2 things I want to know...

[Mr. Ophidian Jones]

Actually, there are planes in the air most hours of the day. There is no time when planes aren't flying.

Except on 9/11. In fact, scientists used this to determine how having no jet contrails in the air affected the daily high temperatures.

Re:2 things I want to know...

[kryonD]

Thursday: middle of the business week. Would you want construction on the road you take to work on a work day?

Midnight: Upgrade to a system that tracks planes in the air. If there are no planes in the air, how can you test it? The upgrade was most likely done around midnight, but didn't see any signifigant use until the heavy part of the travelling day. The article indicated it was due to an interface with a defense system. JIEITAI (Japanese military) usually hit the office between 7 and 8, which was when the delays started. I could be wrong about the work time for the folks in Tokyo, but the two units I've worked with in Okinawa and Hokkaido were usually rolling in to hit their first cup of coffee just after 7.

Despite the lack of information in the article, passengers were not stranded. If their flight was cancelled, they were re-booked and most likely in the air within the hour of their original scheduled flight. Japanese air systems are extremely efficient, although I have no clue how they could be making any money.

Re:2 things I want to know...

[mpe]

1) How the hell did the flights get DOWN once the radar died? It said they disappeared from radar, and you don't keep radar on the planes that are on the ground, so....?

They still had working primary RADAR and radios. Aircraft on the ground are managed by controllers at the airport itself.

2) Whose bright idea was it to do a "systems upgrade" while there were large, flying metal objects carrying many people still in the air?!?!

They did it on a Saturday.

Re:2 things I want to know...

[Anonymous Coward]

2) Whose bright idea was it to do a "systems upgrade" while there were large, flying metal objects carrying many people still in the air?!?! Wouldn't you do a test run, install it on a backup system, or one that's not systems-critical?

So when is there a time when there aren't aircraft in the sky? This is the sort of real-time application where down-time is not an option. The system must be upgraded or switched over while running live. Yes, it takes specially designed systems to do this properly. No, it's not easy. No, you don't want to try this on a PC, no matter what OS you're running.

secret scientology weirdess

[Anonymous Coward]

what are these guys on?

redundancy

[Brigadier]

Am I the only one wondering why there was no redundancy. As in effective. One would think something as important as airtraffic control should have several layers of complete redundancy. As in if a control tower has say a catastrophic failure there is another a.) civilian b.) military control center able to hand off instructions. which would include all flight information. including passengers, cargo flight log, flight plan everything.

Re:redundancy

[Mr Rohan]

Am I the only one wondering why there was no redundancy.

Typically there are redundant systems as well as manual processes - in Sydney Australia there's even a redundant tower, which is used if the main tower stops working (e.g. major power problems).

Re:redundancy

[Anonymous Coward]

Keep in mind simple redundancy doesn't solve all of your problems, especially software problems.

If you've made your application fully redundant - multiple nics, raid, multiple power supplies, failover clustering, multi provisioned and colocated, then all of your systems will still have the same bugs.

Re:redundancy

[oh]

From memory, the backup computer crashed as well.

From the Australian ABC

Due to a reprogramming hiccup, the main system and its backup went down immediately after being switched on

I presume they made a change to both systems, or more likely, the backup system was also connected to the military system and also choked on the data is was being fed.

Re:redundancy

[neier]

From what I heard on NHK news Saturday, there was a redundant system -- but it also received the same system upgrade early Saturday morning. So, two broken systems is really no better than one.

Re:redundancy

[Anonymous Coward]

Actually, there is a lot of redundancy. That the artical did not touch on any, does not mean it does not exist. That flights were canceled, is more an indication of the conservative nature of Japanese Airport Managment. Safety, is far more important then a few yen. Air Traffic Control proceduress has condtingincies for all sorts of nastiness, including something as relativly minor as loosing flight identification information on the displays. One of the reasons for the existance of the system whos upgrade might have coused the problem, is to enable another local ATC facility to take over in the event of a catastrify. This is standard. What most of the /. crowd, does not realise is that Military ATC does periodicly support civilian traffic. A good example of this was during the early 80s with the PACAF strike. A good portion of Air Traffic Control was performed by USAF Air Controlors.

Pedantic Bastard to the rescue!!!

[Spazholio]

His name is Linus, not Linux. Linux is the operating system, Linus is the person.

Re:Pedantic Bastard to the rescue!!!

[Anonymous Coward]

Linux is the kernel, not the operating system.

Re:This is wonderfil news for opensource!

[fewnorms]

Euhhh . . . . wasn't that Linus Thorvalds? =]

ATC and CAPPS II are NOT connected

[MyNameIsFred]

Obviously upgrades to Air Traffic Control (ATC) systems and communication links to ATC can be cause problems. There is a significant safety of flight issue. Therefore, the FAA maintains strict control of these systems. And in fact, has a dedicated network reserved for ATC. Only "essential" programs and systems are allowed to connect to it.

Passenger listings, airline booking systems, and related software are NOT connected to the ATC network. Since CAPPS II looks at booking data, credit card info, and related data, it would not be connected to the ATC network.

Re:ATC and CAPPS II are NOT connected

[mlknowle]

True, but both are, after all, major systems in the air transportation process. A failure in either one has the potential to be catestrophic; if the ATC system screws up, planes could collide, and if security screws up, a dangerous passenger could be let through. The difficulty is in the political nature of the second system; there is no one who disagrees that planes should be kept apart, and no one disagrees that terrorists should be kept off airplanes (well, except the terrorists - they might disagree with both assertions, but that's another matter).

Anyway, the worst hassles of a functioning ATC system are long delays, holding patterns, etc, while the security system has much more troubling implications - which are obvious to anyone. So I guess, in a way, it's better that the ATC system failed, and no one got hurt, than a security failure ...

how do they test the system?

[NotAnotherReboot]

Out of curiousity, how does one go about testing a system like this? Do they test changes to the code in a live system? (not using the newer version, just looking at it along with the old one). Are there flight emulators that will feed fake data to the software which in turn displays what it is receiving? Do they do extensive testing between new systems that perform different functions yet interface as well? It seems to me a large part of the budget for these projects has to be testing.

Re:how do they test the system?

[roalt]

With new systems, they normally do shadow-mode testing: this means, running the new system with ATC controller behind it in parallel. After a while, the new system will become the default system and the old system is also run in parallel but now as backup system (with real ATC controllers checking if everything goes okay). After a while, the new system is considered O.K. and the backup system is removed.

I don't think they do this with maintenance updates, but maybe they should consider it. Unfortunately, air traffic controllers are still rare and (not in the least) expensive.

Overheard, Call to help center

[Anonymous Coward]

1. Sir, Is your computer plugged in?
2. We are going to need some registration information before we get started.
3. Oh, we don't support that OS
4. Anything else, have a nice day

Anyone see the other news on this site?!

[caluml]

Anyone see the other news on this site?!

Police recover rock climber's body after fatal fall
Motorcyclist dies after being hit by a truck
61-year-old jobless man fatally abuses senile mother
Dad dies of shock after son's repeated beatings
Comic questioned over hitting woman in restaurant
Death row inmate dies in prison cell

Can someone in Japan please confirm that this is a freaky, awful day, and that Japan isn't normally this bad?

Although that last one is quite ironic.

Re:Anyone see the other news on this site?!

[kfg]

Looks like a perfectly normal day, pretty much anywhere, to me.

Yes, I'm serious. Read your own home town newspaper if you don't believe me.

KFG

Re:Anyone see the other news on this site?!

[Anonymous Coward]

Here are the first "news" items from Bristol, UK.

Customs officers stop a 46-year-old woman at Bristol airport and find £96,000 in her hand luggage.
Operatunity winner 'floating on air'
Man, 83, charged with murder
Patients' phrasebook aids 999 crews
A future for Concorde?
South Gloucestershire council tax up 6.1%

How dull is that?
http://www.bbc.co.uk/bristol/content/news/

Re:Anyone see the other news on this site?!

[nomadic]

Yeah, I think this paper just made my periodical reading list.

Re:Anyone see the other news on this site?!

[Anonymous Coward]

That's OK. Mr. Shrub will whip out some nukes and give the papers a piece about truth, justice, and the American Way in a few months. Death to infidels!

Re:Anyone see the other news on this site?!

[Anonymous Coward]

There hasn't been a murder in this town for decades.
There hasn't been anyone killed by being hit by a car in the entirety of my life.
Shall I go on? If urban cessholes are all like that, it just makes me all the happier I'm not there.

Re:Anyone see the other news on this site?!

[hey]

You missed... Dominatrix whips up donations for refugees ... now that's a headline!

Smart japaneses

[Anonymous Coward]

They agreed to give access, but did they agreed to not hide names? :)

Re:This is wonderfil news for opensource!

[Anonymous Coward]

pass the crack pipe this way please

The explanation

[anon*127.0.0.1]

I think it's obviously Y2K related. Civilization as we know it should be coming to an end in a week or so.

Who needs computers?

[ignoramus]

"Computers are just no good," said one 51-year-old company manager leaving for Sapporo. "I'm sure they're helpful, but they're just too fragile." Uh, yeah, I also have a feeling they may be a little helpful. Good luck controlling 70 percent of all air traffic in Japan with abacii and the Everyday Memory Builder...

I just dropped a floppy from 2-in above my desk

[baywulf]

and it shattered into pieces. Talk about fragile.

ATC?It's a big tower, but that's not important now

[Alien Being]

Loger Murdock: We have crearance Crarence.
Captain Oveur: Loger, Loger. What's our vector Victor?
Tower voice: Tower's ladio crearance, over!
Captain Oveur: That's Crarence Oveur! Oveur.
Tower voice: Loger.
Roger Murdock: Huh?
Tower voice: Loger, over.
Roger Murdock: Huh?
Captain Oveur: Huh?

some glitch

[LuxFX]

If this was an error in the code, then how were they able to repair it in just 54 minutes? That's a pretty narrow window when it comes to rounding up the programmers, searching through the source, then repairing, testing, redistributing to the entire system, and rebooting the whole thing.

Kind of like how Hugh Jackman can hack into the DoD from a computer he's never touched before in Swordfish.

I'm tempted to think that this was much more human error than a bonefide "computer glitch". Maybe that 54 minutes was the time it took to call in their expert, have him look at the system, and declare "Why, you must have hit F11, which toggles the flight information. Just hit it again and it comes back."

Re:some glitch

[DASHSL0T]

According to the article I read, it took around four hours to fix it. (7 am to 11 am). About 30 minutes after the problme occurred, they began routing flights manually, using 10 minute delays between each aircraft.

Re:some glitch

[Anonymous Coward]

They wouldn't have to round up programmers and do all that, just install the version prior to the crash.

Re:some glitch

[secolactico]

Kind of like how Hugh Jackman can hack into the DoD from a computer he's never touched before in Swordfish.

Keep in mind that they had the advantage of not being "serviced orally" while they were at it.

*ducks*

The real reason the flights disappeared!

[Wiz]

Godzilla of course, obviously he is currently running rampent over Japan!

You have seen the Simpsons episode where they go to Japan right?!

WaiWai

[Anonymous Coward]

Prob'ly mildly offtopic, but whenever I see "Mainichi Daily News", I think of their Wai Wai section, which is full of crazy stories with headlines like "Schoolgirls selling panties open avenue of danger" or "Louis' leathers lure lasses into luxurious lunacy". It's the kind of Jap-weirdness that occasionally shows up on FARK.

Yes, it is that bad

[Anonymous Coward]

I've lived here for several years now, and the above stories really are an average selection. On a true freaky, awful day, you would see stories far worse.

Computer or Programmer error?

[Technomancer]

Was it computer that failed some operation or lousy programmer who made a mistake in the program?
I am sick of people complaining abour "computer errors" when they are at fault.

Re:Ban this evil game!

[cowlum]

huh? what you talkin bout willis...

mayhem, madness and sailor suit fetishes.

[chloroquine]

Wild boar attacks 6, leaves 2 seriously injured
Strong winds strike fatal blow to do-gooder
Public servant disguised as delivery man rob mother, daughter
Osaka legislator slurs Asians
Computer cock-up continues plaguing domestic travelers
Prison death trial to see violent video images
Marine steals from cabby
Man sits by unaware as neighbor plunges to death
Jilted man busted for forcing woman to wear sailor suit
Old man slits own throat with paper cutter
Women call for sex scandal governor's resignation
Education evolves from coeducation to social equality

And the rest of the items were similarly strange. The thing is, you know how you watch the local news on television and they only seem to report items involving spectacular suffering or small fluffy animals. I think they get extra points if they can find a small fluffy animal suffering. The news items seem to be the standard fare but without the feel-good pieces that we're (that's the royal we, if you don't agree with me) used to.

THIS is why you don't upgrade

[NineNine]

... or at least upgrade as little as possible. No matter how much planning and testing is done, upgrades can and will screw things up. I'm always reading about , "luckily, you can recompile the new kernel every week or so", or, "a new version is coming out so I have to upgrade" and I'm thinking... yeah, at home, maybe, if you have nothing better to do. But this is an extreme example of why companies that are worth their salt don't upgrade at the drop of a hat.

Re:THIS is why you don't upgrade

[Anonymous Coward]

Yeah... real smart asshole: "Don't Upgrade". Your box is going to get rooted so many times, it's not even funny. Your pathetic little shit-ass porno site sucks ass Tom. Go fuck your mom, I got her ready for you. Everyone else, skip NineNine's site and go here, it's WAYYYY better and not infected with putrid advertising:

http://www.sublimedirectory.com/

Stupid fucker.

Re:This is wonderfil news for opensource!

[Anonymous Coward]

No, it's Linus Torvoldes

DATELINE: JAPAN

[infonography]

DATELINE: JAPAN (maybe)

Computer related story about a programming error halting Air traffic control system in Japan is entered in a pre-posting queue to Slashdot.

DETAILS: Limited and not noteworthy.

REAL NEWSWORTHYNESS: Not really. No deaths reported.

DATELINE: SLASHDOT HQ

PREPOST WORD SEARCH: code runs check for Important items. - keyword search generate matches for two known hot item words [COMPUTER & JAPAN]

HENTAI AND GIANT ROBOT FACTOR?: n/a

CUTE BABE?: n/a

SEARCH FOR BIG NAMES- JOBS, ELLISON, GATES, TORVALDS, STALLMAN, CowboyNeal?: n/a

Microsoft Bashing Factor: High

PRIMARY ACTION TAKEN: Story authorizes posting of story to Slashdot

SECONDARY ACTION TAKEN: activate Inquisitors of the Holy Order of Linux, First Poster Squad IM'ed, new Sex story featuring Whicky the slashdot cat beta authorized.

STATUS REPORT: Status Quo Achieved.

RESOLUTION: Computer error found between keyboard and chair

Model of a Slashdot Personality

[Anonymous Coward]

to the tune of: "Model of a Modern Major General", with apologies to
Gilbert and Sullivan...

Model of a Slashdot Personality

I am the very model of a Slashdot personality.
I intersperse obscenity with tedious banality.
Addresses I have plenty of, both genuine and ghosted too,
On all the countless topics that my drivel is cross-posted to.
Your bandwidth I will fritter with my whining and my sniveling,
And you're the one who pays the bill, downloading all my driveling.
My enemies are numerous, and no-one would be blaming you
For cracking my head open after I've been rudely flaming you.

I hate to lose an argument (by now I should be used to it).
I wouldn't know a valid point if I was introduced to it.
My learning is extensive but consists of mindless trivia,
Designed to fan my ego, which is larger than Bolivia.
The comments that I vomit forth, disguised as jest and drollery,
Are really just an exercise in unremitting trollery.
I say I'm frank and forthright, but that's merely lies and vanity,
The gibberings of one who's at the limits of his sanity.

If only I could get a life, as many people tell me to;
If only Mom could find a circus freak-show she could sell me to;
If I go off to Zanzibar to paint the local scenery;
If I lose all my fingers in a mishap with machinery;
If I survive to twenty, which is somewhat problematical;
If what I post was more mature, or slightly more grammatical;
If I could learn to spell a bit, and maybe even punctuate;
Would I still be the loathsome and objectionable punk you hate?

But while I have this tiresome urge to prance around and show my face,
It simply isn't safe for normal people here in cyberspace.
To stick me in Old Sparky and turn on the electricity
Would be a fitting punishment for my egocentricity.

I always have the last word; so, with uttermost finality,
That's all from me, the model of a Slashdot personality.

THIS IS YOU

Slashdotted

[banzai75]

At 7am Saturday, the error "caused the names of airlines and flight numbers to disappear from radar screens."

I'm guessing there was an article posted yesterday on Slashdot that linked directly to their system.

Re:Model of a Slashdot Personality

[Anonymous Coward]

Man I gotta read the AC comments more often. That was pretty good till the half way mark.

Re:ATC?It's a big tower, but that's not important

[Anonymous Coward]

I get the Airplane reference, but I don't get the switched 'r' and 'l' letters... there's too many to be considered a typo. Anyone want to enlighten a slightly embarrased laymen?

Nowadays, it's easy!

[Anonymous Coward]

1) type "www.windowsupdate.com" and hit return - 2 seconds
2) download minor patch - 20 minutes, assuming T3 link
3) Read EULA, try call lawyer (not working Sunday), decide "aw fsck it" and Click OK - 12 minutes
4) "Windows is inspecting your hardware" (and software and calling home and installing DRM) - 16 minutes
5) Reboot - 2 minutes
6) Reboot - 2 minutes
7) Reboot - 2 minutes

Total time: 54:02. OK, they exaggerated a little.

Re:ATC?It's a big tower, but that's not important

[Anonymous Coward]

Differences in the phonetic alphabet used in Asian cultures.. no l, so it gets pronounced as r.

Yeah, computers don't make errors...

[commodoresloat]

They do those things on purpose!

Re:This is wonderfil news for opensource!

[Anonymous Coward]

pls to make at least a minimal attempt to make any sense at all.
kthxbye

Ok.

[Anonymous Coward]

That's fine. I like the rough stuff, by my safe word is "apples".

people are more fragile

[wadiwood]

I've yet to see a computer go postal without human help.

trouble is computers are designed by people.

air traffic control and missile guidance are two systems I'd never ever work on.

I'd like to program all the missiles to fly to the sun, but the consequences of a bug in the system are just too scary for me to contemplate.

Re:Model of a Slashdot Personality

[Anonymous Coward]

Another attempt at a funny song. There *are* no funny songs. After a few lines you get the idea..next.

Sort of like those "list" jokes. 10 reasons why a beer is better than a woman. Oh tee hee.

too suspicious

[zogger]

--around a year and a half ago a similar situation happened in japan if I am remember correctly. It looked rather like a military test that got out of hand. I am medium suspicious of two separate systems failing at the same time, it reminds me of that other "accident" that was rumored to be some pretty advanced jamming.

Note to anyone, yes, this is pure speculation, I admit it out loud. My default nature that I am completelycomfortable with is whenever strange occurrences happen with "government"- anyone's government - I am suspicious of it as being more than incompetence or actual random accidents. Too many events over the years that at first looked one way turned out to be completely different, they were either delibarately done, or somehow they were collateral damage from something bigger that needed to occur for some agenda, or allowed to happen, again, for a higher level agenda not readily apparent at the time.

Everything fails.

[T-Ranger]

Water mains under the street break. Suspension bridges collapse. Buildings collapse. Ships founder and sink. Breaks on cars fail causing crashes. Trains derail.

No, computers shouldnt crash. But they will eventualy fail, just like everything else will.

Y2K Bug Comparisson?

[citizenc]

Wait a second.. didn't they say that was going to happen when the Y2K bug was supposed to hit? Flights disappearing from radar, etc? Funny, they seem to have handled it fine. =)

Re:ATC?It's a big tower, but that's not important

[Tablizer]

Differences in the phonetic alphabet used in Asian cultures.. no l, so it gets pronounced as r.

IOW, a culturally insensative joke that will probably be banned or highly shameful 20 years from now and will go on your permanent record. Or is that pelmanent wecord? :-P

This 80's Show

[Tablizer]

If this was an error in the code, then how were they able to repair it in just 54 minutes? That's a pretty narrow window when it comes to rounding up the programmers, searching through the source...

If this was the 80's, I could say: "Their programmers are Samari trained, and if they don't work fast and accurate, they have to commit hari kari (disembowelment) in front of their peers.", and everybody would believe me. Guess I'll just have to make up shit about Islam instead.

Re:This 80's Show

[Anonymous Coward]

Business is war!

Japan will own all of the USA in no time flat!

Man, I miss the late eighties... Bet you the Japanese do, too.

I know what happened!

[wouterke]

Someone obviously broke the coffee cup holder right at the time the sysadmin had clicked on the 'uninstall' button of the old version.

Ah, no, wait...

Japanese software industry.

[coday]

Working in the software industry here in Japan for the last two years I have had my eyes opened to the true state of affairs. Most 'westerners' have an idealogical view of the high-tech world of Japan. This is far from reality. The fact is that software development here is at best poorly done, little design, short timelines (okay that one is universal), and lack of quality assurance. I can't say why this is the case, but shoddy products are in abundance. It may be trying to shove a relatively new industry into an old style organization, or the lack of individualism, I'm guessing at these. This story does not surprise me. All I know is I am looking forward to returning to the industry in Canada.

Re:Japanese software industry.

[TracerJPN_USMC]

Really. I've been living in Japan (Okinawa) now for close to 3 years, and I never ceased to be amazed at the quality and low cost of all the electronics and software available here. The Japanese are varey subserviant and bend over backwards to please the customer and repair any problems. Hell, my DSL line has gone out 1 time for about 3 hours in the last 14 months, I had a NTT tech over to fix it in about 30 min from calling in, and they gave me one month of free service...

Re:Japanese software industry.

[coday]

I agree about the electronics and the desire to please, however when it comes to software development my expeirience has been terrible. I'm in Osaka, don't the Japanese think of Okinawans as non-japanese anyway :)

Re:Japanese software industry.

[kryonD]

Opposite. Older people from Okinawa will correct you and incorrectly refer to themselves as OKINAWAJIN (of Okinawan Nationality). This would be like saying my nationality is Floridian. This is mostly due to them being pissed about the way the Japanese military treated them in WWII and then subsequently having the US "give" Okinawa back to Japan in 1972.

Okinawa used to be a seperate kindom called the Ryukyu Kingdom. I forget the year, but Japan forcefully occupied it a while back without much effort as Okinawan people are really peaceful and laid back, thus not seeing a need for a signifigant army.

Re:Japanese software industry.

[coday]

I stand corrected.

A few thoughts on redundancy.

[muonzoo]

I think this is one of those rare times where I have an opinion that's actually relevant. :-)

• I am a pilot.
• I am a Software Engineer
• I worked on a large air traffic control system for 7.5 years
• I read the article

First, people need to understand that no Bad Things will happen if an ATC system goes offline while planes are under it's jurisdiction. ICAO member countries (and most nations for that matter) have strong procedural rules in place that keep planes separated without the help of radar. This is espcially true in the enroute case. (Area control centres handle overflight and enroute traffic. Eveyone is separated by at least 1000' vertical and 3 miles horizontal at all times. The altitude restrictions and clearances that each pilot receives are chosen specifically so that in the even of loss of communications, the pilot can continue to his "clearance limit" without any problem. Well, you ask, what happens when he gets to his clearance limit and still isn't communicating with air traffic control? They hold. This is all laid our quite clearly. These rules have been around since before RADAR because thats the way it was done.
Just take a look at the RADAR coverage map of Canada (one is visible at the link above). There are lots of places that don't even HAVE radar coverage.
The old tried and true clearance and time/speed based conflict resolutions works and works well.

Secondly, and more imporatantly, there really isn't any news in this article. It's scaremongering. This happens all the time. It's an inconvenience, but rarely a saftey concern.

For those who asked about it; yes, typically a new system is run in parallel with the legacy system for a period of time (sometimes 24 months) before it is used as the primary control. Notice that the old system is live and the new system is shadowing. That way, anomalies that are found do not impact any flights.
[*flame proof underwear on*]
Is it just me, or does the press dig around for 'news' in about as diligent a manner as Slashdot?

Re:A few thoughts on redundancy.

[RDPIII]

Is it just me, or does the press dig around for 'news' in about as diligent a manner as Slashdot?

It's just you.

In related news: Aviation Security Expert Says Outdated Airtraffic Control System Still in Use 24 Months after Better, RADAR-Based System Deployed; Tries to Convince Unaware Public "No Bad Things Will Happen"; Canada Apologizes for Insufficient RADAR Coverage.

Re:A few thoughts on redundancy.

[Microlith]

Mainichi Daily News (daily daily news) is often regarded (especially MDN english) as being a tabloid.

Generally they go for sensational headlines and stories (their "Wai-Wai" section is the most popular).

Re:A few thoughts on redundancy.

[lommer]

Does the ICAO [icao.org] have strong procedural rules in place on what to do in the event of a slashdotting?

Might be time to get out the rulebook...

Re:A few thoughts on redundancy.

[Oswald]

Hmmm. Perhaps I can help with a few misconceptions here, based on over 19 years of air traffic control experience at Atlanta Center.

First, people need to understand that some bad things might happen if enough ATC systems go offline at once. Bad things are less likely to happen, as the poster states, if the outages occur in the enroute (my) environment, because the planes are generally farther apart than in terminal airspace. (Picky notes: enroute separation is 5 miles (not 3) OR 1000 feet--not AND--but I'm sure that was just a misstatement.) But they're not THAT far apart. This post makes it sound like any time we want to we can drop back to good old non-radar control. Well, standard separation in a non-radar environment is as high as 10 minutes flying time (longitudinally, which is to say along the same route). That's a lot more than the five miles I was using when the radar was working. The transition will be a bit tricky, and if I have to do it for any length of time, traffic will slow to a virtual standstill.

What's more, it is simply not true that aircraft clearances cover eventualities like lost communications or lost radar. This is a myth, and one that new on-the-job trainees quickly get de-programmed out of their heads. It's not possible to issue clearances that are good all the way to your clearance limit--every aircraft that departs, deviates for weather, changes destination, or even changes altitude (say, for turbulence) has the potential to screw up everybody else's "perfect" clearances. We truly don't even try to come up with such clearances. As for the idea that everybody will get to their clearance limit (actually, it's the published holding pattern for the route they're on to their clearance limit--probably that was simplified for clarity) and hold, that's great until you get the part about "until their estimated time of arrival" (original poster left that part out). Now you have planes dropping out of holding (and BTW, who assigned altitudes to make sure 6 aircraft didn't hold at the same altitude when the radios went out?), not necessarily from the bottom first, and flying to their destination airport. It's a 5-times-a-day event at hubs like Atlanta for 30+ aircraft to be scheduled over one fix in an hour--what are we gonna use for sequencing? TCAS? Common Traffic Advisory freqs? Get serious.

I'm not trying to scare anybody here. There are redundant systems (and they're pretty well-seasoned at this point anyway, so they almost never break), and ways to get hold of aircraft through company radios, and it really is a big sky. But it doesn't do anybody any good to pretend that it's not dangerous to try to sort out a major arrival rush by looking in your fish-finder and chatting with the other pilots til the controller gets back.

ATC was invented many decades ago because airplanes flew into each other without it. Those were props, flying to destinations with a tenth the volume of a modern hub. Maybe someday we'll have some cool hive-mind software that will allow the airplanes to sort everything out between themselves, and there won't be anymore ground-based controllers. (I won't see it in my career, cause I retire in less than 6 years.) Until that time, controllers and reliable control equipment will continue to be necessary for safety as well as expediency.

Re:A few thoughts on redundancy.

[Rich0]

Disclaimer - while I have read quite a bit on ATC I won't pretend to be a professional.

My understanding is that controllers keep manual flight strips handy just in case a computer fails without warning. They are trained in estimating where a plane will be after flying x minutes at y knots. However, this is not meant as a substitute for the full technology provided by modern ATC systems - it is basically designed to minimize the likelihood of a crash while the computer guys run like mad to get things back up.

Keep in mind that modern ATC systems inflict ulcers and heart attacks on their operators WITH all the fancy technology. Imagine having 100 planes under your control and having to keep calling them all for position updates so you can keep updating their location on a map...

If something like this happened across the entire US at once, the first thing that would happen is all aircraft on the ground would stay that way - no need to add fuel to the fire. Planes in the air would probably be directed away from busy airports as much as possible based on their fuel loads. It is simply impossible to run the JFK or LAX approach patterns without radar at the same capacity they run with it.

You could probably handle something like this without a major disaster if everyone stayed alert, but you can bet that the controllers involved are going to want to take a day off after it is all over...

Re:This is wonderfil news for opensource!

[wouterke]

Oh gosh. If you guys want to correct each other, make sure you get it right.

<checks>

Linus Torvalds

Bean Counters

[Jetson]

Redundancy started to suffer when the bean counters took over. Air Traffic Control is no longer an exercise in absolute safety but one of "risk management". This means that when the system designer says "I want a fully redundant hot standby system in a separate building powered from a different grid feed and on its own battery backup" the bean-counters say "you can have a warm standby (because we wouldn't want to have to pay for two software licenses) in a separate rack in the same computer room (have you looked at the cost of raised flooring lately?)". Instead of asking "what can we do to avoid a failure?" they tend to ask "how long will each failure last and how much will that cost us in lost revenue?"

Re:Bean Counters

[HiThere]

That's actually a somewhat reasonable tradeoff. It's not like this would kill people, the air field just shut down. But you would think that while the system was being upgraded it would be run in parallel with the known working system. That was the standard procedure even for office automation.
Perhaps in this case the separate building and separate power would be excessive. But ... no redundancy? During an upgrade? If you do that, you should expect failure, and allow for it in advance.

(Perhaps they did, and figured out that it would cost them more to notify their customers than to risk just suddenly closing the airport.)

Re:Bean Counters

[Jetson]

Even running a "shadow operation" on a second set of computers can be an expensive proposition when you're talking about ATC. It is far more common to do "in place" upgrades by taking the redundant systems off-line and upgrading them while the controllers continue to carry live traffic on the other half of the system. At some quiet period the two systems change roles (usually with data loss) and then the controllers run with the new software for a few hours to see if it breaks. If all is well, then the other side is upgraded and brought back into redundancy.

The problem with this situation is that there is a significant period of time in which the smallest hiccup can bring the whole aviation system back to the stone age because of the lack of redundancy. There have also been (unfortunately numerous) occasions where latent software defects were not detected until after the second side had been upgraded.

The need for quality control at the design and coding stages cannot be overstated, as it's almost impossible to do live load testing on these systems before they are shipped to the facilities. Sure, the developers can pipe in recorded radar traffic, etc., but nothing can simulate the pseudo-random reality-based pounding of a hundred controllers on the keyboards.

From my experience, the biggest bugs to reach the operational systems are generally race conditions that weren't expose in testing due to the different operating conditions and/or the presence of debugging code that were actually masking the condition they were trying to detect.

Japan is a bizarre place

[coday]

This country is indeed a strange strange place, although they have less violent crime than most western countries, they do have crime, and it is usually that 'creepy kind of crime'. I've been here for two years and I'm heading home before it's too late :)

Mainichi

[Anonymous Coward]

Mainichi is rather famous for its sensationalist journalism. In fact, I wouldn't even call it journalism. It caters toward lecherous Tokoyo expats and slants every article. (Not that's a bad thing.) Take everything you read at the site with a grain of salt.

Re:Was It Linux Based?

[fitten]

If it had been open source, this problem would have never happened. With millions of eyeballs detailing the code, we'd have found and corrected this bug before it ever occurred. Whats more, if the flaw did get thru, the operator could have jumped in and fixed the problem real time.

OMG... man are you brainwashed. First, as impossible as it may seem (gasp), open source software has bugs in it too. Second, even if it were open source, what million eyes would be looking at the code? I bet there isn't any source in the OSS archives that a "million eyes" have looked through. Third, you assume that the operator is an a) programmer, and b) at all familiar with the code enough to debug it and understand just what in the hell the code is doing anyway. Keep repeating your mantras fan boy, may they always give you a warm tingly feeling as you say them.

Similar Story

[Giant Robot]

Hey, slashdot japan is running this story too:

here.

.Net

[art3d]

I take it they weren't running .Net?

This isn't bad at all.

[Ignorant Aardvark]

Let's do a simple trichotomy of possible Slashdot headlines:

The Good: "Flight Software Runs Smoothly In Japan"
The Bad: "Computer Error Grounds 203 Japanese Flights"
The Ugly: "Computer Error in Flight Software Causes 203 Plane Crashes"

It could have been a lot, lot worse ...

The Real Story???

[rm3friskerFTN]

From The DrudgeReport on 02MAR2003 @ 2204 PST

Intelligence reports about the terrorist threat to the Hawaiian harbor bombed by the Japanese in World War II were sent to senior U.S. officials in the past two weeks and coincided with reports of the planning of a major attack by Osama bin Laden's terrorist group.

GERTZ: Terrorists aim at Pearl Harbor; Plan to hijack airliners, fly them into nuclear subs

Re:The Real Story???

[rm3friskerFTN]

The Washington Times has still more details.

While im here you just reminded me

[Anonymous Coward]

Click Here To Save Lives

Computer glitch?

[Orlando]

So when is a problem a 'computer glitch' and when is it a human error? How can you blame something that is entirely of human design for making mistakes on its own? Garbage in, garbage out, surely?

Seems to me we are good at praising ourselves when machines do what we want, but we are quick to distance ourselves from them when they go wrong.

Re:This is wonderfil news for opensource!

[Anonymous Coward]

No, it's Linus Thrwaldos.

Re:ATC?It's a big tower, but that's not important

[Anonymous Coward]

Oh ha ha. I suppose you find those Charlie Chan skits and those cartoon characters with the big front buck teeth a laugh riot as well.

So many @!!((@ racists on Slashdot.

Re:Was It Linux Based?

[Anonymous Coward]

I bet there isn't any source in the OSS archives that a "million eyes" have looked through.

Hmmm, we usually have two eyes, and I can guarantee you at least 500000 people have looked through the linux kernel code at least =)

Re:ATC?It's a big tower, but that's not important

[Alien Being]

I apologize. I didn't mean to offend the humor-impaired.