Slashdot Mirror
FAA Computer Glitch Causes Widespread Airline Delays
seven of five writes with this excerpt from an Associated Press report: "A problem with the FAA system that collects airlines' flight plans caused widespread flight cancellations and delays nationwide Thursday. It was the second time in 15 months that a glitch in the flight plan system caused delays. The FAA said in a statement that it is having a problem processing flight plan information. 'We are investigating the cause of the problem,' the agency said. 'We are processing flight plans manually and expect some delays. We have radar coverage and communications with planes.'"
...stuck in Atlanta...
Some one messed up their $2,000,000 excel macro that list morning.
I only changed one little line of code! It wasn't even important enough to test!
When Fascism comes to America, it will call itself Anti-Fascism, and tell you to give up your guns.
The FAA and their omnipotent employees hate anything new/improved. I worked for a very large FAA contractor developing a modern system. We tried to use Solaris/C++ but they sandbagged it because we could not test each and every line of code in the OS and that even C++ stripped to essentially C may have hidden problems. They are comfortable running systems on ancient mainframes running ULTRA.
By "glitch" they mean "totally offline delaying and canceling flights".
I'm pretty sure they had lots of other bugs and "regular glitches" in this time.
On the other hand, I'm also pretty sure that this kind of software does indeed go through a much better development and verification process than most commercial software around.
-- SouNerd.com
...according to the Wall Street Journal. Wonder if they'll give me a lift home?
I just read a post on Facebook by an Air traffic controller I know. They had to e-mail or fax all icao flight plans to the FAA. The FAA manually typed in every flight plan for every flight in the country.
"A person is smart. People are dumb, panicky dangerous animals and you know it." - K
I don't know a whole lot about in-flight software, but I do know the FAA itself heavily regulates it. There are different "levels" of flight software and hardware, with varying degrees of documentation and testing required for each. In-flight navigation software is obviously the most critical, and the level of documentation and testing they do for that is insane. Back when I was working on my capstone project in college, I got to see a little bit of how Honeywell tests their displays, and it is orders of magnitude more thorough than what I've seen in corporate software wherein money actually changes hands. They even had to create their own proprietary operation system in order to pass muster with the FAA.
I'm not sure about where the FAA's flight plan software falls in, but I'm guessing that since it's not safety critical, and only an operational risk, it probably is fairly solid, but obviously not as solid as the safety-critical software.
I do think it's too soon to blame this on MS, though. We don't even know whether this was caused by a third party vendor, which vendors they use for the particular piece where the error occurred, or if this is even anybody else's fault but their own. I wouldn't be surprised if someone gets fired over this, though.
http://online.wsj.com/article/SB125863837097855555.html?mod=WSJ_hpp_LEFTTopStories
I read Slashdot for the headlines, because the headlines, unlike the articles, are usually original and never duplicated
"That's it, you're all grounded!"
I am officially gone from
According to this article, the glitch is no longer.
Apparently they fixed the glitch so the problem worked itself out naturally.
We will bankrupt ourselves in the vain search for absolute security. -- Dwight D. Eisenhower
FTA: "The FAA said at that time the source of the computer software malfunction was a "packet switch" that "failed due to a database mismatch."
We all know how large out of touch behemoths sometimes structure their IT. By 'packet switch' they mean 'guy who couriers hardcopy flight plans' and by 'database mistmatch' they mean their dewey-decimal-system was mixed-up by some jokesters.
'We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress.' RPF
I don't think MSSQL runs on any other OS??
That bug just flew in under the radar...
**ducks**
~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
"First things first -- but not necessarily in that order"
-- The Doctor, "Doctor
Sorry, this is flight planning/scheduling software. This is the system by which airliners are told what route they are to take when they finally get their flight clearance.
The primary reason this has not received "oh my god the sky is falling" priority is because, well, it isn't. Nor are shiny metal tubes blasting through the sky at 500 miles an hour going to smash into each other because of this. People will be inconvenienced and that's regrettable and needs to be fixed, but this is not a safety issue.
- Radar control systems: Unaffected.
- Air Traffic control: Unaffected.
- Communications: Unaffected.
- Landings: Unaffected.
- Any flights already in progress: Unaffected.
- Ground control: Unaffected.
- Passenger Safety: Unaffected.
This only delays the granting of flight clearance. The planes that are inconvenienced by this are safely on the ground, and the effect is that they stay on the ground longer than they should. Again, this is regrettable, but not fatal. There is no plausible scenario that would lead from this to a failure of traffic control. In fact, with fewer planes in the sky, you could argue that flying is actually safer (for those people who are lucky enough to have gotten clearance, of course!).
In the medical field, this would be a failure of the system that the receptionist uses to schedule your next appointment. In the automotive analogy, this would be a failure of your garage door opener. In the heart monitor analogy, there isn't even an analogy because heart monitors don't have (as far as I know) any non-critical systems.
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
Considering that this did not affect the development of flight plans, only delayed them, I'd rip this back out of the category of "safety critical".
Online development of flight plans is much faster than offline. But that's precisely what makes the offline ones safe - as you delay the development of flight plans you also reduce the number of planes being cleared for takeoff at the same time. Once the flight planning software goes back online, you can start granting flight plans quickly again and get traffic volumes back to normal.
And it's a bit of a stretch to assume a risk of a collision even in the complete absence of flight planning. ATC, Radar, airplane internal collision avoidance, and pilot eyeballs are all unaffected by this. The only thing this could possibly do is put two planes in the vicinity of each other, but even that is unlikely as the sky isn't "the wild west" - it's carved up into clearly-defined highways with speed limits and defined routes which are all used in flight planning. So if you have two planes in the vicinity of each other, they are going in the same direction at roughly the same speed, so there's TONS of time to react.
And the first precaution, as we've seen here, is to reduce the number of clearances. Arguably, this makes the actual flying safer, since there are fewer planes up there for ATC to have to track and communicate with.
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
Even if it's fixed, there will still be delays for a while. There are probably tons of flights that are still awaiting clearance and the requests are being dealt with as quickly as safely possible. Meanwhile, a normal load of requests is still coming in, and those go into queue as well.
Trouble with things like flight planning is that they are, well, planning. If they go down, the manual process can't keep up with the load, and a queue develops. Once the system comes back up, it's gotta munge through the queue and get all the planes that have been waiting for clearance in the air.
And the skies can only handle so much traffic - you can't have infinite numbers of planes in the sky at once. So there will be a period where takeoffs will be done pretty close together, up to the point where the flying lanes are at max safe capacity. Max safe capacity is higher than normal capacity, but it'll still take a while to get all those delayed flights going, and the flights that are just now at this moment getting ready to go will be delayed as a result.
The net result of this is the same as a construction zone on a highway - you have a large stretch of highway with traffic backed up in it (planes waiting to take off) followed by a stretch of highway that almost empty. If you pull the barriers, it will still take a while for the empty stretch of highway to fill up with cars again, and the cars at the back of the line still have a long wait ahead before they can get up to speed.
"This post contains words, known to the State of California to cause thought. Wash brain thoroughly after reading."
Flight plans are for scheduling and routing purposes only. Seperation is maintained by Air Traffic Controlers using surveilance and secondary radar systems connected to automation systems, none of which run M$ Windows I might add.
Yea, these are the guys (the feds) I want building cars, taking over health care...thanks god they are not building the planes. I'm just trying to think one government run organization that works as well as any private one. Any idea?
Its true the flights are delayed. That's as much of the truth we get from this complex system of terribly overworked government and airline employees, national security, information systems, politics, FAA, corporate scum, secret clearance computer firms with bizarre contracts, and the 24/7 news spew with commercial airline as clients. They start by overbooking flights, while trying to account for each seat for security, and the bean counters, vips, and bad weather. To top it off information systems have to share some data, but not other data, while the people working have to comply with all sorts of rigid protocols and odd project management that controls everything except their overtime. I dunno, I have no reason to believe that any news we get about this type of event is necessarily true. It doesn't need to be. Its just the spin that reporters have to make in the absence of any real statement for the record. If they told you the truth you'd stop giving them your money. God forbid airlines make customers a top priority. Ha! Even if they were telling the truth, it would be a fluke, but just as vague and pointless. OK. We know the truth. Its aliens from Orion preparing for 2012. And they work for Delta.
Well, I work w/ the FAA right now, and they're becoming relatively platform agnostic actually. ERAM, for instance, was written by Lockheed Martin to run on top of some flavor of IBM UNIX or Linux (forget which). In the old days, everything ran on custom, purpose-built hardware and OSes, but that really turned out to be a maintenance nightmare. Using COTS Hardware/Software means upgrading systems or providing new capabilities can be pretty easy.
I think ASDE-X runs on top of some sort of POSIX type OS as well... I know its data stream is standard UDP over Ethernet type stuff.
Maxim: People cannot follow directions.
Increases in truth directly with the length of time spent explaining them