FAA Computer Glitch Causes Widespread Airline Delays

seven of five writes with this excerpt from an Associated Press report: "A problem with the FAA system that collects airlines' flight plans caused widespread flight cancellations and delays nationwide Thursday. It was the second time in 15 months that a glitch in the flight plan system caused delays. The FAA said in a statement that it is having a problem processing flight plan information. 'We are investigating the cause of the problem,' the agency said. 'We are processing flight plans manually and expect some delays. We have radar coverage and communications with planes.'"

Comments

Here I sit...

[dtmos]

...stuck in Atlanta...

Re:Here I sit...

[gEvil (beta)]

...stuck in Atlanta...

It could be worse. You could be stuck in Lodi (again).

Re:Here I sit...

[natehoy]

Sorry to hear that. But, as the old saying goes, "it's better to be DOWN HERE desperately wishing you were UP THERE, than UP THERE desperately wishing you were DOWN HERE."

Re:Here I sit...

[fprintf]

No it is supposed to go:

"Here I sit, broken hearted, came to s**t and only farted."

Re:Here I sit...

[icebrain]

Some come here to shit and stink
and pick crabs off their balls;
I come here to sit and think
and write stuff on the walls

re problem

[freddieb]

Sounds funny to me. Packet switch due to a database mismatch?

problems?

[archangel9]

only the second glitch in fifteen months? One would assume that's far better than most commercially written software.

Re:problems?

[Tellarin]

By "glitch" they mean "totally offline delaying and canceling flights".

I'm pretty sure they had lots of other bugs and "regular glitches" in this time.

On the other hand, I'm also pretty sure that this kind of software does indeed go through a much better development and verification process than most commercial software around.

Re:problems?

[geekoid]

No, by glitch they mean glitch. "totally offline delaying and canceling flights" is the result the glitch.

  "On the other hand, I'm also pretty sure that this kind of software does indeed go through a much better development and verification process than most commercial software around."
the scheduling software? probably not. It was done out of house for contract by people who don't give a damn the moment the software releases.

Re:problems?

[ShatteredArm]

I don't know a whole lot about in-flight software, but I do know the FAA itself heavily regulates it. There are different "levels" of flight software and hardware, with varying degrees of documentation and testing required for each. In-flight navigation software is obviously the most critical, and the level of documentation and testing they do for that is insane. Back when I was working on my capstone project in college, I got to see a little bit of how Honeywell tests their displays, and it is orders of magnitude more thorough than what I've seen in corporate software wherein money actually changes hands. They even had to create their own proprietary operation system in order to pass muster with the FAA.

I'm not sure about where the FAA's flight plan software falls in, but I'm guessing that since it's not safety critical, and only an operational risk, it probably is fairly solid, but obviously not as solid as the safety-critical software.

I do think it's too soon to blame this on MS, though. We don't even know whether this was caused by a third party vendor, which vendors they use for the particular piece where the error occurred, or if this is even anybody else's fault but their own. I wouldn't be surprised if someone gets fired over this, though.

Re:problems?

[Osiris Ani]

I'm not sure about where the FAA's flight plan software falls in, but I'm guessing that since it's not safety critical, and only an operational risk...

Considering that flight plans exist to prevent planes from crashing into one another, I'd lump them into your category of "safety critical."

Re:problems?

[natehoy]

Sorry, this is flight planning/scheduling software. This is the system by which airliners are told what route they are to take when they finally get their flight clearance.

The primary reason this has not received "oh my god the sky is falling" priority is because, well, it isn't. Nor are shiny metal tubes blasting through the sky at 500 miles an hour going to smash into each other because of this. People will be inconvenienced and that's regrettable and needs to be fixed, but this is not a safety issue.

  - Radar control systems: Unaffected.
  - Air Traffic control: Unaffected.
  - Communications: Unaffected.
  - Landings: Unaffected.
  - Any flights already in progress: Unaffected.
  - Ground control: Unaffected.
  - Passenger Safety: Unaffected.

This only delays the granting of flight clearance. The planes that are inconvenienced by this are safely on the ground, and the effect is that they stay on the ground longer than they should. Again, this is regrettable, but not fatal. There is no plausible scenario that would lead from this to a failure of traffic control. In fact, with fewer planes in the sky, you could argue that flying is actually safer (for those people who are lucky enough to have gotten clearance, of course!).

In the medical field, this would be a failure of the system that the receptionist uses to schedule your next appointment. In the automotive analogy, this would be a failure of your garage door opener. In the heart monitor analogy, there isn't even an analogy because heart monitors don't have (as far as I know) any non-critical systems.

Re:problems?

[natehoy]

Considering that this did not affect the development of flight plans, only delayed them, I'd rip this back out of the category of "safety critical".

Online development of flight plans is much faster than offline. But that's precisely what makes the offline ones safe - as you delay the development of flight plans you also reduce the number of planes being cleared for takeoff at the same time. Once the flight planning software goes back online, you can start granting flight plans quickly again and get traffic volumes back to normal.

And it's a bit of a stretch to assume a risk of a collision even in the complete absence of flight planning. ATC, Radar, airplane internal collision avoidance, and pilot eyeballs are all unaffected by this. The only thing this could possibly do is put two planes in the vicinity of each other, but even that is unlikely as the sky isn't "the wild west" - it's carved up into clearly-defined highways with speed limits and defined routes which are all used in flight planning. So if you have two planes in the vicinity of each other, they are going in the same direction at roughly the same speed, so there's TONS of time to react.

And the first precaution, as we've seen here, is to reduce the number of clearances. Arguably, this makes the actual flying safer, since there are fewer planes up there for ATC to have to track and communicate with.

Re:problems?

[cyberprophet]

Flight plans are for scheduling and routing purposes only. Seperation is maintained by Air Traffic Controlers using surveilance and secondary radar systems connected to automation systems, none of which run M$ Windows I might add.

Re:problems?

[zippthorne]

And some of which still runs on cryo-cooled Univac machines...

Re:problems?

[jklovanc]

Not receiving clearance means flight delays and flight cancellations. That translates to lost revenue to the airlines; inconvenience and possibly lost wages to the traveller; and inconvenience and possible lost revenue to to the travellers' employers. It is not a trivial matter as the losses could add up to millions of dollars per hour. Suer, no-one will die beczause of it but many will lose money.

To use one of your analogies; your garage door does not open, you are late for a sales meeting and lose a $1M contract. Not perfect but a cancelled flight may delay someone for a day.

Re:problems?

[JumpDrive]

Well I think you implicitly put your finger on part of the issue.
1) Money
2) Management
If nobody see the need for spending a certain amount of money for a system, then you may get a system which fails on occasions.
If once every 15 months is acceptable, then no need for more money.
But if you want a more robust system, then more money may need to be spent.
Which would mean that companies would need to be giving the FAA more money. So suing the FAA would defeat the purpose of what they are trying to achieve.
Now it may mean that the system is fine, in which case the problem may be with better support or more talented support. Which once again comes back to probably needing more money.
The problem may be Management, in which case you must realize it is a government bureaucracy. Good luck, dealing with that. But it actually comes back to Airlines needing to use their lobbyist to work with the government to get a better system in place.
Suing, would be a short term solution ( if it is even possible).


Hello US, welcome to the third world.

Re:problems?

[natehoy]

I was responding to a post that implied that this failure could cause a midair collision. It could not. I didn't address the possible economic impact of the failure because that wasn't the topic I was responding to.

But, since you brought it up, what would be the economic impact of fixing it so it could never, ever fail under any circumstances? Moreover, is that even possible?

Keep in mind that flight safety stuff is standalone. Each zone has their own RADAR, their own control systems, and is a self-contained entity. Most of the handoff is (or can be) done from controller-to-pilot. This is by design, both for safety and efficiency. There's no compelling reason to integrate the systems, and it's not worth the added complexity and risk of failure.

Safety gear needs to tell you where planes are, give you warnings if two planes start converging on course, and give you ways to communicate those issues to an airplane. Once an airplane flies outside your zone of control, it's not your problem any more, so your system doesn't even need to be aware of those aircraft.

Flight planning is, by its nature, integrated across the country. Airway allocation needs to be done as a coordinated national function, not a series of zones. Flight Planning is a separate entity within the FAA. They send their data to the individual controllers, but they don't really need to. Flight Planning exists to reduce or eliminate congestion (clumps of aircraft close together) so the controllers have an easier time managing the flow of traffic (and therefore you can have more planes in the air at a time).

The hardware and software involved in planning are far more integrated and therefore complex. By its nature the system is subject to additional points of failure as a result.

Then you add funding and priorities to it, and you accept that an occasional failure of the planning system isn't going to end up with anyone dead. So you do your best, but accept that you'll probably have a failure from time to time because making the system completely reliable means spending probably more money than is lost by everyone delayed.

So, in order to avoid the chance of missing your $1M meeting, how much additional would you like added to each airline ticket everyone buys every day? Because $50 probably isn't going to even start covering it, and if you make everyone pay $50 on each and every airline ticket to reduce the chances of a once-in-a-year flight delay, it'll never get passed. This isn't even a leading cause of delays.

There are too many other delays for too many other reasons for this one to reach that level of spending priority. Again, even if making the system completely immune to failure is possible.

Re:problems?

[Fulcrum of Evil]

I'm not sure about where the FAA's flight plan software falls in, but I'm guessing that since it's not safety critical, and only an operational risk...

Considering that flight plans exist to prevent planes from crashing into one another, I'd lump them into your category of "safety critical."

Last I looked, this was mainly an issue near airports - airspace is really huge, and I don't see flight corridors as hugely risky.

Re:problems?

[PitaBred]

Not a safety issue? Have you ever been in an airport with significantly delayed passengers? It's a bit of a powder keg when you get enough people trying to find seats in the terminal to wait for their "slightly delayed" planes, fight for power outlets, talk on cell phones...

Re:problems?

[AceyMan]

I ran an airline for a time, and I see where you left off a few critical sub-systems.

- Dispatch Planning (what cooks up the flight plans in question for submission, specifying the fuel burn, max takeoff weight, planned landing weight, operating mach speed, altitude, etc.)

- Operations Control (where every plane and crew is, in real time.). Think of a Gantt chart that would make your jaw drop, it's kind of like that.

These are the things that make an airline run. The FAA only has one responsibility -- keep airplanes apart. The rest (largely) is up to the carrier.

Re:problems?

[natehoy]

LOL, good point.

Let me refine that: Not an FAA safety issue.

FTFM (Fixed That For Myself).

The TSA, on the other hand, might consider it a wee bit of a safety issue.

Re:problems?

[hesaigo999ca]

>This is the system by which airliners are told what route they are to take when they finally get >their flight clearance.
>The primary reason this has not received "oh my god the sky is falling" priority is because, well, >it isn't

You lack of imagination leaves to me to think you would be a bad programmer.
See I could tell you to take off runway 7, which in itself would be harmless, however , the glitch comes from the same software that told the other air traffic controller that it was ok to use runway 7 to land, now when a moving object meets a moving object, which neither are either indestructible or unmovable, I wold say there tends to be a lot of carnage.

Re:problems?

[natehoy]

The system works like this:

1. Pilot works with Ground or calls Flight Services and files a Flight Plan. Flight Plan includes:
  - Airport you are taking off from (A).
  - Airport you intend to land at (B).
  - Airports you will use as a "contingency" if things go badly (C,D,etc).
  - What routes you intend to use to get from A to B.

This allows Flight Services to figure out how many airplanes will be at altitide in different areas at any given time so they can control congestion enroute, and also attempt to control congestion at destinations (so 200 planes aren't approaching Atlanta all at once).

If Flight Planning is down, Flight Plans cannot be filed.

2. Pilot contacts Tower for clearance, and if the flight requires a Flight Plan the Tower checks to make sure the Flight Plan is approved before deciding to allocate a runway to the flight.

The Flight Plan is only a "deny" reason for a takeoff. The Tower can sometimes use it to decide from multiple runways if they have space (more efficient for the plane to take off in a direction that resembles its final course) but that is at the discretion of the controller. The controller is coordinating all takeoffs and landings, and will not give takeoff clearance without a flight plan if one is needed (for a lot of flights, flight plans are not required). They will deny takeoff clearance for a lot of other reasons, including delaying it if the runway is in use.

So, again, if Flight Planning is down, flight plans cannot be filed therefore flight plans cannot be approved therefore takeoff clearance is not granted.

3. The pilot takes off under Tower Control. They fly at a given course designed to maintain separation until they can turn and get on their route. At all times, a controller is directing their operations, and that controller is responsible for separation and safety. The Flight Plan is only used at that time to determine what direction the plane is headed when they leave the Tower Controlled area.

Again, if a Flight Planning is down the takeoff that led to this never happened.

4. Once the airplane leaves the Tower Control area, the Tower hands them off to Flight Services. Flight Services is aware of the flight plan, but doesn't really need it. They use RADAR and the defined airways to maintain positive control and separation of aircraft. The plan is handy, as it allows them to predict which way a plane will turn once it reaches a certain point, but they'd know this by talking to the pilot anyway.

5. As the plane approaches their destination, Flight Services hands them off to the Tower and usually closes their flight plan at that time. The Tower in the destination airport, just like the Tower in the originating airport, is an independent entity that accepts the aircraft under their control and puts them in queue for landing.

At no point does the Flight Plan become relevant in assigning runways for takeoff or landing, nor is it ever relevant (except for helpful supplemental information) in traffic separation.

And even if it was, if Flight Planning suffered a complete failure, NO PLANES WOULD TAKE OFF. You can't have a midair collision when no planes are, well, in the air.

Re:problems?

[natehoy]

PS: You are confusing "Air Traffic Control" and "Flight Services" with "Flight Planning". They are three separate entities.

An "Air Traffic Controller" is a person (or multiple people) sitting in a tower or other control area. They have one primary job - to make sure planes don't smack into each other while in their domain. They sequence takeoffs and landings, taxiing, etc.

ATC has absolute and utter control within their domain. If a Controller tells you to jump, you start jumping then ask "how high?" on the way up. Any aircraft wishing to enter their little fiefdom has to ask permissions.

"Flight Services" controls flights that are at altitude and flying outside an ATC area. They are basically a Controller, but without landing and takeoff responsibilities.

A Controller is not told by Flight Services to allocate a runway to a flight. If a Flight Services agent called from 1000 miles away and tried to tell a Controller what runway to use, that Controller would undoubtedly (and rightfully) hang up on the Flight Services agent, probably after offering a few choice expletives. There is nothing Flight Services can say to a Controller that could affect operations at that Control Tower in any way.

In the same way, Flight Services is not told exactly where a plane should fly by Flight Planning. They are advised which way the plane WANTS to go, but they decide where the plane ACTUALLY goes, and they can tell a pilot to deviate from a flight plan any time they please to maintain separation and safety.

The reason I can't imagine a failure in Flight Services affecting the safety in Control or Services is simple - there isn't any way it could. Flight Planning does not include any traffic separation services. That's the responsibility of the people with the RADAR screens and radios in front of them, in Control and Services.

Planning is only there as a "gatekeeper" to help prevent the skies from getting too crowded, and if they get delayed planes don't take off until they can coordinate the flights properly.

A failure in Planning doesn't mean all planes will suddenly get clearance, a failure means planes stay safely on the ground.

Re:problems?

[hesaigo999ca]

> If Flight Planning is down, Flight Plans cannot be filed.
Already you are wrong in your assumption, as per the article states,
they were doing this manually.
This error on your part would cause all subsequent points to be null and void.
You would be at home with those developers responsible for the glitch!

Re:problems?

[natehoy]

No, if you read some of my other posts I already stated that this outage did NOT mean Flight Planning was down. It meant that flight plans could not be submitted automatically.

Flight Planning was not, in fact, down during this incident. It was severely delayed because the Flight Plans could not be submitted automatically and the manual process takes longer. However, I was making the point that Flight Planning is NOT involved in Flight Operations (Air Traffic Control, and Flight Services) and a theoretical failure in Flight Planning could not, in any way, impact the safety of Operations.

You may believe as you wish, but if you aren't even going to read an explanation of how the aviation system works as explained by a licensed pilot, then, well, you can remain ignorant. Makes no difference to me.

Re:problems?

[natehoy]

To clarify, I was responding to this point of yours:

"See I could tell you to take off runway 7, which in itself would be harmless, however , the glitch comes from the same software that told the other air traffic controller that it was ok to use runway 7 to land"

If you told me to take off on RWY7, you would be an Air Traffic Controller. Your software (if you used any) would be contained within the tower. It is not Flight Planning software, it is Flight Operations software.

The article make absolutely no mention of a failure of Flight Operations software.

A failure of Flight Operations software could be a safety issue, I agree, but that's not what the article said went wrong.

If you want to change the subject to Flight Ops software, fair enough, let's go through those procedures if you're interested in learning how it works.

OK, so say the Flight Ops software went out.

The traffic pattern is set in such a way that controllers can predict where the airplanes will be and can track them on RADAR. They'll also immediately withdraw any takeoff clearances to reduce their workload, and call Flight Services and tell them to divert every inbound flight to their alternate airports.

If RADAR goes out, they'll use binoculars and the radio to control landing traffic. Pilots know they need to fly at 500 foot increments in a series in the pattern and work their way down the stack. Again, you've got a stack of planes all in predetermined flying ovals, so you can take the bottom planes in the stack (those at Traffic Pattern Altitude), land that queue in order, then tell everyone to slowly move down a rung in the ladder. Lather, rinse, repeat until no planes are in the air.

If the radio goes out, they have handheld directional bright lights they can use to signal the aircraft what to do. Any incoming aircraft will not get a response from ATC and get back on the horn with Flight Services for a priority divert to another airport. Any planes above the lowest rung will contact Flight Services for departure instructions and will continue to fly in predictable circles at a set altitude until they get instructions.

If the batteries on the lights go out, the pilots in the lowest rung will fly over the runway at traffic pattern altitude, and exit the tower control area safely so they can get back in touch with Flight Services for a priority divert to an airport that is working.

Pilots spend a crapload of time learning all this stuff. If Air Traffic Control goes away completely, we know to stay in predictable patterns and what those patterns look like.

Re:problems?

[hesaigo999ca]

Sounds like you have experience in the field, I like the fact they have backup plans
for these type of things...I just hope that most of what you say takes into consideration the last time they were stuck using this was way back when we did not have as much air traffic as today...

I remember hearing someone telling me all about their backups, and the technology used, and everyday they made backups...until the day they had to restore, that's when they found out the backups were no good....could have been avoided by forcing a fake situation where you needed to test them....hence here the same logic applies.

Damn Excel

[headhot]

Some one messed up their $2,000,000 excel macro that list morning.

Honest!

[sycodon]

I only changed one little line of code! It wasn't even important enough to test!

Re:Honest!

[Anon-Admin]

HAHAH, around my work you could add "or Document" to the end of that statement. I cant count the number of times people have changed things with no documentation and they did not consider it important.

Re:Honest!

[sycodon]

Document? I don't understand. What is "Document"?

Re:Honest!

[Monkeedude1212]

No you see the problem was that someone tried to comment out a comment. However, their skewed vision on how comments work, and by some MIRACLE of programming and compilation, All the code ended up commented and the comment was compiled instead.

Re:Honest!

[EchaniDrgn]

It's an artifact type in the CM system. Don't feel bad, we don't have any of those around here either.

Re:Honest!

[31415926535897]

I think it's a Microsoft product. That's why not many around here use it.

what do you expect from 70's technology ??

[Anonymous Coward]

The FAA and their omnipotent employees hate anything new/improved. I worked for a very large FAA contractor developing a modern system. We tried to use Solaris/C++ but they sandbagged it because we could not test each and every line of code in the OS and that even C++ stripped to essentially C may have hidden problems. They are comfortable running systems on ancient mainframes running ULTRA.

Re:what do you expect from 70's technology ??

[eln]

Given that the FAA is responsible for the lives of thousands of people who travel by air every day, not to mention cargo planes which represent billions of dollars in economic activity, I'd be extremely alarmed if they were constantly switching to the latest and greatest technology without testing each and every line of code in the entire software stack, up to and including the bare metal. Change is slow in organizations like that because it has to be. Improper testing can easily lead to lost lives, or at the very least huge delays like these ones that can cost millions in lost productivity.

Re:what do you expect from 70's technology ??

[fbjon]

C++ stripped to essentially C may have hidden problems

That sounds about right, yeah.

Re:what do you expect from 70's technology ??

[sexconker]

They are comfortable running systems on ancient mainframes running ULTRA.

I would be comfortable doing the same thing.
Why do people shit on mainframes? They may cost a lot of money and time, but they're orders of magnitude more dependable than any other server solution that exists.

Yes, that includes the "cloud".

Re:what do you expect from 70's technology ??

[DeadCatX2]

This brings up the greater question of how old must a technology be before it is considered mature enough for purposes as mission-critical as air planes. 5 years? 10 years? 20?

I think that reasonably, you ought to be able to trust technology that's been around for a decade.

Re:what do you expect from 70's technology ??

[Nerdfest]

A huge part of their reliability is due to restistance to, and lack of change. Almost every problem that is encountered is due to a programming error. The same hardware availability can be obtained on a 'cloud' using virtual machines, redundancy, automatic fallover, etc, many of the same technologies the mainframes use, with a huge performance increase and cost savings.

Re:what do you expect from 70's technology ??

[jklovanc]

Not necessarily, if the technology was speced to handle 100,00 flights per day and they are trying to pump 200,00 flights per day through it, the software may not be able to handle it no matter how old it is.

Re:what do you expect from 70's technology ??

[happy_place]

The wallstreet journal article mentions the issue was with a new system, specifically noting that it was not related to their antiquated hardware that has historically been the cause of failures in the past. I think it's naive to suggest that old hardware or coding in C++ (over C, as if C's a completely dead language... they both have their places in complex engineering tasks) will eliminate failures in the system, because in this case it was newer hardware, which had a cascading effect, implying involvement of networked databases, etc...

Re:what do you expect from 70's technology ??

[sexconker]

Uh, that's a very small part of the reliability.

The hardware reliability comes from having fault-tolerant, hot-swappable, redundant, well-tested hardware. The "cloud" hopes to achieve this, yet doesn't.

The software reliability comes from having fault-tolerant, well-designed, well-tested software. This takes time, sure, but it can just as easily be done with forward-thinking to allow for growth and change.

The problem is not mainframes, mainframe programmers, or PHBs resistant to change. The problem is people rushing the newest shiny, do-nothing feature, people willing to sacrifice reliability and forgo proper design and testing, vendors end-of-lifing shit for no reason other than planned obsolescence, and the general inability of people to plan ahead - opting instead to throw money at the problem later.

Hell, because of this mentality we don't even have phones that work right anymore.

Re:what do you expect from 70's technology ??

[PitaBred]

Carrying things on my back and walking places is much more dependable than driving a vehicle which can break down. I'm still not gonna start hauling everything in a backpack.

Re:what do you expect from 70's technology ??

[sexconker]

Terrible analogy.

Resolved...

[dtmos]

...according to the Wall Street Journal. Wonder if they'll give me a lift home?

Re:Hmm could it be a windows problem? - of course!

[Anonymous Coward]

Link to deployment announcement about "modernization" of airspace network.

http://www.stratus.com/news/2005/20050314a.htm

Re:Yep...

[ArhcAngel]

I just read a post on Facebook by an Air traffic controller I know. They had to e-mail or fax all icao flight plans to the FAA. The FAA manually typed in every flight plan for every flight in the country.

obviously

[stillpixel]

they must have been transitioning over to Windows 7 this morning.. you know that can take days depending on what your current system is.

Re:obviously

[confused one]

This is the government. They transitioned from Windows 2000 to XP only last year.

Re:obviously

[vegiVamp]

So what you're saying is that this was the reboot halfway the upgrade process ?

Re:obviously

[stillpixel]

Don't you see? The FAA was chosen as the Windows 7 upgrade test bed for the U.S. Govt Then if it doesn't work and people die, they can just blame Microsoft. Everyone hates them anyway right???

Maybe it's like the internet...

[ShatteredArm]

...and Iran was publishing bad flight plans. You know, to block undesirable traffic.

Re:Resolved?

[dtmos]

All true, but note line 1 of TFA:

A computer glitch that caused flight cancellations and delays across the U.S. Thursday has been resolved, the Federal Aviation Administration said. [emphasis added.]

Already fixed before it even got posted on /.

[DigitalReverend]

http://online.wsj.com/article/SB125863837097855555.html?mod=WSJ_hpp_LEFTTopStories

To quote some technology middle manager

[dkleinsc]

"That's it, you're all grounded!"

Glitch is now fixed

[smooth wombat]

According to this article, the glitch is no longer.

Apparently they fixed the glitch so the problem worked itself out naturally.

Re:Glitch is now fixed

[PitaBred]

Just stay the hell away from the guacamole, ok?

Foot TCP

[cosm]

FTA: "The FAA said at that time the source of the computer software malfunction was a "packet switch" that "failed due to a database mismatch."

We all know how large out of touch behemoths sometimes structure their IT. By 'packet switch' they mean 'guy who couriers hardcopy flight plans' and by 'database mistmatch' they mean their dewey-decimal-system was mixed-up by some jokesters.

Re:Hmm could it be a windows problem?

[Jumpin' Jon]

I don't think MSSQL runs on any other OS??

Honest...

[flogger]

That bug just flew in under the radar...

**ducks**

So when will they upgrade?

[Trip6]

We've heard about the antiquated ATC system for over a decade. Does anybody on the inside know if, and when they'll upgrade?

Re:So when will they upgrade?

[horatio]

ADS-B, the next generation of ATC tracking/radar systems was brought online recently over Louisville, KY and has been in testing in several other areas as well.

They are also phasing in more GPS approaches (nav systems specifically designed for landing) and phasing out the NDBs (non-directional beacon) and taking failing VORs out of service that are often used at fields which have (ILS) instrument approach runways.

That said, it is a very slow process. If your car's speedo goes out, who cares. If the airspeed indicator in your plane fails, you could be in deep shit. (GPS only measures your speed across the ground, not your airspeed - which includes factors for wind. Too slow and you stall, crash. Too fast you rip the wings off, crash.) Upgrading the avionics in even the most inexpensive aircraft is not cheap.

Re:So when will they upgrade?

[bill_mcgonigle]

GPS only measures your speed across the ground, not your airspeed - which includes factors for wind.

Assuming that most planes are flying where

    GPS speed - headwind > stall speed

unless they're on take-off or approach, wouldn't a GPS plus a laser wind-speed detection system at airports solve that problem?

Re:So when will they upgrade?

[Starker_Kull]

Wind varies tremendously by altitude, location, and time. Wind shear accidents occurred in the past because the resolution of the wind readings at airports compared to the rapid shear of the wind gave little to no warning of how rapidly the headwind component to an aircraft was about to change. In addition, indicated airspeed is actually measured by taking the ratio of ram pressure to static pressure, calibrated to read true airspeed at sea level on a standard day. This leads to the indicated airspeed being different from the true airspeed by large amounts depending on altitude, temperature, and pressure, but many important flying 'speeds' remain at the same indicated airspeed because of the physics of flying; for instance, best angle, best rate of climb, and best glide speeds stay (approximately) at the same indicated airspeeds even though the corresponding true airspeeds vary greatly.

Anyway, this is a long winded way of saying that the current way airspeed is measured and displayed to a pilot is absolutely critical to aviation operations. It's an emergency when your airspeed indicator is malfunctioning, and it is the most direct indication of the amount of energy relative to the airstream your aircraft has. It's not something you want to muck with by putting multiple layers of indirection/abstraction (and thus additional things that can fail) between the sensors and the indicators.

Re:So when will they upgrade?

[bill_mcgonigle]

It's an emergency when your airspeed indicator is malfunctioning, and it is the most direct indication of the amount of energy relative to the airstream your aircraft has. It's not something you want to muck with by putting multiple layers of indirection/abstraction (and thus additional things that can fail) between the sensors and the indicators.

The GP was mentioning failures too - why not have a backup system available if this is something which happens enough to be concerned about?

Re:So when will they upgrade?

[Starker_Kull]

Your idea of a backup system is an excellent one - and it's already been done, in a much more reliable way! The problem with the GPS + windspeed from some ground source is that:

a) It assumes the wind at the ground station is near the wind speed and direction of the aircraft. This is almost always a wrong assumption, as the wind speed rapidly increases with altitude, and tends to cant in quite a different direction as well. For a sensor that has to be accurate within a knot or two, this is completely inadequate.

b) It assumes the GPS gives an accurate speed second by second. This is again, a poor assumption. GPS fundamentally determines position, and speed is derived from this over time, meaning it typically indicates how fast you were, a few seconds ago. For the second by second resolution you need, this is inadequate.

c) It assumes that a way, in real time, of transmitting this information, deciding which ground stations to use, what interpolative formulae to use, mixed with the GPS data, can be reliably done.

So, it's not that it's a bad idea to have a backup, just that this particular implementation of one has many features that would render it useless.

How it's actually done is as follows. On transport category airplanes, it is required that you have three independent sensors to detect airspeed. Almost universally, this is implemented the same way; a pitot tube to detect ram air pressure, and static ports to detect static air pressure. They generally have three pitot tubes, each with its own independent plumbing, all electrically heated, and hooked up to 1 of 2 independent boxes (called air data computers) that reads the two pressures and calculates the indicated airspeed, which is then signaled on two independent airspeed indicators, one on the Captain's side, one on the F/O's. The third is hooked up to a set of standby gauges that are meant to break ties in a hurry if one of the primary sources goes wonky. The static ports are easy to spot (except on American Airlines, for reasons that will be obvious), because they carefully don't put any paint near them, so you can spot them on the sides of airframes as metallic ovals, with a black paint around the edge of the oval as a 'warning', and a few small holes in the middle where the static air pressure is equalized with the outside air.

On the 737, if you look down and to the left just when you board it on a jetway, you will notice one of these pitot tubes, along with an angle of attack vane perilously close to the jetway 'gasket'. The pitot tube looks like a little forward angled wing with a tube extending along its outside edge and forward, usually with a nice, bluish sheen to the metal from being kept well above the boiling point most of the time to keep water and ice far away. The AOA vane looks like a backward slanted wing, typically cocked at a crazy angle on the ground as there is too little airflow to make it align with the wind.

One of the preflight duties of an F/O is to assure that these tubes and sensors have not been messed with, whether by an overzealous TSA inspector using them as a hand and foot hold and breaking them in the process (yes, this happened), a jetway driver snugging up the the jetway a little off the spot and mashing them (happens often), aircraft washers forgetting to remove the covers put over them after cleaning the exterior (this killed 70 people once)or a baggage cart smearing up against it and driving off, hoping nobody will notice they did it (yup, this one too!) Check out this article on Wikipedia about it - it's fairly good and pretty informative about why it's such a fundamental indicator and why pilots get cranky about it being at all abstracted away from the raw data! :-)

Re:So when will they upgrade?

[bill_mcgonigle]

a) It assumes the wind at the ground station is near the wind speed and direction of the aircraft. This is almost always a wrong assumption, as the wind speed rapidly increases with altitude, and tends to cant in quite a different direction as well. For a sensor that has to be accurate within a knot or two, this is completely inadequate.

That's why I suggested the laser systems above - I don't understand exactly how they work, but by shining lasers through air masses they can determine wind speed, density, etc. Doppler RADAR's could probably be used too.

b) It assumes the GPS gives an accurate speed second by second. This is again, a poor assumption. GPS fundamentally determines position, and speed is derived from this over time, meaning it typically indicates how fast you were, a few seconds ago. For the second by second resolution you need, this is inadequate.

Surely this is a matter of sampling rate? GPS is dealing with microsecond-resolution calculations to even work.

c) It assumes that a way, in real time, of transmitting this information, deciding which ground stations to use, what interpolative formulae to use, mixed with the GPS data, can be reliably done.

Is there some reason to think this can't be worked out?

On transport category airplanes

What about private aviation?

When the Air France plane went missing, why did so many pilots jump to 'iced pitot tubes'? I got the impression from that they weren't very reliable? It sounds like with so many failure modes, they surely need a back-up system? (I'm all for 'unnecessary' redundancy when it comes to safety).

The other trouble with relying solely on pitot tube systems is they can't tell you anything about the air near you (like maybe a microburst forming above you?), only the air you're in.

Re:And ./ just noticed?

[sexconker]

I bet their landlines were still working.

Re:Yep...

[natehoy]

You and dtmos (see post further down in this thread) should get together for a beer. You're both stuck in Atlanta, and it's after noon. :)

Slashdotters meeting in person? Whoda thunk it. LOL.

Re:Resolved?

[natehoy]

Even if it's fixed, there will still be delays for a while. There are probably tons of flights that are still awaiting clearance and the requests are being dealt with as quickly as safely possible. Meanwhile, a normal load of requests is still coming in, and those go into queue as well.

Trouble with things like flight planning is that they are, well, planning. If they go down, the manual process can't keep up with the load, and a queue develops. Once the system comes back up, it's gotta munge through the queue and get all the planes that have been waiting for clearance in the air.

And the skies can only handle so much traffic - you can't have infinite numbers of planes in the sky at once. So there will be a period where takeoffs will be done pretty close together, up to the point where the flying lanes are at max safe capacity. Max safe capacity is higher than normal capacity, but it'll still take a while to get all those delayed flights going, and the flights that are just now at this moment getting ready to go will be delayed as a result.

The net result of this is the same as a construction zone on a highway - you have a large stretch of highway with traffic backed up in it (planes waiting to take off) followed by a stretch of highway that almost empty. If you pull the barriers, it will still take a while for the empty stretch of highway to fill up with cars again, and the cars at the back of the line still have a long wait ahead before they can get up to speed.

Just another great goverment run program...

[round4]

Yea, these are the guys (the feds) I want building cars, taking over health care...thanks god they are not building the planes. I'm just trying to think one government run organization that works as well as any private one. Any idea?

Re:Just another great goverment run program...

[DerekLyons]

Yea, these are the guys (the feds) I want building cars, taking over health care...thanks god they are not building the planes. I'm just trying to think one government run organization that works as well as any private one. Any idea?

Under their control, 87,000 flights a day cross the skies of the US. Despite incredible crowding at and in the airspace surrounding a large number of airports, collisions are one-in-billions events. The vast majority depart and arrive without undue delay. (And anyone expecting no delay in such a dynamic system with so many variable is smoking some good stuff.)
 
In this incident, a problem was detected, backup procedures implemented, the problem was fixed, and full functionality was restored - all in a matter of hours without halting the system.
 
I'd be the first to admit that there are a lot of badly broken government programs - but in this case, you're just blowing smoke.

Re:Just another great goverment run program...

[Anonymous Coward]

Yea, these are the guys (the feds) I want building cars, taking over health care...thanks god they are not building the planes. I'm just trying to think one government run organization that works as well as any private one. Any idea?

This morning I was woken by my alarm clock powered by electricity generated by the public power monopoly regulated by the US Dept of Energy. I then took a shower in the clean water provided by the municipal water utility. After that, I turned on the TV to one of the FCC regulated channels to see what the national weather service of the national oceanographic and atmospheric administration determined the weather was going to be like using satellites designed, built and launched by the national aeronautics and space administration. I watched this while eating my breakfast of US Dept. of Agriculture inspected food and taking the drugs which have been determined as safe by the food and drug administration.

At the appropriate time as regulated by the US congress and kept accurate by the national institute of standards and technology and the US naval observatory, I get into my national highway traffic safety administration approved automobile and set out to work on roads built by the local, state and federal departments of transportation, possibly stopping to purchase additional fuel of a quality determined by the environmental protection agency, using legal tender issued by the federal reserve bank. On the way out the door, I deposit any mail I have to be sent out via the US postal service and drop the kids off at the public school.

After work, I drive my NHTSA car back home on the DOT roads, to a house which has not burned down in my absence because of the state and local building codes and the fire marshal's inspection, and which has not been plundered of all it's valuables thanks to the local police department.

I then log onto the internet which was developed by the defense advanced research projects administration and post on freerepublic.com and fox news forums about how SOCIALISM in medicine is BAD because the government can't do anything right.

Re:Just another great goverment run program...

[MaWeiTao]

I have a lot of problems with government and many departments within that bureaucracy, but the FAA is not one of them. It's pretty amazing what they do and there probably aren't many other entities out there that could do what they do.

Re:Just another great goverment run program...

[ak3ldama]

Considering how many companies have failed in the past 200 years, how the financial industry was central to our current economic crisis, and how we're still living under the same government we had 200 years ago, I'd say your faith in the private sector is severely misplaced.
But don't let that stop you.

I had a longer comment prepared, but I am going to shorten it. The government's failures would be as easily producible as failures of industry. Even worse is that government departments are usually not created until a real need is noticed - which should increase their chance of success. Also, your example of the financial industry is a joke, and not relevant to any discussion of a free market. Lastly the proposition that we are under the same government we were under 200 years ago is only acceptable upon a cursory glance. I would love to go into greater detail but I refuse to against anonymous cowards.

Re:Just another great goverment run program...

[dkleinsc]

A government-run organization that works as well as a private one: US Postal Service.

Seriously. Those guys move tons of stuff around the US every day, and significant portions of daily business would grind to a halt if they stopped. Think about it: for around 50 cents, you can send something from Maine to California and expect it to arrive within a few days. They make extensive use of OCR, have an extremely sophisticated sorting and routing system, and are in many ways more efficient than FedEx or UPS (for instance, FedEx will route packages through Memphis while USPS will send them more directly).

Re:Just another great goverment run program...

[round4]

The radar technology used by air traffic controls is from the 50's. No GPS... Planes are still tracked by little "physical" slips of paper. The entire system has been in need of an overhaul since the 70's. Planes still use land based radio towers - waste tons of fuel due to indirect routes. The FAA is incapable of fixing this decrepit system. Please share with me your on-time stats. I know there few mid air collisions but there are pilots in those planes to stop them from crashing. "without halting the entire system" - maybe you missed the point of the article.

Re:Just another great goverment run program...

[DerekLyons]

Repeating the usual party line like a parrot, all the while ignoring the points I made, very impressive.

Not.

Re:Just another great goverment run program...

[DerekLyons]

Bravo, bravo! A few minor nits to pick and some stylistic points, but against such a performance I'll let them pass. Again, bravo!

Re:Just another great goverment run program...

[Starker_Kull]

The radar technology used by air traffic controls is from the 50's. No GPS... Planes are still tracked by little "physical" slips of paper. The entire system has been in need of an overhaul since the 70's. Planes still use land based radio towers - waste tons of fuel due to indirect routes.

That's true. The U.S. GOVERNMENT built the best air traffic control system in the world back in the 50's, one so robust, well designed and well managed that it has been the safety and operational count leader world-wide for more than half a century despite ongoing neglect under the Reagan, Bush I, Clinton, and Bush II administrations. I wonder what they could do if we actually spent some money, collected from taxes, to upgrade it? Wow. Infrastructure building. My dad always said it was cool.

Re:Just another great goverment run program...

[sahonen]

IANA Air Traffic Controller, but I am studying to be one.

The radar technology used by air traffic controls is from the 50's.

If by this you mean "the basic technology was invented in the 1950s," then sure. If you know of a better general-purpose way to find out where things are in the sky than to bounce radio waves off of them, I'd like to hear it. That said, the systems which process and display radar data however have received several updates since the 50s.

No GPS...

Are you referring to ADS-B? It's a system which allows planes to broadcast their GPS position back to ATC as part of their transponder echo. It works *in *conjunction* with existing radar technology, not as a replacement. In any case, implementation of ADS-B in the US is already underway and is scheduled to be completed by 2013. The entire east coast, parts of the upper midwest and areas in the northwest are already covered.

The major concern of ADS-B is that is is susceptible to hacking (you can broadcast your own "phantom" aircraft if you know the standard) and reduces the anonymity of private and business aircraft users (every ADS-B transponder broadcasts a code that uniquely identifies an individual aircraft).

Planes are still tracked by little "physical" slips of paper.

All of the information contained on the flight progress strips is available on the air traffic controller's console, the strips just serve as a backup system in case of radar or computer failure. They're hardly necessary for actual operations. Some centers have computerized the strips but it's actually less efficient. It's easier to look over at the strip bay and mark it up with pencil than to bring up the computerized equivalent and enter data into it. It's the old "NASA Space Pen" vs "Russian Pencil" problem, sometimes low-tech just works better.

Planes still use land based radio towers - waste tons of fuel due to indirect routes.

The airways are based on the locations of radio navigation beacons, but the planes don't use the airways because they need the beacons to navigate, in fact GPS is effective enough that many of the currently existing beacons will not be replaced at the end of their service life. The airways are used to create a more orderly flow of traffic which is easier and safer for controllers to manage. That said, controllers will often allow aircraft to skip segments of their flight plans and fly more direct routes as workload permits.

Re:Just another great goverment run program...

[RLaager]

The major concern of ADS-B is that is is susceptible to hacking (you can broadcast your own "phantom" aircraft if you know the standard) and reduces the anonymity of private and business aircraft users (every ADS-B transponder broadcasts a code that uniquely identifies an individual aircraft).

Aren't the pilots already doing this with their radio (the code being the flight/tail number)?

Re:Just another great goverment run program...

[sahonen]

Non-commercial flights have the option of flying under Visual Flight Rules (aka VFR), in which case they are not required to be in radio contact with anybody except while flying in the airspace around towered airports. You can fly through 99% of the country under VFR, and you'll only ever have to key your radio at the start and end of your trip to either talk to a tower or on the traffic advisory frequency at an un-towered airport. Also with ADS-B anyone with a receiver can plug it into their computer and leave it running, record a bunch of flights, and correlate the codes with airplane owners fairly automatically. Much harder to do than with just voice communications. It's akin to the difference between sitting by the side of the road writing down license plate numbers and putting up a camera with OCR software.

I'm this many.

[StikyPad]

It was the second time in 15 months that a glitch in the flight plan system caused delays.

Thanks for the arbitrary use of months. Is it a baby, or were you working from fortnights and thought you might as well round to the nearest lunar cycle and then convert to Gregorian? Do you also append "and a half" to your age, as appropriate?

Yeah, right. A cynic's view.

[Bob_Who]

Its true the flights are delayed. That's as much of the truth we get from this complex system of terribly overworked government and airline employees, national security, information systems, politics, FAA, corporate scum, secret clearance computer firms with bizarre contracts, and the 24/7 news spew with commercial airline as clients. They start by overbooking flights, while trying to account for each seat for security, and the bean counters, vips, and bad weather. To top it off information systems have to share some data, but not other data, while the people working have to comply with all sorts of rigid protocols and odd project management that controls everything except their overtime. I dunno, I have no reason to believe that any news we get about this type of event is necessarily true. It doesn't need to be. Its just the spin that reporters have to make in the absence of any real statement for the record. If they told you the truth you'd stop giving them your money. God forbid airlines make customers a top priority. Ha! Even if they were telling the truth, it would be a fluke, but just as vague and pointless. OK. We know the truth. Its aliens from Orion preparing for 2012. And they work for Delta.

Re:Hmm could it be a windows problem? - of course!

[poetmatt]

apparently people didn't understand the difference between the question mark at the end of my sentence and a statement, since people decided to mark it a troll. Do people not know what a question mark means? I was hoping someone would reply with actual informative info.

Re:Hmm could it be a windows problem? - of course!

[pwfffff]

Do you know what 'troll' means? I await you informing me of your informative information, informant.

Info.

the best layover

[nimbius]

ive had was a 7 hour standby at a connecting airport that was 45 minutes away from my house, only to be flown back to my original airport and directly flown to houston. downsides: i ate so much au bon pain at the airport i had these weird pastry farts the entire flight, and so much coffee i thought i was going to barf before we took off.

Re:Hmm could it be a windows problem? - of course!

[Chibi Merrow]

Well, I work w/ the FAA right now, and they're becoming relatively platform agnostic actually. ERAM, for instance, was written by Lockheed Martin to run on top of some flavor of IBM UNIX or Linux (forget which). In the old days, everything ran on custom, purpose-built hardware and OSes, but that really turned out to be a maintenance nightmare. Using COTS Hardware/Software means upgrading systems or providing new capabilities can be pretty easy.

I think ASDE-X runs on top of some sort of POSIX type OS as well... I know its data stream is standard UDP over Ethernet type stuff.

Re:Hmm could it be a windows problem? - of course!

[poetmatt]

maybe you should do a little research, because I did.
http://www.stratus.com/news/2005/20050314a.htm Do you notice something? They installed windows due to defining it as an "open platform". They were deceived. This is the result.

Stratus was chosen as best able to provide an open platform with 99.999 percent uptime reliability – which is mandatory for running an application as important as NADIN 1 – together with the required caliber of maintenance, logistical support, and long service life.