Slashdot Mirror
US Switches Air Traffic Control To New Computer System
coondoggie writes: The Federal Aviation Administration this week said it had completed the momentous replacement of the 40-year-old main computer systems that control air traffic in the US. Known as En Route Automation Modernization (ERAM), the system is expected to increase air traffic flow, improve automated navigation and strengthen aircraft conflict detection services, with the end result being increased safety and less flight congestion. The FAA said the Lockheed Martin-developed ERAM systems “uses nearly two million lines of computer code to process critical data for controllers, including aircraft identity, altitude, speed, and flight path. The system almost doubles the number of flights that can be tracked and displayed to controllers.”
Its still a bandage. I see that ADS-B is still crap and not authenticated at all. My home wifi using years old technology is still better than this junk.
For delays and glitches...
So how does a 40 year old computer system get replaced and only doubles the number of flights capable of being tracked?
40 years & merely "almost doubles" performance? Sorta sad.
/me waits to hear that it's Windows-based...
Moore's law times government equals...
what could possibly go wrong?
Where are we going and why are we in a handbasket?
Oh... and you will need another entirely new system to accommodate drones.
"If any question why we died, Tell them because our fathers lied."
Ha ha
Hmmm. People are still the same size, fuel is still the same, turbines still use the same theories, the planet hasn't gotten bigger, the atmosphere is still the same, our materials are still the same..
Could it be, and this might be a shocker, could it be that the limits on materials have nothing to do with information processing?
For example, you might want to sit down for this and read it a few times, could it be that just because processors got a thousand times faster it doesn't mean that we can somehow actually put a thousand times more airplanes in the air?
I'm just wondering out loud here.
Wait, you write a new application from the ground up to operate on new hardware, in an era of grid computing, ridiculous amounts of possible ram and multi-core compute nodes, with modern programming structures that can hold obscene amounts of data in a single variable.... and you only managed to "double" the number of flights which can be tracked and analyzed?
"The system almost doubles the number of flights that can be tracked and displayed to controllers."
Can the air traffic controllers sort them out on the display in real time?
According to researchers with MITRE and other experts, this hybrid system is the FAA’s first challenge as a system made up of both IP-connected and point-to-point subsystems increases the potential for the point-to-point systems to be compromised because of the increased connectivity to the system as a whole provided by the IP-connected systems, the GAO stated.
“The older systems are difficult to access remotely because few of them connect from FAA to external entities such as through the Internet. They also have limited lines of direct connection within FAA. Conversely, the new information systems for NextGen programs are designed to interoperate with other systems and use IP networking to communicate within FAA. According to experts, if one system connected to an IP network is compromised, damage can potentially spread to other systems on the network, continually expanding the parts of the system at risk,” the GAO stated.
"If any question why we died, Tell them because our fathers lied."
"The FAA is moving steadily toward replacing the old system of ground-based radars to track aircraft with one that relies on satellite-based technologies."
Better at tracking those pesky drones they wont allow.
It was a shortage of computer memory in the $2.4 billion air traffic control system while a U-2 spy plane flew over southwestern US that caused LAX computers to crash and hundreds of flights to be delayed on April 30. “In theory, the same vulnerability could have been used by an attacker in a deliberate shut-down,” security experts told Reuters. Now that the “very basic limitation of the system” is known, experts expressed concerns about aviation cyberattacks.
$2 billion air traffic control system failure blamed on shortage of computer memory
Lockheed Martin, which created the En Route Automation Modernization (ERAM) air traffic control system, claims it conducts "robust testing" on all its systems, yet the lack of altitude information in the U-2’s flight plan caused the automated system to cycle off and on trying to fix the error.
http://www.computerworld.com/a...
"If any question why we died, Tell them because our fathers lied."
It's mostly Ada running on AIX. See http://www.iaeng.org/publication/IMECS2009/IMECS2009_pp1095-1099.pdf.
"Display System (DS), User Requested Evaluation Tool (URET) and ERAM and have been developed mainly in the Ada programming language. " Page 2.
"Product supportability advantages led to the selection of the IBM P series processors, the AIX operating system, and CISCO switches." Page 3.
http://imgur.com/R16EFCd
- In Soviet Korea, only old people loose all their bases to Natalie Portman's petrified hot grits overlords.
ERAM is written in Ada.
I say this as a thirtysomething computer programmer, although I've also always been a minimalist: Given the choice between something that uses software and something that does not, go softbare.
My car, TV, and entire life are now filled with much more software than ever. Now that they can "do" more, they are also slower, flakier, and more complicated. And as a computer programmer, I know why: even the simplest program is amazingly complex. Every keystroke is a pitfall.
Two million lines? I think I'll drive --- no, just walk.
can't they do it in one line of perl?
Simple solution. VFR. Why make things more complex? Contractors are getting rich from public money. https://en.wikipedia.org/wiki/...
Glad I am not flying anywhere for a little while.
Systemd yes or no discuss.
Probably outsourced to chinese coders. Everything is for sale in the corrupted regimes in play these days. From Loral space technology creating chinese missles, to chinese spies at lawrence livermore and sandia and LBNL, to selling our ACA odeathpanelcare to a devloper in canada to whatever - giving away panama canal and the control of our internet - everything we fought to invent and build from scratch we give away to enrich the thieves in government.
You are largley right here, the gains in thoughput in the system will be made by reducing seperation between aircraft, so you can have twice as many aircraft on the same airways. Those reductions in seperation can only go so far, as you have to have a system that can still fail back to stone age (100% down) and still be reasonably safe. At that point controllers fall back to using primary radar, radio and bits of paper in stacks, i.e. how it used to be done before computers.
The improved processing and tracking allows some saftey margins to be compressed, but not many, and not by much.
What protections do the ground stations have from ADS-B being spoofed? Seems to me that this protocol that was slightly extended from the 70's version isn't very secure.
More downtime. More Costly repairs. Less reliable technology. Downtime for updates. Downtime for patches.
New technology is always twice as expensive and half as good as the good old stuff.
No Y\iipppeee! for the FAA!
Please note, the same objection applies to the space fantasies so prevalent among programmers and other nerds.
We live in modern times where all reality is produced by FB. Besides in all movies I have seen for years everything of value is produced by click of a mouse or stealing of fort Knox gold or some other such thing. In other words physical limitations do not apply on reality created by Zuckerberg & Co.
First off, I suspect there were some women working on the project. Even back in the stone age of computers in the 1970s the team I was on doing air defense and dispatch systems was probably 40% women. Get with the program: "work hours".
OK.. a good "all in" ratio is 10 lines/work hour (that counts building requirements, actual coding, test, docs, etc.)
so 2E6 lines of code is more like 200,000 work hours. A typical toiler does about 1700 hours/year (after you take out vacations, holidays, "here work on this other project", etc.) : just around 120 work years. Where I am now, we figure a work year costs about 300k (salary+benefits+taxes+overheads+management), which comes out to around $36 million.
This is what one the old systems looked like
Stopped reading right there.
uses nearly two million lines of computer code .... The system almost doubles the number of flights that can be tracked and displayed to controllers
Nearly two million lines, and almost double the capacity... If they bumped it up to an even two million I wonder if they could've completely doubled the number of the flights that could be tracked.
And what if they expanded it to four million lines of code, could they have quadrupled the number of flights that could be tracked?
And what if they made the code self-replicating? Could they have support an infinite number of flights?
I believe a bunch of venerable DEC PDP's were running the show with some IBM disk controllers using something like emitter something logic, basically EMP proof (not Hitachi HMET?).
Fine, they replaced something that works, with something that also works but cost a lot of money and unproven (those atmospheric tests did prove what worked back then).
The unspoken 'saving' is what happens if a massive EMP goes off? I can vouch that a quorum of PDP's boot up without missing a beat - and never seen anything better.
I guess the new assumption is civilian traffic will be grounded if such events occur.
Given a Heathkit 2Mhz Z80 with CP/M handled 256 aircraft fine, I don't think CPU grunt is the issue - if you stick with Ford model T vectors, one iPhone could handle all USA traffic easily.
Will my baggage have a better probability of following me to my destination in the same time frame.
According to this government site https://itdashboard.gov/investment/evaluation-history/368 the ERAM system was installed at all locations in mid-2009. It took six more years for the sites to get comfortable enough with it that they were willing to decommission the old system.
So this FAA-sponsored project with critical safety tolerances really took 14 years to develop. Or, to put it another way, if you measured Windows development time using the same milestone that says ERAM took 20 years to develop then you'd have to say that a Windows XP has taken nearly 15 years to develop.
FAA enlists the aid of the flying public to complete testing on 2M lines of new software.
Hope the don't break the Engineer's rule. (Stay out of the news.)
And it's not like the new computer system magically gives the airports more runways.
I don't even want to know how much this fucking thing cost, but it's probably 2000% too much considering how much software companies rip off the taxpayers in this country..
Were all developers of the system required to complete training and pass a knowledge check prior to beginning work?
Has the application had manual/dynamic penetration testing performed against it?
Are there any critical/high/medium findings?
What is the timeline to address pen test findings?
How is access authenticated?
Is the application segmented housed in a dedicated DMZ?
Is there firewalling within the application stack?
Are Web Application Firewalls used?
What intrusion detection systems are in place?
What logs are generated and how are logs monitored?
The usual stuff...you know...before we have a shitstorm in congress about the vulnerability of our critical infrastructure which somehow requires billions of dollars to be paid to defense contractors (like Lockheed Martin...hmmmmm) to mitigate.
.. evil lawyers trying to block Progress!! OMG.. Progress is falling...
btw what is average length of their "Lines of Code"?
Sorry if I was unclear, but I wasnt trying to equate one to the other or say that putting this system together should be about as easy as your average iPhone application.
My intent was rather to give people who dont normally deal with enterprise class applications a point of reference for what two million lines of code is. As I have thought more about it, thats actually a pretty efficient code base for the level of functionality being discussed here.
Its not 2mil means FAA system eq iphone all.
Its Hey, 2 mil, gee thats hard to think about: basically they made a new version of the air traffic tracking and display with the level of raw code that you typically see in a well connected enterprise class iphone app.
i am pretty sure they have built more runways in the last 40 years.
But you're leaving out the previous 20+ years spent developing systems that were never finished.
But you're leaving out the previous 20+ years spent developing systems that were never
finished.
Nope, the clock on that 14 year period counts that prior work on AAS as part of its time period. In reality, the contract for ERAM was awarded in 2003 so if you didn't count the groundwork that was laid by those previous systems you'd have to say that ERAM development took from 2003-2009.
Six years for two million lines of code. When you put it that way it doesn't sound so outrageous, does it?
We haven't even begun to tap operation. Wait until the drones and swarm AI takes flight. Then you will see density. Skylanes will be a thing just to not blot out the sun.
I had heard that the next generation (now current) FAA system was to both reduce separation and enable direct flights, rather than standardized routes that essentially created "highway lanes" in the sky.
This has been needed for a LONG time! It was outdated when I was learning to fly in '70, even NASA updated their mission control a few times in those years. Yes, only doubling capacity is not as much as I hoped, but it should mean if we start developing the next version in 5 or 10 years, we can hopefully have it going before another 40 years are up. The trays of paper tracking chits always made me nervious. I always know being an air traffic controller was stressful, one of my flight instructors taught noobs how to fly as a 'stress relief' from being an ATC at the DFW center.
... "When you pry the source from my cold dead hands."
Back in the 1980s, the FAA's shiny new Advanced Automation System project (AAS) was being designed to replace the 1960s-vintage En-Route system, which used IBM 360/90 and 360/50 computers that were getting to be old, unmaintainable, and unreplaceable. (It was getting hard to even get cable connectors for components - imagine coming up with new SCSI-1 terminators these days.)
As with many military aircraft system contracts, they ran a design competition, which had funneled down from 4 bidders to two by the time I was there. I worked for a subcontractor on one of the teams bidding on AAS. We were the lucky ones who lost; IBM were the poor suckers who won the deal. We learned many lessons about how not to do large software projects. The requirements weren't very well-defined, but the one thing that was certain was that if yet another airplane crash happened, the FAA would take lots of political heat, so everything had to be totally bullet-proof, and every bureaucratic ass had to be covered in triplicate. The phase we were working on was already behind schedule and over budget, and once IBM won it got much farther behind, way farther over budget, and it kind of slunk into the 90s, the 2000s, and the articles referenced above make it sound like Lockheed-Martin bought the IBM Federal division that was working on this debacle.
Originally, the requirements were for 8 9s of reliability (so 99.999999%), but what was worse was that there was no definition of what a failure event was. If a failure meant "each individual radar needed to meet 8 9s", that was hard enough, but if a failure meant "ANY radar's connection was down", that meant the system had to meet 10 9s, not just 8, since there were O(100) radars. Everything had to be triple-redundant to meet those numbers, because taking down any component of a dual-redundant system for maintenance for 5 minutes would blow your reliability for the year. We later found out that the existing 1960s-vintage system that AAS was supposed to replace was shut down for 4 hours per night, replaced by EDARC (a ~1970s upgrade to the ~1950s DARC radar controllers), to make sure that the EDARC system was available as a working backup and that personnel stayed trained in using and maintaining it. (And of course the radars only had dual access lines, with a typical reliability of 3-4 9s each, so 8 9s per radar was already overkill. Phone company equipment with the famous 5 9s of uptime got that by using lots of dual redundancy in appropriate places.)
AAS was originally required to use DOD-STD-2167 software development methodology, a 1985 standard that the DOD replaced in 1988 with 2167A because 2167 was unusable. (You're having trouble dealing with Agile? This is way way far out the other direction.) Both were cumbersome waterfall processes, 2167 requiring something like 180 documents over the predicted 3-year development period, so every week, there'd be one or more new documents, hundreds of pages long, that were all ironclad requirements for all remaining development; developers wouldn't have the time to read and analyze each document and still get their work done, and if they determined down the road that a previous decision had undesirable consequences, there was no way to go back and change it. For example, a decision about whether a given calculation should be done out at the remote radar site, or on one of several central processing computers, or on the computer that drove a given operator console, might turn out to make several hundred milliseconds difference in processing time, but any given radar signal had to get from the remote radar to the console in under 1 second. The subcontractor designing the display consoles knew they wouldn't have the horsepower to do it in time, so they bounced it to the central processors early in the requirement process; those didn't even have an architecture that met the redundancy specs yet, so we didn't know if they'd have the resources to do it in time either. (We later offered to move a bit more of thei
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
The article you're pointing to was about how one of the ERAM systems crashed trying to cope with a bizarre flight plan for a U-2 spy plane.
When I was working on AAS in the late 80s, one thing I was mildly concerned about was that the planned "upgrade" our project was trying to design wouldn't really be able to cope with super-sonic aircraft over the continental US. The requirements for how much area had to show on a controller's screen and how fast the radar sweeps were meant that anything at Concorde speeds would kind of blip onto the screen, maybe bounce once or twice more, and then be gone by the next refresh, either to somebody else's screen or another regional center. Economics and politics (sonic booms, restrictions on what nations' airlines could compete for US markets, etc.) meant that it wasn't a likely prospect anyway, but U-2 spy planes operate under different economics and politics.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
It's designed for object-oriented use, with lots of type specification and such upfront, to push decisions into upfront design time rather than coding time, and it's not as terse as C or APL, but it's nowhere near as verbose as COBOL. I wouldn't use it today (mostly because its main uses are for military stuff I won't do, and for antique maintenance, and it doesn't have all the friendly libraries that I'm used to and probably doesn't easily link to non-Ada systems), but it's a fairly cromulent language.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Most of it on the part of the people who started the original project, who thought it would be done in 3-4 years, made way too many incorrect decisions for the wrong reasons, specified lots of requirements without understanding how impossible they were to meet, picked multiple sets of pie from multiple sets of skies, and didn't start with the ability to get kinds of budget they would have needed to do the job right (if they'd picked a definition of "right" that could have been implemented in the 1980s, when they were trying to replace a 1960s system that had much lower ambitions when it was built, but was still a big upgrade over the 1950s predecessor), but the one thing everybody knew was that if airplanes fall out of the sky or crash into each other, the FAA gets blamed, and if the system's late, the FAA gets blamed, and if it's over budget, the FAA gets blamed, and if the budget had been bigger to start with, the FAA would have been blamed, and if the FAA's going to get blamed, then you can be the contractors trying to design the system are going to get blamed a lot, even just for asking questions when they're working on the thing.
Projects with a scope of tens of millions of dollars are much much different than projects with a scope of a few billions or a few tens of billions. A couple of years after I worked on my part of that fiasco, one of the directors for information systems for one of the National Labs was telling us that he was trying to restructure things to be done in small manageable projects, because he'd never seen the government do a billion-dollar computer project that didn't fail. And all that ancient "Mythical Man-Month" stuff said things you probably already knew about projects in the $10m range sometimes being too large; I remember one much less critical project that had 30 people working on it, so it had to grow to 150 people before it totally failed; if it had started with 5 people instead of 30 and had a budget limiting it to a max of 10, it might have worked. But projects that know they're legitimately in the billion-dollar scale are really really hard.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
That's not even counting the huge amount of code that's designed to make sure all the other parts of the code are working, and to do something appropriate if they're not, and the code that's designed to make sure that code is also working. That stuff's a lot harder than the basic code, and getting it right is the difference between a system with double- or triple-redundant hardware that gets you the 8 9s of reliability the FAA naively thought was possible with 1980s hardware and a air-traffic control system that had triple-redundant hardware running an operating system that crashed weekly (that one was in Singapore, but I don't know if it was actually deployed; I assume they killed it long before it hit the field.)
The 1980s attempt at developing this was only going to be deployed at the ~25 En-Route control centers (with simpler components at the several hundred radar sites feeding each one); it's not intended to be at every airport tower, which was a bunch of different systems.
It's interesting to see how much this thing has grown into, beyond the initial "get radar signals onto the board and replace paper flight-strips and never ever ever crash" goals.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Back in the late 80s, when I was working on that decade's failed project to replace the 360/90-based systems, my coworker and I were in DC for a meeting on some phase of the project (or one of the related projects), and we had half a day spare, so we went to the Smithsonian Air&Space Museum to do "research". They didn't have examples of the system we were working on, but they did have some other air traffic control systems (Tracon, I think), and other cool stuff like astronaut ice cream. After that we went to the National Gallery, because Van Gogh.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Truth. Sounds like just about every government project, ever.