Slashdot Mirror
Embedded RTOS Maker Raises Linux Security Issues
drquizas writes "Embedded RTOS provider Green Hills recently delivered an address where they raised the question of whether Linux can be considered secure enough to be used in defense applications. Much of the usual FUD is present in the remarks, although an interesting question is raised regarding what defense and other government contractors are required to do in testing code (in this case anyway): is the closed code here being held to a higher standard than its open-source equivalent, and does this change the 'security through obscurity' argument?"
"Open Source is actually more secure than closed source proprietary software because the oversight of technology content is broader and deeper. Instead of just one company monitoring its own contributions -- or potentially hiding security holes and exploits -- a worldwide community of interested parties actually oversees Linux to make it strong and secure. That's why the NSA -- the most security-conscious organization in the world -- chose to standardize on Linux, and even supplies its own version of secure Linux."
Can't put it much better than that. When you have the contribution of the entire open source development community, so much knowledge and experience comes to the table that it's difficult for any one group of programmers to compete.
Wireless News www.DailyWireless
on a related topic... here.
that Linux can be made pretty damn secure.
If they have faith in it....
http://www.nsa.gov/selinux/
Green Hills calls Linux 'insecure' for defense
By Alexander Wolfe, EE Times
NEW YORK -- A storm has erupted in the embedded community, with real-time operating systems house Green Hills charging that Linux is fundamentally insecure and wide open to security breaches by "foreign intelligence agencies and terrorists."
The explosive charges were made in a speech delivered Thursday (April 8) at the Net-Centric Operations Industry Forum in McLean, Va., by Green Hills chief executive officer Dan O'Dowd.
"Now that foreign intelligence agencies and terrorists know that Linux is going to control our most advanced defense systems, they can use fake identities to contribute subversive software," said O'Dowd, in a copy of the remarks released by Green Hills.
"If Linux is compromised, our defenses could be disabled, spied upon or commandeered," O'Dowd continued. "Everyday new code is added to Linux in Russia, China and elsewhere throughout the world. Everyday that code is incorporated into our command, control, communications and weapons systems. This must stop."
O'Dowd laid out a scenario in which the open source development process -- where thousands of programmers contribute code that's subject to public review before being folded into Linux -- could be subverted via "Trojan Horses" illicitly slipped in the software.
At least one embedded expert thought O'Dowd was overstating his case. "I think it's pure FUD [fear, uncertainty and doubt]," said Rick Lehrbaum, a respected board-level-computing guru and former president of Ampro Computer and currently operator of the developer site LinuxDevices.com. "I think the insecurity he's concerned about is an intentional back door and this [Linux] is the most transparent operating system in existence."
Several programmers on the Linux street are also giving O'Dowd some pushback. In a reader's forum on the LinuxDevices.com Web site, a developer who identified himself only as "Concerned citizen" posted a lengthy rebuttal. "[Linux] has features, security, and strengths that are not easily compromised by a foreign agency," he wrote. "Let's not forget that the terrorists that Mr. O'Dowd refers to used proprietary software for attacks on the USA. They have Windows machines and Flight Simulator, you might recall."
O'Dowd claimed the salient issue is that Linux isn't held to as a high a security standard as is the proprietary "Integrity" RTOS made by Green Hills. "If all they would do is hold Linux to the same standard they hold us to, I'd be happy," O'Dowd said told EE Times.com. "At the [Federal Aviation Administration], they have received from us documentation of every single line of source code and tests of every line of code and boundary condition. It costs us $500 to $1,000 a line to review our source code. It would cost billions of dollars to review Linux."
O'Dowd's tough stance may attract attention because he is also taking an unusual public stab at a competitor -- embedded Linux powerhouse MontaVista Software. "MontaVista is outsourcing their development to Russia and China. That's not wrong if you're building toaster ovens," O'Dowd said in an interview. "If you're building national security applications, that's a different story. Nobody's even checking if there's anybody putting anything [dangerous] into Linux."
In response, said MontaVista CEO Jim Ready said Linux constituted a threat to vendors of proprietary software, because of its robustness, cost-effectiveness and its security.
"Mr. O'Dowd makes the common mistake of confusing obscurity with security," said Ready. "Open Source is actually more secure than closed source proprietary software because the oversight of technology content is broader and deeper. Instead of just one company monitoring its own contributions -- or potentially hiding security holes and exploits -- a worldwide community of interested parties actually over
in-house code, as well.
The advantages of closed source coding seem to me to be a faster development time, stronger integration of components, and more support. The drawbacks, though, are that you are ultimately trusting somebody else.
Open source code, I would say, is more secure overall - there are more people looking at the code, so it is less likely that bugs slip through. The drawbacks would be that open source is less custom-made and possibly less supported than the rest (also, as O'Dowd would have it, people 'contributing' backdoors).
As for simply writing your own secure code (an agency doing this, that is), it's obviously just more expensive.
The best solution, in my opinion, is to make your own custom flavor of Linux that is open to all, but contribution is regulated so no questionable code can be admitted - the tack taken by the NSA.
webpage
First, this isn't the first time that Green Hills has come out complaining about Linux, you may remember a previous slashdot story where they claimed that the embedded linux tools market was a myth. Secondly, this article, like their previous one is through EETimes. If you've ever read EETimes you'll know why that should make you question the quality/validity/truthfulness of all the statements in the article.
Basically, Green Hills seems to be just another proprietary software vendor scratching for ways to try and derail a competitor in their market space. Nothing to see here. Move along now.
My Slashdot account is old enough to drink...
Green Hills isnt comparing Linux with Windows, but rather with their own RTOS, Integrity. Which in that case, THEY ALREADY OWN AND CONTROL THE BLEEDING SOURCE CODE!
They got a Class C license (whatever that means), but only on the condition that it wasn't connected to a network.
I think we've pushed this "anyone can grow up to be president" thing too far.
Some good reading about this topic can be found here.
As someone who's used GHS compilers and knows other industry folks who have as well, I've got to say the almost unanimous opinion is that Green Hills are a bunch of schmucks.
To be fair, their core compilers are actually pretty efficient in terms of code speed and density. However, the products are annoying to use, particularly with non-GHS tools, their sales reps are full of BS, and their technical support is spotty at best. Their toolchains are also quite expensive per seat. Many companies get a seat for doing benchmarks, and that's it, since it'd be prohibitively expensive to deploy to an actual development team.
Furthermore, I have it on good authority that at least as recently as a year or two ago, their internal revision control was pathetic to non-existent. Customers would experience very odd bugs that GHS would have difficulty reproducing. When and if the bug was fixed, that individual customer would get a one-of-a-kind patch from a software dude in GHS. Another company, also a GHS customer, asking about the same problem would usually never find out from GHS that a patch existed, since the patch usually only existed on that one GHS dude's hard disk. No real central software repository or proper version control.
And customers could get in trouble if they passed such a patch directly to another customer! Even the version numbering of software releases was inconsistent and no guarantee that all bugs fixed in a lower version of software would have been included.
I'll take open source tools any day! (Which is not BTW mutually exclusive with doing business with companies like Montavista, Redhat, etc.)
It seems that GHS may be doing a fair amount of business in the defense and government contractor sector. Places where technical recommendations from the design engineers don't always get the proper consideration from the folks who sign the actual contracts.
In the article, he states that anyone can contribute backdoors/trojans into the code because nobody is looking at the code. This is completely and utterly wrong. I'm pretty sure that to insert code into the kernel, you have to sign up to the mailing-list, and send in a diff. There, other kernel hackers can easily see the code, and if Linus accepts it, it goes into the tree. Even though anyone around the world can do this, the process is fairly strict.
Anyone want to place bets that Microsoft paid him to say that?
Founder of Mirror Moon - Tsukihime Game Trans
Actually, this is somewhat misleading. The top players listed are Microsoft (Windows CE), Wind River, Symbian, Palm. QNX, OSE, and Green Hills. But Microsoft, Symbian, and Palm are really selling into handheld devices, not hard real-time control. (The phone and PDA markets are much bigger than real time control, though.) Wind River's VRTX is the dominant player by a big margin, especially in low-end embedded control. QNX is next, and is usable on a broad range of platforms. Wind River is more of a specialist maker catering to the military Ada market.
Following these seven come LinuxWorks and MonteVista, who are moving up. These are the main Linux-based offerings.
Also confusing the issue is Windows XP Embedded, which is basically a Windows XP from which you can delete stuff you don't need. This sells more into point-of-sale applications than hard real time control.
Who audits the open source software that they use? I certainly don't. I rarely even bother to look at the source. So in this respect, it doesn't matter (to me) if the software is closed source or open source since the code isn't looked at even if you (I) had the chance to.
Let's say you're in a major city. Let's say there's a small, narrow street. And let's say you walk down this street twice; once in the middle of the day, when the street is well-lit and crowded. And the second time in the middle of the night, when it is empty, abandoned and dark.
In which of these situations do you feel more safe?
I would probably guess the second. Why? In both the first and the second case, you have no way of knowing if there is someone who has a weapon and wishes you harm. But in the first case, this possibility is not something that worries you.
Open source is kind of like the well-lit, crowded street, and auditing is kind of like being able to tell if someone wishes you harm. First off, you don't *have* to be constantly watching your back; it's more than likely someone is looking at your back at any given moment, and can tell if someone is walking up behind you with a blunt instrument. Second off, the fact that *everyone knows people are watching* means it's less likely anyone will try anything, because they know they'll get caught and have messy problems with the law.
Note that this argument does not go so well unless the open source product is relatively well used. If you're the only user, well, you're not much better off.
---
Anyway, as far as (1) goes, I would imagine that while it may probably be a very important point insofar as you go, as far as the kinds of software discussed in this article go it's not so useful. You probably don't audit the open source software you use. But a propreitary embedded RTOS vendor certainly would, since their demands for security and reliability are much higher.
But wait, you say, wouldn't the need to audit the code at that level be an argument against open source, since it destroys the "free" nature? Well, here we run into the basic problem of the conflation of "free" software with "open source" software, or the conflation of "free" software with what RMS might call "software libre". These two concepts are often described by the same term. However, they are not the same thing!
A program being open source does not mean that it can have a trusted company of some kind behind it! A company such as IBM could be providing an open source program they internally wrote, or you could have a case such as that with MySQL (ok, maybe that's not a great example, but you know what I mean) where a community-developed program passes through a certain central trusted point (MySQL AB) which can be trusted to-- or demanded to-- perform the auditing for you. So this is not a problem with Open Source, this is just a problem with software you downloaded for free off the internet.
Irritable, left-wing and possibly humorous bumper stickers and t-shirts
Having worked as an systems administrator on DoD programs, I can tell you for a FACT that ANY software that goes on mission-critical systems is either developed in-house or very throughly scrutinized. They do code review, bug fixes and testing in a continuous cycle to get all the software bugs out. (This is one of the big reasons you hear about DoD projects going over time and/or over budget).
If COTS products are used, the DoD programmers will test the software for defects and ask the vendor to correct the defects they find. There have been cases where the DoD has signed NDAs to gain access to source code for COTS software to fix bugs that caused problems with the DoD software that the software company WOULDN'T fix. This has even been done to find backdoors, trojans, and other bad things that disgruntled employees of proprietary software vendors have put into that company's products.
OSS gives the DoD the power to make the changes they want to secure their systems the way they want. They WILL go through the code and look for backdoors, trojans, viri, etc. They may even set up their own repository and fork the kernel. Once the DoD has a trusted version of Linux, they'll use it in-house. I suspect that most DoD programs looking at Linux are probably testing NSA's version.
The DoD should be able to release some of the improvements they make back to the community, but don't expect them to release everything. The military still has it's secrets.
If "disco" means "I learn" in Latin, does "discothèque" mean "I learn technology"?
I used to work as a Defense contractor and I spent quite a lot of time going through the various processes. As a Linux developer, I can certainly say that Linux has not been developed to the same standards that the projects I've been involved on have.
For starters, in the DoD, every line of code is reviewed by hand by a team of reviewers (usually 4-5). There are records of all the defects found and verification that fixes were made. After the initial development cycle, there's a rigorious testing phase where all specifications are tested, senarios are ran, and stress tests are performed. Any defect from this testing is recorded, and the software doesn't ship until it's fixed. This usually ends up being a 2-4 year process of just doing bug fixes.
And for those that don't know the difference, Windows is *not* certified for tactical use. Having EAL4 is not the same as being certified for tactical use.
It's really a different type of software. It's not that Linux isn't good piece of software, it's just that it wasn't developed for this type of work. There's nothing wrong with that.
int func(int a);
func((b += 3, b));
It might be more like $2,000/line - or more
There's also the issue of what kernel version you want to run - once you've decided on a certain version, it will be extremely painful to updtae to a new one. You've also have to validate that the compilers are generating the expected code. Compared to a well designed RTOS, Linux is bloatware.
One thing many Linux fans forget is that there are situations where you do not want to use Linux - pretty much the same way explaining to PHB's that there are situations where you do not want to use Windows.
Linux is a general purpose OS and there are places where general purpose OS's don't cut it.
A Shadeless room is a brighter room.
It is worth adding to your point that the 'Cathederal Method' versus the 'Bazaar Method' is not an open versus closed source schism. Raymond wrote that essay as a criticism of the small-group closed way that the GNU Emacs team was doing their work. It's quite possible, and rather common for source-disclosed projects, even ones that are released with the GPL or BSD licenses, to be developed by small closed groups who don't actively solicit outside code.
resigned
oh yeah I forgot. Diebold uses russians. Sequoia is foreign owned. Shouptronics founder served time for Election Machine Rigging, Seqoia execs indicted for bribing election officials, and GEMS VP of research served time for computer fraud.
In contrast OVC is a multi-national effort but its all open source so no one cares there be foreign nationals programming US machines.
Some drink at the fountain of knowledge. Others just gargle.
With all the off-shoring of work that large companies like Microsoft, HP and IBM do there is at least a perception on their part that when selling to the DoD that they should downplay the fact that foreign nationals, in foreign countries, not only have read access to the source code for the OSes (NT/XP/HPUX/AIX) that most DoD contractors don't have themselves, but that these same foreign nationals also, in many cases have write access to that source code too. Whether most DoD contractors care, I don't know, but like I said, the vendors often remind their customer interaction people to gloss over those kind of details.
When information is power, privacy is freedom.
There is a real time variant of linux called RTLinux which is used across a large number of embedded and real time sensitive industries to great effect. Real time scheduling is exactly that, mostly about the scheduler. Linux is modular enough that it has had no less than four different schedulers in the last two major kernel releases, not counting the RT variant. If you can have as large a swath of the code peer reviewed as possible then I don't see where you can really go wrong, if you feel more testing is needed then put dollars into testing, not into creating new code that is then going to need just as much testing!
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
If you've got your real time system in rom inside a piece of equipment, or in thousands of pieces of equipment, you've got to be very careful with it.
Desktop system can be patched and upgraded, but ROMs have to be replaced or flashed. For example, you've got to bring the missle into the hanger/lab and hook up the reflashing unit or swap out the ROM chip. You've got to test the missle with the new chip. Out in the field, the soldiers have new ly upgraded missles (or tanks...) and would really like to know that it will work when they need it. You can field test a tank, but some missles are expensive, especially when all you want to do is prove you installed the right chip in the right way.
When a desktop or server software hiccups, the human user can work around it. Often this is not the case in communications and avionics.
Linux Advantages Don't Translate to Military Embedded Systems
Embedded systems are almost always memory-resident and have no disk for software storage. There are usually no user identities to manage, and the user interface is quite often absent or primitive.
Most of the advantages of Linux do not apply to an embedded, military situation: Licensing fees for software are usually a negligable part of a tank, missle or radar. Embedded RTOS systems are already quite reliable, and do not suffer from nearly as many buffer overruns, neither are they susceptible to hackers. Embedded military systems are almost never connected to the internet.
You could build a reliable, compact embedded software system from embedded Linux, but you'd want to write all your own drivers and you would have to port it to special hardware. This approximately the same amount of work that you would have to do if you were to use a proprietary RTOS.
The vast bulk of the problems users experience with proprietary OS's are 1) expensive to license, more expensive to license across many machines. 2) Security vulnerabilities resulting from using a system designed and built for stand-alone personal use in an internetworked environment. Neither of these problems matters much to embedded, military systems.
I18N == Intergalacticization
What is time any way.....
Each application has different needs, so some might need 1ns accuracy (nuclear detonators/patriot missiles) , and some might need 20ms accuracy.
Besides linux there are other RT free OS's , like TINYOS - http://webs.cs.berkeley.edu/tos/
or mini NetBSD.
Linux isnt the only thing around, theres a lot of choices which are free, just that linux gets the attention, both good and bad.
Liberty freedom are no1, not dicks in suits.
I used to work with FAA certified aircraft software. I can say that whether or not software is open or closed source makes NO DIFFERENCE in whether it can be approved. What makes a difference is the process.
When software is written for aviation, it must have strongly defined requirements. It is wise to pass these past the government for approval to avoid wasting time later. Then, each function must derive directly from the requirements. Each function must have a corresponding test suite written by a different person who does not communicate with the code writer (they can only share the requirements document). Additionally, once the code is 'passed over the wall' to the tester, a certain amount of each for-loop must be exercised in testing, and if or case statements must be tried.
If the software passes this, then it is integrated and tested as a whole. There must be test documents to support how this is done. All test results, data, and code must be sent off to the government for approval. No code changes can occur during the entire process without performing some of the tests again.
If you follow this process, your open source project can be certified. However, it is terribly expensive in terms of man hours. More time is spent on test and planning than on code writing. This is why most FOSS projects will never make it in Gov't apps - but they could if you spent the time.
Since the FAA is pretty stict, I assume the defense process is similar.
What developers can do to help is to modify a web server, a mail server, and a DNS server (the most attacked server side software) to run under NSA Secure Linux, partitioned into levels of integrity.
The idea is that just because somebody attacks, say, the mail server receive program, that doesn't get them power over the whole system. All it should do is let them run their attack code in a jail where it can't do anything except burn CPU cycles, and maybe generate phony incoming mail. Mail associated with the known IP address from which that copy of the receiver was launched, so the problem can be tracked.. When they disconnect, or some monitor program kills the corrupted component off, all the damage should be flushed.
You need to rework key server apps so that about 95% of the code is untrusted and jailed, while the 5% that has to do security-related functions is isolated, identified, and carefully examined.
That's what NSA Secure Linux is for.
"As a cryptography and computer security expert, I have never understood the current fuss about the open source software movement. In the cryptography world, we consider open source necessary for good security; we have for decades. Public security is always more secure than proprietary security. It's true for cryptographic algorithms, security protocols, and security source code. For us, open source isn't just a business model; it's smart engineering practice."
Exactly right - so much so that I've asked my editor at linuxinsider (which will be offering a rebutal of my own soon) to contact him for permission to reprint his whole article on the subject. That may not happen, but you can look at it directly on his website: http://www.schneier.com/crypto-gram-9909.html and sae how an expert handles the somewhat loony argument against open source in high security environments.