Slashdot Mirror
Embedded RTOS Maker Raises Linux Security Issues
drquizas writes "Embedded RTOS provider Green Hills recently delivered an address where they raised the question of whether Linux can be considered secure enough to be used in defense applications. Much of the usual FUD is present in the remarks, although an interesting question is raised regarding what defense and other government contractors are required to do in testing code (in this case anyway): is the closed code here being held to a higher standard than its open-source equivalent, and does this change the 'security through obscurity' argument?"
"Open Source is actually more secure than closed source proprietary software because the oversight of technology content is broader and deeper. Instead of just one company monitoring its own contributions -- or potentially hiding security holes and exploits -- a worldwide community of interested parties actually oversees Linux to make it strong and secure. That's why the NSA -- the most security-conscious organization in the world -- chose to standardize on Linux, and even supplies its own version of secure Linux."
Can't put it much better than that. When you have the contribution of the entire open source development community, so much knowledge and experience comes to the table that it's difficult for any one group of programmers to compete.
Wireless News www.DailyWireless
quote from this raty-os dude
"It costs us $500 to $1,000 a line to review our source code. It would cost billions of dollars to review Linux."
Say whut? It actually costs this? why? where can I sign up???? I'll sub my per-line auditing out, rake it in...
Naw, cmon, really? the government charges this, or he just pays this cost? Because..huh?
that Linux can be made pretty damn secure.
If they have faith in it....
http://www.nsa.gov/selinux/
"It costs us $500 to $1,000 a line to review our source code. It would cost billions of dollars to review Linux."
Hows that any different from if they chose windows? Wouldnt it still cost them just as much? thats assuming they can get access to the windows code. At least with Linux you don't have to pay to get it.
And no the leaked source does not count.
While it is never good to rely on "security through obscurity", it doesn't mean that it is useless. For example, if after all the thorough testing the same number of bugs were left (hypothetically) in the software, they would be harder to find in the closed system where you wouldn't know where to starting looking as opposed to open source where you could scan the source until you came upon what looked like a vulnerability. The obscurity isn't harmful in itself and it provides an additional barrier. Maybe not a powerful, but every little bit helps. I'd feel a little nervous if I knew some terrorist (as a much over used example) could look over the source code (even if it had no holes!) for a nuclear weapon command centre or something of that sort. I think the ultimate question should be whether the open nature of the open source development can lead to the less bugs - and thus greater security- than closed source development plus the small bonus of obscurity. I think the value of obscurity may have been undervalued in the past, it does have some value.
Your CPU is not doing anything else, at least do something.
There was Cowboy Neal at the wheel of a bus to never-ever land.
I know Sun had to have a special version of Solaris just to meet these needs and Solaris was already considered very secure to begin with. I can't remember if MS released a secure NT for this reason as well or if they tried to and failed.
Talking about the openess of the linux code, there's another question I always wonder nobody asks. Sure Linux is open source and that's what helps it get better but I don't see the argument in terms of cost and security. Saying "you have the source you can see how secure it is" doesn't work for me. People buy an OS because it's cheaper to spend a few hundred or a few grand per PC than it is to hire the staff to build their own OS. Having to have the staff that can review, maintain and patch their own linux kernel alone isn't easy. It's something like 1.5 million lines of code right now. People want an OS that just works and is cheaper than building one themselves.
Open Source Java DAO Generator
First, this isn't the first time that Green Hills has come out complaining about Linux, you may remember a previous slashdot story where they claimed that the embedded linux tools market was a myth. Secondly, this article, like their previous one is through EETimes. If you've ever read EETimes you'll know why that should make you question the quality/validity/truthfulness of all the statements in the article.
Basically, Green Hills seems to be just another proprietary software vendor scratching for ways to try and derail a competitor in their market space. Nothing to see here. Move along now.
My Slashdot account is old enough to drink...
This is kind of a side remark that I haven't really thought on too much, but here goes. (I think I'm playing devil's advocate...)
(1) Who audits the open source software that they use? I certainly don't. I rarely even bother to look at the source. So in this respect, it doesn't matter (to me) if the software is closed source or open source since the code isn't looked at even if you (I) had the chance to.
(2) If you're not going to audit the code, will you trust the code developers to have done adequate auditing? Again, the folks who write open source software are, for the most part, as much a stranger as the folks working in some company (at least if you're me). Why should I trust "open source" strangers more than "closed source" strangers?
These points rarely seem to get brought up here. I can certainly see the answers to (2) giving the edge to open source, but what about (1)?.
"Now that foreign intelligence agencies and terrorists know that Linux is going to control our most advanced defense systems, they can use fake identities to contribute subversive software."
The whole story is so absolutely paranoid (The Russians are coming! Beware of the Yellow Peril!) and shows such a complete lack of understanding of the Linux Open Source process that it would make me worry if I were buying Green Hills' software: Do you want to buy something from somebody who is this divorced from reality and has this little understanding of how his competitor works?
For example you'll never see backdoors in commercial software. You can rest easy that they've done their job well and everything is nice and secure. That's why its better to stick with big commercial vendors like Cisco.
btw, why even give a story like this press? What a joke.
If you wanna get rich, you know that payback is a bitch
Some good reading about this topic can be found here.
"Everyday new code is added to Linux in Russia, China and elsewhere throughout the world. Everyday that code is incorporated into our command, control, communications and weapons systems. This must stop."
Does he get this pissed about Microsoft, IBM, Sun, HP and other companies that outsource core dev to those same countries?
The parent post is funny but in all fairness I think the general idea is that he's discussing the cost per line for a very large system. A single line in isolation is easy to debug. But you can't debug them in isolation, can you now? I think it should be fairly obvious the average cost to debug per line of code increases the more lines of codes you have in the system. Since the different lines of code interact, you know.
And this tendency is probably much more pronounced when rather than debugging, you are, for example, attempting to certify something as a failsafe system.
Linux is a fairly large and multifarous system. If his company sells a product that is designed and streamlined to be an RTOS embedded kernel, it more than likely achieves this in far, far fewer lines of code than Linux overall. While he is probably being unfair by counting in the total number of Linux number of lines of code things like desktop video card drivers, it is an altogether reasonable statement to suppose that the streamlined and smaller RTOS kernel this company sells is probably easier to debug and reason about than the Linux kernel, which is relatively larger, more complex, and has more complex design goals.
In the article, he states that anyone can contribute backdoors/trojans into the code because nobody is looking at the code. This is completely and utterly wrong. I'm pretty sure that to insert code into the kernel, you have to sign up to the mailing-list, and send in a diff. There, other kernel hackers can easily see the code, and if Linus accepts it, it goes into the tree. Even though anyone around the world can do this, the process is fairly strict.
Anyone want to place bets that Microsoft paid him to say that?
Founder of Mirror Moon - Tsukihime Game Trans
Actually, this is somewhat misleading. The top players listed are Microsoft (Windows CE), Wind River, Symbian, Palm. QNX, OSE, and Green Hills. But Microsoft, Symbian, and Palm are really selling into handheld devices, not hard real-time control. (The phone and PDA markets are much bigger than real time control, though.) Wind River's VRTX is the dominant player by a big margin, especially in low-end embedded control. QNX is next, and is usable on a broad range of platforms. Wind River is more of a specialist maker catering to the military Ada market.
Following these seven come LinuxWorks and MonteVista, who are moving up. These are the main Linux-based offerings.
Also confusing the issue is Windows XP Embedded, which is basically a Windows XP from which you can delete stuff you don't need. This sells more into point-of-sale applications than hard real time control.
Green Hill seems to be making some unsubstantiated claim that open source isn't held up to the same standards as closed source, and I find that rather funny. I think the real issue is, when Green Hill approaches the FAA or whatever, the FAA will do its own testing of the source. If Green Hill's code is breakable, Green Hill is the one responsible for fixing it. But, if what the FAA is reviewing is open source, it's possible the FAA can just fix the source themselves (and avoid having to pay an outside contractor). So, Green Hill, to avoid the scenario where the FAA might be displeased with Green Hill's RTOS and switch to open source, decides on its *own* to spend $500/$1,000 per line to audit their OS.
In the end, this means to me that Green Hill believes OSS has an unfair advantage. Personally, I think it's perfectly fair for people to offer free software. If Green Hill doesn't like it, tough. Or, they can just make their RTOS so good that the FAA or some other organization will be so impressed they won't bother going over some OSS and possible having to fix bugs or write documentation. Looks like the free market to me.
PS: SAYODF == Self-Analysising Your Own Dog Food; it's like a water bottling plant bitching about there being freshwater lakes because lakes don't have to do their own quality control
Eurohacker European paranoia, gun rights, and h
Who audits the open source software that they use? I certainly don't. I rarely even bother to look at the source. So in this respect, it doesn't matter (to me) if the software is closed source or open source since the code isn't looked at even if you (I) had the chance to.
Let's say you're in a major city. Let's say there's a small, narrow street. And let's say you walk down this street twice; once in the middle of the day, when the street is well-lit and crowded. And the second time in the middle of the night, when it is empty, abandoned and dark.
In which of these situations do you feel more safe?
I would probably guess the second. Why? In both the first and the second case, you have no way of knowing if there is someone who has a weapon and wishes you harm. But in the first case, this possibility is not something that worries you.
Open source is kind of like the well-lit, crowded street, and auditing is kind of like being able to tell if someone wishes you harm. First off, you don't *have* to be constantly watching your back; it's more than likely someone is looking at your back at any given moment, and can tell if someone is walking up behind you with a blunt instrument. Second off, the fact that *everyone knows people are watching* means it's less likely anyone will try anything, because they know they'll get caught and have messy problems with the law.
Note that this argument does not go so well unless the open source product is relatively well used. If you're the only user, well, you're not much better off.
---
Anyway, as far as (1) goes, I would imagine that while it may probably be a very important point insofar as you go, as far as the kinds of software discussed in this article go it's not so useful. You probably don't audit the open source software you use. But a propreitary embedded RTOS vendor certainly would, since their demands for security and reliability are much higher.
But wait, you say, wouldn't the need to audit the code at that level be an argument against open source, since it destroys the "free" nature? Well, here we run into the basic problem of the conflation of "free" software with "open source" software, or the conflation of "free" software with what RMS might call "software libre". These two concepts are often described by the same term. However, they are not the same thing!
A program being open source does not mean that it can have a trusted company of some kind behind it! A company such as IBM could be providing an open source program they internally wrote, or you could have a case such as that with MySQL (ok, maybe that's not a great example, but you know what I mean) where a community-developed program passes through a certain central trusted point (MySQL AB) which can be trusted to-- or demanded to-- perform the auditing for you. So this is not a problem with Open Source, this is just a problem with software you downloaded for free off the internet.
Irritable, left-wing and possibly humorous bumper stickers and t-shirts
I caught this story on OSNews yesterday and posted a rebuttal on my blog. This sort of thing probably doesn't carry a lot of weight with most of the defense types because the military is the very definition of mission critical, no pun intended. Peoples lives are at risk on a daily business in most jobs in the military these days. There is almost no price too high to pay for the freedom to design to specification that Linux provides.
Linux is certainly not ready to take over a lot of things yet, but it is good enough for many things that traditional defense contractors are involved with. I wouldn't trust it yet as an OS for our warships or other vehicles, but I would trust it for communication systems and things like that. For situations like that, a RTOS from a company like Green Hills may not provide enough benefit to justify the cost. Linux is free, their product isn't. They can try to get the military hooked for a while, but Linux will always be free and there are plenty of IT workers in the military who could work on existing RTOS Linux forks for military use.
Another thing that has to be kept in mind is that with the push for homeland security, the laissez faire attitude that has been prevalent toward security has to go. The miltiary wants transparency so it knows it's not getting something bugged all to hell by some Jihadi who wormed his way into Microsoft or Sun via the H1-B visa program. The Debian and Fedora teams are great for that very reason. Everything is open to public scrutiny, from the installer to every package so the military gets a chance to audit everything.
Free markets are great, but in this case the military has to perform a more core mission: defend the US from attack. If that means violating free market principles by pouring taxpayer dollars into a free OS for public use, then they should and most likely will do it eventually.
Click here or a puppy gets stomped!
Having worked as an systems administrator on DoD programs, I can tell you for a FACT that ANY software that goes on mission-critical systems is either developed in-house or very throughly scrutinized. They do code review, bug fixes and testing in a continuous cycle to get all the software bugs out. (This is one of the big reasons you hear about DoD projects going over time and/or over budget).
If COTS products are used, the DoD programmers will test the software for defects and ask the vendor to correct the defects they find. There have been cases where the DoD has signed NDAs to gain access to source code for COTS software to fix bugs that caused problems with the DoD software that the software company WOULDN'T fix. This has even been done to find backdoors, trojans, and other bad things that disgruntled employees of proprietary software vendors have put into that company's products.
OSS gives the DoD the power to make the changes they want to secure their systems the way they want. They WILL go through the code and look for backdoors, trojans, viri, etc. They may even set up their own repository and fork the kernel. Once the DoD has a trusted version of Linux, they'll use it in-house. I suspect that most DoD programs looking at Linux are probably testing NSA's version.
The DoD should be able to release some of the improvements they make back to the community, but don't expect them to release everything. The military still has it's secrets.
If "disco" means "I learn" in Latin, does "discothèque" mean "I learn technology"?
I used to work as a Defense contractor and I spent quite a lot of time going through the various processes. As a Linux developer, I can certainly say that Linux has not been developed to the same standards that the projects I've been involved on have.
For starters, in the DoD, every line of code is reviewed by hand by a team of reviewers (usually 4-5). There are records of all the defects found and verification that fixes were made. After the initial development cycle, there's a rigorious testing phase where all specifications are tested, senarios are ran, and stress tests are performed. Any defect from this testing is recorded, and the software doesn't ship until it's fixed. This usually ends up being a 2-4 year process of just doing bug fixes.
And for those that don't know the difference, Windows is *not* certified for tactical use. Having EAL4 is not the same as being certified for tactical use.
It's really a different type of software. It's not that Linux isn't good piece of software, it's just that it wasn't developed for this type of work. There's nothing wrong with that.
int func(int a);
func((b += 3, b));
How hard will it be for Chinese nationals to poison each of the major linux applications + the kernel?
And no one would dare do that at a closed source vendor. How on earth could it be possible for, say, a Chinese or Russian person to get a job a Microsoft or Cisco working on an operating system? It would never happen.
How right you are, our closed source software is completely safe!
Note to self: If ever pirate Russian gas pipeline control software, look for the "paybacksAreHell" subroutine.
When you buy a RTOS, you usually aren't getting compiled executable code. You usually get source code that you need to port to the hardware you are building.
Data sheets like this implies that Green Hills adheres to this common practice. So all the open source is more trustworthy than a black box arguments don't apply. Anyone who wishes to deploy a system based on Green Hills' RTOS can audit the code, it isn't hidden from them. Also, this PDF linked says:
Which to me implies that it has had a more thorough external audit than most open source packages.One final argument is that an RTOS is usually very small. Their Velocity RTOS can run in 3KB of RAM. When the OS is stripped down to something that small, a full audit seems like a much less daunting task.
This implies that he isn't arguing security through obscurity. He is arguing for the cathedral approach vs. the bazaar. Don't get me wrong, he still is spreading FUD. Its just a different FUD than you think. He is ignoring the role that Linus Torvalds and some of his trusted lieutenants like Alan Cox play in planning a direction, vetting ideas, and protecting the stability of the code base. Patches don't just come out of the blue from anonymous sources and applied without any examination, no matter what Dan O'Dowd may think.
The FAA approves software when it is written according the DO-178B specification. This specification states that software when developed must adhere to a development process.
This is defined within the D) 178b as software requirements, software specification, software design, source code configuration, and software test suites. If one changes one part then all levels affected must change as well.
Simply put a paper trail must exist for every change made in a system. It is stringent anal rententive form of development. It is costly since the amount of book keeping that must be done to incorporate changes.
This is the 'cost' that O'Dowd is refering to. In order to make a 'DO-178B' compliant version of Linux a group of developers/software house would have to:
1) Ensure that a comprehensive set of functional requirements is generated to match the desired platform.
2) Define a kernel that matches desired functional requirement. Any kernel portion that is not needed is defined out.
3) Specify the behaviour for each driver. Ensure the driver is fully specified. Work from the source and ensure that the behaviour of each execution path is documented.
4) Ensure that all changes to this build are reviewed and a paper-trail exists for all changes and changes are made for solid well documented reasons.
5) Use the documented behaviours to generate test cases that validate the documented behaviour.
It goes on and on...
There is nothing inherent within Linux that would prevent a DO-178B build to be created.
Only in the last 3 years has Green-hills has marketed a DO-178B compliant system. DO-178B as a standard has been around for I believe the last 10 years. Hmmm...
Research is what I doing when I don't know what I am doing - Werner von Braun
oh yeah I forgot. Diebold uses russians. Sequoia is foreign owned. Shouptronics founder served time for Election Machine Rigging, Seqoia execs indicted for bribing election officials, and GEMS VP of research served time for computer fraud.
In contrast OVC is a multi-national effort but its all open source so no one cares there be foreign nationals programming US machines.
Some drink at the fountain of knowledge. Others just gargle.
There is a real time variant of linux called RTLinux which is used across a large number of embedded and real time sensitive industries to great effect. Real time scheduling is exactly that, mostly about the scheduler. Linux is modular enough that it has had no less than four different schedulers in the last two major kernel releases, not counting the RT variant. If you can have as large a swath of the code peer reviewed as possible then I don't see where you can really go wrong, if you feel more testing is needed then put dollars into testing, not into creating new code that is then going to need just as much testing!
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
In fact, open-source software may have a slight advantage here, because it's less of a monoculture. Presumably Microsoft always uses their own Visual C++ compiler to build Windows, so if there were a trojan in the compiler that compromised the resulting Windows executables, it would be present in all copies of Windows that Microsoft distributed. But open source software is by its nature built on many different platforms using different compilers, so a compiler trojan would only affect a portion of the deployed copies of the open source software. And it is possible that a trojan introduced by one particular compiler would be found due to the executable it produces being different in some noticable way from the executable produced by a different compiler. For instance, strace might show the trojaned executable making extra system calls.
How does Mr. O'Dowd propose to assure us that his company's operating systems and compilers are more secure than Linux, xBSD, GCC, etc? Is he certain that none of his employess who have written code incorporated into his products have ever installed trojans? If so, how has he gained this certainty? Has he scrutinized every line of source code himself? Including those of the compilers that compiled the compilers, back all the way to the machine-code only origin of the system? Somehow I doubt it.
It is a matter of historical fact that far more trojan and back door exploits have been present in commercial, closed source software than in open source software. Just two days ago Cisco had to issue a security advisory regarding a back door found in their WLSE and HSE products. Would Mr. O'Dowd conclude that foreign agents and terrorists are responsible for that? Would he really have us believe that these shadowy figures can compromise open source software developed in the public eye more easily than they could subvert a commercial closed-source software package for which the source code and development process get no public scrutiny?
One is forced to conclude that Mr. O'Dowd feels his company's business model is threatened, and rather than change that model to reflect changes in the marketplace, he prefers to use "the sky is falling" proclamations in an attempt to scare customers into sticking with his products.
The whole story is so absolutely paranoid (The Russians are coming! Beware of the Yellow Peril!) and shows such a complete lack of understanding of the Linux Open Source process that it would make me worry if I were buying Green Hills' software: Do you want to buy something from somebody who is this divorced from reality and has this little understanding of how his competitor works?
There are a number of open source projects that have had their servers 0wn3d by crackers in the last year or two. In at least one or two cases the source code was tainted.
So, are you saying that, given an appropriate motivation (like linux being used to power Star Wars weapons) that the national security apparatus of a major power, with what are relatively unlimited resources and methods (buy equipment, time, expertise or information, hack, bribe, extort, tait software at source, infiltrate, murder (project lead?)) wouldn't be able to insert code when a pimple faced kid somehwere was able to do so?
Do you think that code would automatically be detected when so many bugs, bad practices, poor design, etc., etc., go undetected or fixed in open source software?
Consider this. Ken Thompson used to be able to login to just about any unix system in the world even if he didn't have an account. People checked and rechecked their systems. It didn't tend to help them. He later revealed his secret. Next, check out the Obfuscated C Contest. Some of the entries have additional functionality that isn't evident. One example is this one which implements 4 functions. I certainly wouldn't put it past somebody to be able to produce pretty standard looking C code that would pass the sniff test but which would, either by itself, or perhaps in combination with other code, implement an entirely different, second level of functionality which could be exploited as needed.
Given the potential stakes of defense work (losing a war, national survival, etc.) there is plenty of potential incentive for the finest minds a nation produces to tackle these problems, and potentially solve them. If you believe otherwise I think you are living in an open source dream world.
If you've got your real time system in rom inside a piece of equipment, or in thousands of pieces of equipment, you've got to be very careful with it.
Desktop system can be patched and upgraded, but ROMs have to be replaced or flashed. For example, you've got to bring the missle into the hanger/lab and hook up the reflashing unit or swap out the ROM chip. You've got to test the missle with the new chip. Out in the field, the soldiers have new ly upgraded missles (or tanks...) and would really like to know that it will work when they need it. You can field test a tank, but some missles are expensive, especially when all you want to do is prove you installed the right chip in the right way.
When a desktop or server software hiccups, the human user can work around it. Often this is not the case in communications and avionics.
Linux Advantages Don't Translate to Military Embedded Systems
Embedded systems are almost always memory-resident and have no disk for software storage. There are usually no user identities to manage, and the user interface is quite often absent or primitive.
Most of the advantages of Linux do not apply to an embedded, military situation: Licensing fees for software are usually a negligable part of a tank, missle or radar. Embedded RTOS systems are already quite reliable, and do not suffer from nearly as many buffer overruns, neither are they susceptible to hackers. Embedded military systems are almost never connected to the internet.
You could build a reliable, compact embedded software system from embedded Linux, but you'd want to write all your own drivers and you would have to port it to special hardware. This approximately the same amount of work that you would have to do if you were to use a proprietary RTOS.
The vast bulk of the problems users experience with proprietary OS's are 1) expensive to license, more expensive to license across many machines. 2) Security vulnerabilities resulting from using a system designed and built for stand-alone personal use in an internetworked environment. Neither of these problems matters much to embedded, military systems.
I18N == Intergalacticization
However, very very little Defense software requires DO-178B level ANYTHING certification.
This certification does not mean that there are not bugs in the software. Based on some limited experience I would say it does not even imply that the compiler and OS that Greenhills provides actuall even works together.
In the end, selecting an environment for any system has little to do with a closed v.s. open source issue and more to do with selecting the tool fits the job. However, the portion of the trade space that deals with open v.s. closed would certainly tip in favor of Open since I have almost no hope of reviewing or discovering holes in a closed system.
--- Liberty in our Lifetime
What developers can do to help is to modify a web server, a mail server, and a DNS server (the most attacked server side software) to run under NSA Secure Linux, partitioned into levels of integrity.
The idea is that just because somebody attacks, say, the mail server receive program, that doesn't get them power over the whole system. All it should do is let them run their attack code in a jail where it can't do anything except burn CPU cycles, and maybe generate phony incoming mail. Mail associated with the known IP address from which that copy of the receiver was launched, so the problem can be tracked.. When they disconnect, or some monitor program kills the corrupted component off, all the damage should be flushed.
You need to rework key server apps so that about 95% of the code is untrusted and jailed, while the 5% that has to do security-related functions is isolated, identified, and carefully examined.
That's what NSA Secure Linux is for.