Embedded RTOS Maker Raises Linux Security Issues

drquizas writes "Embedded RTOS provider Green Hills recently delivered an address where they raised the question of whether Linux can be considered secure enough to be used in defense applications. Much of the usual FUD is present in the remarks, although an interesting question is raised regarding what defense and other government contractors are required to do in testing code (in this case anyway): is the closed code here being held to a higher standard than its open-source equivalent, and does this change the 'security through obscurity' argument?"

Comments

Open source is much better than closed souce

[mindless4210]

"Open Source is actually more secure than closed source proprietary software because the oversight of technology content is broader and deeper. Instead of just one company monitoring its own contributions -- or potentially hiding security holes and exploits -- a worldwide community of interested parties actually oversees Linux to make it strong and secure. That's why the NSA -- the most security-conscious organization in the world -- chose to standardize on Linux, and even supplies its own version of secure Linux."

Can't put it much better than that. When you have the contribution of the entire open source development community, so much knowledge and experience comes to the table that it's difficult for any one group of programmers to compete.

Re:Open source is much better than closed souce

[0x0000]

You're almost certainly wrong in your assertion about Windows. Perhaps you work for a vendor who likes things that way, because they'd rather not build on a Windows platform.

No, he's not, your assertion that "Windows could easily be qualified, by a team under NDA with a source license for Windows." is just plain laffable. And I say that as one who has been working in DO-178b certification efforts for the last 8 years or so. Believe me when I tell you that I have already laffed out loud at that sort of a statement more than once, both on and off the job.

Take a look at the cost per line of code for the typical level A cert and you will find that just the sheer volume of code in Windows makes any such effort impractical. Then there's the issue of the man-hours of effort per line of code...

Also, saftey-critical code is requirements driven. That is, the requirements are defined, then the code is written to implement the requirements. Using this approach, even if you could find something in Windows that fit your requirements, by the time you removed everything else, it wouldn't be Windows anymore. There would likely be no requirement for the user interface, for example.

That said, it is notable that most operating systems that exist today have similar issues. The bottom line is, none of them can be considered an RTOS, so if an RTOS is what you need, they won't do.

Smaller kernels have a better chance if you're trying to make one compliant, and the scheduler is the most useful piece, since it a) is what is needed (otherwise an OS wouldn't be needed, a monolithic embedded application would suffice), and b) it must be completely reliable, incapable of allowing race conditions, priority inversions, lockups, etc.

As for Green Hills, the fact is that Green Hills has the reputation in the industry for having produced the only (there may be another, but this is their rep I'm speaking of) actual OS that has qualified under DO-178b.

There have been several tries to produce others, but to date none has qualified, afaik. I have been, and continue to be, invovled in those efforts, so I am properly a competitor to Green Hills, since I am not a Green Hills employee, and have not worked with their OS.

I have, however, been involved in attempts to bring the Linux kernel into compliance with both DO-178b and ARINC 653 (the document desribing the partitioned RTOS model for "Loadable Software Airplane Parts," that is expected to be used in the future, starting with RTOSs like the one sold by Green Hills). While the Linux kernel port is considered doable, the cost of the effort was more than that client chose to bear. Again, the critical part of the port was to make the scheduler provably deterministic. Without that, the OS can't be considered.

The marketting hype (FUD) included in the piece is standard marketting hype, and is completely beside the point from an engineering standpoint, but plays well with suits, who typically don't understand what they're trying to build (software-wise), anyway.

Also, if you take a look at the experience the US Navy had trying to use Windows NT for engine control applications, you will get an idea of a) the relative simplicity of making it function as an RTOS, b) the degree of effort Microsoft was willing to expend towards the reliability of their software for a critical system, c) what a dumb idea it was, and d) why anyone with any understanding of the problem will not trust their life to software made by Microsoft, NDAs notwithstanding.

Finally, regarding the Open Source/Cathedral/Bazaar argument: When creating saftey critical requirements and the implementation of them, it is pretty much impossible to have too many eyes on the work. Practicality and economics are the constraints. That, and security, in the case of military or other sensitive applications.

The NSA seems to think

[pair-a-noyd]

that Linux can be made pretty damn secure.
If they have faith in it....
http://www.nsa.gov/selinux/

A Few Quick Notes about Green Hills

[pridkett]

First, this isn't the first time that Green Hills has come out complaining about Linux, you may remember a previous slashdot story where they claimed that the embedded linux tools market was a myth. Secondly, this article, like their previous one is through EETimes. If you've ever read EETimes you'll know why that should make you question the quality/validity/truthfulness of all the statements in the article.

Basically, Green Hills seems to be just another proprietary software vendor scratching for ways to try and derail a competitor in their market space. Nothing to see here. Move along now.

Biggest load of sh**.

[chendo]

In the article, he states that anyone can contribute backdoors/trojans into the code because nobody is looking at the code. This is completely and utterly wrong. I'm pretty sure that to insert code into the kernel, you have to sign up to the mailing-list, and send in a diff. There, other kernel hackers can easily see the code, and if Linus accepts it, it goes into the tree. Even though anyone around the world can do this, the process is fairly strict.

Anyone want to place bets that Microsoft paid him to say that?

Look at it this way.

[mcc]

Who audits the open source software that they use? I certainly don't. I rarely even bother to look at the source. So in this respect, it doesn't matter (to me) if the software is closed source or open source since the code isn't looked at even if you (I) had the chance to.

Let's say you're in a major city. Let's say there's a small, narrow street. And let's say you walk down this street twice; once in the middle of the day, when the street is well-lit and crowded. And the second time in the middle of the night, when it is empty, abandoned and dark.

In which of these situations do you feel more safe?

I would probably guess the second. Why? In both the first and the second case, you have no way of knowing if there is someone who has a weapon and wishes you harm. But in the first case, this possibility is not something that worries you.

Open source is kind of like the well-lit, crowded street, and auditing is kind of like being able to tell if someone wishes you harm. First off, you don't *have* to be constantly watching your back; it's more than likely someone is looking at your back at any given moment, and can tell if someone is walking up behind you with a blunt instrument. Second off, the fact that *everyone knows people are watching* means it's less likely anyone will try anything, because they know they'll get caught and have messy problems with the law.

Note that this argument does not go so well unless the open source product is relatively well used. If you're the only user, well, you're not much better off.

---

Anyway, as far as (1) goes, I would imagine that while it may probably be a very important point insofar as you go, as far as the kinds of software discussed in this article go it's not so useful. You probably don't audit the open source software you use. But a propreitary embedded RTOS vendor certainly would, since their demands for security and reliability are much higher.

But wait, you say, wouldn't the need to audit the code at that level be an argument against open source, since it destroys the "free" nature? Well, here we run into the basic problem of the conflation of "free" software with "open source" software, or the conflation of "free" software with what RMS might call "software libre". These two concepts are often described by the same term. However, they are not the same thing!

A program being open source does not mean that it can have a trusted company of some kind behind it! A company such as IBM could be providing an open source program they internally wrote, or you could have a case such as that with MySQL (ok, maybe that's not a great example, but you know what I mean) where a community-developed program passes through a certain central trusted point (MySQL AB) which can be trusted to-- or demanded to-- perform the auditing for you. So this is not a problem with Open Source, this is just a problem with software you downloaded for free off the internet.

DoD, Linux and Security

[thewiz]

Having worked as an systems administrator on DoD programs, I can tell you for a FACT that ANY software that goes on mission-critical systems is either developed in-house or very throughly scrutinized. They do code review, bug fixes and testing in a continuous cycle to get all the software bugs out. (This is one of the big reasons you hear about DoD projects going over time and/or over budget).

If COTS products are used, the DoD programmers will test the software for defects and ask the vendor to correct the defects they find. There have been cases where the DoD has signed NDAs to gain access to source code for COTS software to fix bugs that caused problems with the DoD software that the software company WOULDN'T fix. This has even been done to find backdoors, trojans, and other bad things that disgruntled employees of proprietary software vendors have put into that company's products.

OSS gives the DoD the power to make the changes they want to secure their systems the way they want. They WILL go through the code and look for backdoors, trojans, viri, etc. They may even set up their own repository and fork the kernel. Once the DoD has a trusted version of Linux, they'll use it in-house. I suspect that most DoD programs looking at Linux are probably testing NSA's version.

The DoD should be able to release some of the improvements they make back to the community, but don't expect them to release everything. The military still has it's secrets.

How the Defense industry produces code

[lkaos]

I used to work as a Defense contractor and I spent quite a lot of time going through the various processes. As a Linux developer, I can certainly say that Linux has not been developed to the same standards that the projects I've been involved on have.

For starters, in the DoD, every line of code is reviewed by hand by a team of reviewers (usually 4-5). There are records of all the defects found and verification that fixes were made. After the initial development cycle, there's a rigorious testing phase where all specifications are tested, senarios are ran, and stress tests are performed. Any defect from this testing is recorded, and the software doesn't ship until it's fixed. This usually ends up being a 2-4 year process of just doing bug fixes.

And for those that don't know the difference, Windows is *not* certified for tactical use. Having EAL4 is not the same as being certified for tactical use.

It's really a different type of software. It's not that Linux isn't good piece of software, it's just that it wasn't developed for this type of work. There's nothing wrong with that.

Embedded Software Different from Desktop/Server

[oldCoder]

The costs and benefits of reliability are different
If you've got your real time system in rom inside a piece of equipment, or in thousands of pieces of equipment, you've got to be very careful with it.

Desktop system can be patched and upgraded, but ROMs have to be replaced or flashed. For example, you've got to bring the missle into the hanger/lab and hook up the reflashing unit or swap out the ROM chip. You've got to test the missle with the new chip. Out in the field, the soldiers have new ly upgraded missles (or tanks...) and would really like to know that it will work when they need it. You can field test a tank, but some missles are expensive, especially when all you want to do is prove you installed the right chip in the right way.

When a desktop or server software hiccups, the human user can work around it. Often this is not the case in communications and avionics.

Linux Advantages Don't Translate to Military Embedded Systems
Embedded systems are almost always memory-resident and have no disk for software storage. There are usually no user identities to manage, and the user interface is quite often absent or primitive.

Most of the advantages of Linux do not apply to an embedded, military situation: Licensing fees for software are usually a negligable part of a tank, missle or radar. Embedded RTOS systems are already quite reliable, and do not suffer from nearly as many buffer overruns, neither are they susceptible to hackers. Embedded military systems are almost never connected to the internet.

You could build a reliable, compact embedded software system from embedded Linux, but you'd want to write all your own drivers and you would have to port it to special hardware. This approximately the same amount of work that you would have to do if you were to use a proprietary RTOS.

The vast bulk of the problems users experience with proprietary OS's are 1) expensive to license, more expensive to license across many machines. 2) Security vulnerabilities resulting from using a system designed and built for stand-alone personal use in an internetworked environment. Neither of these problems matters much to embedded, military systems.