Embedded RTOS Maker Raises Linux Security Issues

drquizas writes "Embedded RTOS provider Green Hills recently delivered an address where they raised the question of whether Linux can be considered secure enough to be used in defense applications. Much of the usual FUD is present in the remarks, although an interesting question is raised regarding what defense and other government contractors are required to do in testing code (in this case anyway): is the closed code here being held to a higher standard than its open-source equivalent, and does this change the 'security through obscurity' argument?"

Comments

Open source is much better than closed souce

[mindless4210]

"Open Source is actually more secure than closed source proprietary software because the oversight of technology content is broader and deeper. Instead of just one company monitoring its own contributions -- or potentially hiding security holes and exploits -- a worldwide community of interested parties actually oversees Linux to make it strong and secure. That's why the NSA -- the most security-conscious organization in the world -- chose to standardize on Linux, and even supplies its own version of secure Linux."

Can't put it much better than that. When you have the contribution of the entire open source development community, so much knowledge and experience comes to the table that it's difficult for any one group of programmers to compete.

Re:Open source is much better than closed souce

[Total_Wimp]

Come on. These guys have a valid point. When you rely on high-quality closed source vendors like Cisco at least you guarentee you won't have back doors built into your system.

Oh. Wait. Nevermind.

Re:Open source is much better than closed souce

[cmacb]

"Frankly, even as a faithful Linux user, I still have to agree with him. Our missile defense systems should not be running the same software as my home PC whether it is a commercial or open-source product."

Funny... I feel just the opposite. Whether it's missile control, voting machines or accounting system 99% of what the operating systems components are doing is the same. I'd want that code tested millions of times if possible. Of course some of the code, unique to that application, can only be tested in place, but the less there is of that the better. For every person who would want to introduce a flaw into such software there are hundreds, more likely thousands, who would want to expose that flaw and fix it. It really doesn't matter if their reasons are patriotic or ego related.

It is closed systems after all that produce voting machines with huge bugs in them, and closed systems that crash vehicles into Mars due to metric to English conversion bugs. It is also closed systems that had laptop computers being used in Afghanistan being subverted by pop-up messages from ... well, nobody really knows. The notion that closed systems are superior from the security point of view simply doesn't hold up to any sort of statistical analysis. Heck, it doesn't even hold up to a back of the napkin analysis.

Re:Open source is much better than closed souce

[HungWeiLo]

I develop aircraft safety software, and the FAA's guidelines require that all code and tools must be certified at the same level of competency. Windows cannot be qualified as a valid development tool or environment, because it is closed source.

Re:Open source is much better than closed souce

[0x0000]

You're almost certainly wrong in your assertion about Windows. Perhaps you work for a vendor who likes things that way, because they'd rather not build on a Windows platform.

No, he's not, your assertion that "Windows could easily be qualified, by a team under NDA with a source license for Windows." is just plain laffable. And I say that as one who has been working in DO-178b certification efforts for the last 8 years or so. Believe me when I tell you that I have already laffed out loud at that sort of a statement more than once, both on and off the job.

Take a look at the cost per line of code for the typical level A cert and you will find that just the sheer volume of code in Windows makes any such effort impractical. Then there's the issue of the man-hours of effort per line of code...

Also, saftey-critical code is requirements driven. That is, the requirements are defined, then the code is written to implement the requirements. Using this approach, even if you could find something in Windows that fit your requirements, by the time you removed everything else, it wouldn't be Windows anymore. There would likely be no requirement for the user interface, for example.

That said, it is notable that most operating systems that exist today have similar issues. The bottom line is, none of them can be considered an RTOS, so if an RTOS is what you need, they won't do.

Smaller kernels have a better chance if you're trying to make one compliant, and the scheduler is the most useful piece, since it a) is what is needed (otherwise an OS wouldn't be needed, a monolithic embedded application would suffice), and b) it must be completely reliable, incapable of allowing race conditions, priority inversions, lockups, etc.

As for Green Hills, the fact is that Green Hills has the reputation in the industry for having produced the only (there may be another, but this is their rep I'm speaking of) actual OS that has qualified under DO-178b.

There have been several tries to produce others, but to date none has qualified, afaik. I have been, and continue to be, invovled in those efforts, so I am properly a competitor to Green Hills, since I am not a Green Hills employee, and have not worked with their OS.

I have, however, been involved in attempts to bring the Linux kernel into compliance with both DO-178b and ARINC 653 (the document desribing the partitioned RTOS model for "Loadable Software Airplane Parts," that is expected to be used in the future, starting with RTOSs like the one sold by Green Hills). While the Linux kernel port is considered doable, the cost of the effort was more than that client chose to bear. Again, the critical part of the port was to make the scheduler provably deterministic. Without that, the OS can't be considered.

The marketting hype (FUD) included in the piece is standard marketting hype, and is completely beside the point from an engineering standpoint, but plays well with suits, who typically don't understand what they're trying to build (software-wise), anyway.

Also, if you take a look at the experience the US Navy had trying to use Windows NT for engine control applications, you will get an idea of a) the relative simplicity of making it function as an RTOS, b) the degree of effort Microsoft was willing to expend towards the reliability of their software for a critical system, c) what a dumb idea it was, and d) why anyone with any understanding of the problem will not trust their life to software made by Microsoft, NDAs notwithstanding.

Finally, regarding the Open Source/Cathedral/Bazaar argument: When creating saftey critical requirements and the implementation of them, it is pretty much impossible to have too many eyes on the work. Practicality and economics are the constraints. That, and security, in the case of military or other sensitive applications.

What? What? What11!11?1

[zogger]

quote from this raty-os dude

"It costs us $500 to $1,000 a line to review our source code. It would cost billions of dollars to review Linux."

Say whut? It actually costs this? why? where can I sign up???? I'll sub my per-line auditing out, rake it in...

Naw, cmon, really? the government charges this, or he just pays this cost? Because..huh?

The NSA seems to think

[pair-a-noyd]

that Linux can be made pretty damn secure.
If they have faith in it....
http://www.nsa.gov/selinux/

He's right you know

[bogie]

For example you'll never see backdoors in commercial software. You can rest easy that they've done their job well and everything is nice and secure. That's why its better to stick with big commercial vendors like Cisco.

btw, why even give a story like this press? What a joke.

Pot Kettle

[DAldredge]

"Everyday new code is added to Linux in Russia, China and elsewhere throughout the world. Everyday that code is incorporated into our command, control, communications and weapons systems. This must stop."

Does he get this pissed about Microsoft, IBM, Sun, HP and other companies that outsource core dev to those same countries?

In all fairness

[Anonymous Coward]

The parent post is funny but in all fairness I think the general idea is that he's discussing the cost per line for a very large system. A single line in isolation is easy to debug. But you can't debug them in isolation, can you now? I think it should be fairly obvious the average cost to debug per line of code increases the more lines of codes you have in the system. Since the different lines of code interact, you know.

And this tendency is probably much more pronounced when rather than debugging, you are, for example, attempting to certify something as a failsafe system.

Linux is a fairly large and multifarous system. If his company sells a product that is designed and streamlined to be an RTOS embedded kernel, it more than likely achieves this in far, far fewer lines of code than Linux overall. While he is probably being unfair by counting in the total number of Linux number of lines of code things like desktop video card drivers, it is an altogether reasonable statement to suppose that the streamlined and smaller RTOS kernel this company sells is probably easier to debug and reason about than the Linux kernel, which is relatively larger, more complex, and has more complex design goals.

Biggest load of sh**.

[chendo]

In the article, he states that anyone can contribute backdoors/trojans into the code because nobody is looking at the code. This is completely and utterly wrong. I'm pretty sure that to insert code into the kernel, you have to sign up to the mailing-list, and send in a diff. There, other kernel hackers can easily see the code, and if Linus accepts it, it goes into the tree. Even though anyone around the world can do this, the process is fairly strict.

Anyone want to place bets that Microsoft paid him to say that?

How the Defense industry produces code

[lkaos]

I used to work as a Defense contractor and I spent quite a lot of time going through the various processes. As a Linux developer, I can certainly say that Linux has not been developed to the same standards that the projects I've been involved on have.

For starters, in the DoD, every line of code is reviewed by hand by a team of reviewers (usually 4-5). There are records of all the defects found and verification that fixes were made. After the initial development cycle, there's a rigorious testing phase where all specifications are tested, senarios are ran, and stress tests are performed. Any defect from this testing is recorded, and the software doesn't ship until it's fixed. This usually ends up being a 2-4 year process of just doing bug fixes.

And for those that don't know the difference, Windows is *not* certified for tactical use. Having EAL4 is not the same as being certified for tactical use.

It's really a different type of software. It's not that Linux isn't good piece of software, it's just that it wasn't developed for this type of work. There's nothing wrong with that.

not open vs. closed, cathedral vs. bazaar

[alangmead]

When you buy a RTOS, you usually aren't getting compiled executable code. You usually get source code that you need to port to the hardware you are building.

Data sheets like this implies that Green Hills adheres to this common practice. So all the open source is more trustworthy than a black box arguments don't apply. Anyone who wishes to deploy a system based on Green Hills' RTOS can audit the code, it isn't hidden from them. Also, this PDF linked says:

INTEGRITY178B has been audited and approved by the FAA for DO178B Level A use.

Which to me implies that it has had a more thorough external audit than most open source packages.

One final argument is that an RTOS is usually very small. Their Velocity RTOS can run in 3KB of RAM. When the OS is stripped down to something that small, a full audit seems like a much less daunting task.

This implies that he isn't arguing security through obscurity. He is arguing for the cathedral approach vs. the bazaar. Don't get me wrong, he still is spreading FUD. Its just a different FUD than you think. He is ignoring the role that Linus Torvalds and some of his trusted lieutenants like Alan Cox play in planning a direction, vetting ideas, and protecting the stability of the code base. Patches don't just come out of the blue from anonymous sources and applied without any examination, no matter what Dan O'Dowd may think.

DO-178B and Linux.

[BStorm]

The FAA approves software when it is written according the DO-178B specification. This specification states that software when developed must adhere to a development process.

This is defined within the D) 178b as software requirements, software specification, software design, source code configuration, and software test suites. If one changes one part then all levels affected must change as well.

Simply put a paper trail must exist for every change made in a system. It is stringent anal rententive form of development. It is costly since the amount of book keeping that must be done to incorporate changes.

This is the 'cost' that O'Dowd is refering to. In order to make a 'DO-178B' compliant version of Linux a group of developers/software house would have to:

1) Ensure that a comprehensive set of functional requirements is generated to match the desired platform.

2) Define a kernel that matches desired functional requirement. Any kernel portion that is not needed is defined out.

3) Specify the behaviour for each driver. Ensure the driver is fully specified. Work from the source and ensure that the behaviour of each execution path is documented.

4) Ensure that all changes to this build are reviewed and a paper-trail exists for all changes and changes are made for solid well documented reasons.

5) Use the documented behaviours to generate test cases that validate the documented behaviour.

It goes on and on...

There is nothing inherent within Linux that would prevent a DO-178B build to be created.

Only in the last 3 years has Green-hills has marketed a DO-178B compliant system. DO-178B as a standard has been around for I believe the last 10 years. Hmmm...

Embedded Software Different from Desktop/Server

[oldCoder]

The costs and benefits of reliability are different
If you've got your real time system in rom inside a piece of equipment, or in thousands of pieces of equipment, you've got to be very careful with it.

Desktop system can be patched and upgraded, but ROMs have to be replaced or flashed. For example, you've got to bring the missle into the hanger/lab and hook up the reflashing unit or swap out the ROM chip. You've got to test the missle with the new chip. Out in the field, the soldiers have new ly upgraded missles (or tanks...) and would really like to know that it will work when they need it. You can field test a tank, but some missles are expensive, especially when all you want to do is prove you installed the right chip in the right way.

When a desktop or server software hiccups, the human user can work around it. Often this is not the case in communications and avionics.

Linux Advantages Don't Translate to Military Embedded Systems
Embedded systems are almost always memory-resident and have no disk for software storage. There are usually no user identities to manage, and the user interface is quite often absent or primitive.

Most of the advantages of Linux do not apply to an embedded, military situation: Licensing fees for software are usually a negligable part of a tank, missle or radar. Embedded RTOS systems are already quite reliable, and do not suffer from nearly as many buffer overruns, neither are they susceptible to hackers. Embedded military systems are almost never connected to the internet.

You could build a reliable, compact embedded software system from embedded Linux, but you'd want to write all your own drivers and you would have to port it to special hardware. This approximately the same amount of work that you would have to do if you were to use a proprietary RTOS.

The vast bulk of the problems users experience with proprietary OS's are 1) expensive to license, more expensive to license across many machines. 2) Security vulnerabilities resulting from using a system designed and built for stand-alone personal use in an internetworked environment. Neither of these problems matters much to embedded, military systems.