Embedded RTOS Maker Raises Linux Security Issues

drquizas writes "Embedded RTOS provider Green Hills recently delivered an address where they raised the question of whether Linux can be considered secure enough to be used in defense applications. Much of the usual FUD is present in the remarks, although an interesting question is raised regarding what defense and other government contractors are required to do in testing code (in this case anyway): is the closed code here being held to a higher standard than its open-source equivalent, and does this change the 'security through obscurity' argument?"

Comments

Open source is much better than closed souce

[mindless4210]

"Open Source is actually more secure than closed source proprietary software because the oversight of technology content is broader and deeper. Instead of just one company monitoring its own contributions -- or potentially hiding security holes and exploits -- a worldwide community of interested parties actually oversees Linux to make it strong and secure. That's why the NSA -- the most security-conscious organization in the world -- chose to standardize on Linux, and even supplies its own version of secure Linux."

Can't put it much better than that. When you have the contribution of the entire open source development community, so much knowledge and experience comes to the table that it's difficult for any one group of programmers to compete.

Re:Open source is much better than closed souce

[beacher]

Yeah but he's spewing this crap.. "Everyday new code is added to Linux in Russia, China and elsewhere throughout the world. Everyday that code is incorporated into our command, control, communications and weapons systems. This must stop." ... Cmon he has a vested interest... His own company puts out it's own RTOS Go to that link. Now. Read the TOP of the middle column "Real-Time Operating Systems Must be Highly Reliable"
Microsoft Windows, MacOS, Unix, and Linux often crash, lock up, or go crazy. They indicate this condition by displaying a sad face, an exploding bomb, a red X, a blue screen of death, or by simply refusing to respond to mouse-clicks or keyboard input.

This is FUD and he does have a vested interest.

Re:Open source is much better than closed souce

[Total_Wimp]

Come on. These guys have a valid point. When you rely on high-quality closed source vendors like Cisco at least you guarentee you won't have back doors built into your system.

Oh. Wait. Nevermind.

Re:Open source is much better than closed souce

[Jeremiah Cornelius]

No.

You want custom, quality, made for Govt. spec code! The kind that is produced by either the low-bidder, or corporate crony!

Re:Open source is much better than closed souce

[cmacb]

"Frankly, even as a faithful Linux user, I still have to agree with him. Our missile defense systems should not be running the same software as my home PC whether it is a commercial or open-source product."

Funny... I feel just the opposite. Whether it's missile control, voting machines or accounting system 99% of what the operating systems components are doing is the same. I'd want that code tested millions of times if possible. Of course some of the code, unique to that application, can only be tested in place, but the less there is of that the better. For every person who would want to introduce a flaw into such software there are hundreds, more likely thousands, who would want to expose that flaw and fix it. It really doesn't matter if their reasons are patriotic or ego related.

It is closed systems after all that produce voting machines with huge bugs in them, and closed systems that crash vehicles into Mars due to metric to English conversion bugs. It is also closed systems that had laptop computers being used in Afghanistan being subverted by pop-up messages from ... well, nobody really knows. The notion that closed systems are superior from the security point of view simply doesn't hold up to any sort of statistical analysis. Heck, it doesn't even hold up to a back of the napkin analysis.

Re:Open source is much better than closed souce

[HungWeiLo]

I develop aircraft safety software, and the FAA's guidelines require that all code and tools must be certified at the same level of competency. Windows cannot be qualified as a valid development tool or environment, because it is closed source.

Re:Open source is much better than closed souce

[0x0000]

You're almost certainly wrong in your assertion about Windows. Perhaps you work for a vendor who likes things that way, because they'd rather not build on a Windows platform.

No, he's not, your assertion that "Windows could easily be qualified, by a team under NDA with a source license for Windows." is just plain laffable. And I say that as one who has been working in DO-178b certification efforts for the last 8 years or so. Believe me when I tell you that I have already laffed out loud at that sort of a statement more than once, both on and off the job.

Take a look at the cost per line of code for the typical level A cert and you will find that just the sheer volume of code in Windows makes any such effort impractical. Then there's the issue of the man-hours of effort per line of code...

Also, saftey-critical code is requirements driven. That is, the requirements are defined, then the code is written to implement the requirements. Using this approach, even if you could find something in Windows that fit your requirements, by the time you removed everything else, it wouldn't be Windows anymore. There would likely be no requirement for the user interface, for example.

That said, it is notable that most operating systems that exist today have similar issues. The bottom line is, none of them can be considered an RTOS, so if an RTOS is what you need, they won't do.

Smaller kernels have a better chance if you're trying to make one compliant, and the scheduler is the most useful piece, since it a) is what is needed (otherwise an OS wouldn't be needed, a monolithic embedded application would suffice), and b) it must be completely reliable, incapable of allowing race conditions, priority inversions, lockups, etc.

As for Green Hills, the fact is that Green Hills has the reputation in the industry for having produced the only (there may be another, but this is their rep I'm speaking of) actual OS that has qualified under DO-178b.

There have been several tries to produce others, but to date none has qualified, afaik. I have been, and continue to be, invovled in those efforts, so I am properly a competitor to Green Hills, since I am not a Green Hills employee, and have not worked with their OS.

I have, however, been involved in attempts to bring the Linux kernel into compliance with both DO-178b and ARINC 653 (the document desribing the partitioned RTOS model for "Loadable Software Airplane Parts," that is expected to be used in the future, starting with RTOSs like the one sold by Green Hills). While the Linux kernel port is considered doable, the cost of the effort was more than that client chose to bear. Again, the critical part of the port was to make the scheduler provably deterministic. Without that, the OS can't be considered.

The marketting hype (FUD) included in the piece is standard marketting hype, and is completely beside the point from an engineering standpoint, but plays well with suits, who typically don't understand what they're trying to build (software-wise), anyway.

Also, if you take a look at the experience the US Navy had trying to use Windows NT for engine control applications, you will get an idea of a) the relative simplicity of making it function as an RTOS, b) the degree of effort Microsoft was willing to expend towards the reliability of their software for a critical system, c) what a dumb idea it was, and d) why anyone with any understanding of the problem will not trust their life to software made by Microsoft, NDAs notwithstanding.

Finally, regarding the Open Source/Cathedral/Bazaar argument: When creating saftey critical requirements and the implementation of them, it is pretty much impossible to have too many eyes on the work. Practicality and economics are the constraints. That, and security, in the case of military or other sensitive applications.

Re:Open source is much better than closed souce

[0x0000]

You are correct, medical device manufacturers do in fact use Windows in some cases, and I find it plausible that they use OS/2, although I am not directly aware of an instance.

However, I would also point out that medical device manufacturers are not held to development process standards or testing requirements as stringent as those applied in the aerospace industry. I won't get into the possible reasons for that, but the medical industry is a lot more self-regulating.

In my experience, "critical" in medical industry software means somewhat else than it does in my field. This based on having interviewed for some of those types of positions. ...

Thanks for the 'Government Computer News: US Navy Ship stalled because of Windows NT' Anecdote. That one sure gets recycled a lot.

And for good reason. It was a clear case of Microsoft having bribed a congressional committee, and the first clue that many of us outside Microsoft got that El Senor Gates' ambitions reached beyond mere global domination of the software industry and great wealth. I think that aspect of it was not as widely discussed in the media, though.

What? What? What11!11?1

[zogger]

quote from this raty-os dude

"It costs us $500 to $1,000 a line to review our source code. It would cost billions of dollars to review Linux."

Say whut? It actually costs this? why? where can I sign up???? I'll sub my per-line auditing out, rake it in...

Naw, cmon, really? the government charges this, or he just pays this cost? Because..huh?

The NSA seems to think

[pair-a-noyd]

that Linux can be made pretty damn secure.
If they have faith in it....
http://www.nsa.gov/selinux/

give us a break

[Aneurysm9]

"Everyday that code is incorporated into our command, control, communications and weapons systems. This must stop."

I don't know how they do things at his shop, but if the DoD is pulling code from CVS into their production systems without auditing it, we deserve whatever we get as a result. That said, I highly doubt that's happening and it's more likely this blowhard is just trying to put a good scare into the technophobic jarheads who control procurement.

Higher Standards

[njcoder]

There are much higher standards for security in these situations.

I know Sun had to have a special version of Solaris just to meet these needs and Solaris was already considered very secure to begin with. I can't remember if MS released a secure NT for this reason as well or if they tried to and failed.

Talking about the openess of the linux code, there's another question I always wonder nobody asks. Sure Linux is open source and that's what helps it get better but I don't see the argument in terms of cost and security. Saying "you have the source you can see how secure it is" doesn't work for me. People buy an OS because it's cheaper to spend a few hundred or a few grand per PC than it is to hire the staff to build their own OS. Having to have the staff that can review, maintain and patch their own linux kernel alone isn't easy. It's something like 1.5 million lines of code right now. People want an OS that just works and is cheaper than building one themselves.

A Few Quick Notes about Green Hills

[pridkett]

First, this isn't the first time that Green Hills has come out complaining about Linux, you may remember a previous slashdot story where they claimed that the embedded linux tools market was a myth. Secondly, this article, like their previous one is through EETimes. If you've ever read EETimes you'll know why that should make you question the quality/validity/truthfulness of all the statements in the article.

Basically, Green Hills seems to be just another proprietary software vendor scratching for ways to try and derail a competitor in their market space. Nothing to see here. Move along now.

He's right you know

[bogie]

For example you'll never see backdoors in commercial software. You can rest easy that they've done their job well and everything is nice and secure. That's why its better to stick with big commercial vendors like Cisco.

btw, why even give a story like this press? What a joke.

Pot Kettle

[DAldredge]

"Everyday new code is added to Linux in Russia, China and elsewhere throughout the world. Everyday that code is incorporated into our command, control, communications and weapons systems. This must stop."

Does he get this pissed about Microsoft, IBM, Sun, HP and other companies that outsource core dev to those same countries?

In all fairness

[Anonymous Coward]

The parent post is funny but in all fairness I think the general idea is that he's discussing the cost per line for a very large system. A single line in isolation is easy to debug. But you can't debug them in isolation, can you now? I think it should be fairly obvious the average cost to debug per line of code increases the more lines of codes you have in the system. Since the different lines of code interact, you know.

And this tendency is probably much more pronounced when rather than debugging, you are, for example, attempting to certify something as a failsafe system.

Linux is a fairly large and multifarous system. If his company sells a product that is designed and streamlined to be an RTOS embedded kernel, it more than likely achieves this in far, far fewer lines of code than Linux overall. While he is probably being unfair by counting in the total number of Linux number of lines of code things like desktop video card drivers, it is an altogether reasonable statement to suppose that the streamlined and smaller RTOS kernel this company sells is probably easier to debug and reason about than the Linux kernel, which is relatively larger, more complex, and has more complex design goals.

Biggest load of sh**.

[chendo]

In the article, he states that anyone can contribute backdoors/trojans into the code because nobody is looking at the code. This is completely and utterly wrong. I'm pretty sure that to insert code into the kernel, you have to sign up to the mailing-list, and send in a diff. There, other kernel hackers can easily see the code, and if Linus accepts it, it goes into the tree. Even though anyone around the world can do this, the process is fairly strict.

Anyone want to place bets that Microsoft paid him to say that?

Look at it this way.

[mcc]

Who audits the open source software that they use? I certainly don't. I rarely even bother to look at the source. So in this respect, it doesn't matter (to me) if the software is closed source or open source since the code isn't looked at even if you (I) had the chance to.

Let's say you're in a major city. Let's say there's a small, narrow street. And let's say you walk down this street twice; once in the middle of the day, when the street is well-lit and crowded. And the second time in the middle of the night, when it is empty, abandoned and dark.

In which of these situations do you feel more safe?

I would probably guess the second. Why? In both the first and the second case, you have no way of knowing if there is someone who has a weapon and wishes you harm. But in the first case, this possibility is not something that worries you.

Open source is kind of like the well-lit, crowded street, and auditing is kind of like being able to tell if someone wishes you harm. First off, you don't *have* to be constantly watching your back; it's more than likely someone is looking at your back at any given moment, and can tell if someone is walking up behind you with a blunt instrument. Second off, the fact that *everyone knows people are watching* means it's less likely anyone will try anything, because they know they'll get caught and have messy problems with the law.

Note that this argument does not go so well unless the open source product is relatively well used. If you're the only user, well, you're not much better off.

---

Anyway, as far as (1) goes, I would imagine that while it may probably be a very important point insofar as you go, as far as the kinds of software discussed in this article go it's not so useful. You probably don't audit the open source software you use. But a propreitary embedded RTOS vendor certainly would, since their demands for security and reliability are much higher.

But wait, you say, wouldn't the need to audit the code at that level be an argument against open source, since it destroys the "free" nature? Well, here we run into the basic problem of the conflation of "free" software with "open source" software, or the conflation of "free" software with what RMS might call "software libre". These two concepts are often described by the same term. However, they are not the same thing!

A program being open source does not mean that it can have a trusted company of some kind behind it! A company such as IBM could be providing an open source program they internally wrote, or you could have a case such as that with MySQL (ok, maybe that's not a great example, but you know what I mean) where a community-developed program passes through a certain central trusted point (MySQL AB) which can be trusted to-- or demanded to-- perform the auditing for you. So this is not a problem with Open Source, this is just a problem with software you downloaded for free off the internet.

People like O'Dowd are running scared

[ShatteredDream]

I caught this story on OSNews yesterday and posted a rebuttal on my blog. This sort of thing probably doesn't carry a lot of weight with most of the defense types because the military is the very definition of mission critical, no pun intended. Peoples lives are at risk on a daily business in most jobs in the military these days. There is almost no price too high to pay for the freedom to design to specification that Linux provides.

Linux is certainly not ready to take over a lot of things yet, but it is good enough for many things that traditional defense contractors are involved with. I wouldn't trust it yet as an OS for our warships or other vehicles, but I would trust it for communication systems and things like that. For situations like that, a RTOS from a company like Green Hills may not provide enough benefit to justify the cost. Linux is free, their product isn't. They can try to get the military hooked for a while, but Linux will always be free and there are plenty of IT workers in the military who could work on existing RTOS Linux forks for military use.

Another thing that has to be kept in mind is that with the push for homeland security, the laissez faire attitude that has been prevalent toward security has to go. The miltiary wants transparency so it knows it's not getting something bugged all to hell by some Jihadi who wormed his way into Microsoft or Sun via the H1-B visa program. The Debian and Fedora teams are great for that very reason. Everything is open to public scrutiny, from the installer to every package so the military gets a chance to audit everything.

Free markets are great, but in this case the military has to perform a more core mission: defend the US from attack. If that means violating free market principles by pouring taxpayer dollars into a free OS for public use, then they should and most likely will do it eventually.

DoD, Linux and Security

[thewiz]

Having worked as an systems administrator on DoD programs, I can tell you for a FACT that ANY software that goes on mission-critical systems is either developed in-house or very throughly scrutinized. They do code review, bug fixes and testing in a continuous cycle to get all the software bugs out. (This is one of the big reasons you hear about DoD projects going over time and/or over budget).

If COTS products are used, the DoD programmers will test the software for defects and ask the vendor to correct the defects they find. There have been cases where the DoD has signed NDAs to gain access to source code for COTS software to fix bugs that caused problems with the DoD software that the software company WOULDN'T fix. This has even been done to find backdoors, trojans, and other bad things that disgruntled employees of proprietary software vendors have put into that company's products.

OSS gives the DoD the power to make the changes they want to secure their systems the way they want. They WILL go through the code and look for backdoors, trojans, viri, etc. They may even set up their own repository and fork the kernel. Once the DoD has a trusted version of Linux, they'll use it in-house. I suspect that most DoD programs looking at Linux are probably testing NSA's version.

The DoD should be able to release some of the improvements they make back to the community, but don't expect them to release everything. The military still has it's secrets.

How the Defense industry produces code

[lkaos]

I used to work as a Defense contractor and I spent quite a lot of time going through the various processes. As a Linux developer, I can certainly say that Linux has not been developed to the same standards that the projects I've been involved on have.

For starters, in the DoD, every line of code is reviewed by hand by a team of reviewers (usually 4-5). There are records of all the defects found and verification that fixes were made. After the initial development cycle, there's a rigorious testing phase where all specifications are tested, senarios are ran, and stress tests are performed. Any defect from this testing is recorded, and the software doesn't ship until it's fixed. This usually ends up being a 2-4 year process of just doing bug fixes.

And for those that don't know the difference, Windows is *not* certified for tactical use. Having EAL4 is not the same as being certified for tactical use.

It's really a different type of software. It's not that Linux isn't good piece of software, it's just that it wasn't developed for this type of work. There's nothing wrong with that.

not open vs. closed, cathedral vs. bazaar

[alangmead]

When you buy a RTOS, you usually aren't getting compiled executable code. You usually get source code that you need to port to the hardware you are building.

Data sheets like this implies that Green Hills adheres to this common practice. So all the open source is more trustworthy than a black box arguments don't apply. Anyone who wishes to deploy a system based on Green Hills' RTOS can audit the code, it isn't hidden from them. Also, this PDF linked says:

INTEGRITY178B has been audited and approved by the FAA for DO178B Level A use.

Which to me implies that it has had a more thorough external audit than most open source packages.

One final argument is that an RTOS is usually very small. Their Velocity RTOS can run in 3KB of RAM. When the OS is stripped down to something that small, a full audit seems like a much less daunting task.

This implies that he isn't arguing security through obscurity. He is arguing for the cathedral approach vs. the bazaar. Don't get me wrong, he still is spreading FUD. Its just a different FUD than you think. He is ignoring the role that Linus Torvalds and some of his trusted lieutenants like Alan Cox play in planning a direction, vetting ideas, and protecting the stability of the code base. Patches don't just come out of the blue from anonymous sources and applied without any examination, no matter what Dan O'Dowd may think.

DO-178B and Linux.

[BStorm]

The FAA approves software when it is written according the DO-178B specification. This specification states that software when developed must adhere to a development process.

This is defined within the D) 178b as software requirements, software specification, software design, source code configuration, and software test suites. If one changes one part then all levels affected must change as well.

Simply put a paper trail must exist for every change made in a system. It is stringent anal rententive form of development. It is costly since the amount of book keeping that must be done to incorporate changes.

This is the 'cost' that O'Dowd is refering to. In order to make a 'DO-178B' compliant version of Linux a group of developers/software house would have to:

1) Ensure that a comprehensive set of functional requirements is generated to match the desired platform.

2) Define a kernel that matches desired functional requirement. Any kernel portion that is not needed is defined out.

3) Specify the behaviour for each driver. Ensure the driver is fully specified. Work from the source and ensure that the behaviour of each execution path is documented.

4) Ensure that all changes to this build are reviewed and a paper-trail exists for all changes and changes are made for solid well documented reasons.

5) Use the documented behaviours to generate test cases that validate the documented behaviour.

It goes on and on...

There is nothing inherent within Linux that would prevent a DO-178B build to be created.

Only in the last 3 years has Green-hills has marketed a DO-178B compliant system. DO-178B as a standard has been around for I believe the last 10 years. Hmmm...

Embedded Software Different from Desktop/Server

[oldCoder]

The costs and benefits of reliability are different
If you've got your real time system in rom inside a piece of equipment, or in thousands of pieces of equipment, you've got to be very careful with it.

Desktop system can be patched and upgraded, but ROMs have to be replaced or flashed. For example, you've got to bring the missle into the hanger/lab and hook up the reflashing unit or swap out the ROM chip. You've got to test the missle with the new chip. Out in the field, the soldiers have new ly upgraded missles (or tanks...) and would really like to know that it will work when they need it. You can field test a tank, but some missles are expensive, especially when all you want to do is prove you installed the right chip in the right way.

When a desktop or server software hiccups, the human user can work around it. Often this is not the case in communications and avionics.

Linux Advantages Don't Translate to Military Embedded Systems
Embedded systems are almost always memory-resident and have no disk for software storage. There are usually no user identities to manage, and the user interface is quite often absent or primitive.

Most of the advantages of Linux do not apply to an embedded, military situation: Licensing fees for software are usually a negligable part of a tank, missle or radar. Embedded RTOS systems are already quite reliable, and do not suffer from nearly as many buffer overruns, neither are they susceptible to hackers. Embedded military systems are almost never connected to the internet.

You could build a reliable, compact embedded software system from embedded Linux, but you'd want to write all your own drivers and you would have to port it to special hardware. This approximately the same amount of work that you would have to do if you were to use a proprietary RTOS.

The vast bulk of the problems users experience with proprietary OS's are 1) expensive to license, more expensive to license across many machines. 2) Security vulnerabilities resulting from using a system designed and built for stand-alone personal use in an internetworked environment. Neither of these problems matters much to embedded, military systems.