Theo de Raadt Responds

Here you go: straight answers to your questions from Theo de Raadt of OpenBSD. Read, enjoy, and comment away. (Theo might even check in with a comment or two of his own if he has time -- although he's been awfully busy lately, so please don't hold it against him if he can't, okay?)

A book on code auditing?
by LizardKing

Would you and/or other members of the OpenBSD coders consider writing a book on secure, bug-free coding and auditing? Most programming books feature sample code that is written for pedagogical purposes. Quite often this runs contrary to how secure code should be written, leaving a gap in many a programmers knowledge. A book on audinting and how to avoid security pitfalls when coding would also make your life easier - less code to audit for OpenBSD, and more time top concentrate on nifty new features!!!

Theo:

There is perhaps a split between the two issues you bring up. On the one side is secure coding, as in code written to be secure by the original author(s). On the other side, auditing, which is where an outsider (or an insider) later on goes and tries to clean up the mess which remains. And there is always a mess. Perhaps part of the problem is that a huge gap lies between these two. In the end though, I think that a book on such a topic would probably have to repeat the same thing every second paragraph, throughout the book: Understand the interfaces which you are coding to! Understand the interfaces which you are coding to! Most of the security (or simply bug) issues we audited out of our source tree are just that. The programmer in question was a careless slob, not paying attention to the interface he was using. The repeated nature of the same classes of bugs throughout the source tree, also showed us that most programmers learn to code by (bad) examples. A solid systems's approach should not be based on "but it works". Yet, time and time again, we see that for most people this is the case. They don't care about good software, only about "good enough" software. So the programmers can continue to make such mistakes. Thus, I do not feel all that excited about writing a book which would simply teach people that the devil is in the details. If they haven't figured it out by now, perhaps they should consider another occupation (one where they will cause less damage).

Making the rest secure
by squiggleslash

OpenBSD has a well deserved reputation for security "out of the box" and for the fact the inbuilt tools are as secure as they're ever likely to be. However, the Ports system is, perhaps, an example of where the secure approach currently has limitations - an installation of OpenBSD running popular third-party systems like INN can only be so secure because the auditing of INN, and other such software, is outside the scope of the BSD audit.

My question is, has the OpenBSD team ever proposed looking into how to create a 'secured ports' tree, or some other similar system, that would ensure that many of the applications people specifically want secure platforms like OpenBSD to run could be as trusted as the platforms themselves?

Theo:

We have our hands already pretty full, just researching new ideas in our main source tree, which is roughly 300MB in size. We also lightly involved ourselves in working with the XFree86 people a while back for some components there. Auditing the components outside of this becomes rather unwieldly. The difficulty lies not only in the volume of such code, but also in other issues. Sometimes communication with the maintainers of these other packages is difficult, for various reasons. Sometimes they are immediately turned off because we don't use the word Linux. Some of these portable software packages are by their nature never really going to approach the quality of regular system software, because they are so bulky.

But most importantly, please remember that we are also human beings, trying to live our lives in a pleasant way, and don't ussually get all that excited about suddenly burning 800 hours in some disgusting piece of badly programmer trash which we can just avoid running. I suppose that quite often some of our auditors look at a piece of code and go "oh, wow, this is really bad", and then just avoid using it. I know that doesn't make you guys feel better, but what can we say...

OpenBSD, security, et al.
by jd

With the release of SGI's B1 code, and the attempts by many U*ixen to secure their contents via capabilities, ACL's, etc, ad nausium, how is OpenBSD approaching the issue of resource control?

On a side note, is OpenBSD likely to ever head in the direction of being a distributed kernel? And, if so, how would security and resource management be maintained? (It's hard enough on a central kernel system.)

Theo:

On the first question, I think there is great confusion in the land of Orange Book. Many people think that is about security. It is not. Largely, those standards are about accountability in the face of threat. Which really isn't about making systems secure. It's about knowing when your system's security breaks down. Not quite the same thing. Please count the commercially deployed C, B, or even A systems which are actually being used by real people for real work, before foaming at the mouth about it all being "so great". On the other hand, I think we wil see if some parts of that picture actually start to show up in real systems, over time. By the way, I am surprised to see you list ACLs, which don't really have anything to do with B1 systems.

As to the second issue, I have no idea what a distributed kernel is, nor do I see how anything like that would improve security or quality of a system.

Forks and cooperation
by PapaZit

A lot of people know that OpenBSD forked from NetBSD, and there's still some animosity between the two groups. Personally, I think that the competition has helped both groups (NetBSD now ships with far fewer open services, for example).

Egos are delicate things, but do you see any chance for greater cooperation in the future, or do you see more forking and division as inevitable?

Theo:

Considering that NetBSD has maintained a black-hole route to the OpenBSD project networks for roughly four years, I don't see how any cooperation at higher levels are possible. However, there are developers who work on multiple projects. Some of them used to complain about having troubles from various groups. Nowadays, I think they've got it easier. Politics do not dictate developer relationships these days.

In Linux land, it appears that projects fork for financial reasons. In BSD land, it appears that the forks that have happened were purely political reasons. I don't know what will happen. It's been 5 years since the last fork in the BSD camp. Why are you guys so fork paranoid? Do you want everyone to vote for the same political party, too?

Kernel design
by laertes

I have only been using OpenBSD for a short while now, so forgive me if this question is based upon some incorrect assumtions.

OpenBSD's kernel design seems to be of the monolithic species. OpenVMS (no relation) and NT are two prominent operating systems that use a microkernel archetecture. The microkernel design seems to me to be fundamentally more secure, since there is less priveledged code. Further, if one of the servers is compromised, the damage is minimezed.

My question is this: Is the OpenBSD design fundamentally secure, or is it only a very well done implementation of a basically flawed design?

Theo:

I don't think it makes any difference, whatsoever. I think your computer science teachers are still teaching you from books written in the 80's, when the word "micro-kernel" was associated with a future utopia. We do not think that NT is a microkernel, and are you really so sure that OpenVMS is? A microkernel is not a kernel that does things through loadable modules. As well, I don't think it makes any difference, as long as a system does what it is supposed to do.

Where Did You Learn Your Code Audit Discipline?
by EXTomar

Did the drive to audit code come from the need or the design of BSD? Or was it initially a whim? More imporantly, where did you learn it from? Is their some "mentor" you looked too for ridge design? I have to admire your team's daunting code reviewing...I wonder if I'll ever have that kind of meticulous coding nature.

Theo:

The auditing process developed out of a desire to improve the quality of our operating system. Once we started on it, it becames fascinating, fun, and very nearly fanatical. About ten people worked together on it, basically teaching ourselves as things went along. We searched for basic source-code programmer mistakes and sloppiness, rather than "holes" or "bugs". We just kept recursing through the source tree everytime we found a sloppiness. Everytime we found a mistake a programmer made (such as using mktemp(3) in such a way that a filesystem race occured), we would go throughout the source tree and fix ALL of them. Then when we fix that one, we would find some other basic mistake, and then fix ALL of them. Yes, it's a lot of work. But it has a serious payback. Can you imagine if a Boeing engineer didn't fix ALL of the occurances of a wiring flaw? Why not at least try to engineer software in the same way?

Firewall/NAT box
by yamla

Linux has FreeSco, a product that fits on a 3.5 inch floppy disk and acts as a router and NAT (Network Address Translation). I always thought something like this would be ideal for OpenBSD. After all, I would rather trust OpenBSD than Linux for this.

Are there any plans to produce something like this? Something with a very simple user interface that is quick and easy to get set up? I'd love to play with OpenBSD and do it by hand but I simply do not have the time.

Theo:

I must say that I am not a fan of these floppy-based routers. Essentially, you are taking one of the most unreliable pieces of storage known to man, and trying to build security infrastructure on it. That's madness. Just buy a small disk. Perhaps somethings based on a CD plus some other (non-floppy) persistant storage might be sane. But please. Not floppies. Are you mad?

Code-auditing
by AT

Any advise for code auditers? Can you share any tips or techniques you have found useful in uncovering bugs? What do you first look for in a fresh piece of code? What about a mature piece of code?

Theo:

I suppose the biggest tip would be to become a better programmer. In particular, study what functions that programs are calling, and ensure that the calling code is following the rules of those functions 100%. How many of you understand the complete & correct semantics of every function in libc, or even just the libc functions being called by the program you are looking? (I mean, we went through our entire source tree, and about half the strncat() and strncpy() calls were subtly wrong, even if it only meant they copied a character extra and then zero'd it out -- it is still sloppy).

When you know exactly what the APIs are, you'll spot the bugs very easily. In my mind, it is the same as any other job that requires diligence. Be careful. Humans learn from examples, and yet, in this software programming environment, the tremendous complexity breeds non-obvious mistakes, which we carry along with us, and copy into new chunks of code. We've even found in manual pages where functions were mis-described, and when we found those, lots of programmers had followed the instructions incorrectly...

Dual Processor Support
by dragonfly_blue

Although there has been some indication that people are interested in running OpenBSD on machines with dual or quad processors, it appears that there are not enough resources and volunteers available to make this a reality. Although I use OpenBSD for my web server, I am by no means an expert, at this, but I'm curious nonetheless.

From what I've heard, multiprocessing support is going to be a very tricky thing to implement, because it gives rise to so many possible exploits, particularly with regards to race conditions. I also understand that it would take a remarkable amount of effort and time to rewrite much of the code base for SMP without compromising the OS's integrity.

With that in mind, what kind of resources would you need before you could seriously consider attempting dual or quad processor support? And, if you were given unlimited access to those resources, how long would it take before a -stable release would be ready? I would really like to see this feature get implemented, although I know that at this point your developer team is busy enough as it is.

Theo:

At this time, we are not working on SMP. It's a lot of work, and not considered the most interesting thing to our developers. Sorry.

Time warp
by rho

Thanks for your work, Theo. I use OBSD every day as a workstation and as a firewall, and the Cop-chasing-script-kiddie t-shirt is the best.

If you could time warp back to the beginning of OpenBSD's development (ignoring the scism that brought you to that point), what would you do differently? Would you have chosen a more commercial focus? Pushed SMP development earlier? Run around in circles waving your hands in the air?

On another note, what's your feeling about commercial use of OpenBSD? i.e., do you support it, tolerate it, or what? (better example, I make a set-top box running OpenBSD, and I need the OS to do "X". If I called you and said, "Theo, I need OpenBSD to support 'X'", would I be told to piss up a rope, write it myself, or would the OpenBSD team do it for a price?)

Theo:

The licence on our code is pretty clear. We want vendors to use our code. We want commercial operating systems to ship with OpenSSH. Not shipping with an SSH varient causes great grief, and it is time that ends.

Same goes for OpenBSD. We would prefer if companies building commercial network appliances used OpenBSD, rather than writing their own operating systems. Typically, these companies are very comfortable with solving the problems within their application space. Yet, there is a history of these companies writing their own cruddy operating systems, and at the same time writing worse applications.

It would be better if routers, firewalls, telephone switches, fileservers, and whatever else used reliable components, designed by people who care.

So go ahead, use any parts of OpenBSD as parts of commercial systems.

Full Disclosure And Version Numbering
by Effugas

First of all, I want to thank you for the hard work you've done building OpenBSD. It truly is a wonderful package.

Much of the security in OpenBSD lies under the hood in the work you've done cleansing the source of unsafe library calls. While this work is appreciated, I've become more and more concerned lately about the fact that these changes are not necessarily documented and certainly not reflected in the version number of an application or utility.

Version numbers reflect a snapshot in the life of a codebase. They're used to reference unsafe editions or particularly stable builds. Major number reflect code branches, but minor numbers reflect specific states of the code -- such is the expectation of a user or an administrator when a version number is detected. Without granularity of versioning, I have no reason to trust or distrust a given application by its number; I must personally audit its source -- and end up giving it a number of my own.

You and your team are code auditing masters. Rather than pollute the namespace by making indistinguishable your securely built modified code and the original(and, by extension, your secure code and numerous unnamed distributions' "just get it to compile" modifications), wouldn't it be appropriate for OpenBSD to apply a name extension to any package which it has modified, and in the interests of full disclosure, to provide a reasonable CHANGELOG of the fixes contained therein?

Theo:

Two numbers exist for every component of OpenBSD. One number is the release that the piece came in, ie. 2.8.

The other number exists in each source file that was built. And that number is also in each binary that was built from those files. You can use the what(1) command to determine the revisions of source files which make up each binary.

As to the "original" you talk about, there is no original. OpenBSD uses it's own components. I don't know what packages you are talking about. cat is cat. ftpd is ftpd. tar is tar. It's the one that came with a certain release. In the systems approach, the version numbering that other groups do is sometimes invalid, because pieces (such as libraries) are all part of the picture.

Was the last plane you were on using front wheel version 2.7 or 2.9? You don't care. You do however care greatly that a "systems approach" was used to ensure that it was whole. And in the OpenBSD case, that means pick a version, and install the patches.

Asking for more means that you want us to do less work on the system, and more version numbering.

Where does the money go?
by MrSparkler

I've seen reports of estimated CD sales per release at being as high as 10000. Add in t-shirt/poster sales and donations and a relatively considerable sum of money is flowing around OpenBSD. Combine this with the fact that checks are to be written to Mr. de Raadt and I get curious as to how the finances are handled. Not that I'm suggesting any misappropriation is occurring, I would just like to know who is in charge of the money and whether or not the OpenBSD project is registered as a non-profit organization (and if it is then checks should be made out to - and the CD image should be copyrighted to - that organization). Also, I would like to see a small financial report put out (as would be required if it were a non-profit organization in Alberta) so that users can see where their money is going. Plus, I would also like know exactly how many CDs are sold per release.

I greatly appreciate the work that the OpenBSD project developers have put in, and I plan on continuing to use, purchase, and donate to OpenBSD (and maybe even contribute when I get the technical skills) regardless of the answer to this question: Where exactly does the money go?

Theo:

We've not yet sold 10,000 CDs in a release. Hopefully we will soon. The project ends up with a bit less than 50% of the revenues from CD sales. The tshirt business is doing OK, but you make a lot less selling textiles. With posters we operate just above break-even. Even though some are sold on the web, most turn out to be free handouts at most conferences. That is how I planned the posters to operate.

We have thought about becoming a non-profit organization, but it is not really a good idea. It would not provide any real benefit -- to you -- as the masses. Especially in Canada, there are costs and serious responsibilities associated with doing such a thing. We would be giving up a lot of freedom, and would need to hire someone to do a lot of accounting. Also, since many of our donations come from outside Canada, we still could not really generate taxable benefits to you. (And I must ask, why are people so cheap, that they only give donations when it provides a partial reduction in their taxes, rather than a real donation? I actually find that pretty fake.)

Money from the project goes to various things. First off, it ensures that I can work full time on OpenBSD, and not need another job. I am also hoping to do the same for other developers in the project, who have indicated that they are interested in doing so. Secondly, certain grimey, unenjoyable, and very important development tasks sometimes put a bit of money in developer pockets. Some OpenSSH work was funded by matching OpenBSD money against donations from a Van Dyke. Thirdly, the project buys a fair amount of hardware: In powerpc land alone, 4 machines this year. Fourth, shipping costs to conferences sometimes severely cut into profits from sales. And finally, when developers get together to do hacking, project money sometimes pays for various things, like airplane tickets, accomodation, and sometimes even some beer. And beer results in ideas, which results in new code.

--------------

Before you ask: yes, we'll be doing Slashdot interviews with people from other *BSD projects in the near future - Robin

Comments

Re:No plans for SMP...

[Jason Earl]

SMP is plenty interesting to the Linux crowd. They have spent a huge amount of time working on it. The fact that it isn't interesting to Theo and the folks working on OpenBSD simply highlights one of the benefits to the Open Source way of getting things done.

If you start your project on OpenBSD and decide that you need SMP to get the performance you need "porting" to Linux shouldn't be much harder than moving your source to a Linux box and typing "make." If, on the other hand, you develop on Linux and then decide that Linux's security isn't good enough for implementation, you can just as easily port to OpenBSD.

There is never going to be an Uber OS that is specialized for every task (although the generic Unix way of recompiling the kernel does come close). That's why standardized APIs are so important. That way you can change your OS midstream if it isn't giving you what you need. The Open Source community has done a pretty good job of matching up APIs.

Theo: Version Number Specifics

[Effugas]

Theo:

I don't think it's fair to say, as you did, that "ftpd is ftpd" or "tar is tar" for all of OpenBSD. Examples from version lines throughout OpenBSD:

spork# perl -v
This is perl, v5.6.0 built for sparc-openbsd

bash-2.04# ./troff -v
GNU troff version 1.15

bash-2.04# nawk -V
awk version 19990620

bash-2.04# gcc -v
Reading specs from /usr/lib/gcc-lib/sparc-unknown-openbsd2.7/2.95.2/s pecs
gcc version 2.95.2 19991024 (release)

bash-2.04# ./cvs -v
Concurrent Versions System (CVS) 1.10.7 (client/server)

[vi :ve]
Version 1.79 (10/23/96) The CSRG, University of California, Berkeley.

bash-2.04# ./tcpdump -V
tcpdump version 3.4.0
libpcap version 0.5

bash-2.04# ./httpd -v
Server version: Apache/1.3.12 (Unix)
Server built: May 5 2000 14:44:59

Look. Some of these you modified. Maybe all of em. Maybe one of em(I *know* you touched Perl.) Lets take the example of tires, why don't we. If I've got Firestone Model X432LFR tires on my car, and I run down to the dealership asking why I'm driving a deathtrap, is he allowed to laugh at me because "Of course *we'd* never put the deadly X432LFR tires on your car, we'd only put the *good* X432LFR tires on! Stupid customer."

That's essentially what happened with Debian a while back, and it was infuriatingly unfair.

I'm not asking you to do more work, Theo--you've *done* the work. I'm asking you to admit it, mark it, brand it in such a way that we know you've been forced to do something to it to make it secure. And then all of us can bitch and moan to the author's of whatever package you've taken and say, "Heh, he changed your stuff, maybe there's something you should look at." Maybe we'll be ignored. But, in the end, *you* did the right thing.

Theo: You and your team rewrote much of an early build of SSH. Technically, you could have said, "Here's SSH1.2.1x, as part of the OpenBSD system." But then nobody would have known what you had pulled off, and people would have had trouble finding your specific improvements. I'm not saying you need to rename every package to show how much you've added. But to keep the original version numbers is to conflate your secure and solid version with whatever bugs you *know* lurk in other people's code. When Foobar 1.2 comes out with a remote root, and OpenBSD ships with Foobar 1.2, do you like--or enjoy--when system administrators frantically upgrade your *already fixed version* of Foobar 1.2 with the original author's possibly broken Foobar 1.3?

Because that's what your version numbers cause.

They're easy to fix, Theo. It's just a tag to let us know you fixed something. It's something for us to differentiate your code with. (Incidentally--what does little on my 2.7 Sparc build.) Consider this: As much as you say you've only dealt with the system, I *know* many of the packages from Ports have had patches that didn't modify version numbers--and I have *no* idea if anything's been modified in your packages section. I just don't know.

This is not a problem specific to you, but I think OpenBSD is in the right place to change what I consider to be a particularly pernicious industry practice. I believe in your systems approach, but a secure system cannot be built from insecure parts. If you've secured your parts--show this, and perhaps let us know where to look to find out how.

Yours Truly,

Dan Kaminsky
DoxPara Research
http://www.doxpara.com

Before flaming Theo ...

[OWJones]

... try and think about what his priorities are. I admit that when I read over the interview my first reaction was "What an ass! People are trying to give him serious questions and he's blowing them off!" Which is half true. Subscribers to Bugtraq will be a bit more familiar with Theo's, uhm, interpersonal style. :) In reality I suspect Theo just wants to write, maintain, and release good, bug-free, secure code and not be hassled with feature creep or random things a good number of his users won't care about or really use.

And what's wrong with that?

OK, he's no "Dear Abby", but neither is RMS. I know many here aren't big RMS fans, but are you insecure enough about your own little world that when someone says, "Quit bothering me with stupid details, just write the code" you start flipping out? I attended Supercomputing '99 and went to a talk by Thomas Sterling, one of the original Beowulf pioneers at NASA. A good chunk of his talk was spent complaining about "Linux cruftiness" and "why are you people here when you could be writing code"? I admit I was somewhat pissed coming out of it, but it did have the effect of motivating me to start programming again. I think sometimes we just need a swift kick in the arse from someone (hi Greg!) to get motivated.

Bottom line: ignore the stupid "Are you mad?" comments from someone whose ego is a bit too big to take the time to be polite, and focus on the "learn your APIs, understand your APIs, and stop writing shitty code." Learn the message, ignore the messenger.

-jdm

(I'm ready for my big Mod-down, Mr. Director :)).

Missing the point on floppy-based routers?

[swb]

I think he's missing the point on floppy based routers. The goal of using a floppy is to eliminate the hard disk, both for cost purposes and hardware performance.

Flash RAM would be perferred, but flash disks are hardly ubiquitous and free-for-the-taking x86 systems that work great as routers don't generally have flash-based disks installed. A floppy drive is almost a given in any system.

The hardware advantage of a system without a hard disk is the reduction of heat generation, meaning they're easier to put in heat-hostile environments like telephone closets. System upgrades are a snap, since you just move the disk to another platform.

As far as security goes, other than floppy disks general lack of reliability, what's the problem with them? They're physically write-protecable, which many IDE disks aren't. Sure it's easier to swap a floppy out than a HD, but if your machine doesn't have physical security to begin with you've failed the first checklist item for security.

Do it right!

[devphil]

Can you imagine if a Boeing engineer didn't fix ALL of the occurances of a wiring flaw? Why not at least try to engineer software in the same way?

Hear, hear! Two of my pet peeves right there:

(1) Why is it that the same bugs keep reappearing? Why is it that we assume bugs only occur in one place? Why is it that we hear, "I fixed the bug," as if a programmer can only screw up in one place?

(2) Every other piece of engineering goes through major scrutiny. Teams are brought in from the outside to look over blueprints. For open source software, we assume that just because anybody can look at the code, that everybody is. Even in OSS, we need to go to outside, objective reviewers and say, "Here's some money, and here's our code [or maybe, here's the URL for our code :-) ]. Please review it and tell us where we screwed up."

Mr. de Raadt knows his stuff; the coders do this themselves, and they take it seriously.

Working with microkernels

[Pink Daisy]

His view of microkernels is pretty standard, but I wonder one thing. As he says, many code errors are due to poor understanding of API's. A microkernel should proved a smaller and more uniform set of API's that are easier to locate and work with. Wouldn't this make correctness easier to obtain? Or is this a goal that is lost in the real world?

Forks are Good!

[Kiss the Blade]

In Linux land, it appears that projects fork for financial reasons. In BSD land, it appears that the forks that have happened were purely political reasons. I don't know what will happen. It's been 5 years since the last fork in the BSD camp. Why are you guys so fork paranoid? Do you want everyone to vote for the same political party, too?

I think he is totally correct with this point. The point of a codefork is that you end up with two variants, only one of which, in the long term, will survive (usually). It leads to a sort of Darwinian survival of the fittest, and improves the overall code base in the long term as well as giving people options - they can mould their distro to their needs. I have often wondered why the Linux people are so scared of code forks also - could it be because they look back at the Unix wars of the eighties and shudder? This would suggest that the BSD'ers have not inherited the UNIX philosophy to the same degree as the Linux community - that may give them more freedom.

I am not suggesting that forks be encouraged, though, rather that people stop whining when they occur, and recognise them as an opportunity.

Perhaps forks will not be a good thing for Linux in ten years or so, but given that it is presently a sort of 'primordial ooze', and very creative, I do not think it is bad thing for the moment.

KTB:Lover, Poet, Artiste, Aesthete, Programmer.