Qmail At 10 Years — Reflections On Security
os2man writes "Qmail is one of the most widely used MTAs on the Net and has a solid reputation for its level of security. In 'Some thoughts on security after ten years of qmail 1.0' (PDF), Daniel J. Bernstein, reviews the history and security-relevant architecture of qmail; articulates partitioning standards that qmail fails to meet; analyzes the engineering that has allowed qmail to survive this failure; and draws various conclusions regarding the future of secure programming. A good read for anyone involved in secure development."
I don't mean to be flippant, but this is a really good article. That it appears on Slashdot gives me a lot of hope that this site isn't just a hangout for system administrators but also for software engineers.
The concepts Bernstein discusses regarding increasing security are very interesting, if not exactly obvious. Fix bugs immediately. Reduce LOCs to reduce the probability of bugs. And execute as much code as possible in untrusted mode. His discussion of running untrusted code in "prisons" is interesting, and I wonder what, if any, accomodation for this type of programming Windows has.
It was really nice to see software engineering presented here for once. Thanks kdawson... kdawson? No way!
I'd use Qmail, except that the licence means that in order for Qmail to scale, it has to be patched about fifteen squillion times over ... all thanks to the restrictive licence.
Sure it may be fast and secure... but unfortuantely scalable it is not (and if it is, it is far from obvious how).
Does anybody run an ISP mail system with Qmail featuring predominately as MTA of choice?
READY.
PRINT ""+-0
...writing a navel-gazing paper after ten years or re-licensing the code so it doesn't devolve into an antiquated relic burdened with conflicting and hacked up patches just so it can be barely useful to a modern userbase? This is one more area where the licensing rubber meets the road: when you can do an "apt-get install exim4" because the license allows for modified binaries versus a download / patch / fix conflicting patches / compile / fix broken code / compile / install cycle every time you need an MTA, with future patches for missing features a guarantee, admins will eventually vote with their fingers and stop installing qmail. Much like the free software belief that draconian licensing terms will eventually be the death of the licensors' products, the lack of a license will eventually do the same to djb products. The right licensing is freedom, and freedom adds more to the vitality of a piece of software than any post-decade academic paper will ever do. I guess we should have known the academic would value publishing over community-friendly, viable, real-world-issues-comprise-more-than-security software.
Call it an experiment and warn serious users away. Or free qmail. Until then, I'll be making sure I actively migrate away from it when given the opportunity.
Count yourself lucky that it doesn't all go under /djb
http://michaelsmith.id.au
if you use qmail "out of the box" it might be secure, but its not usable nowadays anymore. You often have to compile in so many patches that at the end there is no security there anymore.
I rather start with an up to date MTA, rather then fight with something like qmail ever (EVER) again.
Just the fact that you have a fixed layout, fixed start tools that need to be there to actually start it, etc etc makes it so horrible, that I wouldn't touch it ever again with a 100 yard pole.
"Freiheit ist immer auch die Freiheit des Andersdenkenden" - Rosa Luxemburg, 1871 - 1919
google html of the pdf (perhaps as bad in some ways as a pdf):
http://preview.tinyurl.com/33lvkr
According to one of DJB's slides it might happen this month that qmail will be placed into public domain. I hope this will become true as it might help qmail to get rid of its obsolescence.
Bill Shupp's patch plus Matt Simerson's Mail-Toaster Perl-library still make a difference.
With postfix or sendmail, you've got to write all the provisioning-tools yourself, but qmail+vpopmail+qmailadmin delivers something out-of-the-box.
http://www.shupp.org/
http://mail-toaster.org/
Windows 2000 - from the guys who brought us edlin
In the PDF at the end of section four he talks about making compromises in the design of the configuration files and the inadvisability of working around file system problems. I can't quote it because my PDF reader is doing strange things with selection but it occurred to be that DJB has some approaches to software in common with Hans Reiser, and that maybe DJB is the right person to drive reiserfs development in the future.
http://michaelsmith.id.au
Where did the submitter get their information from for saying that it's one of the most widely used mail servers ? I suppose if you "widen" your limits a fair way it could come in as being moderately popular.
Sendmail, Postfix, Exchange... sure, they're up there in the high levels.
Anyhow, would love to see a site/page showing the breakdown of mail servers around the net.
Actually, that might be changing in the immediate future. Check out the slides to go with this talk, in particular, page 10 where there's a timeline including:
Bogtha Bogtha Bogtha
The programming model used by DJB is more or less:
,,be strict as possible when sending, and liberal as possible when receiving''. If you can destroy other systems functionality especially designed for email (like multiple mx-es?), huuuge karma boost.
Implement only a subset of protocols, ignore the parts that you don't like, or might be insecure or are too boring to implement. Bonus points if you ignore actual features depended on by the users. Double bonus, if you manage to make it non interoperable by nazi-strict implementation of protocol, ignoring the rule
Then refuse to implement needed features, pointing to third parties and their patches, and offer a prize for successful hack of your software. And ignore the insecurity of the patches. They're third party, after all.
Robert
PS I was so glad when some mature alternatives to sendmail and qmail apeared...
Bastard Operator From 193.219.28.162
I was in a weird situation where there were two of us looking after a company part time. The other guy, a typical djb fanboy, replaced *most*[1] of exim with qmail, vpopmail, and daemontools. Oh what fun this was when he was 'unavailable.' The included 'docs' were garbage. Here's some fun questions for the audience: /etc/init.d/... or delete a file and recreate it to restart.
1. How do you start / stop your MTA?
2. How do you configure software? Config files or adding and removing files from a magic directory?
3. How do you kick the mail queue? Buggered if I can remember.
Having a few years of experience looking after various 'nixes is nothing to being thrown at djb's stuff without warning. Add to this the attitude from the fanboys I've met [2] and I hate anything touched by djb. The other fun thing I can remember from some doc was djb's suggested solution to one problem was to change fork().
[1] mailq ran, but obviously freaked out.
[2] The worst examples of the stereotype, however, I've seen stuff posted online from some very nice people. My sample size was small but annoying.
Already pointed this out, but DJB is just gaining access to chroot, then dropping privileges.
XML is like violence. If it doesn't solve the problem, use more.
That would be too obvious and useful.
I don't usually see much on real software development and design here (outside of book reviews).
While some of the things are "Duh!!" people don't think of it. Many of the metrics programmer skill is based on LOC and bugs per LOC. This type of metric is counterproductive. Articles on the subject correctly reward low bug count more than LOC. But none of this takes into account the efficiency of the code, therefore encourages slow code with no bugs, and lots of lines. He is right, I think some developers (including myself) have the tendency to microoptimize the interesting code.
Fight Spammers!
Regardless of whatever else you might think of him or his software, DJB is a promoter of "security at any cost", for which everyone should give him some respect. If there's anything we should have learned in the past ten years, it's that you can't half-ass security.
Too much software is written as if security concerns are on equal footing with features and performance. That should never be true. If your program deals with untrusted input and has access to sensitive information, then security must be the primary concern during the entire development process. Security is not something that you can "patch in" after the architecture is settled.
There can be no trade-offs when it comes to core internet services. If one mail server is 10x faster than another but also contains a remote execution exploit, it is not 10x better -- it is useless.
You can debate DJB's personal approach to security, but you cannot fault his priorities.
qmail gives the impression of security from a programatic standpoint, and was designed to be so. but unfortunatly wasn't designed for or by someone who understood the security needs/wants of a mail administrator or how those may change. otherwise it would never have been be one of the biggest contributers to backscatter {see end of http://en.wikipedia.org/wiki/Backscatter also security is seldom achieved through feature stripping which appears to be the case the fact it is still used by many non-proffessionals is a testiment more to its original and ongoing hype and buy-in to the myth by the uninformed
Qmail -- whatever its security merits, and it does have some -- is not suitable for use on the public Internet because it fails to comply with not only de jure standards (RFCs) but de facto standards (best practices). The author has refused to correct these defects -- which is certainly his prerogative as an author, but has as a byproduct serious operational impact on not only users of the package, but other mail server operators who communicate with those run by users of the package.
It's my professional recommendation (based on nearly three decades of experience running mail systems) that qmail be avoided in favor of superior packages such as sendmail, postfix, and exim. (Although the latter, unfortunately, appears to be making increased use of an abusive, spam-supporting feature known as "callbacks".) These packages are actively maintained and their authors have made, and continue to make, efforts to make them standards-compliant and well-behaved (despite the increasing stress placed on them by all forms of mail-related abuse).
As an aside, readers interested in the history of qmail should query a Usenet archive for "a tribute to the programming style of Eric Allman".
oops! thats gmail.
I have excellent Karma and I am not afraid to Troll it.
Of course it's secure - it hardly does anything. To even get the most rudimentary features a mail server needs to have, you have to patch the living daylights out of it and link it up with loads of third party software. You end up losing the security anyway when you add these features. It's not a usable mail server in it's native form for most companies - it's just far too basic and takes too long to configure for most real-world setups.
Look at how much extra stuff and TIME it takes to get a small qmail based mail server usable:
http://www.qmailrocks.org/install_rh.htm
I got a comparable Windows based server up and running in under 20 minutes. Try doing that with qmail.
Once again, it's only free if your time has no value.
It's part of the specs I reckon, check out the HTTP specs, same deal. You can't just send \n\n for two lines.
Someday, I'll have a real sig.
I have to say I am sad to see so many negative reactions to djb and his software.
I'm not sure where the accusations of arrogance against Bernstein come from. I've never met the guy but having studied his code I would say, if anything, his programming style exudes humility. He doesn't trust client software, he doesn't trust the standard libraries and he doesn't trust himself. I think if you look closely at his "style" (for want of a better word) you will find a lot worth emulating.
Personally, I still use qmail and tinydns on my own boxes, where appropriate. At work I don't have any problem recommending his software either and have used qmail for projects relaying over 8 million messages a day without issue. Saying it "doesn't scale" is, in my opinion, untrue. Or at least misleading.
Anyway, djb, I for one salute you!
Si
You would be wanting the Postfix source code, then. I've learned a tremendous amount about how secure, well designed software can be constructed. Wietse is a very smart guy, and his code is some of the tightest code I've seen. Go through it, and you'll be a better software developer for it.
I've never looked at the qmail code. It could be just as good, I don't know.
If djb worked at a fire station:
Caller: my house is on fire!
DJB: your sprinkler system is configured wrong *click*
I wrote my first program at the age of six, and I still can't work out how this website works.
> Imagine running the jpegtopnm program in an "extreme ... (working on RLIMITs, do chroot, fork, setuid, etc etc)
> sandbox" that doesn't let the program do anything other
> than read the JPEG le from standard input, write the
> bitmap to standard output, and allocate a limited amount of
> memory. Existing UNIX tools make this sandbox tolerably
> easy for root to create:
>
Hm... even if not allowing the program to reuse the filesystem and other programs in the system is tolerable, it doesn't sound like a good idea that a program that want to *lower* its privilege has to run as root. If nothing else, the system administrators will probably go mad staring at all those suid root programs (much more mad than all the non-standard procedures needed to run the current programs of djb indeed). Unluckily for all of us, POSIX-compliant Unix is still the way to go. I expect that an "extreme sandbox" would be much easier to build (and much more useful!) with a system like Hurd, only that it won't be ready for use outside the Hurd developers' spare PC in any foreseeable future.
Oops, there's the mention of it buried in the paper. Thanks for letting me edit my comments Slashdot
Did you ever notice that *nix doesn't even cover Linux?
Rediffmail uses qmail. 60M users and counting.
Don't piss off The Angry Economist
Sounds like someone has a bad MUA behind their MTA. Upgrade to Cyrus IMAP. It's very smart and culls duplicates by message ID with a sliding window, tunable by the administrator.
This MTA behaviour is, like it or not, the correct behaviour at the MTA level. Postfix (my secure MTA choice for the past 9 years, and [IMO] a far superior one to Qmail) behaves identically regarding duplicates, as does every other MTA I've looked at. I wouldn't be surprised if this was written up in an RFC on SMTP as the correct behaviour.
Now if we're going to talk about incorrect behaviour, we should all jump on the 10.5.0 finder which is completely non-spatial!
--
Internet Explorer (n): Another bug -- that is, a feature that can't be turned off -- in Windows.
http://bash.org/?7565
I'm kinda surprised by all the complaining in this thread. Here's a very competent software engineer who created several highly secure and useful applications that we can all use for free, and he's giving us his retrospective thoughts on the engineering choices and...
Everyone is posting "djb sucks" and such? What a bunch of useless pricks we can be.
DJB - thanks for qmail. It's odd but pretty cool and has never fucked up my system. And I found the paper pretty interesting.
Cheers.
Way to take a single line out of context. You should point out that the chroot is one step out of five or six or so.
Vintage computer games and RPG books available. Email me if you're interested.
I'm really disappointed that you didn't read TFA. "Example where qmail did badly: integer arithmetic."
Don't piss off The Angry Economist
No. Just give us her phone number and I'll see to it.
Religion is what happens when nature strikes and groupthink goes wrong.
What if the file system layout standards are wrong? For example, the same piece of software can have two different installation locations: /usr and /usr/local depending on who compiled it. That's just wrong.
Don't piss off The Angry Economist
DJB makes an unusual and somewhat extreme claim in this paper: he says that the Principle of least privilege is "fundamentally wrong." I thought this was a strange thing for a security person to say, and I wrote a blog entry explaining why I think he's overreacting.
And on my systems, svscan is started by svscan-boot from rc.local.
It makes perfect sense. Your package manager installs binaries in /usr/bin and /usr/lib. You don't want to write to those directories yourself so you don't conflict with the package manager. Binaries you compile yourself go in an alternate set of directories, /usr/local/bin and /usr/local/lib.
Hands in my pocket
This is just my opinion of course, but what I think Alan was trying to say is that chroot(2) environments are no replacement for well-written, regression-tested peer reviewed software, nor a substitute for well-defined security policies and procedures. However I still love the risk mitigation that chroot environments provide.
It's like locking your car or the door to your house at night. It wouldn't prevent a determined intruder (although perhaps it could), but it's better than not using the feature at all. Nonetheless I would agree that if it gives you a false sense of security that's not good either.
It's true no man is an island, but if you take a bunch of dead guys and tie 'em together, they make a good raft.
Of god.. of course it supports TLS. For heaven's sake do you research before flaming something but what am I thinking this is /. ... noone does that
Alan's comment related to a discussion where the process in the chrooted environment still had root privileges. In the example in the article, the process also drops privileges and so cannot call chroot() to get back out of the jail.
This is my biggest complaint about qmail, and I wish I had mod points right now.
You should see the hoops I had to jump through in order to convince qmail to reject spams/viruses during the SMTP session. It can be accomplished with qmail-qfilter, but it ain't pretty.
The problem is, I only run a personal mail server, and I haven't yet found another MTA that is as simple as qmail. This is a low priority for me, so I don't want to spend hours learning a new MTA.
Do you have a recommendation for a simple MTA that integrates well with clamav and spamassassin?
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock
Seems like everybody and his dog has a patchset for qmail. Unfortunately, many of them conflict with each other. Patchset A solves problems 1, 7, and 12, patchset B solves, 2,7 and 12, patchset C solves 1 and 15. You want to solve problems 1 and 7, but don't want the fixes for 12 or 15. Or you want A's fix for problem 7 but B's fix for problem 12.
Qmail was great in its day, but that day has come and gone. Maybe the public domaining of qmail will allow it to be distributed properly and make it viable again.
Infuriate left and right
Who says I like ALL of the patches he chose? Suppose the patch he chose of the several that solve the same problem has side effects that I don't want, or that solves it in a manner I don't want.
The more patches lumped together, the greater the odds there will be something in it that is wrong for any arbitrary situation.
And if you don't lump them together, the greater the odds that they will conflict.
THAT is what is wrong with qmail's patch mentality.
Infuriate left and right
Putting executables under /var is "wrong". Relocatable installation is a feature.
Whats more, if you patch the hell out of qmail (in order to make it actually useful in a real-world situation for example) you effectively lose DJBs security 'guarantee' (such as it is).
(I say 'such as it is' because, IIRC Fyodor (spelling?) found a remote root exploit of the default install of qmail on (IIRC) OpenBSD and when it brought this to DJBs attention, DJB refused to recognise it as such).
In the free world the media isn't government run; the government is media run.
And one heck of a decent guy, too. Unless he's destroying your career for no real reason.
Dewey, what part of this looks like authorities should be involved?
Laws do not persuade just because they threaten. --Seneca
DJB: I think you'll be happy to see that I've completed the construction on my absolutely secure data closet. I can promise you that no one will be getting unauthorized access. In fact, if you can find a way to get unauthorized access, I'll give you a prize.
Bystander: Where is it?
DJB: Why, right here!
Bystander: Where?
DJB: Right here, behind this door! What, are you stupid?
Bystander: That's not a door. At least, if it is, it doesn't appear to have a handle to open it with
DJB: Well of course not, idiot. A handle might allow someone to break in
Bystander: It's not much of a door without a handle, and besides, the fire marshal is going to be upset if you've got a door that someone can't get out of
DJB: It's a very secure door. No one can break in at all, and besides, what do I care what the fire marshal thinks. If he cared about securing facilities, he'd change his tune. I dropped by the fire station the other day, and there were door handles all over the place. I bet they get broken into all the time.
Bystander: So what if someone takes a prybar to the door and opens it?
DJB: Well, to tell you the truth, the door is just a patch...
Check out my sysadmin blog!
Qmail does not provide banner that allows to identify software. 0.17% is for Qmail toaster.
Cool so we can say that the total for qmail is somewhere in the 0.58% that didn't provide a banner.
Assuming that there are other MTAs which do not or are not configured to provide a banner we can safely say that the figure for qmail is *less than* (0.58% + 0.17%)
Yeah that qmail sure is a popular and widely used MTA. Sure is. Yep.
In the free world the media isn't government run; the government is media run.
I'm disappointed that you didn't bother to read the comment I made immediately after I spotted it in TFA.
Did you ever notice that *nix doesn't even cover Linux?
Security is about way more than remote root holes. Sure there is no known remote root exploit in qmail, but I can abuse a qmail server to send out massive amounts of SPAM since qmail late-bounces instead of rejecting messages during the SMTP session.
10 years ago, generating bounces was OK, but that's not OK in 2007. Hasn't been OK for a few years now.
They don't grade fathers, but if your daughter's a stripper, you fucked up. --Chris Rock
I'd have to agree - I just got done porting our mailsystem from the One Big Box(TM) to a more flexible set of 1U's - and Postfix Admin (with MySQL replication) works just as well managing our multiple-MX, separate-mailstore system as it did the previous monolithic box. It's simple to modify (some of the cleanest PHP code I've seen in some time), and not too confusing for our clients, who can manage their own mail and reset their own passwords. Makes for a lot less in the way of service calls.
I've messed with sendmail, exim4, postfix, and a little with Exchange, and I'll say that Postfix has made the effort I spent in learning its ways worth it. I'd hate to try to set up one of our current projects - a spam-filtering front end for an Exchange server, with LDAP lookups, greylisting, RBLs, the whole bit - on anything other than Postfix.
Why can't I mod "-1 Idiot"?
DJB is a competent and smart hacker (and by this I mean programmer, the correct word for people who break stuff is cracker), but I don't use his software for two reasons:
So, for me, qmail is good for reading the source or for hacking (by this I mean playing to learn). Would I put qmail on a production server? No!
I actually use exim, mostly because it's GPL and Debian default. Postfix would be good if it was GPL too (for the uninitiated, Postfix was started by the guy who wrote the old good SATAN).
I see that DJB has unpopular ideas, but I don't think this should make some of you bashing him whenever his name is mentioned in a forum.
Robert
PS Reductio ad Absurdum is a valid tool. It just lets you see if your thesis takes all the borderline cases into account. In this case I am just trying to tell you, that the (lost) functionality cannot be the cost of security.
Bastard Operator From 193.219.28.162
...but it's not the problem you might think.
I'm surprised at the complaints about patching. For your average implementation there are single rollup patches that cover just about everything. We run several qmail based systems where I work. We find them stable and relatively easy to work with.
The only times we take them down is when those 'on-high' want to implement some weird business rule for email. I don't really like DJB's coding style but it is clean and consistent.
"perfect sense" to an insane person. Okay, so when I want to tell somebody what program to run, do I tell them /usr/bin/foobar or /usr/local/bin/foobar? Because, just as you say, there may already be a foobar in /usr/bin which the user didn't want to overwrite. The only way I can tell them is to execute another round-trip of question and answer to find out whether they compiled it themselves or whether it came as a package. With qmail, I can say "run /var/qmail/bin/qmail-qstat" and tell me the second number.
Don't piss off The Angry Economist
Errr, no. The executables in this case are specific to the machine, thus they *belong* in /var. Putting them anyplace else would be wrong.
Don't piss off The Angry Economist
Erm, no. The /var directory is for variable files, not executables of any flavor. Depending on platform either /opt or /usr/local is the appropriate location for locally installed software (or increasingly software not managed by the native packaging system).
Having two versions of the same program is silly. If they want to compile their own version of foobar, they should always remove the one installed by their package manager first. If they don't do that, well, I suppose there are worse ways to hose your system. Overwriting something in /usr/bin would be one of them, because when the time comes for your package manager to update that package, you'll have an inconsistent mess on your filesystem.
What you do is don't specify the path. Tell the person to run "foobar". If their shell is setup correctly, the program will run.
The Linux/UNIX world is perfectly happy with this setup. But we must all be insane, it couldn't just be you lacking understanding.
Hands in my pocket
Only when I get the distinct impression that everyone's forgotten. Consider what I was replying to:
Easily sounded like someone in need of a history lesson.
So what's your excuse?
Don't thank God, thank a doctor!