Sendmail Performance Tuning

Andy Murren submits this review of Nick Christenson's Sendmail Performance Tuning, writing "The first thing that I noticed about this book was how relatively thin it is at only 223 pages. This is compared to the O'Reilly Sendmail book by Bryan Costales and Eric Allman (a.k.a. the 'Bat Book') which weights in at 1232 pages. In this case, thin is good. The second thing that I noticed about the book is the quote from Eric Allman emblazoned across the top that proclaims 'This book is great.' That's rather ringing endorsement from the original creator of sendmail. But great what? A great technical reference? Companion to the bat book? Kindling? Now here is my sound bite about this book: 'One damned fine technical book.'" Read on for the rest of Andy's review. Sendmail Performance Tuning author Nick Christenson pages 256 publisher Addison Wesley rating 9 reviewer Andy Murren ISBN 0321115708 summary Compact but well-constructed book; shows you ways to cost-effectively improve your sendmail setup, and much of the rest of your system along the way.

This book was very easy to read and kept moving along. While I did not find it as much of a page turner as Douglas Adams' Hitchhikers Guide, it did keep me interested. One word of warning though: do not read this when you are tired. You will miss some really good information and have to re-read parts.

I was surprised and very pleased with how much I learned about so many things about Unix, networking and hardware while learning more about sendmail. There is a wide range of information presented that readers at almost any skill level would be able to use. The writing style and use of language was easy to read. The wealth of information packed into the pages of this book I found immediately usable on my Linux boxes and for my job.

I think the only drawback to the book was that there was not more specific sendmail information. Due to the nature of the topic, a lot of the book is devoted to how Unix systems work, more than specific sendmail configuration tasks. Time and detail is spent on other more important considerations such as logging, disk performance, test planning and file systems.

Chapter 1 gets us started with an overview of tuning in general. One of the more important themes of the book is established up front: It is that a cost/benefits analysis should be done for each step.

Sections 1.5, "Tuning Isn't Always Necessary" and 1.6, "Not So Fast ..." establish baseline considerations for making a decision on what, if any, tuning efforts should be made. Throughout the book Christenson reminds the reader to decide on a path that is most cost effective for his organization. Is it the most cost-effective use of the company money and time to have the IT staff hand-craft solutions, or is it better to throw some hardware at the problem?

Let's face it -- if you are running sendmail for a small company and only move a few hundred emails a week, how much performance tuning do you need? If, however, you are running an ISP, a mailing list server or a medium- (or even a large-) sized company mail server, then you need to tune your mail server. This is the book for you. The information in this book, while oriented for sendmail, is actually applicable for tuning any Unix based Mail Transport Agent (MTA) server.

Each solution is an individual matter, that is wholly dependent on several factors. Some of these factors are: volume of email, what the main use of the email system is, how the end users interact with the system, what hardware is being used, how much bandwidth you have and much time and money you have to throw at the problem. Of course, what management considers important is the overriding factor in all decisions.

Chapter 2 is a ten-page introductory overview to sendmail, covering versions, obtaining the (Open Source) code and building sendmail. One of the important things covered is the queue and message spool layouts and permissions. This is helpful for making sense of things later in the book.

A few very important pages are spent on creating the .cf file and why you should use M4, the macro language, for managing the configuration files. Having hand-crafted a .cf file myself several years ago then having to deal with maintaining it, I can vouch for the wisdom of using M4.

The maintainers of sendmail update the M4 macros for new features and changes. The 200 - 300 lines of M4 macro files are converted into a 1500to 2000 line configuration file. While it may be easier to figure out the configuration file to make changes, those changes may not be valid from one version of sendmail to the next.

Christenson admits that he does not always use M4 when in a rush or to test some things. What he does is copy the working configuration file to sendmail.cf.REAL before making changes and updating the .mc file afterward.

The next chapter, Chapter 3 'Tuning Email Relaying' starts with an overview of the email relaying sequence. Most of the discussion in this chapter is not sendmail specific. The importance of data synchronization is emphasized here. Section 6.1 of RFC 2821 is quoted, where it states the email server 'MUST NOT lose the message'. Once that is stated and understood all of the requirements that are discussed in the rest of the chapter are clear.

The next 17 pages are spent discussing how file systems, networking and effective use of file space support, and can detract from, meeting the edict of the RFC. For me, these sections are some of the most interesting, filled with information presented in a concise, readable and detailed manner. I learned a lot about what impact some very basic decisions have on email performance. I also learned how much better I could have made the email servers I have worked on. I will be turning to this chapter the next time I am putting together a box, be it a web server, mail server, file server or even a workstation or laptop. This is a great chapter that can help with any system configuration.

Email reception is covered in chapter 4. Different strategies for verifying recipient, tuning POP and IMAP are covered. Effective use of Local Delivery Agents (LDAs), including procmail, is covered here also.

Additionally, an excellent discussion of storage systems, including disks and solid-state disks, is in this chapter. The sections covering RAID levels, benchmarking and use are well written and informative. Available options on drives, ATA vs. SCSI, Solid State, are just as well done. Section 4.4.2, 'Stupid Disk Drive Tricks,' has some nifty information about how to set up disk drives for even better performance.

Sending email is the next chapter. Here we see some more sendmail-specific information. Tuning of mailing lists and mass mailing is part of the discussion here. One of the more important sections is 'Draining Queues.' How backups are caused and what to do to recover are discussed in this section. This has good information that can be used with any MTA.

One of the most important chapters for me is Chapter 6 'Configuration, Security and Architecture.' Sendmail specific configuration and tuning options are discussed. Section 6.1 covers configuration and is in many ways the heart of the book. This is where sendmail directives that can directly impact performance are covered. How a system's architecture (and DNS) is laid out can have a significant impact on performance.

The section on security is good, but brief. Most of the discussion is on privacy and stopping spam. The use of Transport Layer Security (TLS) is covered in less than a page. With a smattering of security in other parts of the book, this is the sum total of the security discussion of the book. Considering some of the problems with sendmail in the past, I would have liked some more information on this topic.

The next two chapters, 'Finding and Removing Bottlenecks' and 'Load Generation and Testing,' are good, solid, well-written sections that are applicable to most any email system. We are given some effective ways of making systems run better, and how to prove that the system actually does work better. This is where we justify to the boss that the work we have done is really cost effective.

Chapter 9, 'Conclusion' is basically a very brief wrap up and a list of books Christenson thinks we should read.

My own conclusion is that I have learned and relearned a lot of things about Unix and email. Even if you do not use sendmail, I recommend this book without reservation. It is an excellent reference on general system performance tuning, with information on making your sendmail installation run better.

Table of contents

1. Introduction
   • 1.1 Performance Tuning Examples
   • 1.2 sendmail Versions Covered
   • 1.3 Definitions
   • 1.4 Email Server Tasks
   • 1.5 Tuning Isn't Always Necessary
   • 1.6 Not So Fast...
   • 1.7 Email System Profiling
   • 1.8 General Tuning Ideas
   • 1.9 Summary
2. Sendmail Introduction
   • 2.1 Obtaining Sendmail
   • 2.2 Building Sendmail
   • 2.3 Creating a .cf file
   • 2.4 Why Use M4?
   • 2.5 System Setup
   • 2.6 Summary
3. Tuning Email Relaying
   • 3.1 What Happens During Relaying
   • 3.2 Synchronization
   • 3.3 File Systems
   • 3.4 File Space
   • 3.5 Networking
   • 3.6 Summary
4. Tuning Email Reception
   • 4.1 What Happens During Email Reception
   • 4.2 Recipient Verification
   • 4.3 Storage Systems
   • 4.4 Disks
   • 4.5 Solid State Disks
   • 4.6 POP Tuning Specifics
   • 4.7 Message Storage Hashing
   • 4.8 IMAP Tuning Specifics
   • 4.9 Summary
5. Tuning Email Sending
   • 5.1 Mailing Lists
   • 5.2 Command-Line Message Generation
   • 5.3 Draining Queues
   • 5.4 Another Mailing List Strategy
   • 5.5 SMTP PIPELINING
   • 5.6 More Notes on Mass Mailing
   • 5.7 Summary
6. Configuration, Security and Architecture
   • 6.1 Configuration
   • 6.2 Security and Performance
   • 6.3 Other General Strategies
   • 6.4 Summary
7. Finding and Removing Bottlenecks
   • 7.1 Kernel Parameters Run Amok
   • 7.2 The Quick Fix
   • 7.3 Tools
   • 7.4 syslog
   • 7.5 Removing Bottlenecks
   • 7.6 Summary
8. Load Generation and Testing
   • 8.1 Test System Setup
   • 8.2 Testing Tools
   • 8.3 Load Testing Pitfalls
   • 8.4 Summary
9. Conclusion

You can purchase Sendmail Performance Tuning from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

Comments

Oh Yes!

[cca93014]

I will buy this book, and file it under "oxymorons I have known and loved"

apache and sendmail books not quite useful

[stonebeat.org]

i have seen many books on apache and sendmail. and i have quite a few of them. i rarely consult them. when i need to find out anything, i m most like to type it into google, or search newsgroup (deja). this includes apache/tomcat performance tuning.

I have administered NIS+ apache, and sendmail, and these topics are so vast, that buying books doesn't help. However searching through google/deja does help a lot.

But that is just my opinion....

Re:Qmail!

[EnderWiggnz]

i've always started by openning up a modem connection, picking up the phone, sneezing into the receiver and capturing the output as my sendmail.conf

Sendmail tuning?

[zulux]

Tuning Sedmail is about as smart as tuning MS Exchange. Both are fat, bloated, and have a history of secutiry holes. Performance tuning Sendmail is like performance tuning an AMC Gremlin made out of spare junkyard parts - you end up with a cobbeled peice of crap with a spoiler.

Sendmail is so bloated that it apparently takes books to get decent performance out of it.

Sendmail doesen't fit the one of the core Unix ideas - use simple, robut and elegent programs, chained together, to do cool things.

Check out Postfix or Qmail for decent replacements to Sendmail. Enjoy the performance of well toughtout software. Postfix was so easy to use, that I went from reading the man pages to a working system in one hour.

Re:Sendmail tuning?

[spanky1]

use simple, robut and elegent programs

Robut? You sound like Dr. Zoidberg on Futurama.

But to stay on topic: We use sendmail at all of our locations because we enjoy the extremely powerful customizability. Yes, it is a little disconcerting trying to disect sendmail.cf, but once you get the hang of it it's not bad. I have never had to do any performance tweaks either.

Re:Sendmail tuning?

[zulux]

We use sendmail at all of our locations because we enjoy the extremely powerful customizability. Yes, it is a little disconcerting trying to disect sendmail.cf, but once you get the hang of it it's not bad. I have never had to do any performance tweaks either.

There are a lot of good thing about Sendmail - but I'm afraid, for me, the good things are hidden in layers of suckyness.

If someone sat down with the source code and only used the delete key - you could strip out the crap and make Sendmail cool. And, of course, change the sendmail.cf format to resemble somthing human readable.

It's kind of like looking at a fat-girl, and saying to yourself "She'd be cute if she lost a few pounds". You know it's never going to happen, she'll just keep eating the Happy Meals. I fear Sendmail will keep on getting bigger as well...

Re:Sendmail tuning?

[ZoneGray]

I haven't yet worked much with Postfix, but I hvae a couple of servers where I run qmail. I believed the hype at first, but eventually concluded that djb is one very, very disturbed person.

I guess it works okay, but only a badly twisted mind could have devised the configuration system that qmail uses. Once you add one of the various virtual domain hacks it becomes nearly impenetrable. Then you try to integrate a mail list manager or spam tool, and before you know it, you're waist deep in GID's and SUID bits and log files, and you find yourself trying to read a Google translation from the original Slovenian.

Re:Sendmail tuning?

[Bruce Perens]

I wouldn't have stated it quite so aggressively, but I have to agree. I used to run Sendmail on the Debian list server, around the time it got to 100,000 emails per day (it's probably at a Million now). It ran on a 486, with probably 32 meg of RAM (this was 5 or 6 years ago). Sendmail, of course, bogged down. I switched to qmail and noticed a tremendous performance increase. After qmail was fully released and I became disenchanted with Dan's non-licensing scheme, I was no longer the Debian list manager but switched my own operation to Postfix, which works admirably.

The thing that Sendmail did well, address rewriting, is irrelevant for today's net, and can be done more readably in many other ways. Even a Perl script is much more readable :-) Installing one as a Postfix transport is trivial.

I'm surprised that there isn't more business action around Postfix.

Bruce

Re:Sendmail tuning?

[FroMan]

Whoo-hoo! Replying to a Bruce post. :-P And not agreeing(sp?) with him.

I have to say that the address rewriting in sendmail is necessary. And because of it, sendmail was the only option for what I needed to do a short while ago. Basically we were using fetchmail to retreive mail from a single multidrop box from our ISP for our department email. Then, for the relay outbound we had to determine if the email was to be sent locally or back up to the ISP if the user was not in our department. Then, after all that I still needed to add amavis(sp?). Sure, it was not the cleanest/most readable .cf file, but it was the only thing I could get to do the trick. When looking at postfix I did not see anything that allowed me to do the address rewriting to the granularity required. Then looking at qmail, well... to avoid four letter words I'll stop there.

The best part of sendmail is actually its obtuse feature set. Its similar to cars with 4wd, most poeple don't need it, but when you do need it, its mighty nice to have.

Re:Sendmail tuning?

[Zapman]

You (and the others following this thread) owe it to yourselves to give postfix a try. It is clean, and elegant. All of it's configs live in 2 files (main.cf and master.cf), and it is powerful.

Postfix has solid anti-spam measures, fantastic performance, readable config files (that are well commented from the start... heck there are about 10-20 comment lines per config line), and you only need to set like 3 lines in the config to get a working mail server.

Re:Sendmail tuning?

[GooberToo]

postfix exploit

Found this by typing, "postfix local exploit" on google. Seems there were tons of other hits on this too (about 6,900).

Re:Sendmail tuning?

[benedict]

The hole -- which was very, very small to begin
with -- has been closed. Please stop spreading
misinformation.

Nice to see M4

[theonetruekeebler]

It's good to see the author advocates using a macro tool and not hacking sendmail.cf directly. It's scary in there.

I'm trying to remember who said this:

When I first started working with sendmail, I was convinced that the cf file had been created by someone bashing their head on the keyboard. After a week, I realised this was, indeed, almost certainly the case

Re:Nice to see M4

[Just Some Guy]

Absolutely. I'm convinced that people who bash Sendmail because of it's supposedly difficult configuration have never seen a sendmail.mc file.

For example, this snippet from a production server's sendmail.mc will turn on DNS-based spam filtering, enable the use of procmail for local mail delivery, and define a smart host to send email through:

FEATURE(dnsbl, `relays.ordb.org', `"550 Mail from " $&{client_addr} " rejected, see http://ordb.org/"')
FEATURE(local_procmail)
defi ne(`SMART_HOST', `mail.upstream.com')

I won't argue that the equivalent sections of sendmail.cf are, well, opaque. However, I've literally never touched sendmail.cf, other than to look at it and thank $DEITY that a much simpler, human-readable configuration system exists.

Re:Nice to see M4

[benedict]

Sendmail is definitely more usable with m4. But it's still
pretty annoying to have to preprocess one's config file.
If I ran the world, sendmail would optionally preprocess
its config file itself. In other words, you'd be able to use
the m4 file directly.

Performance tuning?

[RyuuzakiTetsuya]

I'd figure a Spoiler, a few 'Type-R' stickers, a coffeecan exhaust, and a bolt on Turbo kit should do the job. Who needs ruddy books?

Re:Qmail!

[Just Some Guy]

One is Free Software, and one is not. That's important to some of us.

What is with all the sendmail bashing?

[mustangdavis]

Seriously people ... if you don't like sendmail, don't come in and bash it on a sendmail book review (btw: all of those posts SHOULD be modded off-topic).

If you like another mail program, DO A BOOK REVIEW ON THAT MAIL PROGRAM and share your "wisdom" with the rest of us.

As for sendmail, it is not meant for Windows weenies or wanna be Unix sysadmins, only for "real" Unix admins and those that wish to use a VERY robust product and are willing to learn the product to reap the rewards. I personally feel that this book is a great quick reference for when you have a brain fart and don't feel like mulling through the O'Reiley "bat-book" for a simple answer ... but nothing can ever replace "the real thing" ....

... and if you don't agree with me, be intelligent and post why ... don't moderate!

Just my $0.02 cents

... preparing to dodge all of the "clippies" that will be thrown at me by the NT admins ... "

Re:What is with all the sendmail bashing?

[zulux]

As for sendmail, it is not meant for Windows weenies or wanna be Unix sysadmins, only for "real" Unix admins and those that wish to use a VERY robust product and are willing to learn the product to reap the rewards.

Are you kidding?

I can get PostgreSQL to master/slave replicate a database cluster over 5 servers in faster time that I can get Sendmail do somthing simple. Really.
The Sendamil config files might as well be binary - there useless to human eye. And, no, you shouldent have to learn a line-noise language just for a silly MTA.

Sendmail is just a bitch. I had it's time in the sun, and it's time to move on.

only for "real" Unix admins

A 'real' Unix admin wants to get work done and learn somthing usefull. Managing Sendmail is just memorising a bunch of trivia about it and hoping it works as you think it should. Managing Postfix or Qmail is like managing any other decent Unix server - and by learning one of them you can take that knowlage and use it somewhere else.

A 'real' Unix admin knows when a peice of software is a cobbled peice of crap and when to move on to somthing sane.

Re:What is with all the sendmail bashing?

[puppetluva]

I can get PostgreSQL to master/slave replicate a database cluster over 5 servers in faster time

I'm looking to do exactly that! How? (I'm not kidding) I'll trade you, you show me the replication trick and I'll show you how to do simple things with sendmail ;)

/

Re:How to improve Sendmail performance.

[Just Some Guy]

The fact that Sendmail is monolithic (one program does it all) where as qmail is modular makes it more secure too.

Since Sendmail hasn't been monolithic for a couple of versions now, have you considered switching back?

Re:Sendmail Performance Tuning - the short guide

[dzelenka]

Us Debian weenies have exim installed by default. This Debian weenie does prefer to "apt-get install postfix". What is this Sendmail thing that everyone keeps talking about?

Large Canadian ISP is using it

[haruchai]

My former employer, AT&T Canada, who has over 150,000 home users and several thousand business
clients, dumped their Sendmail-on-Solaris servers in favor of Postfix-on-FreeBSD and have never been happier.

Comment removed

[account_deleted]

Comment removed based on user account deletion

More MaxRunnersPerQueue speeds sendmail

[petej]

Sendmail's default configuration is very conservative with respect to both your machine resources, and the resources of those with whom you exchange mail.

Run sendmail with

-o MaxRunnersPerQueue=10

(or something bigger, if you want) and you can get past the biggest hurdle in sendmail performance -- by default, sendmail will handle all outgoing messages in a single queue. (Both qmail and postfix use multiple queues and/or runners by default.)

Sendmail can relay messages without doing any disk I/O, thanks to the asynchronous I/O libraries; coupled with multiple queue runners, it's hard to get faster performance out of any MTA.

Nice to see a positive review

[bobcat]

Nick's book is very good, and he has managed to "give away" a lot of the tricks of the trade that are used by Sendmail, Inc. Professional Services.

Sendmail (the app, not the company) scales quite nicely if you know what you are doing - I've installed sendmail on a couple of old dual-proc HP's and they handle about 2 million messages a day. Yes, that's right; 2 MILLION. It's a "simple" matter of tuning and knowing the file system.

Don't hack the .cf directly. There be dragons there. Use M4.

And, if another tool works better for you (exim, postfix, qmail) - use it! I don't always recommend sendmail, either.

Bobcat

disclaimer: I used to work for Sendmail, Inc. - my cubicle/cage was about 40 feet from where Nick used to sit...

Don't forget...

[bani]

...racing stripes

My secret Red Hat sendmail trick.

[emil]

These days, Red Hat's default install of sendmail does not accept remote connections. Red Hat tells you to use M4 to generate a new CF file to enable this functionality.

Here is what you really do...

Find the line in the CF file that reads:

O DaemonPortOptions=Name=MTA

and change it to:

O DaemonPortOptions=Port=smtp,Addr=0.0.0.0, Name=MTA

I picked this up from a bugzilla comment.

Best thing I ever did was *de*tune sendmail

[Ben Jackson]

I added:

define(`confBAD_RCPT_THROTTLE',`1')dnl

to my mc file. This causes (recent versions of) sendmail to add a short delay before responding to each bad recipient after the first one. The delay is hardcoded at 1 second (unfortunately) but I also changed the source to increase it to 10 seconds.

You can run dictionary attacks against domains I handle mail for, but at least it will take you a damn long time! I just wish everybody was doing it. Eventually it would take too long to spam effectively.

Those who don't understand sendmail are doomed...

[Kunta Kinte]

...to repeat it.

Um, wait. That didn't come out right.

Sendmail is so bloated that it apparently takes books to get decent performance out of it.

Seriously, a lot of people just don't understand sendmail and how difficult mail routing is. Sendmail is bloated because it tries to solve a very complex problem in an extremely diverse environment.

Postfix was so easy to use, that I went from reading the man pages to a working system in one hour.

Postfix and sendmail are going in two different directions. Sendmail == configurability, postfix == ease of use.

Sendmail is fast. Not because there's a book on performance tuning does it mean the software is slow. Sendmail is probably as fast as anything else out there.

Sendmail is configurable to no end. I run a sendmail setup with virtual domains, lots of address rewrite rules, spam filters. All my virtual users, and alias maps are in LDAP. I modify all my sendmail virtual users, virtual domains, aliases, etc through a PHP website. Try that with other mailers. They support LDAP, but they are no where as configurable as sendmail.

Then try running these mailers on Linux, bsd, solaris, windows, AIX, etc., etc. No OS lock-in.

Sendmail is well documented.

The prevaling mood on /. these days seems to be if it's not brand new, get rid of it. That's unfortunate.