Slashdot Mirror
Sendmail Performance Tuning
This book was very easy to read and kept moving along. While I did not find it as much of a page turner as Douglas Adams' Hitchhikers Guide, it did keep me interested. One word of warning though: do not read this when you are tired. You will miss some really good information and have to re-read parts.
I was surprised and very pleased with how much I learned about so many things about Unix, networking and hardware while learning more about sendmail. There is a wide range of information presented that readers at almost any skill level would be able to use. The writing style and use of language was easy to read. The wealth of information packed into the pages of this book I found immediately usable on my Linux boxes and for my job.
I think the only drawback to the book was that there was not more specific sendmail information. Due to the nature of the topic, a lot of the book is devoted to how Unix systems work, more than specific sendmail configuration tasks. Time and detail is spent on other more important considerations such as logging, disk performance, test planning and file systems.
Chapter 1 gets us started with an overview of tuning in general. One of the more important themes of the book is established up front: It is that a cost/benefits analysis should be done for each step.
Sections 1.5, "Tuning Isn't Always Necessary" and 1.6, "Not So Fast ..." establish baseline considerations for making a decision on what, if any, tuning efforts should be made. Throughout the book Christenson reminds the reader to decide on a path that is most cost effective for his organization. Is it the most cost-effective use of the company money and time to have the IT staff hand-craft solutions, or is it better to throw some hardware at the problem?
Let's face it -- if you are running sendmail for a small company and only move a few hundred emails a week, how much performance tuning do you need? If, however, you are running an ISP, a mailing list server or a medium- (or even a large-) sized company mail server, then you need to tune your mail server. This is the book for you. The information in this book, while oriented for sendmail, is actually applicable for tuning any Unix based Mail Transport Agent (MTA) server.
Each solution is an individual matter, that is wholly dependent on several factors. Some of these factors are: volume of email, what the main use of the email system is, how the end users interact with the system, what hardware is being used, how much bandwidth you have and much time and money you have to throw at the problem. Of course, what management considers important is the overriding factor in all decisions.
Chapter 2 is a ten-page introductory overview to sendmail, covering versions, obtaining the (Open Source) code and building sendmail. One of the important things covered is the queue and message spool layouts and permissions. This is helpful for making sense of things later in the book.
A few very important pages are spent on creating the .cf file and why you should use M4, the macro language, for managing the configuration files. Having hand-crafted a .cf file myself several years ago then having to deal with maintaining it, I can vouch for the wisdom of using M4.
The maintainers of sendmail update the M4 macros for new features and changes. The 200 - 300 lines of M4 macro files are converted into a 1500to 2000 line configuration file. While it may be easier to figure out the configuration file to make changes, those changes may not be valid from one version of sendmail to the next.
Christenson admits that he does not always use M4 when in a rush or to test some things. What he does is copy the working configuration file to sendmail.cf.REAL before making changes and updating the .mc file afterward.
The next chapter, Chapter 3 'Tuning Email Relaying' starts with an overview of the email relaying sequence. Most of the discussion in this chapter is not sendmail specific. The importance of data synchronization is emphasized here. Section 6.1 of RFC 2821 is quoted, where it states the email server 'MUST NOT lose the message'. Once that is stated and understood all of the requirements that are discussed in the rest of the chapter are clear.
The next 17 pages are spent discussing how file systems, networking and effective use of file space support, and can detract from, meeting the edict of the RFC. For me, these sections are some of the most interesting, filled with information presented in a concise, readable and detailed manner. I learned a lot about what impact some very basic decisions have on email performance. I also learned how much better I could have made the email servers I have worked on. I will be turning to this chapter the next time I am putting together a box, be it a web server, mail server, file server or even a workstation or laptop. This is a great chapter that can help with any system configuration.
Email reception is covered in chapter 4. Different strategies for verifying recipient, tuning POP and IMAP are covered. Effective use of Local Delivery Agents (LDAs), including procmail, is covered here also.
Additionally, an excellent discussion of storage systems, including disks and solid-state disks, is in this chapter. The sections covering RAID levels, benchmarking and use are well written and informative. Available options on drives, ATA vs. SCSI, Solid State, are just as well done. Section 4.4.2, 'Stupid Disk Drive Tricks,' has some nifty information about how to set up disk drives for even better performance.
Sending email is the next chapter. Here we see some more sendmail-specific information. Tuning of mailing lists and mass mailing is part of the discussion here. One of the more important sections is 'Draining Queues.' How backups are caused and what to do to recover are discussed in this section. This has good information that can be used with any MTA.
One of the most important chapters for me is Chapter 6 'Configuration, Security and Architecture.' Sendmail specific configuration and tuning options are discussed. Section 6.1 covers configuration and is in many ways the heart of the book. This is where sendmail directives that can directly impact performance are covered. How a system's architecture (and DNS) is laid out can have a significant impact on performance.
The section on security is good, but brief. Most of the discussion is on privacy and stopping spam. The use of Transport Layer Security (TLS) is covered in less than a page. With a smattering of security in other parts of the book, this is the sum total of the security discussion of the book. Considering some of the problems with sendmail in the past, I would have liked some more information on this topic.
The next two chapters, 'Finding and Removing Bottlenecks' and 'Load Generation and Testing,' are good, solid, well-written sections that are applicable to most any email system. We are given some effective ways of making systems run better, and how to prove that the system actually does work better. This is where we justify to the boss that the work we have done is really cost effective.
Chapter 9, 'Conclusion' is basically a very brief wrap up and a list of books Christenson thinks we should read.
My own conclusion is that I have learned and relearned a lot of things about Unix and email. Even if you do not use sendmail, I recommend this book without reservation. It is an excellent reference on general system performance tuning, with information on making your sendmail installation run better.
Table of contents
- Introduction
- 1.1 Performance Tuning Examples
- 1.2 sendmail Versions Covered
- 1.3 Definitions
- 1.4 Email Server Tasks
- 1.5 Tuning Isn't Always Necessary
- 1.6 Not So Fast...
- 1.7 Email System Profiling
- 1.8 General Tuning Ideas
- 1.9 Summary
- Sendmail Introduction
- 2.1 Obtaining Sendmail
- 2.2 Building Sendmail
- 2.3 Creating a .cf file
- 2.4 Why Use M4?
- 2.5 System Setup
- 2.6 Summary
- Tuning Email Relaying
- 3.1 What Happens During Relaying
- 3.2 Synchronization
- 3.3 File Systems
- 3.4 File Space
- 3.5 Networking
- 3.6 Summary
- Tuning Email Reception
- 4.1 What Happens During Email Reception
- 4.2 Recipient Verification
- 4.3 Storage Systems
- 4.4 Disks
- 4.5 Solid State Disks
- 4.6 POP Tuning Specifics
- 4.7 Message Storage Hashing
- 4.8 IMAP Tuning Specifics
- 4.9 Summary
- Tuning Email Sending
- 5.1 Mailing Lists
- 5.2 Command-Line Message Generation
- 5.3 Draining Queues
- 5.4 Another Mailing List Strategy
- 5.5 SMTP PIPELINING
- 5.6 More Notes on Mass Mailing
- 5.7 Summary
- Configuration, Security and Architecture
- 6.1 Configuration
- 6.2 Security and Performance
- 6.3 Other General Strategies
- 6.4 Summary
- Finding and Removing Bottlenecks
- 7.1 Kernel Parameters Run Amok
- 7.2 The Quick Fix
- 7.3 Tools
- 7.4 syslog
- 7.5 Removing Bottlenecks
- 7.6 Summary
- Load Generation and Testing
- 8.1 Test System Setup
- 8.2 Testing Tools
- 8.3 Load Testing Pitfalls
- 8.4 Summary
- Conclusion
You can purchase Sendmail Performance Tuning from bn.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
I will buy this book, and file it under "oxymorons I have known and loved"
Invoicing, Time Tracking, Reporting
Because no self respecting geek has ever taken the easy road. This coming after I just wrote my first sendmail configuration file from scratch. It's a right of passage.
(B) + (D) + (B) + (D) = (K) + (&)
i have seen many books on apache and sendmail. and i have quite a few of them. i rarely consult them. when i need to find out anything, i m most like to type it into google, or search newsgroup (deja). this includes apache/tomcat performance tuning.
I have administered NIS+ apache, and sendmail, and these topics are so vast, that buying books doesn't help. However searching through google/deja does help a lot.
But that is just my opinion....
Consensus is good, but informed dictatorship is better
i've always started by openning up a modem connection, picking up the phone, sneezing into the receiver and capturing the output as my sendmail.conf
... hi bingo
Tuning Sedmail is about as smart as tuning MS Exchange. Both are fat, bloated, and have a history of secutiry holes. Performance tuning Sendmail is like performance tuning an AMC Gremlin made out of spare junkyard parts - you end up with a cobbeled peice of crap with a spoiler.
Sendmail is so bloated that it apparently takes books to get decent performance out of it.
Sendmail doesen't fit the one of the core Unix ideas - use simple, robut and elegent programs, chained together, to do cool things.
Check out Postfix or Qmail for decent replacements to Sendmail. Enjoy the performance of well toughtout software. Postfix was so easy to use, that I went from reading the man pages to a working system in one hour.
Moneyed corporations, non-working 'poor' and criminal prisoners are turning productive citizens into tax-slaves.
I'm trying to remember who said this:
This is not my sandwich.
I'd figure a Spoiler, a few 'Type-R' stickers, a coffeecan exhaust, and a bolt on Turbo kit should do the job. Who needs ruddy books?
Non impediti ratione cogitationus.
.. always needs tuning. The strings need replacing, the wood frame warps in the humidity. It's a real bitch.
Trolling is a art,
One is Free Software, and one is not. That's important to some of us.
Dewey, what part of this looks like authorities should be involved?
Seriously people ... if you don't like sendmail, don't come in and bash it on a sendmail book review (btw: all of those posts SHOULD be modded off-topic).
... but nothing can ever replace "the real thing" ....
... and if you don't agree with me, be intelligent and post why ... don't moderate!
... preparing to dodge all of the "clippies" that will be thrown at me by the NT admins ... "
If you like another mail program, DO A BOOK REVIEW ON THAT MAIL PROGRAM and share your "wisdom" with the rest of us.
As for sendmail, it is not meant for Windows weenies or wanna be Unix sysadmins, only for "real" Unix admins and those that wish to use a VERY robust product and are willing to learn the product to reap the rewards. I personally feel that this book is a great quick reference for when you have a brain fart and don't feel like mulling through the O'Reiley "bat-book" for a simple answer
Just my $0.02 cents
HallmarkOrnaments.Com
Since Sendmail hasn't been monolithic for a couple of versions now, have you considered switching back?
Dewey, what part of this looks like authorities should be involved?
Yes, I've been using psotfix for almost 2 years now. It's a heck of a lot easier to configure than sendmail for sure (and secure too)!
Karma: The shiznight, mostly because I am the Drizzle.
Us Debian weenies have exim installed by default. This Debian weenie does prefer to "apt-get install postfix". What is this Sendmail thing that everyone keeps talking about?
Bah!
Then use Courier MTA or MTA+IMAP server, which is same maildir compatible as qmail, fast and beside: it's GPL - what can be more free than GPL?
Less is more !
Bruce
Bruce Perens.
No reason. I use qmail since I had a look at sendmail configuration 2 years ago and turned away in horror from that configuration interface nightmare.
Although many others in my department, including the main mail server, use postfix.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted and ignored otherwise.
My former employer, AT&T Canada, who has over 150,000 home users and several thousand business
clients, dumped their Sendmail-on-Solaris servers in favor of Postfix-on-FreeBSD and have never been happier.
Pain is merely failure leaving the body
Us Debian weenies have exim installed by default. This Debian weenie does prefer to "apt-get install postfix". What is this Sendmail thing that everyone keeps talking about?
LOL! A pity I just wasted my last moderator point on something not half as funny some minutes ago!
Qmail is a little more difficult with Debain, but an qmail-src package exists and works for me.
Most ACs are not even worth the keystrokes to insult them. Be generically insulted and ignored otherwise.
I'm still partial to Sendmail (probably because I'm used to it by now), but that would definitely be my second choice (for technical and license reasons).
Dewey, what part of this looks like authorities should be involved?
Comment removed based on user account deletion
And when was the last serious security problem in sendmail?
There are a lot of wannabes going around repeating old news
and thinking they're hip.
Ben "You have your mind on computers, it seems."
Comment removed based on user account deletion
I have to disagree.
.cf file, the flexibility, and tweakability of sendmail is astounding. Not everyone needs it, but when you do, sendmail rocks.
Sendmail kicks ass.
In a former life, I had to build a mailing list manager that handled content generation, sub/unsub, bounce management, etc. for a large number of mailing lists that had to do about 1.3 million messages in 4 hours.
Without going through the whole set up, there were slave boxes that just delivered mail. They used pieces of postfix, which did a good job (I like postfix, BTW, my current company's primary mail server uses it).
The primary machine used sendmail. Once you get over the horror of writing a
A lot of people who don't really know what they're talking about rag on sendmail, echoing some very valid complaints that are mostly only of historical interest now. The most valid complaint these days is that it is arcane to configure. My take is, sure, it is, as it should be. Handling large volumes of email is second only to nntp for placing a heavy load on all sections of a network. If you don't know what you're doing, you shouldn't be doing it.
I use postfix because it is easy now, we're only handling a few thousand messages a day.Should that change, I'm back with my old pal sendmail.
I forget what 8 was for.
Sendmail is an MTA (Mail Transfer Agent) and thus does not need or want to speak POP3. In Red Hat (at least, versions up to 7.3) you need the ipop3d daemon for pop3.
/etc/mail/sendmail.mc, which contains useful comments. This is not a bug - MTAs that relay without requiring any user configuration are BROKEN.
However, all POP protocols are brain-dead and lame. You should be using IMAP, which is supported by all decent end-user mailers (MUAs, or Mail User Agents) as well as by Microsoft's latest MUAs.
In v5-7.3 RedHats, type ntsysv from the command line and turn on IMAP (or, better yet, IMAPS, if you comprendhez crypto) and SENDMAIL in the runlevel of your preference (I recommend 3, since Xwindows is bloated and unreliable).
If you want an exchange server clone (NOTE: only really useful if you run Outlook, the world's absolute worst MUA for security and reliability, which runs only on proprietary opsystems that have high hardware requirements) use HP Openmail. Or wait for Miguel de Icaza to write something better.
One thing to remember, regardless of what MTA you use, it should NOT relay Email from any address other than 127.0.0.1 unless you specifically configure in the addresses/domains you want to relay from. So you will need to edit
I ran 400 users on redhat5.2+sendmail+ipop3d server and win98+pegasus clients, the clients popped the mail server every 7 seconds, and the server was a pentium 133 with four ethernet cards on it. No performance problems in our real-world shop.
For my users, reliability + performance = ability to generate profit = paychecks for workers. Features that cannot be obtained without sacrificing reliability or price/performance will not be implemented, because the goal is to earn enough to feed the kiddies, not to be 1337.
All that being said, I recommend you install Postfix rather than sendmail. I use sendmail because I already know how to do it, and it works with OpenLDAP. I don't think you fit the profile.
Sendmail is BSD-licensed. From a corporate perspective, it's even freer.
Dewey, what part of this looks like authorities should be involved?
Sendmail's default configuration is very conservative with respect to both your machine resources, and the resources of those with whom you exchange mail.
Run sendmail with
(or something bigger, if you want) and you can get past the biggest hurdle in sendmail performance -- by default, sendmail will handle all outgoing messages in a single queue. (Both qmail and postfix use multiple queues and/or runners by default.)Sendmail can relay messages without doing any disk I/O, thanks to the asynchronous I/O libraries; coupled with multiple queue runners, it's hard to get faster performance out of any MTA.
The fact that you cannot distribute modified versions of it pretty much moves it into the non-Free camp.
Dewey, what part of this looks like authorities should be involved?
You mean, like, a version that installs binaries in /var? Oh, wait...
I really don't understand the non-free label attached to Qmail by some people.
You can't change it and restribute the modified version. Without that ability, how is it significantly different than the "freeware" that floated around on BBSes in the 80s and 90s?
Just because something is GNU/Free doesn't mean it's good, and something that doesn't meet the GNU definition of "free" isn't necessarily bad.
First, I never said a thing about GNU - I tend toward BSD, personally. Second, I made no judgement of its relative goodness or badness, which is entirely different from declaring it Free or non-Free. It may very well be a great program, but if it's not Free, then it's not useful for me in my applications.
Dewey, what part of this looks like authorities should be involved?
Yep. Sendmail isn't trendy or cool, so it's popular to talk about how terrible it is, regardless of whether the common complaints are still (or ever were) valid.
Dewey, what part of this looks like authorities should be involved?
Nick's book is very good, and he has managed to "give away" a lot of the tricks of the trade that are used by Sendmail, Inc. Professional Services.
.cf directly. There be dragons there. Use M4.
Sendmail (the app, not the company) scales quite nicely if you know what you are doing - I've installed sendmail on a couple of old dual-proc HP's and they handle about 2 million messages a day. Yes, that's right; 2 MILLION. It's a "simple" matter of tuning and knowing the file system.
Don't hack the
And, if another tool works better for you (exim, postfix, qmail) - use it! I don't always recommend sendmail, either.
Bobcat
disclaimer: I used to work for Sendmail, Inc. - my cubicle/cage was about 40 feet from where Nick used to sit...
-- Ziggy Sig Sig
Because qmail doesn't speak LDAP?
In FreeBSD, there exists an /etc/mail/Makefile. After you edit the .mc file, you just "make install; make restart". Yes, you still have to process the config file, but at least it's 100% automated.
Dewey, what part of this looks like authorities should be involved?
...racing stripes
If your SMTP server has to route mails to different mail servers depending on the recipient address, LDAP support is very useful. You can also do things like move your alias tables into the LDAP server, use the LDAP server to authenticate people for routing (SMTP AUTH), use the LDAP server to masquerade senders etc etc etc.
LDAP is supported out of the box with sendmail, no need to download extra patches and stuff.
All I want is a secure system where it's easy to do anything I want. Is that too much to ask ~~ Randall Munroe
These days, Red Hat's default install of sendmail does not accept remote connections. Red Hat tells you to use M4 to generate a new CF file to enable this functionality.
Here is what you really do...
Find the line in the CF file that reads:
O DaemonPortOptions=Name=MTA
and change it to:
O DaemonPortOptions=Port=smtp,Addr=0.0.0.0, Name=MTA
I picked this up from a bugzilla comment.
You have no idea what you are talking about.
What you describe is sendmail as it existed about 5 years ago. Any reasonably intelligent sysadm could get a simple sendmail config downloaded from the distro site and running in a couple of hours.
The security problems are largely a thing of the past too. Apart from a couple of minor obscure and possibly unexploitable problems (and that embarrassing trojan in the build process), there have been no security problems with sendmail for years.
Sendmail might not be the fastest, but how fast does your mail server have to be? I was able to get eight messages / second out of my G4 powerbook without any tuning whatsoever. This is not really a lot, but otoh represents nearly 30,000 messages per hour. How many do you get every day?
All I want is a secure system where it's easy to do anything I want. Is that too much to ask ~~ Randall Munroe
You can run dictionary attacks against domains I handle mail for, but at least it will take you a damn long time! I just wish everybody was doing it. Eventually it would take too long to spam effectively.
...to repeat it.
Um, wait. That didn't come out right.
Sendmail is so bloated that it apparently takes books to get decent performance out of it.
Seriously, a lot of people just don't understand sendmail and how difficult mail routing is. Sendmail is bloated because it tries to solve a very complex problem in an extremely diverse environment.
Postfix was so easy to use, that I went from reading the man pages to a working system in one hour.
Postfix and sendmail are going in two different directions. Sendmail == configurability, postfix == ease of use.
Sendmail is fast. Not because there's a book on performance tuning does it mean the software is slow. Sendmail is probably as fast as anything else out there.
Sendmail is configurable to no end. I run a sendmail setup with virtual domains, lots of address rewrite rules, spam filters. All my virtual users, and alias maps are in LDAP. I modify all my sendmail virtual users, virtual domains, aliases, etc through a PHP website. Try that with other mailers. They support LDAP, but they are no where as configurable as sendmail.
Then try running these mailers on Linux, bsd, solaris, windows, AIX, etc., etc. No OS lock-in.
Sendmail is well documented.
The prevaling mood on /. these days seems to be if it's not brand new, get rid of it. That's unfortunate.
Based on upvotes, Ageism is the only "-ism" Slashdotters care about and think isn't SJW
Yeah, I switched us to Courier recently.
I was running qmail, but then needed an IMAP server. Courier-IMAP was the best for Maildirs. Then I needed a web mail server. Sqwebmail (part of Courier) was the best for Maildirs, and used the same subfolder format.
Then I needed mail filtering, and wanted to use Maildrop because it works well with Maildirs. So I just bit the bullet and installed Courier.
Had some initial issues with configuration setup, but after that it's run pretty smooth. Now I just need to get SpamAssassin installed (need a new version of Perl on the mailserver).
Sendmail scared me off in the mid '90s. Haven't touched a .cf file since.
How does it make it freer? Do you think that any corporation wants to fork the source code of MTA server and resell? They wanna use it, not fork it. Besides, you still can fork GPL code, just don't forget about source code availability if you resell it. But again, who wants to resell any MTA server without sources?
Less is more !
It would be a reasonable business proposition to build a GUI-controlled mailserver appliance powered by Sendmail, with all sorts of proprietary backend stuff.
Dewey, what part of this looks like authorities should be involved?
Better yet, go with a distro that doesn't foist Sendmail on you by default...you can then do something like this:
# emerge qmail
20 January 2017: the End of an Error.
The mere fact that you have to link to a web site called "lifewithqmail" to point out a patch to a 5-year-old MTA to incorporate even basic LDAP functionality highlights why I use Sendmail.
Sendmail's not NEARLY the problem it used to be. It's a whole lot better than ever before, and it works out of the can without Dan Bernstein's mess all over it.
Philosophically, I couldn't use software written by such a jerk anyway.
absolutely. I guess the parent poster hasn't even run a *REAL* mail server where quantity_of_users > 5.
--NBVB
Because while Qmail works great, it's just plain weird. It uses a bizarre directory structure, requires several local users and is quite simply an obnoxious program to deal with.
Conformity is the jailer of freedom and enemy of growth. -JFK
> and is quite simply an obnoxious program to deal with. ...and then there's the author...
dave
Sendmail as Rice-Boy -- what a concept :)
Esli epei etot cumprenan, shris soa Sfaha.
I've always pharted my sendmail.cf into the handset...