Slashdot Mirror
E-Mail Server Setup Advice?
dhammala asks: "I am responsible for setting up and maintaining a mail server for small web-hosting type business. We currently host about 75 domains, around 100 mailboxes and due to the efforts of our sales team, we are wanting to get ready for some great increases in those numbers. I am worried about my current configuration and ease of administration. More importantly (well, at least to the customers) is email deliverability -- it seems that messages delivered to some big players are being marked as SPAM or disappearing altogether. I am asking the Slashdot community for it's insight and advise on 1) if my current choice of software/configuration is a good match for this situation and 2) if there any additional measures I might take to ensure email deliverability?"
"Here is an overview of our current setup:
I have not yet been able to get DomainKeys to work with Postfix. It was during my configuration attempts that I started to question this setup and wondered if this was the best setup for our situation.. this inquiry has lead to this posting.
In a perfect world, I would have an email server that:
Are there any other technologies or configurations that I need to implement to support the best deliverabilty rates?"
- We lease servers at ev1servers.net.
- The servers are running RHEL ES3.
- We chose to use Postfix and have it configured to support virtual users and domains mapped in MySQL tables. The reference I used to configure this setup is located here. We initially chose Postfix over qmail because it was open and over sendmail because the config files are actually readable.
- I have added in SQLGrey grey-listing for Postfix to provide a simple level of SPAM detection for our users. We are not wanting to deal with the customer service and higher box loads of mail scanning at this time. We might choose to use a 3rd party vendor to do this as needed.
- Messages are delivered locally via maildrop in maildir format.
- Courier IMAP is running to support both IMAP and POP access to the mailboxes.
- Postfix Admin was setup for easy mailbox administration.
- I have verified that our reverse IP records are correct
- I have created SPF records for all of the domains
- I have verified that our server is not listed in any blacklists (great scanner at dnsstuff.com)
- I have started to install DomainKeys for Postfix
I have not yet been able to get DomainKeys to work with Postfix. It was during my configuration attempts that I started to question this setup and wondered if this was the best setup for our situation.. this inquiry has lead to this posting.
In a perfect world, I would have an email server that:
- is easy to administer,
- supports automated mailbox setup/removal (currently I can just insert rows into my tables and the mailbox setup is done)
- supports current technologies, like grey-listing, DomainKeys, etc
- is secure
- makes the best use of system resources -- I want to get the 'best bang for the buck'
Are there any other technologies or configurations that I need to implement to support the best deliverabilty rates?"
It's not free, but great support and full everything right out of the box, including IMAP, POP, SMTP, HTTP, authentication, account management, quotas and everything else you could possibly want.
v r/home_messaging.xml
If your company can't afford it, that sucks, but I'd rather use that than try and get courier, postfix, pop3d and squirrellmail or whatever to all work together.
http://www.sun.com/software/products/messaging_sr
I recommend setting up ClamAv with FreshClam to filter out virus/worm type email. I have found it performs very well on my server. I have also found they have a very fast responce to new viri as they appear.
http://www.clamav.net/
postini. its more expensive, but its worth EVERY CENT.
evil too. they have a free trial, and the filtering is of such high quality youll be hooked.
You do realize that when something breaks or is misconfigured or *you* do something wrong and it affects your coworkers or your customers - you will take full blame, right? You're not going to have th ability to call one person that will support you and fix you and take the blame for anything that went wrong?
Not only that, but with so many different groups involved in your email solution, just getting good support even in the community will be hard to do. It's not like there's any QA done between X and Y and Z on A and B hardware and C environments.
Not to mention, nobody is obligated to help you out in a certain timeline (say, you may or may not ever get an answer rather than getting an answer and help from someone live within an hour).
Sometimes you get what you pay for. And in big-business email centers - it's such a case.
Personally, I use postfix and qpopper and spamassassin, but I'm also not providing email for a hundred or a few thousand customers.
Please do my work for me.
Sincerely, Slashdot reader.
Karma: Positive (probably because of superiour intellect)
So, I have a mail system setup, it's running around 70 domains, and 500 email accounts.
I am using courier as the mta, and courier as the pop, and courier as the imap.
The courier makes a fine MTA, but you do have to tweak a few of it's settings to make it more wideopen to allow it to connect to misconfigured exchange servers.
Other than that it has been great. I have a email account management system that I wrote that lets each domain have admin users that can add and delete accounts as they please.
I have SpamAssassin setup for some users (most of them post their email addresses in plain text on their websites) and even with that load, there is still plenty more capability in this little server.
Now, if you want a system that scales to tens of thousands of users, you are going to need to get something a little bigger than this, you are going to need to get a mail system that can distribute the messages over a number of servers. That is something I have not researched.
I would have done it myself but you, a MENSA babe, had already done it.
Courier IMAP is running to support both IMAP and POP access to the mailboxes.
I would switch to dovecot. I found the performance to be quite a bit better than Courier, and it seemed more stable as well.
I support email servers for a living. I have for almost seven years - exclusively on Solaris, AIX and NT (though I do so on linux for my personal use).
While I think that your deployment is a reasonably sane one - as far as going the OSS/free route is concerned - I agree with the other poster here who said that having nobody to blame will be an issue in the future. When your job is on the line, it's good to have someone else who is supposed to know and fix everything for you when you are hard-up for solutions. Email administrators for the largest and biggest corporations in the world don't do it all in-house. Even they contract out for support for their enterprise level products. Because their customers and bosses expect great reliability and performance and features and they don't want to wait for several days (or longer) while you read some half-assed documentation on a website, chat up some gurus in IRC and post to some web forums and usenet groups hoping for help.
Also, there is nobody certifying that the products you are using will absolutely work together. And on whatever platform you're using. They may say they've tried it on it - but I doubt in many cases they will say it's been certified through a thorough internal QA process that weeds out a lot of bugs and such.
Also, when you really must have something fixed, you will either have to write the code yourself (laborious to do, without even talking about testing and implementing). If you have a commercial product and a contract, you can present a business case to get your issue some priority and have a fix. And you can always threaten to drop the product if they don't do what you want (it works more often than you'd think).
Even when full-fledged, thorough, all-encompassing high-capacity commercial servers - the position of email admin is a full time job for at least one or more people. Using a dozen different open source products and maintaining everything and keeping a constant sandbox environment to work in (you don't want to introduce upgrades or patches or changes on production, of course!) will consume all of your available time. If you are the full-time email admin here and that is your only responsibility - have at it. But if you have other responsibilities... I think the commercial path might be better for you.
Again - I'm an OSS advocate. Yet, I feel strongly that there are some cases in which commercial software and support is valuable. Depending on the specifics of your duties and position, this may or may not apply to you. But consider it. Especially if you're going to be fairly huge some day.
Another solution would be to contract with a third party. There are companies that do nothing but provide you with email solutions. They can do this based on very strong commercial products. These companies themselves will host and run the hardware for you. They will do all of the configuration and deployment and maintenance and administration for you. I'm not familiar with their prices, though - but do look into it. The upgrades and crashes and migrations are their responsibility. Meeting QOS is their responsibility. They will deal with the commercial mailserver vendor(s) for you. They already have support contracts with them. All you do is tell them how big of a deployment you want and you're set.
After working with commercial mailservers for several years, I was ready to setup a deployment of my own for my own personal project. Not having any funds, I decided I was going to go the OSS rout. Just figuring out what would work together and what wouldn't (you have to make sure your POP, IMAP and webmail servers all use the same mailbox formats. You have a gazillion options for accounting from LDAP to MySQL, countless authentication mechanisms, etc). It drove me nuts. It was at that point that I started to see the light and the real value in what I did with commercial products. Having an entire server that supports everything you could possibly need or want in an email solution through one install and one configur
Courier IMAP is pretty poor. UW-IMAP is a piece of crap. Binc might be good now (it has the right goals), but was too immature when I looked at it.
Switch to Dovecot. Also, if you haven't already, switch to Maildir for your storage format. The mbox format is a disaster when dealing with IMAP clients like Apple's Mail, which opens multiple folders at once, thereby locking them all and blocking mail delivery.
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
It is easy to use, can be expanded to cluster servers and is reliable.
Works with squirellmail, and a bunch of other cool features. Plus the name of the company is kind of cool.
http://www.stalker.com/content/solutions.htm
I read Slashdot for the headlines, because the headlines, unlike the articles, are usually original and never duplicated
If I were you, I would ask myself at least this:
"If the building the server lives in falls into the center of the earth, but my boss wants the mail back up (not necessarily with their data, just live again), would I be able to put Postfix, SQLGrey, LDAP auth and Courier back together in less than 4 hours except for user accounts?"
If you are sufficiently detailed enough to pull that off within 4 hours except for user accounts, you probably have the bits you need to wing all the rest of the bells and whistles (webmail, MAPI integration, upgrading one piece and making sure the other 11 don't die, etc.) which eat up your time.
The main problem with going commercial (which you should consider if you're sure it'd take you at least 8 or 9 hours of a day to put this together from zero) is that the extra features (Outlook integration via MAPI, etc.) tend to cost.
OTOH, if you go commercial the extra features are not yet another thing you need to spend time on because presumably you're paying for parts that already work.
Choosing a path starts with your honest appraisal of your own skills.
(Personally, I find Postfix + SQLGrey + Postfix virtual delivery agent to rock. But I don't mind going into the guts to add a new user or set a quota on an abuser or whatever. QMail I've found to be a tad dangerous. Often the slightest typo either puts qmail-send in a mad CPU loop (which is annoying) or delivers it in the wrong place (which is really annoying) or 5xx's it (bad). Adding extra steps into the qmail handling pipeline is a nightmare and should never be done on a production box first. But maybe I suck.)
"The Devil does not know a lot because He's the Devil, He knows a lot because he's old." -- unknown
Check out inter7.com
They use Qmail which is open source. Who told you it was not?
Qmail is highly scalable and I think www.qmailtoaster.com and a few other sites provide great setups that allow you to set quotas and such.
large game sites use qmail.
Hell hotmail.com uses qmail to send emails. Not sure about the rest of it.
Inter7 can get you setup properly and provide maintenance if you have problems but otherwise their setups are self manageable.
Then, assuming you know how to write PHP code, throw away the php. It's not that good. It can't handle fields being added to the database. But writing php for database manipulation is trivial, so I'll assume that's what you're already doing.
Anyway, what you need from 'vmail' is the 'maintain' perl script. It's fairly easy to understand. Basically, you want a 'new' table in your database with new email addresses, a 'deleted' one for deleted addresses, and a 'moved' one for moved emails. So in addition to editing your main mail table, you also put email addresses in those tables when those things need to happen.
The script pulls these out of each table and does the things. It doesn't mess with your main table at all.
Now, the perl script needs to turn that email address into a directory. It starts out as hostname.dom/u/user/ from user@hostname.dom, but you can change that however you want with a bit of perl study.
Or, instead of putting the email address in the tables, you can just put the directories, and tell it not to try to make that into a directory at all. That's probably simplier if you already have the rest of the system set up with other pathnames, and you don't know perl.
I experimented with cgi scripts and whatnot, but this was much easier. You can either put in cron to run every minute or so, or you if it is important updates happen instantly, you can make it suid and run it from a cgi script or from php.
As an added bonus, that script is so nicely written you can make other tables and make other things happen. It's a nice way to keep restrictive permissions on your webserver, but have nice, protected php pages that can make 'requests' that get executed at certain times.
I have a copy of it that lets people change users listed in .htaccess files, although I don't currently have an interface to it. And I have one that will create apache config files and empty directories with the right permissions, and then restarts the web server, for when domains get added.
If corporations are people, aren't stockholders guilty of slavery?
We initially chose Postfix over qmail because it was open
Who told you this little chestnut? Are you planning on redistributing qmail? No? Then you don't even need a license (which is good because it doesn't come with one ;-).
qmail is just as "open" as postfix if you're just using it, and has the advantage of being simpler and guaranteed secure.
I have set up qmail on several BSD boxes and some of those boxes have not been touched (rebooted, upgraded, hacked, whatever) in *over 2 years* (verified with Tripwire). Try that with windows sometime...
I have nothing against postfix, mind you, I use it on my home machines and in certain situations that need flexible setups, but qmail is great for "set up and forget" situations.
I'm surprised nobody has mentioned dbmail yet. It's a nice prepackaged substitute for maildir/courier/mysql virtual users, etc... It's not a substitute for everything you require, but it will simplify your most common admin tasks.
If you do something that's going to drastically alter your server's behavior, do 'inet_interfaces=localhost' to test, then restore to 'all' when you're sure it's working.
This sig no verb.
... after hosting using Exim3 and Exim4, Postfix, and Sendmail... if i were doing a "Large" config again (read 1000+ domains, 30k+ accounts) I wouldn't consider anything *but* sendmail. It's not the easiest, newest, or anything like that, but it does scale extremely well. The setup I'm currently using (about 10 domains, 70ish accounts) is:
/16 netblocks of ip addresses to catch a single spammer... some of the standard rbl's are nutzo.)
Exim4 SMTP
Dovecott IMAP and POP3
Bogofilter
Spamassassin (SA-Exim)
Clam-AV
It's a rocking system, I'm currently having about 18000 messages a day tossed at me of which about ~17000 are spam. My personal accounts were getting about 2500 spam/day until I enabled all the anti-spam software and virus removal. I now get about 1-2 Spam a day and I've not had a single false positive.
For a small mid range setup I would probally use exim4. It's simple, has great features, and it's nice to have spamassassin at smtp time instead of having to process the entire message.
I don't recommend standard RBL's, however, the URI RBL's are *extremely* effective and an order of magnatude more sane in what they block (eg: if the message contains a link to viagraforyou.com it blocks the message, rather than blocking random dsl servers and
Theres a nice tutorial and informational link about using all the good features of sendmail and several additional ideas and theories on what is effective and what isn't at http://acme.com/mail_filtering/ the guy gets *insane* quantities of mail (mostly spam) and tells how he deals with it.
Synopsis: Large site- Sendmail, Medium/Small Site- Exim4.
Alot of people like qmail and postfix over sendmail and exim, but I just don't care for them having used them. Although if forced to choose between postfix and qmail it would be qmail.
Shadus
I *LOVE* Dovecot!
If only it did shared mailboxes, it'd be perfect.
I would suggest setting up a number of outgoing relays. Group your customers into tiers of trustworthiness. Everyone goes on server 1 to begin with. Anyone who behaves for 3 months gets moved up to server 2. Anyone who behaves for 3 months there gets moved up to server 3. Anyone who misbehaves gets moved back down to server 1, and anyone who continues to misbehave on server 1 gets disconnected. This ensures that your non-spamming customers end up with more reliable delivery.
I'm a courier user myself, but I was talking with one of my co-workers about IMAP servers and he said that Cyrus - though a bit more of a pain-in-the-ass to configure - was generally more efficient under scenarios with a high userbase.
I can't recommend it from experience, but I would trust the advice of this particular individual.
For smaller userbase, I'd have to say that courier was pretty painless to configure and reliable as well.
... Microsoft Exchange Server 2003 Enterprise Platinum "This One Is Really Secure" Edition. Nothing like good old Microsoft products.
Sig: I stole this sig.
I really wouldn't worry about it. Hardly anyone uses it to block email since he states on his site that it has a lot of colatteral blocking.
Hey, you want job security, right?
Write your own mail server software, preferably in an unpleasantly horrible language, such as Threaded Intercal. Make sure it keeps all the mail and account information in something inherntly tied to the implementation language, such as stored procedures, disk-based monads, persistent lexical closures, or the like.
Did I mention the part about not supporting POP3 or IMAP, but rolling your own protocol and client? You wouldn't want some hotshot hiree coming along, extracting all the mail that easily, and moving the company over to Exim or Postfix.
Oh, and you want there to be a lot of resistence to moving away from your solution, so make it do something executives will like, such as have the server authenticate clients by MAC address so they don't have to have passwords.
Also, just to raise the bar for potential replacement systems, roll in some features that have nothing whatsoever to do with mail. For instance, you could tie the mail server into the company accounting system and put user interface in the client for viewing up-to-the-minute charts showing revenues, remaining fund levels in various funds, and so forth. Arrange it so that users can send each other these charts (actually just magic tokens that pull them up) by email.
Cut that out, or I will ship you to Norilsk in a box.
You might want to check out QmailToaster. It's free, supports multiple domains, has a web interface, and has SPF and ClamAV integration.
I own a small hosting company. I have setup my business so that all accounts (except shell accounts) are stored and authenticated against MYSQL databases.
For that reason I chose Cyrus as the actual local mail system. It supports IMAP / POP3 can be scaled pretty easily. And despite reports that it is hard to configure, I have found that it really is not too bad if you keep things simple.
Currently I host about 3000 domains, and roughly 5000 email accounts, though most are nothing more than SPAM traps.
If you do go this route, the key is a reliable and robust MYSQL server(s).
The main advantage of MYSQL based virtual acounts is web-based management is trivial. ADD / UPDATE / DELETE can be done simply by updating a record.
The draw backs I have found are: a database/DB Server is an additional point of failure. Replication has been a bit tricky at times. Do not run DSPAM in the same database as your user / hosting accounts.
-MS2k
You might want to ask your question also at the forums at emaildiscussions.com. There is a subforum there for "setting up an email service" and there are several active participants that are email admins running operations like yours or bigger (or smaller) that can give you good advice.
and so does qmailrocks.org
that site is a very complete step by step, and you can stear away from the sendmail/postfix nastiness..
qmail in my professional opinion, runs the fastest, and is the most secure.
Considering it's the most insecure piece of software on the internet, well next to BIND DNS anyway...
Why would you recommend it to anyone?
95% of your customers would probably be happiest with addresses forwarded to GMail accounts that are configured to put the forwarded address in the From: field. Think about it.