Slashdot Mirror

Proprietary file formats, databases, and such... But outside of Unix no one came up with a standardized mail format.

Thunderbird uses the cross-platform MBOX standard. I've even been in a triple-boot scenario where the Linux, OS X, and Windows Thunderbird clients could all access the same network share that held the email. It was great! Here is a list of other standards.

I use Claws Mail. It's light on resources, fast, stable, and can deal with gigabyte-sized mailboxes without a hiccup. Moreover, it uses the MH mailbox format, where each email message is a single plaintext file so it's very flexible and if necessary it allows for straightforward manipulation directly from the shell. There's even a nice book available on it.

A tip: Dovecot has a nice sync tool http://wiki2.dovecot.org/Tools... Perfect to get your email from different IMAP sources to your own system. It can also change mailbox format etc. Combine that with Dovecot itself to give you IMAP access and you have access. You can also use it to keep it in sync with an off site archive.

Dovecot does have full body search, but it is quite CPU intensive. No problem if you just run it for a few users and except that it may take a while on a large amount of emails. Not too great if you're hosting for lots of users.

If I'm configuring some sort of local mail store, I don't just need to know how to set up Dovecot. I need to know how to set up Dovecot, Postfix, Roundcube, and so on, and I need to know how to set them up together.

That type of information is typically also in the docs because you're right. It's important, and if everyone has to hammer it out themselves, it's reinventing the wheel.

Sorry for advertising my own product, but pretty much on topic here. :) Buy two (cheap) servers from completely different networks / data center providers, and keep them replicated with http://wiki2.dovecot.org/Replication. You can set up MX records to both of them, and use DNS to switch between the replicas for IMAP/POP3 as needed. Either one of the data centers can die and your mail won't stop working. Or keep one of the replicas in local network and your mail keeps working even if your internet connection dies.

(Then you'll only need to hope that there are no software bugs bringing down everything.)

Run your own IMAP server. For the past decade or so, Dovecot has by far & away become the best choice. If you've set up any other daemons before it's really not very complicated software.

I'm not sure what you mean. It's possible to have arrays of ints, floats and everything. Looks to me like your idea would assume that array works only on structs? Anyway here's the implementation: http://hg.dovecot.org/dovecot-2.1/file/tip/src/lib/array.h

If mobile devices had decent terminal emulators, I might still be using local mail on a machine somewhere.

Have you heard of IMAP? Use the local email clients on your laptop, mobile phone, desktop and whatever to access a single email account. I've got Dovecot running on a virtual server, and access it from numerous devices. I've also got Roundcube installed on the server, for a webmail interface.

(I've been running my own email system for some years now - and it's been surprisingly straightforward to admin. I wouldn't recommend it to everyone, but I like having an alternative to all the big, ad-funded webmail services...)

It's virtually impossible to replicate the functionality and ease-of-use of gmail. However, I've recently looked into this, and here are my comments (note: I haven't yet implemented any of this, so take this with a large boulder of salt):

* For obvious reasons, you need an IMAP server. Dovecot is among the most compliant and best (my ISP happens to use it :-). Should you want to choose something else, make sure you check out the IMAP server compliancy page.

* For push email on the iPhone, z-push seems to work, and people have gotten it to work with dovecot (note: this is a bit old, and so these instructions might need some tweaking).

* You do, of course need an MTA like postfix or exim, but choosing one may be a matter of personal preference.

* You're unlikely to find a spam filtering solution as good as gmail's (it's crowd-sourced, after all).

* Finding a replacement for gmail filtering rules is a big problem. You'll probably have to go with procmail.

(However, as a programmer, I happen to prefer something with a bit more power and flexibilty, and so I'd probably port over the ancient-but-likely-still-usable "deliver" mail handling program. Deliver takes mail received from postfix, exim, or sendmail and feeds it to a program that you write (a shell script, ruby script, C++ program, or whatever you like). Your program then tells deliver what to do with the message (deliver it normally, refile it, delete it, etc., etc.). Also, since it's a program, you can do behind-the-scenes stuff like saving of attachments, vacation autoreplies, mail archiving, etc., etc.. It's the ultimate in power in flexibility, if you can program.)

However, this still doesn't address the issues of contacts and calendars. Unfortunately, there's no good solution for these:

* You might want to check out Zarafa. The free community version seems decent, as long as you're happy with access via the web or iPhone. Mail filtering capabilities are limited, and you'll have to use Outlook if you want to use a desktop client for contacts or calendars (the free version limits you to three Outlook users). However, Thunderbird might be usable via CalDAV for calendars and z-sync for contacts.

* As others have mentioned, Zimbra is a possibility. However, if you need iPhone support, it appears to be horribly expensive for home use -- as in multiple hundreds of dollars expensive. From that I understand, the Zimbra network edition, starter version is the cheapest iPhone-supporting deal, at ~$400/year or $840 for a perpetual license.

That's why I recommended Dovecot - it uses indexes which make searching 20 years of emails very possible.

As far as a "cloud" webmail interface goes Gmail has the best search features (which probably contributes to why so many Slashdotters prefer Gmail), but the search features introduced into the Thunderbird 3.x mail client are the best of any e-mail interface. To even rival the customizability of searches that is available in Thunderbird 3.x would require one to be fluent with command-line commands like find and grep, but acquiring such fluency is temporally expensive.

Timothy (OP) says that he has already tried Thunderbird though, but since his first complaint is that moving the "hundred of thousands of emails" that he has hoarded over the past two decades between the email systems that he has already tried takes "forever to process", Timothy appears to have some unreasonable expectations regarding data sets that large (specifically in regards to migrating and indexing such sets).

For those who do not feel comfortable keeping their e-mails in the cloud, they could always use Thunderbird 3.x as the interface and administer their own IMAP server at home using software like Dovecot.

If this is really important to you, and you want it all to work across multiple workstations/OSes, your best bet will be to store it all in IMAP. If you have the means and motivation to run this yourself, I would recommend Dovecot. If you don't have the means and motivation, then you can use a service like Gmail to run your IMAP although you give up certain freedoms in doing so. For example, I use Dovecot coupled with Maildir++ as the physical storage format - as a result I can (if I wanted to) change to any email client I wish very quickly, use different email clients at the same time, etc.

Many distributions moved on to Dovecot

And I'm probably a nerd since I actually understand it ;-)
I am actually still using courier-imap...
http://www.courier-mta.org/imap/

Many distributions moved on to Dovecot

Can you give examples of good Exchange replacements?

Yes, for that see DVL. Seriously, though you have to define what activities you need to do before you can ask for a replacement. MS Exchange is marketed in many niches and fails (on the surface) in most. The most spectacular is its failure as a mail server replacement, if you look at it as such. If you look at the wonderful cover of plausible deniability it gives executives by randomly losing and delaying mail, then that is a success.

Anyway, try looking these. Keep in mind that, unlike with M$ products, you can combine pieces of several packages.

• Kolab — http://www.kolab.org/
• Citadel — http://www.citadel.org/
• Dingo Calendar Server — http://andrew.triumf.ca/dingo/
• Darwin CalendarServer — http://trac.calendarserver.org/
• Bedework — http://www.bedework.org/
• Zimbra — http://www.zimbra.com/
• OpenGroupware — http://www.opengroupware.org/

If you are simply looking to improve reliability of e-mail they a plain Mail Transfer Agent (MTA) will do. Before it became too embarrassing for M$, it used to be recommended practice to put one of these in front of MS Exchange to improve reliability and security. Also look up ClamAV, Spamassassin and how to do greylisting.

• simta — http://rsug.itd.umich.edu/software/simta/
• Dovecot — http://www.dovecot.org/
• Postfix — http://www.postfix.org/
• Exim — http://www.exim.org/
• Sendmail — http://www.sendmail.org/
• qmail — http://www.qmail.org/

However, before you can think about "replacing" MS Exchange, you will have to get rid of the staff that selected and deployed it in the first place. They ignored all the licensing shortcomings, the bad reviews, high price and ongoing technical failure to instead push ideology over technology. People making decisions based on ideology are not going to accept any technical or economic arguments...

Can you give examples of good Exchange replacements?

Yes, for that see DVL. Seriously, though you have to define what activities you need to do before you can ask for a replacement. MS Exchange is marketed in many niches and fails (on the surface) in most. The most spectacular is its failure as a mail server replacement, if you look at it as such. If you look at the wonderful cover of plausible deniability it gives executives by randomly losing and delaying mail, then that is a success.

Anyway, try looking these. Keep in mind that, unlike with M$ products, you can combine pieces of several packages.

• Kolab — http://www.kolab.org/
• Citadel — http://www.citadel.org/
• Dingo Calendar Server — http://andrew.triumf.ca/dingo/
• Darwin CalendarServer — http://trac.calendarserver.org/
• Bedework — http://www.bedework.org/
• Zimbra — http://www.zimbra.com/
• OpenGroupware — http://www.opengroupware.org/

If you are simply looking to improve reliability of e-mail they a plain Mail Transfer Agent (MTA) will do. Before it became too embarrassing for M$, it used to be recommended practice to put one of these in front of MS Exchange to improve reliability and security. Also look up ClamAV, Spamassassin and how to do greylisting.

• simta — http://rsug.itd.umich.edu/software/simta/
• Dovecot — http://www.dovecot.org/
• Postfix — http://www.postfix.org/
• Exim — http://www.exim.org/
• Sendmail — http://www.sendmail.org/
• qmail — http://www.qmail.org/

However, before you can think about "replacing" MS Exchange, you will have to get rid of the staff that selected and deployed it in the first place. They ignored all the licensing shortcomings, the bad reviews, high price and ongoing technical failure to instead push ideology over technology. People making decisions based on ideology are not going to accept any technical or economic arguments...

As does Dovecot

I visited Directi a couple of months ago when they wanted to see me and begin sponsoring my open source IMAP server. The company certainly didn't feel like a sleazy spam harboring company, so I find these accusations a bit hard to belive. The summary could have also made it a lot clearer that Directi is saying these are all lies instead of just a small "BTW here's their reply" link.

I agree, i've seen the term abused throughout both business and open-source scene alike. My favourite is Dovecot which managed to get 32 release candidates throughout whole year, completely changing module/plugin API and major data structures somewhere inbetween. Well at least they followed the release cycle - just week ago one salesman told me that his company's product successfully progressed from gamma to beta stadium.

Aww, wtf, Google ref stuck in my link. That'll teach me. Here's the right link.

There are two text files on the Dovecot site. One is about secure code and the other is about writing good C. Unfortunately, the one I can't find is the more useful of the two.

> What exactly can't IMAP do that is IMAP (the protocol's) fault?

It can't receive notifications of new messages arriving in multiple folders (such as if you've got a bunch of procmail rules sorting stuff on the server.) There was some discussion on the Dovecot mailing list a while ago about whose fault it was for that feature not working. =)

OfflineIMAP would fix most synchronization problems. Dovecot is a fast IMAP server and Maildrop coupled with your favourite smap filter could take care of the server part. Couple that with a good mail client (mutt) and a way to synchronize contacts. mutt can be customized with own keybindings, so that way one could add support for training the mail filter. I keep my home directory in a darcs repository to keep it in sync between machines. Other people use Subversion.

Sun is in trouble, and according to FSF Lawyer Eben Moglen's (wild) allegations in his talk at a recent Free Software Foundation Associate Members [fsf.org]hip meeting, they previously (2005?) took a bribe from Microsoft to keep OpenSolaris incompatible with the GPL

Wow, I would really like to see some evidence of that. As it stands, it's just an absurd claim with no support. Having (over a period of 15+ years) used Sun equipment and software, and having worked with the company as a customer, and having known people who were employees at Sun, I would say that the chances of this being true are about 0.01%. Sun doesn't like being told what to do by Microsoft, or have you forgotten that Sun once famously sued Microsoft over Java and also once banned PowerPoint presentations? In fact, here's a Scott McNealy quote about the PowerPoint thing:

We had 12.9 gigabytes of (Microsoft) PowerPoint slides on our network. And I thought, 'What a huge waste of corporate productivity.' So we banned it. And we've had three unbelievable record-breaking fiscal quarters since we banned PowerPoint. Now, I would argue that every company in the world, if they would just ban PowerPoint, would see their earnings skyrocket. Employees would stand around going, 'What do I do? Guess I've got to go to work.'

So, it's really hard to believe that a company with this history would be in bed with Microsoft and on some kind of anti-GPL crusade. Anyway, you also said:

Sun is now flip-flopping like a struggling politician; they caved to the pressure of GPL'ing Java

I hardly see how this can be considered flip-flopping. Two years, neither Solaris nor Java was open source in any sense. A year ago (approximately), Solaris was open-sourced. Today, Java was open-sourced, and they mentioned they are thinking of adding GPL to the list of licenses for Solaris. This seems like a steady trend in the direction of open source.

The Free Software Foundation has made no announcements on either of these developments.

So what? Dovecot is a really cool GPL-ed POP/IMAP server, and I don't recall the FSF making any announcement when it was released. Nor do I recall the FSF making any announcement when many other things were released under the GPL. It's not necessary because the GPL speaks for itself.

How would GPL'ed Solaris utilities impact use and development of the GNU utilities? (Yes, I realize that the Solaris utilities share code with BSD utilities given their common ancestors . . .

What common ancestors? Solaris is based on System V Unix from AT&T, not on BSD. Yes, Solaris 1.x (a/k/a SunOS 4.x) was based on BSD, but it was pretty much totally rewritten before Solaris 2.x, so Solaris 2.x (and 7, 8, 9, 10, etc.) have very little in common with Solaris 1.x, and thus very little in common with BSD. You will notice that /usr/bin/ps on Solaris takes options like "-ef" rather than options like "aux", and you will also notice that sh's echo command needs "\c" in the string instead of "-n" as a separate argument if you want to supress the newline. So you can see that the Solaris command-line utilities are not very BSD-like.

I use Postfix. It's simple to set up in a simple configuration where it relays outgoing mail to a smarthost and delivers incoming mail locally. If you are doing something more complex than that then I have no idea, since that's all I use it for, alongside Dovecot for IMAP access to the local mailboxes.

Having said that, I'm sure Exim is just as easy for a simple config like that. I really just use Postfix because I've always used Postfix. :)

I recently designed and built a mail system for a six-digit ISP userbase.

Before I feed you the design, let me tell you a *crucial* concept that you must carry with you at all times.

EMAIL SYSTEMS ARE PROTOCOL SPEAKERS BETWEEN USER DIRECTORIES AND STORAGE.

Read that and inwardly digest it before you even start to design your system.

For the design, first, I'm going to proselytize a particular piece of software.

DOVECOT IS THE FREE POP/IMAP SERVER OF THE FUTURE. It leaves the Cyrus codebase rotting in the slime. It already kicks Courier's butt in performance and ease of deployment. It's beautifully coded; it has the most elegant authentication architecture; it's exceptionally fast. It isn't complete yet but it's featureful and stable enough that I have successfully deployed 1.0-betas into production. http://www.dovecot.org/ for the last IMAP server you'll ever need.

Here is the design:

1 x OpenLDAP 2.3 master server
2 x OpenLDAP 2.3 read-only replicas
2 x world-facing mail servers running Postfix 2.3
4 x mail scanning servers running amavisd-new 2.3.3, ClamAV, SpamAssassin, Sophos SAVI and Sophos PMX-ENGINE. LMTP in from the mail front-ends; ESMTP out to the mail storage.
2 x mail storage front-ends running Postfix 2.3 and Dovecot IMAP/POP3 1.0-beta. These servers also run mysql for amavisd-new quarantine and squirrelmail user options. Actual storage is over NFS to the NetApps. Using Dovecot's Sieve-based delivery agent for server-side filtering.
2 x Squirrelmail webmail servers. We have our own skin, and our own sqm plugins as the user interface to our various system options - which are all in LDAP. We have integrated MailZu into sqm as a quarantine view/release interface.
2 x NetApp FAS3020c heads w/4TB NFS storage allocated to mail.

Everything is load-balanced using foundry hardware LBs. It's very high-throughput and very reliable. It's also easy to monitor (we're using Nagios).

Base OS is Debian Sarge with applicable backports. I'd prefer FreeBSD but this happens to be a Debian shop, and I wasn't out to change their world, just their mail system.

Probably the most borderline item is mysql's performance as a quarantine DB; however much RAM and index/query tuning we throw at it, I'm yet to be satisfied with InnoDB's performance on this 100GB+ INSERT-heavy database.

If I could change one thing about it, it'd be to use the extremely pretty and surprisingly good value @mail (a commercial choice) rather than SquirrelMail. I'd also consider Fedora Directory Server over OpenLDAP, but it wasn't looking ready for this design at the time.

I have to say there is some bad advice in this thread; now for the hatchet:

Cyrus: difficult to configure, doesn't support shared storage, horribly ugly codebase, and has some nasty-ass failure modes.
Qmail: stale, poorly integrated MTA software from the bitchiest developer in town.
Sendmail: doesn't scale. Even the developers think so, which is why Sendmail X is a rip-off of postfix.
Communigate Pro: if I don't get to futz with the source for integration and value-add, I'm not interested.
GFS/GPFS: you don't need the complexity or interesting failure modes of shared-block-storage filesystems. Stay away.
Linux NFS: isn't reliable enough. We've had problems with data corruption to Linux NFS, both kernel and userland. Right now the only NFS server implementations I trust are NetApp's and Solaris's. No doubt the Linux one can/will improve, or already has, but trust is a hard thing to build :). Plus NetApps are shiny, marvelously reliable, and I love their support.

Wow, that is a bunch of incorrect assumptions!

>Just today I saw KDE goes wild on an SLES9SP2 system and nearly freeze it - the same >fucking thing that used to happen back in 2000. Five years past by and not much has >changed.

Who was talking about KDE? If you are running a mailserver, there is no need for such eyecandy. Why not install in CLI mode only. On the other hand, if you don't like KDE, why not try GNOME? Both are lightyears ahead of the Windows GUI as far as really weird flaky disasters. Windows people crack me up.

>You need directory services, scheduling, global address book, forms and sophisticated IMAP >folder sharing even in a very small company (100 employees), so even in small-and-medium >enterprises, people do need Exchange-like functionality and not only SMTP/IMAP/Webmail.
>Dovecot: it's in alpha, for Christ's sake (http://www.dovecot.org/)

Did you even check out zimbra.com?
1) Directory Services - OpenLDAP.
2) Scheduling - Handled with iCal
3) Global Address Book - again, OpenLDAP
4) Forms - ?
5) Sophisticated IMAP folder sharing - I could not find which IMAP server they were using, but if it complies to the spec, should be taken care of.
About Dovecot - so what if it's alpha? I have run many programs over the years which are pre-alpha quality (hint: it's mostly from a company in Redmond).

>1. Software for managing AD: not really that expensive. On Linux you need to spend as much >to write and maintain custom scripts, Webforms and what not.

Pretty much evens out, you're right. Unless someone has written the scripts you need already. Ah, the power of open-source.

>2. Backup software: yes, because Exchange has its internal database format (i.e. it does >not use only flat files). You can't back that up without suspending I/O to a consistent >state which means you have to have an application-side plugin.

You say it yourself: Exchange blows. Having a proprietary database format is silly when dealing with e-mail. Why not use flat files - oh! I know! So they can keep your data hostage.

>3. LVM: can't create crash-consistent snapshots of database files so what you say is >incorrect, unless you meant snapshots of ordinary IMAP directories (incorrect comparison - >database format vs. flat files). Besides, if you have VSS H/W Provider agent on Exchange >server, you can take snapshots (on storage or the server itself), re-mount them and backup >them using the regular Windows software.

LVM won't need to rely on snapshots of database files... are you even trying at this point?
It is an incorrect comparison. Correct. You can do this thing you mention in the second sentence, but why bother? Wouldn't you rather get on with admining rather than setting up hacks and workarounds to Exchange and Windows' many problems?

On another tangent, why can't Microsoft make server software that works?
Bah! The pain of having to work with non-standards compliant products.

-Nick

To comment on the article: wouldn't it be great if /. had a regex filter so that we can get rid of these "exchange replacment" articles....
Just today I saw KDE goes wild on an SLES9SP2 system and nearly freeze it - the same fucking thing that used to happen back in 2000. Five years past by and not much has changed.

> That said if you need a "farm" of computers to run your mail and your company has fewer than 100,000 employees, I think the benefit of moving off Exchange should be obvious: you wouldn't need the farm any more.

You need directory services, scheduling, global address book, forms and sophisticated IMAP folder sharing even in a very small company (100 employees), so even in small-and-medium enterprises, people do need Exchange-like functionality and not only SMTP/IMAP/Webmail.
Dovecot: it's in alpha, for Christ's sake (http://www.dovecot.org/)

>If you were moving to a newer Exchange you already know the hidden costs: software for managing Active Directory quirks (from CA or whomever), special backup software that interfaces properly with exchange (possibly licensed per mailbox) and so forth. With the usual Linux setups you would backup mail the same way you backup anything else: with an LVM snapshot.

1. Software for managing AD: not really that expensive. On Linux you need to spend as much to write and maintain custom scripts, Webforms and what not.
2. Backup software: yes, because Exchange has its internal database format (i.e. it does not use only flat files). You can't back that up without suspending I/O to a consistent state which means you have to have an application-side plugin.
3. LVM: can't create crash-consistent snapshots of database files so what you say is incorrect, unless you meant snapshots of ordinary IMAP directories (incorrect comparison - database format vs. flat files). Besides, if you have VSS H/W Provider agent on Exchange server, you can take snapshots (on storage or the server itself), re-mount them and backup them using the regular Windows software.

I would advise you to read this essay. Being written in an unsafe language does not intrinsically make something insecure - it just makes it a bit harder to write secure code. Likewise, a bad coder can write insecure code in a safe language.

Well, that does show pretty clearly that mysql is slower, WAY slower for deletes. It doesn't even do normal operations like reads unfortunately. All it really did show was that searching is faster if you have indices to search. This didn't really need to be shown, everyone already knows this. However, you don't need a database to get an index, any imap server could impliment indexing if they wanted to, and in fact dovecot has:

http://www.dovecot.org/

Courier IMAP is running to support both IMAP and POP access to the mailboxes.

I would switch to dovecot. I found the performance to be quite a bit better than Courier, and it seemed more stable as well.

From what I have read, it does seem to affect some users and not others. There is an extensive thread on it over at Apple's discussion forum. For what it's worth, I'm using Dovecot for my IMAP server.

Not if you use a good imap server

Greylisting is a very powerful spam reduction technique (with no false positives). This can be done at the firewall, and will consume little resources. It will stop much of the e-mails sendt by viruses that has it's own SMTP engine. Your other spam filtering daemons will have less work to do.

OpenBSD has a daemon spamd that can do greylisting. Just put an OpenBSD box in front of your mailserver, and you can test it out for yourself.