Slashdot Mirror
What Mailbox Format Do You Use And Why?
"I currently store all of my e-mail in a local mbox-style IMAP store in ~/mail/, so that I am not tied to any particular mail client. However, I am planning on syncing my mail across multiple machines (home, work, and soon a laptop) so I need to have mail in a form which can be synced easily. MBox is bad for this because if I grab mail on one machine, and later delete some mails from the same folder on another machine, then sync, the new mails will be lost. This is where maildir is good - each message is a separate file. But why do so many people hate it? If I do change over to mailbox, what IMAP/SMTP servers should I use? A hacked sendmail/UoW IMAP? Courier-IMAP + QMail? Something else? How do other people keep their mailstores synced across many machines, and what software do they use?"
And this is a perfect example of what IMAP was designed for -- a stateless client such as a web browser.
If you have > 32768 mail messages in your inbox you should be slapped by your sysadmin.
Then your sysadmin should be slapped for running Solaris 2.5.1
And finally, WTF was your sysadmin thinking even allowing you to run without a quota?
Put a friggin limit on the size of a lusers inbox and issue a warning when the limit is reached, next start wiping emails after the warning period expires...
Lovely,
BOFH
I had that problem, then I put on my BOFH hat and fixed it. We tell users to keep their mailboxes clean. That means 3 megs of crap at the most. Once they cross it, qpopper's custom hack starts renicing them one point per meg. At 23 megs or more, they REALLY go into the toilet at a nice level of 20.
...
Once in awhile someone calls up and complains that it's going too slow. The proper response is to say "does (person near you) have the same problem?" - when they say "no, of course not", you simply reply "that's because they keep their mailbox clean" and let them deal with it.
Yup, lusers can definitely handle the old trick of "it hurts when i do this" / "so don't do that, then"
Maildir seems elegant at first, but it has one problem: our filesystems suck. We need a filesystem that is good and fast at creating, opening, and deleting files, even when there are 20000 files in a single directory.
I wonder if the Right Thing to do might be to forget about ext3, Reiser, etc. and write a custom filesystem that is deliberately designed and optimized for this special case situation.
...Which means that you can't leverage your already existing tools (e.g. grep) to extract data. Sounds anti-Unix.
I'm sorry, perhaps I sounded more inflammatory than I meant to. I wasn't aware of a POP alternative; I'm still learning the security part of administration, and while everyone and their mother tells you "don't run telnet", I've had a hell of a time finding anything decent about securing mail services, other than "just wrap it with SSL, it's the wonder cure!"
Try a search for "secure IMAP" on google and then try acting suprised at my response. :(
1st Law Of Networking: Loose ends are bad, termination is good.
WWJD? JWRTFM!!!
1st Law Of Networking: Loose ends are bad, termination is good.
WWJD? JWRTFM!!!
You're using MH folders and you say can't read it from a shell? I thought this was the point of using MH :-). Get NMH and read stuff from the command line to your heart's content. That's the only way I read mail now.
For more information, man nmh and look at the manpages for each program, or better yet, read the ORA book on the subject. It's very helpful.
Quick starter guide though:
Messages are referred to by number. There are plenty of other commands that do other interesting things (such as pick which lets you query the current mailbox with regexps on a per-component behavior), and every aspect can be customized (see again the ORA book).
The coolest thing is that because these are all shell commands, they can be scripted to do complex things your "conventional" mail client never could. As an added benefit, since they're not a monolithic program, there's no resident size.
Finally, if you want a nice monitor, I hacked MH-style mailbox support into Sjoerd Simon's Mailwatch plugin which you can use with gkrellm. (If you're in the console, use flists to show new mail.)
Have fun. :-)
Don't think of it as a flame---it's more like an argument that does 3d6 fire damage
Well, one (probably undesirable) solution would be to fetchmail it all to one location (like your home box). Of course, that defeats the purpose of IMAP. I'm sure you've considered this already though. Hmm... too bad NMH doesn't do IMAP.
Don't think of it as a flame---it's more like an argument that does 3d6 fire damage
Distinguish between Outlook and Outlook Express. There are a bunch of people who feel they can't live without Outlook's shared calendering, contacts lists and so forth. They don't care if Exchange is a piece of shit.
You've got it the wrong way around. Outlook is a powerful client with a bunch of neat neatures. Exchnage is an unreliable piece of shit with the standards compliance of IE4.
There's a lot more value in something that allows people to use Outlook with a sane back end (eg, OpenMail).
You were probably reading from the point of view of a client. The client interface to IMAP is relatively nice and quite capable. Implementing the functionality on the other side (the server), is a completely different story altogether. It seems as though the designers of the IMAP protocol made some assumptions about the server implementation which influenced the design. So server implementations tend to have to work hard to provide all the features required by the RFC.
However procmail (and maybe other local delivery agents?) can be used with the maildir format, so you get all of the advantages without worrying about running qmail if you're not ready yet.
-Peter
== Just my opinion(s)
Damn right that news abuses the filesystem. Why do you think that products with cyclical file systems for news overtook INN a while back until they developed their own?
Mind you, with reiserfs, it may become feasible to do the "one article per file" thing again.
-Dom
Just curious, what caused you to migrate from FreeBSD to Linux. I would have thought that FreeBSD (with softupdates) would have been the better choice...
-Dom
I notice the story isn't about POP3, but qmail is getting a lot of attention for being as slow as it is. I just gotta put in a mention for the Cubic Circle POP Daemon, cucipop (search freshmeat). Small, fast, perfect for small- to mid-size email servers. As always read the dox before installing.
/. ID is lower than the real Bruce Perens'.
(Moderators: I have a +1 and didn't use it. Consider this post already modded down.)
The real Threed's
--Threed
If I may offer a suggestion ... you need to use the -print0 argument to find, and the -0 argument to xargs. This will allow filenames with spaces in them to work properly. This comes in handy quite frequently especially with Samba mounted shares, as Unix people would never do something as stupid as put spaces in a filename :) but Windows people do it all the time ...
Actually, what it means is that you can use a high-performance, industry standard query language like SQL to extract data, instead of having to kluge together a patchwork of file and stream manipulation tools.
I'm no database expert, so here's a question: would it be possible and practical to have a virtual, read-only file system based on a database so database records could be viewed as files in a special hierarchy? Then you could have the advantages of both.
Ooh, a sarcasm detector. Oh, that's a real useful invention.
I totally agree. That's why I said it's ugly.
I was just addressing the concern that "from" in the body of a message would cause a problem. Yes, it would, so it gets "escaped."
Geoff
Computers are useless. They can only give you answers. -- Pablo Picasso
That's why you see things like
Yeah, I heard yesterday
>From the banker and he
says my loan is approved.
Ugly, yes, but it does prevent confusion. (The mailer should escape "from" at the start of a line automatically.)
Computers are useless. They can only give you answers. -- Pablo Picasso
you've just gotta hate a format where the common English word "From" at the beginning of a line is used as a delimiter.
I may be smoking crack here but I believe the delimeter is a blank line followed by "From:"...
Even if you had a message which had "{cr}From:" it would be stored as "{space}{cr}From:" and would not be taken as the start of a message. It's been a while since I've looked at the RFC but I believe this is how it's done. It's no harder to scan for than \x6e\x1e\x77\x0a\x5f or some other "pulled out of the air" delimeter.
Do you have a pointer to his views on GPL ?
The way (to me) an RDBMS makes sense for storing email on the server side are increased speed of retrieval (maybe), and speed of development. It makes more sense to me when you've got 1000 or 10000 clients getting their email all at once... then I think the indexing and concurrency features of the database would make up in speed for the increased complexity.
* And remember, it's spelled N-e-t-s-c-a-p-e, but it's pronounced "Mozilla."
A true enterprise-grade message store will use an embedded database with transactions support.
Why is it important that the database be embedded? I agree that from a client/server standpoint, using an RDBMS makes far more sense than the standard sendmail mbox format. But it seems to me that having an external database would be more flexible than a linked-in system like Berkeley DB.
* And remember, it's spelled N-e-t-s-c-a-p-e, but it's pronounced "Mozilla."
actually, most people will need to run it as so :
the xargs you provided will act the same as grep * - passing all of the files as parmaters instead of each in turn.
--
You ever hear of a filesystem? Your forcing your server to do something it was not designed to do. I see this all the time, be it on a pop3 server, IMAP or Exchange. People use their email as a repository for EVERYTHING. With IMAP and Exchange it is not as dangerous, but the users out there that save ALL their mail on the local computer always beg for lost data. If you have things you need to save, do so. Save them out somewhere other than Email. When that server goes down for a few hours and your rushing to get a presentation printed for a meeting the sysadmins have NO sympathy for your predicament. The more you stress the server the more chance you have of bringing it down. The better you treat it, the better it will treat you.
GeekBoi
Pres. Admins for the ethical treatment of servers!
01010100 01101000 01101001 01110011 00100000 01101001 01110011 00100000 01101101 01111001 00100000 01010011 01001001 010
Your being generous. I would not run Exchange on anything less than a dual pIII with at least a gig of ram. Remember it is full of "FEATURES" and "FEATURES" take up resources. Exchange, as with all MS products is like a 2 month death sperm whale sitting in the hot coastal sun, it is nothing but bloat and it stinks like the dickens.
01010100 01101000 01101001 01110011 00100000 01101001 01110011 00100000 01101101 01111001 00100000 01010011 01001001 010
The I LOVE YOU virus showed the idiocy of Outlook and I for one, do blame Outlook for being so bad that it runs .vba scripts without asking. Outlook like most M$ products, has a very pretty look and feel with lots of bells and whistle but no care about about how it does its job. As someone that administers an Exchange server, my greatest problem is that I would not dream of using it myself so I always have to fire up another machine when someone has a problem (I would rather give the job to someone else but eveyone else that is capable feels the same way).
I love stacking my barbecues in the shed at the end of summer - you can't beat a bit of grill on grill action.
Congratulations...
A lovely bit of troll. Well written and it looks serious enough to get people incensed and replying.
I love stacking my barbecues in the shed at the end of summer - you can't beat a bit of grill on grill action.
If you use rsync to copy the files back and forth
then you can use mbox no problime it will only copy parts of the file which changed. You would run in to a problime if there were changes in more then one file thoe. You could also consider CVS as a tool to distrubute you mail. It could handle the changes for multypul copys of the mbox and merge them. Plus you would never loose a message by acident.
You can get similar speed to mbox with maildir, you just have to attack the problem in a different way. Usually when you're running a grep on your mail, you're searching for something that you recieved a while back. Thusly, the simple solution is to use glimpse.
Every night, I glimpseindex my Mail directory. Although that means that I can't use glimpse to search the emails I've recieved that day, in exchange, I get linear-time searching of my entire archived mail area. (Currently at ~650MB)
The Mail(1) command won't work anymore (real nice for all sorts of customer support stuff).
In the context of qmail, using the sendmail-wrapper provided, mail(1) still seems to work fine for me... I find it handy with uuencode to throw file between systems with no other connections.
"don't fall into the fallacy of believing that Perl can solve social problems. Maybe Perl 6 can, but that's a ways off"
I'm so sure it is, but either way, the mbox size is the total of all your mail, not just one message.
(thinking about it - I think we're both right. For maildirs, you would do a mv(1) into the tmp subdir, which is essentially free, rather than qpoppers copy to a different fs (/tmp) which is slow and expensive)
"don't fall into the fallacy of believing that Perl can solve social problems. Maybe Perl 6 can, but that's a ways off"
see subject
___
___
If you think big enough, you'll never have to do it.
--
Set your MTU on your ethernet interface to
576 instead of 1500 - GREATLY improves latency
over cablemodem/DSL connections.
I agree. I used to use Netscape both under Win and Unix. The IMAP worked well, and I had consistency between the platforms. But I ended up needing one feature that only OE provided well - the ability to switch between personalities quickly. If I reply to a message in a folder on one account I appear to come from there, if I reply to the other, then from that one. Mozilla is nearly there, but it's still a bit too unreliable for me, and it's *so* ugly...
I would avoid copying mailboxes/messages directly if at all possible; keeping this stuff sync'ed is what IMAP is for, and there are IMAP-aware mail clients for just about every platform including PalmOS. The problem is that you would need to use a client that supports IMAP disconnected mode (which synchronizes a local and remote message store for offline reading), and unfortunately very few do this properly; the only ones I know of is Netscape 4.x on the Windows platform (it doesn't on Unix, and Netscape 6 doesn't at all), though I think HandMail for PalmOS does as well.
There's a utility called isync that synchronizes a local maildir-style message store with a remote IMAP server, but it only supports maildir, and I've found it somewhat tricky to configure. I'm in the thinking-about-it stage of writing a similar utility to support mbox.
I'm not sure what problem you have with Netscape 4.x and and the web interface of exchange, but I just tried it with Netscape 4.08 and Exchange 5.5 and it works fine for me.
This server didn't scale past that as you can guess... but the server in question was a two proc PII-350 with 1gig of ram, all UW2 scsi drives off a raid with 16M on board cache. Performance was fine for people with the small mailboxes, but for people with large (ie 500M+) it was quite slow.
Again with MBOX this wasn't even remotely possible, with MBX it just worked.
--- I do not moderate.
I use MBX... and have used it in a pretty decent sized IMAP implementation (120 users/average 180M INBOX). MBX simply adds a line at the start of every message telling the client exactly how long the message is. So instead of scanning until the next header it can just jump. Occasionally there are breaks but they are easily fixed with any text editor (well editors that can open really large text files).
--- I do not moderate.
http://ftp.botik.ru/rented/namesys/ftp/pub/linux+r eiserfs/gif/postmark/postmark.html
Postmark benchmark: "PostMark was designed to create a large pool of continually changing files and to measure the transaction rates for a workload approximating a large Internet electronic mail server."
Well, I think InterMail (used by Excite, AT&T, GTE and others) is at least 'Enterprise-Grade', and it stores the messages in the filesystem. The metadata (envelope, not headers) for messages is stored in a database, but the messages themselves are just files, one per message.
Awesome furniture, accessories and cabinetry in Santa Rosa, CA: http://humanity-home.com/
I have all my emails in mbox format right now. What is the best way to migrate to the Maildir format? At the same time, I would like to start using procmail. Where could I find info about Maildir with Procmail?
-- Slef
Having tried to run a database mail back-end for an ISP, I can categroically state that this just won't work. You can just about hack it with fixed space storage for users, but once you start deleting mail you're in line for some very interesting table corruption issues... Performance was pretty bad, too. We ended up porting everything to maildir. I've since used this format in 2 further projects. One a Hotmail style service for the UK's largest ISP, and the other in a large image storage service. Maildir worked great for both services - handling images just as easily as mail! S.
I agree - Postfix is awesome. It's compatable with programs that do stupid things like running sendmail directly (postfix installs a binary called "sendmail" for that purpose), and is dead easy to configure. I ran sendmail for a couple of years without knowing exactly what lal of the stuff did. Then I needed to change something, and did - I went to postfix. It's running a 12,000 student email system right now with cyrus IMAP (using maildir on a reiserFS RAID) using IMP for web access, and the system just barely knows stuff's there. Try getting sendmail to deliver to maildir boxes while looking up names from LDAP - tain't gonna happen easily...
I agree that from a client/server standpoint, using an RDBMS makes far more sense than the standard sendmail mbox format
Okay, wait. This isn't my area of particular expertise, but isn't one of the inherent advantages to using an RDBMS (for anything, not this application) its ability to cross-reference records and, in the case of the high-end varieties, to store frequently-called queries in memory in order to reduce disk access?
It seems to me that using an RDBMS for indexing and storing email is overkill. The typical pattern of an email client is to check for new messages over and over, or occasionally move a large file. This usage doesn't leverage any of an RDBMS's abilities at all. You're expending lots of resources in memory and management complexity for little gain. After all, an admin's job (from an employer's position, especially) is to most efficiently use the resources available, not nesessarily to come up with the most elegant solution, right?
If anything, using a lightweight system like DB to seperate header info from message content seems like all you'd need. If filesystem access times are that bad, work on that. Check out filesystem options, or seperate the users onto multiple drives or something.
Am I missing something?
While I agree with most of your points about Exchange, it does offer a "work offline" method that does exactly what you're suggestion.
You sync your messages, go offline, reply, send, delete, move, etc... and when you reconnect and sync, all changes are replayed on the server.
-Hunter
RateVegas.com - Vegas Reviews
I thought we had technical people on this forum?! Not a lot of technical arguments WHY mbox is better or worse than maildir...
I run a small ISP that's been around for 8 years, and we've been using POP3 all that time. We've occasionally had to restore files, unblock email boxes, and so on. But the worst issue we've had to deal with is Sendmail's piss-poor security and maintenance. It seems like there's a new crack for sendmail every week.
We're moving to qmail. I haven't seen a good argument why maildir isn't vastly superior than mbox, except for MUA support. Since most of my customers use dummyware like Outlook, MUAs are not such an important issue for us.
With a filesystem that uses a linear search to find a file in a directory, maildir is an awful beast. With a filesystem that has fast directories, such as reiserfs, I would bet maildir would beat mbox hands down in almost anyy comparison you could think of.
The only problem maildir would have then is that you have to open every file in the directory to get a header list.
Need a Python, C++, Unix, Linux develop
Not even Microsoft is insane enough to suggest Exchange for an ISP solution. (They used to have a product called Commercial Internet Mail Server or something, but that was dropped.)
If you are looking for a pure mail solution, $30-$90/client (+ NT seats if you haven't bought already) is a ripoff. But if you plan to make use of the calendaring and collab features, it really isn't - Lotus/Novell/iPlanet all cost about the same amount, and there aren't any 'open source' solutions that operate on their level yet.
--
Business. Numbers. Money. People. Computer World.
I use OWA 5.5 for 9 hours a day at work. It crashes Netscape 4.76 within 5 minutes and 6.0 immedeately.
But it also crashes IE 5.01 about once a day. Turns out that IE can't reliably handle lots of javascript day-in and day-out.
--
Business. Numbers. Money. People. Computer World.
---
Modern shells and systems cope with reasonably large shell command lines.
I use nmh so all each mail is in a seperate file, the biggest folder I have has 18958 messages in it at the moment (it's a mailing list that I've been subscribed to for a few years).
grep whatever * works fine in the folder's directory...
Your comment about speed is very true though.
grep whatever * takes over 6 times as long as grep whatever foo (where cat * | cmp - foo).
That's 1:50 as opposed to 17 seconds.
However, I glimpseindex my Mail folder early every morning (which I must admit chews about 50Mb of disk space just for the .glimpse* files - then again the mail takes up about 550Mb). Which means most of the searches I end up doing are reasonably fast (obviously searching for 'a' or something will take a _long_ time)
i'm probably not the only one that'll say this, but you know, you can do a:
grep "stuff here" *
and search through all the files in the directory.
i personally think it's a better idea to put new, and old mail in different places. maildir does this, but mbox does not. mbox also suffers when the filesize gets real large. the whole file has to be read in. with maildir, only small files need to be read. this cuts down on disk i/o. on large capacity mail servers, this is a pretty large bonus.
maildir also makes it easier to delete a specific message. since everything is just a file, just delete the file, and the message is gone. no more searching through a 15 meg mbox file looking for a specific e-mail.
oh well..
Why read the article when I can just make up a snap judgement?
it's a small point but the live backup solution is free (no money down)
stay frosty and alert
It abuses the filesystem with one file per message in the same way that mh folders do. Unless you're running a decent btree structured filesystem like XFS, ReiserFS or JFS, expect a performance hit if you get thousands of messages in a single mailbox.
You can rightfully argue that filesystems shouldn't be so pitiful but that still doesn't change the fact that today's most popular ones are.
mbox uses a large flat file that takes even more processing to read, index and search. how braindead is that? but it works up to a point on todays stupifiyingly fast hardware.
Disclaimer: I use and like Maildir.
> Instead of simply reformatting your drive how about using ReiserFS?
Because he might be looking for a general solution, not a Linux specific one?
(jfb)
To spur "enterprise Linux," Big Bang, the distributed two-phase commit.
But that's exactly the problem. Any format that has to examine the payload of a message to prevent total breakage makes Baby Jesus cry. It's an idiotic thing to have to do, when there are a metric trillion better ways to accomplish the same thing.
It's one more good reason to stake that damn mbox through the heart.
(jfb)
To spur "enterprise Linux," Big Bang, the distributed two-phase commit.
Naw:
% find . -type f | xargs grep -il "string" | xargs less
I was raised on MH, and will never move away from the one mail, one file philosophy. Currently, I use Gnus and nnml, which is the most beautiful mailer I've ever seen. nnslashdot is tres cool, too.
(jfb)
(jfb)
To spur "enterprise Linux," Big Bang, the distributed two-phase commit.
You ought to check out Gnus. I've never used VM, and I know that people love it, but I moved from mh-mode to Gnus a couple of years back and have never regretted it.
exmh does rock, though.
(jfb)
To spur "enterprise Linux," Big Bang, the distributed two-phase commit.
Well, there you are. Always good to learn something useful -- thanks.
(jfb)
To spur "enterprise Linux," Big Bang, the distributed two-phase commit.
The real killer is the local database engine. Sometimes I forget to read a high-traffic maillist routed to an mbox on a unix system. To simplify matters, I'll just rm the mbox file on the unix system.
When switching to the imap "folder" in OE5.5, OE has to resync its local database of headers (mainly purging records). The database engine is REALLY kludgy; its capable of pinning both PIII650 CPUs for something like 20-30 seconds if the old mbox was really big.
It's also doing something like direct file access, because hacking the registry to move the OE database(s) to a network drive (Win2k, Netware, Samba, I've tried them all) fails miserably and results in a corrupted/empty DB.
What security issues does IMAP have that POP doesn't?
Why are subscriptions stored on the server? That is client data.
It would completely defeat the purpose of IMAP if I had to resubscribe every time I use a new client.
Why does the RECENT flag exist? First of all, it's client data. Second of all, if it didn't exist at all the client is perfectly able to calculate RECENTness just be storing the UIDs from the last session.
What if there was no last session for the client?
it's no wonder it's made almost no inroads against POP3
POP3 and IMAP4 are targeted at completely different environments, they aren't competitors. In its target environment IMAP has made lots of progress, especially when you consider that Exchage is essentially IMAP.
If you are running Exchange, you could turn on IMAP support, and then suck all your messages out into a local folder using Netscape (I've gone the other way when my last employer mandated Exchange email for everyone). If you are using Outlook Express for your home email (as I was, since my wife wouldn't use UNIX), your mail is trapped in Outlook Express' proprietary binary format forever.
Microsoft does not provide an easy (or even moderately difficult) method for you to discard their products, and that's something that needs to be accounted for in the total cost of ownership of anything MS.
With rsync you have to remember where you last made your changes. Use something much smarter, but using the same diff/compression protocol, like unison
God forbid that you have enough memory. God forbid that you be able to do live backus. God forbid that your machine is secured. Take exchange or leave it, but it a lot more than a POP server.
Yeah, the Courier-IMAP description looks great. I wanted to use it on an OpenBSD setup but it didn't handle folders correctly.
What turned me off was that their FAQ on the topic was no help - just an unintelligible rant on IMAP clients.
So screw 'em - I installed imap-uw-2000 and never had a problem.
Not since, what R3?, has the BeOS filesystem been a database.
PS. Don't talk to me about security or whatever
Don't talk to you about security? Er... Oh well, never mind. I do hope that you keep full, staggered backups and that your place can stop working for a couple of days to figure out where the last uncorrupted backup is and to install it back.
Kaa
Kaa
Kaa's Law: In any sufficiently large group of people most are idiots.
Minus useful things like Biff, keyboard shortcuts...
It never seems to work quite right with browsers other than IE, at least in my experience. IIRC, it even crashed Netscape under Linux when I tried it.
It is a hell of a lot faster than Outlook, though :-)
This is true. qmail can deliver mail to mailbox and maildir. If you want to know how to use qmail and mailbox, look at /var/qmail/boot/home in a qmail installation.
Do some resarch, maybe qmail can do the job for you after all.
I have been using gnus/nnml for many years. .active file etc.
However I recently switched to the imap (mbox) format as I wanted to be able to read my mail from other machines without loggin in through my firewall. I did not use gnus to filter mail put just some simple mh commands and some scripts that automatically generated the
What I miss most is ;
> grep xxx ~/Mail/foo/*
It is a pain to search through mailboxed w
the ml format (which is used by nntp) is really nice and I can not see why it should not be significantly much faster than mbox. However it is not supported by nnimap, and until I find another mailreader that reads mail as if it was nntp-news I will stay with gnus.
mvh
arne
Copyright 1998 arne Verbatim copying and distribution is permited as long as this message is preserved
qmail natively supports Maildir and mbox. Read the documentation. mbox is dangerous because you can't have multiple readers and writers without locking, and over NFS, the file locking is unreliable. Maildir is safe for multiple writers over NFS with NO locking.
I use a mix of both, but Maildirs are so easy to use that I prefer them when writing software dealing with mail storage.
--
Twoflower
I can't tell you how cool maildir is...I hadn't been exposed to it before my current mail account. It's the ideal hook for a programatic interface of your own design, i.e., you get to use the file system as your friend, not your enemy. Perfect for true hackers :)
Mike
Slashdot 's editors are dickheads
Not even Microsoft is insane enough to suggest Exchange for an ISP solution. (They used to have a product called Commercial Internet Mail Server or something, but that was dropped.) :-)
Oh yeah?
$ telnet pop3.[myisp].co.uk pop3
Trying xxx.xxx.xxx.xxx...
Connected to pop3.[myisp].co.uk.
Escape character is '^]'.
+OK Microsoft POP3 Server Version: 5.5.1877.117.11 ready <228.1555433360@[myisp].co.uk>
Microsoft POP3 server, version 5.5?
Well, I must say that I am impressed with the reply here! :-)
Anyone who suggested M$ Exchange formats to go and stick their head in a pig :-) it took me long enough to get all of my mail _out_ of a .pst file.
Maildir does seem to be winning the public debate - I am planning on upgrading to kernel 2.4.1 once Mandrake Cooker have a source RPM ready - so reiserfs is available to speed up directory searches.
So: how easy is it to migrate from UoW IMAP and Sendmail to Cyrus and QMail? As this is a home solution for now, are there any other SMTP servers which support maildir I can use? Sendmail at the moment is basically a relay server, I send mails to it and batch the sends in a cron job, and fetchmail forwards the mails it collects to sendmail for delivery.
Thanks for the comments!
Ross Burton
The Maildir format used by qmail...it will therefore never be supported in the official distribution
I read the docs (docs/format.txt) and my statement stands - UoW IMAPd does not support maildir.
UW imapd supports a file format called mbx. It is similar to mbox except that each mail record has a length pointer to the start of the next record. It is fast to use even on fairly large mailboxes.
My mail pipeline works like this.
I don't know how you're supposed to create mbx files normally. However, I stumbled upon this great little Perl script a while back, called creatmbx (don't know the author):
#!/usr/bin/perl .= sprintf "\015\012"; }
# Ported from imap/src/osdep/unix/mbx.c
my $NUSERFLAGS = "30";
my $HDRSIZE = "2048";
my $string;
die "ERROR: creatmbx <file>\n" unless ($#ARGV == 0);
open(FILE,"> $ARGV[0]");
$string = sprintf ("*mbx*\015\012%08lx00000000\015\012", time);
for ($i = 0; $i < $NUSERFLAGS; ++$i) { $string
# Pad the rest of the file with nulls as expected
$string = unpack("a${HDRSIZE}", pack("a${HDRSIZE}","$string"));
print FILE $string;
close(FILE);
To use it, say "creatmbx MyMailboxFile". At that point, dmail will recognize it as being an mbx file when it delivers the mail, and UW imapd will, too; using it is then a lot like using an mbox file, but far faster.
It works pretty well for me. Your Mileage May Vary.
I know this was a problem with earlier versions of Outlook, dunno what version you're using or if the problem has since been resolved, but a PST larger than 750mb will reduce Outlook to below a snail's pace. We're talking 30-45 minutes for Outlook to completely start, the entire time being reported by Windows as "not responding." Once it's up it would tend to be pretty speedy. Only saw this a few times, and it was a couple years ago...most people are sane enough to not have a 750mb+ PST.
"That's Tron. He fights for the Users."
Exchange and Outlook are not just simply email applications. I would consider them a suite of applications that include a personal information manager, directory services, calendar and scheduling, etc.
Exchange very obviously is not targeted at small or even large ISPs. Aside from the software licensing costs, the system itself doesn't scale anywhere nearly as cheaply as POP or IMAP. No ISP is going to spend $20k each for 5 PC-based servers just to support 1000 users.
Exchange is targeted for enterprise use. It actually works pretty well if the user-base is mostly Windows and requires the additional functionality. It also requires a decent administrator, which is something more difficult to find. Particularly someone clueful enough to be able to understand Internet-connected email systems and DNS in addition to all the undocumented "features" of NT or 2000.
Try out Mutt. It most closely resembles elm, so if you're comfortable with elm the switch should be easy.
Mutt will also read mh-style folders, and can be used to convert those folders to Maildir format.
http://www.mutt.org/
If you telnet into your friendly, neighbourhood POP2 server you can use commands found here which are explained in excruciating detail in RFC1939. Who needs a mail client.:)
IMHO, as per
J:)
Oh well, no point in steering now.
Yeah, that's why. The new security model; Security through binarity.
-segfault
Binary mail formats are pure evil, unless they are accessible through ODBC.
-segfault
I've got Courier-IMAP running on a RH7 box, using the maildir format. This setup works great for mutt, which I use 95% of the time. It also works great w/ Netscape, which the wife uses.
The best part, though, is Squirrelmail. IMHO, it's the best web-based email program out there. It works with just about any IMAP server, and there's even a plugin to fetch POP mail. But, with Squirrelmail, my wife and I can always check our mail on the road. And when we get back home to our mutt and netscape mailer's, we just pick up right where we left off, since the folders are stored on the server.
The cool thing with IMAP is that the client becomes transparent -- it doesnt matter which one you use (as long as it supports IMAP, obviously).
They're also rather vulnerable to buffer overflows, especially when the owner of the mailbox is unable to read his or her mail for an extended length of time.
if( read(this) ) { you = programmer; }
All you have to do is run procmail. procmail supports writing to all of the popular mailbox formats. So, if your mailer doesn't support a particular format, use procmail to convert it.
That's not true. It just means that the files would be named 1, 3, 4, etc.
That is pretty much what cyrus does.
The headers are stored in a db file for quick searching, a cache file, and and index file is kept. It does store each individual message in a file, so a decent filesystem is essential for an enterprise operation.
It ends up working incredibly well. The University where I work handles 120,000 accounts, and the software side never has a problem.
>Denial of Service attacks (rednecks driving by in pickups with baseball bats)
hmmm, they'd have to come all the way up my lawn and up my front steps to get to it... I s'pose yous country boys got mo' pro'lms widdat...
--
"It's tough to be bilingual when you get hit in the head."
What I want to know is why you think you need to sync your mail.
There is a whole set of RFCs with different protocols and standard interfaces, the whole idea of which is that it doesn't matter how your mail is stored. If you use IMAP instead of trying to hack around with different mail storage formats you have everything you need.
There's a reason you don't FTP your mail off a server, you know?
MAPI does not run on Unix does it? If not then by all means go ahead and reinvent it.
War is necrophilia.
why is every ms employee always talks in the future tense. Exchange will do this NT will do that .NET will do the other thing. By the time the future arrives MS will have already dumped it thing and moved on the next great acronym.
Tell us what it can do now because we don't believe MS can deliver on it's promises.
War is necrophilia.
Huh? I don't think I understand your usage of the word "so" in your sentance. I would expect it to be used in the form "A so B" where A is a set of facts, and B is a logical conclusion from A. Your A does in fact seem to be a set of facts, but your B does not follow from those facts.
What about "Exchange 2K supports a very nice web interface, as well as IMAP and POP" would "it's actually a lot more cross platform than UNIX mail...." follow from?
-- The act of censorship is always worse than whatever is being censored. Always.
> For MIME processing, mhn isn't that great,
> so I just bounce copies to Yahoo or
> Mail.com mailboxes so I can view them
try munpack. It's bliss.
"My God...It's full of ads!" -Fry, about the Internet, Futurama
MH doesn't have a problem with large mailboxes. One-message-per-file doesn't have the problems with content-length or lines that begin with "from " that mbox does. I like those features. What I don't like is the separate meta-data files, the need to run special utilities to file mail, the locking issues, etc. It seems that the features of the mh folders I don't like are exactly what maildir was designed to solve.
Unfortunately, I don't think any of the mail-readers I'm confortable in using (exmh, elm, gnus, mailx) have decent support for maildir -- and converting 15,000+ messages in 50+ folders is no easy task!
Interesting you say that .. I use Outlook Express (5.5) and its IMAP support is a real dog. (On a modem connection, this is, not some fat pipe). When you click 'Send', the message stays there until it's contacted the server and moved it to 'Outbox', so you think it hasn't noticed your click, and you click again, and if the connection is down then what happens to it? and if you don't sync every one of your folders each time you miss messages and get messages in the wrong place. It tkaes upwards of a minute to check mail on an IMAP server, but about one second on POP. Ugh
ufs and ext2fs sure aren't, and I believe they constitute the filesystem on most Unix boxen.
Using 'From:' to indicate the sender seems quite natural; it's an easy thing to search for, and it means you can read the mail as a textfile if you want, and don't even have to use a mail client.
Some filesystems that have appeared recently are databases, with SQL for searching etc., rather than the tree hierarchy that was first started by Multics. Oracle's iFS is probably the most notable.
It's surprising that nobody had this idea 10 years ago.
>My A500 is STLL faster than a 486Dx66
.. let's see Doom2 on your A500 and we'll compare it to my 486.
Oh rubbish
Unless this is some kind of joke that I've missed?
"Some filesystems that have appeared recently are databases...It's surprising that nobody had this idea 10 years ago"
They did, it's called OS400 (for those AS/400 iron boxen)
~Any apparent grammatical or typographic errors are caused by defects in your display device.
(storing email in a database that is)
I think you meant to say that "Technically, it may be possible, but practically, neither you nor your customers are going to like what really happens."
The phrase "Lotus Notes email" comes to mind.
We used GroupWise 4.1a on a Netware 3.11 P60 with 128 MB RAM. Only 4 GB of disk space for messages, which was kept on a DEC Pathworks file server! So the wonderful designer of this kludge had to put the post office agents on a P75 running DOS, which actually needed to be rebooted a lot less than our NT servers. We ran over *300* users with three post offices on this. Thankfully we have now gone to GW 5.5 on a dual PII Netware 5 server. Too bad Netware isn't as stable as it used to be, we need to reboot every few months.
Gamingmuseum.com: Give your 3D accelerator a rest.
First of all, let me just say that the person who designed the mbox format should be shot without a trial. I can just imagine what went on in his head "hmmm....I have to find some format for RFC8222 messages together. Oh, I know! I'll just throw them in one big file. Wait, but how will I know where e-mails end? What's one of the popular words in English? From! I'll use "From " to distinguish e-mails, and let people quote from-lines". Really!
;-), or a plain old database (cool feature: no need to retrieve headers the mailer won't show).
MBox is also very little suited to programs like MH or PMS, because it's very dangerous to store offsets into mbox files.
MH has its own format, which is particularily cool because you can have e-mails and folders in the same folder.
Other formats inclue ZODB (if you read all your e-mail in Python, of course
PS.
IMAP is *evil*. It's way, *way* bloated.
Another disadvantage of mh is that files are named sequentially -- so, if you delete message #2 of 5000, it then has to rename 4998 files. Maildir avoids this problem by naming files uniquely, while retaining all the advantages you described above.
------
I store all my email in what's called the "Information Store" on the MS Exchange server. It's transparent and works very fast. It's about 10 times faster than imap and it's secure because if you are not in the Exchange Admin group you can't see other peoples emails. The Exchange server is the industry standard in businesses.
I've been using MS Exchange for over 3 years now and I have never seen the Information Store get currupted.
Qmail supports both vsm (/var/spool/mail/username) AND maildir - not just maildir. I hope someone corrects the article headline.
Even further, in the support of standards, Qmail is distributed with default instructions for mbox format to maintain compatilibility.
Qmail has not been distributed specifically antagonisitic to current mbox users, considering it has to convert mbox users to become a useful ubiquitous standard.
unfortunately they aren't reliable. the 'snowblower bug' has shown that flooding mailboxes with large amounts of snow can cause them to crash.
Maildirs rock, Courier-IMAP rocks.
I am using qmail and courier-IMAP on a p133 with 40megs of ram mailserver with about 5000 messages on stored on the server, I use Outlook 2000 and a web interface mainly but have used nearly every mail client out there that supports imap at one time or another. I never see a slowdown or corrupted emails or folders.
to tell you the truth, I honestly have no idea...
however, I would imagine it would not handle the load too badly, afterall it was designed to handle the requirements of a university (CMU).
Mind you.....so was the WU-IMAPd, and that just doesnt scale at all.....so I guess the only way to tell would be to try it....
but..the poster seems to imply that it'st just for personal use...so in the scope of this AskSlashdot....it probably doesnt really matter....
Advanced users are users too!
Not really.....you have the same problems you mention regardless of the where the stuff is saved...
If I save stuff out to my workstation, then it's not available from other machines....if I put it on a fileserver then I still have the problem of the server going down.
And as I mentioned in my post, this sort of use doesnt stress Cryus at all.....
I've been running it for more than two years with exactly this sort of usage, and have never had any downtime. (Other than hardware upgrades etc., of course)
Advanced users are users too!
Wake the fuck up. There's multiuser support in Beos. Not to mention there's no way to scale.
I use an IMAP based webmail client (Squirrelmail or SM for short) for all of my email (~50-100 messages per day)
I believe it makes (good) use of every feature you name. SM "looses it's mind" every time the browser refreshes or you click a link. Yet it is still able to notify me of recent unread messages without telling me HEY! THERE ARE UNREAD MESSAGES every time it refreshes, and without having to have a DB back-end to keep track of which unread messages it has notified me of.
Same goes for subscriptions.
-Peter
"There is no number '1.'"
I balked when one of the sysadmins at my work suggested trying out our experimental exchange 2000 server. But, it supports IMAP and Webmail, so I checked it out. Man, is that great! The webmail is really what puts it over than just plain IMAP, although I guess an IMAP server could have a webmail client as well.
The cool thing is that I can use Netscape (Navigator or Messenger) to read mail on my main Linux box, and then if I get a word document, I just flip over to a windows box, check mail, and view attachment. No more copying to a network drive (or worse yet waiting for Star Office to start up).
Plus, no annoying POP downloading/local storage.
http://www.naildrivin5.com/davec
Exchange 2K supports a very nice web interface, as well as IMAP and POP, so it's actually a lot more cross platform than UNIX mail....
http://www.naildrivin5.com/davec
I don't think it was the same thing. It sounds as if the mailbox you're talking about was booby-trapped. That is, unfortunately, illegal. The other poster was simply suggesting that you reinforce the mailbox and videotape anyone trying to damage it. I can assure you, there's no way to file a lawsuit if that's all you do.
--
And the men who hold high places must be the ones who start
To mold a new reality... closer to the heart
Moron, read the post above. Outlook simply provides you the ability to do things faster. If that means executing an attachment which will do great things then that's what it does. It also means if you are an idiot it will take you to hell faster. Same thing as having a car. You can use it to go where you want to and you can use it to drive off a cliff. Don't blame the car.
Mmmm.. Donuts
Are you completely incapable of critical thought? We are talking about the ILOVEYOU virus here. That was NOT Outlook's fault. The discussion is not about whether Outlook is good or sucks. The discussion is whether Outlook is to blame for the ILOVEYOU virus.
Mmmm.. Donuts
That's like saying you'd rather use a horse-drawn buggy because it cost you so much time when you had that accident with a car.
The ILOVEYOU mess was because Outlook made it easy to execute an attachment. This also made it easy for users to shoot themselves in the foot but don't blame Outlook.
Mmmm.. Donuts
I'm already using Courier POP, IMAP and Web-access, and I really believe it *is* Enterprise ready.
I would like to see a proof of database is better then maildir before I would want my mail stored in one.
If I remember correctly nameplanet.com is using Reiserfs and maildir for their mail. And I believe they have a "true enterprise-grade store". Thousands and thousands of users.
http://www.millnet.se/ GO/U d- s+:+ a C++ UL++++ P- L+++ E W+++ N+ w++ M-- PE+ t+ X++
you're right that "From: " indicates the sender of a message in the message headers. however that is totally different from what the poster is talking about.
what the poster was talking about is the message delimeter in mbox format mailboxes which is "\nFrom " (note the newline and the lack of a colon).
--
Twivel
Mutt supports Maildir, MH, MMDF, and mbox formats out of the box. Works fine, been using it for several years.
I'm doing the same thing - ssh to my university machine and pine, on a cable modem with about 70ms ping.
I'm masq'ing this connection with 2 housemates and ssh is not even breaking a sweat. Pine's very snappy. Every keystroke echoes immediately.
Shit happens tho, when everyone's doing their assignments in the labs - they just CLOG bandwidth. But then even telnet becomes non-responsive.
Therefore, I think your university needs to upgrade. It's not ssh's fault.
I use a web-based e-mail service (AmExMail). I don't have to use a mail reader, and it's available just about anywhere that has a TCP/IP connection (since most people think that the http protocol *is* the Internet). If I want to use a mail reader, my e-mail provider allows me to connect to its POP servers and download my mail.
That's great until you start writing a LOT of email. Then the latency of a SSH session can start really getting on your nerves. Even on my cable modem with a 50ms ping to my university box it's still semi-annoying.
I definitely wouldn't write anything but short replies while SSH'ing.
Game... blouses.
Databases may be ok for small businesses and personal use. But for enterprise level it is completely out of the question. Why do you think places like Yahoo! and Hotmail use a filesystem instead of database? Because using databases for e-mail is very IO-intensive, especially when you have thousands, or even, millions of people accessing it. If you are designing an email system for enterprise level use, avoid databases for the email storage at all costs.
---------
---------
Swearing is the crutch of inarticulate mother fuckers.
Hmm I'm using it with Netscape 4x, Mozilla and Opera. What's your problem?
Ring brother, ring for me | Ring the bells of hope and faith
Ring for my damnation | I am at the gallows end
At my work we used Exchange 5.5 on a P100 with 96Mb RAM. 150+ users, no speed problems at all.
Of course I would prefer some opensource solution, but my boss likes Microsoft :/
Ring brother, ring for me | Ring the bells of hope and faith
Ring for my damnation | I am at the gallows end
Yes, Opera does work,
;)
But, Netscape & Mozilla don't. In Windoze or Linux.
It may be the size of my inbox (which doesn't make a lot of sense, since it only shows the first few messages).
Other people say it works fine, others have the same problem. Even when it does work, it's much harder to use in a non-IE browser; I'm sure that's by design ("embrace & extend"...).
I don't know what the problem is, I gave up debugging Bill's code for him long ago
When I die, please cast my ashes upon Bill Gates -- for once, make him clean up after me!
I've got a great deal of experience using UW IMAP (home) and Exchange (work) IMAP servers simultaneously from my Linux client.
The Exchange server won't do secure IMAP.
The Exchange servers web interface only works with IE (it says it works with Netscape 4.x, but it neither works with 4.x or Mozilla).
For all the bad things I've heard of UW IMAP, I've been using it for about 3 years, and am very satisfied. It used to corrupt large mbox files occasionally, but I haven't had that problem in a long time (and that's repairable). It also used to have problems with multiple mail clients referencing the same mail file simultaneously. I haven't seen that problem in a long time.
For me, email is my memory. I have huge email folders that I constantly reference. It provides me with a bread-crumbs trail of all that I've done. It provides an on-line research archive for all the lists I belong to.
Secure IMAP is the best mail interface. My memory works from anywhere!
Without secure IMAP, I can only read work related email from work. Exchange sucks.
When I die, please cast my ashes upon Bill Gates -- for once, make him clean up after me!
Also, with MH-style mailboxes (and maildir, if I understand maildir correctly), you have the additional advantage of being able to search the folders using glimpse. Glimpse allows you to do efficient full-text searches (it's light-years beyond grep). It's only "disadvantage" is that you've got to build an index (database) of your mail messages. However, disk space is cheap, and I have a nightly cron job that auto-indexes my messages. It's VERY nice.
Note that exmh contains glimpse support. Check it out.
Yeha, but those of us who use OpenBSD should all be used to rants anyway... Arawak
It's nice to see the efforts being made on the front-end, but now that most of us have huge HDs and SQL, why should we be using MUAs that start crapping out with a few thousand messages per folder?
How the hell is HTML supposed to crash a browser? That's a fault of the browser, not the HTML. Would you use a browser that crashed every time you ran across some guy's personal homepage because he forgot to close a couple of tags? Sometimes browsers just crash. Netscape especially.
murphy's mail law: no matter whatever mailbox you use, the other one tends to crash less often. or, the grass is greener on the other side. agree with me then go here else, here
I have more than 5 years of mail, instantly accessible, either via web browser, grep'ing the mail files or IMAP mail program. All of it is full text searchable even though the outgoing link from my mail server is only 128kps (@#@!#@! @HOME!)
:-)
Easy to back up, easy to restore. Easy to convert between format, Cyrus rules. Plus it's fast & works great w/ GUI mail programs.
People see pine open my 3500 msg inbox & ask me why I don't delete messages. The reason is simple, it takes about 2 seconds to open that mailbox.
My archive mailboxes tend to be much larger.
With the rpms available on redhat's powertools CD's it's also pretty trivial to install, just rpm, then reconfigure sendmail & you are set to go. (Though sometimes you need to play w/ cyrus's authentication library a little)
-Jay Thomas
http://www.uiuc.edu/~jthomas2
My mailbox and those of four neighbors were recently crashed into by an SUV. Pretty impressive, 4x4 posts splintered like matchsticks! :-)
Not exactly a DoS attack - we got hand delivery to the door for a while. (Special delivery of junkmail - oh boy!)
I put the new box next to the telephone pole for protection - makes a good 'firewall'
Confronted with this same problem, I wrote a program to synchronize sets of mailboxes. It's a little like rsync, but it considers the messages individually, instead of treating them as files. It will propagate deletions as well as new messages, and it treats the two mail stores symmetrically. It is based on the UW IMAP c-client code, so it can handle many formats; you can then choose your format based on your MUA's preference or performance considerations. I've used it to duplicate my mail on several machines at once. You can also keep your mail in multiple formats on the same machine. http://mailsync.sourceforge.net
is to just pile all my mail, opened and unopened, on a table in my apartment. Makes it hard to find things, but at least I know its all in one place.
Going on means going far
Going far means returning
Going on means going far
Going far means returning
I keep all mail on one machine, to which I ssh when I need to access mail. :)
Then I use mutt to read/write mails, which I can highly recommend (especially if you like 'vi'
In your mail profile you can replace the servername with it's IP address and it speeds the startup quite a bit...
"Bugger this, I want a better world." - Jenny Sparks
Other people have already pointed out the usefulness of glimpse, so I won't stand on my high horse about that.
./ -type f | xargs -n 1 grep 'whatever'
But xargs is your friend if you're worried about truncation of wildcards:
find
--
no sig
this is a sig.
I'm still waiting for nmh to support maildir, which is similar to but better than mh folders. On my main server I use exim with UW imap, which has ok mh folder support. I have a secondary server running postfix/courier, both with lovely maildir support. My only fault with maildir (this applies to mh folders as well) is having many small files being an inefficient use of drive space. I often archive old email into mbox format for this reason. For email clients, I use nmh and mutt (which supports everything) locally, and OE with IMAP whenever I am in Windows.
Because it makes harder for root to read everyone elses email?. Actually as a sys-admin myself i'd love to have mail stored in a way where only the recipent can read it. I don't want the responsiblity and i don't want the chance of management asking me to pass them the bodies of other peoples email. Todays e-mail is way too insecure and laking in privacy. I think e-mail should be automatically encrypted end-end as standard. All mail-clients should include public key encryption, and all Mail transports and temporary stores should use further encryption.
Instead of simply reformatting your drive how about using ReiserFS? No more problems due lots of lots of small files. Performance should be much better with very small and very big files. Of course I assume you're currently using ext2 instead of ReiserFS, XFS etc. What comes to mail(1) it shouldn't be that hard to fix. In fact I'm pretty sure there's replacement already.
_________________________
_________________________
Spelling and grammar mistakes left as an exercise for the reader.
yeah a few scripts would be nice. For us it involved writing a few scripts, and pulling a lot of our hair out in the process. a few nights of leaving the office at dawn to return at 9 am... but it got done.. I'm sure someone with more experience could do it pretty easily.
One of the first things i did in my current job, was from sendmail to qmail and Maildir format. If you have to deal with a lot of annoying clients with email problems in a day, its a Great change.
.avi file. He doesn't want to download the .avi, but gets dangerously angry when you suggest wiping out his mailbox and starting fresh. With maildir format, you can simply erase the large one, and the others are still file. This also applies to people that want certain files removed due to virus protection programs flagging their messages before they download them.
My main reason is simply this: Joe user gets 20 emails in his inbox, and one of them has a large (5 meg)
There are a few other purposes, that are pretty useful... You can tell, at a glance, how many mails any given user has in their box at any given time, without going in to any files.
All in all, The maildir format has had amazing results for me.. simply being able to run a script that will erase any horribly large mails every few hours without wiping out entire mailboxes is an amazing power to have.
uw imap will never natively support maildir because one of the primary authors of it (and
the imap rfc, pine, etc) does not like the guy who came up with maildirs.
yeah, that's it. a personal conflict.
it's the same reason wget doesn't support publicfile's ftp server's listing format.
Ummm Buffer overrun on date field to execute arbitrary code from HERE or just the fact that "Enterprise level" software has a vulnerability like a scripting host in the default install. Perhaps the fact that the same company that brought you this HTA Exploit might make some one wary. To partially use your example just because all of this companys other lines of cars explode dosn't mean this one will. No really!!
Very simply, Maildir and qmail is a much stronger security typed system than Sendmail and the way it stores (mbox). This does not mean Maildir is better, but it does mean I use Maildir as my domain mail router.
Wait, Maildir SUCKS when it comes to supported apps. I also love to use IMAP and the ability to hook into it, and I run procmail to filter crap.
So... I have a personaly mail system using IMAP that I connect to, and I use Maildir to actually handle all the mail server stuff.
I can program myself out of a Hello World Contest!!
`head -1 /var/mail/$USER`
depending on what format/OS you're using.
-f
-f
www.blackant.net
Wow! what's so bad about IMAP? I read that very RFC and enjoyed it immensely. especially with the the XTHREAD and XSORT extensions.
Explain.
Yes, please, let's discuss this. I'm sure something could be worked out that might even prevent spam.
Er, this is FUD. Imap, most notably UW's implementation, has been the subject of several buffer overflows, but if properly patched is as secure as any POP implementation. And since IMAP connections can be protected with SSL (it seems that Messangers ssl over POP don't work.)
:wq
I am proud to use cscmail which stores its mail in a sql database like postgresql or mysql.
So when my mail arrive in mbox format I give it to cscmail which eat it and produce a very nice, fast secure sql table.
If cscmail stop working (it happened) I can reach my mail with: "select * from mail;".
I am happy now!
--
--
We have fought the AC's, and they have won.
I use postfix and Maildir. Postfix also supports Maildir and I like it better syntax-wise and config-wise than qmail. Not a flame, just a personal preference. I like Maildir cause I can impose quota on a box with multiple users and have their mail in their home dir fall under the quota. Also, I am sure someone will mention (if they haven't already) that Maildir works better when used with nfs (file locking) and mutt supports it. I also use courier imapd (which is the only IMAPD that I have found that works with Maildir.) and Outlook express works fine. Honestly, my /var is usually a lot smaller than my /home and I just find the concept of ~/Maildir per user to be a lot neater.
YMMV.
SealBeater
-- Its survival of the fittest...and we got the fucking guns!!!
This is FUD. Both IMAP and POP have alternatives to cleartext passwords. Pop has APOP. IMAP has several alternatives. Most modern clients support these alternatives.
Please don't spew misinformation - read up before posting...
It rocks...
(please God kill me)
---
This
I believe mbx format is better than both maildir and mbox. I think part of it is in binary, which makes it faster. Unforutantely, I was unable to use it in qmail because qmail doesn't support it natively. Ialso tried plugging in a module into qmail, but I had trouble doing it. So I resorted to using maildir. Maildir is faster if you have a few large files, but mbox is faster when you have a bunch of users with a bunch of small emails.
I don't know too much baout mail server though, but this is what I gather from other people
Where are unread marks stored, anyway?
In three places, actually. The unread list itself is a special, named object (whose name is the canonical user name) in the database itself. The unread list is a compressed list of unread note IDs. In order to gain compression efficiency, the note IDs chosen are specific to a given database replica, and not applicable to other database replicas. A cached copy of this unread list is kept at the Notes Client in DESKTOP.DSK, mainly for performance optimizations. Finally, a log of client-initiated operations to mark notes read or unread is kept at the Notes Client in CACHE.DSK - this log is called the Unread Journal. This journal stores UNIDs, which are applicable to all of a database's replicas. The proper conversions are done to convert note IDs to and from UNIDs when processing the journal.
Do unread marks replicate?
Unread marks do not replicate in the standard way that documents replicate in Notes. Attempting to replicate database unread lists this way would often lead to replication conflicts that an end user could not resolve. To avoid confusion, we say that unread lists are synchronized, not replicated. The major way that unread marks are synched is via the Unread Journal. Additionally, there is a NOTES.INI setting which tells the Replicator to sync unread marks during a client to server replication. There is no way to sync unread marks during server to server replication. The reason for this is that server to server replication does not run with the identity of a user or users for which to sync unread marks. Since many Notes databases have a large number of users who access them, synching unread lists for all users' unread lists in the database would be a serious drain on Replicator performance.
So when do unread marks work flawlessly?
Well, we have all learned not to say "flawless" when it comes to computer software. But we believe that all cases of a user using one Notes client system to access multiple replicas of a database works correctly in R5. We also believe that all cases of a user using multiple Notes client systems to access a single database (with no replicas) works correctly in R5. In the single client to multiple replica case, the Unread Journal assures that mark operations that were done in one replica are kept outstanding for the other replica until that replica is accessed, at which time all of the outstanding operations are "replayed" to that replica, thus synching the unread marks. In the multiple client to single replica case, the R5 redesign, coupled with the fact that the definitive unread lists lives in the database itself, keeps the unread list correct.
When do unread marks potentially not work, and what can I do about it?
When the same user uses two different clients to access two different replicas of a database and does mark operations in each, without intervening steps, inconsistencies in unread lists can arise. If the user were to subsequently access both replicas from both clients, then both unread journals would replay their outstanding entries to the respective replica, and the unread marks would then be consistent. If one of the replicas is a local replica on one of the clients, using the REPLICATOR_SYNC_UNREAD=-1 NOTES.INI setting on the client will force unread marks to be synched between the client and server replicas on each replication. Both of these mechanisms are workarounds to the problem.
If REPLICATOR_SYNC_UNREAD is set to a non-negative number, unread mark synching may not be done on each and every replication. The value of this variable is the minimum number of half hours between database replications which will synchronize the unread lists between local and server replicas. If you have scheduled the Replicator to run more often than the interval specified by this variable, interim replications will not sync unread marks. Use this if you don't want the performance hit of synching on every single replication. By setting REPLICATOR_SYNC_UNREAD to -1 (negative 1), you assure that each and every replication will sync unread marks.
The Unread Journal is a Notes Client mechanism only. Other components or API programs doing mark operations will not get the benefit of synching unread marks that the Unread Journal provides. They will certainly work correctly on the particular replica (or single database) on which they operate, but will not get the automatic replay of their mark operations to other replicas. The Unread Journal is also finite in size, though it does hold a large number of entries, around 20,000 entries to be used for all databases accessed by the client. If you go a real long time without accessing a replica, some of its outstanding marks can get overwritten as the journal wraps around with newer entries. As a result, the recently added or modified documents in the database will be marked correctly, but the older ones may not be.
Another factor is that a "Mark All" operation is one that affects a specific set of documents based on exactly when it is performed, so for complete accuracy it can not be journalled as a simple "mark all" indicator, but rather must be journalled as a set of mark operations. Because of the finiteness of the journal and the possible negative effects of wrap-around, Notes imposes a limit of 1,000 on the number of individual journal entries for a "Mark All" operation. A "Mark All" when there are more than 1,000 documents affected is still journalled, but slightly less accurately. It is journalled as a "mark all before this time" operation, which occupies just a single journal entry. This won't work quite right when the time of the servers hosting other replicas is off, nor will it work properly if operations performed on a partial replica are journalled and later replayed against a fuller replica.
Problems relating to the fact that the Unread Journal is not exposed outside the Notes Client can be worked around by using REPLICATOR_SYNC_UNREAD=-1 if one of the replicas is a local replica. Problems relating to the finiteness of the journal can be addressed by encouraging users to access all of their replicas occasionally, or to repeat mark operations on the other replica when it becomes necessary. At any one time the journal contains at most 20,000 entries which are the most recent mark operations across all databases accessed by the client. There is no per-database limit on the number of journal entries. Generally speaking, it takes a lot of Notes usage to cause the journal to wrap around, in part because of the limit of 1,000 imposed on "Mark All" operations.
The following feature may be obsoleted in a future version of Notes, and should be used only as a last resort for attempting to fix unread mark problems. The R5 Notes Client currently still supports the R4-style desktop user interface via the "Workspace" page. On the Workspace page, you can unstack replica icons, select two of the replicas, and select Edit - Unread Marks - Exchange Unread Marks. This forces the equivalent of REPLICATOR_SYNC_UNREAD, but it is more powerful in that it allows you to exchange unread marks at any time, and it allows you to do it between two server-based replicas.
There is no recording by Notes of the time at which a document was marked read or unread. Thus, the synching mechanism enabled by REPLICATOR_SYNC_UNREAD has no way of knowing how to resolve a read/unread conflict. Since notes are initially marked unread for a user when they are created, it is more common for a note to go from unread to read than vice versa. Thus, REPLICATOR_SYNC_UNREAD always resolves a conflict by making the note read in both replicas. Occasionally, this may not be what the user expected. The Unread Journal does not have this problem because, although it does not have precise times when mark operations were done, it does know the order in which they happened.
I use the MH mail maildir format. MH, nmh, xmh, exmh can all handle this. I especially like the fact that it uses the underlying filesystem for structure. You can grep through messages more readily, do misc. file operations on a single message. nmh is available on standard RH 6 disribution media. and easy to find with a search on the web.
Were that I say, pancakes?
I currently store all of my e-mail in a local mbox-style IMAP store in ~/mail/, so that I am not tied to any particular mail client. However, I am planning on syncing my mail across multiple machines (home, work, and soon a laptop) so I need to have mail in a form which can be synced easily. MBox is bad for this because if I grab mail on one machine, and later delete some mails from the same folder on another machine, then sync, the new mails will be lost. This is where maildir is good - each message is a separate file. But why do so many people hate it? If I do change over to mailbox, what IMAP/SMTP servers should I use? A hacked sendmail/UoW IMAP? Courier-IMAP + QMail? Something else? How do other people keep their mailstores synced across many Ignore this bullshit I'm just testing the karma scores machines, and what software do they use?
That depends. Did you use your own data store API, or use one which is already becoming a standard? If you used your own, then I am not happy. In order to create the largest interoperability, someone will have to write an interface between your API and a more widely used API; Or, a module must be developed for each db, which seems wasteful.
Then again, I haven't yet done a lot of research into what's out there. A quick search indicates:
Again, I really don't know how far any of them are along.
Also, to those who say they don't want to install a RDBMS to store mail; Yippee. There are a number of small, free, and functional SQL databases these days. MySQL, for example, is LGPL, and if your product does not require it to operate (IE, if you're supporting more than one RDBMS) then you don't need a license -- Which is the idea. MySQL is small, fast, functional, simple, and easy to set up; A large MySQL db should be faster (one would hope) than a large set of dbm files :)
Consider my two cents pitched in.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
Why the Berkeley DB? Why not a whole bunch of databases? This seems really short-sighted to me. I personally would like to store my email in mysql, since I already have a MySQL db. I'm sure many others would like to use mSQL, Oracle, Sybase, or even M$ SQL Server.
I think the best way to go about this, though, is to develop a local delivery agent which will put mail in a real live database, preferrably through something like ODBC so that you can specify arbitrary databases, and then have a POP3 and/or IMAP agent which will serve the mail to clients from the db. I'd go for IMAP first, and worry about POP3 later, personally, because the idea (to me) is to keep the mail there, not to just house it there temporarily.
"You're right," Fisheye says. "I should have set it on 'whip' or 'chop.'"
- stores mail in users' own /home/directories (the default). no lusers whining about tiny little mailbox quotas, and no more admins whining about users emailing files to themselves to bypass said quotas.
.DOC format. Rather than giving each user a separate copy each, the mail can placed in a global location and symlinks created in each user's Maildir (automated by scripts of course).
- much fun can be had with Maildir and symlinks. eg. the CEO occasionally has a yodle to the entire company in
Maildir has nice parallelism for POP/SMTP-only access, but is a poor choice for IMAP.
The Cyrus mailbox format used by the Cyrus IMAP server is the best open-source format for IMAP or mixed IMAP/POP/SMTP access.
Easy to use, well integrated with other system tools. Very stable, both tools you can trust.
Just check the features !!
Exchange is great. But you'd be crazy to go out and buy a fresh copy from Microsoft for home use. Well, unless you have a thousand dollars to throw away. If you do, throw some my way, please..
/. is a commercial entity. goto slashdot.com
Personally, I like wooden mailboxes the best; you know, the ones that look like a little bird house?
Were I more cool, I'd put a little X.10 wireless sensor in it to let me know when it has been opened. I might link it to an e-mail/pager that says "You have mail!".
Just kidding. I've been using MH forever (late 80's). It's just so darn flexible for programmers to search their e-mail. I like my ability to search with Unix command line tools through zillions of messages I've sent/received to anyone. I've been using it since 1988. For MIME processing, mhn isn't that great, so I just bounce copies to Yahoo or Mail.com mailboxes so I can view them with my web browser du jour and then delete them. Instead of POP or IMAP, I suck my mailbox from a mail server using ssh and incorperate mail into MH on my local secure desktop.
Not so....From the INN (Internet News Faq):
CNFS is the Cyclic News File System, written by Scott Fritchie. It is a high-performance method of storing news articles, designed to avoid the high overhead involved in interacting with the file system when storing articles in individual files. From INN's INSTALL file:
CNFS stores articles sequentially in pre-configured buffer files. When the end of the buffer is reached, new articles are stored from the beginning of the buffer, overwriting older articles.
INN can either store it the "traditional" way or as described above, in a rotating spool file. Typhoon, Cyclone, and Breeze (from bCandid, formerly Highwind) all use a similar format to this. This WORKS for news. This WOULDN'T WORK for mail, unless you want a "most recent x bytes of mail" type of storage. Diablo does something similar, although I am not a diablo expert by any means.
This has less to do with the fs as it does with freebsd having faster file access, unless you are reaching the 1024 file limits of ext2.
As to jfs/xfs give me a break. These are NOT production ready filesystems. reiserfs has only the benefit of faster boot times. As for its file access times they lag dramatically behind ext2 in most cases. ** according to http://www.namesys.com/ **
Thanks for the tip! The developers of phpGroupWare will be looking into it ...
until (succeed) try { again(); }
until (succeed) try { again(); }
but really, wouldn't an Ask Slashdot story on the future of SMTP and its much-needed overhaul be more interesting?
...my 2 pesos
--
I know of several ISPs in town that use M$ for various things, but aren't exactly playing the licensing game fairly either.
apt-get install redhat please god - Me (take it easy, I love Debian)
People who live in glass houses .... well you know the line.
/var/spool/mail. ...
...
..."
... is inherently unreliable. ...
...
.... before you flame people for apparently not getting their facts right at least make sure you're on more solid ground yourself.
If you know so much about Qmail you would know that INSTALL.vsm and INSTALL.maildir which come with the qmail source have something to say about this. What in fact they have to say is this:
(INSTALL.vsm)
"UNIX has traditionally delivered mail into a central spool directory,
* It's slow.
* It's insecure.
(INSTALL.maildir)
"This file points out some reasons that you might want to switch from mbox format to a new format, maildir.
1. The trouble with mbox
The mbox format
Other formats, such as mh folders, are just as unreliable.
qmail supports maildir, a crashproof format for incoming mail messages.
maildir is fast and easy for MUAs to use. Even better, maildir works wonders over NFS
I don't want to cram maildir down people's throats, so it's not the default. Nevertheless, I encourage you to start asking for maildir versions of your favorite MUAs, and to switch over to maildir as soon as you can."
SO !
NOTE: I do not claim that the above extracts are true, merely that they exist and clearly indicate as suggested by the initial post that qmail does encourage use of maildir rather than mbox.
In fact it is because of the documentation with qmail that I use maildir in our offices although I use mailbox (qmail's version of mbox) at home.
Having experience with both formats and being a Linux user I prefer maildir because I like the way it keeps all my messages separately.
However at work where all our clients are windows users they wouldn't know the difference and wouldn't care as long as they get their email. I use the maildir format in this environment because of the claims of reliability.
I have a 1.0 GB .pst file ... maybe I should look in the mirror for why it's so damn slow!
sulli
RTFJ.
This works very well. Until the server admins decide to move me to another server and don't notify me! (oops) But I always tell people to do this, particularly for dial-up access, where for whatever reason name resolution is WAY SLOW on my network.
sulli
RTFJ.
Aside from the hacked/Kerberized version of POP (which not that many people support), isn't IMAP more secure by allowing SSL connections?
Ditto for Netscape Messenger. The problem lies in the 'features' of Outlook that let an attachment access the addressbook? Too many of these insecure features are what enemies call 'weaknesses' (which is a frightening prospect when you realize that the DoD and the US Government use M$ products almost exclusively...)
Although I agree that Outlook does have some advantages, my office wasn't very productive during the WEEK we lost while they tried to fix the ILOVEYOU mess.
No, it's more like saying I'd rather use a lower technology car because the high-tech car, which thought it knew I "where I wanted to go" kept driving itself into walls. Don't get me wrong, I appreciate the features of Outlook et al, but I can't stand it when software thinks it knows what I want. That leads to problems, especially when it starts opening holes in the system.
* Copyright (c) 1994-2000 Carnegie Mellon University. All rights reserved. * * Redistribution and use in source and binary forms, with or without * modification, are permitted provided that the following conditions * are met: * * 1. Redistributions of source code must retain the above copyright * notice, this list of conditions and the following disclaimer. * * 2. Redistributions in binary form must reproduce the above copyright * notice, this list of conditions and the following disclaimer in * the documentation and/or other materials provided with the * distribution. * * 3. The name "Carnegie Mellon University" must not be used to * endorse or promote products derived from this software without * prior written permission. For permission or any legal * details, please contact * Office of Technology Transfer * Carnegie Mellon University * 5000 Forbes Avenue * Pittsburgh, PA 15213-3890 * (412) 268-4387, fax: (412) 268-7395 * tech-transfer@andrew.cmu.edu * * 4. Redistributions of any form whatsoever must retain the following * acknowledgment: * "This product includes software developed by Computing Services * at Carnegie Mellon University (http://www.cmu.edu/computing/)." * * CARNEGIE MELLON UNIVERSITY DISCLAIMS ALL WARRANTIES WITH REGARD TO * THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY * AND FITNESS, IN NO EVENT SHALL CARNEGIE MELLON UNIVERSITY BE LIABLE * FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN * AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING * OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. What's your problem?
Watch this Heartland Institute video
Watch this Heartland Institute video
qmail does support mbox; in fact it does so by default.
mbox and to a lesser extent maildir, are supported by multiple mua's, so you could use a different mua every day.
what's missing is a cross-mua format for *indexing* these mbox's/maildir. indexing would speed things up *a lot*.
If you are using Outlook Express for your home email (as I was, since my wife wouldn't use UNIX), your mail is trapped in Outlook Express' proprietary binary format forever.
Unless you figure out how navigate through File>Export>messages , although that only gives you the option to export to Exchange/Outlook. But you can then import that into other 3rd party clients.
If Snapple is "made from the best stuff on earth", why don't any of their beverages contain jerked chicken?
With big IS shop managers always wanking themselves off spending gobs for the biggest baddest servers they can get their hands on, I don't see why any of this is a problem. Exchange 2000 coupled with Active Directory is a mean solution. Yes, I think AD is the best implementation of LDAP I've seen yet.
Qmail only supports maildir, and claims that mbox is slow and dangerous! Who is right? Why?
Qmail can use mbox, though mbox is definitely slow. Why? It's a flat file of untokenized data, wtf do you expect?
IIRC, it even crashed Netscape under Linux when I tried it.
/able/ to crash the browser, even if it is nasty MS-HTML.
I don't like Exchange anymore than you do, but Netscape is prolly the worst piece of software on *NIX.
To blame Nutscrape's crashing on Exchange is stupid - HTML shouldn't be
RG
----
Deepthroat my submarine, swallow my seamen.
Don't blame outlook or exchange - blame your companies infrastracture. Outlook uses whatever name resolution you have configured on your machine and it will use DNS if it is available. Try adding your Exchange server to your local hosts file. And stop using .pst files...
Like anything - if properly managed and maintained - exchange/outlook is as rock solid as anything else out there is.
Now if you do want to talk about message storage database formats, that's a different story. And, I'm sorry, the Microsoft JET Database isn't top on my list. Probably something MySQL-based, heavily indexed... Enjoy.
http://www.invisik.com
I have exmh running on my home system virtually all the time, and use mh remotely (via ssh) to keep up with things when I'm away. As long as I do a "rescan folder" first thing when I sit down at the exmh console, having mh and exmh sitting on the same maildir seems to work fine. It's a pretty handy setup. Now, if I could just find a simple web interface to my maildir...
Here at GE, we use an Exchange server and Outlook for our email software. So far, we've never had any problems other than it being sometimes slow.
I think the slowness is part to blame for all the chick sending their stupid forwards to all their friends. If only we could find a way to bust them without stealing their password...
...All I can say is that my life is pretty strange...
From what I've heard it's security model is better than qmail, but it's as efficient and clean as qmail is. Anyone who'se used both have any comments? (Preferrably comments regarding usage in high-volume environments.)
I use the plain old Mailbox format, but I have a script making a backup of this file every week. That way, the Mailbox file doesn't get too big, and it's still easy to view old messages.
I'm an administrator of both a sendmail and qmail server (qmail running Maildir); and have found Maildir to be absolutly wonderful. Very straightforward and easy to keep working properly; if there's a large message that a user can't download, simply navigate through the FS and remove it. I can't speak for any IMAP features, as we are only doing POP; but at least from the administrator standpoint, it is wonderful.
yes, but what about the off-chance that from is on it's own line in the body of the message, would that be recognized as a new message?
/* oops I accidentally made a comment, sorry */
So the question is when will they change?
========================== pipe(13) -- can you figure it out?
"It never seems to work quite right with browsers other than IE, at least in my experience. IIRC, it even crashed Netscape under Linux when I tried it."
I use Outlook Web Access almost everyday and I have run it from IE5, IE5.5, Netscape 4.x/6, Opera and Mozilla 0.6. It has some small problems in Opera (everytime a new window is opened the exisiting ones goto half size) but in all the others it works fine. Im interested if anyother people have had problems with it.
Jon
Umm. IMAP allowes for Authentication Extentions, like KERBEROS. To find out what extensions are supported you would use the capability command. Plain RFC2060 IMAP however does use plain-text authentication via login as opposed to authenticate.
Other than that, IMAP is different conceptually from POP. It allowes for server-side permanent mail storage AND management, which includes flags and so on. This means that you can synchronise messages between computers without any difficulties. There is also server-side searching and other nice things which make IMAP cool.
From the client side IMAP is thus far superior. Server side however...well the server has to store all the messages all the time. I could see definite storage issues over time (if you consider attachments and so on). Plus you have to rely on the server to make proper backups.
with...
"I'd go for IMAP first, and worry about POP3 later, ..."
First is support for The Berkeley DB package, then the developers or others extend to other DB's. Thats the way of software project development. You develop for your needs/requirements/enviorment then extend to other things.
'grep important *' may fail
'grep -r important .'
"Patriotism is your conviction that this country is superior to all other countries because you were born in it." -- GBS
"Enterprise-grade messaging" goes far beyond what Citadel will ever offer. IMAP is far advanced beyond POP3, but has a long way to go when compared to Microsoft's MAPI.
Exchange 5.5 and 2000 are leaps and bounds ahead of the industry in terms of messaging. Issues I see w/ Exchange architecture:
-Scalibility. An average Exchange server can only server upwards of 2,000 users until you reach the physical limits of what disk arrays can server.
-Administration. An Exchange server is far more complex than a POP server.
-Proprietary. It's really too bad that so much of Exchange is closed off and inaccessible for 3rd party vendors. How can we access the MTA data before it's processed by Exchange? You can't. What about "seeing" what the store does with your message before handing to the MTA? You don't. It's really unfortunate and I think Microsoft would be wise to open up the innards and let some outside people have a look. As always, there would be some real innovation and Microsoft could either 1) buy the innovator, or 2) use the idea themselves. It would only build on an already great product.
Obvious advantages to MAPI & Exchange:
-Collaboration. Calendaring, tasks, chat, etc. It's all in there and works fantastic.
-Industry. You won't get fired for being the IT guy that chose to go with Microsoft.
--Joe
Sorry - my work has lots of people running Eudora who tried to execute that attachment...nobody had problems except for the outlook users
Never underestimate the bandwidth achieved by a stationwagon full of DLTs travelling down the interstate.
I use SIMAP, prefer it over POP
"It's better to be a pirate then join the Navy"
What we did was that we maintain a cache that is updated each time the IMAP servers enters a folder and finds new messages, or status flags changes. This is trivial to implement, as you just needs to read whatever is in the new/ directory in a folder, and it gave us a tenfold increase of speed for some operations, while maintaining the simplicity and security of maildir (if the cache file is deleted, it's simply recreated the next time the user logs in, so we don't need to care about it).
Heck, I use the multiple mailboxes on multiple machines in multiple format principle. That way I can really say, "I guess I either never got your email or I lost it" when I need too.
What is pirate software? Software for inventory of stolen treasure?
ooookay. Who said anything about the mail client? He was asking about the format.
ignorance spawns fear, and the leaders are scared.
When I'm at work I set up a secure TCP tunnel, using SSH, to the IMAP port on my Linux box at home. This enables me to access my personal e-mail both from work and from home. All messages are stored in mbox format on my Linux box.
If I send personal e-mail when I'm at work I use the SMTP server of my employer, although I could set up another secure TCP tunnel to the SMTP port on my computer at home if I wanted that.
The IMAP port on my Linux box is blocked for access outside my local network at home, so security flaws in the IMAP server is not an issue.
This solution works just fine for me.
-- Tomas Hellberg
ls | xargs grep "search string" xargs is cool
If the poster was looking for a nice cross platform easily synchronized mailbox format. I don't think Exchange is going to be right for him, especially since he apparently runs (gasp) Linux.
I read the internet for the articles.
I think you forgot to read the message. The poster wants a mailbox format that he can download on a laptop/PDA or some other device not connected to the net, read and respond to email, and the synchronized the system back up when he gets back online (IE, copy over the mbox or maildir).
Exchange doesn't like that last time I used it, in fact it acts more or less like mbox in that respect, although it depends on how your mail is localy stored. Plus Exchange is pretty expensive for people who don't have large expense accounts and have to support a large base of people.
Finally, how does supporting POP and IMAP make you a "lot more cross platform than UNIX mail"? Especially since UNIX based mail systems can do the same thing (and can share mailboxes between other non-Exchange servers if need be). My biggest beef with Exchange is the binary message format. Just try to resurrect a slightly damaged file, or search/modify something without having to fire up your mail client or web browser.
I read the internet for the articles.
maildir's speed is far better under reiserfs. I can't say that there's no slowdown, but it's certainly much smaller. You can also do a significant amount of filtering based on filename, rather than mbox (where you've no choice but to grep through every message in the file).
Re the wildcard expansion limit, xargs can handle that.
ReiserFS has been designed to be easily portable. That Linux is its {current,primary} target means little -- throw $20K (or whatever they charge) at Reiser and his team and a port will get done.
It being that some organizations spend that much on a single server, this is pretty damn reasonable.
And he's entirely right -- my experience confirms that using ReiserFS makes maildir handling much faster than under ext2.
...you reinvented a wheel. In particular, the MAPI architecture.
While much M$ software is poorly designed, MAPI is an exception. MAPI is a pretty flexible, intelligent architecture for all things messaging.
MAPI allows you to do things like substitute message stores, address books stores, etc., by treating them as abstract components. Exactly what you're claiming to have done with your "data store API"
I don't want to be too critical, but I hope you folks looked at MAPI before you went out inventing another API...
Let's try not to let fact interfere with our speculation here, OK?
I just moved my qmail/courier-imap mail system from freebsd to linux (ffs to ext2) and performance is not so good when dealing with large directories.. I'd definitely recommend using an advanced fs on your mail server like sgi's xfs, ibm's jfs, or reiserfs. xfs is especially cool, because if the file is small enough, the file itself is stored in the inode and no space is allocated. That sounds great for storing messages to me! SGI also have created a redhat7/i386 installer cd, which allows for xfs-only systems (with 2.4) from the get-go. Tried it our last night, works like a champ.
as far as mta's go, does anyone know if qmail supports secure sendmail (using sasl)? I'm running an old version of postfix on my relays, time to update.
cheers,
-o
Actually, what it means is that you can use a high-performance, industry standard query language like SQL to extract data, instead of having to kluge together a patchwork of file and stream manipulation tools.
1) It needs to be *trivial* to use standard unix tools on the mailbox to find things.
e.g.,
rmm `scan
should remove all messages with badthing[12] in the heading, where f4 is an alias for
sed -e 's/\(....\).*/\1/' |tr '\n' ' '
[I'll admit that I was briefly worried the first time that this was
my reaction to a bunch of messages from a mailer gone nuts . . . ]
2) it would be nice for the system to be hostile to abusive mailings--not by content, but from the idiots that send plain text messages in html and mime. That's not a user preference; it's *wrong*.
3) Must be command line friendly. MUA's are for sissies. Real men read from the command line
hawk
Actually, the Cyrus IMAP Server is open source and takes a similar approach. It's been deployed and in use at Carnegie Mellon for quite some time, and I believe that the older version of the engine has the bases for a number of commercial web servers out there, including the iPlanet mail server.
sigs are a waste of space
It used to be that qmail was only allowed to be distributed in source code form, and the CDB database system (it's a cool thing well worth looking at) used a license that was somewhat incompatible with the GPL. There seemed to be some rancor to the effect that the GPL wasn't a "free" license; seemingly an independent recreation of the "BSD bigot" approach to software licensing.
If you're not part of the solution, you're part of the precipitate.
o_dev_t st_dev;
o_ino_t st_ino;
o_mode_t st_mode;
o_nlink_t st_nlink;
o_uid_t st_uid;
o_gid_t st_gid;
o_dev_t st_rdev;
off_t st_size;
time_t st_atime;
time_t st_mtime;
time_t st_ctime;
};
which adds up to around 48 bytes, and add to that the size of the directory entry that attaches to the inode.
It's not forcibly "ludicrously big," but it's space overhead nonetheless.
As for "flaming," it's somewhat unfortunate that Dan
If he tried to find some places for agreement, his software would probably get used more. Some of it's really very neat, cdb and the microscopic DNS server being particular examples...
The fact that he comes from a pretty strongly "pure math" background means that he comes up with substantially different ideas than most people. The PM factor adds in two particularly useful things:
If you're not part of the solution, you're part of the precipitate.
This may be important on a big mail server where inodes or disk space may wind up being scarce commodities.
There are then nontechnial issues.
The creator of Maildir , Dan Bernstein, is a, um, "somewhat prickly character." Take a look at his criticisms of Postfix for some mild material. Comparative discussions of Postfix and qmail have resulted in extremely inflammatory discussions. And Bernstein's attitudes towards the GPL seem similarly "inflammatory." This appears to have put some people off his software, whether rightly or wrongly.
Personally, I use Postfix as my MTA, and push messages through Maildir as interim step to pushing them into MH, which is only a fairly small step removed from Maildir...
If you're not part of the solution, you're part of the precipitate.
Exchange has its good points, this is true, but the biggest problem I have with it is that it holds my data hostage. I can't get at the mail spools if something dies and, if it does, you're fucked unless you also bought support contracts.
We've been running qmail + vpopmail for over 1500 people with Maildir formatted message stores without a problem for over two years now. When something breaks, I can fix it. Data is stored either in the database or in regular old files. It seems to work very well on a mediocre P2 and has all the good stuff: (A)POP, IMAP (courier-IMAP), selective relaying (relaying is allowed after a successful POP or IMAP authentication), user-run mailing lists (ezmlm) and web configuration (vpopmail has a web client). Oh yes and Squirrelmail for the web based mail reading folk.
There's one thing I learned early on and that's that I don't like having my data held hostage. The software I reccomend for the companies I advise for is pretty much any software is alright so long as either a) it's open-API b) opensource or c) I get copies (and updates) of the data formats. Surprisingly few companies balk at this.
"'Tis great confidence in a friend to tell him your faults, greater to tell him his." --Poor Richard's Almanac
Your commandline does not solve the problem that the original invocation of xargs was intended to solve - passing a *huge* number of files to grep on the commandline (grep * in a directory with a ton of files) causes it to break.
xargs works two different ways depending on how you invoke it.
is the equivilant to
or
Whereas invoking xargs like this :
is congruent to :
So, umm, there, and such.
</pedant>
--
filesystem level tool work well with maildir. you don't need special "formail" type tools to work wirh them, bash scripting is capable of doing it all by itself.
Yeah, being to grep to find a particular message properly is really handy - as is being able to kill all the messages containing 'University Diploma' with just find, grep and rm...
The other thing I've found in the past with mbox is that if you're really unlucky, the POP3 server will make a temporary copy of your (whole!) mailbox before doing a UIDL/LIST. qpopper used to do this at least, and you really knew about it when someone had a 30Mb mailbox. Maildir has a minimum of file shuffling and reading/rewriting.
"don't fall into the fallacy of believing that Perl can solve social problems. Maybe Perl 6 can, but that's a ways off"
The Citadel/UX project is developing a robust communications server that will compete with products like OpenMail, Groupwise, and Exchange.
On the face of it, this statement makes no sense at all. The big mail communications servers these days are the Internet MTAs, which in all the major ISPs handle typically many millions of messages per day on behalf of millions of customers per ISP. As others on this thread have mentioned, Exchange runs out of steam if you push it beyond some 2000 users per server -- it just doesn't scale, so it's not "Enterprise Grade" by any stretch of the imagination, it's out by 2-3 orders of magnitude. You've got to stop believing manufacturer's propaganda.
You should compare Citadel/UX to qmail or Exim installations in large ISPs, not against toy systems. Server farms with dozens of hierarchically-organized, multi-CPU MTAs which provide the massive underpinning to the world's Internet mail traffic, those are the "Enterprise grade" systems of today, not the relatively puny corporate systems of yesteryear being portrayed as "Enterprise grade" by manufacturers of personal computer software with more money than experience.
I feel I must also comment on your novel use of the word "robust". If one compares the reliability, availability and robustness of a flat file to that of even the simplest database system, the mind boggles that anyone could consider the database system as anything but the less reliable of the two by a collosal amount.
We run massive database systems here from the best regarded RDBMS manufacturer in the industry and configured with their help, yet even our DBAs will admit that the reliability of their databases is not brilliant. In contrast, the reliability of Exim is, er, well, it has never failed, so I guess the reliability is infinite. And I hear that qmail is likewise excellent in that respect. How the hell is a database going to improve on that kind of reliability and robustness?
Even the best databases crash and corrupt data every once in a while, and a new database could easily be less stable rather than more. But I've never had a flat file crash on me.
If it makes you feel any better, Unix is a sort of combined I/O multiplexer and storage mechanism, which inevitably makes it a particular kind of database too. To get the most out of it you should leverage its capabilities instead of trying to impose a totally different semantic on top of it. You'll never gain robustness by adding complexity.
"The question of whether machines can think is no more interesting than [] whether submarines can swim" - Dijkstra
However, you'll be happy to know that we've wrapped all of the database calls into a data store API. Recently we made the transition from GDBM to Berkeley DB without having to rewrite everything -- just drop in a new data store module and re-import the data (yes, there's an import/export utility). It would be quite straightforward for someone to write a data store module that uses MySQL, Oracle, or whatever.
--
Tired of FB/Google censorship? Visit UNCENSORED!
I'm very concerned about security, so I configured Courier-IMAP to ONLY provide SSL/TLS secure POP and IMAP. I set it up to provide insecure (non-SSL) service only on localhost (127.0.0.1), but not visible over the network. That way SquirrelMail or MUAs running on my server can get to it without SSL, which is OK because there's no way for someone else on the wire to eavesdrop. Of course, I also have the .htaccess file for SquirrelMail set up to only server over SSL/TLS (see below), and I don't allow telnet, rlogin, or non-SSL'd FTP. into my server.
I'm somewhat interested in developing up with a database back end for the IMAP server, so that old archived email can be stored more efficiently than either a maildir or mbox, but still be readily accessible.
# .htaccess for SSL-only services
# Options -Indexes
<IfDefine HAVE_SSL>
SSLRequireSSL
# insert the https: URL of the service in the next line
# for automatic redirect if the user attempts a non-SSL connection
ErrorDocument 403 https://host/webmail/
</IfDefine>
<IfDefine !HAVE_SSL>
# this is to make sure that if the web server is accidentally started without
# mod_ssl, the web pages won't be served up insecurely
Deny from all
</IfDefine>
refile +foobar `pick -from foobar`
will move all messages from "foobar" into my foobar folder in about 15 keystrokes (with autocompletion).
refile -link +foobar `pick -search project6` +project6
will refile messages in my foobar folder containing the text "project6" to my project6 folder using hard links. Now the messages exists in both folders.
I can type inc, show, next, comp, etc. in any terminal window at home or at work, and the right thing happens (with a few ssh tricks and gnuclient). No fumbling for some icon to click on, or waiting for the gui to come up, or finding the window running my mail agent...
The only drawback is that after a few hundred thousand messages scattered in hundreds of folders indexing the files for backup can take a bit of time, "what do you think I'm running here, a news server?"
maildir format does not scale well to large mailboxes on large servers because it has no sort of overview cache information. Mark Crispin (author of UW imapd) correctly deduced that MH format sucked for the same reason that qmail format sucks, and refused to implement it. Without a way to do overview information, getting headers to do the message list is excessively slow.
Yes, I'm aware of that. The problem is that it's dog-slow. Opening and scanning 2000 files for one mailbox alone is just darned painful. Even if the mailbox is hundreds of megabytes in size, 'grep' will operate on it faster if it's a single file than if it's zillions of separate files.
Also, when your mailbox grows to thousands of messages, the wildcard expansion in the shell ('*' in your example) may overflow or truncate, and you may not actually scan all the messages. Yes, you can resort to foreach, but then not only are you opening zillions of files, you're discretely launching 'grep' a zillion times as well.
Like I said, I admire 'maildir's reliability, and it's certainly more flexible in certain ways., and if I could get the same or similar search speed out of 'maildir', I'd switch. But for the moment, 'mbox' serves my purposes.
Schwab
Editor, A1-AAA AmeriCaptions
I have experience working for a company that hosted millions of users with a maildir format. There are some problems with it. First, some filesystems are just not built for having zillions of inodes and tiny files. WAFL, used by Network Appliance, can fail under this sort of load. Secondly, maildir file names can be quite long. There was a bug in a version of Solaris where the operating system would not cache file contents of an NFS-mounted file whose name was longer than 31 characters. This can result in very poor performance.
IMHO you seem to mix up a few thinks. You shouldn't compare Exchange and qmail. since the latter is written with the "do-one-thing-but-do-it-right" paradigm in mind. it is only an MTA, that means it delivers mail to your mailbox. while exchange does a whole lot more. using the right tools (e.g. courier-imap, procmail, fetchmail, etc.) you can get most (if not all) functionality you get with exchange. and each of those tools uses the DOTBDIR paradigm. you just need to combine the appropriate tools, and voila...
IMAP? courier-imap.
security? you have to make a tradeoff since you refuse to use proper products, there's no tradeoff using qmail-courier-imap-ssl-mutt-whatever.
workgroup facilities? there are a lot (Evolution, many webbased) so that's a moot point. you get everything.
Resources? qmail is a lot more resource friendly than Exchange...
still there's a tradeoff using OS-tools, you need somebody put all this together. a smart guy...
This combines the best of both worlds. This also means that while it's easy to corrupt your database with a single bug in your code, you can always re-build it from the on-disk messages.
Yes, it is great until the two get out of sync. If you can limit access to the raw filesystem, then that'll eliminate most of the problems, and most of the advantages.
Besides, databases are a lot better (these days) at storing large hunks of arbitrary data, so I'd just stick everything in the database.
That or use a future version of reiserfs, which could give you a database-like view of your filesystem.
What I do is configure maildirs for everyone on the mail server, using either qmail or postfix (both can deliver to maildir; qmail is more minimalistic but a bit confusing, postfix is about as good and a lot more understandable), and then setup qmail's pop3 daemon (even if using postfix to deliver). This combination has worked so well for me that I use it both on server and on my desktop computer (getting mail from pop3 with fetchmail, delivering into maildirs, reading with mutt).
The only thing to make sure with maildir is that you have enough inodes. But that's easy to handle when formatting the partition, and (even better) you could use reiserfs, which has dynamic inode allocation and handles large directories of small files very well.
--
Couple of considerations have to be made regarding choice of mbox formats. Here are my thoughts:
Flat mbox file:
pros: easy to set up, accessible.
cons: subject to locking issues,
not scalable, limited to local fs
Maildir format:
pros: fast, highly scalable, good
performance, very few locking
issues, reliable
cons: limited user access to directory
Proprietary db format:
pros: transactions, scalable
cons: expensive, corrupts easily,
word of warning:
backup frequently if you are
using MSexchange.
University politics are vicious precisely because the stakes are so small. -- Henry Kissinger
Not FUD: The Exchange guys at my old job were sharp, loved Microsoft products, and generally kept Exchange up.
Their points:
1) No version of Exchange had a stable message store until 5.5SP1. According to them, that's at least 3 years on the market, corrupting mail all along! But it does work fine now, and Ex2000 solves the '1 big database' problem.
2) They had weekly maintance downtime to handle the database issues. That meant they took turns coming in on Sunday mornings. Whoop for them.
3) Even so they still occassionally had niggling database consistancy problems which they never could quite work out. When these things were happening, people would get nervous because basically the server could crash anytime. Many times they had to go offline and restore the entire messagestore from tape to solve these things.
Meanwhile, I used to do some Notes stuff. Notes has it's own problems, but at least you could backup and restore mailboxes with the COPY command, as well as solve DB corruption and whitespace issues (which cropped up rarely) with the server online. I never had to come in on the weekends at least. But to prove this isn't FUD, I'd take the Outlook interface over Notes or Netscape any day of the week
--
Business. Numbers. Money. People. Computer World.
Courier-IMAP works with it. So every IMAP client works with it. Of course, mutt works fine with it, too.
I use exmh as my mail client. The mh tools use a separate file per message. Here are the issues with it as I see them:
Advantages:
* Easy to access any message with standard Unix text utilites (grep, more, and such).
* No worry about corrupting the entire mailbox if one message gets clobbered by a broken client (or broken file system or whatnot).
* Incremental backups and syncronization is easier
Disadvantages:
* Uses lots of storage. [Oh wait, I work for a storage company, so this is an advantage.]
* With one file per message, you can get more files in a directory than your shell will allow you to use as command line arguements. (e.g., `grep important *` may fail)
I guess the big safety issue is how well it behaves if you have more than one mail client accessing your email at a time. I don't see this as a very likely situation, but still something that should work.
I used to work in a place that stored about 20 terabytes of certain documents it worked with, which varied in size from 1K to 5G each. Median size was about 40M. All the meta data, like what customer it applied to, dates of processing, and so forth, were stored in a database. But the actual document file never was. The network path to the document was in the database, but the documents were stored on hundreds of Novell (ick) file servers. The database was still the major bottleneck of the whole operation. All these wonderful database facilities like SQL don't mean squat when the main functionality was to get the document, process it, and store it back, which is what happened most of the time. Of course it was nice to have the SQL when you needed to manually check on things or do some odd searches. But I would never store bulk data in a database; only the pointer to it would go in there. Databases are faster at complex searching, but not at bulk delivery of data.
now we need to go OSS in diesel cars
Why are they concentrating so much in a single box like that anyway? Why not a few separate smaller boxes?
now we need to go OSS in diesel cars
You have got to be fucking kidding me!
I haven't used Exchange 2000 but Exchange 5.5's mailbox format is a piece of shit! Its one huge flat file. And I mean HUGE. Plus the "Jet" database format it uses is slow as balls. And to top it all off, you have to take the service offline to defrag it! Unless you love getting up several Saturday mornings a month because your users can't check their email, then exchange isn't for anyone.
-Lee
-----BEGIN GEEK CODE BLOCK----- Version: 3.12 GIT d? s: a-- C++++ UL++++ P++ L+++ E- W++ N o-- K- w--- O- M+ V PS+ P
That or use a future version of reiserfs, which could give you a database-like view of your filesystem.
I find future versions of trendy software to be pretty impossible to use in building a workable solution...
"The future's good and the present is nothing to sneeze at." - Roblimo's last
Three different string encapsulations. That should be enough to tag it as bletcherous.
-russ
Don't piss off The Angry Economist
Doesn't seem to hard to me. As far as consistency checking goes, you can ignore the on-disk text except for displaying the message. If you want to use the headers from the file to refresh the database in the event of coruption, fine, but it's not a big requirement.
Any backups of my data that I keep are also stored in the same physical universe, but I don't use this as an excuse not to keep backups. Having the headers lying in a plain text file to sanity-check against can only help. Generally when one assesses risk, one works with cost/benefit tradeoffs. What you propose is very costly in terms of database resources, whereas duplicating headers on disk is very cheap. This cost comes in terms of disk space, time used to duplicate the data (which in a very large system could be staggering for every message body), etc.I think you will find that the benefits of storing headers twice will far outweigh the cost of having done so. I can't say the same for storing open-ended (in terms of size) message bodies in a relational database.
Nice idea, but we're talking about software design here, not system administration procedures. Clearly a sysadmin should be backing the data up, but to tell the user, "something looks odd here, go chase down a sysadmin and make him restore a backup," is a lot less friendly than, "I found some courupt headers in message 501719, correcting..."UW IMAP has different requirements. It doesn't just place mail on the system and leave it at that; it needs to read all of the mail that is there. Maildir will suck for that task. With maildir the IMAP server will need to open every single file and read some info and then close the file again. If you have a folder with a couple hundred emails in it it will very quickly thrash your system(100s of opening and closing of files for just a couple bytes of data from each one). MBox on the other hand, you just open one file per folder and parse through that. For the delivery this can cause problems because you need to open it and append to the file. For reading it is much easier on the system, but you take up a lot more memory and have the possibility of corruption(if you are careful that is a pretty low possiblility).
Except that mail "readers" don't just read. They also do things such as add metadata, delete, move mail around, etc. With mbox metadata is commonly done through adding extra headers into the existing file inserting stuff into the middle of a file is expensive as well as meaning that anything other than exclusive access probably isn't possible. With maildir it's simply a matter of renaming the file. To delete a message with mbox you either have to leave holes in the middle of the file (and "compress" it later) or rewrite as you go. With maildir simply delete the file. To move with mbox it's a matter of a file append followed by a delete. With maildir it's simply a rename.
For maildirs, you would do a mv(1) into the tmp subdir, which is essentially free, rather than qpoppers copy to a different fs (/tmp) which is slow and expensive)
Actually the latter is probably even more expensive since it isn't a simple matter of copying the data a chunk at a time from one file to another. The code doing the copying needs to look at the data being copied, either generating an index or verifying an index... As well as adding metadata by inserting extra data into the file (or the copy).
It abuses the filesystem with one file per message in the same way that mh folders do.
/var/spool/news must really "abuse the filesystem" then. Odd that in nearly 20 years noone has come up with an alternative.
Guess
Unless you're running a decent btree structured filesystem like XFS, ReiserFS or JFS, expect a performance hit if you get thousands of messages in a single mailbox.
Expect an even bigger performance hit if you have lots of messages in the same file. You must use lots of locking (and it must be reliable otherwise the whole thing will get corrupted). Things such as index files must be understood by every piece of software which does anything with the file, etc. Effectivly you will end up trying to enumatle a file system in user space software.
1) it is more reliable over nfs. Maildir is designed to not need file-level locking, which sucks over nfs.
There is another consquence of this maildir supports an arbitary number of processes reading and writing at the same time. The mailbox format requires complex locking, even then adding new messages has to be strictly serial.
Maildir is also a better analogy with paper mail. Mailbox would be something like you have a scroll of all the messages pasted together which you periodically have to hand to the postman for more bits to be stuck on the end...
The other thing I've found in the past with mbox is that if you're really unlucky, the POP3 server will make a temporary copy of your (whole!) mailbox before doing a UIDL/LIST.
It's not just pop3 servers which do this, indeed it's almost the standard way of processing a maildir file.
You CAN use NFS, if you want -- without getting real paranoid.
You could even use SMB to access the mailbox from a Windows workstation. (Or at least you could if the software existed.)
A point which hasn't been mentioned is that accessing email from a workstation using file sharing (the same file sharing which is in use anyway) means no need for additional password entry (or storing passwords in plain text/reversable encryption formats.) User simply needs to log in and there mail is there. If they log in on more than one machine everything still works fine too...
Actually, what it means is that you can use a high-performance, industry standard query language like SQL to extract data
SQL being so "standard" that software vendors demand a specific implimentation... Pull the other one, it's got bells on!
The only objection I can see the Linux camp having is qmail is released under a "non-free" (as in freedom) license.
The licence is likly to upset both GPL and BSD diehards. Also who the author is may be an issue too...
Both formats have problems. A true enterprise-grade message store will use an embedded database with transactions support.
.INI files. With the same problems, it's an "all eggs in one basket approach" and difficult to deal with when things go wrong.
Sounds like big iron propaganda... Both mailbox and maildir have the advantage of being conceptually simple. The database solution is complex, probably more complex than is needed for storing email in the first place.
Arguing for a database looks to me quite similar to the arguments as to why the Windows registry is better than
Using mailbox means that a problem with John's mailbox probably won't afffect Jane's. Using maildir means that a problem with one of John's messages probably won't affect the rest of them. Using some kind of DB could easily mean, John has a problem with mail, everyone has the same problem.
Coz if you put everything in one or two big proprietary boxes people can charge you a lot for consultancy and support.
;) ) Unix admins.
As well as these boxes being expensive, since they need to be reliable and hot swappable redundant. Not that even that will help when something external such as a router, cable, etc fails.
Whereas if you distribute the mail load to 10x the number of boxes (albeit cheap of the shelf boxes), you just need maybe one or two decent (backup/redundancy
Unfortunatly RAIC or RAIB dosn't quite have the ring of RAID.
With mail the load can be distributed. In fact I believe people don't really mind having their email addresses being user@tag.domain.com. It's the marketing/PR guys who'd complain. Heck market it as user@neighbourhood.domain.com
Assuming the distribution needs to be that obvious in the first place...
We had issues with databases becoming corrupt (and hey, 150000 users like it when they lose all their mail), the database being overly bogged down (guess what, fopen is faster than going through a database) amongst other things.
As opposed to one user out of 150,000 losing their mail with mailbox or one user out of 150,000 losing some of their mail with maildir.
While granted I'm sure that bugs such as these can be worked around
What's the point of applying a work around to get a complex system to work when you could simply apply a KISS aproach?
Sorry to tell you, but this doesn't work.
(storing email in a database that is)
I worked as lead programmer at a mail provider and was in charge of the system's design from the start. The ingenius idea to store email in a database, while it sounds good...is rather horrific. We had issues with databases becoming corrupt (and hey, 150000 users like it when they lose all their mail), the database being overly bogged down (guess what, fopen is faster than going through a database) amongst other things.
While granted I'm sure that bugs such as these can be worked around, databases were meant for holding fields of data, not whole files - especially binary ones (and before you say that email is ascii, thing other languages where they use multibyte encoding etc.)
this is basically X.400, upon which OpenMail is based. But instead of having a JDBC/ODBC/Whatever link to a relational database, the architecture is to have a mail-specific "mail store" which stores the mail in SOME way, and then client tools which just talk to the mail store (and client tools in this case are things like an IMAP daemon). It's basically this model, assuming that you are dealing with your mail store as a Mail-specific interface rather than raw SQL.
Wasn't there a recent article on a MySQL file system? Wouldn't this be the best of both worlds?
It's 10 PM. Do you know if you're un-American?
If you're using POP, typically the MUA downloads and removes all mail from the server. If that's the case with you folks, then you're taking advantage of the client's CPU and storage, rather than the server. That's swell if you already have heavy-duty clients with good backups and your people don't move around much. IMAP makes more sense if you need a central message store, though.
Both are reasonable choices, but it's unfair to compare them in an apples-to-oranges fashion.
I'd agree that "prickly" is a good word to describe Bernstein; a while back I wrote a chapter of a book on email and while I was writing it I had nightmares about him reaming me for a minor error. Luckily, the book seems to have escaped his notice.
Having used qmail for a few years, I can indeed say that it is a safe and reliable product. But I wouldn't recommend it for a novice sysadmin; DJB is a really smart guy, and he seems to have little patience for those who aren't.
As to his views on licensing, here is the distribution policy for his software. He strictly forbids distribution of qmail except in forms approved by him:
http://cr.yp.to/qmail/dist.html
Exchange defrags itself online, but deleted items become white space. The only way to remove that white space is by an offline defrag. But if you have your act together from the start (planning, esp. mailbox limits), tons of whitespace shouldn't be an issue.
Stop spreading FUD.
ostiguy
Your statement was true until about 12 months ago.
MS is going to enterprise style per CPU licenses with its xxxxx 2000 products. Exchange 2k will be pushed heavily at ISPs and especially ASPs. MS will expect companies to scale highly, with x,000's of users per box.
ostiguy
I can think of two advantages right off:
/Maildir/ instead of messing around with the MAILBOX file.
1) If your MAILBOX file gets trashed, you're out your entire e-mail directory. If one MAILDIR file gets messed up, you've only lost one e-mail.
2) If you get a messed up e-mail that you can't read in a mail program (and this DOES happen) you only have to delete the corresponding message in the
I know that UofW claims Maildir take a performance hit, but I've not noticed one. There's all sorts of web resources on tweaking UofW to pump out e-mails faster. I'm currently using Qmail + IMAP-2000 (with the Maildir patch on Qmail's site) on a P100 w/ 32Mb of RAM and I've got it pumping out IMAP as fast as my work's commercial server does.
Some people take their .sig way too seriously
A few points:
Tune your file system for what its used for. Your /home directories (where the mail will be stored by default) should be set to have a relatively large number of inodes because of a tendancy toward small files in there.
Read the docs on updatedb -- set the execlusions to include "/home/*/Maildir" if you wish.
Maildir also allows for multiple processes accessing a 'mailbox' because it uses per-file locking on per-message files, not a lock on an entire mbox itself. This allows for situations where 6 people all have the same IMAP shared folders for shared incoming mail (like an accounting office, or tech support) without locking problems for the MUA or IMAP server.
- Michael T. Babcock (Yes, I blog)
Use rgrep or GNU grep's -r option to do a recursive search:
grep -ri "slashdot" ~/Maildir/*- Michael T. Babcock (Yes, I blog)
Does someone want to explain how mbox is better for concurrent access than Maildir? If you do some good coding, they're equal. For Maildir though, you just do read locks on individual files in your Maildir when opening them to present them to the user, and you create new files to write new messages, which doesn't have any effect on (eg 25) other processes accessing that Maildir.
- Michael T. Babcock (Yes, I blog)
Take a look at Courier-IMAP. It handles Maildir quite efficiently.
- Michael T. Babcock (Yes, I blog)
What he may not realise is that distributiors and VARs are the ones who (usually) get the first calls from their clients. If those people (like RedHat) can modify the software (a la GPL) to behave like the rest of their software, they'll find it easier to support themselves. RedHat distributes a version of the Linux kernel with several patches added so that their customers will be happy (based on their own presuppositions). Does the kernel mailing list still get questions from those users? Of course. Does it take much to tell them to contact their distributor instead? No.
- Michael T. Babcock (Yes, I blog)
That was in 25-user bundles, new users (no upgrades)... I image there are volume discounts... my previous employer got Exchange 5.5 down to $37/mailbox at 150,000 user volumes...
Of course, $35 is the current rate for Exchange 5.5.
Meanwhile, Sendmail.com's advanced server (SMTP/POP3/IMAP4) is $3/user (500user minimum).
-- You can't idiot-proof anything, because they're always coming out with better idiots.
An even better solution is a length of 132 lbs (to the yard) rail.
:)
And a largish electromagnetic coil, to make a neato rail gun. Add some optical recognition software to identify the Rednecks on your video camera feed and away you go
I have to agree here.
I used the Washington Uni IMAPd on our branch mailserver for a little while, but it became painfully slow due to the mbox format it used.
Try this simple mbox experiment - delete one email from a folder containing 1000 mails, or a few emails with megabyte attachments.
with only 15 - 20 users the (admitedly underpowered - P133 16meg) server was on it's knees most of the time. It was rare to see a load average of less than 3
So I upgraded to a PII 266 with 64meg, and installed the Cyrus Imapd, and since then, the userbase has doubled, and since quite a few of us don't ever delete most of our emails (you never know when you might need to recall a conversation with a client from 2 years ago), the server now handles around 50,000 emails totaling nearly a couple of gig of data, without breaking a sweat. I've never seen the load hit 1, and nobody has ever complained about speed since.
IMAP is the easiest and safest way of making your mail available from any machine. And since the mail is stored on a central server, there's no synchronisation problems at all.
But if you don't want go to the (rather small) trouble of installing an IMAPd, or your favourite mail program that you just cant be without doesnt support IMAP, then I would say that maildir is the way to go unless you tend to delete messages as soon as you've read them, in which case it probably doesnt matter much. (although I noticed another comment that maildir works well over NFS due to not needing file level locking).
Advanced users are users too!
Now where did you got that load of crap from? Certainly not from the documentation, because those (QMail FAQ) state:
Is qmail compatible with sendmail? Answer: Yes. qmail supports .forward, /etc/aliases, binmail deliveries to a central mail spool in the usual mbox format, the /usr/{lib,sbin}/sendmail interface for mail injection, and the normal UNIX user database in /etc/passwd. There is a checklist for large sites moving from sendmail to qmail.
Do get the facts right before drawing conclusions whether something is or isn't possible.
To delete an email in mbox you have to read the entire spool file, then write it out without the one email. Very slow, very resource intensive.
Wu-Imap supports maildir, however the problem is that imap needs information such as the subject and sender from the header of the email. That brings you right back to open/close/read on every mail. Why bother?
Where I work we use wu-imap for pop. We didn't realize the performance implications until the last minute, then wrote a patch to the code that doesn't load the body of the email until it is needed. With that patch, you can't run imap, only the pop gateway but the performance is great. We might have used qmail or something similar if we had it to do over again, but at the time we had a lot invested in wu-imap and went with it, _with_ our patches. Otherwise wu-imap loads every mail from the mailbox into memory... kinda defeats the whole purpose of maildir. (btw.. we have 54k named users popping two 300mhz single cpu suns.)
Maildir also performs better over nfs. That's because 1) stat isn't expensive, 2) don't have to load the enitre file over the network just to delete one email, and 3) no locking issues.
Finally, note that most people who pop their email don't even have new email. This is the most important fact. Mbox format requires that you read the whole spool just so that a client can say, OK, I already have these emails. Very wasteful.
So go with pop, shun imap, go with maildir. Imap would be great if you had a integrated MUA/MDA which could save headers in a database or somesuch upfront. Otherwise imap requires too much as a protocol I think.
Bullshit. Yes, Outlook pretends to know where you're going and is often wrong but this virus has nothing to do with that. The ILOVEYOU virus was spread simply because users could simply double-click on an attachment to execute it. The difference between Outlook and other clients in this regard is that other clients make it a pain in the ass to do so because their developers never thought anyone would want to execute an attachment. So it's more like giving you a car. If people give you the directions to hell which you choose to follow, Outlook will simply take you there faster. Other clients might save you from going there because they make it too far away.
Mmmm.. Donuts
I have had too many security issues with IMAP. POP may suck, but it does so the same way every day.
Take a look at what sorts (and counts) of messages / files your users tend to have. Choose your mailstore / filesystem based on that.
--
Free Software: Like love, it grows best when given away.
Despite a campaign to get rid of the worst {ab,}users (many had 'leave on server' accidently turned on), there were still many users with mailboxes over 50MB (In one case, a customer with dialup accounts had a friend that sent them a digitized home video (3 copies). They never managed to get their email after that because they kept on concluding that the connection had timed out.
In any case, I could notice the system load go up when a user with a large mailbox tried to POP their mail. Even with server mode turned on, mailboxes regularly got thrashed back and forth. Had I been free to do so (and understood qmail at the time), I would have definitely considered going to maildir format for performance reasons. (we had customized the software to remove just about every other performance issue).
Problems that I can see with maildir:
- On NFS, directory searches tend to be expensive if you have 'full' directories. (then again, there are problems with POP and NFS
- You're going to eat more inodes (so reformat the drive already!)
- You have to open a file for every message... but you only need to read the beginning of it to get a message list (RFS, here we come!)
- The Mail(1) command won't work anymore (real nice for all sorts of customer support stuff).
- It's gonna hurt you if users have lots of small messages (they way that email used to be).
- Regular file commands can be used for all sorts of mailbox work (pointed out by someone else).
- you don't thrash files around whenever a user checks email, and/or deletes one or two messages (of hundreds)
- You CAN use NFS, if you want -- without getting real paranoid.
-
I see MailDir format being the biggest winner when you have users with mailboxes full of big messages (15
.DOC attachment anyone?)
I don't see how mailbox format can be a security consideration. I can see a specific program that uses that format having problems. Related, but different issues.On the other hand:
--
Free Software: Like love, it grows best when given away.
Let's see now: A medium-small ISP with 10,000 customers, at $87 each... that's almost $1M. (minus discounts). In that space, the cost of a dual PIII with 1MB ram and firewall is going to be trivial.
Bluntly put, that pricing is obscene..
Another thing: This is about mailbox formats, not software (though exchange would mandate their prefered format).
--
Free Software: Like love, it grows best when given away.
How does it work on something much bigger?
We have about 40,000 users spread over 6 POP servers (Pentium 600s or something). And we have people like me with about 1/4 million messages in our folders...
I'd love to see us do IMAP, but I think it would choke without racks and racks of servers.
Shut up, be happy. The conveniences you demanded are now mandatory. -- Jello Biafra
First of all, databsses can handle large amounts of arbitrary data, such as BLOBs or big chunks of text.
Secondly, I think you missed the fact that maintaining two separate data stores (database for headers, filesystem for message content) will certainly be more work than just using one or the other.
Lastly, a filesystem and a database are both stored on a magnetic disk (for the most part). How is it easier to corrupt one than the other? Back in the good ol' MS-DOS days, I lost data when BOTH copies of the FAT were corrupted. I've also had to manually rebuild NTFS partition data. In both cases, there were relatively small points of failure.
So, what do you do in th ereverse case from what you described? Suppose the portion of the filesystem containing the mail is lost. Can you rebuild those lost messages from the database which you would have store only the headers?
I say do one or the other, and then BACKUP OFTEN!
In addition to the advantages mentioned above, it fully supports accessing the same maildrop by multiple clients at the same time, which UW only partially does (the last time I looked, anyway). It also doesn't have the innumerable security problems that UW had forever that gave IMAP a bad name. You can send mail directly to a maildrop other than your inbox, if you set the permissions on it. You can even share your maildrops with other users on the same system with a nice ACL system. They've recently implemented a deliver-time filtering system (that I haven't used yet), which fixes the one drawback that it ever had.
You should definitely go with Cyrus if you think you will ever have the need to access your mail from multiple places.
Fuck 'im up, Tim! His views are invalid! -Pirate Corp$
Despite that, I'm cutting my teeth on qmail, well, because that O'Reilly's Sendmail book is just too fucking big for me to spend my valuable time reading. I may be getting a bit O/T here - qmail seems to be more like a "real" unix program than sendmail. Small discrete modules that pipe output to other small discrete modules, each mutually untrusting, instead of one all-inclusive behemoth of a program. There are also some easy to use tools for it like vpopmail, makes virtual domains a cinch. Some of the "big boys" are using it too, so it seems to be proven. Of course, qmail does not enjoy the what, 80-85% market penetration that sendmail does.
cat
>
/bin/grep: Argument list too long
./waters.roger/Bring_the_Boys_Back_Home
./shakespears.sister/_Hello_(Turn_Your_Radio_On)
./pink.floyd/Is_there_anybody_out_there?
/bin/ls | xargs grep "foo"
.: directory
./Roger_Waters-It's_a_Miracle.mp3: symbolic link
../../.mp3/3155
/bin/ls | xargs -i grep "foo" {}
Main Entry: pedant
obsolete : a male schoolteacher
2 a : one who makes a show of knowledge b : one
who is unimaginative or who unduly emphasizes
minutiae in the presentation or use of knowledge
c: a formalist or precisionist in teaching
Which one of these applies exactly? Because your post is incorrect.
>Your commandline does not solve the problem that
>the original invocation of xargs was intended to
>solve - passing a *huge* number of files to grep
>on the commandline (grep * in a directory with a
>ton of files) causes it to break.
Actually, it *does* solve the issue that xargs was intended to solve - shells have finite (often 100 or 255 character) commandline buffers.
For example:
bash-2.04$ find . -type f | wc -l
35416
bash-2.04$ grep -il "is there anybody out \
there" `find -type f`
bash:
bash-2.04$ find . -type f -print0 | xargs -0 \
grep -il "is there anybody out there"
I've done this on Linux, Solaris, OpenBSD, HP/UX, Openserver and Unixware without any problems.
> $
> is the equivilant to
> $ grep "foo" `/bin/ls`
This is almost, but not quite true.
First difference is that `/bin/ls` will be executed and substituted into the commandline buffer of the shell you're using, thus giving the possibility of an overrun.
The second difference is that you don't have to worry about escaping out special characters when using the backtick operators, but you do when using xargs.
For example, I have a directory containing a file with a single quote in its name. So this happens:
bash-2.04$ ls | xargs file
xargs: unmatched single quote
To work around this, its usually much better to use find.
bash-2.04$ find . -print0 | xargs -0 file
to
You brought up syntax like:
> $
This is mostly for cases where you need to repeat a command with a series of arguments, but again where the argument would be too long.
It is roughly equivilent to
for I in `/bin/ls`; do $I; done
Matt
Actually, if you want maildir support, you don't have to go with qmail. Postfix does it as well. While I haven't used qmail so I can't compare the two, Postfix is very easy to configure and get up and running. I gave up on sendmail after two weeks when I couldn't get it do what I wanted. Postfix on the other hand took me maybe four hours to completely install and configure to my liking. It also has the same benefits of qmail from a discrete module standpoint -- definitely not the big behemoth-all-in-one that sendmail is. Anyway, check it out at here. An article that I found to be helpful when getting started was here.
The program supports around a dozen different mailbox formats, easily user selectable with an ~/.imaprc file. Some are just legacy stuff, but a few are clearly suitable in different situations.
I think UW recommends the "mbx" format for most situations - fast, safe in concurrent-access situations, etc. Clearly unlike either UNIX mbox or Qmail maildir.
Of course, if what you REALLY care about is fast IMAP access instead of being able to bypass the IMAP server with client-side solutions, you should look at Cyrus IMAP server as well.
Ask a silly person, get a silly answer.
If I'm creating security problems for myself, I'd like to know what they are.
__________________
/. is a commercial entity. goto slashdot.com
I'm going to put on my Captain Obvious costume and state that the administrator has more to do with the system succeeding or failing than the package itself (Unless of course it's MS Exchange, then failure is assured).
I'm not trying to be cute, it's just that any system can be slow and dangerous if not properly administered. Get the best talent you can and give them the support they need; you'll end up with a good system.
He who joyfully marches in rank and file has already earned my contempt. - "Big Al" Einstein
Sound just like.... BeOS! Amazing what duplication exists in the technical world....
(Except BeOS has an advantage: the database is the filesystem, which brings unparalleled speed and stability, not to mention that you can edit the messages with a resource editor.)
I use mbox for no special reason. It's the default for sendmail, exim, postfix, procmail, and all that good stuff. In fact, I don't know what SMTP server defaults to maildir.
That's about the only reason, but let me tell you why I stick with mbox, instead of switching to maildir.
Clean trees. Remember, every time you run updatedb (or locate.updatedb if you come from FreeBSD), it's looking for files. The more messages you have, the more files it's got to hunt down. That means time and size. Keeping clean directory trees is generally a good thing, it helps keep things where they belong.
Inodes. Many people keep thousands upon thousands of messages. This means thousands upon thousands of files. Multiply this by a few horde-happy users, and you're quickly running out of inodes in your /var filesystem. Then you start getting horrible messages like "Could not create /var/tmp/tmp.file". Not a good thing.
Cluster waste. Don't get clusterfucked! Remember, each file needs to fill a cluster, no matter how big the file is. Thousands of files can waste significant amounts of diskspace.
The only disadvantage I see is filesystem damage. If you trash your filesystem, it is easy to wipe out a lifetime of messages by ruining your mail file. Maildir reduces the damage to (potentially) only a few messages. However, I think this is insignificant.
A new year calls for a new signature.
Databases are not without problems. Most importantly, they can easily leave you stranded with data in a format you can't read, they guarantee data integrity only under a very limited set of failures and will corrupt your data otherwise, they have their own set of bizarre resource limits, and they require considerably more complex software and maintenance.
It's also a common fallacy to assume that scaling up requires a database. But mailboxes of different users don't interact with one another. If a file-system-based mailbox system works fine for one users, it will work fine for each of 1000 users. In fact, in many cases, a database makes the problem worse, because if messages for different users are stored in the same database, using a database introduces interactions between users that weren't there before.
Both maildir and the standard mailbox format work fine and do scale up. They do have their limitations, problems, and failure modes. But, you know what, so do databases. In practice, in my experience, mailbox-based mailers seem to work more reliably than Exchange, so being an "Exchange-killer" isn't much of an advertisement.
As for mailbox and maildir based mailers, you can avoid almost all problems by turning on message size limits and per-user quotas; even very liberal limits will keep accidents from happening while not interfering with normal mail traffic.
...until I started implementing.
Example: Why does ENVELOPE exist? It just lists header fields that could be obtained from BODY[HEADER]. All it really does is provide them in a different format--is that a job the server should be doing?
Example: Why are subscriptions stored on the server? That is client data. What if different clients (of the same user) want to subscribe to different folders? What if IMAP is being used as a front end to an existing mail system that can't have mods made to it?
Example: Why does the RECENT flag exist? First of all, it's client data. Second of all, if it didn't exist at all the client is perfectly able to calculate RECENTness just be storing the UIDs from the last session.
And these are the larger issues. It's a huge mish-mash of bizarreness--it's no wonder it's made almost no inroads against POP3, whatever the user-side usefulness (and I have to admit IMAP should be nicer than POP3).
--
MailOne
Non-meta-modded "Overrated" mods are killing Slashdot
(Hey Ryan! Here's your proof!)
I've worked in the past with Isocor (now Critical Path) N-Plex Ultra, a high-end mail server product for ISPs.
One of the ways they reached very high performance (200 messages/second on a Sun E450, when a good conventional Unix mailer like Postfix or Qmail will more typically be running at 20 messages per second on equivalent hardware) was their proprietary message store organized as an object database.
Using this database and a multi-threaded server architecture, they could batch multiple email commits in a single disk write before acknowledging a SMTP transaction. The one file per message in the queue architecture of sendmail, postfix, qmail and the like means each email will require at least two disk writes (one for the file itself, one for the directory entry).
Obviously, using the standard Unix format has benefits in terms of ease of customizability, but there is a performance tradeoff to openness.
This is less of an issue than it appears, as there aren't that many companies or even ISPs that require such as high volume of email handling capacity in a relay, and those who do can easily expand capacity using a load-balanced server farm of cheap machines for much less than the licensing costs of proprietary software alone.
Cyrus http://asg.web.cmu.edu/cyrus/ seems to use a hybrid approach. Messages are stored in individual files, but the envelope information is stored in dbm format. So opening up a mailbox and listing messages is very fast. So is searching unless you want to do a full body search on all emails. Give it a try. It supports IMAP, POP, and LMTP.
If you look under the hood of the UW Imap server, you will see that it supports many more formats than straight mbox. I don't think that maildir is one of them, unfortunately, but there are a few (mbx comes to mind) that overcome some of the more blantant shortcomings of mbox.
Is UW Imap free software? If so, someone should feel free to give it maildir, db, sql, or other mailbox support. For some reason I seem to remember that IWImap was not free software, even though the source is available (some weird academic license hostile to commercial use?). The author is a good programmer and active in the standards process, but can be abrasive to work with.
Thus
might end up invoking, if you have thosands of files, something like and so on. Using the -i flag to xargs just means it has to create a seperate process for each grep, taking a lot of extra time.--
Ok. First off, Outlook is the client. Not the mail server. The mail server is called Exchange. Try not to mix the two. I can use Outlook against MANY back ends, including HP's Openmail, (almost) any IMAP/POP3 server, or no backend at all.
Second, you site three 'benefits' to Exchange:
Fast: Define fast. The Exchange/Outlook RPC is great over a 100MB network, but try it over a dial-up line, or some line with a high latency. They performance goes right now the crapper, because the protocol is very 'chatty'. The client and server communicate back and for repeatedly to get a task done. IMAP/POP3 are infinately better in adverse environments, because their protocol is 'batch' oriented. A couple of commands, and you have data streaming to the client. Another example is over that same high-latency connection, try forwarding a message with an attachment. The attachment has to be uploaded to the server before you can COMPOSE YOUR MESSAGE. On the server side alone, every internet message has to be 'decoded' into MAPI body parts for storage in the database. If it pukes on a body-part, it'll crash your information store. the IMAP servers do/can parse the messages based on MIME body parts, but that is only when necessary. Exchange parses EVERY internet message, and at a lower level that the MIME body parts.
Second, you site 'scalability'. I ran a 7000 mailbox UofW POP3 server on a dual 166Mhz Solaris box with 256MB of RAM. The concurancy was about 25%, and the server ran with a load-average of about 1.2. My previous employer is having trouble running 2500 users on a quad PII-450 with 1GB of RAM at a 50% concurency. How is that scalability?
Third, you mention 'workgroup features'. True, Exchange includes a fairly decent calendar service, this discussion is about e-mail. If you want to talk about workgroup functions, we can do that... (btw, voting is a client function, as it the task management. There is no true 'workflow' in that because there is no central process tracking the work. It's all source-routing/message updates.)
You also said that Qmail is technically correct, but it's not going to do my company's productivity any good. This may be true. But talk to me when your company starts to interact with OTHER companies, and tell me how well Exchange does. Internet software is designed for interoperability, and when you're dealing with other companies, THAT'S what will make your company productive.
As for security, I'll leave that to the rest of these guys. I already like the comment about the 5 days w/out mail due to the I Love You virus.
-- You can't idiot-proof anything, because they're always coming out with better idiots.
I ran into the "sync" mail issue a while back and came up with the following criteria:
1) I want to be able to read mail both from a GUI-based mail prog (Outlook, Eudora, Netscape, whatever) **AND** from a shell
2) I want to be able to access live and "older" mail anytime from (at least) home and work, preferably both my home and work email accounts.
3) I do not want to send any cleartext passwords
What I came up with is the following:
At home I run the UW-IMAP server, and store my incoming mail in MH folders. Stunnel does a fine job of adding SSL support to IMAP.
At work we run Netscape's Mail server which actively supports SIMAP.
Either at home or at work, both servers (and all the mail in all the folders) are available.
Just about the only thing missing is the ability to read my work mail from a shell, but that's where most of the big ugly attachments are, anyway...
I've been using 'mbox' for -- gawd, can I say this? -- fifteen years, and it's served me well. 'mbox's advantages for me are that it is efficient with disk space (you don't eat an inode per message), and that it is quick to search.
9 times out of 10, when I'm searching my mail, typically with 'grep', I'm looking for something in the body, not the headers. With 'maildir', you have to open each message and search it. This is preposterously slow. There is also the danger that the shell's wildcard expansion limits may be exceeded if you have a lot of messages. With 'mbox', 'grep' opens the one file and slurps through it quickly.
Remote synchronization is not an issue for me. All my email resides on my laptop, which follows me everywhere.
However, I'm hip to 'maildir's increased reliability. I have over 2000 messages in my outgoing box alone, and I'd hate to have a system hiccup destroy any of it. If I could search the bodies of a 'maildir' spool as quickly as an 'mbox' spool, I could be convinced to switch.
Schwab
Editor, A1-AAA AmeriCaptions
Originally, the reason we switched to maildir was that even without NFS, mbox was corrupting our filesystems. Not just the files, mind you, but the filesystems themselves. It was a total pain in the ass, and we damn near left Linux for FreeBSD. This was using 2.0.36 and Sendmail. We had to put /var/spool/mail on it's own partition so we could unmount and fsck it until we found a solution. Between that and problems with files > 500MB, my opinion of Linux 2.0 is very bad.
:)
Our solution was moving to qmail and using Maildir mailboxes for our users. We never saw the problem again.
Recently, I've switched to courier mail server (http://www.courier-mta.org/) on all my non-production machines to evaluate it. I'm really, really happy with it. Courier is a complete mail system, not just an IMAP server, so you might take a look at the whole package. The whole thing is RFC compliant, which causes troublte for software that isn't, but that's a fault in the other software.
As a final rant against UW-IMAP: I hate it. It loads the whole damn mailbox being checked into memory (regardless of the type), which creates a huge load every time someone with a large mailbox checks their mail. This problem affects the POP3 server as well, since that also uses the c-client code.
That's just plain wrong. Qmail supports both maildir and mbox. I've been using qmail with only mbox files for years...
ArsDigita has a great article on using Oracle as a backend for your mail and ACS as a front end.
Prevent email address forgery. Publish SPF records for y
"Mind, as manifested by the capacity to make choices, is to some extent present in every electron." -Freeman Dyson
My mailbox works just fine, and it hasn't changed in over 20 years! It sits at shoulder height just to the right of my front door. Here's the advantages:
-No encryption techniques neccesary
-rarely have to waste time with forwarded jokes
-Best of all, the spam it collects is occasionally useful (I know all the pizza deals available in town).
From hell's heart I fstab at /dev/hdc
As someone who is, as we speak, supposed to be implementing an IMAP server, let me say this: If the person who dreamed up RFC2060 says that X is "slow and dangerous" run, DO NOT WALK, to leap onto the X bandwagon--it'll be the wave of the future.
--
MailOne
Non-meta-modded "Overrated" mods are killing Slashdot
(Hey Ryan! Here's your proof!)
I think the guys who wrote Cyrus IMAP server got it right. I have been using Cyrus for about 4 years now and I rarely delete mail. The server is still responsive and full body text searches are pretty speedy, even on the P133 server that it is running on. I think keeping each mail in a seperate file, and making a directory for each folder is the way to go. It also makes it very simple to restore a lost mail message and to index the whole mailbox. Anyway,.. thats my two cents.
http://www.jwz.org/doc/
has a number of essays about mail on Unix systems, including problems with mail box formats.
I use Xemacs/Gnus/nnml so all my mail is stored as individual files, which is handy (as other posters have said) and has it's downsides, as they have said too (grep now bitches if passed all files in my main mail box). Still, I like it, best system I've used. Not so great for the multiple hosts thing though.
Or you could run your mail and xemacs on one machine, and either read your mail in a terminal, or open X windows on your local display. Look up gnuserve to do that, I think.
Plato seems wrong to me today
Both formats have problems. A true enterprise-grade message store will use an embedded database with transactions support.
Fortunately, a solution to this problem is being developed right now. The Citadel/UX project is developing a robust communications server that will compete with products like OpenMail, Groupwise, and Exchange. SMTP and POP3 are already in place; IMAP will be available by the end of the year. Web-based access works as well. After that's done we'll be writing plug-ins for both Evolution and Outlook, in order to facilitate all of the 'shiny things' working as well: calendars, address books, etc.
So, you might ask, what mailbox format does it use? None of the above. Messages are stored in a database, like they should be. The Berkeley DB package from Sleepycat Software (yes, it's open source) is used for robust back-end storage, including transaction and logging support.
I'd encourage any developers who are looking for the open source world's "Exchange Killer" to get involved in this project.
--
Tired of FB/Google censorship? Visit UNCENSORED!
Email messages are a specifically interesting topic. They're (for the most part) text, and tend to be larger than database fields want to be (on the order of 1+ kB each ranging all the way up to many megabytes in common practice).
This makes most mail messages poor choices for database storage (for example you want to be able to use "grep" on mail or compress in-place. Headers on the other hand are a major win in a database ("select messageid from headers where user = 'me' and date > yesterday and fromaddr = 'taco@slashdot.org'" should be fast even if I have tens of thousands of messages).
The easy solution is to keep the headers in the database, and then just keep maildirs with the original messages in the normal filesystem with the filenames in the database with the headers (something like message.headerid => headers.id and message.text is a path to the maildir entry for this message.
This combines the best of both worlds. This also means that while it's easy to corrupt your database with a single bug in your code, you can always re-build it from the on-disk messages.
Maildir is better because:
1) it is more reliable over nfs. Maildir is designed to not need file-level locking, which sucks over nfs.
2) maildir is more resistant to catastrophic corruption since each email is a seperate file.
3) maildir keeps metadata about the email in the emails filename, rather than a seperate index file. This helps prevent the metadata, such as "replied-to" and "forwarded this" from getting out of sync
4) filesystem level tool work well with maildir. you don't need special "formail" type tools to work wirh them, bash scripting is capable of doing it all by itself.
5) maildir is better positioned to take advantage of advanced new filesystems like reiserfs. when reiserfs has a plugin for file-level transparent compression, maildir will be able to selectivle and invisibly compess emails to the disk without requiring other programs/scripts to decompress them before use.
Study maildir, it's just plain better.