Slashdot Mirror
Improving Unix Mail Storage?
At first, there was mbox, then there was Maildir, and Bill begat Outlook and .mbx. CaraCalla wonders if there is a better way to store mail than the way we currently store it today. I admit, with the changes that email has undergone over the past 5 years (changes in what is being sent, not necessarily in how it is sent), it may be time to reinvent the mail format. Read on for CaraCalla's analysis of the current mail options, and his thoughts on where we may go in the future. If you were to design your own MUA, how would you design its mail storage?
CaraCalla asks: "Does anybody know a good, free solution for storing mail on unix hosts? The reason that I ask this question is my discontent with available techniques:
- mbox: There are problems with locking, corruption, access-times, and bloat.
- Maildir: Do you really want to clutter your system with millions of small files? That's waste of inodes, space (unless perhaps you use Linux/ReiserFS or SGi) and just try to open a Maildir with 1000+ mails and see how long it takes your favorite Mailprogram to only display the subjects.
- Cyrus: Basically the same as Maildir with database features.
- UW-Imap mbx: That's classical mbox with extensions allowing multiple access.
- Evolution: Basically mbox with database features.
- Windows clients: Typically some proprietary db-format. Pathetic.
But the thing that bugs me most is disk space. Typical inboxes are made of 5% to 10% of Text including Headers and HTML. The rest are BASE64- (or UU-) encoded pictures, word documents, zip archives and so on. The problem here is the encoding which wastes considerable amounts of space (at least one third).
Some ideas about the ideal mail-storage:
- One file per Mailbox-folder, allowing multiple folders per user. Should those files reside in one central location or in users Homedirs?
- Compression: Should messages be broken into pieces and the MIME-attachments stored separately (thus searching of the text parts would still be possible without decompressing the whole file)?
- File format: gdbm, Sleepycat db? Something new?
- Should the security model allow users to directly access their files, grep them, copy them around?
- Shared folders, virtual domains?
- Unicode support in folder names? Imap message-IDs, flags, useragent specific state-information?
- How would MTAs deliver mail? How would clients access? File-locking (NFS)?
- What about backwards-compatibility? Writing libmailstore (anyone)? adopting UW c-client?
Does my ideal mailstorage exist somewhere? Is somebody working on a project addressing this? Does anybody have some other hints? And please no mbox/Maildir flamewar!"
But put multiple indexes (by sender, subject, date, whatever key-classes you want to assign messages) and the possibility to restrict the range displayed. With careful programming, you can manage many users who won't be able to read each other's mail, except as required.
This way, you can arrange your mail as you please.
No more message duplication. Send a memo to 250 people? Just send it once, but tag it as readable by the 250 sendees.
Of course, this calls for an SQL database... :) :) :)
Well. My solution for storing ALOT of BIG email but still browsing fast is to use MySQL. My mail client is Pronto! (written by Muhri, in perl, gtk, etc).. I have several 10's of thousands emails in about 10 different folders. Reaction time is immediate and searching is pretty damn quick aswell.
The mysql server is at work, and I can view my mail from anywhere simply by pointing my client at my IP. Presto.
I'm also slowly writing a MySQL-based IMAP server which will hopefully be compatible with Pronto!... But as with so many projects, itl probably take some time to complete...
David
stuff
If you're so worried about encoded binaries, why not try yEnc instead of base64 or uuencoding? It works well in newsgroups. It might work well for email storage as well.
A single database to hold of the user's email. Single instance storage ensures that only one copy of any attachment is in the database at once, no matter in how many email messages it was sent in. API's for back up let you back up the whole database or individual mailboxes. And depending on your backup solution you can restore mailboxes and individual emails. Anti-virus software that integrates into the server side of the software. In Exchange 2000 if you accidently delete a mailbox you can easily bring it back with all emails without restoring from tape. Only files to worry about on the user end is a personal address book and archived email. Unless you use POP3 or it's archived in personal folders the email always stays on the server preventing problems like accidentaly downloading important emails you need at the office being on a home PC. And it's stable. Not as stable as UNIX I admit, but it stays up for months without a reboot. And in my experience most problems are solved by a simple reboot. In 4 yeas of exposure to exchange, the only non-admin related problems I've seen were 1 database corruption where I needed to run a utility and wait 45 minutes for it to work again. And a corrupted MTA that needed a reboot to get it working right again.
MS Exchange has been doing this for almost a decade now. Version 5 was the first decent one to get and 5.5 is great.
Something like Maildir .. if the FS is slow and can't handle that kind of application, then we need to improve our filesystems!
Lots of applications need lightweight databases with indexes, locking, and atomic operations. Why not bake this into the filesystem, and it won't have to be just for email, it will have many uses.
I was thinking about this the other day as I was working on a logging system for a large in-house email filtering system.. similar problem, except instead of storing emails, I'm storing small XML fragments describing the structure of each email and what was done to each. So far the easiest solution was large monolithic XML files, and an external index pointing in the large file (i.e., like mbox + a DB index). As it grows we'll probably have to move it to a "real" database.
There is a need for something like sleepycat DB + ReiserFS on steriods..
I've followed ReiserFS development for years now, shipping our first servers with it some two years ago (and every box we've shipped since then), and I believe they have the best long-term plan for this kind of thing. Hans has written some excellent white-papers on making small files extremely cheap.
The eventual goal of Reiser is a filesystem that is indistinguishable from a powerful database (if a special purpose database). The plan is to make small files so cheap that every extension of a file, directory, etc. is just another file. Another interesting turn is that files would no longer be, necessarily, of the form '/big/long/path/to/some/file'...because the filesystem is a database, one could also access it by a category, so that one file read pulls in all of the data of that category (from any number of files). Directories become just one view of the data available, with any number of other views possible depending on the application.
As was mentioned in the parent, this would lead to things like 250 email recipients and only one actual file. But of course, this leaves out the copy-on-write functionality needed to make this seamless.
So I think the solution is probably to fix the filesystem--not to fix the email storage mechanism. A number of very smart people have 'fixed' email storage in the past, leading to all of the options we have today, none of which works extremely well on really large mailboxes. Yes, many are good enough, and many work fabulously for small to mid-sized applications. But the day will come when they do not work so well, due to the higher volume and growing average size of emails.
A good place to start for information about these ideas (which are primarily a consolidation of the most interesting research in the field of filesystems and databases):
http://www.namesys.com/whitepaper.html
ReiserFS is good stuff. Give Hans' papers a read sometime.
BTW-Don't gripe at me about ReiserFS instability, etc. I know better. As I mentioned I've been shipping servers with it for 2 years, and we've never had a single ReiserFS-caused corruption. Not one.
The great advantage of the current system is that it is very easy to move your e-mail from one program or computer to another with little hassle and/or risk. With any type of database system, you introduce a level of complexity that virtually assures that only one e-mail program will be able to read your e-mail. I think the best solution as far as I am concerned is to just stick with current mbox format, but allowing attachments to be deleted independently though that is just personal preference. But I think we should be wary of adding any complexity that endangers the portability of mail. Also, the other thing to be said for the mbox format is that worst come to worse you can still access your e-mail with a text editor and/or grep.
/. punchingbag jwz has some strong opinions about using databases (etc.) for mail storage. I tend to agree: everything can read from and write to files, there no versioning issues, they can be easily transported among different operating and file systems, they can be backed up easily. But it's another wheel to reinvent, so everyone hop to it at once and then lose interest in two or three weeks!
Chris
M-x auto-bs-mode
I've been joking for years about getting two shell accounts on opposite sides of the planet and setting each up with procmail to bounce all my mail between the two (always rewriting the header so as to avoid a loop.) I figure at any given time, my mail would be in both places and neither simultaneously.
.procmailrc for a few seconds and change it back. Plenty of mail storage without chewing-up precious file-system quota.
If I want to read some, I'd just chmod
Some people have a way with words, and some people, um, thingy.
Hello! Why don't you talk to IBM about this? It's called Domino.
That is all.
Attachment Converted: "C:\EUDORA\ATTACH\NEW YORK.pps"
Click on that in Eudora and the attachment opens.
This keeps the actual text in the mbox file lean. I've got almost a decade of correspondence that totals about 20 MB, if it included all the attachments it'd be much more.
Also it allows you to edit messages after receipt, (this might trouble some people, but it just simplifies what I used to do by opening the mbx file in a text editor). I can select all the text, then paste it back in. This has the effect of removing all the HTML coding that is especially crufty from Word generated mail -- a 20k message reduces to 1k.
The next generation of mail storage should definitely work on taking optimal advantage of compression technologies. Preferably in a way that compresses the data from end to end, not just in the recieving mailbox. As to managing the kind of data sent, I'd suggest using a twofold approach. Save binary attachments in the natural state in a subfolder linked to the message itself, which would be kept in a compressed database format.
As to the database format itself, I'd like to see a form of redundancy in the structure of it. Give the design some self-healing ability in case flaws develop as the information gets shuffled around. Media isn't perfect, but mail stability should try and be as good as it can get.
If you want to speed searches, index the data in a seperate file and use that. Just keep the actual data storage as simple and reliable as possible, anything like searching or sorting is just a bonus.
My own pointless vanity vintage computing page
One file per Mailbox-folder, allowing multiple folders per user. Should those files reside in one central location or in users Homedirs?
Depends on how the user accesses their mail. If they read their mail only on the local machine, it should be in their home dir. If the server allows multiple forms of access (like local + IMAP), central storage makes sense. There's a lot of other issues here, like backup methods.
Compression: Should messages be broken into pieces and the MIME-attachments stored separately (thus searching of the text parts would still be possible without decompressing the whole file)?
No. Separating a single mail into its component parts is just asking for trouble (not to mention that it massively increases your locking problems).
File format: gdbm, Sleepycat db? Something new?
Personally, I like Maildir, since it lets me use standard tools like grep to find particular mails. I admit that a more efficient method is probably required these days.
Should the security model allow users to directly access their files, grep them, copy them around?
Yes, of course. It's their mail - let them do what they want with it. The mail app must be able to deal with that.
Shared folders, virtual domains?
Shared folders would be nice - IMAP can do that now, although it's overengineered and not necessarily fully implemented in any particular IMAP server. Virtual domains I've never had any use for myself...
Unicode support in folder names?
Why not?
Imap message-IDs, flags, useragent specific state-information?
As you say, IMAP does that already...
File-locking (NFS)?
More the fault of NFS than the mail software (and I believe NFS4 handles locking better).
No. This is Unix we're talking about, not Windows. Unix doesn't have any problems.
I have been pwned because my
Of course, with traditional UNIX file systems, this is a bit slow. The thing to do is to fix the file system, not to kludge ever more complex mail formats on top of it. ReiserFS goes much of the way; we now also need some system calls to open and read multiple files with a single call.
Until file systems catch up, one kludge is as good as another. UNIX mbox format is at least simple, so I stick with that.
People have been arguing about the balance between standard formats that are easy to parse and move between systems and complex formats that make searching easier.
What we need is a standard DTD or schema for mail data that all well written email systems can understand. If everything can import and export XML representations of email, the internals aren't so important.
Maildir : Do you really want to clutter your system with millions of small files? That's waste of inodes, space (unless perhaps you use Linux/ReiserFS or SGi) ...
... ...
... ...
In case you haven't noticed, the default settings for the Linux ext[23] filesystems is to allocate one inode per 4096 or 8192 bytes of disk space. Which happens to be pretty much the size of an average E-mail message. So, in other words, you are unlikely to run out of inodes before you run out of disk space, since both are going to be used up pretty much at the same clip.
It may come as a shocking surprise to some, but the average large filesystem is just littered with small files here, and small files there, all over the place. Here's my workstation -- a fairly large box with all sorts of crap loaded:
Filesystem 1k-blocks
/dev/sdb5 8159388
Filesystem Inodes
/dev/sdb5 1036288
I'm using up almost exactly 8192 bytes per inode.
and just try to open a Maildir with 1000+ mails and see how long it takes your favorite Mailprogram to only display the subjects.
How about instantly? Most GUI E-mail clients cache mail headers, so they don't have to go and wait for the server to reply each time you click on the folder index window to re-sort, or scroll the folder index.
...
Some ideas about the ideal mail-storage:
* One file per Mailbox-folder, allowing multiple folders per user.
Using one file per folder essentially forces you to use some form of locking each time folder access is necessary. Locking of any sort has been problematic for years whenever NFS (or pretty much any other network filesystem) is involved. A single circuit will now take out your entire network spool, as all clients are now spinning on lock requests out on the unreachable server.
Compression: Should messages be broken into pieces and the MIME-attachments stored separately (thus searching of the text parts would still be possible without decompressing the whole file)?
I thought you wanted to save everything in a single file per folder, and using multiple files for messages is supposed to waste inodes, remember?
File format: gdbm, Sleepycat db? Something new?
Ask an Exchange admin about joys of a corrupted Exchange database. If mail are stored in simple, plain, files, a single instance of corruption will affect at most one mailbox, instead of taking out the entire monolithic database.
Unicode support in folder names? Imap message-IDs, flags, useragent specific state-information?
IMAP already uses Unicode to encode folder names. Not sure what "useragent specific state-information" means...
I believe that UW-IMAP .mbx also includes indexing in the mail file, along with the concurrent access stuff. It's definitely WAY faster than mbox.
Now I'll never get to sleep tonight.
If it ain't broke, you need more software.
yEnc isn't all that great. See http://www.exit109.com/~jeremy/news/yenc.html.
Life is not that simple. All databases are limited by the size of the basic block, and if you can't fit your data into that block performance takes a hit.
With PostgreSQL this a compile-time option, default 8k and it can go up to 32k.
It *is* possible to store larger items, esp. if they're 'TOASTable' or blobs, but this often just pushes the problem of dealing with thousands of files onto the database. Only now it's a lot harder to figure out why performance sucks.
Does this mean that database solutions won't work? Of course not. But it does mean that simple solutions won't scale well when you're dealing with massive amounts of data.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
i concur. there's nothing wrong with maildir or the linux filesystem, at least for me. my mailbox has about 3000 messages, and it opens pretty much instantly, using Maildir, Courier-IMAP and EXT2, from a server running a 700mhz Athlon and 7200 RM IDE disks.
the author's comments about Maildir make it sound like they've been using it and having problems. perhaps the problem is with their imap daemon? or their client? or their hardware? if running out of inodes or space for small files is such a problem, why not use ReiserFS? reformatting your filesystem is probably a lot quicker than inventing another new UNIX mailbox standard and getting people to support it.
i use the OS X mail client, and it indexes my messages in the background as they arrive, so i can do instantaneous (i mean in-stan-tan-e-ous!) searches through my 3000 message mailbox by subject, to, from, or the entire message text. i can't imagine how this could work much better.
your experience is clearly different; but i think there are other factors you should consider before blaming the mailbox format.
You can go a step further - don't bother with setting up a new compression layer, just encrypt it with existing tools. Most encryption routines compress it first, to make cryptanalysis more difficult (and for performance, since there's less data to encrypt), but this is partially offset by the continuing need for 7-bit safe transport layers.
For every complex problem there is an answer that is clear, simple, and wrong. -- H L Mencken
Check out the Citadel system. (Disclaimer: I am one of the developers, so my opinion on this is kind of strong.) We use Berkeley DB from Sleepycat Software for the data store. Yes, this is the same Berkeley DB that Sendmail uses to store its alias tables, access tables, etc. But it's capable of so much more than that. It's a robust, non-relational database that is hugely scalable and even has transactions/logging support!
We store all messages in the database.
Works like a charm. No pounding through ugly directory hierarchies or insanely long flat files. No need to escape out the word "From" when it appears at the start of a line. None of the cruft.
Ok, so it's a black box. But it's an open source server that uses an open source database backend, and since it supports SMTP/POP/IMAP plus webmail all by itself, you can still plug your favorite utilities into it (Pine, elm, fetchmail, etc.) and you don't have to graft together Sendmail+IMAP+whatever to make your mail system work.
The traditional Unix mail utilities are getting a little long in the tooth. I'm going to get flamed for saying this but look at what's happened to the email world: Lotus and Microsoft have run away with most of the market because Unix traditionalists won't give up their flat files. It's time for us to evolve, folks.
Tired of FB/Google censorship? Visit UNCENSORED!
So far I have blocked about 94% of the SPAM coming in through our mail server. It only misses a couple and is highly configurable! Download and install it!
OFFTOPIC!? With that great deal of spam reduction, the space required to store the emails is greatly reduced!
Cheers!
Tom
http://tomgould.com/
You're newfangled system better have DRM built in. I don't have any data, but it must be obvious that artists are losing billions in revenues every week due to mp3s being sent as attachments. This criminal behavior must be stopped or the practice of free expression will come to a screeching halt.
We have to have the remote hammer to pop out of the monitor to whack the end user. This is a must for any admin that works with more than 300 people. Hammer trigger from e-mail, pager, SMS, or telephone number.
Power mains must be connected to the user's chair, see above for trigger.
The MUA must forward all p0rn to the admin account. Likewise with credit card info.
The MUA must know when the user is about to do something to tick off the admin, like sending a "me too" to everyone in the office, or replying to a confidential e-mail to the whole office and prevent the user from reproducing. X-rays are fine for impromptu sterlizations. The side effect of loosing all your body hair is no problem, as it alerts others to a stupid co-worker.
The MUA must alert the admin when a coworker he has got the hots for changes her home phone number. Just to be fair, if the the admin is female, the reverse applies.
The MUA must analyze the admins e-mail and throw a bucket of cold water if (s)he attempts to send a really stupid e-mail.
Also, the MUA must be able to launch nuclear missles at spammers automatically. After that, it should refer the e-mail to the admin to see if a stronger response is warranted. Better yet, the MUA should employ a time machine to go back and choke the spamming creep when the spammer is still a baby, then use X-rays on the parents as above.
The MUA should have a hypnotic effect on the object of the Admins desire and cause that person to preform disgusting oral acts on the Admins body each time a new e-mail arrives. (HOORAY FOR KELZ!)
For the PHB, he should (by the same hypnotic effect) do a "Full Monty" when the big cheese walks in. Twice.
The MUA should be able to cause back dated confirmation messages from HR approving a 51 week paid vacation upon pressing a special key combination, unless it's the PHB pressing the keys, then it should cause an e-mail to be sent to HR from the PHB's account turning in notice.
Sorry, if you had a day like mine, you'd need a laugh about now...
Necessity is the plea for every infringement of human freedom. It is the argument of tyrants; it is the creed of slaves.
What is the problem with Maildir? I mean if you're going to store email, might as well use reisterfs. I don't have any problems with big mail boxes, and the extra integrity of the email messages, are worth the (non-noticable) dealy.
Used to get the mbox corrupted once in a while. Never had problems with Maildir.
Je ne parle pas francais.
The problem with using database formats is that you can't access them with vi. How many times has your mail client crashed attempting to read an email, but you still _need_ to get access to it? If it's in a database (proprietary or not), you're up the creek. If it's stored in a flat file, you at least have the option of using vi/emacs/grep to find and read the email, and then excise it.
This has happened to me in Netscape, Kmail, Outlook, Evolution, Eudora, etc. Every single one has had problems at one point or another. The best programs are the ones that are _truly_ open, and let you get at the mail from other directions.
Don't doubt the power of the text utilities in Unix. :)
Jason Pollock
Courier IMAP uses an enhanced Maildir format (original Maildir didn't support subfolders)
Je ne parle pas francais.
I actually found a nifty little package called dbmail which uses an SQL messagestore. I've been playing with such things at work since they wanted me to write them a web-based mail client, and I wanted something which would let me deal with a MySQL database on the web client, but also allow people to connect to it via IMAP or POP3.
Of course, the whole replication part of it might be a bit more difficult, but it could probably be arranged as well. I'm pretty sure there are tools in existance for doing replication on a MySQL database (of course, don't ask me the names of any of them...)
Dogma: Dead (mostly because your Karma ran it over)
Shredding and compressing mail messages is almost always a bad idea. Essentially *nobody* does it correctly, and you can't reconstruct messages in their original byte-for-byte formats, which trashes digital signatures. You won't save much disk space, because real text doesn't take up enough space for anybody except a big ISP mailsystem to worry about, and binary attachments usually only compress well if they've been encoded in some non-8-bit-transparency format like base64 or uucode. About the only time it wins is when one person on your keep-mail-on-server mailsystem is sending an attachment to a bunch of people who can then all use the original, which is to say they should probably have stored the file on the web and mailed a URL. If you're going to do things like this, get yourself a compression-equipped filesystem and just store your raw mail messages there.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Why waste CPU cycles on parsing a human-readable text file format such as XML?
There should be a standard byte-compiled representation of XML (CXML), which has been flattened into an easily readable data structure. It would be portable, with byte orders indicated in flags (or would just use network byte order, i.e., big-endian), and with fixed-length element start/end headers, and could be used in lieu of XML for machine-machine communications. If a human wants to inspect the data, XCML could be trivially converted to and from XML.
Why go to the trouble of running a parser for files that 99% of the time no human will ever look at?
As part of my job, I've written software to send out HTML mails to people (no, it's not spam). When these messages pass through an Exchange server, Exchange does us the "service" of creating a text version of the mail from the HTML. I guess this is so that people without HTML-capable mailers can have a readable version...
The problem is, we include our own text/plain version alongside the HTML (ain't multipart/alternative great?). Nicely formatted and everything. Instead of leaving our mail alone, Exchange rips out the text version and creates a new one from the HTML. The result is an ugly mess of URLs because we use some graphics in the HTML version. Our nicely formatted text version ends up in the bit bucket so that Exchange can dump it's url-barf on people.
This is really stupid behaviour for an MTA. And for some reason, it's always CEOs of important clients who use text-based MUAs while sitting behind an MS Exchange server. They call us up asking which URL to click on.
This, combined with other mail-rewriting bogons, has lead me to the conclusion that Exchange has no respect for the messages passing through it.
Current versions of PostgreSQL no longer have such limits (they're much higher, a single field can use up to 1GB ...).
"I love my job, but I hate talking to people like you" (Freddie Mercury)
If you store your messages in an usenet server you get all kinds of neat features like auto expiration and tools that can put the binaries together and let the server deal with the file format.
Back when C-news was new, there was a systems called "Notes" that keep usenet posts in a database. From what I can tell that became the ancestor of lotus notes at some point.
Quite right. Just try it. You might be a bit surprised by the results.
News for Nerds. Stuff that Matters? Like hell.
A couple things:
:-)
1. Evolution is NOT "Basically mbox with database features". It can use Maildir or MH as the backend (and you can write your own plugin to extend this if you like).
2. Evolution's body indexing and summary files are extremely fast and efficient, about the best you'll get. I hear MySQL has text indexing capabilities that are extremely fast, but I'm not sure if they are faster than Evolution's indexer or not. Might be interesting to check this out.
3.
> But the thing that bugs me most is disk space. Typical inboxes are
> made of 5% to 10% of Text including Headers and HTML. The rest are
> BASE64- (or UU-) encoded pictures, word documents, zip archives and so
> on. The problem here is the encoding which wastes considerable amounts
> of space (at least one third).
It's theoretically possible, if you wrote your own Evolution storage plugin, to change the Content-Transfer-Encoding header value of binary attachments to "binary" (and text attachments to "8bit") before writing the message out to disk (or wherever) thus magically making it so that you no longer save the encoded text of the attachments but rather in-line binary data content. (Yes, it's as easy as setting an enum value in the CamelMimePart structure).
However, you have to be aware of the consequences of this. Most importantly, you will not be able to validate any of your PGP/MIME or S/MIME signed messages as according to the RFCs for these types, the signed MIME parts MUST be treated as opaque (meaning that you may not modify them in any way).
Now on to your ideas...
> One file per Mailbox-folder, allowing multiple folders per user.
> Should those files reside in one central location or in users
> Homedirs?
How is this different from mbox? (btw, CVS Evolution can handle mbox files and directory trees in external locations - ie, not within the
~/evolution directory).
> Compression: Should messages be broken into pieces and the
> MIME-attachments stored separately (thus searching of the text parts
> would still be possible without decompressing the whole file)?
If you break apart the MIME parts, you run into the same problem I described above about not being able to verify signatures.
However... if you took a normal mbox and gzipped it, you would certainly save space (at the expense of speed). I've been thinking about writing a CamelMimeFilterGzip class for gzip compresing/decompressing streams which would allow Evolution to read and write to gzipped mbox files for example.
Once the class is written (which should be fairly simple), allowing Evolution to read gzipped mboxes should be as simple as doing:
camel_stream_filter_add (MboxStream, GzipFilter);
...before feeding 'MboxStream' to the MIME parser.
> File format: gdbm, Sleepycat db? Something new?
Please not Sleepycat. If you are so sure that a generic database backend will be better than what Evolution's got, at least have the sense to use MySQL or PostgreSQL.
I'm personally against using a generic database as a storage and heres why:
1. The average user does not have an SQL database installed on their desktop systems, and so this is a completely rediculous dependency for them. If you think library dependencies are bad, just wait till you have to go installing, configuring, and maintaining a multi-user database running on your system. This may be fine for a company solution, but not the average end-user.
2. I'm not too familiar with MySQL or PostgreSQL, but I recall there being problems with mailers that use SQL database backends that tried to store the content of the messages as part of the table (due to them making the size of the table too small or whatever). If you can set the size to be "infinite", then I guess that's not a problem.
If your plan is just to have the database index the folder and actually store the contents as separate files, then you've instantly gained nothing over Maildir except that now you have a hefty database that you have to maintain and very little to no speed improvements (especially if you have a well designed/implemented summary index like Evolution does).
The only improvements you might gain here is body indexing? As I said earlier, MySQL supposedly has a REALLY good text indexer and so it might be a little faster than Evolution's. I'm really not sure on the comparison here.
> Should the security model allow users to directly access their
> files, grep them, copy them around?
Is there a reason NOT to? I don't see one. It's their mail.
> Shared folders, virtual domains?
This doesn't really have anything to do with folder formats and everything to do with features of the client itself.
(Evolution can do this).
> Unicode support in folder names? Imap message-IDs, flags, useragent
> specific state-information?
Unicode support in folder names I'd say is a pretty important feature. I'm not sure what you mean by "Imap message-IDs". Do you mean UIDs? Evolution, for example, has a UID assigned for each message whether it be in an mbox folder, Maildir folder, MH folder, or IMAP folder. So this isn't necessarily dependant on folder format (though it could be if you used a database backend for example, you might want a UID in the table).
I don't feel that UIDs are a must though, but I would suggest them. They are definetely useful especially for folders that can be accessed by multiple clients at once.
Flags are good. I'd go so far as to say a MUST have.
As far as user-agent specific state-information, it'd be nice to not need it. But if the client needs to keep it's own info, it'd be nice to be able to map the info to UIDs and keep it's own state file somewhere else (not necessarily alongside of the mail storage).
For example, IMAP doesn't have any means for the client to store state information on it, but that's perfectly fine. If a client chooses to
have it's own state, then it can save it locally.
It would be nice if the storage could handle user-defined flags/tags though. This would allow the client to extend the native features of the format (Flag-for-Followup, message colouring, etc).
> How would MTAs deliver mail? How would clients access? File-locking
> (NFS)?
This is one reason to just stick with what's available
File locking is a MUST have (or a scheme to make it not needed, such as Maildir).
--
You know, I have one simple request...and that is to have messages with freakin' laser beams attached to their headers. Now evidently my MIME specification informs me that that can't be done. Uh, can you remind me what I pay you people for? Honestly, throw me a bone here. What do we have?
I have in some times used a custom made MTA and indexed the incoming mails, headers and first message body part, into MySQL database. Attachments are compressed and stored within "regular filesystem". The whole kludge is then interfaced to IMAP. User authentication is also done via MySQL, thus making it unnecessary to create "real users". The solution has lasted without problems for years now already. MySQL in general, works like a dream, I have never had any problems with it.
It is good (and fast) for some purposes, which I am not going to discuss here, everyone is probably very well equipped to figure out the plusses and minuses of this way of doing it.
But unfortunately, the Notes client does not. We still need to dick around with wine to access the corprorate Notes server. If anybody from IBM (who likes to show their committment to Linux...) is listening: are there any plans for a native Linux Notes client? If so: when? If not: why not?
Say no to software patents.
Plain and simple. Switch from mail to Usenet. Maildir-like structure, but with a .overview (XOVER) file to help out with indexing.
Storage is another problem, though... but Usenet messages can be sidetracked a bit with the encoding.
--
# Canmephians for a better Linux Kernel
$Stalag99{"URL"}="http://stalag99.net";
Easy solution: Build a list of "VIP" users who will get a text-only version. Or who will get the text and the HTML version in 2 separate mails.
Say no to software patents.
Life With Qmail
Building a Linux Qmail Toaster
Same thing, but with FreeBSD (more scalable, in my experience)
have fun
Remember that what's inside of you doesn't matter because nobody can see it.
Alen, your experiences with MS-Exchanges are so many worlds of difference away from mine that I nearly suspect that you've written a troll. Rebooting a mission critical service like a mail server during working hours is unsatisfactory. If other mission critical services like file and print sharing are also disabled during that reboot, then it's time to look for a more robust product.
I have worked closely with three shops in the previous three years that used Microsoft Exchange. Each had at least 3 full time equivalents of MSCEs to babysit their Exchange servers, probably more if you count overtime. This is not counting the occasional high priced consultant. None of these shops could keep Exchange running for a full week. Nor could they keep it from losing mail (When I measured it was 10-15%, ). Nor could they get it to communicate well with other mail servers. Nor could they keep it from getting wiped out once every three months by MSTDs (especially worms and virii).
In contrast, Novell servers run years at a time unattended (nearly every consultant has at least two such anecdotes of their own) and many UNIX-based MTA's need only a few hours of non-hardware maintenance per year, when set up tight. I guess running MS-Exchange is a new status thing to flaunt resources, like having a tuburcular wife was during the Vicrotian era.
Needless to say the managment's support was/is a real PITA for anyone doing work via e-mail with people outside of the house's MS-Intranet. In one case it even delayed a publishing a book by several weeks. In house use of Exchange was fine -- when it was down for you, it was down for everyone else so it was a nice time out and a chance to go have coffee with the others. When put to the test, file sharing couldn't, wouldn't, didn't function often enough to be useful either. For file sharing, those without access to a Novell or Unix file server, used sneaker net or mailed attachments. Yes, Exchange does look good in the 4-color glossy marketing brochure, but that's were it ends and reality sets in.
Puh.
Back to mail databases. RFC 2822, Internet Message Format specifies the general structure of a message. This can be over simplified as a header with its standard and non-standard fields and one or more message bodies. RFC2049 specifies multipart bodies. These structures do seem very well suited to a relational database.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
Oh dear, another file format debate. I'm glad there was a library suggestion though ... that allows us to change our mind when we do it wrong the first time ;)
First, you need to consider the possibilitiy of moving the mailbox. To a different computer, or a different platform. This means it must be easy to access in any environment, and the tools must be portable.
This doesn't completely rule out a database solution (like mySQL), but it certainly makes it less-than-ideal.
Second, having used many mailers which separate out attachments ... Please Don't Do It! You can't easily move your mailbox, because there are a host of associated attachment files. There is ALWAYS a synchronisation issue between attachments and messages, so you end up scanning and cleaning out the attachment folder every so often to prevent dead files from accumulating.
Compression is nifty, but isn't really important. Disk space is seldom a concern these days, and the really big stuff (binaries) is often already compressed or don't compress well.
The real issue with most mailbox formats is how do you deal with the problem of removing dead space from the mailbox? Some program just leave it there until you hit "compact", which is wasteful and confuses users. Others rewrite the entire mailbox every time, which causes the software to "hang" for a while on shutdown.
The best suggestion I can come up with off the top of my head is this: One file per mailbox folder, and that file is its own filesystem. The "root node" contains a group of summaries (from, to, subject, date, etc) and node links. Other nodes are chained to contain the message and attachments.
Handling attachments: attachments are separated out and stored as binary in the mailbox. This conserves space but keeps the attachment with the message.
Compacting: is avoided. When a mail is deleted, it is merely flagged in the root node (index). So each mailbox has its own deleted items folder, so to speak. When the deleted items folder is empties, the index is rewritten and nodes freed - every node not at the end of the file is overwritten with a node from the end of the file (and appropriate reindexing done), so the file is automatically compacted.
Ideally the file needs some sort of transation logging area to ensure its integrity at all times.
Shared access to files is best handled through a library or a service. File locking is notoriously prone to bugs and security issues, and avoiding multiple implementations in different mail clients would be beneficial.
i-name =twylite [http://public.xdi.org/=twylite], see idcommons.net
For Pete's sake, leave mail alone. If I can't fix it in less than 20 minutes with grep and perl, I don't want to know about it.
Divide mail into 20-30 logical "folders" (files), use procmail to help sort/scan/unspam, do IMAP to get to it from Win machines, archive mail out of your working files once it gets a year old, and you're all set. Strive to keep your inbox empty (you need a proper "action" orientation with your mail folders to accommodate this). No big deal.
Son, I'm posting at 2.
I have been pwned because my
There is no need to toy around with the mail as-is. It's a little like IP packet, doesnt matter what's in it, but the essential thing is that it has a destination and source addresses and it travels in the net. No techinal solution will _ever_ overcome the fallacies with current emails, becuase the current email is as unrestrictive as IP packets.
./ said it really well once, he said that the best solution would be "Cheap plasma handguns and justice for us all"
Take spam for example. The problem will always be present theoretically when you want to receive mail also from people you've never received before and/or havent given your public crypto key for example. Another side of the aspect is when _we_ allow people to send source address spoofed spam.
The problems with email are people, as with almost every damn problem in the IT sector, it's always us. People bend towards stricker rulesets, to avoid abuse, which in many cases is not the way to go, let alone the solution to the root of the problem. Somebody here in
1 Earth is warming, 2 It's us, 3 it's royally bad, 4 we need to take action NOW
I lost the plot half way through this, but here's some food for thought anyway. Now I should get back to work ...
;), but it works reasonably well, and we've had a chance to try and deal with users with lots of email.
...
...
...
... and you dont want 1-tier applications), so it doesn't matter what format you use under the belt - you can choose the format which best suits what you're trying to do.
Z
I think that this is looking for the solution to a problem that doesn't really exist in the first place. Although I guess it depends somewhat on what you define as 'Unix mail'.
I'm a developer on Evolution, and primarily on Camel, evolution's email library. I'm not sure i'd rave about it (although I think Camel is a mostly beautiful piece of code
What IS 'Unix mail'?
I would define Unix mail as mail (rfc822 format) downloaded and stored locally on a per-user basis. IMAP, Exchange, and other remote protocols are very different beasts.
Why are DBMS's not suitable for 'Unix mail'?
Once you have a remote server you have to do things differently than if you have local access. Using a DBMS, and having a trained administrator to manage it are practical considerations, as are the benefits you might get from this configuration. These solutions dont really make sense for standalone users. They shouldn't need to install and manage databases, complex backup prodedures, and so forth, just to read their email.
i.e. rdbms's are:
hard to setup
hard to maintain
another major point of failure
If however, I was to design a multi-user groupware server, then a DBMS would come into serious consideration - at the backend at least. It allows you do to things like easily consolidate authentication outside of the operating system (the idea of having a 'shell account' to access mail is somewhat outdated), it allows you to save space by storing common data, like attachments and email content in a single place, and redirecting it to multiple recipients (which is a common practice within organisations). It may be practical to use a mixture, a RDBMS to store textual parts or indices to data stored in a more conventional filesystem.
But even with a RDBMS backend, I would personally probably still stick to IMAP to serve it to actual clients. The IMAP protocol is a bit heavy, but not really that bad, and it serves email, I dont think there's really any need to reinvent the wheel here.
So
If you define unix mail as I have, and separate it from a *mail server*, then you rule out full blown RDBMS's, and are left with:
single file database
multiple file database
I'm not even going to mention XML because I think it is the single most stupid idea anyone's come up with. It is completely unsuitable for this purpose.
And well, there's really no reason not to use MIME to store the messages. MIME already does everything you can possibly do with email (since, uh, it is how the email *will* be sent), any client will already have to deal with it, and mime decoding is for the most part really quite simple and fast anyway. Translating the mime format into some other storage format really doesn't make sense.
single file databases
mbox
Mbox is a single file database. Its just that everyone that uses it generally writes their own access code. This is where problems with 'locking' come about, either because the underlying filesystem doesn't support it properly (e.g. some nfs implementations), or everyones clients don't use the same locking mechanism. This really just an implementation issue anyway. There would be nothing to stop someone writing a common 'mbox.db' library that stored everything in completely compatible mbox files, which took all the work out of it, and then you'd have an mbox DBMS
mbox scales ok, without any caching of header information it handles in the order of 2K messages in an interactive timescale, and quite a lot more if you dont mind some short delays (i.e. in the order of the time it takes mozilla to start up).
Appending and reading is quick, and reliable - assuming the filesystem works, which is a pretty safe assumption to make. This is assuming the mailbox is first summarised at first opening, otherwise looking up messages can be slow, because you have to scan the whole file first.
The only operation that is slow is expunging messages, and at worst case isn't really any slower than copying a whole file across to another file.
The only other issue is agreement on the 'standard' for what constitutes an mbox file. For example. Solaris uses and honours the 'Content-Length' header, and thus it does not translate any lines beggining with "From " into the conventional ">From ". Some mail clients translate "(>*)From " into ">\1From " (using sed syntax) and visa versa, others do not. There is no standard, just some conventions, some of which aren't easy to determine either.
Because you need to keep the whole index in memory at once, this can become expensive, but you could use a secondary database as an index into the real file. But eventually you hit a point where the cost of expunging does get too expensive. You could just archive the mail regularly, or use a format like maildir instead.
gdbm/db/etc
db files wrap the single file in a common api that handles all of the locking issues and access issues for you. Some have different features, e.g. querying capability, logging and transactions, etc.
We've never tried to use db for this purpose, more just because we didn't think it was worth it. All you really get with a minimal implementation is the ability to store and retrieve a blob of data using a single key. Writing is fairly slow because the database has to manage more details for you (locking, allocating blocks, unlocking, etc). You could use multiple db files as indices to perform multiple-key searches, but they are quite slow at creating them (we tried using db for the content indices and it was way too slow).
i.e. even if you store the data in a db file, which gives you a slight benefit of inbuilt referential integrity, you still need to provide additional indices to actually be able to use it in any useful way. Evolution suffers this problem with the addressbook which stores vCards in db records.
Most db libraries (all?) also dont provide any mechanism to stream data. You either get the whole lot into memory, or you get none of it. So for large messages you're limited by memory (well, evolution is anyway, but it doesn't have to be). Yes, memory is cheap, but it is still a consideration, and it would certainly rule out a simple database in a multi-user environment.
db files are also slower than native files, especially for large objects. You're mapping an arbitrarily sized chunk of data to some 'database blocks', which are then stored in an arbitrarily sized 'database file' which the operating system is then mapping to its 'filesystem blocks'.
multifile solutions
Well I guess this comes down to mh and maildir. mh isn't really suitable for anything, because of its just plain bad design and lack of defined semantics. There's no way to guarantee anything about its operation.
maildir - i like. It moves the scourge of trying to implement a reliably, scalable, multiple access database almost entirely into the operating system layer. Operating systems already do this very well - they manage hundreds of thousands of files randomly written across your disks, without skipping a beat.
No operation requires more than a single message size of data, and the operating system already indexes the message, via its filename. Sure, ext2 doesn't do such a swell job with long directories, but that can be addressed (and the same problem can be addressed on just about any platform). For 'free' you get concurrent multiple-reader, multiple-writer database access, without any of the considerable problems you have to solve to implement it otherwise.
The maildir 'protocol' is simple, reliable, and it works.
Again, it can easily be augmented by a client with additional indices, but for things like delivery agents who dont care about existing email, they dont need to suffer that overhead at all.
Some other comments specific to the question:
Compression. Personally I dont see the point. But a maildir-like structure would fit well with compression. Flat files would be the worst (e.g. mbox), and block-file formats (like db files) would also work well with compression. The good thing about email is it is 'write once', you don't edit or change the messages in the mailbox.
External attachments. I guess its possible, but again, it isn't really worth it in most cases. Parsing MIME is *fast*. It is much faster than parsing xml, and besides, people rarely look at an email more than once or twice. There isn't much use going off and storing the attachment in a high-performance reading format if it isn't going to be accessed often, and it just places a greater burden on your server.
base64, etc. Well, its entirely possible simply to store the messages as 'binary' format. Assuming the boundary markers are checked properly, Camel can work with binary encoded mail messages, and probably at least some other mail clients can too. There are some problems with some of the extremely broken openpgp/pgp/mime specs which suddenly say that mail transports aren't allowed to alter the *transport* encodings of some parts, but well, these specs are just braindead, and can be worked around.
Security model. Well, talking about Unix mail, not server mail, the filesystem is adequate.
Shared folders - is not an issue for unix mail.
Unicode. Well you can write unicode filenames to most unix filesystems, evne if 'ls' doesn't show it right.
MTA. Nothing could be simpler or safer than maildir as a delivery format. The mta doesn't have to care about any client-side indices, the mua will simply update them when it incorporates the new messages, etc.
Writing libmailstore? Mate, its called Camel, and its already written. Camel already does mbox, maildir, mh, it can read spool files directly (it doesn't create a summary file or build any indexes), it can talk imap, pop, and partial support for nntp. If someone gave me a decent RDBMS table schema and a carton of pale, I could probably write a MySQL backend in a couple of days, well, assuming the MySQL api is mt-safe.
Finally, some comments on evolution.
Evolution isn't reinventing any wheel. We use standard mbox format (if such a thing really exists anyway). We use standard maildir format, etc. Yes we may optionally create body indices, and we do usually create on-disk binary/compressed 'summaries' of the data, but these are really just on-disk caches of in-memory data structures, rather than anything to do with the mail storage format.
We put mail in another location, but everyone else has done that too, elm:Mail, pine:mail (or is it the other way around?), netscape:ns_mail, etc. At least we now offer the option to read most of this 'in place'.
The main problems evolution has with scalability is:
indexing.
Indexing is quite costly. The original index code was written somewhat like a database, it handled all internal data structures, used blocks of data, etc. It was slow, it scaled poorly. Definetly some of the algorithm choices and the implementation wasn't that hot, but it shows that such a solution isn't as simple as at first thought. Using libdb was impossibly slow (like several orders of magnitude slower).
The new stuff is a lot better, but can still use a lot of resources while indexing, and copies the whole file (well 2 files) across when performing expunges, but they are only performed occasionally, and the indices are smaller than the original indices, so in practice it scales much much better.
the summaries
The summaries are indices of a sort anyway. They are an in-memory tree of a subset of the information on each message. Enough information to display a list of messages, and perform vfoldering operations. Even though we do some tricks, like sharing common strings, the summary can get very large.
But, its a tradeoff I thought was worth it, rather than using on-disk summaries. The api's are much easier to use, and the problem gets pushed to the user - if they want to have folders with 100K messages, they should expect it to use a bit of memory. The on-disk size of the summaries is very small too, although I guess it could be made even smaller if we consolidated common strings.
per-message memory use
Currently, a lot of data gets copied around in memory. Every time you read a message, at least 1 whole copy of the (decoded) message is in memory at a given time (yes, including attachments). For IMAP this can get even worse (2-3 copies of a given attachment at a given time), because it doesn't stream enough. Most of this could use a disk-backing without changing any api's though, and well, i'm rewriting IMAP.
Wrapping up
And yeah, we're talking 100K messages here, not 1400. My 500Mhz celeron laptop has about 35K messages stored over about 10 mbox files, and it starts up in under 10 seconds, and that includes all of the bonobo/activation overhead (which is very significant). Yeah it uses a bit of memory, but memory is cheap on a personal workstation.
In short. The current mailbox formats we have suffice for "Unix mail". Add some archiving abilities to your mail client (even RDBMS backed mail clients need archiving), and you'll never have to delete a message again, and still get work done and still use mbox.
If you want to talk about writing a server - well who cares, you can do whatever you want, because everyone has to go through your interface anyway (you DO NOT want clients accessing data under you, thats what DBMS's are all about in the first place
It seems some people think using 1-tier applications (client code talking directly to a database) are the way to go for multi-user environments. They're not, they dont scale and are impossible to maintain. Nobody writes any real software like that anymore, unless you're writing dodgey vb toy apps.
_
\\/ are accustomed' - First Lensman
There is absolutely no reason to abandon the standard e-mail file format, including uuencode for file formats. Doing that, you would end up with a file format that depends on certain versions of the e-mail file format to work optimally. If you want to reduce harddisk space, zip it like OpenOffice.org does.
E-mails are documents. Documents belong into the home directory, and so do e-mails. If you want to do something new, you should use the harddisk folders as e-mail storage, so that e-mails, spreadsheets and documents mix. This probably requires inventing a new ".e-mail" file format so that e-mails can be properly recognized and indexed.
Storing one e-mail in one file is not a problem as long as you index the filenames properly, for which you can use gdbm.
Dybdahl.
I don't think things are that bad - for example, Cyrus with its indexes works pretty well and large (20,000+) folders. And things like searches are pretty fast with a client like evolution that does a lot of cacheing.
I would take the simple structure of Cyrus over the easy to break "database" files of Exchange server any day.
This is all very interesting because I'm slowly writing an IMAP server at the moment..
But here's the setup I'm currently using:
Inbox:
/var/mail/$USER
/var/mail/$USER-folders/$FOLDER/.messages
Subfolders
Eg:
|-- root
|-- fred
`-- fred-folders
|-- 1ZB
| `--
|-- Friends
| `--
|-- Games
| |--
| |-- Rune-Beta
| | `--
| `-- Tribes
| `--
`-- Mailing Lists
|--
|-- EFNZ chat
| `--
`-- Hard News
`--
I started with uw-imap but I want to store messages and subfolders together. Plain uw-imap doesn't do this and last time I checked, neither does Maildir. So I did a [kludgy, incomplete] mod and produced the above. Works for me :)
Get the patch: http://home.y3m.net/uw-imap-2001a-nested-folders.p atch
(diff against imap-2001a)
In the server I'm working on you will be able to implement a relatively simple C++ API to do your own storage. So you can use Maildir, mbox, PostgreSQL, whatever. We'll see.
flame away :P
No, I did not read the f***ing article!
There seem to be two discussions going on in the comments today, one about mail storage for an MUA and one for storing mail on servers. ;)
I've never used OpenMail/Domino/Notes/whatever, but I guess they do roughly the same thing, which is a pretty good idea. However, these things all have the distinct disadvantage that they use propritary protocols and aren't particularly cheap. There's always IMAP, which many people really like, but I feel is too complex a protocol (compare with the infant levels of complexity in POP3).
As far as the client end is concerned, from the point of view of writing an MUA, having an SQL backend is a complete godsend because you have to write virtually no IO code, you can put all the logic in the queries. However, there are some tricks you need to use to keep up the speed, most importantly to use two tables, one for metadata and one for the mails themselves. This keeps the speed up by keeping the metadata table small (maybe on a better RDBMS than MySQL this wouldn't make a difference, but I found that >10,000 mails all in a single table in MySQL got quite slow until I moved the metadata into a seperate table).
The obvious downside of using a DB for client end storage is that you have to have a centreal DB server, or one on each client and you need to admin one more set of authentication/permission details, plus you can't move the mail very easily to other MUAs. IMO a much better solution would be to keep the use of SQL/RDBMS, but move the DB into the filesystem so you can just have a bunch of files with metadata stored in the fs. Need to make an mbox? "cat ~/mail/* >>/tmp/my_new_mbox".
From the server point of view, many people have been mentioning Exchange/Domino etc. Personally I can't stand Exchange, I've had to admin it on several occasions and it's generally done everything it can to stop me from having an easy life (just thought I'd air my predjudice against Exchange in the spirit of fairness and honesty
With a colleague of mine, I'm working on a set of POP3 extensions that give some IMAP like features, but is really designed to keep multiple mail clients in sync with each other by way of a transaction log. There are still some limitations, but I think I know what they are and how to fix them (e.g. not enough metadata can be associated with each mail yet). It adds about 6 or 7 commands to POP3 and currently lacks any decent client support, but I have written a fairly usable library and patch to gnu-pop3d for it. I've just submitted it as my University final year project, so I'll try and get the protocol description documentation online soon. In the mean time, if you're interested, it's on SourceForge
Chris "Ng" Jones
cmsj@tenshu.net
www.tenshu.net
So NNTP solved this IMHO a rather elegant way...
.overview, which is just the summary information for all the files.
.overview file. Or grep through it, if you like.
You have directories corresponding to newsgroups or mail folders or whatnot. i.e. alt.swedish.chef.bork.bork.bork is really alt/swedish/chef/bork/bork/bork
Articles are numeric, i.e. \d+ for Perl types. The raw message is stored in each file.
In each directory, there's a file called
Thus, you can have zillions of small files, and happily grep and copy them to your heart's content. But you never do a 'ls' on a huge directory, you always just look through the
So, in that sense, it's very much the best of both worlds. And, on the same box, you can specify rules on who can access the folders, so one file can be read by multiple people. Ooh.
GNUS, an Emacs based mail/news reader, uses a variant of this called nnml, which rocks.
Of course, when you get down to it, JWZ arguments aside, databases start to really look like what you want, especially on a corporate level when you're tossing the same piece of mail around to tons of different folks.
-e
better yet, make the files downloadable via a secure web server.
Are you trying to tell me that a 5MB empty mailbox is asking too much? A text message that says "Hi!" costing 1.2MB is somehow wasteful?
Lotus Notes Uber Alles!
This
What bugs me the most with current mail technology is the problems with distributed mail handling.
:(
I access my mail on all kinds of devices, sometimes online sometimes not.
My main problem is not so much witch mail-server / retrieval / presentation to use, since they all have the same inability to give me a working distributed solution.
For online usage imap is sufficient, but if I go ofline with my laptop or ipaq, Im lost.
POP isnt very efficient either, since only one of my clients can be the deleter, I must make sure that I synced all my other devices before the deleter removes the message.
Since I use tons of folders for my mail, some of my stored mails data back to the late '80s, it basically forces me to use imap so my folders are insync on all the devices, but again that only works online
Further it only works if my imap server is online. That can be a trouble if Im in some far of part of the world and for some reason or not I have no contact with my mailserver.
What I would like is a concept I call SyncMail
A distributed db-system. First I set up some 3-4 primaries, spread out on the net with completly different access routes. Each of them gets a MX record.
The sending mta is happy to deliver to a secondary mailserver if the primary is ofline.
But here comes the magic!
The system regarded as a secondary MX by the rest of the world is in fact a primary!
It sucks the message instead of queing it into its db, tags it with it's own internal server id, and tries to sync it to all other SyncMail primaries.
Sooner or later the new mail is propagated to all the primaries.
On the client side, the SyncMail app, contacts all the primaries, and cheks against a private index, and syncs all new mails, first trying with the closest server.
Since all mails are tagged with what primaries it's been delivered to, no mail is retrieved to the client more than one time.
Now I have a complete local mail-tree in my client, regardles of which primary I was able to contact, sure if a mail was delivered to a primary that goes ofline before the client syncs, and it hasnt been able to sync it to the other primaries, I wont get it until that primary comes online, but - what the heck, in pop/imap is my mailserver ofline im completly out of buisness, so the loss is defenetly smaller in this case.
And for my ipaq i just configure the client to work with a few important folders, and to skip attachments, to save storage
And for sending, all clients stores it in a outbox, wich is then synced to the primaries, once it gets to a primary it is sent in normal SMTP
this way I solve the problem of being able to send mail with propper originating SMTP headers. Of course the outbox is synced as well, so I get a ref copy of my mail on all systems.
I have started on a SyncMail application and someday I might be able to complete it, but there is so much work all the time
Would anybody else be interested in this concept, maybe we could complete it together.
Or if this is a realy stupid Idea, I'd be glad if someone would point it out, so that I can focus on finding a better solution.
I once tried benchmarking Maildir vs mbox for my mail archives (mailboxes with ~3000 messages). On ext2 Maildir was a loss:
- Mutt took twice as long to open a Maildir than mbox from cold cache.
- Mutt still took a bit longer to open Maildir than mbox from hot cache.
- On ext2 with 4K blocks mbox ate 13 MB of space, Maildir ate 21 MB.
- Small UI degradation: Mutt wouldn't show the number of lines in a message from a Maildir, and it wouldn't show percent progress indicator while reading the Maildir.
Basically for my situation (read-only mail archives with large numbers of messages, which are rarely in filesystem cache, ext2 and constant disk space shortage) mbox was better. But my situation (personal mosty static mail archives) is remarkably different from running IMAP server.I did this test in 2000. I should probably try again some day with Reiserfs, but I heard various people telling me it doesn't improve Maildir performance. Can't say anything until I try myself.
I therefore recommend you to try it yourself and see if Maildirs really help in your situation.
here's a little transcript:
% cd /mail/fs/mbox
/mail/fs/mbox/318/2/body is a jpeg file, viewable directly by any usual jpeg viewer).
% lc
Directories:
1 113 128 142 157 171 186 20 214 229 243 258 272 287 300 315 33 344 359 373 388 401 416 430 445 46 474 56 70 85
[...]
% cd 318
% lc
Files:
bcc date filename info messageid rawbody sender type body digest from inreplyto mimeheader rawheader subject unixheader cc disposition header lines raw replyto to
Directories:
1 2 3
% head raw
Return-Path:
Received: from punt-1.mail.demon.net by mailstore for rog@vitanuova.com
id 1021665470:10:17045:138; Fri, 17 May 2002 19:57:50 GMT
Received: from psuvax1.cse.psu.edu ([130.203.4.6]) by punt-1.mail.demon.net
id aa1016828; 17 May 2002 19:57 GMT
Received: from psuvax1.cse.psu.edu (psuvax1.cse.psu.edu [130.203.6.6])
by mail.cse.psu.edu (CSE Mail Server) with ESMTP
id 27DA4199BE; Fri, 17 May 2002 15:57:13 -0400 (EDT)
Delivered-To: 9fans@cse.psu.edu
Received: from acl.lanl.gov (plan9.acl.lanl.gov [128.165.147.177])
% head body
This is a multi-part message in MIME format.
--upas-mbyuptynpdsmbjuyeermihdgur
Content-Disposition: inline
Content-Type: text/plain; charset="US-ASCII"
Content-Transfer-Encoding: 7bit
Hi,
If you seek excitement and thrills you need to look no further than
Plan9 -- it gives you everything and then some, but in a good way (or
% cd 2
% lc
Files:
bcc date filename info messageid rawbody sender type
body digest from inreplyto mimeheader rawheader subject unixheader
cc disposition header lines raw replyto to
% cat mimeheader
Content-Type: image/jpeg
Content-Disposition: attachment; filename=iostats.jpg
Content-Transfer-Encoding: base64
% page body
reading through graphics...
%
"raw" contains the raw data that makes up the message. "body" contains the data after the encoding formats have been applied (hence in that case
the beauty of this scheme is that it hides the underlying storage scheme from the mail clients. if i wish to change things so that the underlying storage format is many files [currently it uses a traditional mbox format], none of the mail client programs have to change.
plus i can use grep, diff, shell scripts, etc directly on the messages in my mailbox. procmail eat your heart out.
... in the form of 8 different machines, all of which were running reiserfs on various GNU/Linux distros ranging from Suse to Mandrake to Debian, all of which suffered data corruption, data loss, and even the mysterious vanishing of entire directory trees (while disk usage exploded). In short, all had unrecoverably corrupted filesystems, not as a result of unscheduled shutdowns (which journalling is supposed to help protect against anyway), but on machines that were operating normally, without interruption. None of these filesystems survived more than 9 months of normal, everyday activity (without improper shutdowns, I will stress once again).
... I have hard data to back up my claims, and, quite frankly, a filesystem is sufficiently important that "your milage may vary" should be an unacceptable answer. By all accounts, if those who haven't (yet) suffered data loss with ReiserFS are to be believed, with ReiserFS YYM indeed V.
These machines were located at three disparate sites, had different base configs, and in two cases were installed and maintained by different people.
The only things they had in common were that they used Reiser, they lost data (severely), and had to be reconstructed from backups (this time without using Reiserfs).
You may believe that you can trust ReiserFS, but I know for an absolute fact that I cannot, and I think it is very possible you will discover that at some point as well. Of course, having relegated everyone else's experience to mere anecdote, it is clear you won't learn this until it hits you in the face, personally. That's OK, not everyone is willing to learn from the experience of others.
However, to those who are interested in learning from the experience of others I will say this: tread very, very carefully with ReiserFS. It is not ready for prime time, and should not be used in any production system. If you really need journalling, use XFS. It is very stable and quite difficult to damage (so far it has survived every stress test I've been able to throw at it).
Now, go ahead and relegate this to anecdote if it makes you feel better
The Future of Human Evolution: Autonomy
I think it's the best of both worlds. Your 'INBOX' is like MailDir, where each 'new' message is a seperate text file. Once you've 'Filed' that message, however, it's compressed into a single file along with the rest of the emails for that folder.
Personally, I think you're looking at the WRONG aspects of mail servers. You're getting way too technical. Nobody gives a shit about wasted inodes. When's the last time you defragmented ANY disk?
The reason I use Mercury, is because of it's exceptional Netware NDS integration. Combine that with Pegasus Mail's NDS integration, and you have 'Roaming' users without all the profile garbage (Pegasus will use NDS calls to see 'who' you are, and read your email from your home directory). Oh, and it's free.
To bad it hasn't been ported to Linux.. along with the PAM stuff needed to keep up the kick-ass user integration :)
"I can't give you a brain, so I'll give you a diploma" - The Great Oz (blatently stolen sig)
I came up with mboxdir. It was actually a preliminary specification for a Win32 client.
VMSmail's storage format is instructive. Each message is represented by a single record in an indexed file. A short message body is simply tucked into the record along with the headers and other metadata. Long bodies (more than around 2kb IIRC) are stored as individual files and their header records point to the files by name.
Of course you all realized at once that the main file can get out of sync. with the directory which holds the external bodies. It does, sometimes, and fixing it up can be a pain. Any storage method which partitions a single message among multiple files is going to have similar problems. But it works pretty well, and it shouldn't be too hard to write a tool to groom the message store in case of inconsistency. It's worth study.
It was a natural choice on VMS, which has really good multi-indexed file support in the base package. It works well with text messages, which often do fall within the size limit for avoiding external storage of the message body. Today it suffers the same problem that mbox does -- people use email differently now.
I have never used Exchange, but a friend of mine admins a large (50,000+ users) Exchange system. Even a few years ago, running on NT4, their servers did NOT go down, ever. They scheduled a reboot for patches etc every 6 months, that's it. I have had lots of Netware boxes up for over a year, but not Netware 5 running mail. I inherited such a box & it needed to be rebooted every month or two. Now I've replaced it with a Linux based mail server & I'm much happier. Still have a 4.11 box cranking along happily, even happier since the 5 box is no longer giving annoying messages about it's licences. And my 2000 Server has been up for coming up on a year with no problems.
I have in excess of 46K email messages in my account alone, not to mention everyone elses accounts on my company's mail server. We use cyrus IMAP and qmail, both of which use the Maildir format mailboxes ... every client I've used (Mozilla, Communicator, Outlook/OL Express, Mail.app on OS X, Eudora, and Papi-Mail on PalmOS) seem to have absolutely no problem with this setup. Most MUAs are intelligent enough not to download all your headers every time you connect, so unless you're getting 1000+ new emails everytime you open a particular folder, you're generally not going to need to read all those headers every time.
... long live Rackspace).
... optimize the file subsystem beneath it, maybe allow for compression/encryption or that sort of thing, but otherwise, the folks that put together Maildir have certainly done a decent job!
The server that runs this is a measly 600MHz PIII w/ 128MB RAM running RedHat 6.2 w/ a 20GB hard drive. I haven't gotten even close to running out of inodes, to my knowledge, and my server never goes down (really, the only times its gone down is when power has been cut to it and this has only happened twice in the past 1.8 yrs
Maildir is specifically designed to handle mailboxes with large numbers of emails in them, contrary to other formats such as mbox. The problem with any sort of DB approach is the waste of space, even if you compress. A basic course in file structures will teach you a wealth of knowledge in this regard.
Imagine this: you have a table that stores everything you need to know about an email. You have a few distinct fields for commonly accessed headers (subject, from, to, cc, etc.) each of which would need to be 'text' blobs, since you cannot limit their size (you've seen the emails that have to/cc fields that are miles long, right?) - well, 'text' fields are notoriously poorly optimized in database engines and quite difficult to search (you can create an index on a part of a text field, but that might not be enough, right?). Next you have the message body which would also need to be a text field since you don't limit it's length, either.
Now, since the space for these fields (which don't *ever* change) is not optimized in the slightest, you might think that compressing them is a good idea, right? Well, what if an email is deleted - then you start looking at fragmented space in your database table which would need to be compacted periodically (much as mbox/.mbx files do today, if I recall).
All in all, storing each message to its own file is not really *that* bad
You are going to run out of inodes at exactly the same time you run out of disk space, because they are one and the same thing.
No.
Running out of inodes is not the same thing as running out of space. Some of the symptoms of the two are the same ("can't create new files"), but they are completely different failure modes.
Consult your local man pages for further details.
News for Nerds. Stuff that Matters? Like hell.
when i read your post, i immediately thought of a Jamie Zawinski article that i read a few weeks ago:
http://www.jwz.org/doc/mailsum.html
he talks about this very thing. quite interesting if you ask me.
There are two excellent reasons that so many people use Exchange.
1) In general, it works out of the box. A company with someone with meager knowledge can set up a fairly complex mail handling system without much help.
And that same person with meager knowledge is going to get hacked six ways from Sunday when the next Exchange exploit comes around, because what's not included in that meager knowledge is that you have to keep up on security patches if you want your easy-to-install mail server to not be an easy-to-hack mail server.
2) It does A LOT. In it's most basic configuration it does what you need 10 or more programs in Linux to do, not to mention that most of those 10 don't exist.
And God help you if one (or many) of those pieces of Exchange are broken or don't do what you want to do. Can't change it, it's part of Exchange! At least if one of those 10 linux programs are broken or doesn't work right, you can replace it with something better without affecting all the other parts.
These are simple philosophic differences between Unix and Borg. Borg stuff usually has a shallow learning curve at the beginning, but then it ramps up as you discover things that are difficult or impossible to do. Whereas, the initial Unix learning curve may be steep, but it flattens out further in.
At least mafia-owned pizzarias make excellent pizza. Compare to Bill Gates.
Well, how many MCSE's on paper equal one person who has actually done the stuff?
We have an exchange server - one person managers it along with tons of other stuff. It pretty much runs itself. We just moved to Ex2k, but were on Ex5.5 for quite a while - I can think of only one time it crashed and that most likely had to do with a 3rd party virus scanner intergrated onto the server. Removed that and no more problems.
I don't think it makes sense to store email in dbm files. It's too sketchy - what happens when the dbm file gets corrupted? The nice thing about flat files is that if something goes wrong, you can fix it with vi.
I think the right solution to the problem is to key off the message ID, which is supposed to be unique. Then define a mail folder as simply a list of message IDs. Messages can appear in more than one folder, but hopefully not in no folders.
To make this efficient, I'd hash the message ID, and use a hierarchy of directories, because Unix doesn't do well with large flat directories. The hierarchy could auto-extend, so that as one subdirectory fills up, you do a sub-hash and split it into more directories.
The problem of tiny files is a real one. The solution is probably to make the bottom of a hash a file rather than a directory, and store more than one message in each such file. You don't have to store a lot of messages in these files to win - even ten messages would produce a big win, and would be pretty efficient.
The format of the individual files should probably be indexed sequential access - that is, a TOC at the front, and then the contents as plain text, nothing fancy. The TOC should be in ASCII, not binary, and you should be able to rebuild the TOC by looking at the file.
Babyl used to use a control character as a delimiter, which worked pretty nicely - much better than using "^From ". Ever seen >From in an email message? That's because Unix mail uses "^From " as an inter-message delimiter, so it has to quote it, and it does so stupidly. So use ^_ as a delimiter, and if ^_ appears in the email message, just double it. Take a doubled ^_ out when reading a message.
As for compression, I don't think it's worth doing at first. Disk space is cheap. Yes, my email folder is pretty huge, but it's really not a major problem. Making the storage system extra-complicated by uncompressing MIME is something to add on after you've got something more basic that works - you don't have to solve every problem all at once.
As for folder scan performance, you can make a cache, and have the mail program scan the cache from time to time when it's idle to clean up errors. This is much better than trying to come up with a format that's optimized toward folders - if you try to optimize toward folders, you wind up creating all kinds of problems, IMHO.
Until about 3 days ago, I had 1700+ messages in my Maildir, and pine (patched to support Maildir) opened my inbox in about two seconds. Compare this with my sent-mail folder, which had about the same number of messages in it. This folder is stored in mbox format and it took 5+ seconds to open AND CLOSE this folder. I believe that Maildir is the fastest option, short of keeping a seperate database.
Searches of google and google groups turns up no one else that shares your experences of "unrecoverably corrupted filesystems" with reiserfs.
ahem. You really didn't look very hard, did you?
filesystem corruption (2.4.18, reiserfs)
Bug#122230: reiserfsprogs: filesystem corruption with reiserfs
Re: ReiserFS / 2.4.6 / Data Corruption
ReiserFS desaster - advice please !
and about 829 other matches. Need I go on?
Oh, BTW, as I noted, two of those systems didn't belong to me, they belonged to people I know who experienced similar difficulties (and documented them as well).
Enough people, of enough diverse walks of life, are having issues like this with Reiserfs that it is clearly not something that is safe to be deploying in a production environment. Even if only 1% of the people using it are being so bitten, that number is way too high (and based on my own experiences and those of several people I know, I suspect that number is a lot higher than 1 per cent).
The Future of Human Evolution: Autonomy
I don't think "unix mail" is all that useful a handle, but if I was going to use it I'd be referring to mail that stayed on unix hosts - usually in mbox format - as opposed to mail downloaded to user PCs with unknown operating systems.
Corporations and other profit-making legal entities can't dedicate specific PCs to single users cost-effectively in most situations, and they certainly can't effectively manage storage and back-up email stores if the Email messages are scattered over many failure-prone end-user hard drives. IMAPv4 and whatever the proprietary boyz are shopping this week purposely keep the email on the server, so that evidence can be extracted (or destroyed, if you work for Enron) from server backups, and so that filtering and surveying of mail data is easily possible.
For example, some corporations sweep their drives for return & delivery receipts over a month old and delete them.
Another example, corporations doing highly sensitive government contracts will sweep their email stores for classified information leaks.
Another example, I need to get my Email regardless of whether I'm on my laptop at a remote site, at my desk in town, or at home tunneled through SSH. Downloading it to one of these boxes makes it inaccessable to the others.
The list goes on, but basically downloading email to a local drive is primarily for AOL users and basement hackers. That being the case, your points about maildir are excellent - let the filesystem handle most of the details. I'd add that if you must run a db for speed reasons (such as a subject line db used by an IMAP server) do it so that it can be deleted and/or recreated on the fly from the contents of the maildir. No need to create additional dependencies.
No they are not. The parent post is correct.
In fact, I believe all the inodes are created when you create your filesystem, all space is mapped to an inode (though of course one file can use multiple inodes).
What you believe has nothing to do with reality. I suggest you take an OS course. Or read up on how Unix filesystems work.
It's usually said that if you have 4k inodes, you'll lost 2k (on average) per file.
There is no such thing as a "4k inode". You got your terminology wrong. You are thinking about blocks. On average, you waste 1/2 the block size for each file on your filesystem, since the last block is, on average, half-full. An inode is not the same as a block! They are two completely different things, which is why your entire post makes no sense. Think of an inode as a "file header". I don't have time or energy to post the full description but I already mentioned where you can get relevant information.
___
If you think big enough, you'll never have to do it.
The poster of the article just assumes that filesystem must be slow when working with 1000+ files per directory and we need a database to save us. That's nonsense, from my experience.
Apart from that, there are some very important reasons why maildir is much better than a DB. With maildir you can use standard Unix tools to manipulate your email. With a DB you can't do that. Mailbox corruption is not a problem with maildir -- even if corruption were to happen it would be limited to one message, or a small number of messages (not even a mailbox). With a corrupted DB storage, you lose everything -- all the mail of all the users in all the mailboxes. Ask an Exchange admin about it some time.
___
If you think big enough, you'll never have to do it.
If you are looking at a file system as a heirarchial structure, why can't you have more than one such table.
The idea being that some mail clients would be only in the "person" tree, and that others would only be in a "function" tree. One could then be given access to both the person and function trees, and shunt mail between them for others to see.
The other thing that we should do is do things that encourage the use of these things. Make the tools for doing this easier to use and understand, and make the concepts easier to grasp.
OS/2 - because choice is a terrible thing to waste.
Eh? Your 'counter' to the factual claim that it doesn't is... an unsupported claim that it does?
It is not Maildir format. Maildir specifies the delivery method as well as the file format; by your logic, Maildir is nothing but mh. But it's not just "single file per message," and neither is Cyrus; and they're not mh in different ways. Cyrus does not use the new/tmp/cur subdirectory setup, Maildir does not use CRLF to represent newlines.
Cyrus mailboxes are not designed to permit multiple processes, unaware of each other, to access mail without failure - that was the primary design consideration of Maildir. Cyrus side-stepped that problem, and was therefore able to improve performance more (do an 'ls' in a Maildir with a thousand messages - that's what a client has to do to read that folder).
In short, there are superficial similarities of design, but they are different.
--Matthew
common sense tells you to store binary files in the file system. Include a URI or path or or whatever in the DB. Believe it or not, direct file access is faster (on most OSes) than a database. You gain nothing by including a blob in the DB. It's not searchable, and it slows other searchs down. The only draw back is that you couldn't do this if you were trying to code your solution completely in SQL or for some other reason are not able access the filesystem directly.
Perhaps because at some point you as a user are likely to switch mail client, and may have mail you want to migrate. But anyway, this discussion doesn't exactly seem to be a discussion of whats nice for end users, does it? It's a useful discussion for anyone designing or deploying mail systems of various types, including MTAs and MUAs.
I wonder...
would a tarred maildir decrease the number of disk reads (renames would be trickier, but possible) and inodes, or would the tar overhead be greater than that of the filesystem?
I see the problem here. You are attempting to use Evolution when the mail client you were actually wanting to install is called "mutt".
If you don't like GNOME and GTK+, for the love of pete don't use a mailer that says in big flaming letters "I am a GNOME program!".
News for Nerds. Stuff that Matters? Like hell.