How Do You Store and Reconcile Email Archives?

heyitsjustme wants to know how you deal with old email. "I delete most of what I get but keep the stuff from friends and relations as an archive. Unfortunately I have these email archives from the late 80's through today in the form of macintosh, linux and windows mailboxes including AOL 1.0 mailboxes. What does everyone use to archive email across multiple platforms and non-standard mailbox formats? Is there an easy solution out there? Does anyone archive IM?"

Comments

Upon Searching..

[yuriismaster]

EmailMan has the answers to your problem.

More utilities than I want to bother with, but hopefully they'll have the converter(s) you need.

Good Luck!

Your favorite online storage

[Alien54]

Hriders.com gives unlimited free 1 Terabyte email accounts that include 500 Megabyte attachments. We have been asked why we would do such a thing. The answer is simple to help people store large amounts of information in a safe and secure environment. - - - We decided that yes a Terabyte of space may sound rather extreme to some, others will not think so. If you have a free membership with Hriders.com then you will receive a free 1 Terabyte 500 Megabyte attachment email account. You will be able to store over 40 million emails, videos, games, mp3s, or pictures.

This might be useful, if they don't collapse under /.

Don't change e-mail clients

[rueger]

That's one of the many reasons why I have stayed with Pegasus Mailfor many years. Because they were created in the same program I know that I can still access my old mail files without problems.

What I do at year end is move all of that year's messages to a new folder and reset my filters so that the new year's messages go into a new set of folders.

Periodically I just copy off previous year's messages to CD.

At least few times I have been able to back a couple of years and find information that I lacked.

formail, mairix, and mutt

[rsw]

Convert everything into mbox format. formail will help you with that.

Use mairix to search through email.

mutt is the best mail client ever.

-rsw

Or use maildir

[suso]

Whatever you do, I think its best to keep it in an open and obvious format like mbox or maildir. The nice thing about maildir though, is that since all the messages are seperate, it might be a little easier to write a program to put them into a new format.

Personally, since 1999, I've been using a combination of maildir and procmail to archive and save my mail. Every message that comes in, goes to a folder called .saved-messages-YYYY-MM and also to my inbox. I simply don't touch the saved-messages folders and when I am done with the message in my inbox, I just delete it. This has worked well for me and makes it much easier to deal with archiving old mail. In the end, having categorized folders and such is just a waste of time. Its kinda like the wm2 (window manager) way of thinking, but for mailboxes.

How I do it

[Matt Perry]

I use a procmail recipe to archive my mail. I put it after filtering mailing lists and before I filter spam:

OLDMAILDIR = $MAILDIR
MAILDIR = $ARCHIVE_DIR
:0 cW: archive.lock
| /bin/gzip >>mailarchive-`date +%Y%m`.gz
MAILDIR = $OLDMAILDIR

I use grepmail to find old emails that I might need. Grepmail lets you use perl regular expressions to find messages and then outputs the entire message where a match was found. You can use grepm to open grepmail matches as a mailbox in mutt. grepine does the same for Pine, which I use.

At the end of each year I clean the spam out of my archives using a procmail recipe and spamassassin. This recipe marks messages as deleted in the mailbox. I open these in pine, sort by deleted, and double check them. Once I'm sure they're all spam, I delete them:

# vim:ft=procmail:

LINEBUF = 8192
SHELL = /bin/sh
MAILDIR = $HOME/mail

:0 fW: spamclean.lock
| spamassassin -e --prefs-file=/home/matt/.spamassassin/user_prefs-s pam_clean 2>/dev/null

# If the message was deemed to be spam, set the status to "deleted" so that
# we can delete it easily and optionally review it.
:0 e
{
:0 fhw
* ^^rom[ ]
| sed -e '1s/^/F/'

:0 f: formail.lock
| formail -I 'X-Status: D'
}

# Fix the mangled "From" line
:0 fhwE
* ^^rom[ ]
| sed -e '1s/^/F/'

# Remove the last of the SpamAssassin headers
:0 f: formail2.lock
| formail -I 'X-Spam-Checker-Version'

# File message in temporary mailbox
:0: sandbox.lock
z-cleaned_mbox

The special spamassassin config turns off bayesian filtering and sets the threshold high:

required_hits 15
clear_headers
fold_headers 0
use_bayes 0

The rest of the spam I clean out by hand.

Re:How I do it

[Matt Perry]

Almost forgot. I archive my sent mail as well. This might be harder for you if you don't use a single email client on a single machine. IMAP can help with that.

Put this in ~/bin/rotate-sent-mail.sh:

#!/bin/bash

# This script takes sent mail in $HOME/mail and moves them into
# $HOME/.mailarchives/sent. It will also rename the file to have the date of
# the log file included.

MAILDIR=$HOME/mail
ARCDIR=$HOME/.mai larchive/sent
year=`/bin/date +%Y`
month=`/bin/date +%-m`

# updating last months mail
month=$((month-1))

# if this is last years mail, set the date correctly
if [ $month -eq 0 ] ; then
month='12'
year=$((year-1))
fi

# if the month is less than 10, add the leading zero back
if [ $month -lt 10 ] ; then
month=0$month
fi

mv $MAILDIR/sent-mail $ARCDIR/sent-mail-$year$month
touch $MAILDIR/sent-mail && chmod 600 $MAILDIR/sent-mail
bzip2 -9 $ARCDIR/sent-mail-$year$month

Now add the following to your crontab:

0 0 1 * * $HOME/bin/rotate-sent-mail.sh

Outport & recursive IMAP folder creation

[ahbi]

I strongly recommend Outport. It does an extremely good job of converting MSFT Outlook attachments into something more readable (mbox I think, it has been a while). MS Outlook usually mangles attachments into some wrapper called TNEF.

Also, anyone know of a client program that will recursively create folders on an IMAP server (maybe a server issue. In which case, what server?)
I had gotten over translating my years of Outlook email into something more universally readable, but I have so many nested folders that the inability to have the client recirsively create IMAP folders is an issue. Suggestions?

Re:Kinda Sorta OT

[gunfleet]

Is there a package that can read the mbox, the other box-formats, plain text, pull from pop, old tar.gz bundles, categorize (sorta), tag and make such things searchable?

Yes there is, check out

http://www.greenstone.org/cgi-bin/library