Slashdot Mirror
How Do You Store and Reconcile Email Archives?
heyitsjustme wants to know how you deal with old email. "I delete most of what I get but keep the stuff from friends and relations as an archive. Unfortunately I have these email archives from the late 80's through today in the form of macintosh, linux and windows mailboxes including AOL 1.0 mailboxes. What does everyone use to archive email across multiple platforms and non-standard mailbox formats? Is there an easy solution out there? Does anyone archive IM?"
No need for rear view mirrors. What is behind you is not important.
Some drink at the fountain of knowledge. Others just gargle.
I archive all my pr0n on DVDs these days. It's really easy and oh wait... fsck!
rm -rf
Save it all. With the exception of some mail archives lost to catastrophic disk failures (I keep archives for my own convenience, not for any official purposes, so I don't back them up), I keep all my email.
Thunderbird is able to import all my old mail archives (from years and years of Eudora) and search it effectively. If I were inclined to export all my archives from my Mac to my Windows machine, I could use Google Desktop Search to really search through it all.
...so I just delete everything after a major deal falls through.
the best way to consolidate various types of emails may be to email them to a common source. Then archive from there.
I delete most of what I get
You must work for microsoft
----
Squirrel
I use the basic Unix mail format, essentially plain text series of messages. Eudora does fine with it; and most anything else can read/import it. I have email going back to the 80's in this format. The one time I had to convert was when I was working for a company that used "Quickmail" on the Mac. I wound up reverse engineering their format and hacking up a program to convert it to plain text.
I delete almost everything, and only save a few very important or personal emails. For those I do keep, I print to PDF, and archive by date and person/subject. It works exceptionally well for me. It is all electronic, takes very little disk space, and keeps the clutter to a minimum, and eliminates most of the cross platform nightmares.
One word: IMAP. If you can read your email using any decent email client, it should support moving it to an IMAP server. If you are using web-based email or some crappy client which can't export emails to a standard/raw format, you'll have to write a script to convert the messages.
Ever since I first got acquainted with e-mail on my Apple IIe in the '80s, I've used e-mail programs that offer plain-text storage as at least an option. It's one of the most universal formats in existence, and can be read one way or another on computers both decades old and brand new. I encountered some weird proprietary clients in the '90s that still stored e-mail in this format, because from a corporate perspective, this stuff was still in its infancy, plus HTML hadn't yet mucked everything up. To this day I still store in plain text from Eudora 6.2.
I burn it to CD-Rs that I know won't get moved around or scratched. They stand a good chance of lasting the rest of my life.
The coolest voice ever.
EmailMan has the answers to your problem.
More utilities than I want to bother with, but hopefully they'll have the converter(s) you need.
Good Luck!
This might be useful, if they don't collapse under /.
"It is a greater offense to steal men's labor, than their clothes"
That's one of the many reasons why I have stayed with Pegasus Mailfor many years. Because they were created in the same program I know that I can still access my old mail files without problems.
What I do at year end is move all of that year's messages to a new folder and reset my filters so that the new year's messages go into a new set of folders.
Periodically I just copy off previous year's messages to CD.
At least few times I have been able to back a couple of years and find information that I lacked.
Three Squirrels
but ...
... just wondering as the Submitter did what i like /. Submitters to do: make me think and look for new, better stuff ... or better ways to do old-stuff.
:)
Along these lines, is there an OSS package that can read the varied formats the Submitter is referring to, tag and drop them in a DB with a nice, friendly, web-enabled (secure) front-end for searching?
My former employer kept *all* of his email from the last 20 years in tar.gz files. Let's just say it wasn't easy to find an email from er, 15 years ago very easily.
Is there a package that can read the mbox, the other box-formats, plain text, pull from pop, old tar.gz bundles, categorize (sorta), tag and make such things searchable?
Totally a shot in the dark here, i'm not a mail guy at all
It is the "drink" that makes me wonder, sorry
Almost every email client around can import and export mbox formats. Getting your email in a format that is going to be readable in 20 years is the first step, otherwise why bother?
Worse comes to worst mbox is readable as plain text.
Gmail?
I don't know about you but I generate about 6GB of email archives per year. Besides that having my email potentially available for searching doesn't sit well with me. I'm not sure where it stands now but there were a lot of potential privacy issues with Gmail.
No I don't receive hords of email, just a lot of engineering related with source code,research, white papers attached. If you do anything business related it's important to keep all of the original emails received so there is an electronic paper trail.
Don't ask Microsoft.
Democracy is two wolves and a sheep voting on lunch.
I never keep emails, or archive IMs or any other form of communication. Once a email is read, it is deleted. Same goes for normal old-skool mail, I read it and then trash it. The only exceptions are of letters/email of some importance such as information I need to keep handy, or if it has some kind of sentimental value (letters from deceased relatives for example.)
Sure, HDD space is cheap; but I tend to equate people who archive every single form of written communication to those who have an Obsessive Compulsive Disorder, in that they hoarde everything in sight: newspapers, snail mail, magazines, boxes, etc..
Commit to memory and destroy the evidence. Thats my way of handling archives.
Convert everything into mbox format. formail will help you with that.
Use mairix to search through email.
mutt is the best mail client ever.
-rsw
Combine this with spotlight/tiger in mac os. Spotlight indexes PDF content. print it to pdf and it will be searchable. Assuming you have a Mac that is.
Some drink at the fountain of knowledge. Others just gargle.
I log and keep all my traffic including IRC logs going back to '94.
Hey B5_geek, here's a trick to free up a lot of disk space *and* raise the S/N ratio in your logs:
mv irclog.txt irclog.txt.fat && grep -vi lol irclog.txt.fat > irclog.txt && rm -f irclog.txt.fat
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
Whatever you do, I think its best to keep it in an open and obvious format like mbox or maildir. The nice thing about maildir though, is that since all the messages are seperate, it might be a little easier to write a program to put them into a new format.
.saved-messages-YYYY-MM and also to my inbox. I simply don't touch the saved-messages folders and when I am done with the message in my inbox, I just delete it. This has worked well for me and makes it much easier to deal with archiving old mail. In the end, having categorized folders and such is just a waste of time. Its kinda like the wm2 (window manager) way of thinking, but for mailboxes.
Personally, since 1999, I've been using a combination of maildir and procmail to archive and save my mail. Every message that comes in, goes to a folder called
One word: IMAP
...who knows what else. I've got freedom to try whatever I want at any given moment without losing my current or past mail.
Absolutely. I use no fewer than two mail clients on two different machines on any given business day. Every email I've sent since 1995 or something like that, and received since 1998 is available and searchable. Over this time, I've accessed this archive with the following clients:
* pine (lots of pine)
* mac mail
* thunderbird
* various netscapes/mozillas
* ML (some random IMAP reader)
* My phone (my old Sony/Ericcson speaks IMAP)
* My palm (two different apps)
* python
* a java webmail system I wrote
* three or four other webmail systems
* mutt
-- The world is watching America, and America is watching TV.
I have several CDs worth of stuff archived with ForKeeps:
http://www.fkeeps.com/whofor.htm
It's a bit of an old program and the interface is clunky, but it works reasonably well once you work through it.
--Steve
That way it won't be subject to a sub poena. You'll regret it one day if you don't. Do you realize how much incriminating stuff you have in there?
Doesn't it make you feel good to know that our freedoms are protected by politicans, lawyers and journalists.
10 Your mother told you to stop being such a pack rat.
9. Disks fill up, no matter how cheap they are. Low cost doesn't excuse gluttony.
8. Backups take forever.
7. Restores take an eternity, especially if your not confident.
6. Mail client gets slower and slower.
5. Searches take too long.
4. Mail clients make mistakes, especially on big stores. See #7
3. Your CYA evidence may be used against you.
2. A mail store is not a file system and SMTP is not a file transfer protocol.
And the number one reason to delete your old email...
1. IT'S ALL A BUNCH OF USELESS CRAP JUST AS IT WAS WHEN YOU FIRST RECEIVED IT!!
6GB yearly? Holy shit...
Do you actually sign up to those free porn places?
I use grepmail to find old emails that I might need. Grepmail lets you use perl regular expressions to find messages and then outputs the entire message where a match was found. You can use grepm to open grepmail matches as a mailbox in mutt. grepine does the same for Pine, which I use.
At the end of each year I clean the spam out of my archives using a procmail recipe and spamassassin. This recipe marks messages as deleted in the mailbox. I open these in pine, sort by deleted, and double check them. Once I'm sure they're all spam, I delete them:
The special spamassassin config turns off bayesian filtering and sets the threshold high:
The rest of the spam I clean out by hand.Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
I've found the easiest way to handle EMail when it's in multiple formats like that is to just print everything out and store it in boxes in my garage.
I am NOT a man!
I am a free number!
I strongly recommend Outport. It does an extremely good job of converting MSFT Outlook attachments into something more readable (mbox I think, it has been a while). MS Outlook usually mangles attachments into some wrapper called TNEF.
Also, anyone know of a client program that will recursively create folders on an IMAP server (maybe a server issue. In which case, what server?)
I had gotten over translating my years of Outlook email into something more universally readable, but I have so many nested folders that the inability to have the client recirsively create IMAP folders is an issue. Suggestions?