Slashdot Mirror
Ask Slashdot: Handling and Cleaning Up a Large Personal Email Archive?
First time accepted submitter txoof writes "I have a personal email archive that goes back to 2003. The early archives are around 2 megabytes. Every year the archives have grown significantly in size from a few tens of megs to nearly 500 megs from 2010. The archive is for storage only. It is a mirror of my Gmail account. The archives are both sent and received mail compressed in a hierarchy of weekly, monthly and yearly mbox files. I've chosen mbox for a variety of reasons, but mostly because it is the simplest to implement with fetchmail. After inspecting some of the archives, I've noticed that the larger files are a result of attachments sent by well-meaning family members. Things like baby pictures, wedding pictures, etc. What I would like to do is from this point forward is strip out all of the attachments and only save the texts of the emails. What would be a sane way to do that using simple tools like fetchmail?"
Storage is cheap and 500MB are hardly worth worrying about. The damage done by reducing that amount will likely be far larger then any temporal benefits you might get. If you want to have it smaller so that you can have faster search, look for a tool that is better at searching and indexing the mails instead of trying to cut the mail into pieces.
You have. Thunderbird includes archival folders and a Lucene search engine.
There is DBMail.
If you're not following Sarbanesâ"Oxley, just delete it. Fuck the pack-rat mentality.
Many people have a larger email store than you.
It is not a sign of status.
More likely, it is a sign of your incompetence to filter and save relevant data.
Congratulations.
Now back to the OP, who perhaps is smarter than you, since he has has just 500MB of email to back up.
IMAP is another potential answer.
I run Dovecot locally, and it stores every mail I've ever received, indexed for quick searches.
This way I can get my mail with all history and a fast search index on all my devices also.
Blessed are the pessimists, for they have made backups.
There is probably some email that you need to keep, but chances are that you don't need to keep most of your email. So just read, respond, then purge (when appropriate).
As others have pointed out, disk space isn't really a concern this day in age. But managing data that you don't need is a concern. A minute spent filing, backing up, etc. of unnecessary data is a minute wasted. Add enough of those seconds together, and it may amount to a good chunk of your life spent doing more interesting/productive things.
As a side note, I notice that people sometimes get attached to things that don't really matter to them. I've known people who have lost all of their data due to circumstances beyond their control, then they became very distressed about that loss of data. The problem is that only a tiny fraction of that data was actually valuable, but they were worrying about all of the data. In some cases it was so traumatic to them that they spent more time worrying about the irrelevant stuff than the stuff that they would need to continue on in the future. So if you don't keep the irrelevant stuff, you can focus on what is relevant.
Even at today's post-Thailand-flood inflated hard drive prices, your entire e-mail history occupies less than a dollar's worth of disk space. I fail to see the issue.
For my own mail archives I just use mutt and weed things a bit by hand. I find that 90% of the mbox size is in fewer than a dozen attachments, so I can hand-filter those out in ten minutes once a year. Beyond that disk is too cheap to care and time is too valuable to make a really comprehensive solution. So what I do:
'mutt -f archive.mbox'
':set pager_index_lines=6' (Lets you see the message index split above the body)
'o' (Order), 'z' (siZe), End (last entry), Enter (Open).
while(mbox.size > acceptable_size)
{
'v' (View attachments)
'jjj' (down a few times to the attachment I want to nuke)
'd' (Delete)
while(more attachments) { 'd' (Delete more attachments) }
'q' (Quit back to the message view)
'k' (previous message)
}
'q' (Quit back to index)
'$' (Sync changes to disk)
'q' (Quit mutt)
Note the 'j' and 'k' are vi-style up/down. The arrow keys work too if you're not a home row junkie like me.
I don't know a good fully automated way to do this that's ready to slice it right out of the box. If you want to roll your own, just pick up a library like RMail or TMail for Ruby, or equivalent for the language you prefer. That's 80% of the work done but you'll still probably find a dozen corner cases involving oddly-named HTML-alternatives named things that look like binary attachments or terribly malformed spam.
Sup uses Xapian, it's pretty fast too.
Dilbert RSS feed
Google keeps a permanent copy anyway...
"I love my job, but I hate talking to people like you" (Freddie Mercury)
Who still uses e-mail?
People who get stuff done instead of being interrupted every 5m? And who want to receive messages even while offline? And have decent systems for archiving, tagging and searching them?
Dilbert RSS feed
back to ARPA mail and UUCP mail days...
for a while I used Eudora and every month religiously took each piece of email and filed it away in suitable mail folders. After Eudora started declining and I got too busy, I stopped that, but even now, religiously every month I clean out my mailbox of all junk and unwanted attachments (trimming 60-100MB to usually 20-30MB) and then stack that months email away as a single mbox file, and start fresh with a new Inbox.
the old mailbox files are on an IMAP server that I can easily read emails from at least 10 years ago -- older with a little more effort. As single mbox files each, I can do greps on them also. Seems to be an okay way to keep the stuff, some of which has proven to be important over the years....
another big help: all semi-junky and non business emails I let Hotmail do the work (vendor stuff, Amazon orders, etc). Have been using Hotmail since before MS bought it. Works well as a place to direct mostly junky vendor stuff.
Google for "procmail remove attachments":
http://osdir.com/ml/mail.procmail/2002-11/msg00091.html
That will get you started. You can do most anything with Procmail after you figure out the rather odd configuration file format.
Make sure you have it backed up first because it's also quite easy to destroy data with Procmail.
After you spend a lot of time futzing with Procmail scripts and sed and formail and the like, you'll wonder why you didn't go on Amazon or Newegg and buy a $10 flash drive that will hold all your mail several times over.
Penny - plain text accounting
Amateur. When you get to 8+ gb then we can talk about 'large archive'. Until then, just stick it on a CD.. you don't even need a DVD for that.
---- Booth was a patriot ----
Why delete when disk space even today is 14 cents in "Salesman Gigabytes"?
Someone back there said he has 16 GB of mail going back over a decade. That's what Two Bucks And A Quarter.
It's less than a cup of coffee at Starbucks or even a Large at Dunkin' Donuts. It is fully irrational to worry about this.
Anyone worrying about personal mbox size has OCD issues. Full stop.
--
BMO
Enables you to save everything off line as a pdf. Personally I don't get the question or see the point. My archive is about 6 Gig, all backed up all searchable. Anyway the company that makes the software is www.spotdocuments.com Just back up
Watch those corners
We all think you're crazy, but here it is:
#!/bin/env python
from mailbox import mbox, mboxMessage
orig_mb = mbox(path/ot/orig/mbox)
new_mb = mbox(path/to/new/mbox)
for key,msg in orig_mb.iteritems():
new_msg = mboxMessage()
payload = msg.get_payload()
if msg.is_mulltipart():
payload = payload[0].get_payload()
for header in msg.keys():
new_msg[header] = msg[header]
new_msg.set_payload(payload)
new_mb.add(new_msg)
new_mb.flush()
this signature has been removed due to a DMCA takedown notice
FTS: "I have a personal email archive that goes back to 2003. The early archives are around 2 megabytes. Every year the archives have grown significantly in size from a few tens of megs to nearly 500 megs from 2010."
So, the total space required thus far is definitely less than (8 * 0.5 GB) = 4 GB. A USB flash drive with that small a capacity is practically classified as electronic waste these days.
Even if his or her annual e-mail archive size doubled every year for the next 10 years, it would only be 1+2+4+8+16+32+64+128+256+512=1023 GB.
A 3 TB hard drive he buys *today* for $100 would probably solve his "problem" for 10 more years.
Hopefully, in the year 2021, we will have tiny 3 PB SSD drives for $100... But maybe we will be ruled by an A.I. by that time, if we haven't already destroyed ourselves with viruses, nanomachines, robots, nuclear weapons, etc.
Email isn't stored in SQL, because typically it's rather pointless. Full text search indexing doesn't require SQL, and it's more efficient without SQL anyway. There are some good use cases for storing emails in SQL database, but efficiency isn't one of them.
2000? Hell, I have my email back to the early 1980s.
The real problem is that back then it was OK to put all messages in one file, and having one message per file is far more useful for searching with grep.
Actually I find this less of an issue. Check out grepmail and mboxgrep. I use these pretty regularly and they're very useful for doing e.g. grepmail 'foo.*bar[a-z]' ~/Mail/mbox.gz >/tmp/messages; mutt -f /tmp/messages
You're in good luck! The same author is developing a similar system in a client/server model, with the server (called Heliotrope) doing the actual work. You just need to write the Linux and Android clients. A client already exists, but it's ncurses based.
Dilbert RSS feed
Is a project a friend my mine started. Interfaces w/ gmail's API, quite easy to use.
www.findbigmail.com
Ask yourself: When are you ever going to read all those email again? When is *anybody* ever going to read them again.
1) I need to order ink recently. Now, I don't print much but I vaguely remembered a good supplier that I had used in the past. But what was it called? A few moments of greping and I found it: in a confirmation email from three years ago.
2) I met a woman on a Meetup hike recently that I seem to have met before. Was this the blind date from four years ago? The smoking gun was in an email from 2007.
3) I've had occasional need to look up old acquaintances. While I might have created a contact file at various points, odds are I have forgotten what I named it or where I put it. But I am quite sure the information is in email.
The real treasure is email that is ten or more years ago. You think you remember what happened in the right order? Trust me. You don't. An email archive is like a diary except it is less work and more complete.