Slashdot Mirror
Ask Slashdot: Handling and Cleaning Up a Large Personal Email Archive?
First time accepted submitter txoof writes "I have a personal email archive that goes back to 2003. The early archives are around 2 megabytes. Every year the archives have grown significantly in size from a few tens of megs to nearly 500 megs from 2010. The archive is for storage only. It is a mirror of my Gmail account. The archives are both sent and received mail compressed in a hierarchy of weekly, monthly and yearly mbox files. I've chosen mbox for a variety of reasons, but mostly because it is the simplest to implement with fetchmail. After inspecting some of the archives, I've noticed that the larger files are a result of attachments sent by well-meaning family members. Things like baby pictures, wedding pictures, etc. What I would like to do is from this point forward is strip out all of the attachments and only save the texts of the emails. What would be a sane way to do that using simple tools like fetchmail?"
Storage is cheap and 500MB are hardly worth worrying about. The damage done by reducing that amount will likely be far larger then any temporal benefits you might get. If you want to have it smaller so that you can have faster search, look for a tool that is better at searching and indexing the mails instead of trying to cut the mail into pieces.
My email archive dates back to 1999 and is 2GByte in size which isn't much considering the attachments.
I "handle" it by making a backup of it.
I do not clean it up. I do clean around it by deleting mail archives that contain mails that have no personal value.
I do not delete personal mails since it is precious like photos.. In 2011 nobody has to delete his personal mail..
This news is stupid
Surely a tool exists to keep email in a SQL database, so the envelope fields, plain text, and attachments are separately searchable. I have email back to 1996 with the same frustrations.
One would think that Thunderbird would have done that a decade or more ago, but no. Nor does any of the standard IMAP servers seem to support SQL (MySQL, Postgres) as a backend: This seems like a serious project waiting to happen. Or have I overlooked an obvious solution?
This problem seems almost too simple, text-only and only up to 500MB per year.
I have a much tougher problem, a mailbox that is growing about 5GB per year that I still need searchable. And, stripping out the attachments is not okay, I need a way to still access them since many of them are receipts in PDF or edits on documents where the e-mail trail is the only record of changes over time. Thus ideally, the attachments should be indexed as well.
I guess you could do what many apache.org sites use: mod_mbox to make a web-accessible version of your your mail folders, possibly pre-processing them with an mbox splitting tool to get them into bit-sized chunks.
Then, overlay a search tool like Lucene Imagination (which is what lucene.apache.org uses) or any other local web indexer of your choosing in order to build searchability.
Nothing to see here. Move along.
Deleted
What rubbish! Don't talk about stuff you know nothing about. Go do your homework kid.
If you're not following Sarbanesâ"Oxley, just delete it. Fuck the pack-rat mentality.
Many people have a larger email store than you.
It is not a sign of status.
More likely, it is a sign of your incompetence to filter and save relevant data.
Congratulations.
Now back to the OP, who perhaps is smarter than you, since he has has just 500MB of email to back up.
My gmail inbox is using 2.7GB, or roughly 34%. I know someone using more than 70%. They provide a way to get more room for a reason.
Just keep it all, and as other people have said, index it.
What rubbish! Don't talk about stuff you know nothing about. Go do your homework kid.
yep, creating a friggin web app sure isn't the easy solution.
the question asked here is actually "anyone know a script that would go through mbox and detect where an attachment starts and ends and just strips those away".
the common sense answers are though "don't strip them away, you'll lose the content and besides storage space is cheap".
storage space is cheap, both local and cloud. I wouldn't trust too many cloud startups to be out there in a while though..
world was created 5 seconds before this post as it is.
fetchmail + procmail sorted into different folders in maildir format.
I don't auto strip attachments on large mails just sort them out from the rest, but It would be easy to add.
I have a maildir folder for inbox, outbox, notme (e.g., stuff addressed to distribution lists), large (here go all the mails with massive attachments).
large, for me, is manageable to go through manually. It only has a few tens of messages / yr.
If I had more to go through, procmail could call a simple script to strip the attachments on all the mails that are large enough to currently get sent to the "large" folder. I have something setup like this already to train a Bayesian filter on mail dropped to certain folders.
Here is the relevant procmail (pretty simple to do):
# if the message is huge, probably don't want to archive it, even if directly :0:
# to me
* > 200000
${DUBIOUS}.large/
IMAP is another potential answer.
I run Dovecot locally, and it stores every mail I've ever received, indexed for quick searches.
This way I can get my mail with all history and a fast search index on all my devices also.
Blessed are the pessimists, for they have made backups.
There is probably some email that you need to keep, but chances are that you don't need to keep most of your email. So just read, respond, then purge (when appropriate).
As others have pointed out, disk space isn't really a concern this day in age. But managing data that you don't need is a concern. A minute spent filing, backing up, etc. of unnecessary data is a minute wasted. Add enough of those seconds together, and it may amount to a good chunk of your life spent doing more interesting/productive things.
As a side note, I notice that people sometimes get attached to things that don't really matter to them. I've known people who have lost all of their data due to circumstances beyond their control, then they became very distressed about that loss of data. The problem is that only a tiny fraction of that data was actually valuable, but they were worrying about all of the data. In some cases it was so traumatic to them that they spent more time worrying about the irrelevant stuff than the stuff that they would need to continue on in the future. So if you don't keep the irrelevant stuff, you can focus on what is relevant.
Even at today's post-Thailand-flood inflated hard drive prices, your entire e-mail history occupies less than a dollar's worth of disk space. I fail to see the issue.
For my own mail archives I just use mutt and weed things a bit by hand. I find that 90% of the mbox size is in fewer than a dozen attachments, so I can hand-filter those out in ten minutes once a year. Beyond that disk is too cheap to care and time is too valuable to make a really comprehensive solution. So what I do:
'mutt -f archive.mbox'
':set pager_index_lines=6' (Lets you see the message index split above the body)
'o' (Order), 'z' (siZe), End (last entry), Enter (Open).
while(mbox.size > acceptable_size)
{
'v' (View attachments)
'jjj' (down a few times to the attachment I want to nuke)
'd' (Delete)
while(more attachments) { 'd' (Delete more attachments) }
'q' (Quit back to the message view)
'k' (previous message)
}
'q' (Quit back to index)
'$' (Sync changes to disk)
'q' (Quit mutt)
Note the 'j' and 'k' are vi-style up/down. The arrow keys work too if you're not a home row junkie like me.
I don't know a good fully automated way to do this that's ready to slice it right out of the box. If you want to roll your own, just pick up a library like RMail or TMail for Ruby, or equivalent for the language you prefer. That's 80% of the work done but you'll still probably find a dozen corner cases involving oddly-named HTML-alternatives named things that look like binary attachments or terribly malformed spam.
The Eudora Mail User Agent (i.e. email client) stores attachments in a directory as binaries but yet keeps the text of emails intact. Thus you should be able to import the email into Eudora, then when you export it the attachments should be stripped.
This is also exactly why I don't use Eudora anymore, because attachments get stripped off when exporting the email (or at least that's the way email export or import from/to Eudora worked last I used it).
Now, although this explains one way attachments can be stripped from email, I don't recommend doing that, because it alters the email. Generally I want email intact because otherwise what you're storing might refer to an "attached file", but yet not even knowing what the filename is that the email refers to. Plus it's actually useful to have sent mail attachments intact too, because it means you get to see what version of what file you sent to someone at the time.
There are some other interesting options; the KMail MUA has an option of "delete attachment" when right-clicking on an email attachment, which does delete the attachment but not the reference to it, so you at least know the filename of what used to be attached. I just sent myself a test email and deleted the attachment and then viewed the email raw, but unfortunately Slashdot's filter won't let me send the result. But if you do that yourself and look at it, it should give you an idea how to re-form emails to strip attachments but not the references.
Google keeps a permanent copy anyway...
"I love my job, but I hate talking to people like you" (Freddie Mercury)
Who still uses e-mail?
People who get stuff done instead of being interrupted every 5m? And who want to receive messages even while offline? And have decent systems for archiving, tagging and searching them?
Dilbert RSS feed
We're worrying about 500MB?
Even at today's outrageous price-fixed (you know it's true) hard drive prices, you're talking 14 cents a GB. For your situation, we're talking 7 cents.
You're complaining about 7 cents worth of storage space. And to cut down on this you want to mangle the archive?
You're tight on space? Buy another drive, burn to CD/DVD.
For those of us who grew up with a Corvus shoebox hard disk costing thousands on the Apple ][ network, this is a ridiculous "ask slashdot" question.
--
BMO
back to ARPA mail and UUCP mail days...
for a while I used Eudora and every month religiously took each piece of email and filed it away in suitable mail folders. After Eudora started declining and I got too busy, I stopped that, but even now, religiously every month I clean out my mailbox of all junk and unwanted attachments (trimming 60-100MB to usually 20-30MB) and then stack that months email away as a single mbox file, and start fresh with a new Inbox.
the old mailbox files are on an IMAP server that I can easily read emails from at least 10 years ago -- older with a little more effort. As single mbox files each, I can do greps on them also. Seems to be an okay way to keep the stuff, some of which has proven to be important over the years....
another big help: all semi-junky and non business emails I let Hotmail do the work (vendor stuff, Amazon orders, etc). Have been using Hotmail since before MS bought it. Works well as a place to direct mostly junky vendor stuff.
Google for "procmail remove attachments":
http://osdir.com/ml/mail.procmail/2002-11/msg00091.html
That will get you started. You can do most anything with Procmail after you figure out the rather odd configuration file format.
Make sure you have it backed up first because it's also quite easy to destroy data with Procmail.
After you spend a lot of time futzing with Procmail scripts and sed and formail and the like, you'll wonder why you didn't go on Amazon or Newegg and buy a $10 flash drive that will hold all your mail several times over.
Penny - plain text accounting
Amateur. When you get to 8+ gb then we can talk about 'large archive'. Until then, just stick it on a CD.. you don't even need a DVD for that.
---- Booth was a patriot ----
Why delete when disk space even today is 14 cents in "Salesman Gigabytes"?
Someone back there said he has 16 GB of mail going back over a decade. That's what Two Bucks And A Quarter.
It's less than a cup of coffee at Starbucks or even a Large at Dunkin' Donuts. It is fully irrational to worry about this.
Anyone worrying about personal mbox size has OCD issues. Full stop.
--
BMO
you will be very sorry you deleted those pictures. don't do that. Even right now, you could make many people very happy by giving as gift one of those digital picture frames that display different stored photo every several seconds, with your pictures of those important to recipient.
My entire mail store is over 16 Gb. I have single mbox files that are larger than 2 Gb.
My entire mail store is over 1TB.
I have single LZMA compressed mbox.xz files that are larger than 16 Gb.
Photos are one of the most treasured things in many families. Keep in mind it's highly unlikely Aunt Petunia is keeping great backups of her photos, and when it all goes south, you might be one of the family members who actually has a photo of a relative who has passed on that she wants to print when her hard drive gives up the ghost.
Enables you to save everything off line as a pdf. Personally I don't get the question or see the point. My archive is about 6 Gig, all backed up all searchable. Anyway the company that makes the software is www.spotdocuments.com Just back up
Watch those corners
We all think you're crazy, but here it is:
#!/bin/env python
from mailbox import mbox, mboxMessage
orig_mb = mbox(path/ot/orig/mbox)
new_mb = mbox(path/to/new/mbox)
for key,msg in orig_mb.iteritems():
new_msg = mboxMessage()
payload = msg.get_payload()
if msg.is_mulltipart():
payload = payload[0].get_payload()
for header in msg.keys():
new_msg[header] = msg[header]
new_msg.set_payload(payload)
new_mb.add(new_msg)
new_mb.flush()
this signature has been removed due to a DMCA takedown notice
2000? Hell, I have my email back to the early 1980s.
The real problem is that back then it was OK to put all messages in one file, and having one message per file is far more useful for searching with grep.
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
yay, my bad. I always forget this is a site where the men are still men and anything that doesn't involve writing an obscure script only known by its creator is forbidden (and a browser to share? Jesus!)
Anyway, if are there any other pussies like, me there are tons of ways and utilities to convert mbox to maildir, and then it's easy to parse them to an DB (i'd recommend sqlite, it still is a simple file(s) in a directory, and the overhead would be minimal)
FTS: "I have a personal email archive that goes back to 2003. The early archives are around 2 megabytes. Every year the archives have grown significantly in size from a few tens of megs to nearly 500 megs from 2010."
So, the total space required thus far is definitely less than (8 * 0.5 GB) = 4 GB. A USB flash drive with that small a capacity is practically classified as electronic waste these days.
Even if his or her annual e-mail archive size doubled every year for the next 10 years, it would only be 1+2+4+8+16+32+64+128+256+512=1023 GB.
A 3 TB hard drive he buys *today* for $100 would probably solve his "problem" for 10 more years.
Hopefully, in the year 2021, we will have tiny 3 PB SSD drives for $100... But maybe we will be ruled by an A.I. by that time, if we haven't already destroyed ourselves with viruses, nanomachines, robots, nuclear weapons, etc.
Many attachments are in the mailspool lots of times. This is how google started allowing massive amounts of email storage, they only store a given attachment once (or so) even if its in a million email messages.
It would be nice if there was a better way of going through the archives and moving the attachments off to one place to deduplicate things.
As others have said, the headache you will have if you do want to come back (potentially years later) to that one email you know you had only to find your attachment-stripping program has foobar'd the whole archive up (or that you need the attachment after all) probably isn't worth the hassle for saving 500MB per year this year (even taking into account reasonable growth rates - I'd note that bandwidth per $, which will be the factor limiting your email size, has been growing rather more slowly than storage capacity per $ over the past decade and things are likely to continue that way).
If the problem is that you have significant duplication between emails (e.g. the same attachment being emailed several times), gzip and bzip2 may well miss the opportunity to de-dupe this because the distance between duplicated sections is large. One solution to consider if this is an issue may be to use something which is better at compressing over long distances. I would suggest trying something like lrzip to compress tarballs of the annual sets of mbox files before archiving those.
Of course, if you just have lots of attachments which *aren't* duplicated (which is probably more likely), that won't really help much.
2000? Hell, I have my email back to the early 1980s.
The real problem is that back then it was OK to put all messages in one file, and having one message per file is far more useful for searching with grep.
Actually I find this less of an issue. Check out grepmail and mboxgrep. I use these pretty regularly and they're very useful for doing e.g. grepmail 'foo.*bar[a-z]' ~/Mail/mbox.gz >/tmp/messages; mutt -f /tmp/messages
Actually I find this less of an issue. Check out grepmail [sourceforge.net] and mboxgrep [mboxgrep.org]. I use these pretty regularly and they're very useful for doing e.g. grepmail 'foo.*bar[a-z]' ~/Mail/mbox.gz >/tmp/messages; mutt -f /tmp/messages
HOLYCRAPTHANKYOU!!!! You have no idea how much time and headache you've saved me. Well, maybe you do, given that you can imagine how much of a pain it is to use plain grep on mbox and similar files.
Put my fist through my alarm clock with its ding-dong death inside my ear. - The Blackjacks.
These are a few of the tools that I use (Unix/Linux, of course):
formail (part of the procmail distribution) is very useful for rewriting mailboxes.
uuexplode is useful for discovering and yanking out attachments.
grepmail is REALLY useful for discovering messages which match certain criteria.
csplit is useful for more than mail, but it also has applications with mailboxes.
You're in good luck! The same author is developing a similar system in a client/server model, with the server (called Heliotrope) doing the actual work. You just need to write the Linux and Android clients. A client already exists, but it's ncurses based.
Dilbert RSS feed
My personal email archive goes back to 1996, and is still only 262MB.
My Google archive uses 164MB.
I've no idea what my Yahoo account uses.
But 500GB of email?!?!?!?! Are your relatives sending you entire videos as attachments or accidentally copying their entire music archives?
I do not fail; I succeed at finding out what does not work.
Deal with the superfluous attachments first and then see how you feel. Attachments are often unnecessary baggage.
You are talking about $0.035 in storage costs
Literally. I bought 3TB for $200.00 at Fry's yesterday. You probably have more than 500M of "Angry Birds" on your cell phone.
If the point is to pull the data off your gmail account and not have it stored there (maybe you want to migrate away, maybe, you are trying to get us to design a product for your file hosting service adjuntct to gmail, whatever), fetchmail is a terrible tool, particularly since gmail permits IMAP4 access, and you don't have to worry about decoding headers in precedence order.
In general, if you want to leave the email on your gmail account, and delete only attachments, sorry, but the mail is not stored that way on the gmail server, messages are stored as units, and attachments are not separate things, they are different sections in the same flat file using MIME encoding.
You would have to pull the mail down with IMAP4, process it, and push it back up, again with IMAP4. You would then need to take additional steps via IMAP4 commands to make it not look like newly arrived mail to the gmail server, or you're going to see them all as new messages the next time you log into gmail (basically, after the put, you will need to mark the message read again).
If you additionally use POP3 access, the message IDs as reported by POP3 will change, and since it does its "leave on server" functionality by maintaining a local database of message IDs, it's going to be seen as new email there. There's no getting around that, unless you own the source code to your POP3 client and are willing to do correlation after the put operation on the IMAP4 connection to translate the ID.
You also realize that if you are ding this for reasons of quasilegality of message contents, those messages aren't really gone, right? People accidentally delete things all the time and want them back, and that same recovery process in that case would be usable in discovery by subpoena for the email provider (there is in fact additional requirements for ISPs under Patriot II to maintain records of sent and received emails for up to two years).
The bottom line here is that you are engaging in a pretty useless exercise here, unless you are trying to hide illegal activity or build a service and have us design it for you. In either case, good luck, you'll be writing a lot of code to get what you want.
-- Terry
Is a project a friend my mine started. Interfaces w/ gmail's API, quite easy to use.
www.findbigmail.com
I have an issue with some of my old Thunderbird mail archives. They are infected with various viri (w32 swen.A, N32 Netsky.T, Trojan Zbot) The anti-virus software I've tried just wants to delete the entire file not clean it. Would like to clean the files without risking infection but haven't been able to find a way to clean an offline mail file. Any ideas? Thanks.
I keep my email in maildir format (the default for Claws-Mail), and rotate every six months. The whole process is entirely manual, but since any given step only takes a few seconds, it works fine for me.
Emails are sorted on receipt according to source or content via ordinarily filters. Every email I receive that's worth keeping goes into a catch-all folder after reading. I probably should be preserving the sorting when I move something into that catch-all, but I don't receive enough email to bother with it yet.
The start of the current six-month range is always part of that folder's name, e.g. "2011-07-01 to Current". Every six months, I rename the folder to add the end date (e.g. "2011-01-01 to 2011-06-30") and move it into a separate storage folder (still within Claws-Mail's folder tree). Then, I simply create a new Archive folder for the new 6-month period. Fill, rinse, repeat every six months.
When the mood strikes me (roughly every couple of years), I'll compress the latest six-month block(s) and move the results into a long-term storage directory. I generally keep only the most recent few years' worth of emails at hand, so the oldest stuff gets deleted from time to time, leaving only the compressed files. If I need to search the older stuff, it's a small matter of extracting to a temp/work directory, doing whatever needs done, and deleting.
On top of that, I run an incremental backup of my home directory and storage areas to a USB-connected disk every so often (the time between backups varies - usually once a fortnight or more often). So eventually, every email I decide to save ends up with one online and at least two offline backups. Since I use Gmail, technically they serve as an off-site backup of the most recent stuff (until I delete it anyway).
I figure with this setup, it's easy to find whatever I need, and it would take a pretty big screwup to actually lose an email.
How about procmail (for new mail) or formail (for iterating through messages in existing mailboxes) and a perl-Email-MIME-Attachment-Stripper script?
Strip (or separate and save) the attachments from a mail message.
$ du -hs ~/Mail /home/spugglefink/Mail
1.9G
Ask yourself: When are you ever going to read all those email again? When is *anybody* ever going to read them again.
1) I need to order ink recently. Now, I don't print much but I vaguely remembered a good supplier that I had used in the past. But what was it called? A few moments of greping and I found it: in a confirmation email from three years ago.
2) I met a woman on a Meetup hike recently that I seem to have met before. Was this the blind date from four years ago? The smoking gun was in an email from 2007.
3) I've had occasional need to look up old acquaintances. While I might have created a contact file at various points, odds are I have forgotten what I named it or where I put it. But I am quite sure the information is in email.
The real treasure is email that is ten or more years ago. You think you remember what happened in the right order? Trust me. You don't. An email archive is like a diary except it is less work and more complete.
If you use, or have access to a Mac, the Apple Mail client has for some years had a Remove Attachments option in the Message menu. Simply select all your mail in a folder with Cmd-A and select that menu option and it'll do exactly what you want. I use it regularly to prune my database.
It's a Unix system - I know this.
Look...
You've never had to look back thru your email for stuff
You don't feel the need to have email from 1994 searchable
You've been lucky enough to never have to find a contact that you lost
BECAUSE
You're too young to 'have a past'.
You've never held a job with any type of seniority
You've never held a job that was important
You don't have friends or intimate relationships
You, are not interesting or important enough to temporally affect
anything beyond a few dozens of hours
and you could probably disappear or die in your hovel and |except|
for bills and rent not getting paid and perhaps a (more?) foul odor
than before...
NO ONE WOULD MISS YOU
But see... that's why this question exists... because we do
have friends, family, loved ones, jobs that are or were important
and people would miss us, if we fuckin took a 1hr nap in the
middle of the day! So, do us a favor and STFU... because us
important folk with friends want to save our emails til the
angels blow their trumpets.
Thanks for playing!
-@|
...and forward everything there.
Or set up (an) imap server(s) on additional machine(s) and use imapsync, or similar, to back everything up.
I have all my emails since 1990 or 1991 :-/ and it's true that I use less and less emails...
I saved them in a mbox format and used a script to remove binary attachement (yeah, cat pictures and things like this). It's mainly text and can be highly compressed.
"Science will win because it works." - Stephen Hawking
I still have the tapes, but the desire to read them. Since the mid-1990s I've have had cloud-email (hotmail) and havent really lost anything.
Email is extremely convenient for file transfer, but I prefer not to have my mail store so bloated.
What happens when my friends want to start emailing me movies?
I haven't figured out the true nature / fundamentals involved here.
http://www.mhonarc.org/ is a perl script that will recurse though folders of emails, unpack the attachments and generate html versions of the mail, with links. A simple find can them delete the attachments, or you could just keep them as the result will be (mostly) smaller.