Slashdot Mirror
How Do You Store and Reconcile Email Archives?
heyitsjustme wants to know how you deal with old email. "I delete most of what I get but keep the stuff from friends and relations as an archive. Unfortunately I have these email archives from the late 80's through today in the form of macintosh, linux and windows mailboxes including AOL 1.0 mailboxes. What does everyone use to archive email across multiple platforms and non-standard mailbox formats? Is there an easy solution out there? Does anyone archive IM?"
No need for rear view mirrors. What is behind you is not important.
Some drink at the fountain of knowledge. Others just gargle.
I archive all my pr0n on DVDs these days. It's really easy and oh wait... fsck!
rm -rf
I've been going with the Gmail philosophy of storing everything. Until someone gets hold of my password of course. People should generally be more careful with the storage of their online communication. Print what's important and stick it all in a drawer. That's the safe way to do it.
and you are done!
I save backup of my emails and other stuff (including IM).
Then I trash them when I change my computer. (every 4 years)
Save it all. With the exception of some mail archives lost to catastrophic disk failures (I keep archives for my own convenience, not for any official purposes, so I don't back them up), I keep all my email.
Thunderbird is able to import all my old mail archives (from years and years of Eudora) and search it effectively. If I were inclined to export all my archives from my Mac to my Windows machine, I could use Google Desktop Search to really search through it all.
...so I just delete everything after a major deal falls through.
the best way to consolidate various types of emails may be to email them to a common source. Then archive from there.
I delete most of what I get
You must work for microsoft
----
Squirrel
...with fetchmail / procmail / cyrdeliver for sorting and storing from other sources. How can 5GB of mail can't be wrong?! I can slice and dice my all my email (including about a gig of spam...) for choice bits of information.
My God! It's full of Voids!
I use the basic Unix mail format, essentially plain text series of messages. Eudora does fine with it; and most anything else can read/import it. I have email going back to the 80's in this format. The one time I had to convert was when I was working for a company that used "Quickmail" on the Mac. I wound up reverse engineering their format and hacking up a program to convert it to plain text.
I delete almost everything, and only save a few very important or personal emails. For those I do keep, I print to PDF, and archive by date and person/subject. It works exceptionally well for me. It is all electronic, takes very little disk space, and keeps the clutter to a minimum, and eliminates most of the cross platform nightmares.
One word: IMAP. If you can read your email using any decent email client, it should support moving it to an IMAP server. If you are using web-based email or some crappy client which can't export emails to a standard/raw format, you'll have to write a script to convert the messages.
Ever since I first got acquainted with e-mail on my Apple IIe in the '80s, I've used e-mail programs that offer plain-text storage as at least an option. It's one of the most universal formats in existence, and can be read one way or another on computers both decades old and brand new. I encountered some weird proprietary clients in the '90s that still stored e-mail in this format, because from a corporate perspective, this stuff was still in its infancy, plus HTML hadn't yet mucked everything up. To this day I still store in plain text from Eudora 6.2.
I burn it to CD-Rs that I know won't get moved around or scratched. They stand a good chance of lasting the rest of my life.
The coolest voice ever.
I log and keep all my traffic including IRC logs going back to '94.
My email isn't in quite the mess yours is, (I used Eudora for almost all of my emailing since I first got on the net, and have just imported from one version to another, and now into Thunderbird.)
I would simply start copying & pasting, or see if you can try importing into excel.
"The price good men pay for indifference to public affairs is to be ruled by evil men." ~Plato (427-347 BC)
I use an IMAP server
EmailMan has the answers to your problem.
More utilities than I want to bother with, but hopefully they'll have the converter(s) you need.
Good Luck!
Of course people archive IM. I archive everything, but in particular conversations with customers (I do contract programming.)
See if one of the desktop search products, such as Beagle or Google's will index will index your archives. That might be all that you need.
If not, the first step is to convert everything to a real format. Eudora and Thunderbird can read in some of the non-standard mail formats and convert them to unix mbox.
and store the files by correspondant. It's not fancy, but it's foolproof.
REM Old programmers don't die. They just GOSUB without RETURN.
setup a local imap server, you can generally setup imap from almost any modern mail client, let the client do the work of conversion.
then put all of your mail into imap folders, sort by year etc.. Each folder will then be in a large text based file. That you can rsync, tar , etc..
every email I get has a copy forwarded to a google gmail account. I also forward everything to an account on my server as well, but I like the gmail account because I can get at it anywhere.
If a job's not worth doing, it's not worth doing right.
This might be useful, if they don't collapse under /.
"It is a greater offense to steal men's labor, than their clothes"
Comment removed based on user account deletion
I give mine to Microsoft for the safe storage and instant retreval they are renoud for. Oh wait..
That's one of the many reasons why I have stayed with Pegasus Mailfor many years. Because they were created in the same program I know that I can still access my old mail files without problems.
What I do at year end is move all of that year's messages to a new folder and reset my filters so that the new year's messages go into a new set of folders.
Periodically I just copy off previous year's messages to CD.
At least few times I have been able to back a couple of years and find information that I lacked.
Three Squirrels
but ...
... just wondering as the Submitter did what i like /. Submitters to do: make me think and look for new, better stuff ... or better ways to do old-stuff.
:)
Along these lines, is there an OSS package that can read the varied formats the Submitter is referring to, tag and drop them in a DB with a nice, friendly, web-enabled (secure) front-end for searching?
My former employer kept *all* of his email from the last 20 years in tar.gz files. Let's just say it wasn't easy to find an email from er, 15 years ago very easily.
Is there a package that can read the mbox, the other box-formats, plain text, pull from pop, old tar.gz bundles, categorize (sorta), tag and make such things searchable?
Totally a shot in the dark here, i'm not a mail guy at all
It is the "drink" that makes me wonder, sorry
I only use e-mail clients that store mail in ascii with standard headers. This means no Outlook mail. I still use the Netscape e-mail client to view and organize my mail. Also I have various perl scripts that can access the e-mail archive. I have 22 years of e-mail, archived on my PC. It gets backed up with the nightly backup onto a swapable firewire drive. I swap the backup every morning and have one of the drives with me.
Almost every email client around can import and export mbox formats. Getting your email in a format that is going to be readable in 20 years is the first step, otherwise why bother?
Worse comes to worst mbox is readable as plain text.
Every week I do a backup of everything important on my SuSE box, the entire public dir on my live web server and two testing servers plus the MySQL databases on all four. My email and document files go back to about 1995 and still everything fits on a single CD. (I also burn another copy for off-site b/u.) So, I have multiple copies of everything that's more than a week or two old. As soon as I have more data than will fit on one CD, I'll move to an incremental system... and start to worry about CD rot.
Gmail?
I don't know about you but I generate about 6GB of email archives per year. Besides that having my email potentially available for searching doesn't sit well with me. I'm not sure where it stands now but there were a lot of potential privacy issues with Gmail.
No I don't receive hords of email, just a lot of engineering related with source code,research, white papers attached. If you do anything business related it's important to keep all of the original emails received so there is an electronic paper trail.
Closest thing to a helpful feature I've seen is the importing features of some programs. Unfortunately, that has mostly favored Microsoft. They have the most resources to devote to this, both in terms of making sure the competitors email can be imported into their system, and in terms of making sure that their OWN email can't be exported successfully to the other systems. (The last one is indirect these days, in terms of the file system. Yeah, you can export to one giant flat mess. At least that was the situation the last few times I tried to escape from Microsoft's clutches, but I'm really loathe to abandon my very complicated filing system.)
Farthest thing and least helpful was a shareware program that would eat the email when the trial period timed out. I don't remember the details except that I'll never voluntarily deal with blackmailers.
Freedom = (Meaningful - Coerced) Choice != (Speech | Beer^2), and sad sock puppets' bad mods avail them naught.
The post is clearly on topic. It's also a quote from a movie on philosphy. Stupid moderators modded it down.
I use Trillian, which logs all my IM conversations, and I can't tell you how many times I've looked up a URL from a year ago, or a friend's address, or some other bit of obscure information I might have passed on, but didn't think it was worth saving at the time.
~~~
Drupal themes from TopNotchThemes
I've been using emacs to read mail for 18 years.
:)
from rmail to vm to gnus, no conversion problems.
(what's an AOL mailbox?)
I second this.
I started running my own IMAP server on an old machine a year or so ago - and synced all my old mail archives to various folders.
My mailserver also solves another problem - multiple POP accounts. I have my IMAP server set up so that each one of my POP accounts gets automaticly tagged and sent to it's own folder.
A third common problem this solves is having multiple machines. Now my desktop's email client is always synced with my laptop's email client. Before I had run into problems when ever I traveled and fetched my email from the road.
I dub thee... Sir Phobos, Knight of Mars, Beater of Ass.
Don't ask Microsoft.
Democracy is two wolves and a sheep voting on lunch.
I use Adium on OS X for AIM, and I have it automatically set to log everything. Currently both my G5 and Pismo log to boot drives, but I back the logs up on an external hard disk on a regular basis.
It basically acts like an automatic journal/diary, as I tend to talk about most significant, or even sometimes insignificant events with friends and family.
Store everything in mbox format (that is, raw text with headers and all).
:-)
Every email client worth the name understands that, for the good reason that it's the format they receive emails in
For nonstandard forms of archives (perhaps old AOL clients and whatnot), you're probably left either (1) perl'ing a convert script or, if you can (2) fire up the old client in Win95 in VMWare or something and fwd the mails to yourself (tedious).
"A door is what a dog is perpetually on the wrong side of" - Ogden Nash
All of the links in my daily pr0n-in-your-mail messages go bad if I dont check them every day so they're useless after being read :)
I delete all mail that has not been directly addessed to me. Usually all mail from mailing lists, unless the message is really interesting or it is a thread I've been participating. I didn't use to do that, but when I changed to this method I deleted the old unimportant messages as well. It brings down the number of messages to a manageable level.
Messages are not sorted into separate subject folders. They are all in a single mailbox, the mailbox. Every month I back up this mailbox to the name of the previous month. I do the same with my sent mail. Messages are then kept in individually monthed mbox files, independent of the email program I am using. All mail programs to date I've used are able to import messages. I keep in the mail program only the mailboxes of the last few months.
If I need to find a message, I first search through the month or, if I am not sure, the year that the message was received. I "grep -i" in a directory with all my mboxes. It usually doesn't take too long, just a few minutes in the worst case. After I find the mailbox, I import it in my current mail program (Mail.app now), to forward, save attachments, etc.
I've tried keeping all these mboxes as a Cyrus imapd spool, but the trouble was larger than the benefit.
I keep all (personal, things I've been directly involved with) my mail since 1996. It works for me. It is around 300MB compressed with gzip.
I've been using mbox format since the eighties, and never had a problem with it on any platform. It's pretty much been THE standard for email for as long as email has existed. If I ever were to switch, I'd probably switch to maildir, which has nearly as wide-spread support these days.
I never keep emails, or archive IMs or any other form of communication. Once a email is read, it is deleted. Same goes for normal old-skool mail, I read it and then trash it. The only exceptions are of letters/email of some importance such as information I need to keep handy, or if it has some kind of sentimental value (letters from deceased relatives for example.)
Sure, HDD space is cheap; but I tend to equate people who archive every single form of written communication to those who have an Obsessive Compulsive Disorder, in that they hoarde everything in sight: newspapers, snail mail, magazines, boxes, etc..
Commit to memory and destroy the evidence. Thats my way of handling archives.
IMAP isn't really a word, its an anocrym. But I agree, IMAP is the way I use, it helps for the relevant email and on my network I use both Linux and Windows (with a dedicated Linux box). I have Evolution set up to continually check and sort my email into IMAP folders and I generally read them off my linux box. If I need to click on links, I generally open up any Windows email client (from Thunderbird to Outlook) and it'll connect to IMAP and my emails will all appear (nicely sorted too!). If I need webmail, I have squirrelmail (which I use) to access my IMAP system remotely using any web browser and I can get at my hotmail email (from the old days, but my accounts are still active) using freepops or some other Web Email to POP3 gateway. Everything (but gmail, my mailing list archive), is in my IMAP server - I just backup one area.
I always wondered where this setting was...
So how do I use evolution and a browser view without keeping double copies?
I do security
Convert everything into mbox format. formail will help you with that.
Use mairix to search through email.
mutt is the best mail client ever.
-rsw
I archive IM convos to blackmail people......I mean...
Combine this with spotlight/tiger in mac os. Spotlight indexes PDF content. print it to pdf and it will be searchable. Assuming you have a Mac that is.
Some drink at the fountain of knowledge. Others just gargle.
This is simple. Just click "Send Error Report" and the wizards at Microsoft will be more than happy to help!
At work we're forced to use an internal MS Windows Messaging for internal IM. It's not a bad program, but it has no archiving functionality. I've found that Messenger Plus! (http://www.msgplus.net/) is a great compatible add-on that add's message logging and other features to Windows Messenger.
However BEWARE, Messenger Plus will want to install spyware by default. Be sure to select the, "I don't want to support this company, don't install spyware". hehe Funny but the option is something like that.
Why would you do that? Do you record all your phone conversations? IM is just like that, unless it's important (rarely happens) you'll never get to look at it even if it was archived anyway. We just feel the need to archive every piece of information as geeks as long as it's electronic and hence possible.
The following statement is true
The preceding statement is false
I archive my mail on /dev/null. Send it there daily.
Whatever you do, I think its best to keep it in an open and obvious format like mbox or maildir. The nice thing about maildir though, is that since all the messages are seperate, it might be a little easier to write a program to put them into a new format.
.saved-messages-YYYY-MM and also to my inbox. I simply don't touch the saved-messages folders and when I am done with the message in my inbox, I just delete it. This has worked well for me and makes it much easier to deal with archiving old mail. In the end, having categorized folders and such is just a waste of time. Its kinda like the wm2 (window manager) way of thinking, but for mailboxes.
Personally, since 1999, I've been using a combination of maildir and procmail to archive and save my mail. Every message that comes in, goes to a folder called
I keep logs of all messages sent through DoorManBot, which allows offline messaging on AIM, but I do that through special software only for the system.
For normal IM logging, I use Middle_Man, an AIM plugin. Other AIM plugins do this too, so take a look around.
Send offline messages on AIM with DoorManBot
One word: IMAP
...who knows what else. I've got freedom to try whatever I want at any given moment without losing my current or past mail.
Absolutely. I use no fewer than two mail clients on two different machines on any given business day. Every email I've sent since 1995 or something like that, and received since 1998 is available and searchable. Over this time, I've accessed this archive with the following clients:
* pine (lots of pine)
* mac mail
* thunderbird
* various netscapes/mozillas
* ML (some random IMAP reader)
* My phone (my old Sony/Ericcson speaks IMAP)
* My palm (two different apps)
* python
* a java webmail system I wrote
* three or four other webmail systems
* mutt
-- The world is watching America, and America is watching TV.
then it gets copied & pasted to a text file and saved, then deleted from the email client...
Politics is Treachery, Religion is Brainwashing
I have several CDs worth of stuff archived with ForKeeps:
http://www.fkeeps.com/whofor.htm
It's a bit of an old program and the interface is clunky, but it works reasonably well once you work through it.
--Steve
That's what I have been doing in recent years: keep your regular "recent" emails in one profile, and in another all your archive stuff, using the same client and formats.
...
When you switch mail clients (you allways do in a few years), make sure you import all current *and* archive email in a new set of profiles. Backup from your current app.
Still have to figure gmail in the equation, but with pop3 access should be just a matter of importing it in a app and backing it up - but downloading up to a GB over pop3 may be problematic, not sure what provisions are there to resume downloads.
just an idea
I only forwarded the important/funny stuff to my other email addresses when I switched addresses.
Now I use Gmail, and everything gets archived. I don't see Gmail/Google collapsing anytime soon, (or me running out of storage space) and so I think in 10 years, if you ask this question, I'd say I just archived it.
That way it won't be subject to a sub poena. You'll regret it one day if you don't. Do you realize how much incriminating stuff you have in there?
Doesn't it make you feel good to know that our freedoms are protected by politicans, lawyers and journalists.
Does anyone know what format and where outlook stores e-mails? I need to archive a friends e-mails and she only used outlook and I have no clue where to start with Windows 98 and outlook..
I like muppets.
If you know basic programming (file access and working with a given API), then you can try writing a plugin for Google Desktop Search. This way, you can add all your old files to the GDS database, and automatically archive the new ones.
I keep everything on my IMAP server. When the mailbox is full, I delete everything.
Ronaldo Faria Lima
E-mail:ronaldo@ronaldolima.eti.br
Home page: http://www.ronaldolima.eti.br
My car is bigger than yours. Move it or lose it!
Thanks, but I'll pass.
I've been looking for a solution to this same problem for a long time. For now, I still have all my email from the last 10 years stored in my MS Entourage database. Scary. What I would like to do is convert them all to HTML and/or plain text so that it's searchable, and won't need a special app to view them. Who knows what formats will still be around in 10, 20, 30+ years? But I suspect HTML and plain text will survive.
What I don't like about the mbox archiving technique is that you have to import the mbox into an email app to read it. To me, that sounds like a chore because I'd want to keep those messages separate from my current email box, and then there's processing time it takes to import it, etc.
It seems to me that an ideal situation would be that the file system can search it (i.e. Spotlight in Mac OS X 10.4 Tiger) or use a web browser.
There is a solution for archiving emails to HTML: MHonarc. I've played with it in the past and it does its job fairly well. What annoys me about the script is how difficult it is to customize the output, especially if you want it to validate and use CSS.
There's nothing else out there that I could find to my satisfaction. Maybe some resourceful developers want to embark on a new OSS project??? :)
Roger Wong
Graphic Designer
Oakland, CA
Moral of the story: Get a gmail account.
As for conversion, you can google for programs to convert your mail (or upload it to gmail) from pretty much any format into a Thunderbird compatible format.
It was also nice to get out of Outlook Express with it's 218 MB limit on folders. It was deleting email at random (presumably for years) without ever even mentioning it to me. Three cheers for Micros#%@!
10 Your mother told you to stop being such a pack rat.
9. Disks fill up, no matter how cheap they are. Low cost doesn't excuse gluttony.
8. Backups take forever.
7. Restores take an eternity, especially if your not confident.
6. Mail client gets slower and slower.
5. Searches take too long.
4. Mail clients make mistakes, especially on big stores. See #7
3. Your CYA evidence may be used against you.
2. A mail store is not a file system and SMTP is not a file transfer protocol.
And the number one reason to delete your old email...
1. IT'S ALL A BUNCH OF USELESS CRAP JUST AS IT WAS WHEN YOU FIRST RECEIVED IT!!
That's four words:
I use Eudora 5.1 - I have every e-mail since 1995 at my disposal. It works very well. Upgrading doesn't hose mailboxes; you can move the entire subdirectory to a new machine and it works perfectly. I can't say much for the newer versions of Eudora. Version 6.2.1 seems to have some bugs in it, but if you can get earlier releases of this software, it's excellent.
How about a quote from a movie on e-mail archives, next time?
6GB yearly? Holy shit...
Do you actually sign up to those free porn places?
I use grepmail to find old emails that I might need. Grepmail lets you use perl regular expressions to find messages and then outputs the entire message where a match was found. You can use grepm to open grepmail matches as a mailbox in mutt. grepine does the same for Pine, which I use.
At the end of each year I clean the spam out of my archives using a procmail recipe and spamassassin. This recipe marks messages as deleted in the mailbox. I open these in pine, sort by deleted, and double check them. Once I'm sure they're all spam, I delete them:
The special spamassassin config turns off bayesian filtering and sets the threshold high:
The rest of the spam I clean out by hand.Slashdot: Failed Car Analogies. Amateur Lawyering. Anecdote Battles.
and just store the whole thing on cd/dvd/tape..
....
...
... and hey, i even have stuff from the bbs times with quickmail, or bluemail .... or whatever it was called :)
:)
maybe run a nice script to have an index in a db on the sender/title
or standard mailbox, and then make a bootable minilinux with pine (mozilla/whatever) on the same media (use vmware or similar to access it from non unix systems
actually my mailbox is just a growing junk collection, and have to do something with that too
or just export the mail addresses and delete the whole thing
Well, if you're going to be on this topic, a mention of ZOË is pretty much required.
ZOË is a sort of an archiving proxy that sits between your mail client and your mail server. It stores and indexes everything, so you can pop open a browser window and do a search on anything you've ever sent or received. Naturally, this was created before gmail.
With ZOË you don't need to worry about those pesky email folders and waiting for long searches.
Naturally, spam filtering before ZOË is a good idea.
Signatures are a waste of bandwi (buffering...)
I still haven't finished setting up my own system yet but I think dbmail is a good way to go
it allows mail to be stored on a SQL database
I've kept all my mail for the last 10 years or so using stand unix mailbox files. Pine moves each read and sent folder into individual monthly folders which are very easy to grep through. In fact to use "modern" email programs I setup a cron script to do the same thing that pine used to do as well as move mail more than a year old into an "old" archive subdirectory. In addition I have procmail eat the binaries into an attachments folder. Its simple to setup all the directories for "viewing" via imap though to be honest I still just ssh in and use pine. As far as old "aol", "gmail" account etc email them to a proper account w/ a tag in the subject header or something and let procmail file them away for you in one of your folders.
-bloo
I do data mining research, most recently on the Enron email dataset, and I've actually been having to roll my own multi-mailbox storage, access, and retrieval systems. It's taking way more time than I'd like, at this point I've gotten a database and web-based viewers made up (beware, they're quite slow).
If anyone has an idea of an open-source application similar to what the submitter is looking for, it would help my research quite a bit. There's practical research applications in this stuff, if someone's interested in making it.
Just about every email program that I've used has managed to export to CSV. A few web-based email systems didn't allow such imports and some hunting on the web found some sort of convertor (like YahooPOPS!, etc.) that converted to POP and then I exported them to CSV using Eudora or Outlook, or whatever program I was particularly enamored with. .PST files that are archived on CDs and on an external hard drive. .PST in case I needed them for anything later on. ...), since I can usually remember roughly when I got an email that I was looking for - alternatives include perhaps by name of sender or company. :)
Admittedly, sometimes the column names didn't match up ("Sender" v "From"), etc., but for the most part that how I did it. I also made an effort to keep the number of email accounts that I had to a minimum. At this point of time, most everything is stored in the form of
I also made an effort to keep my email accounts to a minimum, which probably made this entire process significantly easier and when I did close an account (like when I finished work at a company), I exported the emails from there and kept them in
As far as indexing works - I have them stored in 6 month segments (Jan97-Jun97, Jul97-Dec97,
I do archive IMs - Trillian worries about it for me.
Hope this helps.
The best archiving option is to post all your emails to usenet. You bet it's archived there for all eternity!
Spotlight - as any other similar technology - also indexes your *e-mail* in the first place. There is no need to transform your mbox files to pdfs, mp3 metatags, iCal entries, or whatever Spotlight may index as well...
My next comment will be ready soon, but moderators can beat the rush and mod it up early.
Since we are not alone facing this problem I suggest we join as a task force to setup a bunch of virtual machine with the proper tools installed to read those archives.
Some people reading us may have the required software.
We can get a vwmare trial licence and setup a Linux drive and a Win 3.11 drive (guest drives are host files, philes are easy 2 share and plug somewhere else, got it?).
I can't tell everything right away here, but we need a common keyword to find each other safely: let say MAIRCHIVEL (0 at google today).
hum... BTW does anybody remember this mail client running with the scheduler on windows 3.11 ? it would be nice to see it with a brand new name ;-)
(if you find me too cryptic you may be not enough interested by a solution, perhaps some other nerds here will explain more zan me)
I'll start you off.
bash
zcat
grep
|
>
Now lets see what you can do.
This is only practicle if you use something like Gaim that automatically saves conversations for multiple IM services. Every few months I'll copy my old IM logs to a backup location, usually when I upgrade the client software in case they change the format of the file or what not.
It's actually pretty convienient to be able to search through old conversations. It makes a better journal than trying to thoughts out to yourself.
are targets...
I've found the easiest way to handle EMail when it's in multiple formats like that is to just print everything out and store it in boxes in my garage.
I am NOT a man!
I am a free number!
If it's worth printing out: it's worth saving. If it's not, I don't save it. Paper: one of the cheapest data storage comodities available.
Can I be a Luddite too?
one year of e-mails. I keep all important e-mails forever until I don't need them.
Ant(Dude) @ Quality Foraged Links (AQFL.net) & The Ant Farm (antfarm.ma.cx / antfarm.home.dhs.org).
I use IMAP and mailbox/maildir.
IMAP allows synchronization between many different servers and clients.
Mailbox and maildir both are open formats. Maildir uses one file per message, which makes management a little easier but some less advanced file systems have trouble with so many files in a directory. Mailbox is a reliable standby. Either way, I keep annual archives.
I think it's inappropriate to keep permanent records of IM conversations. IM is more like a face-to-face talk and people don't stick a tape recorder into each other's faces when they talk either.
Can someone step me through getting an IMAP server running on my network, and moving existing mails to it?
Assume I have Linux or Windows boxes available (Would prefer Windows, yeah blah blah but other people don't like me 'wasting' hardware on *nix), and I use Outlook for my primary mail client.
How many people can read hex if only you and dead people can read hex?
What I do is have an IMAP server running on a Unix machine, and archive my e-mail into IMAP folders. The server turns those into vanilla Unix mailboxes, so I can deal with the messages as plain text if I want to. Almost every e-mail client out there supports IMAP, so compatibility isn't much of a problem. It also lets me use different clients and different systems to access all my e-mail in one place.
I use this.. it is fully cross platform too (I'm not affliated at all) and supports the various quirks in different versions of Eudora too.
Linkey
Online backup with Mozy, sounds like Ozzie, but more!
I keep the emails in mailbox format (that is, in plain text as it is stored in most UNIX systems), in several files. The reason I do that is that most email readers (MUA) can read mailbox format. I keep them in several files to make it more manageable.
The tools that I use to manipulate emails are mostly "from", "procmail", "grep", and "less". There used to be tools from the "elm" era (still remember them?), such as "frm" (which is better than "from"), "reademail" (to read individual email, given the number of email in the archive), "deletemail" (which can delete an individual email in the archive). Too bad, these tools are gone. At one point I slapped a simple Tk interface as a front end to those tools. But it didn't scale well.
At one point I did experiment to store emails in indiviual files. But the tools to manipulate them are limitted. I used MH.
The next experiment I did was to take all those email headers and put them in a database. (I used msql, which was popular at that time.) Then, I had a Java applet and perl script to make queries to the database (and actually did an analysis of my reading habit). The actual emails were stored as plain text files. Each email was stored in individual file. Basically, the original email was untouched. I got bored and never continue the project.
Now ... I am stll searching for the perfect email tools.
I've always found IMAP fantastic (properly configured, thousands of emails searchable instantly, from any email client anywhere in the world (if you use imaps or ssh tunnels), only downside is you need a machine to run it on, though if youve got more than one PC in the house, it's not a bad idea to put a 3rd one in to run a few network services, like email and file sharing.
:)
I'd recomend for those without a limited budget to get a low-power machine, onboard video, slowest 90nm processor you can find, one hardisc with a backup medium eg cd/dvd rewriter, or two hardiscs for a raid etc, for those on a limited budget, any PC your friends or relatives want to get rid of
I've been looking to do the same although my email archive isn't nearly as massive but I would still like to consolidate it all. One option I've been considering is GMail Loader. All it does is forward your mailboxes from a variety of clients to a Gmail account and you can read and search it from there. At this point all the Gmail privacy concern post should begin...
(This sig intentionally left blank.)
I strongly recommend Outport. It does an extremely good job of converting MSFT Outlook attachments into something more readable (mbox I think, it has been a while). MS Outlook usually mangles attachments into some wrapper called TNEF.
Also, anyone know of a client program that will recursively create folders on an IMAP server (maybe a server issue. In which case, what server?)
I had gotten over translating my years of Outlook email into something more universally readable, but I have so many nested folders that the inability to have the client recirsively create IMAP folders is an issue. Suggestions?
Does anyone archive IM?
/. ?
AOL does, don't you read
I keep all of my IM logs. Gaim logging has been on ever since I started using it. Unfortunately, the changed formats at one time, and I think I lost some. It currently dates back to late 2003. I have all my email since 2002 in Evolution, I only delete the spam and mailing lists... well some lists. Back in the days of Windows, I had an unfortunate battle with format, and format won. Then I saw a penguin in the far distance, and I knew my destiny.
Kernel Krunch - Part of a Complete OS
Outlook pst file. about 4GB's now. Converted to outlook 2003 so PST's can grow larger than 2GB. Save every email.
1. Print out emails.
Man, it's so nice not being burdened by the embarrasing history of all the emails I sent. Besides, that was a long time ago, way before Score: 5, Funny.2. Shred them.
#!/bin/sh
tar cvzf mail-$DATE.tar.gz ~/Mail ; gpg -e $self mail-$DATE.tar.gz && cp mail-$DATE.tar.gz ~/archive/.
So IMAP sounds interesting, but how can i cheaply install it on my already present Windows server?
Hivemind harvest in progress..
Ummm...
urpmi imapd (or apt-get imapd, or whatever)
man imapd
Or you might want to actually say what system you want to install to, in what context, what mail clients you use, or you could go read the mail HOWTO (if using Unix/Linux), or use Google, or the Microsoft knowledge base (or whatever it's called) or whichever of a hundred ressources which might be suited to your problem instead of asking a very vague question on a non technical forum.
May contain traces of nut.
Made from the freshest electrons.
/dev/null of course.
Have a look at bincimap. It works well, installs easily and seems to be quite secure.
See http://www.bincimap.org/ for more details.
It runs on my small linux server without problems and I can access my emails securely over ssl from anywhere. The only limit is the hd size, so even a couple of GB should be no problem.
Sure. From my experience the best IMAP server for Windows is MailEnable. Unfortunatly for IMAP support you'll need the Pro version which isn't free, but you can try it for 60 days, and my windows installations rarely last that long ;)
On the client side, it's easy - just add your server to Outlook as an IMAP server, and that's it. You'll probably find that IMAP support in Outlook sucks (it's a lot better in Outlook Express), but it's still OK.
Amiga floppies. All my email from 1986 to 1990 fit on 3 Amiga floppies downloaded from a Xenix system. They are of course unreadable... although, it jsut occured to me I have a 10 meg Amiga scsi A1000 drive in the barn that has it all. Hmmm...
Need Mercedes parts ?
I save the important one which are about 10 percent of them. The rest of it gets deleted with impunity. Anything over 5 years old that i still need I still keep a hold of, but the oldest message in my mailbox is 3 years old. You will find out if you actually go through your box, about 90 percent of them you don't really need.
Gorkman
I have an archive going back to 98 when i got my first computer and I have always moved to the newest technology and imported my mail at that time, so even though I have used Outlook Express, Entourage, Netscape, and now Apple's Mail I keep all my mail in the .mbox format inside Mail, I get rid of my old archives and just backup my current mailbox format. I also started saving all my IM sessions. I use Mac OS X and the new Spotlight search feature is what influenced me to start saving those. Being able to search through loads of emails and IM sessions seems like a nice thing to have and storage is cheap, I don't mind saving many many pages of what is mostly text. It didn't seem like it would even make an impact of any kind. Hundreds of typical chats are only about 1 MB.
...and let mutt sort out.
:0 c:
I had multiple folders, sorted by people/project. I got in a complete mess and finally snapped when I spent half an hour looking for a simple message.
Use procmail to write all incoming messages to 'all-mail-YYYY-MM' and use Mutt hooks to write out to the same directory.
At the end of the year, cat them together and make 'all-mail-YYYY'. Accessing and reading this mailbox can be done with 'mutt -R -f all-mail-YYYY' as this opens read-only. Use 'l' to do 'limit' searches and use ~t, ~f, and ~b in AND combinations to limit on To: From: and body of messages. It's lovely only having to look in one place!
Procmail:
INCOMING=all-mail-`date +%Y-%m`
# now I want to keep a copy of EVERYTHING in a dated directory
$INCOMING
Muttrc:
set record="+all-mail-`date +%Y-%m`"
Works for me!
Dr Fish
Perhaps Mailbag Assistant might do the trick.
I just collect stuff, I don't have any motivation to reconcile it!
Seriously, it's convenient that Evolution saves in the same format as Mutt. A prudent number of folders to file by and I'm thinking annually I dump to CD. Searching? Grep, of course.
I used to use Mozilla, which successfully imported my Eudora mail. However, I now use Thunderbird but have not been successfull in automatically importing Mozilla mail into Thunderbird. I have manually transfered a few emails but not managed to do any kind of automatic import.
My father was concerned about the longevity of his e-mail a few years ago, so I created a small batch file that converts his Outlook Express mail archive into mbox on a monthly basis. Last month he asked if I could convert them "into a web site" so he can get an idea of a thread history without parsing a huge file. When I get a moment I'm planning to write a script that outputs each message to a new file in html tags and use the message subject and date to create a rudimentary index.html.
I'm surprised no one has tried this before. It's a good low-tech solution for people who require information in a hurry and is more immediate than a flat file.
ZOE: http://zoe.nu/ is a nice mail archiver/indexer/server. It has a nice web user interface, it is written in java and it is easy to install.
If it only handled IMAP...
IMAP is the answer. I don't use IMAP on a regular basis, but it did let me export mail from outlook over to Evolution on linux.
I used the UW IMAP server, which is a little easier to set up than the Cygnus one.
The UW IMAPd keeps its folders in mbox format, so it's a great tool for converting oddly formatted mail.
Moving email is pretty easy -- it's harder to move calendar entries, address books, notes, and the other sorts of data that ends up in a program like outlook. I think the easiest way to do it would be to sync to a palm device, on windows, and then do it again under linux, although I haven't actually tried that.
Mailbag Assistant from fookes software will take care of all your concerns. Great piece of software. Sorry, Windows only.
Print out all the emails. Photocopy all the printouts. Laminate the originals. Lease fire/water proof storage space & store it there. Keep the copies under your bed.
If you have saved them since the 80's, I am sure
they warrant such special care.
use to archive email across multiple platforms and non-standard mailbox formats?
A printer?
What?
Good choice, mine too. I copy everything into IMAP, just works and works.
On the UNIX host, I periodically burn them to CD/DVD.
The best part is if you delete something from the spinning plater and you need it all you have to do is copy it back in from the CD.
Just like you install any other piece of software, with a Windows based IMAP Server install package.
"I use a Mac because I'm just better than you are."
I have personal email going back 10 years or so. I'm also on a couple of high volume email lists and get ~500 emails/day.
I keep everything in mbox format. I archive high volume mboxes with archivemail, so everything older than 90 days gets gzipped. Procmail sorts all my mail, spammassassin strips out the garbage, etc.
I use mutt as my email client. It's as powerful as any other program I've used, and because it's text mode I can ssh into my home machine and check my mail from anywhere.
I suppose this is somewhat of a stoneage type setup, but I've been using it for years and I've never seen a reason to upgrade.
As a bonus, you can tell which emails are worth reading by how they get moderated. All your work related emails will probably be modded Troll, except for your performance review, which will be modded +5 Funny. Email from your illicit lovers will be modded Insightful, since that type of thing is new to most of us. Email from your family will be conveniently modded down so you will not have to deal with it. Your friends won't need to send you any email at all, since they are probably already on Slashdot, and therefore, know enough to post in your threads.
Problem solved. Ah, Slashdot... Is there anything it can't do?
Show me on the doll where his noodly appendage touched you.
Maybe this hasn't been mentioned because there is no email client supporting an SQL database already. I think that Thunderbird should show the way and offer the possibility to store emails as an SQL database.
That searches give instantaneous results would make a part of my job way easier. It seems to me that many email programs fail at that. There is Spotlight from Apple, but it's a feature of the OS. I like the way iTunes lets me navigate through MP3/AAC files. The whole interface of iTunes is built around the idea that you're going to look for a tune. Much more often than search a tune, what I find important is to quickly find what my friends and co-workers told me, and what I told them.
There are many big limitations in email programs, but they often are subtle in that they leave the responsability to the user. For example, Thunderbird with the default options puts the cursor at the end of the text when replying to an email. So I often receive an email response starting with a whole page of the very message I wrote! Only at the end do I read: "Agreed. John." Some might say it's the fault of whoever wrote the response; I think the email program is flawed.
Another example is the discouraging difficulty of create filtering rules. I don't know many people who have filters programmed. I think that 1% of email users is a realistic figure.
A good email program should make it easy to write good emails, not just tons of emails. All email programs are archaic. As, I said, they make searches a hassle.
A possible explanation for the very little sophistication of email programs is that typically, people that design software (1) are not the best communicators, (2) are not very good at improving their own tools.
For those interesting in the design of Social Software, Joel Spolsky wrote an interesting article on his blog. Recommended reading.
I print it out, seal it in an envelope and mail it to myself. Then, in 20 years, I can pretend to get new mail and open it as a suprise!
Not to rain on the parade and all, but I wouldn't like to have all the e-mail I've received for the last 10 years available on an online server. While it sounds like a decent enough transition strategy for modern clients (I don't know about AOL 1.0, though), I would strongly recommend still backing things up manually, and taking stuff off the IMAP server periodically. 'course, I just download all my e-mail to my machine and archive it in various folders. Haven't had to switch e-mail clients in like ever, so.
Why would you? Old email can only be used against you. Trash everything and burn your hard drives every few years.
This
If you are interested in backing up message history look no further than Trillian (www.trillian.cc). The history of every IM is backed up in very easy to access and searchable (through the program) txt files.
Way to go. Now it's five words.
Jerk.
Information wants to be free.
Entertainment wants to be paid.
You just want to be cheap.
There are some mildly irritating things about this message format, e.g. if I've got a back up version of ~/Mail/BOZOTECH, I can't just restore it to the directory, because the numeric file names are arbitrary and reused often (e.g. an old version of "~/Mail/BOZOTECH/1666" would get backed up on top of a new one).
But then, as is often pointed out, with MH format, small errors will only destory individual messages instead of big stashes in a single mbox...
And this format is simple enough that file system tools like find/grep work very well with it. You can write your own custom tools by sticking together mh commands in shell scripts (I still refile mail that way rather than using procmail... I like being able to clean-up a folder full of mail after it's arrived, rather than having to do it all when the mail is coming in).
One of these days I'm going to experiment with indexing it all with swish-e, maybe play with slyphweed claws as a front-end, see if there's a way I can switch to maildir format, and so on...
I realize that this answer is completely unhelpful, but then the given question was a little silly ("how do *you* do this?").
Take an old machine too slow to run M$ warez on and install SME/E-Smith on it. http://www.linuxiso.org/distro.php?distro=13 This is a free linux server distro designed for small groups, it has a full mail server with pop and imap support. It builds the entire system and applications from one CD. It has a wonderful "backup to desktop" feature and runs for years without a reboot.
One of the reasons SME runs nicely on old hardware is that it does nor run a GUI but is administered throug its own web server.
I went to mutt.org.
First thing on the page:
Latest News
Note: The mutt-users list recently suffered catastrophic failure. Please resubscribe.
Hmmmmm.
Insert witty sig here.
I have kept virtually all the email I have ever received in the last 25 years. They are almost all stored in unix mbox format. I also keep logs of all IM conversations in gaim or gabber formats.
No email application that uses non-standard formats has been allowed to touch my email for about the last 20 years. Before that, I saved mail in plain text format, writing utilities to convert proprietary formats (such as novel and corvus) as needed. Online mail systems such as compuserve or MCI mail were downloaded in some flavor of ASCII. Mail much older than ten years ago is disorganized and on floppies. I think I lost some mail about 10 years ago when a backup across the net failed. Mail since then is saved in subdirectories of my mail folder. When I upgrade computers, the old mail tree is brought over as is or relegated to a subdirectory.
mbox and maildirs have been the defacto standard for the last three decades and virtually all decent mail software uses some variation of those. The mbox file format has been used on unix systems since 1975 and was adopted on other operating systems as those gained internet access. Both formats are basically the mail message in industry standard RFC-822 format. In mbox, the messages are concatenated separated by ">From" lines and (since 1995) any line starting with "From", ">From", ">>>From" is changed to ">>From" has a ">" prepended so you can accurately revert to the original. In maildir, each message is stored in a separate numbered file in a subdirectory. There are some variations on mbox. One is to add a bunch of nulls before the ">from" line. Another is to add a Content-length: header. Some programs add a dummy message at the begining of the mailbox to store application specific data. Programs may add their own proprietary indexes in separate files. Many programs use standard mbox format including: PC-NFS, unix mail, elm, pine, pc-pine, netscape, mozilla, thunderbird, pegasus, Mac OS X mail, and many others. The primary problems with mbox format are that it is slow to delete a message in the middle of a large file and that some programs have not handled lines beginning with ">From" properly. Each folder is stored in a separate mbox file or maildir. Mime digests are somewhat similar to mbox format. However, they have an extra RFC-822 header at the begining and use a separator line defined in the file header which normally starts with some number of dashes instead of ">From" lines.
If you have any programs that use non-standard formats, the most sensible thing is to ditch them and if you are prevented from doing that by a pointy-haired boss then install a gateway or find or write a conversion utility.
There are tools out there such as imap-utils that can save pop3 or imap mail to mbox format (many mail clients can also do this), convert maildirs to mbox format, etc. There are tools availible to convert proprietary formats such as microsoft outbreak, outbreak express, and eudora into mbox files.
Web mail services can be problematic. Choose one that gives you adequate disk space (so you don't need to delete before backing up) and allows you to retreive your mailboxes via IMAP or allows mailbox downloads in a sensible format.
I keep my email for a month.......after that I dump it. I usually receive 600+ a month (no spam) and send 200, so one months worth is all I keep.
The late 80's? Hell you might as well print it out, scan it and save the pictures to DVD. Or, just get medicated to relieve yourself of these obsessions.
Requirements:
- an email store that I can easily add messages to, and that will handle duplicates gracefully. Like you, I have lots of different archives floating around, mostly from all the different email programs I've used over the years. I want to be able to throw all these into the mail store, and adding the same mailbox twice shouldn't screw things up (message-id can be used as a unique identifier).
- an easy and seamless way for new mail to go into the archive. Once I create this archive, I want all my mail to go into it, without having to think about it.
- I need to be able to reply to recently received email, using whatever client I want. This is somewhat at odds with the previous requirement, because once your mail is moved from your mail reader's world into the archive, it might not be easily accessible from your mail reader any more (depending on your approach).
- Fast, indexed searching, from any web-accessible computer. If I'm at a friends house, and want to see a picture or something that I know I have in my email archive, I want to be able to pop into a web browser and instantly have my answer.
- Fast, convenient, thread-capable browsing.
The most promising solution I've come across so far is OSS project Lurker. It's an archival program primarily designed for mailing lists, but also works pretty well for personal email. You add messages to its data store in mbox format from the command-line, and it indexes them and gives you very fast browse/search on the web. The software scales extremely well, and its web interface is quite snappy.So what are its shortcomings? I don't believe it handles duplicates. I can't reply to email in its data store, because it's all web-based (there might be a way to create a link that will open up your mail reader and set all the appropriate headers for replying, but this didn't exist last I checked). There's no way to delete a message from the archive once it's there. There's no IMAP interface, and you can't open its mbox in a mailreader because it depends on knowing file offsets for each message. And there isn't a really seamless way to put messages into it from your mail client, though one good suggestion I heard was to have a special mailbox that a cron would pull messages from.
Why not just IMAP? A couple of problems with that. Again, it doesn't handle duplicates. I haven't found a web client that will provide fast search of an IMAP mailbox (by caching and indexing headers). And I've never really found a web mail client that I like much.
Currently I do use IMAP, but it's just a little less than optimal. I keep hoping something better will come along (or that one day, writing email archiving software will be the most exciting thing I can imagine doing
Most people said that they simply delete email that they no longer need. But what about your children/grand children/great-grand children etc? They won't have any letters from the past to read and see what thier parents were like.
Just something I came across:
:) has a database format, copies are cheap and, presumably, the same message can exist in a number of places under INBOX without adding any extra storage requirements. I guess procmail or sieve coud do this.
Procmail stuff - lots of links http://www.ii.com/internet/robots/procmail/qs/
Anyway - I once came across a recipe to use Cyrus to copy every incoming email into folders indexed under
-date received
-FROM: domain
-size?
-other things?
Because Cyrus (it's strange dealing with software I share a name with
Anybody know where this page went (or has come up with a recipe for something similar)? I think it's a wonderful idea.
You could look under "1995, April" or under "from mit.edu" and find the message you're looking for. Zoe I guess would do just as well but I thought this was a neat idea when I saw it a few years ago?
Is anybody using Sieve out there?
Here's a page I found on Cyrus IMAP on FreeBSD.
I have a machine that runs a dedicated IMAP server with one account on it (mine) which has my 2GB+ of e-mail since 1996. (minus the spam of course). That way I can easily switch between different clients and not have to worry about converting my e-mail all the time.
You have to print it with something. Ink: one of the most expensive ways to put stuff on paper. Heck, they say it costs seven times more than champagne per drop! That, plus the costs of cartridges and printer maintenance and, and... oh the horror! ;)
Me? I obsessively reinstall my operating system and reimport old mailboxes into my mail client, so I have a dozen copies of 5-year old email, ten copies of 4-year old email, 8 copies of 3-year old email, etc. No need for backups... plus when I search my computer for old email, I get a dozen copies of what I'm looking for!
And that just made my list of top 100 favorite oxymorons.
Omnes stulti sunt.
I've been considering an experiment: create a new domain name, and email address, and retaining every email sent and received during a ten year period. Mainly, the point would be to track usage and spam reception across a ten year period.
Of course, this will use some disk space but email compresses pretty well. The only thing I can't decide is if mbox or maildir would be a better choice.
Just kidding - I am certain AOL 1 doesn't support IMAP. I feel safe saying AOL 2, 3 etc also don't support IMAP. Since the original poster specifically mentions getting mail from AOL 1 mailboxes, I think there's a step missing in the IMAP solution. Remember, AOL (any version) is not a decent email client. I suppose that if it's possible to take mail from an AOL mailbox back up to the server then perhaps a POP client could DL it and then upload it back to IMAP...
It would be helpful if the original poster had mentioned what platform he is using right now.
In any case, there is a Java shareware program called Emailchemy that might help you considerably with converting email formats, although it doesn't seem to do anything with AOL mailboxes. Check it out here
One assumes, perhaps wrongly, that pdfs are a more durable format than mail. This of course is what the entire "ask slashdot" question was about. How do you deal with past mail in different mail programs. If you keep it in pdf format then it probably will be readable regardless of the mail program that generated it. However then the problem is wading through 10,000 old e-mail pdfs. Spotlight solves this. Now that spotlight exists one assumes no operating systme in the future will ever be without something like spotlight.
Some drink at the fountain of knowledge. Others just gargle.
If you have a Mac, save it all as plain text files and just throw it into Boswell. It doesn't just archive it, it cross references it for you.
Can take care of all of the text in your life.
Check out www.boswell.com.
You mean, besides the NSA?
I've used Mozilla for mail for the past several years, and I've had no problem trasnferring my POP folders to a new machine. I don't get all that much mail either, so the storage space wouldn't even be an issue, except...
Attachments that I no longer care about take up I'm sure the majority of the storage. I'd like to be able to delete the attachments while keeping the messages. Does anyone know of any tools that can go through mail in POP3 format and allow selective deleting of attachments. Ideally I'd like it to be a separate tool from an actual email program, but if there's an email program that can do this (and runs on Linux) I might consider setting it up on my existing folders just to allow deleting of attachments on occasion, while uisng Mozilla to actually download, send and read email.
The ultimate plays for Madden 2006
I been using Netscape and Mozilla for email for 10 years. I don't have the problem you ask about because I stick to a standard format.
Does anyone archive IM?
Only when it's really, really good cybersex.
...I should boot up my VAX and pull off the years of DEC email (sans spam TYVM) I have stored on there.
What's my Karma Mr. Burns? "Excellent"
You open all your email with an email client and move all the disparate inboxes into a big IMAP store on your own computer or one provided by a joint like Fastmail.fm or Runbox.com
Then, you keep a local backup on any computer that you move to with offlineimap, a wonderful utility that doubles as a multi-inbox syncronizer and backup utility. I have been using it for the past two years and can attest to its reliability.
EnjoyPragmatism as an ideology is not particularly pragmatic in the long term. Keep it in mind when you dismiss Free Software
I actually have two backups of my mail:
Mostly they were in Outlook PST formats. I backed them up to CDR disks. Then after reformatting my system, Outlook said they were not valid PST files. I think I did this from 1997 to 2005. I am slowing trying to move to Thunderbird so I can avoid this problem. Only Thunderbird does not sync like Outlook does to various devices and services. Also the Thunderbird Calendar is weak, and while Sunbird claims to interface with Thunderbird, I cannot see how it is supposed to do that as it does not interface for me.
:(
Prior to 1997, I used old versions of Eudora and Netscape, which I stored on Zip Disks, which now suffered from Bit-Rot and lost the data and they are unrecoverable.
I do not think I did anything wrong, but if I did, they would have to get the mail off of old servers if it is still there, as my copies have all gone up in smoke.
All I saved to text files, where some software registrations that I paid for. There are some software registrations that I paid for and did not back up that way.
Anyway Microsoft Outlook PST files are not the best way to save or archive email into. I even tried recovery programs, and all they were able to recover was the trash directory which was empty, and nothing else. Fbog!
Remember, Slashdot does not have a -1 disagree moderation, and no, troll, flamebait, and overrated are not substitutes.
Once every year I spend a day or two separating attachments, and filing them into my virtual filing cabinet by topic and date, And then I export the entire email into one big text file. That way it is easily searchable and is easily imported into any email program later on.
:)
I started doing this in 1996 starting with Pine (does anyone still use Pine?) at the University of Washington. All my text is still less than a CD. And my contact database has grown to more than 1,000 names.
Anal, aren't I? But it works, and I can find anything in my history. Just waiting for Spotlight
Does anyone archive IM?
Yes. I would tell you who they are, but then I would have to kill you. Ok, ok, I might just drop your priority instead. Consider yourself warned...
I've about 15G of emails now dating back to the early 90s, all stored in a locally-installed Cyrus IMAP server (maildir format, technically). Never used AOL's mail or free webmails so that was never a concern of mine.
I archive nothing.
I did in the beginning, but then later I realized I never ever needed to find anything older than a year.
You want to keep a particular email - forward it to your Gmail (or other) account. If Gmail (or whatever) goes bust one day, so what. It's unlikely you'll actually need any of that crap.
6gb? You should investigate compressing your inbound/outbound attachments. I'd imagine you could reduce their size substantially. If you were using Outlook (unlikely as it would choke at 2gb) I would suggest http://www.pstcompactor.com/ . Disclaimer: its my product. And I concur with your Gmail privacy concerns.
IMAP isn't really a word, its an anocrym.
acronym - n. A word formed from the initial letters of a name, such as WAC for Women's Army Corps, or by combining initial letters or parts of a series of words, such as radar for radio detecting and ranging.
Personally I just insert newer HD's every few years and keep the old ones, including OS and stuff.Need old email? Plug in old HD, reboot: viola! My own timemachine. Including cluttered desktop, silly backgroundschemes and once o so funny error-sounds ('I can't let you do that, Dave' anyone?)
All those moments will be lost in time, like tears in rain. Time to die.
All my messages are belong to Gmail.
When you look at the state of the world, how can you not become a radical, liberal anarchist?
Outlook 2003 removes the 2gb barrier on pst files. :-)
Now if I could just get Apache 1.xx to write to log files larger than 2GB I'll be in heaven.
Darin
This is why the House of Lords was resistent to the prosecution of Nazi war criminals for so long, incidentally.
Wikileaks, no DNS
Does anyone archive IM?
I do, but that's because I use Trillian which makes logging a breeze, and Trillian's Activity History feature makes these archived logs actually useful, being able to "bookmark" points in the conversation as well as search the logs in several useful ways.
Dossy's Blog
I have never changed the basic underlying format - MH - Rand Mail Handler, one file per mail, in a directory, numbered.
I used to use mh, then nmh, then xmh. Nowadays I use mutt. Evolution will handle mh format, but it is slow.
Andy Rabagliati
beagle , lucene or swishe for *nix, MS-Land etc.
peterrenshaw ~ Another Scrappy Startup
I keep my mail in Unix mbox format. This is the format used natively by pine and many other Unix mail programs, but more importantly it's text-based and common enough that it'll never be indecipherable in the future.
Non-Unix mail programs often have the ability to export messages in a text format that is a bit like mbox but non-standard. I have written outlook_text_to_mbox to decode the text and CSV files written by MS Outlook.
-- Ed Avis ed@membled.com
Faced with the same problem, I found that a nagware program: MailNavigator (www.mailnavigator.com) was finally able to help me consolidate my mailboxes into one common format. That's all I'd recommend that one particulary tool for however. Apparently it does all sorts of other stuff, but from what I've seen, it doesn't do it well. Once your mailboxes are in one giant mailbox of some standard format, it becomes much easier to convert/grep using more readily available open-source tools.
This is how my mail infrastructure works:
:)
I have set op my own Imap server (bincimap) and use the combination of fetchmail + procmail to get all mail from my various pop accounts to that Imap server.
Now the Imap server is configured to use the Maildir format and the dir is located on an mirroring raid device (just in case of an hd crash).
Now for offsite backups I just create a tar.gz of the Maildir directory. If dirs get to big I can rotate them. Using thunderbird I can move mail from any imported format to the Imap server.
And If thunderbirds search function doesn't satisfy me (or I want to script something with it) Swish++ does an excellent job indexing and searching the mail
* ML (some random IMAP reader)
Gracious, I've not thought about that mail reader for many many years. It must be 10-15 years old easy. As I remember it was written by a guy who eventually stopped developing when he joined Netscape in their early days. Don't remember his name.
It was a GUI app based on Motif, and was the first MUA I ever saw that implemented the concept of "Virtual Mail Folders". I was completely blown away by it at the time.
And now, here I sit on my Mac all these years later and I'm waiting for Apple to give me that functionality back again with Smart Mail Folders in Tiger. Hmm
thanks for that, i was wondering why it looked wrong when i wrote it
I always wondered where this setting was...
I have a small script that exports all my mail from Outlook (minus attachments) to a FileMaker database. This DB holds nearly all my written communication (outgoing E-mail, faxes, letters, incoming E-Mail) plus notes about important meetings and phone calls (I could scan imcoming letters and faxes, too, but I am too lazy to bother with OCR) from the last eight years and weighs around 100 MB. I can live with that.
This way, my Outlook PST file stays small (just a week's worth of mail plus contact and calendar data), so I can easily sync it with my phone.
When I open the FileMaker DB (trusty old 3.0, BTW), it prompts me to enter a name and then displays all communication with a person/organization, chronologically sorted. As a bonus, this is linked to my address DB, so I can easily call up additional data. This has saved me several times when an almost-forgotten ex-client called.
I guess there are more professional CRM products for this kind of stuff, but I feel better with a homebrew solution. And should I ever want to move to a different format, FileMaker can export CSV and TAB. Good enough for me.
Most of the emails I need to store are decided mail by mail. I keep 20-50 folders on my macs, and store most ork related mails in .mbox format every now and then. .. I really rarely find reading old personal emails or IMs interesting. But it must be something "wrong" with me, since I don't keep any personal photos anywhere eitehr.
Very rarely I have an urge to store any personal emails.
I have saved a few IMs, however my bf has saved practically all IMs for a long time, for work purposes. That includeds a very interesting amount of old IMs with me and him,
If you're serious about archiving or migrating your email, take a look at Mailbag Assistant and Aid4Mail for Windows. Mailbag Assistant makes it easy to read email from many different formats. It can search and display your email archives from CD-ROMs and any other location accessible to Windows Explorer. Aid4Mail supports even more mail clients and can archive your messages into highly compressed ZIP files. It can also help you migrate your email to another mail client or a database. Aid4Mail is very accurate; it can correctly migrate status information and is capable of rebuilding Eudora mailbox files and MS Outlook message folders into standard mbox files.
Mailbag Assistant 3.8:
http://www.fookes.com/mailbag/
Aid4Mail 1.0:
http://www.aid4mail.com/
--
Eric Fookes
http://www.fookes.com/
I love Bloomba verymuch, but now they are acquired by Yahoo and they don't want to continue their e-mail client. !$#$@!
Now I need to go back to my previous e-mail client.. MS Outlook. (ok, ok.. I know..)
Anybody know how to export Bloomba e-mail?
Thanks!
Gmail. Duh. Forward your email to gmail.
Need an invite?
Need 50?...
Deja Vu
n. 1. The sensation that you've read this very article before.
I assume this experience has lead you to evaluate the various options for your IMAP server software. What do you use?
On my systems I'm using:
* fetchmail
* procmail
* courier-imapd with Maildir
* any email client.
This goes the more efficient solution to read and sore mail.
http://www.michel.eti.br
I hate to be redundant but I can't stress how useful this has been for me.
I've been running courier-imap using maildir for ages now, and typically use kmail and squirrelmail as clients, but I can use anything I want at any time and not get out of sync.
Per a suggestion in this thread I just switched to dovecot in about 15-20 minutes, and that seems to work just fine after a little tweaking of my imap root settings.
Cyrus seems to be a good option, but I'm hesitant to try it since documentation and howtos seem to be lacking. One thing I like about my choices so far is that my mail just stays in maildir format, and I could read that with grep and less if I had to.
Anybody have any luck with a single-user scale instllation of cyrus? Can it handle a maildir where the mail delivery agent and procmail just stick mail in folders? If the various indexes get messed up can you just blow them away without losing anything?
One of these days, I'll get around to importing all the old MMDF mailboxes from my Waffle install of yore.
Mail? Put "slashdot" in the subject to pass the spam filters.
I do something similar to the maildir approach, converting e-mail (from mbox files from long ago and from collected Usenet messages too!) to individual text files. Each text file contains the raw message text, encoded attachments and all, so nothing is lost.
However, I'm using BeOS's BFS to store these files so they are a more quickly searchable than plain text files (subject, from, to, dates are all indexed by the OS). Plus I file them away in directories by category. I also wrote a conversion utility [MailboxFileToBeMail] for mbox and Usenet messages that automatically attaches attribute tags to the message files, and even does international character set conversion on the subject and other key text.
- Alex
Most email solutions have some form of IMAP connector so run a local IMAP server on a box. COnnect to that via you email client and drag your email into the imap folders. I had an old outlook PST file and few old OE relics that I migrated onto a more flexible UNIX solution just by connecting Outlook (imported OE into outlook easily enough) to IMAP and draggging the emails into the IMAP folder.
:).
Basicly there should be least some form of migration/upgrade available to most formats that least allows you a way to get it onto IMAP. Worst case you may have to script something but I'd guess that would be some real perverse format and would very very old, but there again by definition it wouldn;t be that complicated (ie pre attachm,ent days type affairs). Also if your realy stuck you could always use a macro program to copy and paste them all out
BTW, it's also the only way to reconcile an enforced corporate suffering of /barf/ Outlook and more sane programs like Evolution (slooow) and kmail (not quite as pretty) or pine (good for SSH on slow dialups ;-).
With an IMAP backend you can try it all without tying yourself into one format - that's Open Standards for you!
Insert
Keep all of it ESPECIALLY the spam.
:).
The more spam the better.
When they ask for your emails, give them ALL the emails, including the PENIS ENLARGEMENT ones, the nigerian scams etc.
Even more fun if you use unreliable mail software so that you lose the metadata from time to time, or it gets corrupted.
--
Alternatively, delete everything, but the spam
http://gmail.google.com/gmail I remember back in the day.. we had to write down each 1 and each 0 with a chisel into stone, and then lug the tablet up all 22 flights of stairs to the filing cabnet! And we liked it too!
http://freshmeat.net/projects/yahoopops/
There are other similar projects listed on Freshmeat & SourceForge that fill the same role. YahooPOPs happens to be the one I use.
"No Bill of Attainder or ex post facto Law shall be passed." - U.S. Constitution, Article I, Section 9.
So Steamboat Willie is NOT actually protected by copyright, since at the time it was created copyright protection only lasted fourteen years!! The constitution says so!!
I wonder if that has been tried before, ie arguing in court that copryright extensions such as the Disney Bono Act. Because copyrights are supposed to encourage creativity, Thomas Jefferson using actuary tables said copyrights should only last at most 28 years. Tbe way things are now, though most writers will have one, one big seller can mean the person never has to write again. This is the opposite of encouraging creativity. Not only does it not encourage more productions but it also locks out others who could take an original work and create another body of work from it as Michael Cunningham did with Virginia Woolf's "Mrs. Dalloway" in writing his "The Hours".
Ooh, speaking of Bono, what does everyone thing of Bono heading the World Bank? While I can't say he'd be the best choice he's at the top.
FalconShould there be a Law?
The 5th Ammendment says that you cannot be forced to testify against yourself. However, you can be forced to turn over other evidence that is very damning to your case.
My other first post is car post.
>>> IMAP isn't really a word, its an anocrym.
>> acronym - n. A word formed from the initial letters of a name
> thanks for that, i was wondering why it looked wrong when i wrote it
Maybe it was not only the spelling which made it look wrong. The excerpt from the dictionary actually says that an acronym is a word.
With the advent and subsequent improvements of LiveCD distros, it should be relatively painless for the average /.'er to:
* Create a multi-session CD/DVD with your favorite Linux LiveCD distro
(or roll your own and create an ISO for future use)
and
* Backup email files to said CD/DVD
(I suggest a set of re-writable media of good quality to play it safe.)
Further suggestions:
1. It would be advisable to split your archives (ie. Mail2004, etc.), especially if you plan to retain a sizeable amount of mail.
2. Convert archives from older mail clients before creating backup, or use a newer mail client that can read the old files with ease.
Good luck!
that was a serious question, your reaction doesn't help :|
Hivemind harvest in progress..
For example, create a mountable r/w image with the best encryption supported by your OS. FIll this with your email archives. umount it, then encrypt the image file with a standalone tool, such as gpg. Then rename the file to something absolutely inane yet credible, possibly with a bogus filename extension.
Keep a copy on your HD, with a CD or DVD backup.
Don't forget to use encrypted swap, or your strong passwords might be easily recoverable.
If like many people you have financial information in your mail boxes, be prepared! Your house could be broken into tomorrow, your computers stolen and your HD's hocked on eBay to someone who buys carloads of surplus drives for the purpose of identity theft. Encryption protects you from this and other dangers.
I have a 10GB partition saved for archiving all my data, between AOL Instant Message conversations (courtesy of DeadAIM -- JDennis.net), and I also log what webpages I go on because i tend to forget easily.
As for e-mail, I keep all of it. I have over 10,000 e-mail messages in my archived folders (encrypted of course). I just need to figure out how to back it up and burn it to a CD using Mozilla Thunderbird.
Posts not to be taken literally. Almost everything is sarcasm.
So far, it's very simple. But it needs a lot of paper. Really. I began to do like that in a time where I didn't had so good backup solutions... I stopped after around 500 pages. That's not 'that' handy. It's expensive, and very big. Sure, it's then much more stable than any other electronics storage ; but then you lose all the rapid search capabilities that make the email so convenient...
Avoid the email, it's just a loss of time. (for the argumentation, take a look at this page )
all my mail and news archives (messages I wanted saved for posterity) are stored in mbox format in my cvs. It does mean I have to resolve some conflict from time to time, due to the way diff matches changes, but that's only really on high volume lists like lkml and beowulf, or identical mails which might have come in in a different order on different machines.
Just like my homedir.
'I am become Shiva, destroyer of worlds'