Eisenstadt's Analysis Of 8 Years' Worth Of Email

Hylton writes "Thought this might be of interest: Marc Eisenstadt's saved every email he's gotten over the past eight years, including spam, and run an analysis of it."

Comments

Apparently the analysis is still running

[Anonymous Coward]

on their webserver.

Spam

[thinkliberty]

I have received more spam in the past week than I have legitimate email in the past 10 years.

Re:Spam

[PopeAlien]

Ah! but did you save and analyze them?

Re:Spam

[BosstonesOwn]

I have analyzed it all and apparently the people sending me these spam messages know my plight.

I need a bigger penis

I need teen sluts who suck **** on webcams

And apparently I shouldn't be telling anyone this but this nice man in nigeria , who is the lawyer in charge of my long lost grand father mutambi wikimbo is trying to get me $5 million american dollars but I have to pay a tax of $5 thousand american dollars to nigeria and he will gladly handle it for me.What a swell guy.

Remembering when..

[Bite-lover]

Must be nice to be able to look back on porn-spam and feel old. 'Hot XXX - Newcomer Jenna!'

Slashdotted

[Anonymous Coward]

If its already slashdotted, he's also probably saving all of his server logs as well.

hah

[usernotfound]

my yahoo account i use to collect spam gets 1700 a month, while my "real" email account i've recieved 1566 since august of 2003, only 10 of those being spam.

Indeed

[mboverload]

I used to NEVER get spam, I didn't know what people were complaining about. I was on a mail server that no spammer really knew about, so there were no dictionary mailings. However, once I posted that paticular email on just a few websites I have been getting ~50 spam a day. There is no way they got my email through me signing up for things because I use a seperate address. I'm just glad the kind of practices they use (trawling the internet for emails) are illegal, although that doesn't mean much.

I will never buy anything from spam, and whoever does has got to be a complete moron.

Re:Indeed

[iced_773]

I should point out that you shouldn't respond to spam under ANY circumstances - it just verifies to the spammer that your address exists.

Article Text

[Anonymous Coward]

February 11, 2005
Eight years of email stats, pass 1Email This EntryPrint This Entry
Posted by Marc Eisenstadt

What's the reality behind the 'email overload' talk? Let's look at some numbers... personal numbers.

To kick things off, I've got a huge email archive. I started emailing in the early ArpaNet days, around 1972, and haven't stopped since. My archive has been extremely thorough for at least the past 12 years (and, in case you think I'm nuts for keeping all of these, my actual regret from a scientific/archive perspective is that I don't have the earlier ones too!). Why? Let's just say that one day I planned to do an analysis of it all... types of mails, social networks, the whole works. But things got a little out of hand.... (anyone lookin' for some data, give me a shout... but first read on)...

Most of this 'storage mania' was triggered by a casual comment in around 1992 or 1993 by Ron Baecker, of the University of Toronto, a longtime research colleague and acquaintance and someone whose work I have long admired and respected. Ron asked me, "given ultra-cheap storage and ultra-fast search, both clearly on their way, why would you ever need either to delete or indeed to accurately file/categorize your emails?"

OK, so as a little personal experiment, I decided to keep 'em, and to see what happened. The quick story is that migrating across machines, operating systems, and preferred email clients, plus being a bit cavalier about the whole thing, has meant that although all the emails are 'there' in various archive files, it takes a little work to get 'em all back in a harmonious form, that is with all headers intact and no duplicates (the main formats are Vax mails, Unix mails, Mac Eudora, PC Eudora, Outlook Express, and Outlook).

The longer story, with some data and preliminary analysis, begins like this:

Even though I haven't had the time or motivation thus far to put in the harmonization work required to get all the data in one format and with duplicates eliminated, I nevertheless thought that a little 'first pass' set of totals (with my estimate of their accuracy) would be interesting, and maybe even provide a little coarse empirical support for Stowe's "Just Say No To Email" campaign.

So I quickly eyeballed-and-tallied the most coherent of the archives, spanning eight years of emails, from January 1st 1997 to December 31st 2004. The totals are real enough, but the 'eyeballing' was needed to assess the approximate propotion of spam and duplication involved in the emails. A more detailed analysis later will enable me to do these more accurately. I've indicated my estimate of the margin for error in the third column, and my estimate for the percentage of spam received (and I mean real spam: i.e. either 'greedily-lookin-for-suckers' or 'low-down-mean-and-nasty spam', not conference announcements - you know what I'm talkin' about). For 2003, this number is precise, because I filtered off such spam using SpamAssassin, and counted them! 2004 spam numbers are an extrapolation, but the totals are accurate, as explained below. Here goes:

TABLE 1: Eisenstadt's 1997-2004 email totals
Year

Emails received Est. Error Est. Spam

1997 4320 20% 2%
1998 3996 20% 3%
1999 6821 10% 5%
2000 7580 5% 6%
2001 6125 5% 7%
2002 6497 5% 10%
2003 13092 1% 37.6%
2004 13889 1% 40%

2003 is the most accurate, because (unlike earlier years when I was changing clients and machines) I have all emails in one clean format and all spam preserved, auto-filtered by SpamAssassin into a folder that I look at only a few times a year, scanning rapidly for false rejections. Incidentally, that falsely rejected email rate appears to be roughly 1 in 5000: good enough for me! By 2004, although I kept all emails, I got fed up keeping the spam even for analysis purposes, and can't even be bothered to scan it, so stuff auto-filtered by SpamAssassin is now deleted without my looking at it - so the column 4 '40% spam' in the lower

Einstein?

[kristopher]

Don't misread like I did. I was like, what the hell was Einstein doing with email..

every month on lug radio

[QuantumG]

One of the guys reminds us that people who send those "increase your penis size" emails and other spam don't just do it because they think it is fun to piss off the world, they do it because they make lots and lots of money from it.

That's what anti-spam laws should be targeting, the morons who use the services offered by spammers.

Re:Link seems to be down...

[Vario]

This is the google cache linked with slashcode: http://64.233.183.104/search?q=cache:GshwWambHvEJ: www.corante.com/getreal/archives/2005/02/11/eight_ years_of_email_stats_pass_1.php

It still tries to access the original site, so it rather slow but you can read the article.

GMail

[GNUALMAFUERTE]

This is pretty interesting (sadly i can't access TFA)
Google should have such a program, there should be a preference in you GMail account, where you can allow /deny google to take stats out of your email. Many interesting information can be collected, like, for example, Ammount of SPAM / Legitim E-mail, % of each kind of spam (viagra, drugs, porn, etc), spam by countrys, % of Text / HTML email, and even other interesting stats not e-mail related, for example, language analisys, frequent mispells, toppics of interest by age, etc,etc,etc. I Would gladly allow google to make such stats, it can be done in such a way that no personal / sensitive information would be leaked.

(Thinks about what has just said, and puts tinfoil hat on)

ALMAFUERTE

So he....

[Robotron23]

Saved it for what exactly? Maybe vintage 1997 pr0n e-mails are now worth something to antique pr0n collectors...

Raymond Chen's Analysis...

[ticklish2day]

Microsoftie Chen's analysis, slashdotted a while ago, has pictures too!

Back in the old days...

[Sheepdot]

I remember a time when the size of my genitalia wasn't an issue.

I remember when I never had any Korean friends.

I remember a time when I went to the pharmacist for a drug I needed, not the pharmacist asking me which drugs I wanted to buy online.

I remember when consolidating a loan was a big decision instead of "just a click away!".

I remember a time where when I left high school, there was no chance in hell I'd ever have to hear from those nitwits again.

God, I miss those days.

Re:Back in the old days...

[Ian Action]

I'm sorry you're so upset...

So, would you like to buy some ink cartidges?

If you think this article is about spam, read end

[linuxbaby]

If you think this article is about spam, make sure you read it all the way to the end. It's not.

He's questioning the entire technology of email as an effective way of communicating.

Analyzes not just the spam-count in his email, but the work-time needed to respond to the non-spam emails, too.

This is one of the most thought-provoking articles posted on Slashdot in a long time.

Femto's Law of Email

[femto]

I did a similar analysis in 1998. I came to the following conclusion:

Given enough time, nearly every email becomes irrelevant.

This 'law' is base based on the fact that of many thousands of emails, there were only about 3 or 4 that I judged to be of value (worth keeping) after three years.

A corollary:

You can safely ignore your email and suffer minimal long term consequences.

Here is an example of the application of "Femto's Law". The boss sends you an email asking you to do something. If you ignore the email, the boss will either a) if it is important come and tell you personally or, b) find someone else to do the task. Ultimately I think the law is based on the fact that email is mainly used for trivial stuff and important stuff will eventually be presented to you in a form which is harder to ignore.

I guess the applicabililty might have changed since 1998, if email has come to be used for non-trivial stuff, but I reckon it's mostly still true.

Side note: the reason I ended up doing the analysis is because the 'delete' button stopped working on my mail client and I had to sort my emails when jobs. AT the time I posted my conclusions to the rest of the University department, to other people's amusement.

PS. No, I'm not brave enough to ignore my email!

Re:Femto's Law of Email

[dvdeug]

Given enough time, nearly every email becomes irrelevant.

Given enough time, nearly everything becomes irrelevant. That job resume you're writing up now is going to be pretty irrelevant in 3 years; but that doesn't mean you can ignore it now.

Not very much

[brunes69]

I rotate my email folders every 6-9 months to increase performance.

Even so, I have 2 folders with over 9000 Emails in them. My work Inbox alone has 1015. None of these are spam - I filter those out through a combination of SpamAssassin and manual filtering.

Anyways - my point is that the numbers in this article are small potatoes. He talks about 250 Emails in a week - I easily get 300 -400 Emails **a day**, probably 40-50 of which are directly work related, the other 350 related to various other side projects of mine, so they are just as important.

I would say I read around 25-50% of my Emails. The rest I only give a cursory scan. His numbers for reply times are way off for a number of reasons:

- Hardly anyone replies to every email they recieve. Most of it needs no reply.

- He basically says that the time spent reading the emails and responding is a waste. Well, what do you think managers did to communicate with you before email? You had faxes, daily memos, daily reports to file... it is just more streamlined now. It is not like this stuff is new.

Newsflash - work is difficult. People are distracting to your work. Shit happens. Deal with it, just like everyone else has for the past 150 years.

Re:If you think this article is about spam, read e

[Tony Hoyle]

It depends on what you use it for.

I work for a company on the other side of the globe.. couldn't do that without email. I also support an opensource project with 10,000 downloads a week... that generates 'a few' support queries :) Heck, without email I don't even think I could do that by phone without hiring a call center.

Gelfling's Axiom of Irrelevant eMail

[gelfling]

Is this:

90% of all eMail is useless the moment it arrives in your inbox.

The First Corollary of eMail age is this:

All remaining eMail is useless no more than one year after the moment it arrives in your inbox.

The Second Corollary of eMail age is this:

eMail accidently deleted will become instantly irrelevant or it will be resent without your request.

Own domain offers new methods

[4Lancer.net]

Having your own domain offers a neat way of tracking where spam comes from. For example, if you see the email I use here, I will know any spam that comes from someone getting my address from here. Of course, /. isn't the best example. Say I sign up at a website, misfitriprapper.com. I will use misfitriprapper.com as the username before the @4la... I use this method EVERYWHERE. I just sent an email last night to Epson support. My email address? epson.com@4la... We've all learned years ago to not trust anybody, so, I don't even trust the big companies like Epson.