Eisenstadt's Analysis Of 8 Years' Worth Of Email

Hylton writes "Thought this might be of interest: Marc Eisenstadt's saved every email he's gotten over the past eight years, including spam, and run an analysis of it."

Comments

Apparently the analysis is still running

[Anonymous Coward]

on their webserver.

Spam

[thinkliberty]

I have received more spam in the past week than I have legitimate email in the past 10 years.

Re:Spam

[PopeAlien]

Ah! but did you save and analyze them?

Re:Spam

[BosstonesOwn]

I have analyzed it all and apparently the people sending me these spam messages know my plight.

I need a bigger penis

I need teen sluts who suck **** on webcams

And apparently I shouldn't be telling anyone this but this nice man in nigeria , who is the lawyer in charge of my long lost grand father mutambi wikimbo is trying to get me $5 million american dollars but I have to pay a tax of $5 thousand american dollars to nigeria and he will gladly handle it for me.What a swell guy.

Re:Spam

[slimak]

Thats a great idea! I have many fond memories from college that involve signing my roomate up for hundreds of explicit mailing lists using some simple application that let you enter and email and check categories. It backfired somewhat though as he really enjoyed many of the porn services.

Remembering when..

[Bite-lover]

Must be nice to be able to look back on porn-spam and feel old. 'Hot XXX - Newcomer Jenna!'

!42

[Anonymous Coward]

Apparantly the computer spent months compiling and cross referencing only to spit out this cryptic message: Host not found

Slashdotted

[Anonymous Coward]

If its already slashdotted, he's also probably saving all of his server logs as well.

hah

[usernotfound]

my yahoo account i use to collect spam gets 1700 a month, while my "real" email account i've recieved 1566 since august of 2003, only 10 of those being spam.

Indeed

[mboverload]

I used to NEVER get spam, I didn't know what people were complaining about. I was on a mail server that no spammer really knew about, so there were no dictionary mailings. However, once I posted that paticular email on just a few websites I have been getting ~50 spam a day. There is no way they got my email through me signing up for things because I use a seperate address. I'm just glad the kind of practices they use (trawling the internet for emails) are illegal, although that doesn't mean much.

I will never buy anything from spam, and whoever does has got to be a complete moron.

Re:Indeed

[iced_773]

I should point out that you shouldn't respond to spam under ANY circumstances - it just verifies to the spammer that your address exists.

Re:Indeed

[autopr0n]

Trawling for email addresses isn't illegal at all, certanly not in the USA which actualy legalized spam with the CAN-SPAM act.

The domains I use for email arn't even up right now, and I'm using gmail these days anyway. I had been using 'throwaway' emails for everything, and then a spammer started jo-jobbing me. Meaning that they started using fake addresses @mydomain. So I was getting tons and tons of bounce messages. It was awful.

These spammers are horrible people, but they're not even close to stupid. They're obviously making money off of it, or they would have stopped doing it a long time ago.

Re:Indeed

[A nonymous Coward]

Try your own domain name on a dialup connection :-) My own account gets around 200 spams a day. It annoys me, but doesn't take long to delete, since so much of it is 5 or ten copies of the same thing, which sticks out like the proverbial sore thumb when viewed once or twice a day.

But starting last summer, maybe 9 months ago, some spammers realized they had an untapped (fools') gold mine to plunder, and my simple little home domain has been receiving more and more spam to accounts that don't exist, like bill123 and so on. My poor little dialup domain has been receiving around 50-60,000 spams a day to those bogus accounts. It hit 120,000 one day.

It's easy enough to deal with since it is known to be spam by definition of going to bogus accounts. I never see it unless I am curious. I collect stats daily on how many unique account names were used, around 3000. It just amazes me that those bozos would send so much pure crap with no hope of ever getting a response.

Re:Indeed

[Anonymous Coward]

And also set your email client not to load images, or anything remote for that matter, off the net. They can just add a image.jpg?id=123456 and know that the email address in their db with the id of 123456 read their spam message.

Re:Indeed

[joschm0]

> I should point out that you shouldn't respond to spam under ANY circumstances - it just verifies to the spammer that your address exists.

Wouldn't they know if your email address is good by the fact that it wasn't rejected as an invalid address?

Re:Indeed

[Grakun]

> > I should point out that you shouldn't respond to spam under ANY circumstances - it just verifies to the spammer that your address exists.

> Wouldn't they know if your email address is good by the fact that it wasn't rejected as an invalid address?

It verifies that the user has read the spam. There are a lot of old inactivate email addresses on the web, which still exist but are never read. This way the spammer knows that their spam is actually being viewed by a user, and not just wasting space in an inbox.

Re:Indeed

[arbitraryaardvark]

You should respond, if you can do so in person.

Re:Indeed

[erikdalen]

As the From: address usually is forged they will just bounce to the forged address. So most spammers don't check if the mails arrive or not, they can use that bandwidth for sending more spam.

I've given up trying to not get spam, I filter it instead. Usually it's aroung ~400-500 spams/day.

I've only saved all my legitimate email for the last 10 years though :)

Re:Indeed

[Just Some Guy]

My poor little dialup domain has been receiving around 50-60,000 spams a day to those bogus accounts. It hit 120,000 one day.

Two words:

1. DNSBL
2. Greylisting

Add those to your setup and see that drop to about 30-40. Let SpamAssassin clean up the rest and forget about it.

Hotmail

[Dash'n'SlashDot]

I have managed to maintain a hotmail account for 10 years, which I consider a feat. It isn't clean of spam or anything, but 10 filters based on keywords in the subject or body make a HUGE dent in the amount that reacheas my inbox.

Re:Hotmail

[SamSim]

where is this spam phenominon? i get maybe two a week....

Really? What's your email?

Article Text

[Anonymous Coward]

February 11, 2005
Eight years of email stats, pass 1Email This EntryPrint This Entry
Posted by Marc Eisenstadt

What's the reality behind the 'email overload' talk? Let's look at some numbers... personal numbers.

To kick things off, I've got a huge email archive. I started emailing in the early ArpaNet days, around 1972, and haven't stopped since. My archive has been extremely thorough for at least the past 12 years (and, in case you think I'm nuts for keeping all of these, my actual regret from a scientific/archive perspective is that I don't have the earlier ones too!). Why? Let's just say that one day I planned to do an analysis of it all... types of mails, social networks, the whole works. But things got a little out of hand.... (anyone lookin' for some data, give me a shout... but first read on)...

Most of this 'storage mania' was triggered by a casual comment in around 1992 or 1993 by Ron Baecker, of the University of Toronto, a longtime research colleague and acquaintance and someone whose work I have long admired and respected. Ron asked me, "given ultra-cheap storage and ultra-fast search, both clearly on their way, why would you ever need either to delete or indeed to accurately file/categorize your emails?"

OK, so as a little personal experiment, I decided to keep 'em, and to see what happened. The quick story is that migrating across machines, operating systems, and preferred email clients, plus being a bit cavalier about the whole thing, has meant that although all the emails are 'there' in various archive files, it takes a little work to get 'em all back in a harmonious form, that is with all headers intact and no duplicates (the main formats are Vax mails, Unix mails, Mac Eudora, PC Eudora, Outlook Express, and Outlook).

The longer story, with some data and preliminary analysis, begins like this:

Even though I haven't had the time or motivation thus far to put in the harmonization work required to get all the data in one format and with duplicates eliminated, I nevertheless thought that a little 'first pass' set of totals (with my estimate of their accuracy) would be interesting, and maybe even provide a little coarse empirical support for Stowe's "Just Say No To Email" campaign.

So I quickly eyeballed-and-tallied the most coherent of the archives, spanning eight years of emails, from January 1st 1997 to December 31st 2004. The totals are real enough, but the 'eyeballing' was needed to assess the approximate propotion of spam and duplication involved in the emails. A more detailed analysis later will enable me to do these more accurately. I've indicated my estimate of the margin for error in the third column, and my estimate for the percentage of spam received (and I mean real spam: i.e. either 'greedily-lookin-for-suckers' or 'low-down-mean-and-nasty spam', not conference announcements - you know what I'm talkin' about). For 2003, this number is precise, because I filtered off such spam using SpamAssassin, and counted them! 2004 spam numbers are an extrapolation, but the totals are accurate, as explained below. Here goes:

TABLE 1: Eisenstadt's 1997-2004 email totals
Year

Emails received Est. Error Est. Spam

1997 4320 20% 2%
1998 3996 20% 3%
1999 6821 10% 5%
2000 7580 5% 6%
2001 6125 5% 7%
2002 6497 5% 10%
2003 13092 1% 37.6%
2004 13889 1% 40%

2003 is the most accurate, because (unlike earlier years when I was changing clients and machines) I have all emails in one clean format and all spam preserved, auto-filtered by SpamAssassin into a folder that I look at only a few times a year, scanning rapidly for false rejections. Incidentally, that falsely rejected email rate appears to be roughly 1 in 5000: good enough for me! By 2004, although I kept all emails, I got fed up keeping the spam even for analysis purposes, and can't even be bothered to scan it, so stuff auto-filtered by SpamAssassin is now deleted without my looking at it - so the column 4 '40% spam' in the lower

Einstein?

[kristopher]

Don't misread like I did. I was like, what the hell was Einstein doing with email..

every month on lug radio

[QuantumG]

One of the guys reminds us that people who send those "increase your penis size" emails and other spam don't just do it because they think it is fun to piss off the world, they do it because they make lots and lots of money from it.

That's what anti-spam laws should be targeting, the morons who use the services offered by spammers.

Re:every month on lug radio

[QuantumG]

obviously I disagree. If you're willing to fund people who piss off millions, just so your penis can be larger, you should spend some time in jail.

Re:every month on lug radio

[NanoGator]

"...don't just do it because they think it is fun to piss off the world, they do it because they make lots and lots of money from it."

Funny thing is, they don't necessarily make money from people buying it, but rather the people advertising it. "Give me $10,000, and I'll get your message out to 10,000 people!" "Okay! That's a lot cheaper than buying a banner on a big site!" (Note: The numbers are made up.)

Re:every month on lug radio

[QuantumG]

ahh the irony of your post combined with your sig (Ferion being amway for geeks and all).

Re:every month on lug radio

[NanoGator]

"ahh the irony of your post combined with your sig (Ferion being amway for geeks and all)."

Referall rewards != Amway. Besides, that's not what irony means. Perhaps if I had said "It's all a huge scam" and if Ferion were actually what you claim it to be, you could call me a hypocrite. :)

Re:every month on lug radio

[QuantumG]

Ferion is like Amway because they both bait-and-switch you.

Ferion: "It's free to play"

Amway: "Make $10,000 a week!"

Ferion: "well actually, it's only free to play if you sucker other people into paying for you."

Amway: "well actually, you can only make $10,000 a week if you sucker other people into selling for you."

Re:Link seems to be down...

[Vario]

This is the google cache linked with slashcode: http://64.233.183.104/search?q=cache:GshwWambHvEJ: www.corante.com/getreal/archives/2005/02/11/eight_ years_of_email_stats_pass_1.php

It still tries to access the original site, so it rather slow but you can read the article.

GMail

[GNUALMAFUERTE]

This is pretty interesting (sadly i can't access TFA)
Google should have such a program, there should be a preference in you GMail account, where you can allow /deny google to take stats out of your email. Many interesting information can be collected, like, for example, Ammount of SPAM / Legitim E-mail, % of each kind of spam (viagra, drugs, porn, etc), spam by countrys, % of Text / HTML email, and even other interesting stats not e-mail related, for example, language analisys, frequent mispells, toppics of interest by age, etc,etc,etc. I Would gladly allow google to make such stats, it can be done in such a way that no personal / sensitive information would be leaked.

(Thinks about what has just said, and puts tinfoil hat on)

ALMAFUERTE

So he....

[Robotron23]

Saved it for what exactly? Maybe vintage 1997 pr0n e-mails are now worth something to antique pr0n collectors...

Raymond Chen's Analysis...

[ticklish2day]

Microsoftie Chen's analysis, slashdotted a while ago, has pictures too!

Re:Raymond Chen's Analysis...

[ticklish2day]

From the post...

The big red splotch in August 2003 around the 100K mark is the Sobig virus.

Back in the old days...

[Sheepdot]

I remember a time when the size of my genitalia wasn't an issue.

I remember when I never had any Korean friends.

I remember a time when I went to the pharmacist for a drug I needed, not the pharmacist asking me which drugs I wanted to buy online.

I remember when consolidating a loan was a big decision instead of "just a click away!".

I remember a time where when I left high school, there was no chance in hell I'd ever have to hear from those nitwits again.

God, I miss those days.

Re:Back in the old days...

[Ian Action]

I'm sorry you're so upset...

So, would you like to buy some ink cartidges?

If you think this article is about spam, read end

[linuxbaby]

If you think this article is about spam, make sure you read it all the way to the end. It's not.

He's questioning the entire technology of email as an effective way of communicating.

Analyzes not just the spam-count in his email, but the work-time needed to respond to the non-spam emails, too.

This is one of the most thought-provoking articles posted on Slashdot in a long time.

Shows value of backups

[confusedneutrino]

I had a very similar setup going on for a while, but I lost it over a year ago. 6 years and 2 gigs of emails lost to a faulty power supply. Scouring turned up nothing usable and I didn't have backups of my emails.

I felt like I lost a part of my past...

Goes to show the value of backing up your data.

Femto's Law of Email

[femto]

I did a similar analysis in 1998. I came to the following conclusion:

Given enough time, nearly every email becomes irrelevant.

This 'law' is base based on the fact that of many thousands of emails, there were only about 3 or 4 that I judged to be of value (worth keeping) after three years.

A corollary:

You can safely ignore your email and suffer minimal long term consequences.

Here is an example of the application of "Femto's Law". The boss sends you an email asking you to do something. If you ignore the email, the boss will either a) if it is important come and tell you personally or, b) find someone else to do the task. Ultimately I think the law is based on the fact that email is mainly used for trivial stuff and important stuff will eventually be presented to you in a form which is harder to ignore.

I guess the applicabililty might have changed since 1998, if email has come to be used for non-trivial stuff, but I reckon it's mostly still true.

Side note: the reason I ended up doing the analysis is because the 'delete' button stopped working on my mail client and I had to sort my emails when jobs. AT the time I posted my conclusions to the rest of the University department, to other people's amusement.

PS. No, I'm not brave enough to ignore my email!

Re:Femto's Law of Email

[dvdeug]

Given enough time, nearly every email becomes irrelevant.

Given enough time, nearly everything becomes irrelevant. That job resume you're writing up now is going to be pretty irrelevant in 3 years; but that doesn't mean you can ignore it now.

Re:Femto's Law of Email

[wmspringer]

You can safely ignore your email and suffer minimal long term consequences.

I dunno...in 3 years I might not care that my boss wanted to see me this Friday, but if I ignore her email there's a pretty good chance I'll be changing jobs :-)

Not very much

[brunes69]

I rotate my email folders every 6-9 months to increase performance.

Even so, I have 2 folders with over 9000 Emails in them. My work Inbox alone has 1015. None of these are spam - I filter those out through a combination of SpamAssassin and manual filtering.

Anyways - my point is that the numbers in this article are small potatoes. He talks about 250 Emails in a week - I easily get 300 -400 Emails **a day**, probably 40-50 of which are directly work related, the other 350 related to various other side projects of mine, so they are just as important.

I would say I read around 25-50% of my Emails. The rest I only give a cursory scan. His numbers for reply times are way off for a number of reasons:

- Hardly anyone replies to every email they recieve. Most of it needs no reply.

- He basically says that the time spent reading the emails and responding is a waste. Well, what do you think managers did to communicate with you before email? You had faxes, daily memos, daily reports to file... it is just more streamlined now. It is not like this stuff is new.

Newsflash - work is difficult. People are distracting to your work. Shit happens. Deal with it, just like everyone else has for the past 150 years.

Re:Not very much

[ricka0]

Wow... how do you get anything done? ;)

You have a good point about it being more streamlined... however, I suspect that since e-mail is easier to send than a memo, fax, etc, etc, there would be more e-mails than the other mediums in the past. Also, more of them seem to be written with less thought put in. You always hear stories about people wishing they hadn't sent that e-mail or how the number errors in e-mails vs memos, etc are so much greater. If there are indead more errors in e-mails, does a poorly written e-mail with various errors take longer to read? (On Slashdot at least, I think that is definitely the case with poor spelling/grammar in posts, since you then must skim past 2 pages of spelling/grammar fanatics arguing with themselves!)

Re:Not very much

[baalz]

Yeah, this sort of analysis seems fairly frivolous. Everybody uses email differently. I've noticed fairly substantial "email culture" differences in the jobs I've worked at. At my current job I usually get about 10 emails a day from people I've never met in other departments telling the whole world they're stepping out early for a doctor's apointment, a dozen reminders every month to fill out your time cards (sent to everybody regarless of if they're already filled out), etc. Like the parent poster, I do more than glance at less than 50% of my email, and this is internal email with no real spam. At previous jobs the email culture was such that managers would send time sensitive requests by email, we discussed more detailed technical issues, and there were much fewer "worthless" mails. Email was used in a very different fashion. When I got far less numbers of email, I spent more of my time on it. Not only did I have a much higher rate of reply, but I also had mail notification turned on and you've got to figure on the cost of context switching from whatever I was already working on when that little chime goes off (more than 3 minutes if the thought required is not trivial).

Re:Not very much

[kjamez]

i'm curious how much of it is second-hand-spam (you submitted email, they sold it) vs. bot/spidering/harvesting (eg: wholly unsolicited email) ...

i have a catch all tld i use to watch. signup like kjamez-slashdot@tld.com which comes to all the same box, so when i start getting unsolicited emails to kjamez-slashdot@tld.com from random people, i can at least see the origin to some degree. i do the same with magazine subscriptions and credit cards and the like. all slight variations on my real name, some even wholly ficticious.

i'm just curious like that.

Re:If you think this article is about spam, read e

[Tony Hoyle]

It depends on what you use it for.

I work for a company on the other side of the globe.. couldn't do that without email. I also support an opensource project with 10,000 downloads a week... that generates 'a few' support queries :) Heck, without email I don't even think I could do that by phone without hiring a call center.

Gelfling's Axiom of Irrelevant eMail

[gelfling]

Is this:

90% of all eMail is useless the moment it arrives in your inbox.

The First Corollary of eMail age is this:

All remaining eMail is useless no more than one year after the moment it arrives in your inbox.

The Second Corollary of eMail age is this:

eMail accidently deleted will become instantly irrelevant or it will be resent without your request.

Re:Gelfling's Axiom of Irrelevant eMail

[gelfling]

No I'm afraid not. It's actually a variant of Gelfling's Axiom of voicemail which is:

You don't really need it, if it's important enough they'll call back.

Mods On Crack (M.O.C.)

[Lehk228]

*Sacrafices karma to protest idiotic mods*

Re:If you think this article is about spam, read e

[Bonhamme Richard]

He raises a good point. I'm a college freshman at one of the most wired campuses in America (Carnegie Mellon). I get NO SPAM since I'm on the campus account, and my packrat mentality has forced me to save all of my emails. I've deleted a (very) few, but have saved 2,200 +/- 50 since September. I am in Navy ROTC, have a long distance girlfriend, and some of my profs really like email. Between these three I've got a lot of "work related" mail. If I spent 3 minutes reading/responding to each of the 2,200 emails, then I've spent nearly 4.5 DAYS on email in the last 5.5 months.

It's interesting to think of where the time goes...

Own domain offers new methods

[4Lancer.net]

Having your own domain offers a neat way of tracking where spam comes from. For example, if you see the email I use here, I will know any spam that comes from someone getting my address from here. Of course, /. isn't the best example. Say I sign up at a website, misfitriprapper.com. I will use misfitriprapper.com as the username before the @4la... I use this method EVERYWHERE. I just sent an email last night to Epson support. My email address? epson.com@4la... We've all learned years ago to not trust anybody, so, I don't even trust the big companies like Epson.

Re:Own domain offers new methods

[aCC]

I used to use your system. Unfortunately it has 2 flaws which made me change it again:

First, the spambots also send a lot of mail to fantasy names with your domain or-- even worse-- they use a fantasy name with your domain as the sender address so you get the millions of error mails.

Second, I once received a cease-and-desist letter from a lawyer because I used their domain as part of my email address to subscribe to their newsletter. It was something like lawyer.com@my-domain.net.

I then decided to have only some emails addresses like public@my-domain.net, lists@my-domain.net etc. to roughly know where they come from.

For useless one-time email addresses, I use the Mailinator. Excellent for that purpose.

four main folders

[RalfM]

I've started filtering my email on what is basically a Steven Covey 4 Quadrants principle:

Urgent is email that is important to _my_ goals in life, where there is a deadline. Usually that means other people are involved. For example, email from my PhD students who should be working on research that furthers my interests as well. (Covey quadrant 1)

Important is email that is important to my goals, with no deadline. The stuff that is good for me if I read it, but I didn't used to because of the deadline issue. I now make sure to read through the Important folder once a day. An example is conference announcements in my area. (Covey quadrant 2)

Distracting is stuff that is important to other people, but not really me. Most of my Staff mailing lists go in here. (Covey quadrant 3)

Timewasting is stuff that is fun but not really important to anyone. Friends mailing lists talking about the latest in computer games or eclectic news stories, for example. Stuff I can read for 5 minutes to get a chuckle before meetings. (Covey quadrant 4)

Other email gets put aside for me to find out how to not get it again. For example mailing lists I subscribed to once thinking they'd be useful for me, but really I'm better off searching the web when I need that info rather than wasting my time keeping on top of it every day/week.

It works very nicely, and I only have a couple of filters for the lot. I get 400+ emails a day, incidentally.

Try it -- just set up 4 filters copying rather than moving the emails, and run it in parallel with your current filters...

R

.

Uh..

[danielrose]

Did anyone else find this article INCREDIBLY boring?? I just couldn't care how a balding man analyzes email..

A great love affair ...

[joe_n_bloe]

I have every email that I've ever received since 1992, with some exceptions for work accounts, and if I'd just thrown them out on the "if it's more than a year old..." principle, I'd have missed one of the best email romances any geek has ever had.

Email Address

[Mr. Jax]

Does anyone know if he used the same email address throughout those 12 years? If he switched addresses the spammers might not have known about the new address, thus reducing the amount of spam he will receive the coming years.

Only 40% ??

[lcsjk]

Last year I kept all my email at work for 6 months. I called all mail that I had not personally signed up for to be SPAM and that includes conference announcements. Approximately 51% was SPAM of about 3000 total. I don't have the exact numbers in front of me anymore. During the summer and fall, I let some graduate students use my computer, and now I get approximately 75 % SPAM. I don't read it all, but I also get email to my computer that has a different user name and email address.

An automated analysis.

[john187]

I posted an automated weekly analysis of the language used in my email some time ago.

http://www.2ad.com/~john/spam_zeitgeist/

This focuses more on language used rather than on message type. So it reveals some of the patterns used in marketing messages.

John