A Visual History of Spam

Cristiano writes "Microsoft employee Raymond Chen has saved every spam message and virus-laden e-mail he's received at work since 1997 and graphed the spams and viruses to create a cool visual representation of one man's malicious traffic."

Comments

"one man's malicious traffic"

[Anonymous Coward]

"one man's malicious traffic"

Sounds like a cool title for a future book about Gill Bates.

Re:"one man's malicious traffic"

[00Sovereign]

...is another man's successful honeypot.

Re:"one man's malicious traffic"

[Anonymous Coward]

There is another example of one man's visual SPAM reporting here along with a bunch of other interesting visualisation techniques.

He's got a virus

[SlowDancing]

Bless you.

Obvious

[Anonymous Coward]

If only MS employees spent more time working on their software, and less time doing these kinds of things...

Re:Obvious

[DaHat]

Do you have a problem with programmers being able to spend a little time here and their on their own projects?

Re:Obvious

[DaHat]

I always take it as a complement when someone attacks something like that rather then the content of what I said. Thank you for agreeing with my earlier statement.

Re:Obvious

[Anonymous Coward]

That's compliment, dickwad. Gasp - does this mean I agree with your earlier statement?

Re:Obvious

[Anonymous Coward]

Do you have a problem with programmers being able to spend a little time here and their on their own projects?

You might want to spend some time learning to spell their(sic), chief.

Hey yourself Chief, I believe your parent spelled their correctly, but in an editing oversight meant to delete the first occurence.

Re:Obvious

[wuice]

I guarantee this comment was only made because he is a Microsoft employee.

Re:Obvious

[Anonymous Coward]

Actually, Chief, I think the original post meant to say this:

Do you have a problem with programmers being able to spend a little time here and there on their own projects?

So this is not an editing oversight regarding the occurance of "their" but instead a mispelling as the poster originally noted.

Re:Obvious

[postgrep]

They spent time On it? I thought they just spilled coffee on it, and then a blue pen leaked on it and then claimed it was a graph?

Re:Obvious

[Anonymous Coward]

You missed his point. This guy's a Microsoft employee. Did you get that? I Microsoft employee!

Re:Obvious

[DaHat]

I did not miss his point at all.

I am a bit of a fan of the comments of Raymond Chen on blogs.msdn.com and follow it closely. You can find it at http://blogs.msdn.com/oldnewthing/

Even if you are not a fan of Microsoft due to it's products, ethics, morals, connections, etc, it is still quite the interesting read and applies to many things.

Re:Obvious

[Anonymous Coward]

If it's Microsoft, then yes, because they're the devil, and everything they do is automatically bad.

...it was a slow day on slashdot...

[Fjornir]

...pretty pictures though, did anyone else try the "magic eye" deal and see what I saw?

Re:...it was a slow day on slashdot...

[Phat_Tony]

did anyone else try the "magic eye" deal and see what I saw?

Was it a giant bee guy stepping on a penguin?

Re:...it was a slow day on slashdot...

[derrith]

A sailboat?

Re:...it was a slow day on slashdot...

[Xzzy]

It's a sailboat!

Re:...it was a slow day on slashdot...

[Holi]

No, it's a schooner!

Re:...it was a slow day on slashdot...

[Stuart Gibson]

You know what? THERE IS NO EASTER BUNNY! OVER THERE, THAT'S JUST A GUY IN A SUIT!

Stuart

Insert random amounts of stuff to evade the lameness filter. Yes, I know it's like yelling, that's the whole point.

Re:...it was a slow day on slashdot...

[Loligo]

A schooner IS a sailboat, stupidhead.

Re:...it was a slow day on slashdot...

[Loligo]

Wow, teach me to read further down the replies before jumping the gun...

D'oh.

Re:...it was a slow day on slashdot...

[rasz]

>It's a sailboat!

It's a YACHT !

Re:...it was a slow day on slashdot...

[lcsjk]

Yes! If you had read the replies to your response first, you would not have made such a stupid mistake.

Wait! Something is wrong here!

I used to work with this guy

[optisonic]

I'll post a pic if I can find one to show just how geeky you need to look if you want to do this yourself. :P

Graphed or plotted ?

[Anonymous Coward]

just looks like a bunch of random dots ? perhaps smoothing the data and joining the lines would give a better "graph"

Re:Graphed or plotted ?

[rs79]

I got pretty good meaning from it by just staring at it. It's usefull.

What exactly do you want to know from it that you can't discern?

Raymond Chen in Linux source CREDITS

[Anonymous Coward]

An interesting aside: Raymon Chen is mentioned in the Linux kernel's source 'CREDITS' file:

N: Raymond Chen

E: raymondc@microsoft.com

D: Author of Configure script

S: 14509 NE 39th Street #1096

S: Bellevue, Washington 98007

S: USA

Re:Raymond Chen in Linux source CREDITS

[sriram_2001]

This blog (post) has some interesting info on this.

...This post wouldn't have been possible without Kaushik - he called me up this morning and said that he had spied a familiar name on the Linux 1.0 contributors file. And since the chances of 2 people with the name Raymond Chen and working at Microsoft were pretty slim, we got pretty interested. A bit of Googling lead us to this page (http://grumbeer.dyndns.org/ftp/mail/v5/digest363) which has an email that Raymond Chen has typed out back in 1993. The first thing that strikes you is his Microsoft id. I was taken aback - a Microsoft employee contributing to Linux code? That too kernel level stuff - not some fringe OSS project? Seems like things were a lot different back then. Here's a snippet from that mail From: raymondc@microsoft.com (Raymond Chen) Subject: New Configure script (and some console patches) Date: 05 Jun 93 20:23:30 GMT This patch kit is really *THREE* patches in one. 1. A new Configure script, hopefully easier to use and more flexible than the current one. 2. A kernel configuration switch to enable high-intensity background in lieu of blinking foreground characters. 3. A kernel configuration switch to control the destination of kernel trace messages (printk's). But the part which I really found interesting was this...the way he signs all his mails. Thanks. -- Raymond (just another linux hacker) Chen Definitely not something you would see nowadays. These days, the very mention of the word 'GPL' might get you into serious trouble in Microsoft - and contributing code is definitely unthinkable.I guess back then , Linux was considered more of a hobbyist-thing rather than a future competitor. But I'm only guessing here. An interesting question that arises is the effects of the viral nature of the GPL. If he had worked on GPL code back then, is he 'infected'? Well - I'm no expert in these issues, but its interesting all the same. Before all the Linux supporters jump to any conspiracy theory, I would just like to point out that the only thing this points out is the amazing versatility and skill exhibited by most Microsoft devs and Raymond in particular. This is a guy who knows both Windows and Linux inside out.Awesome!!! I would really like it if Raymond comes and tells us a bit about his past - especially the 'just another linux hacker' days :-) ....

Re:Raymond Chen in Linux source CREDITS

[kaleco]

Yes, he used a PERL script to generate the graph. Less than orthodox for an MS employee. His blog's 'not a .net blog' caption also hints at a certain cynicism being harboured.

Re:Raymond Chen in Linux source CREDITS

[hackrobat]

Yes, it's confirmed.

Re:Raymond Chen in Linux source CREDITS

[professorfalcon]

Will this cause trouble with SCOble?

What does this figure represent?

[kaalamaadan]

As far as I understand, this is the plot of distribution of *size* of the email vs. time. The "darker" color is not enough of a visual hint to determine the *number* of spam messages over time, which is what is important. Also interesting is the large splotches of computer viruses suggesting (maybe!) that variants are roughly the same size, but not exactly the same.

Re:What does this figure represent?

[echeslack]

I think the variation in the size of the virus emails probably has to do both with variants (sometimes) and with the amount of trash the put in the body of the message besides the actual payload.

Re:What does this figure represent?

[Cymsdale]

I think it depends on what you hate more about spam, the amount of messages that clog your inbox, or the amount of bandwith taken up from downloading useless messages.

I agree that this chart doesn't visually represent the amount of spam as much as I would like, it would be simpler (and more informative) if it was broken up into two graphs: Total size over time, and number of messages over time.

Of course the author doesn't intend this to be any kind of serious study. I think he just wanted something that looks pretty.

Re:What does this figure represent?

[ESqVIP]

Agreed. This is especially true for a logarithmic graph, since most small e-mails got condensed in the bottom (while they may be even 4 times bigger than their surrounding neighboors). Those excessively big messages are distorting our view on the smaller ones.

One nice solution would be a not-so-tall version, so we can catch the density of each column more easily, while still making some idea of e-mail size.

But if we just wanted to check the load, a more conventional amount/time graph would be much better.

Re:What does this figure represent?

[Xilman]

More likely, it's the differing size of the headers. He is measuring the total size of the mail, not just the size of the content.

Paul

Only 19000 spam messages?? That's nothing.

[joshuao3]

My primary account receives nearly 500 spam messages a day, and the number is growing. It would only take me 6 months to get that amount of spam. It seems like Raymond Chen is less than average in the amount of spam received. The data analysis is intriguing, nonetheless, and I'm glad he had the forsight to do this project.

Good work

[Anonymous Coward]

Now he'll get even more spam.

Single worst spam day by number of messages: Augus

[FePe]

Single worst spam day by number of messages: August 22, 2002. 67 pieces of spam.

I normally get around 60 spam mails *per day*, so I guess he is rather lucky. The spam mails I receive are fortunately not full of images like the 41 images he got.

It's the Visual part that scares me

[superbondbond]

I think if I were to actually see what went into Spam I'd never be able to eat it again.

Re:It's the Visual part that scares me

[AndroidCat]

So .. if you were able to see the entire process involved in producing a hamburger made from 100% top-grade ground beef, you grab it, take a big bite and go mmmmmm?

There might be steam-seperation and less popular bits of meat involved, but I'd take Spam over a hotdog any day. (Meat marshmallows made from lips and .. the other lips.)

Re:It's the Visual part that scares me

[Garion Maki]

eat or think about what you eat...
never both at the same time :D

Jose nazario might have more spam graphs

[Anonymous Coward]

Jose Nazario arguably has a much more extensive collection of spam, you can see some of his research here: http://www.monkey.org/~jose/wiki/wiki.php?page=Spa mAnalysis.

One of several talks of his on spam (complete with more graphs): http://www.linuxchile.cl/docs.php?op=verVersion&do c=64&id=1 And he's even done generated some really really horribly insane spam collages, but I'll let those interested dig around for them on their own.

Re:Jose nazario might have more spam graphs

[moonbender]

They're holding talks in IRC now? (The document AC linked to is an IRC log.) Cool. I never would have thought of that, but I guess why not? Is this commonly done? I'd like to have something to read. :)

Sorry for posting off-topic, but it's a slow news days, anyway - none of the stories today has gotten more than 250 comments.

Not exactly "reader friendly"

[Hockney Twang]

I would have much preferred to see the volume of email, represented in terms of the size of messages received, displayed on a nice looking bar graph, with viruses in the foreground, spam in the back. Maybe even show legit email as another row in front of the viruses. Or even just a line graph. As it is, the information is occluded by his presentation. He took some raw data, did very little to interpet it, and put it on his blog. The information could be interesting, but the presentation is very lacking.

Re:Not exactly "reader friendly"

[Anonymous Coward]

He took some raw data, did very little to interpet it, and put it on his blog.

Yeah, I really hate it when people don't do my thinking for me.

Single worst day was only 67?

[Shayde]

Single worst spam day by number of messages: August 22, 2002. 67 pieces of spam. The vertical blue line.

This guy needs to get out more. I set up monitoring of all my spam and total message traffic for the last couple years. My current average is around 350-450 spams per day. Check out the spam report I run every night.

Virii? That's a different report. I seperate my virii out of the entire mail feed for the 3-4 domains I run (yay amavisd and postfix). The virii report is a lot more variable, with as many as 1600 viruses a day, and as few as 10, though that's pretty rare.

Spam filtering here is done via amavisd + postfix + spamassassin + some custom rules.

Re:Single worst day was only 67?

[syrinje]

67 that made it past the corporate filters. I ahve to admit that makes it sound like MS has pretty good filters though.

OTOH, he could just be a man with low span suseptibility :)

Re:Single worst day was only 67?

[Anonymous Coward]

This guy needs to get out more. I set up monitoring of all my spam and total message traffic for the last couple years.

Uh-huh. He really needs to get out more :).

Re:Single worst day was only 67?

[MrWa]

This guy needs to get out more. I set up monitoring of all my spam and total message traffic for the last couple years. My current average is around 350-450 spams per day.

It looks like you need to get out more!

Re:Only 19000 spam messages?? That's nothing.

[goneutt]

IF the articel he said that this is what gets through the corporate filters.

Weak!

[schnarff]

Man, this guy really doesn't get much spam at all. Before I threw SpamAssassin on my mail server, I was getting close to 1,000 spams a day on my personal e-mail address at its height. I saved my spam from 2001-2004, and I had over 250,000 messages for the whole period; the volume totals around 1.3GB. So dude's totals are small, if you ask me. ;-P

Re:Weak!

[Anonymous Coward]

you really need to stay off those sites, man. Or give Debbie and Dallas a fake address...

How I avoid spam.

[here4fun]

Here is what I did and I get next to no spam. Actually, I have none. I got an account at yahoo, and I made a login which has numbers mixed in, and is not a word from a dictionary. Think taking the first three lettes of your first name, a couple numbers, the first four lettes of your last name, and a couple more numbers. I never post my email address anywhere on the web, and just use it to communicate with people I know. I have a second email address that I give out to everyone, and that one is not bad with spam either. The account that gets 100 spam messages a day is my account that I used to reply to offers from websites, or that I used when posting on the web. It is a shame, because I don't check that last account except once every other month when I have nothing_better_to_do. And every once or twice a year I get an email which is important.

When I was back in school I never had spam in my university account, but that was before the 2002 spike shown on his graph. I wonder if school email accounts are still off limits. When I was in school, I did not get spam there, it was my "free" email accounts that had spam.

Re:How I avoid spam.

[DuctTape]

I've done the mix-numbers-and-letters routine, and it works rather well. I get no spam on the account that I primarily use for e-commerce, of all things. On my regular easy-to-spell email account, unfortunately, I put it on a buddy's publically-viewable guest log, and someone else entered it on a dubious website to have it email me a funny Flash file that they liked, so now I get about 30 spams a day on it.

Unfortunately you can't put the numbers-and-letters email address on resumes, 'cuz then you look like a luser, according to an HR type I interviewed with.

DT

Re:How I avoid spam.

[iamcf13]

Your approach will work if the people you correspond with with using this 'secret' email address DO NOT add it to their email client's address book (likely Outlook/Outlook Express) or save it on the computer they connect to the internet with. If they do, you may fall victim to the 'fallout' of a Mass Mail Virus like I was a while back. I was accused of sending malware to people who were total strangers to me that I *never* emailed at all in the first place.

As for me, I filter out all my spam and malware with software I wrote after much research and thought into the spam/malware problem.

Please keep the above in mind while other antispam solutions get coverage on Slashdot and this post is (likely) moderated into oblivion for being an 'ad' and 'just like SpamAssassin'--I tried to offer a clearly effective antispam/antimalware solution to all interested parties....

Re:How I avoid spam.

[iamcf13]

Unfortunately you can't put the numbers-and-letters email address on resumes, 'cuz then you look like a luser, according to an HR type I interviewed with.

An email address is a point of human contact on the internet. Nothing more--nothing less. To show such bias against the mere spelling of an email address indicates a mindset at work at that particular business I find...totally unacceptable.

Kept all his spam!?!

[nurb432]

Man, i couldnt if i wanted too.. i get 10mb a day of the crap..

Re:Kept all his spam!?!

[InfiniteWisdom]

So 3 years of the crap would be a little over 10 gb of space. Not a huge amount for today's hard drives. Certainly won't be a huge amount for your hard drive 3 years from now. And this is uncompressed. Its definitely feasibly... whether you'd actually want to is a whole different story

Re:Kept all his spam!?!

[iamcf13]

Man, i couldnt if i wanted too.. i get 10mb a day of the crap..

Then you can delete it all automatically either before or after it gets to your email inbox like I do.

Please keep the above in mind while other antispam solutions get coverage on Slashdot and this post is (likely) moderated into oblivion for being an 'ad' and 'just like SpamAssassin'--I tried to offer a clearly effective antispam/antimalware solution to all interested parties....

Re:I just don't understand

[goneutt]

Theres that AC post a few messages ago that says Mr. Chen shows up in the Linux Credits. Is it possible he's using a linux box in richmond.

Invasion of the penguins.

1997?

[startleman]

There seems to be a disproportionate amount of spam in late 1997 (as compared to the following few years) . . . anyone know why this might be?

Let's get this message out!

[Phat_Tony]

Hey, that graph's some important news (it made it on Slashdot!)

I think we should all email it out to everyone we know.

In case of /.ing

[OO7david]

Here's the Coral cache page.

MS employee rotation

[gspr]

MS employees stay aboard that long? Wow...

Re:MS employee rotation

[stevesliva]

Sure, he's still waiting for his stock options to recover their value.

Re:I just don't understand

[targo]

How did he manage to keep track of this on a M$ box without catching a few of those viruses?

Beause contrary to the popular opinion on Slashdot, you actually have to open and run the attachment yourself in Outlook in order for it to do anything. None of the big e-mail viruses have been able to spread without active help from the user. I have been running Outlook for 6 years by now and never had any problems.

Re:I just don't understand

[tesmako]

Comedy genius, a perfectly done parody of a open source zealot, exaggerated in just the right way.

Well, or completely mornic trash which has somehow gotten moderated interesting anyway.

I guess what I am saying is that no matter how you look at it, that moderation is insane.

Re:Single worst spam day by number of messages: Au

[DrEldarion]

On my "spam account", I currently get approximately 200-300 per day. Unfortunately, Yahoo deletes them after a month, and this has thwarted my plan to see how many I could rack up.

Currently my monthly record is around 7,000.

Obligatory virii correction

[Anonymous Coward]

Viruses!

Re:Obligatory virii correction

[The Ape With No Name]

Explanation: virus is a New Latin word. Well, it is old, but it is used by us as a neologism. The rule of thumb is this: if an ancient coinage it declines according to Latin rules of grammar. If it is a word with no ancient meaning left (virus once meant 'venom' but was used rarely), then it should decline according to the rules of grammar of the adopting language. Virus has no meaning close to venom, and only has a modern sense of an microorganism that causes disease, or a malicious computer program. I guess there is a literary sense in which you can use virus to mean 'venom', but that is not a defintion, per se, but a synonym. So, not virii, but viruses. For example, in my native language, it is not virii, but 'virusov.' Always.

Re:Obligatory virii correction

[Anonymous Coward]

Virus has no meaning close to venom, and only has a modern sense of an microorganism that causes disease, or a malicious computer program

Additionally, Virus as a Latin word has no plurual uses in any known source texts. It is believed that it was used similarly to the way we would use 'water' - we would say "some water" or "the water" but not "some waters" or "the waters" - put two lots of water together and you still have water. Put two lots of venom together and you still have venom (if you don't distinguish types of venom as we would nowerdays)

In fact, the class of latin words that Virus comes from (neuters of the second declension) has no known plurals, so it is uncertain what form a plural of virus would take if there were one in Latin. Most latin scholars agree that 'vira' is more likely than 'virii'

If we really need that cache....

[PedanticSpellingTrol]

1. Welcome our new microsoft-owned-server-slashdotting overlords
2. ???
3. Profit

Doing something similar

[Anonymous Coward]

I made a script that parses procmail log and creates a graph with rrdtool.
The log only goes half a year back, but it's pretty interresting to see. mailstats

Re:Single worst spam day by number of messages: Au

[mrak and swepe]

Although we all hate spam, at least we can engage in some harmless macho posturing re the amount of it that we get.

I'm a mere minnow in comparison to your good self: Just 57 per day, on average.

Me off to stuff a pair of socks into my pants...

Decline in spam volume explained

[asb]

The reason why the second chart shows that the amount of spam has been decreasing is given in the first sentence after the chart: "This particular email address has been inactive since 1995; all the mail it gets is therefore from harvesting done prior to 1995."

So the reason why it shows a decline is because that particular e-mail address is not on the newest e-mail address lists.

But that's only my theory...

RTFA instead of looking at the pretty picture

[Anonymous Coward]

From the page:

Note that this chart is not scientific. Only mail which makes it past the corporate spam and virus filters show up on the chart.

*DOH*

Re:RTFA instead of looking at the pretty picture

[VB]

I also looked at the pretty picture and embellished my personal knowledge base by reading the article, as well.

I agree with the person you responded too; this guy's level of spam is nil. And, the true irony is that M$ (for whom this suck-ass works) is probably 99% of the reason spam exists anyway!

The true question is why is this even remotely "news for nerds. stuff that matters." Don't understand how it made the press when most of us here get as much spam on some days that this prick has received since 1997. News?

What was it you said... Doh? Telling response....

Re:RTFA instead of looking at the pretty picture

[gordgekko]

I agree with the person you responded too; this guy's level of spam is nil. And, the true irony is that M$ (for whom this suck-ass works) is probably 99% of the reason spam exists anyway!

Eh? Care to explain that statement? Microsoft didn't write the first spam, didn't create the open protocol that enables spam to be sent so easily, and doesn't run the biggest ISPs where spam is sent from, though its Hotmail users seem to be quite susceptible to receiving it. So how is Microsoft responsible for virtually all spam?

I don't much care for many of Microsoft's products but let's be a little more circumspect with the accusations we throw around.

Re:RTFA instead of looking at the pretty picture

[VB]

Well, let me try to be more concise (since being circumspect does not apply), but Micro$oft has taken what open source developers created from the Arpanet and turned it into a playground where every icon on a desktop looks like a treat without consequences.

That is exactly what they've contributed to without giving enough attention to encouraging people to be responsible with the tools they use to communicate in an open environment where virtually everyone on the planet can review firsthand the behavior of all others. I stand behind my 99% estimate. Care to offer something different?

Also, take a look at the definition of circumspect before using the term. My argument has no impact on others. It implicates no one but myself since I'm merely offering an opinion. M$ has created software and an environment where 95% of the world is forced to use one tool to do the job of communicating in the Internet medium. I'd argue the term applies much more accurately to the proliferation of their software due to the way they've manipulated the market to ensure it was the only tool in place for most people.

Incidentally, Slashdot is the rebuttal. What are you doing here? >:)

Missing graph

[Skiron]

I would like to see the OS graph of machines sending spam/virus 1998 -> / 2004 -> |

another graph

[DuctTape]

Cristiano writes "Microsoft employee Raymond Chen has saved every spam message and virus-laden e-mail he's received at work since 1997 and graphed the spams and viruses to create a cool visual representation of one man's malicious traffic."

I'd like to have saved every BSOD that I've received since 1997 and make a cool visual representation, too, but the system crashes each time I get one... so much for data retention.

DT

Re:another graph

[caluml]

Or we could all collectively graph our kernel panics.

Wierd...

[dalmor]

I noticed the gap of spam right around New Year's Day 2004. That was when the CAN SPAM act was taken into effect. I guess there are spammers out there that DO follow US laws.

Re:Wierd...

[Fenris Ulf]

No, that's when they turned on corporate spam filtering.

Re:Only 19000 spam messages?? That's nothing.

[Holi]

It seems like Raymond Chen is less than average in the amount of spam received

Umm.. so your the average? Have you ever thought that maybe you are on the high-end of the bell curve.

Raymond Chen is less then you in the amount of spam received, who knows maybe he is exactly the average.

Why don't you poll people and find out.

I would but I dodn't care.

This has been done before

[Prince Vegeta SSJ4]

THIS site even has an animation of the propagation of spam.

Irony

[thrills33ker]

A Microsoft employee keeps a record of his ever-increasing levels of spam and viruses?

Aargh! My irony meter has gone off the scale!!

Re:Irony

[HedonismBot]

It must be from Alanis Morrisette Industries Inc.

MSN and Hotmail

[bayerwerke]

I want to see a graph of the percentage of spam that has headers identifying it's origin as msn.com and hotmail.com (yes, I know headers can be forged).

Re:Single worst spam day by number of messages: Au

[imsabbel]

Yeah, spams/day seems to be an integral part of the common ePenis.
My mail-account is online since 1998. I didnt keep it secret, just didnt do stupid things with it (like sign up adult sites or so).
get 3-7 spams per day. annoying, but thunderbirds only lets 1 or 2 per week slip, so its ignorable.

The only ways people get 500 per day must be in their own stupidity.
(btw: this email-address is also in the whois database. IN fact i only started to get spam regularly after i registred my domain. coincidence?)

Should we rewrite SMTP

[Slinky Saves the Wor]

It boggles the mind to think about how much bandwidth is wasted on the useless trash that spam is. Not to mention just time spent with dealing with that. How much money is lost each year overall due to spam... the number must be huge. This is an unnecessary loss of money and time.

I think this problem will just escalate for as long as we have SMTP in use. So maybe SMTP as a protocol needs a rehaul, or a revision to rewrite it completely (and call it something different). I think it wouldn't be impossible to pull off.

Re:Should we rewrite SMTP

[AndroidCat]

Excellent idea. You go first.

Re:Should we rewrite SMTP

[mabu]

I think this problem will just escalate for as long as we have SMTP in use. So maybe SMTP as a protocol needs a rehaul, or a revision to rewrite it completely (and call it something different). I think it wouldn't be impossible to pull off.

Waste of time.

Every month someone suggests that there's a technological solution to this problem. But there isn't. This isn't a tech problem. It's a law-enforcement/sociological problem.

You can only go so far technologically as long as spammers are allowed to compromise peoples' computers and use them for improper activities.

Ever ask yourself why this doesn't happen with POTS? If someone called your phone once every two minutes to sell you something, the phone company is compelled to do something about it. We don't have this mandate in cyberspace. And spammers commit felony crimes in the tens of thousands every single day, yet law enforcement doesn't go after these guys.

I contend there are approximately a dozen or so spamming outfits that are responsible for 80% of the spam on the Internet. These criminals are in the United States; they're well known and easy to track. They may use foreign systems to partially hide their tracks, but their tracks can easily be traced. If law enforcement would prosecute a few of these outfits and put some spammers in jail, I think we'd see a lot less spam. Some might argue that this wouldn't deter future spammers. I disagree.

Re:Should we rewrite SMTP

[Slinky Saves the Wor]

You can only go so far technologically as long as spammers are allowed to compromise peoples' computers and use them for improper activities.

So, rewrite the mail system in such a way that each mail sent requires the sender's computer to crack a small computational puzzle, which takes e.g. 10 seconds. That's a technological solution. It restricts you so that you can only send 6 mails per minute. For normal use, this is more than enough: in 10 minutes you can send 60 mails. However, you cannot achieve throughput in the rate of many tens of mails per second. Rate of spamming is thereby reduced.

This would hopefully drive the spammers to better focus their "offerings" (the reason we have spam is that some idiots do buy something based on spam!!) and leave the rest of the world out of the collateral damage of their mail bombing.

Ever ask yourself why this doesn't happen with POTS?

But it does. Magazine salesmen, organizations, donation drives, etc. call people, and some people receive more calls. Of course this harassment is not in the level of e-mail systems, because using POTS for spamming is way, way more expensive than email.

Spam doesn't cost so much for an individual person, since the individual person doesn't pay per mail, but it does cost a lot in terms of time spent. Also, the bandwidth usage is significant, considering that this is just "empty" traffic which gets nuked in the receiving end...

Re:Should we rewrite SMTP

[mabu]

So, rewrite the mail system in such a way that each mail sent requires the sender's computer to crack a small computational puzzle, which takes e.g. 10 seconds. That's a technological solution. It restricts you so that you can only send 6 mails per minute. For normal use, this is more than enough: in 10 minutes you can send 60 mails. However, you cannot achieve throughput in the rate of many tens of mails per second. Rate of spamming is thereby reduced.

How is this any form of improvement? Penalize everyone on the planet because of spammers? Force an entire worldwide network systems upgrade? Slow down mail service exponentially?

And for what? A gamble that it will make any difference whatsoever? It's doubtful it would. The idea is totally impractical as well as 99% likely to be ineffective in the first place.

The closest thing we have to a technological solution is very simple, and all effective methods of spam control ultimately gravitate towards this solution, which is SMTP whitelisting. Mail servers should be "licensed" to operate on the Internet and regulated by some centrally-located body that follows very strict standards mutually agreed upon. People argue that finding a body that could legitimately regulate such a system would be impossible. I disagree. We have a TLD system that, for the most part, performs a similar service in handling DNS. We could do one step better with the licensing of SMTP relays and offer networks the ability to only accept mail from whitelisted relays.

This can be done without having to rewrite any smtp protocol and it would actually result in *improved* mail service performance. It would also almost completely eradicate the propagation of worm/virus mail which almost always involves turning the client PC into an (unauthorized) SMTP relay.

There's your technical solution. I'll bet good money if there ever is any major dent done technologically to spamming, it will be based on a form of mail relay whitelisting. Right now RBLs are THE most effective method across the board; the next step is to move from blacklisting to whitelisting. If we don't do this before we migrate to IPv6, we're all screwed.

All of this notwithstanding, we still have nonexistent law enforcement in cyberspace. No technological solution will make up for the fact that the Internet is full of criminals who operate without fear because law enforcement for the most part refuses to take action against these criminals - this is the other half of the "solution" and the onc in which the least amount of progress has been made.

Re:Single worst spam day by number of messages: Au

[Deorus]

Usually most spam which arrives here receives a friendly "550 Go away! Find someone else to spam!" message from Postfix. As a result I never get more than 3 or 4 spam messages a month, and even though my maillog is plenty of rejections, few of them are false positives.

A few well thought regular expressions can do the trick. Most spam comes as HTML formatted messages, and since such messages are usually generated by WYSIWYG editors, they come with all sorts of formatting crap, which makes them easier to spot because their authors care about things such as the border thickness around images inside hyperlinks and so on. Images inside hyperlinks, imagemaps, frames and iframes are also common things in spam, but not in regular messages, so these are other indicators worth of attention.

Of course this doesn't work for everyone, especially for people who don't host their own MTAs, or others who receive a lot of HTML crap in their mailboxes (not my case, I love 7bit us-ascii encoded text messages), but it does a pretty good job for me.

Oh please

[shrewmy]

67 messages a day?! I get that a minute. I saved all my spam since 1982 on tape backups and I have about 3 terabytes of spam!! Not only that but I hand plotted it to show the subject and size of spam received in relation to the date on a 3d graph!! What a wuss!

Is there an over/under on how many more posts like these are made and get modded up??? I'd like to get in on the action.

I wonder how many Raymond Chens are out there ...

[dougmc]

Probably a lot. I remember a Raymond Chen who went by the name of BustrBuny on IRC something like eight years ago ...

Quite the troublemaker he was, but he was fun too :)

He saved copies

[Anonymous Coward]

Does that mean that Bill Gates will be sending me the money he owes me for forwarding all those emails?

Re:Single worst spam day by number of messages: Au

[1Oman]

Uh, so are you going to post your well thought out regular expressions or what?

RTFB

[daytrip00]

Read the blog. This guy is one prolific programmer. He's the guy who ensures that all the old windows apps (like the ones from 10 years ago) keep running on the latest versions of windows. He has all sorts of stories about windows bugs and idiosyncracies and explains how they all came to be. It's a fascinating read and I have an RSS subscrption to his blog.

Read this article which is all about his quest for windows and developer backwards compatiblity.

He give this story about Sim City: It deallocated memory, and then used it right after deallocation. It was a bug that windows 95 allowed. So his code make a special check that you were running sim city and if you were, you could use memory right after you deallocated it. It's pretty amazing to see all the hoops that he and his team jump through. But he's a MSFT ledgend.

PS. That blog entry I linked to sent Shockwaves through Microsoft. It's changed the new XML api design, and resulted in the backporting of Avalon to Windows XP.

Re:RTFB

[Anonymous Coward]

yeah right, that precise guy.

oh, and, you know, if you read Microsoft typical EULAs, they forbid you to do any kind of decompiling, reverse engineering and whatsoever of their product.

and for Simcity and hundreds of other applications, they did just that.

I don't think they asked each editor for the permission to do so, and even if they are Microsoft, they had NO rights to do that otherwise.

(not to mention that more than a few editors went out of buisness and could not be asked for permission)

so who is the pirate hacking sowtfare now ?

Re:I just don't understand

[AndroidCat]

Over several years there have been a number of exploits that worked just by reading the email, or the preview pain displaying it.

They keep saying that it's safe to turn on the preview pain, and that the water is warm--come on in, but previous times that ended with people disappearing in bubbles and pink water.

Still, the people that open a "H0T CH1QUES" email, open the attachment, open the passworded zip file in the attachment .. and run it. There's just no hope for them.

Re:Single worst spam day by number of messages: Au

[Anonymous Coward]

The thing is, he wrote this:

Note that this chart is not scientific. Only mail which makes it past the corporate spam and virus filters show up on the chart.

So his results are really rather meaningless. They show a drop in spam in the last year--likely because Microsoft has installed better spam filters or whatnot--when spam has actually been increasing exponentially.

GMail and spam

[PerlDudeXL]

is the spam on Gmail analyzed in some way?
feed to an RBL or so?

lucky SOB

[Anonymous Coward]

"Single worst spam day by number of messages: August 22, 2002. 67 pieces of spam."

Geez, if I could get my spam down to 67 per 1/2
day I would be doing great.

Of course, he says he is behind a corporate firewall... I suppose my yahoo accout spam filter sux0rs.

email retention

[ballista]

I see he is following Microsoft's email retention policy to the letter!

Re:hmm, how many gigs has he used to store spam?

[leav]

"Totals: 227.6MB of spam in roughly 19,000 messages. 61.8MB of viruses in roughly 3500 messages."

uhm...
nope...

not gigs at all...

think about it: let's assume 10k per spam (according to the graph) times 22,500 messages...

thats 225 MB.. not too off than what he posted...

Math Rules.... :)

You Know You've been recieving too much spam when.

[Anonymous Coward]

In honour of the 'You Know When You've Been Hacking Too Long When' list, I'd like to start a 'You know when you have recieved too much spam when' list. Here's some entries to start it off.

• You actually know what a Bukkake is.
• You personally block all e-mail originating from a Nigerian IP address.
• You start to see 'Enlarge your vagina' mails (think about it...).
• You have friends over and you check your e-mail, and up pops a spam-mail with a pornographic image. You think nothing of it and are not in any way embarassed because you're certain they know it's just the spam.
• You actually bought something from a spammer.

This should be a startr. Could anyone else please add something to the list?

Re:Only 19000 spam messages?? That's nothing.

[joshuao3]

Read the rest of the posts... many people have indicated they get a lot more spam than Raymond Chen.

I generally never presume that my experiences in this world are exceptional one way or the other. I always presume that I am average unless told otherwise.

Mailinator

[R.Mo_Robert]

Have you head of Mailinator?

Basically, you can make up any e-mail address, say foobar2004@mailinator.com and go and check it later. All you have to do is type in your chosen name and check for mail. It's useful for websites you don't really trust (but not for those you might continually receive useful mail from). And, of course, it's incredibly unsuitable for any personal information, since anyone can check any "account" if they can guess its name. And e-mails only stay for a certain number of hours/days. But for quick signups that just require some sort of e-mail address, it works.

School accounts off limit? Hah.

[Daniel Ellard]

I wonder if school email accounts are still off limits.

More likely your school has a kick-ass spam filter or something like that. My school account got hundreds of spams a day, and my classmates seemed to think that was about average.

I'm waiting until Monday

[appleLaserWriter]

When we will find out that Raymond Chen has been fired for blogging about internal Microsoft SPAM statistics.

Re:I'm waiting until Monday

[M.C. Hampster]

When we will find out that Raymond Chen has been fired for blogging about internal Microsoft SPAM statistics

Heh, you obviously don't know who Raymond Chen is.

Re:I'm waiting until Monday

[Anonymous Coward]

Heh, you obviously don't know who Raymond Chen is.

You, Sir, obviously don't know what a joke is.

Re:Only 19000 spam messages?? That's nothing.

[wuice]

I'd say you're probably the abnormal case here. That's a lot.

2002 was a big year for spam

[Basehart]

2002 must be the year when Florida got connected to the internet.

Obvious by it's Absence

[Anonymous Coward]

I want to know what was going on around the last month of 2003. There is a vertical bar of greatly reduced traffic at the end 0f 2003. Traffic begins to pick up again but it's not near as great at 2003.

zero

[zogger]

none, nada, last several months. Not sure how I pulled it off other than not using my email address with only a few people and trusted companies, but I haven't received one in I think 6 months now. Haven't even had any to "train" my moz mail with.

I used to use email all the time and was cavalier about posting it, contacts in newsgroups or forums, etc, last I remember I was approaching a couple hundred spams a day with that insecure technique. Since then got a different ISP and new email addy, and been real particular who I give the addy to, and it seems to have worked. I miss using email *more*, but the time (and aggravation) I save makes up for it for me.

I don't think so

[GCP]

The only ways people get 500 per day must be in their own stupidity.

I probably get 500 spams a day, but I don't think it's because I'm stupid.

I have an email address (MyFullName@MyCompanyName.com) that I've been using for well over a decade for a personal business. I don't plan to change either my name or my company name.

When I would be a speaker at some event or teach a seminar, the organizers would always include my email address as part of the speaker bio, which started going up on the Web when the Web was born. Also, in the early 90s, when the problem of spam was trivial, my address was mentioned in the industry press from time to time.

I can use an alternate address for close personal friends, but I have a lot of professional and personal inertia behind this very basic email address, so I can't stop using it. Friends I haven't heard from for years wouldn't know any other address, and I continue to get new business through that address.

I get more spam at this address, which I don't think has been publicly posted for six or seven years, than at my Hotmail spam magnet address, which I use roughly once a week for public postings, online product ordering, services that require a valid email address in order to register, etc.

It appears that the age of an email address matters even more than how much it is currently being exposed.

I go thru SpamAssassin on the server, and then thru SpamBayes on my laptop, and still about five spams get all the way thru to "ham". That amount is tolerable, but I also end up with an "Unsure" directory [SpamBayes sorts into three categories: spam, unsure, and ham] containing about 50 per day that I have to look through because occasionally I find an important *real* email in the "Unsure" pile.

Without SpamBayes, I'd be in big trouble. Some of us just can't change our email address frequently or even hide it very well, and that's not the same as being stupid...I think. ;-)

Cry Me a River

[VB]

I want to know why this guy has only received 3,500 spams since 1997?

1-800-WAA-AAAH!

Cheez!

My own plots....

[menscher]

About 6 months ago, I decided to disprove the claim some people were making about spam increasing exponentially. So I started on a project of plotting my personal spam over the past few years. I was rather disturbed to discover the exponential fit was better than the quadratic fit. Since then, it's tapered off, but you might still check out the plot. Also, I started plotting spam and viruses system-wide. Lots more plots are available (though only for a few months history, rather than years).

Re:My own plots....

[Sch0pehauer]

You seam to have a rather new mail account. As for my spam stats the spam grow seam really stochastic.
I was wondering if someone is interested in gathering spam stats through XML-RPC or similar. I know that SpamHaus gives some general statistics of the number of spam, but they are not accurate and differentiated between the different types of people (people that read UseNET and have a website receive more spam).

Re:My own plots....

[menscher]

I'd argue that you don't have a long enough history to be making predictions. Your stats begin in April. You have about 5 months of data, and, as seen in my own plots, spam has actually leveled off in those 5 months.

Also, I notice you're plotting daily stats, so of course you're going to see a huge randomness to it. Try binning your stats a week at a time (to improve the statistics and eliminate the "weekly cycle") and you may find it easier to pick out a trend.

It's good to see that spam plots are becoming more common, though. If the large ISPs did this, we'd have a lot more weight to use in convincing politicians that the problem is real.

Re:My own plots....

[menscher]

You seam to have a rather new mail account.

Not sure why you say that. The account was in use 5.5 years before the analysis started. And it gets mail forwarded from my previous account. I don't see how 5+ years can be considered "new", considering how the internet has changed during that time.

Re:Only 19000 spam messages?? That's nothing.

[Vicsun]

I get 2-3 spam messages per week on my most active mail account, but then again I bring new levels of paranioa to the playing field when giving out my e-mail. I wonder, am I alone in receiving so little spam?

Re:Only 19000 spam messages?? That's nothing.

[GileadGreene]

Anecdotal evidence doesn't exactly give you good statistical information. I average about one spam every few weeks (across several different email addresses). But it didn't even cross my mind to comment on this (until I saw your post) because the fact that I get far less spam than Chen just didn't seem like that big a deal.

Re:Single worst spam day by number of messages: Au

[gordgekko]

According to the stats my spam filter keeps, I receive an average of 92 messages a day, 71 per cent (or about 65) of which are spam. I'm rather surprised I receive so "few" considering my email address is listed on about 6,000 pages on the web.

I thank God for Bayesian filtering every day, I usually only see 1 or two spam every few days.

Re:I just don't understand

[Xilman]

Theres that AC post a few messages ago that says Mr. Chen shows up in the Linux Credits. Is it possible he's using a linux box in richmond.

Do you seriously believe that there are no Linux boxes in Redmond? If so, I suggest you wake up and start paying attention.

Paul

Only 67 spams on his *worst* day? Wow!

[sakeneko]

I think it was before 2000 that I last had that few spams in a day. <wry grin> That's what happens when you have an old email address and like to post to Usenet....

Re:Only 19000 spam messages?? That's nothing.

[Anonymous Coward]

In all honesty, I've received perhaps half a dozen spam messages in the 15 years I've had an email address. I know spam's a problem because I'm a sysadmin and see the shit that goes through the mail servers but none-the-less, I find it amazing that I personally don't get bothered by it. Seriously, I would love to know why. I don't have any spam filters between my account and the world at large so the only thing I can think of is; I'm careful who I give my address to. Now there's a thing. Old fashioned paranoia, nurtured before the days when spam even existed, saving my sanity and my mailbox.

I think there's a lesson for everyone here, people. Be more like me and the world would be a much nicer place.

Engineers. Gotta Love 'Em

[reallocate]

>>"...saved every spam message and virus-laden e-mail he's received at work since 1997."

O-o-kay. Step away from the keyboard.

19000?

[jedrek]

roughly 19,000 messages [...] 3500 messages

Since 1997?

I've gotten 16000 spams and viruses since *APRIL*. That doesn't count the accounts I've cut off because I was getting nothing but spam.

Re:19000?

[iamcf13]

I've gotten 16000 spams and viruses since *APRIL*. That doesn't count the accounts I've cut off because I was getting nothing but spam.

Then you can delete it all automatically either before or after it gets to your email inbox like I do.

Please keep the above in mind while other antispam solutions get coverage on Slashdot and this post is (likely) moderated into oblivion for being an 'ad' and 'just like SpamAssassin'--I tried to offer a clearly effective antispam/antimalware solution to all interested parties....

Only 500 spam messages?

[MMaestro]

If you've read some of the other comments made on their number of spam messages its hard to take anyone's claim seriously. Some people are reporting getting just one type of virus an average of 1 e-mail every 5 minutes. 288 copies of the same virus in a day? Possible, but doubtful. Others are even claiming a having months where their spams/viruses would reach the 1 gig mark. Who can believe some claims online with numbers like that?

"Quiet period" in early 2004

[hereisnowhy]

Can anyone explain the drop-off in early 2004? The funny thing is, the same thing happened to me -- my spam-ridden hotmail account received next to no spam for a period of four or five weeks.

Re:"Quiet period" in early 2004

[Graspee_Leemoor]

"my spam-ridden hotmail account received next to no spam for a period of four or five weeks."

You know how credit cards, mortgages etc. have those "payment holidays" where you don't have to pay that month? Well this is the same thing for spam.

In the case of credit cards and so on it makes more interest for the company. Maybe with the spam they're trying for that psychological trick of stopping the pattern for a while then starting it again, because when you get spam continuously your brain filters it out.

Then again, maybe I'm talking bollocks.

graspee

Re:I just don't understand

[Nyder]

targo says after a cheap shot at Slashdot, "...you actually have to open and run the attachment yourself in Outlook in order for it to do anything..."

That used to be the case, then those clever people figured out how to do it so you don't have to click on anything.

evil, evil world...

500 per day?

[Sivaram_Velauthapill]

500 per day? You must be one popular fellow ;)

(As an aside, the article on Raymond's site says that this is the e-mail he receives after it passes through the corporate filters).

Re:500 per day?

[joshuao3]

The first respondant was kind enough to point this out as well--it is indeed a valid point.

Since that is the case, would the graphs presented by Raymond not represent the rate at which spam methods evolved and the successive onslaught of new spams using that method, rather than the volume of spam present on the web for a particular day?

A spike on the graph would indicate that the filter failed to keep up with the changing spectrum of spam, and would only partially reflect a volume increase... as the volume increase is only asit relates to the new spam method not picked up by the filter.

The hole in the graph, which he contributed to the CAN-SPAN movement, may instead be a result of a lull in the changes to span techniques.

Thoughts?

Re:500 per day?

[Sivaram_Velauthapill]

I personally don't think the charts are very helpful. Showing size vs time isn't that benefitial. I think Raymond should have shown a histogram or the frequency (vs day/time) or something like that.

I agree with most of your observations. For instance, the gap in the graph could indeed be a change in technique more than anything else.

I also think the graphs are misleading and are simply measuring the effectiveness of Microsoft's corporate filters rather than actual spam. Since we are not looking at the original spam data (i.e. we are only looking at what went through filter), the main point to draw from the graphs is whether the spammers are getting mroe effective in defeating corporate filters.

So, I agree with most of what you are saying. I have the same feelings...

Graph could have been better

[Sivaram_Velauthapill]

I think the graph isn't too helpful. Size vs time may be interesting to look at but it doesn't really say much. I think a more useful plot would be a frequency chart or a histogram or something like that.

I'm not dissing the work--just saying how it could have been better...

Re:Single worst spam day by number of messages: Au

[Deorus]

I just posted a few hints. You must be able to work them out and pay more attention to little things you find more often in spam than in regular messages.

I will not post my header/body_checks files here since they are far from being a general miraculous solution to the spam problem. They just work for me and the people who send me mail.

What I wanted to say with my previous post is that under specific conditions there are ways of dealing with the problem fairly well, and I am fortunately under such conditions.

As someone who likes us-ascii encoded text messages, I could, for example, add a "/content-type:\stext\/html/ REJECT Go away! Find someone else to spam!" rule to my header_checks file and get rid of all HTML crap (which would therefore block most spam). It wouldn't harm me much because the mailing lists I subscribe are all text-based, but would be too limited for regular users. Obviously I am not using such a rule since some people forward me HTML formatted junk mail (which I read) sometimes. Similarly I have rules which block messages with specific attributes in <img> and <table> tags, imagemaps, <form> tags, etc. These rules are good for me since they still let junk mail in but scrap most spam. Although such rules would probably block your favourite newsletter, so they would not apply to you.

Learn to observe your messages and find common things in the spam you receive which generally do not appear in your regular messages and try to figure what's better for your mailing needs.

Give me a break on how good this guy seems

[Anonymous Coward]

From his page... "This particular email address has been inactive since 1995; all the mail it gets is therefore from harvesting done prior to 1995."

Emails are re-harvested from existing lists and re-sold and reused every minute of the day...the mail to that address is no reflection of date-limited harvesting. The address is still in circulation, regardless of when the owner thinks it went offline. I'd like to see that comment removed from the page, please...it can prompt readers to distrust the entire article.

Re:Give me a break on how good this guy seems

[TomServo]

However, some of the more scrupulous (did I just say that?) spammers will remove emails that don't seem to ever open those little tracking images and the like before selling them off. Old email addresses *do* eventually lessen their spam load, but they'll never lose it.

In the end, I think his point is that that email is probably less likely to have been picked up by all the more recent spam-related worms and the like, so it shouldn't be viewed in the same light as currently used addresses that potential virus victims might have in their address book. As is, I get many, many bounces sent to my (now retired) work e-mail because business contacts, with that email in their address book, got infected by the multitude of spammer worms.

Spam vigilante

[Anonymous Coward]

If I had the money (I don't), I would pay for a professional hit on a few of the most notorious spammers. I'm not kidding.

I would pay big money for an experienced and expert hitman, to do the job carefully, patiently and thoroughly.

Once a couple of the well-known spammers were iced, I think we would see a serious decline in spam.

I don't fell all vigilante about other, more serious crimes. I don't think violence solves anything. I oppose the death penalty. I know this is an irrational position, but I don't care.

Does this make me a bad person?

Re:Should we rewrite SMTP-No, FILTER THE CRAP OUT!

[iamcf13]

I think this problem will just escalate for as long as we have SMTP in use. So maybe SMTP as a protocol needs a rehaul, or a revision to rewrite it completely (and call it something different). I think it wouldn't be impossible to pull off.

Waste of time.

Every month someone suggests that there's a technological solution to this problem. But there isn't. This isn't a tech problem. It's a law-enforcement/sociological problem.

You can only go so far technologically as long as spammers are allowed to compromise peoples' computers and use them for improper activities.

My approach addresses these issues.

Spammers continue to spam and hijack other people's computers via email to relay their trash and law enforcement does little or nothing unless you are 'big' enough for them to take an interest in your plight.

As an individual email user, I finally got so angry and tired of all the spam, phish attempts, scams, and malware I got by email, I did something about them that made them all go away for good! The only time I ever get spam and (rendered inert) malware now via email is when I have to 'lower my defenses' temporarily for a good reason.

I am trying to share my solution with others here on Slashdot but I am constantly accused of 'advertising' and 'reinventing SpamAssassin'. Yet aren't all the stories posted on Slashdot advertising in the end? The public at large freqenting Slashdot is being told about information, products, and services found noteworthy by the Slashdot editorial staff whether there is a pricetag attached to said information, products, and services or not.

Until the day comes when spammers and scammers finally give up sending out their trash via email (probably never), I will continue to automatically delete it 'on sight' using my approach. Maybe one day, I'll be able to have use of the email mailserver I wrote and then I'll never have to waste time downloading and subsequently deleting this unwanted crap ever again!...

Re:Only 19000 spam messages?? That's nothing.

[Anonymous Coward]

Read the rest of the posts... many people have indicated they get a lot more spam than Raymond Chen.

This is hardly scientific. Aside from the vagueness of the "many people" part, there's obviously going to be huge selection bias here -- people who get similar levels of spam to Raymond aren't going to post about the fact.

Incidentally, I get maybe 1/10th as much as Raymond does.

My SPAM vs valid e-mail feb-sep 2004

[Anonymous Coward]

I collected all spam vs valid email from Feb 2 through Oct 31 2004. The account was my work account at the University where the firewall is supplied by the University. The browser and email is Mozilla. All e-mail is delivered. I recorded approximately 2100 spam emails and about 1700 valid emails. No attempt was made to chart by date. When I started it seemed my SPAM was about twice the good mail, but that turned out to be wrong.
At home, I work behind ZoneAlarm. Both locations use up to date antivirus, and both remove cookies at the end of the session. At home I do not have to log in to companies to get data or to order parts. Apparently, being security conscience at home pays off.

Re:I just don't understand

[iamcf13]

targo says after a cheap shot at Slashdot, "...you actually have to open and run the attachment yourself in Outlook in order for it to do anything..."

That used to be the case, then those clever people figured out how to do it so you don't have to click on anything.

evil, evil world...

For maximum safety if you must use Outlook for email is to rename or carefully delete the Windows Scripting Host Program.

As an alternative, you could use my approach to email which is unaffected by any kind of scripting exploit.

Please keep the above in mind while other antispam solutions get coverage on Slashdot and this post is (likely) moderated into oblivion for being an 'ad' and 'just like SpamAssassin'--I tried to offer a clearly effective antispam/antimalware solution to all interested parties....

I CANT BELIEVE YOU WASTED A MOD POINT!!!

[shrewmy]

You modded me troll when theres all these other post more deserving of a Redunant!!! I remember a time when I used to post at +2, those were the days :(

Spam archive

[Richard W.M. Jones]

It's down at the moment (too many people tried to download the whole thing for their Baysian filters or whatever), but I've collected all my spam since Aug 1997 here.

Internet Archive version.

Rich.

Problem with deleting on servers

[nurb432]

My problem with doing this, is that i often get things tagged from maililng lists as spam..

Only my whitelist ( which runs before the spam filter ) saves me on this..

OK, I'll go first

[Slinky Saves the Wor]

Sure. In the meanwhile, why don't you have a look at how X.400 mail was done, for some perspective. At the protocol level, SMTP works but only if everyone plays nice, I'm sorry to say. The protocol state machine is also too complex, it could be much simpler: 1. here's the recipient, 2. here's the mail. The server could disconnect the sender in either 1 or 2. Sender and other stuff is matter of the message representation (if you need signatures to prove the identity, or what ever).

HELO/EHLO is a hack in SMTP. It works, but it's a hack nevertheless.

I do think there's a technological solution to spam. Spam is not a social problem. If you take away the means to send meaningless unwanted mails, there will be no spam.

So you just have to make mass-spamming impossible. And do it in the receiving end, so that the first hop is where the unwanted mail stops, when the unwanted mail goes out from a spammer ISP, a zombie machine, whatever. For this, we could utilize systems which are based on brute-forcing a certain space of a one-way function when receiving the mail (like hashcash).

Legal bulk emailers should of course be re-thought too. Perhaps we could use a RSS feed leecher at the ISP (clueful people could of course run their own RSS feed checkers), which would then deliver to their customers who are subscribed to some feed. Something like the Usenet News, but a more modern one. Offer a web interface for users to subscribe to whatever place.

So.. in other words, bulk emailing is really useless for anything. So replace it with methods which disallow spam. Sorry, but it can be done way better with different methods (like RSS). Use the hashcashed email (whatever kind of email system it's based on) only for private correspondence (or with just a few recipients).

Here's just some ideas from the top of my head.

Doesn't work

[Slinky Saves the Wor]

How is this any form of improvement? Penalize everyone on the planet because of spammers? Force an entire worldwide network systems upgrade? Slow down mail service exponentially?

How many times do you send more than 100 mails per day? How many times do you send more than 5 mails per minute? A normal user doesn't. And those who legitimately do, are so few that a new kind of system could be worked out for them.

Make it impossible to send large numbers of mail. That's a solution which works. Systems upgrade, yes, since SMTP is broken and it cannot be fixed. I also argue that it shouldn't be fixed with some hack. Rewrite it to be better!

Mail servers should be "licensed" to operate on the Internet

This doesn't work. Think zombie machines in some ISP's network.. Windoze machines which the ISP considers trusted, most likely, since it's their customers we're talking about. The mail server is licensed, all right, but the zombie client can pump out a million messages through that licensed server.

Whitelists and blacklists just don't work. Then when you end up blacklisting an entire ISP block due to the aforementioned problem, there will be no mail service for others in that ISP block who attempt to mail to a place which blocks that ISP.

So no, white/blacklisting is not a solution. It helps, but it's not a solution.

You see, the problem with spamming is that the spammers do not follow the system: they'll break into a licensed SMTP box if need be. To beat spam, you just have to make it (physically) impossible to send large numbers of mail messages. It's that simple.

Re:Doesn't work

[mabu]

Mail servers should be "licensed" to operate on the Internet

This doesn't work. Think zombie machines in some ISP's network.. Windoze machines which the ISP considers trusted, most likely, since it's their customers we're talking about. The mail server is licensed, all right, but the zombie client can pump out a million messages through that licensed server.

It does work. Like you said earlier, smart relays should trigger an alarm if any single client starts to send out too much mail, but that should be a voluntary (responsible) practice adopted by ISPs who handle large user bases. It should NOT be part of some goofy new protocol standard which requires everyone on the planet to upgrade.

If the ISP can't control their internal clients, then they deserve to lose their SMTP license.

This is the same topology that's threatening the more irresponsible ISPs. If they don't start controlling port 25 traffic on their DULs, their whole IP space gets blacklisted... as it should be, until they start losing business because nobody wants to take their smtp traffic.

You see, the problem with spamming is that the spammers do not follow the system: they'll break into a licensed SMTP box if need be. To beat spam, you just have to make it (physically) impossible to send large numbers of mail messages. It's that simple.

You obviously don't have much experience in this area.

Spammers aren't breaking into known legit relays any more. Because those relays are locked solid and monitored. They're very hard to control.

Besides, let spammers break into inner networks and create zombies through legit SMTP relays. I GUARANTEE you in such cases the ISP will act A LOT FASTER to fix the problem, lest they get their main relay blacklisted. As it stands, they don't give a crap about zombies polluting the internet autonomously.

Wake up. It's not 1995 any more.

Re:Doesn't work

[Slinky Saves the Wor]

If the ISP can't control their internal clients, then they deserve to lose their SMTP license.

How the hell do you expect some ISP to control what's being run and downloaded in some Windows box of a home user who has no clue of security? It's impossible. The ISPs can't even keep each Windows box in their network up-to-date with security patches! So it's just not going to happen. The ISP can shut the box down, but that is after the damage has been done.

You obviously don't have much experience in this area.

Maybe so. I've once implemented a minimal RFC2821/2822 SMTP implementation (with some extensions and MIME supported as well) for a protocol converter. I became quite familiar with the specification. From that experience, it is my opinion that SMTP is fundamentally broken especially with regard to the spam problem.

I would like to ask you, how did you feel when Telnet was replaced with SSH? That required phasing out a (security-wise) broken protocol with something that works a whole lot better. You could have insisted on policies which say "thou shalt not eavesdrop" but that clearly doesn't work. A better solution is to just make it technically/computationally impossible (or at least as hard as possible) to eavesdrop.

Do you see from the previous example the difference between "thou shalt not send spam" vs. "thou cannot send spam"?

Always think 10 years to the future.

Re:Doesn't work

[mabu]

I would like to ask you, how did you feel when Telnet was replaced with SSH? That required phasing out a (security-wise) broken protocol with something that works a whole lot better. You could have insisted on policies which say "thou shalt not eavesdrop" but that clearly doesn't work. A better solution is to just make it technically/computationally impossible (or at least as hard as possible) to eavesdrop.

Telnet was not replaced with SSH. That's an invalid analogy. SSH was an *alternative* to Telnet that was an improvement. It didn't force anyone else to abandon telnet as a protocol; it didn't require global standardization. This was only relevant between two specific hosts that wanted to connect to each other - it's not basically a public-serving protocol like smtp or http.

I choose SSH over Telnet because I completely control who I want to have shell access on my server. This is different from SMTP which is a global protocol that continuously accepts connections from unknown hosts.

Do you see from the previous example the difference between "thou shalt not send spam" vs. "thou cannot send spam"?

You're living in a dream world if you ever think you can completely stop people from sending spam.

What do you hope to accomplish by redefining the protocol? What?? Enforcing accuracy in e-mail header/source information? What good will that do? Do you think spammers aren't capable of operating from legit hosts and have reverse DNS control? Of course they can. So then what? You have your fancy-schmancy new protocol and the spammers are using it, and they're just as legit as anyone else. So how does this change anything? Oh, you say, if they can't forge header information then you can stop them? How? Oh wait.. by blacklisting them. Well, that's what we already do and we don't need any stupid rewrite to the protocol!

It's not about "thou cannot send spam". It's about: "thou can send spam, but not to my system *click*"

That's why RBLs are the ONLY way to go. Your idea is counterproductive, wastes exponentially more resources, consumes more bandwidth, costs more money, requires more manpower, slows down mail service, slows down all internet-based services, requires an unrealistic mass change in system standardization, and in the end, STILL WON'T MAKE ANY DIFFERENCE.

The only real change in protocol that will work is one which checks the validity of the remote relay against a list of acceptable/unacceptable hosts - everything else can be subverted.

Re:Doesn't work

[Slinky Saves the Wor]

Take a deep breath and this time please READ at least the following three paragraphs before answering, since what you answered to was definitely not the reasons why I consider SMTP to be obsolete.

1. By redefining the protocol I want the protocol to be simpler, and utilize a hashcash-like system in its very core. (If you don't know what hashcash is, Google it up now or read paragraph 2)

2. By using a hashcash-like computational puzzles, it just is PHYSICALLY IMPOSSIBLE for anyone to send large numbers of mails. (The machine which is to receive the message gives a computational puzzle for the client to solve, and will accept the message only after the result has been verified. This can be done in such a way that it WILL require e.g. 7 seconds of calculation per mail. And it cannot be bypassed. For instance the client has to bruteforce a collision for an n-bit hash function, given m bits, where m < n (m chosen suitably to make the computation of desired length).)

3. Why RBLs are not the "ONLY way to go": RBL systems break down the moment someone 0wns the machine and uses that zombie box to send the mails. You cannot avoid this. It's a reality that systems are insecure. If you allow large numbers of mails to be sent by any machine, you will have spam. Also, RBLs are not fair, because you will end up blocking entire networks just because there was one compromised machine as the origin of spam.

Of course you could retrofit SMTP to use hashcash-like systems, and those exist already, but I bet it would be simpler just to rewrite the mess and make it more modern in the first place.

The basic idea, however, was not to dick around with the SMTP header validity, as you misunderstood my intentions, but rather to make it physically and computationally and algorithmically impossible to send mass mails. Period. Do you now see the difference between "thou shalt not send spam" vs. "thou cannot send spam"?

Adding headers which can be trusted (through cryptographical means) is one aspect through how SMTP and its message representation could also be modernized.

The only real change in protocol that will work is one which checks the validity of the remote relay against a list of acceptable/unacceptable hosts - everything else can be subverted.

If you answer, please give an answer how a hashcash system can be subverted. I'm sure many people would like to know.

Re:Doesn't work

[mabu]

By using a hashcash-like computational puzzles, it just is PHYSICALLY IMPOSSIBLE for anyone to send large numbers of mails. (The machine which is to receive the message gives a computational puzzle for the client to solve, and will accept the message only after the result has been verified. This can be done in such a way that it WILL require e.g. 7 seconds of calculation per mail. And it cannot be bypassed. For instance the client has to bruteforce a collision for an n-bit hash function, given m bits, where m n (m chosen suitably to make the computation of desired length).

Like I said before... that's the most ridiculous, wasteful idea I've ever heard. Let's make e-mail rival snail mail in terms of delivery efficiency as a solution to solving spam?

One of the great values of e-mail is that it travels at the speed of light. To deliberately slow this down is offensive and stupid IMO. That's like creating a web site that can only handle one visitor every seven seconds. What's the point? Why even bother?

I got an idea for you that uses the same analogy. If you're worried about burglars breaking into your house, why don't you BURN IT DOWN? That way there won't be anything to break into. It's as good an ideas your computational hash challenge smtp circle jerk protocol.

3. Why RBLs are not the "ONLY way to go": RBL systems break down the moment someone 0wns the machine and uses that zombie box to send the mails. You cannot avoid this. It's a reality that systems are insecure.

Like you said, all systems are ultimately insecure, so your system doesn't solve the problem either - it just puts the entire Internet e-mail service on slo-mo... yes, you get less spam, but you also get less legitimate mail - and ultimately the preportion to spam-to-legitimate mail is exactly the same. Zero Sum Gain with the added bonus of almost completely destroying the usefulness of the e-mail system in the process. Congrats!

If you answer, please give an answer how a hashcash system can be subverted. I'm sure many people would like to know.

Do I even need to explain this to you? Like you said, every system can be subverted.

In any case, your idea only strenghens the value of spammers employing massive armies of zombie PCs which would send a spam e-mail at whatever interval the new protocol allowed.

In fact, your idea might actually increase the spam-to-legitimate mail ratio, because the spammers would adapt to the nature of the boneheaded protocol whereas end users would just arbitrarily send mail without thought as to the limitations of the protocol.

Think man. Why would you propose such a ridiculous standard that makes everyone suffer and slows down all mail service? It could just as easily be subverted... You're basing the value of your idea on the erroneous premise that we have small numbers of relays sending out massive amounts of spam - that's not the case any more - now we have wide arrays of relays bursting little bits of spam at programmed intervals... your idea would only make things worse. I'm curious, are you the author of the CAN-SPAM act?

Re:Doesn't work

[Slinky Saves the Wor]

That's like creating a web site that can only handle one visitor every seven seconds. What's the point? Why even bother?

If you have a web server running on a Gameboy with 10 bits per second bandwidth, you might want to do this. In other words, if the bandwidth resource or server resource is very scarce, you might want to limit the usage.

Also, I don't see how the mail service would be slowed down beyond usage. People poll their mail (POP/IMAP/web interface/whatever) with intervals being in minutes, so as not to place a load on the servers. If the entire chain of mail coming would take one minute more time, it would not make any difference whatsoever to the end user.

yes, you get less spam, but you also get less legitimate mail

No, no, no, no! Less spam, yes. But the legitimate mail would not be affected that much. Why? BECAUSE a normal person does NOT send 100s of mails per hour. They send maybe 2 or 3 (amortized over the duration of the day). For them the new kind of system wouldn't make any differences.

Of course going through relays would be problematic, but afaik some Israeli cryptographer (Biryukov? Can't remember) had a solution to exchangeable hashcash. Unfortunately I don't remember the details, but I think I saw it as a preprint on IACR. Anyway it would be the best to connect to the server nearest to the recipient, and skip the relays altogether.

I'm curious, are you the author of the CAN-SPAM act?

No, I am not. Do you have some vested interest in allowing spamming to continue or are you afraid of losing your job as an SMTP box admin, or what's the issue here? You seem to be very temperamentic about this discussion.

I got an idea for you that uses the same analogy. If you're worried about burglars breaking into your house, why don't you BURN IT DOWN? That way there won't be anything to break into. It's as good an ideas your computational hash challenge smtp circle jerk protocol.

That's not the same analogy. The analogy would be about taking the stuff worth stealing somewhere else (such as a bank vault), so that there would be nothing to steal. Of course burning the house down would work too, but only an idiot would even think of such a thing.

Re:Doesn't work

[mabu]

No, no, no, no! Less spam, yes. But the legitimate mail would not be affected that much. Why? BECAUSE a normal person does NOT send 100s of mails per hour. They send maybe 2 or 3 (amortized over the duration of the day). For them the new kind of system wouldn't make any differences.

Yes, a normal person doesn't send 100s of mail an hour, BUT a normal mail server DOES handle 100s of legitimate mail an hour. And if you impose a stupid computational hash routine on each connection, you cut down on the efficiency of the mail server exponentially.

The whole concept is completely ridiculous and you have not demonstrated in the slightest that this would actually reduce spam any more than it would definitely cut down on legitimate e-mail efficiency.

I assumed you knew this but maybe you don't.

Servers have limitations on how many concurrent clients they can serve. These limits vary but every server and ever service, be in web or mail, has a threshold. This determines how fast and efficient information can be delivered. The most critical factor in performance is the speed at which the transaction can be concluded. If you impose a deliberate delay to this process that is ten thousand times greater than what it normally takes to complete, you effectively cripple the performance across-the-board of the service, and create a virtual "traffic jam" of pending requests - basically a "slashdot effect" on a mail server - and it won't distinguish between legit and spam mail.

Here's another example. Let's say your protocol is in force. You refuse to accept mail from spammers, but they have a large zombie army hitting your mail server from 50 different IPs at the same time. Under the old system, if your server only allowed 20 concurrent connections, you'd be unavailable for a second. Under your system, this zombie army, even though they might not be able to send spam, would still tie up the mail server and create a denial-of-service condition.

Your plan has no way of ever working.

I work in the trenches every day with this stuff. I've tried every solution you can imagine. I know what I'm talking about here. There's no way in hell I'd cripple my mail server on the goofball premise that it might reduce spam, when I know it won't.

Abandon this idea or else continue to lose credibility. There is NO WAY it would ever work.

Re:Doesn't work

[Slinky Saves the Wor]

Yes, a normal person doesn't send 100s of mail an hour, BUT a normal mail server DOES handle 100s of legitimate mail an hour. And if you impose a stupid computational hash routine on each connection, you cut down on the efficiency of the mail server exponentially.

Yes, relaying would be problematic, and that's why the protocol itself would have to undergo some changes. Or the "exchangeable hashcash" could be utilized as a Proof-of-Work in the receiving end (still can't remember the author!!).

But you did realize that all the calculation is done on the client end? Not in the server end. The server does not bruteforce anything. The server just makes a large random number r, runs MD5 (or what ever one-way hash function) to get H, sends H to client along with some bits of the r and tells the client to figure out the original r.

For example, using 4 bits, the server creates 0011, H(0011)=111. The server sends 111 and the last bit 1 to client. The client then knows H(???1) = 111. Then it can go through all those question marks in order and see which bits fit. There are no shortcuts to finding that in a proper hash function, because you cannot invert a one-way hash function. (MD5 was maybe not so good an example but it's perhaps the most widely known so I used it). The example used 4 and 3 bits, in reality it would be like 128 and 90 bits, perhaps.

The client can (and should!) work off-line, it does not have to be connected to the server while it does the computation.

When the client connects back, the server can verify that r,H was actually sent to the client at some time in the past (e.g. by looking at a random session id and verifying r, H in its database), and that MD5(r) == H in an instant.

If it's OK, the mail is accepted. Otherwise, sorry no bonus.

So, in the new system as those zombies connect, the server is unaffected, it can process 20 connections simultaneously easily. The server just makes the puzzle and gives it to the connecting client, then kills the connection. But the zombies end up being tied up each for a short duration of time as they build the hashcash, i.e. bruteforce the hash. So the system is not about tarpitting the spammer by keeping the connection open for 7 seconds (or whatever duration).

Only if they provide properly the correct r do they have any chance of getting the mail through.

For a normal person sending mails not too often, 7 seconds of wait (or whatever the length of time is) is acceptable.

We have had a good discussion here! That is nice.

Re:Doesn't work

[mabu]

But you did realize that all the calculation is done on the client end? Not in the server end. The server does not bruteforce anything. The server just makes a large random number r, runs MD5 (or what ever one-way hash function) to get H, sends H to client along with some bits of the r and tells the client to figure out the original r.

Whoever does the cpu work is irrelevant. It would be reasonable to assume that during the process of the client calculating the proper response, a socket would remain open to the server, thus wasting precious resources. This in quantity creates a denial-of-service condition. But then you address this...

The client can (and should!) work off-line, it does not have to be connected to the server while it does the computation.

And in the stateless environment of the Internet, how does the server allow the client to reconnect and validate itself? I guess the server has the burden of maintaining an elaborate database of IP state information... creating yet another drain of resources and an additional point of attack and method by which the server can be brought to its knees.

Beyond this, you're proposing that a single mail transaction require twice as many TCP connections, which cuts performance in half and doubles peak resource requirements. In reality, such a transaction would require much more than double the server resources because the server now has the added burden of validating all inbound connections against some sort of state database.

I agree, it's interesting debating with you on this issue. I apologize if I was overly acerbic with my words, but seriously, I really think this idea is totally counterproductive and I have trouble understanding how anyone with experience in all aspects of the spam problem would not agree with me.

You need to understand some basic premises to the spam problem... what most people think of when they think of "spam" are just a bunch of junk e-mails in their inbox. The spam problem affects much more than this.. and a true anti-spam solution has to do more than just limit junk e-mail.

There are several casualties of the spamedemic - here are them listed from most to least significant:

1. Internet Bandwidth - first and foremost, this is the big problem that most client-side spam solutions ignore. It's conservatively estimated that more than half the bandwidth on the Internet is unsolicited crap traffic like spam. This slows down performance of all other net-based services.

Your idea appears to address the issue by creating a condition upon which mail is accepted (or rejected prior to any more bandwidth being wasted), but it compounds the problem by doubling the amount of TCP connections needed. Result: no advantage.

RBLs are THE most bandwidth-efficient method of stopping spammers wasting bandwidth PERIOD. Under your scheme a challenge-response is requested. Using RBLs, as soon as the server determines the IP is invalid, the connection is closed. It's many times more efficient.

2. System resources - ISPs and companies that run their own mail servers have to build systems that are capable of handling ten or more times the mail traffic they would normally expect, just to not interrupt legitimate mail flow. This is a huge burden on legitimate companies that is passed along to consumers that most people don't consider. The degree to which an ISP can effectively handle mail determines the efficiency, speed and value of the service. If you've ever used AOL, you know that sometimes it may take hours or days for mail to arrive. This is the result of anti-spam systems being bogged down.

Your idea doesn't address this critical issue. In fact, your idea compounds the problem by requiring more client and server side resources. In fact, I'd say your idea probably increases the necessary server-side resources by a factor of 20-100 or more just to maintain the status quo.

In the last five years, I've personally spent over $20,000 on reso

Re:Doesn't work

[Slinky Saves the Wor]

Whoever does the cpu work is irrelevant. It would be reasonable to assume that during the process of the client calculating the proper response, a socket would remain open to the server, thus wasting precious resources.

No, it is not quite irrelevant. The client has to do it - it has to be done by the one who is sending. It would also not be reasonable to keep the socket open due to the DoS possibility, as you mentioned.

And in the stateless environment of the Internet, how does the server allow the client to reconnect and validate itself? I guess the server has the burden of maintaining an elaborate database of IP state information... creating yet another drain of resources and an additional point of attack and method by which the server can be brought to its knees.

There is no reason to be overly complex, a database of IP state information would be like shooting a mosquito with a cluster bomb. A simple randomly created session id would suffice: the server gives it to the client, and the client then gives it back to the server (along with the response) once it has calculated things. That's the basic idea, some simple cryptography can make it more robust.

Beyond this, you're proposing that a single mail transaction require twice as many TCP connections, which cuts performance in half and doubles peak resource requirements.

No, I am not proposing such things. There would be two distinct connections, but the performance would not be halved.

First of all, the TCP connections are not simultaneous. Second, the first transaction is very simple, something along the lines of 500 bytes needs to be transmitted. This payload can easily fit inside one packet (of course depending on exact OSI layer 2 technology used). Third, the second transaction contains the actual message and it is larger, but it would be very easy to design a protocol which would, in overall, require less transmitted bytes and therefore bandwidth than SMTP uses for a mail transaction. It would perform at par, or even better, when measured with accepted transactions per second. (Of course the offline calculation of the challenge would make the overall transaction take more time).

The whole thing would go like: 1. connect to server, receive the challenge and session id. 2. Disconnect. 3. Calculate offline. 4. Connect back to server, send the responses and session id. There's two distinct connections in steps 1 and 4.

Connection setup in modern TCP/IP networks is not an issue, and does not really burden the server excessively. But in the normal SMTP, for instance the tribal dance of HELO/EHLO is a nonsensical remnant which wastes bytes needed to fully complete the transaction. HELO was useful in the time when all Internet trusted each other, when nobody played evil. But it is not so nowadays, and the HELO greeting cannot be trusted. So for instance clearly such a "HELO blah", "220 Nice to see you" thing would not be needed in a new kind of protocol.

I agree that RBL is more bandwidth efficient if the recipient has been blacklisted, as the connection can just be dropped immediately and there will be few bytes transmitted. But RBL does not work well with zombies, since zombie machines can be anywhere.

I can assure you, the database would not be a bottleneck in this kind of system. Accessing a couple of values based on the session id needs just a simple hash table variant, can be done very fast and easy without the need of complex DBMS solutions.

As an end user, or someone who doesn't manage mission critical mail services, you may not be aware of some of these issues, but they indirectly affect you each and every day, in the general performance of the Internet; in the efficiency of your ISP's mail service, and more.

I have not done much system administration of a mail system, but in the past one consulting-like gig did involve me getting administrator access to the platform running the production mail systems in the biggest ISP of my country. From that