A Timeline Of Spam And Antispam

Haak writes "American Scientist has a fine article by Brian Hayes summing up the history of spam and proposed measures to deal with it." A shorter article along the same lines is running at The Economist.

Comments

Re:TIME...

[Anonymous Coward]

Spam, on the otherhand, is stoppable.

But, so many /. postings about spam recently... too much!

Interesting Perspective

[ankleteeth]

The article sums it up well, but is this something that is going to ever stop? SPAM to me seems like another one of those things in life like drug dealing for instance. Whatever tactice we take to stop or outlaw it, people are always going to find a way around it. The stronger we make our SPAM filters, the more normal desired mail that is going to get blocked. DOn't get me wrong, I hate Spammers, but I dont see how any of these solutions are going to work. Thats my opinion at least, but as the article says, I suppose suing spammers might have a good effect.

Re:Interesting Perspective

[lpontiac]

SPAM to me seems like another one of those things in life like drug dealing

I know it's not the main point you were trying to make, but I don't think this is a valid analogy. Drug dealing is a 'victimless' crime in which all parties concerned are consenting. Spamming is not.

Re:Interesting Perspective

[letxa2000]

The article sums it up well

And that's all it does. The article itself doesn't have any new information, insight, or anything to help in the process of eliminating spam. But I guess it's good reading for someone who hasn't had an email account for the last 9 years and is just now becoming interested in what happened to the spam-free email of 1994.

but is this something that is going to ever stop?

This article won't as it doesn't provide anything new. Paul Graham's articles of last year and this year are, to-date, the best real work published on effectively eliminating spam. I've implemented his "plan for spam" and I can say it works as advertised. I'm getting better than 99.5% of spam caught with my only false positives being when my corpus was pretty dang small.

This article says that the real test for Bayesian filtering will be when it becomes so widespread that the spammers need to come up with countermeasures. I agree this will be interesting, even though Paul Graham thinks he already has the answer (see "A Plan For Spam"--find the link yourself). And it seems that if spammers try to implement countermeasures against Bayesian filters, the results are going to be messages for which the countermeasures themselves are going to be easy to add to "charactertistics" in the Bayesian filter. If the spam of the future is "Check this out: http://www.spammersite.com" then you start doing things like calling "Messages where 50% or more of the body are HTTP links" a characteristic, etc. Or you look for the countermeasures and call that a charactertistic.

Bayesian is the answer to spam. Once you try it you won't go back. :)

Re:Interesting Perspective

[Steve B]

SPAM to me seems like another one of those things in life like drug dealing for instance. Whatever tactice we take to stop or outlaw it, people are always going to find a way around it.

There are two basic differences between spamming and drug dealing:

1. Spamming produces relatively modest profits for its most successful perpetrators. Major drug dealers routinely make millions of dollars a year. Thus, the former are far less likely than the latter to accept a prison term as an acceptable cost of doing business (particularly since the former are far less likely than the latter to have the sort of "muscle" contacts that will keep them above the bottom of the prison totem pole).

2. Drug dealing generally takes place between a willing seller and a willing buyer. Thus, neither party has an incentive to cooperate with the police. Spamming, on the other hand, takes place between a bandwidth thief and a bandwidth theft victim. The latter has every reason to cooperate with the police, if he is only convinced that the police will actually do something about the problem.
(This difference is what makes drug dealing more respectable than spamming.)

Re:Interesting Perspective

[sfe_software]

Bayesian is the answer to spam. Once you try it you won't go back. :)

I agree to an extent. I started using Bayesian classification since Mozilla 1.3a (I think) implemented it.

After a couple weeks of "training", it was dead-on accurate. Very little slipped through.

It's been a few months now, and it's gotten worse. Much of my spam seems to be one-liners like "Here's that URL we were looking for: ..." Others contain mis-spellings in common spam-related words, and slip by the filters.

Marking the ones that slip through as Junk causes more problems with false-positives. It's also too easy for a notification from a company that you *did* request, to get marked as spam.

In all, Bayesian is great, and far better than anything else I've ever seen -- but it isn't the holy grail unfortunately. Plus, it's fairly easy for a spammer to tweak his message against a relatively common corpus. I believe that most people would come to the same conclusions as to whether or not something was spam -- and thus an "average" corpus is trivial to create, and tweak your spam against.

I think it'll only get worse if Bayesian classification finds its way into more people's mailboxes. I'd almost prefer that it remain a "geek only" thing (though one could say the same about the Internet itself, and we wouldn't have this problem ;)

Re:Interesting Perspective

[fferreres]

Your drugs / spam analogy may seem to hold, but does not really. The economics of drugs is that if you ban it, but a lot of CONSUMER desperately want the product, then you'll STILL have drugs beign sold. Note I am not saying that if the drug dealer wants to sell it, they will sell it, the one that's putting the money, the consumer, will be the one rasing the offer so that he can have it.

Spam, on the other hand, only pleases the distributor (the spam services provider) and the producer (seller of the penis enlargement program). It does NOT pleases the consumer, they are not going to pay more for the penis enlargement program just because they seem to start receiving the email less often.

So, in the one hand, you have unstopable offending marketing, on the other hand you have a product that people will literaly kill so the can get doped. Stoping spam is much easier economically. Just make the spammers revenues harder. If you want to stop drug dealers, then burn colombia, bolivia and your corrupted agents (among other places / guys).

Re:Interesting Perspective

[letxa2000]

Agreed. Bayesian all by itself is not perfect. But Bayesian can do 95% of the work reliably, and a little extra filtering can take care of the rest.

I personally advocate Bayesian along with some simple keyword filters that contain mostly known spamvertised domains or spam sources. If it is kept up-to-date that helps.

It's been a few months now, and it's gotten worse. Much of my spam seems to be one-liners like "Here's that URL we were looking for: ..." Others contain mis-spellings in common spam-related words, and slip by the filters.

First, with a sufficiently large corpus the mis-spellings shouldn't slip through. The fact that they slip through means your Bayesian filter is still "learning." At some point, "VIAGRA" might be a 98% chance of spam but V1AGRA will essentially be a 100% chance of spam. The mis-spellings often make it easier to detect spam with confidence and the rest of the email should generally be enough to let Bayesian calculate a good spam percentage.

The one-liners can be caught by improving the Bayesian filter itself: Perhaps a new characteristic considered by Bayesian is "Is the message 1-line long?" or "Is the line 2-lines long?" or "Is more than 40% of the body of the message used to convey an HTTP address?" Things like this are valuable characteristics that will help Bayesian catch even 1-liners. Perhaps 90% of your 1-line messages that have an http reference in it are spam--that's something Bayesian can work with.

Marking the ones that slip through as Junk causes more problems with false-positives.

Really shouldn't.

Plus, it's fairly easy for a spammer to tweak his message against a relatively common corpus. I believe that most people would come to the same conclusions as to whether or not something was spam -- and thus an "average" corpus is trivial to create, and tweak your spam against.

While pretty much everyone will agree on what IS spam, not everyone will agree on what isn't--and that's what's great about Bayesian. Sure, they might avoid the word "Viagra" or "slut" but the headers themselves can be damning, the fact that they have 15 images being loaded off an external site is damning, and the fact that a message with a 60-character body consists of a 30-character HTTP address is also probably damning. They're not going to know I have a best-friend named Fred (which is something that will lower the spam score when it is found in my email). As Paul Graham said, if spammers have to stop using all the words (viagra, porn, slut, etc.) and techniques (images loaded from external servers) that they are using to make their pitch, they're going to be significantly limited in what they can say.

If it gets to a point where they totally mangle their emails with SMS-like substitutions to convey their message, you can also add new characteristics for Bayesian: "Are more than 40% of the tokens unknown?" "Are more than 50% of the tokens unknown?" You can assume that if you have a halfway-decent corpus and more than X% of the tokens in an incoming message are unknown, that may be a good indication of a spammer trying to use mangled words to get their message accross.

Sure, Bayesian as proposed by Paul might not be the final solution. But the countermeasures that spammers use will end up being such that the simple use of those countermeasures will probably be something which can be considered a characteristic of the message which will be further used in identifying it as spam.

In my opinion, the trick will be keeping Bayesian "up to date" in terms of identifying new characteristics that can be used to identify spam. For now, tokens in the message are sufficient.

personally I think..

[sickboy_macosX]

I feel that SPAM should be considered like Telemarketing. And I think we should be able to opt out without notifying someone that our email address is life and getting filled. Interesting Article..I knew about the Green Card lottery, but I didnt know about DEC sending emails to people.

For you Viewing Pleasure

[Michael's a Jerk!]

We Present the world's first Make Money Fast Spam

Re:The solution to getting rid of spam...

[GnarlyNome]

But..But ..my delete key wore out

Well....

[Gefiltefish]

I'm gonna need all that money from Nigeria to afford the necessary penis enlargement and credit rating accentuation!

Examined from the inside, the world of spam has created its own perverse little self-sustaining ecosystem.

Re:Unique? Sorry, but....

[jnana]

The ones *they see* could be unique if just one person from that mailing reports it to them. Also, they could be including the random text in every email, which actually does make it so that every spam is unique.

Re:Unique? Sorry, but....

[Tralfamadorian]

Uhm, why do you say that? According to Merriam Webster spam is: unsolicited usually commercial E-mail sent to a large number of addresses.

Why can those messages not be 'personalized' and still fit that definition?

Ever notice that spam now-a-days has random strings of characters placed throughout it? That's to make it unique to prevent spam filters from looking the checksum of the message up in a database and marking it as spam.

Look at the stupid spammer

[NewtonsLaw]

Today's Aardvark Daily shows exactly why spam is the problem it is -- there are too many stupid people out there who believe they can get something for nothing.

Check out just how lame the spammer in question is and how, in his world, the word "free" has a whole different meaning to the one most people have.

Despite his blatant misrepresentations and the fact that he's promoting his scam via spam, this guy has got people queuing up to hand over their "stupidity tax".

What's worse though is that the spammer is so lame he's effectively exposing the credit card details of *all* those who sign up. You even get to look inside his two email accounts because he doesn't have a clue about choosing sensible passwords.

We're quick to blame spammers for the problem but maybe the truth is that the tide of spam is driven more by the stupid and greedy people who respond to these "too good to be true" emails.

Antispam

[Tuxinatorium]

*scotty voice*
Captain, the spam/antispam reactor is gon ta blow!! I cant give ya any more porno!

Origin of Spam

[Shazow]

The beginning of spam:
Moses brought down the ten commandments.

Result:
Related spam has grown exponentially into dozens of religions.

Anti-spam mail service = id yourself

[192939495969798999]

I don't see how to create anti-spam without some form of identification, simply because without an ID, anyone could use a mail type system to send junk messages to people and not get caught - because there's no ID, of course!

We owe a lot to anti-spam fighters

[bigberk]

Anti-spam activists go to a lot of trouble to help locate and identify people and groups responsible for flooding the net with spam (or who provide spamware to misinformed laypeople). These same good-doers are often sought out by spammers, sued by groups of them, have their privacy invaded (release of home phone, address) in effort to scare them into shutting up.

I am not kidding here. Take a look at some of the projects that scare the hell out of professional spammers:

spamhaus keeps an exhaustive list of major spam operations.

SPEWS lists areas of the Internet that have frequently be used for spamming, including detailed evidence files and histories of ISPs that turn a blind eye to spam.

Spamware vendor list has a listing of sites that sell spamming software -- without which we would have little or no spam.

There's a Reasonable, Albeit Draconian Solution:

[NeuroManson]

About 7 years back, when the WWW was still cutting its teeth, I had an epiphene; The best thing about the internet, is now everyone can use it. The worst thing about the internet, is now EVERYONE can use it.

Simply put, we should require some form of an operators' license to own or operate a computer. Despite there being radical differences between the types of machinery, an adequate comparison would be to either automobiles or firearms licensing legislation.

Before anyone makes the claim that this is not an adequate comparison, if could be eventually, the financial costs of such practices is matching, and quickly overtaking those of firearm and auto related damages. With time, eventually it could cross over to life threatening potential (for example, if someone decided to make a virus with a specific angle, wiping out or modifying records for grandma's prescription drugs).

(1) The majority of abuses involving computers involve people who consider themselves "above the law", with no care in regards to potential damages that abusing the system can incur. Virus writers, spammers, script kiddies, warez distributors and DDOS operaters often fall under this category. For sake of comparison, lets file this under speeders, reckless drivers, drunk drivers, or road rage. Similarly, the comparison can be made for firearms.

(2) The majority of problems that occur within the computer industry and most media involve people who are poorly trained (or not trained at all), or poorly advised in using their computers. People who do not patch their systems, do not operate a firewall, and open e-mail attachments to unleash every iteration of klez upon the net. This one can be filed as those who pretend a car or a gun is a toy, and treat them accordingly.

(3) Despite the whole "for the children" trend in regards to the internet, there is no practical method to truly enforce it without trampling every detail in the constitution. Therefore, unlike most offered solutions, informing and training the young'uns in how to go about using a computer responsibly would be ideal. Similarly, do the same with new computer users. Give them a basic course, then a test, and upon passing said test, they can purchase their own computer.

The problem is, as illustrated by current tech problems, along with the e-commerce industry's shortcomings and varied collapses, Joe Sixpack tends to think of the computer as an appliance. A new magical alternative to the TV that can make all their dreams come true. They need to be informed that the computer is a tool. And just like any tool, it can be abused, and that there could be consequences, something that most of them are for the most part either ignorant to, or even defiant of. Therefore, if they have this knowlege, then they cannot claim ignorance, and as such could finally be enforced, then charges can be pressed, and at least for the short run, problems can be avoided.

After all, if they could lock away Mitnick (sp?) for over 5 years for downloading a few files, why can't they lock away a virus author or spammer for operating without a permit? At least that way they can set a precedent. Hell, I'm sure a good deal of spammers out there are in violation of other things, such as unpaid taxes, working without a business license, et al. And how many of them use their proceeds towards drug use, pornography, etc? Make the bill tough enough and at least the spammers in the US can be eradicated virtually overnight.

There. The can of worms is open. Feel free to bait a hook.

How I've Cut Down My Spam

[MBCook]

My e-mail address is plastered all over the internet, and I don't feel like changing it. I have been getting more and more spam, but I've got it pretty much under controll. For the record I get 20-30 e-mails a day, only 0-2 of which is ham. Here is my little anti-spam journey.

First I ignored it. This worked for a while, but my paitence didn't grow nearly as fast as the spam volume (I've been on the net for years, so I remember when spam was a rare occurace). These are only the major things. I've tried others here and there.

Next I started using MS Outlook's built in spam catcher. This is basically a blacklist that you maintain that you can easily add things too. This actually worked somewhat well, but as the use of forged addresses (and just plain random ones) grew, this became less effective.

Next I started to use SpamNet. I used this up untill about last week. This used to be somewhat effective, and in the last month or so has been almost completely effective. This is the most wonderfull anti-spam device I've used. It was great near the end of the beta. But now it's out of beta and I'm not going to pay $5 a month to stop something I shouldn't get in the first place. Sorry Cloudmark.

When Spamnet started, it was pretty effective, but still left a decent amount to be desired. So I searched around and found SAProxy. This program let's you run Spamassassin on Windows, and the combination of this and Spamnet worked wonders. As Spamnet got better, this became more or less useless.

Unfortunatly, I had to get rid of Spamnet, due to the afformentioned monthly fee. So now all I have is SAProxy. It does work great, and it does get better with each new release. Now only about 3 messages a day get through, which is quite fantastic. Only 5% or so of the spam I get gets though. I could set the limit lower (to catch more spam) but right now I don't have to worry about it catching ham (it never has for me) and I don't want to have to start wading through my spam folder to check for ham. I thought I was using this stuff to not have to do that in the first place?

So in short, I'm now using SAProxy and quite happy. If there was a free version of Spamnet, I'd use it, but there isn't. If you're on Windows and have a supported e-mail client, get SAProxy, and save yourself a huge headache.

So what will I use next? I've been thinking of setting up a perl script to automatically find the home address of people who spam me and sending them a few ICBMs with notes attached like "HOW TO WIN AT EBAY WITH FREE CHEAP ICBMS THAT INCREASE YOUR SEXLIFE AND GROW HAIR."

And Now...

[Michael's a Jerk!]

According to This Site, The earliest spam was sent by DEC in 1978.

Einar Stefferud, a longtime net hand, reports that DEC announced a new DEC-20 machine in 1978 by sending an invite to all ARPANET addresses on the west coast, using the ARPANET directory, inviting people to receptions in California. They were chastised for breaking the ARPANET appropriate use policy, and a notice was sent out reminding others of the rule.

Interestingly, a young Richard Stallman argued that spammers had every right to send spam.

Re:And Now...

[jnana]

Interestingly, a young Richard Stallman argued [templetons.com]that spammers had every right to send spam.

But he retracted in the very next email:

Well, Geoff forwarded me a copy of the DEC message, and I eat my words. I sure would have minded it! Nobody should be allowed to send a message with a header that long, no matter what it is about.

Spammers are trying harder

[waynemcdougall]

I would have thought that spammers wouldn't expend much effort at trying to get around sophisticated anti-spam techniques. After all, if you go to the trouble to block spam, you're probably not going to respond.

But of course some of the spammers get paid based on how many 'eyes' (or HTTP requests) are generated, so if they can just get through to an Outlook Express preview pane, it's worthwhile....until 'marketers' wise up.

By virtue of having my own domain name, outside of the United States, I now receive 1200+ spams a day (and noticeably increasing). People who advocate 'just hitting the delete key' make me fume. That's a lot of delete key. And a lot of time. I've now reached the point where false positives on spam detection by automated software are less likely than me hitting delete one too many times. Thanks to DNSBL I can reduce spam from 1200+ a day to 10 a day, and Paul Graham's Bayesian filtering reduces that down to 2 or 3 a week.

I'd like to share some recent observations I've made - I haven't seen this referenced elsewhere but maybe I don't know where to look (so feel free to point me where this is mentioned elsewhere).

First a minor observation that spam increases markedly on the weekends - because peop,e aren't around to close down open relays or spamming accounts?

Secondly, spammers have started adding non-spammy words (eg capacitor) and constrcuted nonsense words (capacitorsggg) inside their messages. I can only see this as a direct response to Paul Graham's approach. I don't see it as working - the rest of the message is just TOO spammy - but it sugegst to me that spammers see such an apprroach as a threat. I've seen these words sprinkled at the start of plain text emssages and after the /body> /html> of HTML messages.

Thirdly, what I've recently noticed is that a spammer will connect to my mail server, say HELO, do a MAIL FROM: and then QUIT. Then they connect to my system again and use a HELO command that is my OWN IP address. They also include a fake Received header that makes it look as though the message originated from my own machine. Nice try you scummy spammers. SpamCop is smart enough to see through that ploy. I wonder how other system's will respond.

Fourthly, I've noticed that often when I complain to SpamCop I become the victim of a JoeJob. Currently I'm getting all the delivery failures coming back to random alphanumeric usernames at my domain. Sigh. Time to strip off my domain when I lodge SpamCop submissions eh?

From the article...

[ktakki]

A few consulting firms and foundations have also surveyed the volume of spam. Jupiter Research estimates that the average e-mail user gets about 2,200 spams a year, and the Gartner Group says that corporate e-mail is 25 to 35 percent spam.

In 2002 I received over 18,000 pieces of spam, for a total of 163 megabytes. Compare this with the year 2000 (6 MB) and 1996 (183 KB). Based on the spam I've gotten so far this year, 2003 should see a bumper crop of 25 to 30 thousand pieces. This is just my POP3 account, and not my venerable Hotmail account that's now a smoking hole in the ground.

If I'm ever lucky enough to meet a spammer in person, I will kick him in the nuts repeatedly, until he sings soprano. Of course, I'll be chanting "Just hit Delete...just hit Delete" the whole time.

k.

Re:From the article...

[sholden]

In 2002 I received over 18,000 pieces of spam, for a total of 163 megabytes.

You get much smaller spams than me...

In 2002 my spam filter (which didn't come close to catching everything) caught 4546 spams for a total of 78 megabytes.

My spam has noticably increased this year with more of it getting past my filter (which I think I've fixed now). My spam folder for March has 854 spams in it, for 18 megabytes. Multiplying those number by 12 is too scary to even contemplate.

I use a seperate email address for things like online shopping and web site account creations and the like - it's a spam magnet, but I only check it when I'm actually expecting something (because I just ordered a book or something) so I don't care.

I do seem to get at least 2 copies of each spam since sholden@*.cs.usyd.edu.au ends up in my mailbox.

the anti-spam measures

[jnana]

should be as effective as slashdot's anti-troll measures... **ducks**

Just like anti-virus...

[SpyderFan]

Spam is not going to stop. It will continue despite laws and regulations which do not apply world-wide and are difficult or impossible to enforce.

There will not be a "new" SMTP because the existing one is too well established.

There have been many wonderful suggestions posted as previous stories and also as responses to previous stories. Many, perhaps most, of the great solutions require a critcal mass of people to adopt a technical solution at the server. None of those will happen.

The best solution will be individuals or companies adopting products like Spam Sleuth or Spam Sleuth Enterprise which have a variety of detection methods including Bayesian (statistical analysis), EMail Stamps (shift cost to sender), Bouncing (trick the spammers), as well as the usual Whitelists, IP Blacklists, e-mail address Blacklists, etc.

Just like computer viruses, those people who use the technical solutions will be immune, and those that don't will continue to suffer. The tools exist. Slogging through spam each day is a choice.

Re:Just like anti-virus...

[kimgh]

Spam is not going to stop. It will continue despite laws and regulations which do not apply world-wide and are difficult or impossible to enforce.

Oh, I dunno. Fax SPAM was effectively stopped by law; is there any reason to believe that an effective Federal law won't work to at least reduce the volume?

Larry Lessig's proposal for a law, which is actually being introduce by my own Representative, Zoe Lofgren, may very well reduce the flow

I would like to see that law include provisions for going after companies that hire spammers, rather than just the spammers themselves. I don't believe that there is such a provision in the current proposal, but it's been a few weeks since I read it, so I might be wrong. But that might be a helpful addition, if it's not already there.

Finally, I read recently that there are only about 180 major spammers responsible for most of the spam we get. 180 people is not an impossible number to arrest, charge, and shut down. The remaining bit players will probably dry up if the major guys and gals are gone...

Re:Just like anti-virus...

[stevey]

Oh, I dunno. Fax SPAM was effectively stopped by law;

That's possible because it's easy to trace the sender of a fax message, and they were almost always sent from within the same country as you.

Sadly with spam these two things are not true; spam is sent via forged addresses almost all the time, and even if you track down the true sending machine/user it's somebody in America or the far east - which wouldn't be covered by any law created over here in the UK.

I'd be happy with spam if it was possible to identify the senders; a nice big lart would do the trick - but for a law to come into effect it would have to be global in nature.

the money trail

[deus_X_machina]

What makes no sense about spam is that it seems like the only people really making money off of it are the spammers themselves. It's a shame I don't have it on hand, but more than 75% of the services being offered according to one account, aren't even legit, the main exception being pornography websites. (I'm sure many of you will remember the article, and someone will respond to this with it).

Case in point, what I'm wondering is, who are the companies funding spammers? Judging by the relativly low success rate of bulk email, I'd imagine you're actually losing quite a bit of money to pay a company money to send out emails for your company, emails that potentially damage the reputation of your company due to the vast amounts of illigitimate business and anti spam sentiment on the net.

Simply stated, it sounds like:

Step 1: Send mass emails
Step 3: collect profit

I willing to bet their business model was derived from the underpants gnomes...

Re:There's a Reasonable, Albeit Draconian Solution

[dbitter1]

After all, if they could lock away Mitnick (sp?) for over 5 years for downloading a few files, why can't they lock away a virus author or spammer for operating without a permit?

Simple.

Money.

Mitnick's foes' lawyers claimed billions of dollars (that's laywer dollars, not real dollars, of course) of damage to the people padding the politician's pockets.

When spam gets there, we could count on the jack-booted thugs raiding a place or two in the night. Unfortunately, the spammers are getting richer, and trying to make laws that favor them...

Skimpy article.

[mrsam]

This article does not really gives much of an overview on the history of spam wars. The article leaves out more stuff that it mentions. I couldn't find any references to:

* Evolutionary progress from your garden-variety, run-of-the mill carpetbombing from the sender's ISP to hijacking of external mail relays, leading to most mail relays now being closed; to repeated gang-banging of every mail relay on the Internet, in the late '90s, that was running the completely fucked up Sun sendmail 8.6, which fails to record the sender's identity, turning it into a somewhat efficient anonymous spam forwarding service; to direct-from-dialup spamware that doesn't need mail relays and delivers directly to the recipients' mail servers; to spamware that scans and hijacks open proxies, and spam-forwarding trojan zombies that take over and infest Windows-based clients.

* The rise, fall, and bankruptcy of Apex Global Information Systems, the first commercial attempt to make a business model out of providing dedicated spam connectivity; with Cyberpromo, Nancynet, Marynet, and Sallynet spam factories as their charter "customers".

* The rise and fall of MAPS. The article makes out MAPS as the leading champions, but those in the know sadly know that MAPS is now a shadow of its former self.

* The rise and fall of ORBS, and a gaggle of similar open relay blacklists that sprouted up to supplement and replace.

* The rise, and hopefully the fall, of the trend where large backbones quietly agree to accept premium connectivity and hosting fees, in exchange for ignoring complaints about their spamming parasites, all the while flouting their supposed "anti-spam" Acceptable Usage Policies/Terms Of Service (documentation and proof available per request).

* The rise of the trend where spam farms are set up in third world countries, whose hosts completely ignore spam complaints and are generally better resistent to spam blacklists, since they don't send much mail to the US.

* The rise of SPEWS, as a partial response for a need for a successor to MAPS, and a surprising accept of SPEWS, which has an aggressive blacklisting policies, which flew in the face of conventional thinking that network providers will tremble with fear, run to hide in the nearest closet, and become completely paralized at a mere prospect of rejecting a single non-junk message.

There's plenty more subject matter for anyone who really wants to provide an overview of spam wars. This article seems a bit skimpy on the facts...

Several problems

[ajs]

First off, the article is WAY behind the times on anti-spam techniques. SpamAssassin's statistical techniques far outstrip the simplistic features discussed. For example, it mentions obfuscation techniques, and yet SA is known to detect almost all of them one way or another, and even when it doesn't it catches the mail because it's in Razor2, comes from a BLed site, has obviously forged bits, doesn't look like valid mail to Bayes, etc, etc, etc.

Second, the article is also a bit naive on several points regarding blacklists. Many blacklists are good and useful, many are not. But taken as a whole, they present a spectrum of data that can be interpreted through a number of classical techniques that are applied to noisy data sources. Trusting any one BL or a small list is almost always a mistake, you need to build a sample set and determine who you trust and how much. SA does this, but it would be easy enough to build a BL-only SA-like tool for high-speed analysis on high volume ISPs and pipe-providers.

I'm getting worried that the problem of spam erradication is starting to look like the most divisive problem the net has faced to date. There are an awful lot of angry people, and those pitchforks and torches are starting to point in some very "infrastructurish" directions. Articles like this one, really don't help much....

Re:Starting over

[OpenYourEyes]

Well, in Houston they did get an energy boom in the 90's. And they messed it up same as they did in the '80s. Enron is the visible example, but all of the energy companies in Houston are suffering as well.

So to continue your analogy - even if we start over with a new idea, it won't work, because we seem to have the infinite capacity to make messes. Any solution to the spam problem that involves starting over would probably also cause one or more of the following (draw the analogies to Houston and Enron if you wish):

• Raising the price of communications, which the Internet has forced to drop. Even the refutable payment schemes would increase the cost of basic use, which is contrary to some of the Internet acheivements.
• Putting power in the hands of a few large companies instead of distributing it amongst everyone, which the Internet has done. Insert paranoia as appropriate.
• Prevent information from getting to the people who want it. Filters are a good stop-gap for now, but I think they will start to cause problems soon.
• Resistance to migration. Competing designs. Lack of adoption due to cost and effort. IPv6 has been out for how long? Email has been forgeable for eternity, PGP has been available for almost as long - and people largely don't use it.
• A very good chance that spammers would find a way to exploit the new system same as the old. Consider, for example, that even with the authentication available in AIM, the ammount of spam is tremendous. Amazing how fast they adapt.

Good for you, then...

[ktakki]

I can't say that I haven't given my address to people who "aren't stupid or assholes", but I doubt that this is the vector for most of the spam I receive.

You say you've had your POP3 account for over a year; I've had mine for nine years. In that time, I've posted it to Usenet, used it as a mailto: on web pages, signed up for things with it, and used this address to register domains, always unmunged, sans "NOSPAM" or "remove this" or "@@@".

Even if I'd done none of these, or did munge my address, I'd still get spam, albeit somewhat less. Spammers use dictionary attacks, too, or they create a list from /users, or your DSL provider "shares" your address with their "strategic partners". Never tasted that canned pink meat? You Will.

Enjoy it while you can, for I see a million penis enlargers in your future.

k.

Re:Drug dealing is victimless?

[vorpal22]

You do realize that legalizing drugs would address every single problem that you've mentioned in your post, don't you?

Re:Drug Dealing is NOT a victimless crime

[Dyolf Knip]

Drug deals being conducted with guns is entirely a consequence of their being illegal. You may note that gang warfare over alcohol is rather hard to find nowadays, but it was quite intense during Prohibition and in fact is a going concern over in Saudi Arabia where (purely coincidentally, of course) it's illegal.

The DMCA?

[jtdubs]

Shouldn't we be able to prosecute spammer's under the DMCA?

Spam filters are obviously a device used to regulate what mail you receive. They used to effectively block spam. However, spam has evolved to beat the filters.

This implies that the spammers determined the method the filter used, so that they could beat it. In other words, they reverse-engineered it.

So, aren't spammers circumventing an access-control device via knowledge they gained by reverse-engineering a product?

It's that the epitome of illegal under the DMCA?

Justin Dubs

Re:Drug Dealing is a victimless crime

[billstewart]

Tobacco dealers don't go shooting each other on street corners. Tobacco is more addictive than heroin, and costs about as much per user per day to make (all the rest is the cost of the black market.)

Liquor dealers don't go shooting each other on the street corners, though people do rob liquor stores and drunks do get into fights. A day's worth of medical-priced opiates is cheaper than a half-bottle of bad gin.

Zucchini dealers don't go shooting each other, though there are the occasional Midwestern terrorist events (leaving bags of zucchini on other people's doorsteps during the growing season); marijuana's about as easy to grow as zucchini if you're not trying to hide it from the cops.

If we legalize drugs, street gangs may not stop carrying, but they'll mostly stop dealing, because you'll be able to get better-quality pharmaceutical drugs at the drug store and marijuana at the tobacco or liquor store, and at that point drug dealing turns into honest work, not significantly more profitable than selling flowers on the street corners except for a bit of low-markup business selling to minors along with selling them cigarettes. Might as well go back to stealing hubcaps.

Re:HELO forging and detecting

[dmeranda]

I too have noticed that the vast majority of spammers now seem to forge the HELO/EHLO greeting. And as most non-spammers don't, this is actually a wonderful way to catch them. I've even seen them send the IP address of my secondary mail gateway in hopes that my primary mail server would fully trust it (obtained probably by looking up my MX records). I run a mail gateway for a corporate domain an get on average 30 to 40 thousand spams per day. Using sendmail with it's milter programming interface I put the HELO greeting though a very strict check. For those contemplating doing the same...

• Per RFC 2821, the HELO greeting string should be either the FQDN of the sending hostname, or the IP address of the sending system in SMTP syntax (e.g., [1.2.3.4] or [IPV6:abcd::1234]
• Most spammers don't even bother with a domain name, using a random greeting like "sqss7e". If it doesn't have a domain, throw it away. Same if you see an IP address without the [] brackets; it's another dumb spammer that can't read the RFC's.
• Sometimes spammers don't even hide their spammy-sounding names in the HELO greeting even though they go to a lot of trouble to make up legitimate From headers. A good regular expression check for common words like "offers" or "optin" in the HELO greeting can work wonders (but use caution).
• When checking if a spammer if forging your own address, be sure to check for ALL hostnames under your domain (say you have acme.com, then check for both "acme.com" and "*.acme.com", and use a case-insensitive comparison). Also check for ALL your possible IP address even if you don't use them all. A remote site using your own IP or hostname is never legitimate.
• If you are running a gateway, you need to treat outbound versus inbound messages differently. This can usually be done by checking the connecting IP address to see if it is one of yours. Also be sure to check for 127.*.*.* and ::1 (IPv6).
• Be aware that some mail clients are broken and don't send conforming HELO greeting; this includes Mozilla (see Bug 68877). So don't be too agressive with your HELO checks for mail originating from the inside of your organization.

One last note about Forged AOL Spam after talking to one of their postmasters...all their legitimate mail by corporate policy is always sent from within the *.aol.com or *.aol.net domains. This will be in both the HELO as well as a reverse DNS lookup of the connecting IP address. If you don't see this in the HELO and DNS but you see a MAIL FROM for aol.com, it's probably spam.

I wish more big ISPs would provide public information about how to better detect forged mail claiming to come from their sites. For instance if I see a MAIL FROM *@yahoo.com, then should the connecting IP address always be from a *.yahoo.com host? Some ISP's like hotmail seemingly always add in a known predictable header whose absence indicates spam. But I can't reliably make these calls unless the ISPs provide that information. Also, beware that some semi-legitimate sites, like Monster.com forge the sending address on purpose; so if you want to receive resumes you may need to whitelist them.