Spam Solutions from an Expert

Mod N writes "SecurityFocus has posted a nice survey of anti-spam technologies by spam expert Neal Krawetz, in which he delves deeply into the specifics and pitfalls of the numerous proposed solutions. Krawetz makes it obvious that securing the email infrastructure is a very complex problem that many of the current (simple) solutions can't solve alone."

Comments

Proof?

[monstroyer]

The marketing myth emphasizes two misconceptions: (1) a human must perform the challenge, and (2) these problems are too complex for automated solutions. In truth, most spam senders ignore these CR systems because they do not account for a large recipient base, not because the challenge is difficult. Many spam senders use valid email addresses for their scams or for validating mailing lists. When CR systems begin to interfere with spam operations, spammers will automate the responses to these challenges.

Excuse me, what? Where's the proof? That's quite a brave statement to be making considering i've never seen this cracked, ever.

I challenge someone to find an automated response to C/R.

I did hear of a theory where C/R was being cracked by taking the C/R image, posting to a porn session, and letting a seeing person do the work. However, i've yet to witness this in practice. Show me the automated response to C/R that exists beyond a blog theory, and i'll believe. Until them, i hardly consider it "marketing hype".

Re:Proof?

[LostCluster]

That's like saying a all theoretical attacks is not worth securing against somebody's fallen victim to it. Sure, there's some way-out ideas that can be dismissed that way, but this one seems so simple I'm pretty sure somebody who runs both spam and a porn site could pull it off...

Re:Proof?

[ender-iii]

Is this a joke? He just asked for proof and you got modded up by offering none?

Re:Proof?

[ookabooka]

I cant even get my scanner to correctly identify a regular text document, it gets most of it, but it still misses a lot of letters. A computer program could do this, but you would need either a very large database of the letter pictures (most places use all different kinds of text pictures, and add in a degree of randomness). Or you would need a very developed algorithm to detect the letters (in which case you would be making oodles of money from the scanner industry. . . spam would be the least of your worries.
In the end i think it is inevitable that software will eventually break this system, but as soon as it does, there will be another system in place. . . .

Re:Proof?

[michaeltoe]

Yeah, true... I didn't really provide proof... but proof would require me to go out and program something fairly complicated, and I'm in the middle of midterm exams.

The point I was making is that, while noone has done it yet, there's no theoretical reason why it shouldn't be possible.

It's like saying "Oh, that mountain's to big, no one will ever climb it." -- If people are motivated enough, they can accomplish just about anything... and spammers seem clearly motivated.

Re:Proof?

[LostCluster]

Yes, but such a human-check is unlikely to be beaten by a computer 100% of the time. If a log of the failed challenge attempts is kept, the source of repeated failed challenges can be ruled out from getting any more challege attempts, or even just one failed challenge with hundreds of successful ones coming from the same IP space... then the hacker source cna be flagged and ruled out.

The best defenses involve several lines so that when the first gets beaten, another one tightens up against whatever the first line learned from its defeat...

Re:Proof?

[michaeltoe]

True, or you could just come up with an authentication method that doesn't involve visually identifying numbers and letters... like, showing a picture of an apple, and having people type in the name of the fruit.

Then again, that becomes less reliable and more ambiguous. You could keep on pushing for more difficult to interpret puzzles, and the technology to interpret it can just push back. People will just end up getting annoyed by it.

Sooner or later that idea runs out of gas... it's only a temporary solution.

Re:Proof?

[silentbozo]

If a log of the failed challenge attempts is kept, the source of repeated failed challenges can be ruled out from getting any more challege attempts, or even just one failed challenge with hundreds of successful ones coming from the same IP space... then the hacker source cna be flagged and ruled out.

Unfortunately, this is one area in which the spam gangs already have a leg up on the rest of us. Trojaned machines provide them with a distributed set of machines (and hence, distributed set of IPs) from which to launch their attacks. While you may be able to block some zombies machines, there are many more from which the spammers can continue launching attacks, many of which overlap with IP space of actual (non-spam) users.

Unless you're being extremely unforgiving (in which case, you WILL get false positives), all the spammers will have to do is continue rotating machines to prevent exposing an IP long enough to get it blacklisted.

Re:Proof?

[jazman_777]

The point I was making is that, while noone has done it yet, there's no theoretical reason why it shouldn't be possible.

I think you have a future in marketing.

Re:Proof?

[Elwood P Dowd]

Challenge / response systems are broken anyway, even if spammers can't break it.

Why? Because from: is forgeable, and viruses use other people's real addresses constantly. Every day, one of my 40 spam emails is a C/R email from someone that I've never heard of. Am I going to click the link and authorize my email address? Fuck no. But I'll never be able to send email to that person. I realize that's a *tiny* incidental, but it's still broken by design.

If your C/R system includes a solicitation to purchase said C/R system, you're a fucking spammer. Fuck you.

Re:Proof?

[michaeltoe]

As I stated in the beginning, if the human mind is capable of doing it, so should a computer...

We're not talking about astronomically difficult calculations beyond the grasp of any mathematical deduction... and we can infer this with relative confidence, given that your brain is doing these calculations even now, as you read this text.

I'll admit I was simply being humorous in implying that I, myself, if not burdened by exams, could slap something together. It would require a great deal of work to get this operate properly... but it is by no means an unreasonable goal. Science has tackled far more difficult problems than this.

Re:Proof?

[LostCluster]

If you're under attack from all corners of the web, then the only logical thing to do (and maybe you should have a doomsday script to recognize that) then the only logical thing to do is to do what the world would want you to do, block everyone from 0.0.0.0 to 255.255.255.255. "Sorry, DDOSers, we're closed. We'll be back when you all get patched."

Re:Proof?

[Saltcreek]

I challenge someone to find an automated response to C/R.

Students at Berkeley have already beaten the C/R system setup by Yahoo! and with a selection of 191 different version of text obfuscation they were able to return a 92% success rate. In much more detailed images, with random background textures and overlaying text they were only able to achieve a 33% success rate but I am sure with time they would be able to do better.

In a paper published by Greg Mori and Jitendra Malik they explain the methods used to defeat the system. For the full write up you can visit their site on Breaking a Visual CAPTCHA

Re:Proof?

[chrisbtoo]

Well, this is by no means a proof, but maybe a method.

1) Get image. I followed your link and got given this image.

2) Pre-process. I loaded it into the GIMP and did Image->Mode->Greyscale, which yielded this image. Then I did Layer->Colours->Threshold, which yielded this image.

3) Match characters. At this point, you have a monochrome image, in what appears to be a known font. The chars don't even appear to overlap, so a simple 1-for-1 match is achievable. Scan left-right, top-bottom until you see a 10x10 (or whatever) section with a black pixel. Scan down and right from that pixel until you see a character.

I don't have the time to code it up right now, but if someone wanted to pay me to do it, I'm pretty sure it's acheievable - not least because a whole bunch of the more difficult code is available for me to use under the GPL.

Oh Well

[dirkdidit]

With the way the Chinese government keeps making their own versions of everything, maybe they'll have their own version of the Internet. That shoud alleviate a good deal of the spam right there, given that their Internet will probably be incompatible with ours.

Re:Oh Well

[_Sharp'r_]

The Chinese government will probably solve any internal spam problem pretty quickly.

I mean, if you start by shooting all convicted spammers, the profession tends to stop attracting replacement members.

Re:Darth Vader

[Anonymous Coward]

pishhhhh *breathe*
I find your lack of junk mail disturbing.

Don't forget SMTP+AUTH

[RT Alec]

Good overview, all things considered. I would like to add to one of his conclusions (from part 1):

IMAP can be used with SSL and supports secure authentication, but not all servers support this. SMTP also supports SSL or TLS but again, many organization's servers do not support this or use only server-side certificates.

This conclusion is correct, but why is this considered a stopping point? Mail admins-- get off your collective butts and add encryption and authentication to your mail servers! The author also forgot to mention that server side certificates are not necessary for SMTP, SMTP+AUTH addresses this quite nicely.

Note that such measures are not necessary for most users. Home users that use their ISP's mail server don't have to implement any of this, since the ISP can already account for the user. Let us not forget that "most users" do not have the e-mail needs that many Slashdot readers do. For those needing roaming access and multiple addresses, use IMAPS and SMTP+SSL+AUTH.

Re:Don't forget SMTP+AUTH

[zcat_NZ]

TCP is NOT flawed. Sure you can spoof a packet or two, but (assuming reasonably strong sequence numbers) you can't fake a whole connection unless you are actually getting the reply packets.

mail is likewise not flawed; It is fairly hard to find an open relay these days; it is all-but-impossible to find one that doesn't put your IP address in the headers. That's your _REAL_ ip address. The one that ends up in RBL's so nobody accepts your mail any more.

The big flaw is home users; they keep getting pwn3d. And you can't even blame Microsoft for this any more. The viruses are arriving as a zipped, passworded attachment FFS. We've long since passed the realm of just clicking on an executable!

Here's how I see it; the antispam community were on the right track from the beginning. Blacklisting has made it impossible for spammers to spam from their OWN connections, even overseas, and pushed them to finding home users (to spam from, or to attack the blacklist sites). Now they're talking about changing the entire mail system, persuade thousands of users to change the way they do email? Hell no, we've almost won. We just need to educate enough END USERS not to get pwn3d, with the result that the DDoS attacks get cut down and the remaining much smaller number of spam sources can be more efficiently blacklisted.

Or we can force one more 'wafer thin' kludge onto the entire mail system, which the spammers will just find a way around next week anyhow.

Cut Your Junk Mail By 50% !!!

[Snagle]

Just buy porn in magazine format instead of registering for it online :)

Re:Cut Your Junk Mail By 50% !!!

[redJag]

What is this buy? *squints suspiciously*

Solution: Stop Spam at the Source

[ElliotLee]

According to the article, there is no good lasting solution to spam. Indeed, there isn't, but we need to consider more the reason behind the spamming.

Why has spam grown to what it is today? It is an undeniably effective means of cheap marketing. What we need to do is come up with a way to stop this not on our end, but by looking at as a social problem or making it non-worthwhile to the spammers. If nobody ever responded to spam, spammer wouldn't bother.

Deterrents

[erroneus]

At this point in the game, I am honestly surprised that we haven't heard of violence resulting from spam affliction.

I don't know about anyone else, but I'm pretty sure I'm not alone in this. I have, at times, felt utterly enraged at all the spam flying about and further all of the innocent and naive people that are being abused by all of this.

I know if I feel violent internally, then surely there are those with less self-control out there who will eventually act on his or her rage... perhaps the parent of a child afflicted with porn spam?

I think if two or three spammers are attacked physically, it might give them pause. Frankly, I'm amazed it hasn't happened.

Re:Deterrents

[LostCluster]

Of course, the worst spammers make it impossible for the average user to ever identify the true source. I guess you are just giving them another reason why they need to do that.

Re:Deterrents

[AndroidCat]

Type "two dead spammers" into Google. You might even get a link back to Slashdot where it was covered. (Stock spammers likely killed by their business .. partners.)

Re:Deterrents

[Elwood P Dowd]

At this point in the game, I am honestly surprised that we haven't heard of violence resulting from spam affliction.

I'm surprised you haven't heard about it either. Some senile twit that got defrauded by a Nigerian "409" scam email figured that all Nigerians were in on the scam, or something, and killed a Nigerian diplomat.

Obviously, not what you were talking about: it was fraud more than spam, and the spammer didn't suffer, but... that's certainly violence resulting from spam affliction. (Also, note from this article: According to State Department figures (PDF), 25 murders or disappearances of Americans abroad have been directly linked to 419 fraud.)

Open Relays

[QuePasaCalabaza]

The truth is 90% of spam comes from open relays, that is SMTP servers that can be tricked (a bit like lying to a 5 year old) into accepting and sending out massive ammounts of mail. Simply blocking open relays using The Open Relay Database at http://www.ordb.org/ or other open relay checking utility will save you lots of time if you run your own mailserver. When we can bascially negate the usefulness of open relays to spammers, they will then have to rely on their own bandwidth for the most part providing they cannot comprimise other "closed" relays.

Re:Open Relays

[SSpade]

The year 2000 called, they miss your opinions.

In other words, your data is so out of date as to be positively misleading.

Open relays are dead. Open proxies are so 2003.

All the cool kids are using virus distributed trojans these days, some of 'em proxies, some dedicated spamware.

Let's use the Patriot Act for the benefit of good

[mao che minh]

I am in full support of using the broad-powered, freedom crushing Patriot Act in apprehending and imprisoning spammers. We might as well get some good out of it.

white list / web of trust similar to PGP?

[newdamage]

A nice fool proof system, while a bit of a hassle, would effectly remove spam. PGP uses a white list of sorts, that only allows people to send you encrypted messages that have your public key. This in a sense could be done with email. Someone wants to send you an email, and has your email address. They send the small request to your mail server (1-2 KB in size) with their name, email address, and name of their mail server. The mail server holds this information and notifies you that a new sender is awaiting access. You then:

1. Verify the identity of the sender, okay then, and the sender is then given the return request, and is notified that they will be allowed to send emails.

2. Deny the sender, and all their emails will be bounced back.

Yes, spoofing problems still exist, but this system could be expanded, and guess what, you only recieve email from people you want to, and the mail server acts at the first point of defense.

This would require more complex and smarter mail servers, but it would make the every day user's life so much more simple.

There's one billion people in India...

[3770]

that the challenge/response could be outsourced to.

Only kidding (I think).

Re:There's one billion people in India...

[fembots]

This might be a joke now, but it may well happen in the future if we're really into this C/R thing.

At the moment spammers are already paying people to send emails from home, obviously it is profitable enough to pay someone to do the dirty job for you.

As a result, if recepients are less defensive against spams in a C/R system, those slipped spams might get a greater response rate. And this is good news to spammers, and they might very well be able to afford to outsource to deal with C/R.

Good old fashioned riddles

[KalvinB]

My free anonymous (as in they can only be traced back to a common e-mail account on my server) e-mailer uses a simple quiz to keep spammers out.

The form page records the IP address of the visitor along the with the question number they were given in a file named with the IP address. That number is never sent to the client. When they hit submit the file of their IP is opened, the question number is read in and the answer given by the user is compared to the stored answer. The file is then deleted and if the answer was correct the e-mail is sent. Otherwise it's not.

This forces my custom form to be used to be able to send the e-mails. And it's not possible to simply keep refreshing the submit page to keep sending the message.

And the challenge is in the form of old riddles and a couple new ones like "what's your favorite color?"

Things a bot would never get but that anyone who knows how to use Google can. Someone would have to program a custom bot with the answers in order to even attempt to spam. And even then since everything goes through my mail server nobody is going to sneak garbage past me for long and I know who your ISP is.

I also include a disclaimer with every e-mail. It'd be quite silly for me not to.

Ben

He's right, we're doomed

[Zork the Almighty]

I think the author of the article is correct. Having a system whereby anybody can communicate at virtually zero cost without unsolicited commercial messages are mutually exclusive goals. I think that for most people, a simple whitelist is good enough, along with the understanding that there is a small chance that email between new contacts will be blown away.

This will never end

[superpulpsicle]

SPAM is like popups. The one day you find a solution to stop it, the next day they find a new solution to send it. It's a never ending cycle get used to it.

Re:This will never end

[The Cookie Monster]

No it's not.
No other medium has this problem (not in my country anyway)

• The telephone does not have a spam problem.
  
• My instant messanger does not have a spam problem (it used to but they fixed it).
  
• SMS does not have a spam problem.
  
• My postal mailbox does not have a spam problem - "No circulars".
  
• The fax does not have a spam problem.
  

email is the only communications medium that has a spam problem, you are suggesting there is something magical about email that makes email and spam a law of nature.

The only thing special about email is it uses a protocol that was designed with different goals to what is needed now (ie security) and switching is hard, so hard that instead we cop out and just bolt more shit onto SMTP.

A secure protocol with existing anti-spam technology in combination with legislation (which mostly exists already) is all that's required.

Hopefully Microsoft (Hotmail+Outlook+OE) will one day join Yahoo and a few others and together they'll have enough momentum to make the jump to a protocol designed for todays environment. Then SMTP email will go the way of usenet - ie you can still use it if you like, but most people won't have a clue what it is.

If the jump isn't made then email will become less and less useful until it is entirely replaced in our lives by a better (and spam free) communications medium. I'm guessing this will be instant messaging (we already use it more than email), and if I had to put money on the future I'd say the gradual death of email and its replacement by another medium is more likely than actually seeing people stop kicking a dead SMTP uphill and adopting a secure protocol.

Re:This will never end

[0x0d0a]

No other medium has this problem (not in my country anyway)

* The telephone does not have a spam problem.

I live in the US, and we *do*. Do you never get telemarketers?

My instant messanger does not have a spam problem (it used to but they fixed it).

IM systems do. The only reason that problems aren't worse than one might expect is that it's easier to pick up peple blasting out masses of messages because everything in centralized. Centralized systems have their own associated problems (easy monitoring of everything you say, easy abuse by monopolies, single point of failure).

# SMS does not have a spam problem.

I don't carry a cell, but I've certainly heard about people getting SMS spam.

# My postal mailbox does not have a spam problem - "No circulars".

*I* get junk mail in my postal mailbox. Admittedly, a manageable amount, but the majority of the mail I get is junk.

* The fax does not have a spam problem.

True. Up until not all *that* long ago, it *did*, though, at least in the US.

Fix SMTP!

[schnarff]

Well, at the risk of sounding like a broken record, SMTP itself is the problem -- it's badly broken, security-wise, and needs to be fixed. It's going to be painful to move to a new mail standard, or to change SMTP so that it's not broken, but that's what needs to happen to stop spam. Thankfully, our friends the Russian Mafia and the ever-growing number of Windows zombie machines are making spam levels so great that, sometime soon, spam will represent such a large percentage of e-mail traffic that fixing SMTP will be necessary, not just something mail admins like myself wish for.

BTW, does anybody have a good figure on what percentage of all e-mail spam represents these days? I'm talking about *all* traffic, too, not just what ends up in peoples' Inboxes after all the filtering going on out there has done its job.

More details in Part 1

[fembots]

The linked article is part 2, Part 1 is here.

Having experience, I can answer 1.2.1

[snakecoder]

I am not recommending mailblocks, I belive there is a sourceforge project called TMDA which does the same thing. Having said that, my experience comes from using mailblocks:

-cr deadlock: This does not exist because when you e-mail someone in a challenge and response system, it automatically assumes they are friendly. So if they have a challenge and response system, it will make it into your inbox, because you e-mailed them first

-automated systems He is correct here. Personally I hate when friends submit my e-mail to third parties without my consent so I do not mind missing these e-mails. I have caught a few while searching my pending folder, and inform my friends I rather have them e-mail me directly.

-interpretation challenge I believe he is wrong here because of a fundamental issue. When dealing with spam filters, the onus of working out refinements is left to the spamee, to make sure they filter out all spam. If a spammer adds a new technique, they get around the filter. With challenge systems, you have a few methods waiting as backup. When a spammer finally figures out how to read your words through AI, you simply change the challenge system and they are back to square 1 in trying to figure out how to defeat. As long as you have a few methods waiting in the wings, the spammers can easily be defeated, and have huge amounts of work to do.
if you doubt this, write an AI system to defeat hotmails gifs. Now what if the next day instead of showing a word, they show you a picture of 3 fire trucks and 2 police cars and ask you how many police cars are in the picture, etc ...

Re:Having experience, I can answer 1.2.1

[vanyel]

He's also wrong about using certificates:

1. certs don't require a connection to the cert authority. You get their CA cert ahead of time and then trust certs signed by it.

2. Responsible CA's won't grant certs to spammers because people will stop trusting their certs

3. If spam does come in signed, then they are trackable and the backlash will quickly shut them down.

Re:Having experience, I can answer 1.2.1

[vanyel]

Those certs will simply not be trusted for purposes of accepting email. Thawte has a very thorough process for getting a cert with your name in it. Even their "Freemail" certs require some level of data input, but it's not verified. It takes enough time to keep it from being a viable option for spammers though.

Requiring certs would spell the end of anonymous mail, but spam has already done that, and the Beagle virus has shown another reason why everyone (ISPs in particular in this case) should digitally sign their email.

I managed to appall a colleague today...

[Ungrounded Lightning]

Was out to lunch with three colleagues today and the subject of anti-spam measures came up.

I managed to appall the one from Berkeley by suggesting that the most practical solution was probably a moderate-size bomb.

B-)

But seriously:

In an arms race, weapons eventually defeat armor. Spam will continue until two real-world things are BOTH brought to bear on spammers:

- Economics
- Muscle

If a governmental solution applying both is not forthcoming soon, I predict that there WILL be vigilantism.

In fact we're already seeing it.

For instance: Subscribing the Detroit area spammer and his lawyer to enough real-world junkmail lists to bury his bills and other US Main correspondence in several daily truckloads of catalogues and other solicitations.

Soon to come: Retaliatory information-war software directed at DDoSer / spammer zombi-net machines. (As discussed in a recent Slashdot article.)

Re:I managed to appall a colleague today...

[mabu]

Good for you. I feel sorry for all your other neighbors who suffered because of your little "arms race."

I'd give even odds that if you try the "get back and them with the same strategy" you can just as easily end up on the receiving end of punishment by the authorities as them, probably sooner.

Re:I managed to appall a colleague today...

[dave420]

Muscle never solved any argument - it just stopped one side from arguing. The only way to win an argument is to win the other person to your side.

Basically, to get the spammer to stop spamming, stop people buying their product. It's legal, ethical and will stop spam in seconds. Instigate laws that outlaws spam as a method of selling products. Any company found trading via spam can be brought before a court. The beauty with that system is the company has to be reachable via the email somehow (otherwise they wouldn't sell anything, so the spammer wouldn't spam for them), whereas the spammer remains hidden. That lack of anonymity the company posesses means you can find the perpetrator, and press charges. Most likely, the company will release the information about the spammer (including financial information, which can be used to persue the actual spammer).

To reach the spammer you have to go through the only route possible - the vendor.

Newest anti spam technology

[cluge]

**note location of tongue**
Of all the odd places to find anti-spam technology, was this killer solution in WalMart. Yep, it turns out they have a remarkable tool that convinces spammers to stop spamming! I was AMAZED. This tool usually only has to be applied once, and the affect lasts for years. It doesn't require updating or re-installation. I was also suprised to find these very same tools in other places, like sears, and even in a "sneaker" store. What is this tool you ask? An aluminem baseball bat. It seems sadly though that there is a law protecting spammers. I believe useing this AWESOME anti spam technology falls under something called assault There is hope that exceptions for spammers could be provided for in a constitutional amendment!
**note location of cheek**

AngryPeopleRule

dont forget ...

[segment]

I don't bother getting too deep into downloading too many 'new improved!...' filters. I block entire damn countries/netblocks. Besides I don't know anyone in korea, brazil, china, nor any other one of the massive spamming countries. I configure postfix to filter out a lot and the minute I receive one spammed message, I always whois -h whois.apnic/arin/ripe/lacnic offender and block their entire range. I also have spam assassin running and I have to admit I get about maybe... maybe... 4 spams a week not kidding. Again though this is my personal machine.

block return-icmp (8) in proto tcp from 24.76.0.0/14 to any port = 25
block return-icmp (3) in proto tcp from 81.208.64.0/18 to any port = 25
block return-icmp (4) in proto tcp from 163.121.163.0/22 to any port = 25
block return-icmp (4) in proto tcp from 82.77.83.0/24 to any port = 25
block return-icmp (4) in proto tcp from 61.247.224.0/19 to any port = 25
block return-icmp (4) in proto tcp from 217.132.0.0/17 to any port = 25
block return-icmp (4) in proto tcp from 62.103.204.32/27 to any port = 25
block return-icmp (4) in proto tcp from 210.111.224.0/17 to any port = 25
block return-icmp (4) in proto tcp from 144.135.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 195.166.224.0/18 to any port = 25
block return-icmp (4) in proto tcp from 61.228.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 207.144.229.0/24 to any port = 25
block return-icmp (4) in proto tcp from 193.252.22.160/28 to any port = 25
block return-icmp (4) in proto tcp from 200.0.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 209.202.192.0/18 to any port = 25
block return-icmp (4) in proto tcp from 83.32.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 68.38.64.0/8 to any port = 25
block return-icmp (4) in proto tcp from 219.240.0.0/10 to any port = 25
block return-icmp (4) in proto tcp from 195.57.218.0/25 to any port = 25
block return-icmp (4) in proto tcp from 129.79.245.98 to any port = 25
block return-icmp (4) in proto tcp from 24.150.0.0/19 to any port = 25
block return-icmp (4) in proto tcp from 24.205.28.0/21 to any port = 25
block return-icmp (4) in proto tcp from 220.116.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 200.128.0.0/9 to any port = 25
block return-icmp (4) in proto tcp from 212.81.64.0/17 to any port = 25
block return-icmp (4) in proto tcp from 32.10.58.0/19 to any port = 25
block return-icmp (4) in proto tcp from 210.183.110.0/20 to any port = 25
block return-icmp (4) in proto tcp from 134.196.0.0/16 to any port = 25
block return-icmp (4) in proto tcp from 24.60.88.0/23 to any port = 25
block return-icmp (3) in proto tcp from 24.190.8.0/24 to any port = 25
block return-icmp (2) in proto tcp from 24.98.77.0/23 to any port = 25
block return-icmp (2) in proto tcp from 24.173.29.0/23 to any port = 25
block return-icmp (2) in proto tcp from 205.206.176.0/23 to any port = 25
block return-icmp (2) in proto tcp from 172.128.0.0/10 to any port = 25
block return-icmp (2) in proto tcp from 200.171.99.0/24 to any port = 25
block return-icmp (2) in proto tcp from 200.171.97.0/22 to any port = 25
block return-icmp (2) in proto udp from 200.171.97.0/22 to any port = 25
block return-icmp (2) in proto tcp from 68.62.80.128/25 to any port = 25
block return-icmp (2) in proto udp from 68.62.80.128/25 to any port = 25
block return-icmp (2) in proto tcp from 218.76.0.0/17 to any port = 25
block return-icmp (2) in proto udp from 218.76.0.0/17 to any port = 25

Missing from the article?

[Anonymous Coward]

One "solution" which seems to be missing from this article is the "verify each stage" solution. You know, close down all open relays and implement a C-R system between the mail client and the server (password authenticaton to send?) and perhaps between servers too (a public-key challenge before transfers between servers, e-mail transferred in bulk after said challenge for speed reasons). The idea being not so much to make spam disappear, but to make all e-mail clearly and easily traceable so that no spammer would want to keep operating, and allowing any spammer who continues to operate to be tracked down.

Perhaps one of those SMTP fixes or SMTP alternatives mentioned at the end implements this idea? Anyone have more info?

Maintenace the problem

[powerpuffgirls]

As stated in the article's summary, the main problem with most spam-filter is the need for constant maintenance. We need a solution that requires ZERO maintenance by the joe-users, and yet cost-effective enough to implement.

My ISP seems to have a so-called "Watch Dog" spam filter, where they actually hire people to read spams and filter them manually, that's probably the most effective way to filter spam, but I wonder if it is cost-effective though.

Do not call ...

[Ephboy]

Prior to this October, telemarketing calls were a national scourge. Amazingly, since we signed up for the Do-Not-Call list, we've only received 2 illegal calls. I'm rather surprised, in fact, at the relatively uniform acquiescing to this law. While spam, coming from all corners of the earth and is more anonymous, will be harder to enforce, some law with real teeth may be a good start.

Re:Do not call ...

[mabu]

There are a few problems with your comparison:

* It's a lot easier to jack into the Internet than it is to get a phone line

* It's more expensive to perform telemarketing than cybermarketing; you have to pay people and you're not nearly as anonymous - there are costs in launching telemarketing efforts, whereas with spamming, all you have to do now is jack into a network or open proxy and unload your spam.

A spam do-not-e-mail list won't work, because at the present time, the spammers can hide much more effectively on the Internet than they can using POTS.

Not to mention that you don't see telemarketers engaging in the fraudulent practices that spammers employ, so that should tell you something.

Re:Do not call ...

[dave420]

The problem is, it's expensive to call from outside the US, and easily traced. Those two problems alone means it's next-to-impossible for a company to make illegal telemarketing calls to the states. As soon as they did, the complaints reaching the telco would make them track down the telemarketers, and at least stop routing their calls. The cost of international calling also means the percentage of callers who purchase their products has to be highter, meaning slimmer profit margins. That must be a very risky game to play.

Unfortunately, with spam, sending a mail to anywhere in the world is free, and very easy to obscure the true origin. As no-one's paying per-email fees for passing the spam along, no-one's that interested if it's spam or not. There's certainly no vested financial interest in stopping it. Just ignoring it is cheaper than actively trying to cut it out.

The real problem with spam is the relative cheapness and anonymity behind it. The only things that stop people spamming via phone/fax/SMS/etc is the fact that the spammer is easily traced. As we all know, with email it's not that simple.

Re:Nothing really works 100%

[root-kun]

Thats what alot of theyse bayesian analyzer attempt to do. They statistically learn your patterns by what emails you like and what you dont like, and then try to "intelligently" discard the bad ones for you. I mean obviously the worry exists (mostly for companies) that good email may get stopped, but in my experience its very uncommon, aslong as the user has taught the spam bot/blocker properly.

Reputations

[grotgrot]

The only thing that will work in the end is some sort of distributed reputation management system. To a certain extent that is what RBLs do, except they are on or off. SpamAssassin does offer shades of grey to the RBLs (differening weights to each one).

To a certain extent this is what we already do in real life. We 'judge a book by its cover' as a first pass (for example people will often walk past a beggar in the street completely ignoring them) and then include other factors. How polite they appear, where they are from, recommendations from friends etc

All other mechanisms suffer from a determined spammer being able to get around them as the article pointed out. Any mechanism that prevents some spammers makes things more lucrative for the rest.

Re:Reputations

[chriskenrick]

Have you checked out WPBL, as linked in my sig?

It basically attempts to classify IPs as primarily spam senders or not according to the ratio of spam/non spam they send.

The more signed up, the merrier, so feel free to check it out.

Re:Reputations

[leviramsey]

I just devised a setup that might be interesting:

• Users (sysadmins) of the blacklist submit two lists of IPs, good (non-spammers) or bad (spammers).
• When a server receives a mail, it checks with the list to see on which lists the IP appears as good and on which it appears as bad.
• The user marks the mail as ham or as spam. A Bayesian algorithm then determines which lists are trustworthy for marking spam hosts.
• Filters could then /dev/null mail based on this bayesian score

The idea is essentially to allow a collaboratively developed decentralized blacklist and whitelist to develop. Spammers will either submit the IPs they use to this list or not submit them; if they do submit them, then a "good" report from them will eventually be taken as a strong sign of spamminess. If they don't, then nothing happens, but presumably "trustworthy" blacklists would list them.

Thus, a user in Brazil, where they would be receiving lots of legit mail from Brazilian IPs would not find a blacklist that listed all of LACNIC to be a strong indicator of spamminess. The effects of blacklisters who maliciously put enemies into their blacklist would also be reduced, if not eliminated.

A suggested implementation detail on the blocking would be to make it random; that is to say that 100% of the mail with a 100% probability of being spam gets dropped, 99% of mail with a 99% probability gets dropped, 97% of mail with a 98% probability gets dropped, 94% of mail with a 97% probability gets dropped, 90% of mail with a 96% probability gets dropped, etc. according to this function:

d(p) = d(p+1)*p/100, where d(100) = 100, and 73<=p<=100

This would allow for a degree of "retraining" in the event of false positives (since a /dev/null'd mail cannot be retrained from!).

Another partial solution

[PapayaSF]

1) Tap the Slashdot and creative communities to produce a series of anti-spam TV/radio/print ads on the theme of "Spammers are Scammers." Smear all spammers as scam artists who sell fake merchandise and steal credit cards, and their customers as stupid losers.
2) Get media outlets to run them for free as public service ads.

Yes, I know this isn't a 100% solution. However, it is relatively low cost, and requires no new laws, software upgrades, or Internet standards.

What works for me

[gregwbrooks]

I know not everyone can run their own mail server*, but here's what has reduced my inbox spam to about 1 miss out of every 400-500 messages:

• I run SpamAssassin and ClamAV on my server and check all inbound mail against a series of RBL lists; and
• All mail POP'd into my Outlook (yeah, I gotta use it - no flames!) gets checked using the free-and-excellent SpamBayes.

Works in the bakcground with damn-near zero false positives, and doesn't require Microsoft-pushed e-mail postage, changes in the e-mail RFCs or anything else.

The tools are out there. If you use them, spam isn't nearly as much of an issue as the press makes it out to be.

*Well not everyone in the Real World anyway -- here on /. we all run our own boxes, right?

Dueling Challenges

[The Monster]

I just copied that challenge into IrfanView and had it reduce the number of colors to 2. It came out quite readable, which suggests that OCR would be able to take it from there nicely. I bet someone could throw together some Script Fu for the GIMP to convert those pictures to text with a reasonable accuracy rate. Bear in mind that the technique doesn't have to be anywhere near 100% accurate to be worth the effort for the spammer, who already has a business model based on a fraction of a percent of his emails actually generating a response.

What I take issue with is this paragraph from the article:

CR deadlock. Alice tells Bill to email her friend Charlie. Bill sends an email to Charlie. Charlie's CR system intercepts the email and sends a challenge to Bill. Unfortunately, Bill's CR system intercepts Charlie's challenge and issues its own challenge. Since neither user actually receives the challenge, neither user will receive the email. And since the emails are unsolicited and unexpected, neither user knows to look for the pending challenge. In essence, if two people both use CR systems, then they will not be able to communicate with each other.

This is leaving out a key feature of any decent challenge system... When Bill tries to send an email to Charlie in the first place, Charlie's email address is automatically added to Bill's whitelist. So Charlie's challenge, showing his address as its source, flies straight to Bill's Inbox without a hitch. If Bill were so arrogant as to think he could send email to someone not on his whitelist, then he deserves not to have his email go through.

Re:Dueling Challenges

[RollingThunder]

Not so much that it would come from Charlie, but that the C/R would have an In-Reply-To that referenced the unique Message-ID of Bill's mail.

When the mail goes out, Bill's system would record the Message-ID (and probably the recipient, but that could screw up on forwarders if you try for a hard match on the two) and then allow Charlie's C/R because it matches the whitelist.

Re:Dueling Challenges

[Tony-A]

Charlie's email address is automatically added to Bill's whitelist. So Charlie's challenge, showing his address as its source, flies straight to Bill's Inbox without a hitch.

Now all I need to do is know or guess anything on your whitelist (or have some means to automatically add something to your whitelist;).

Methinks all a CR system would do is add hassle to legitimate traffic and give the spammers an even easier time of it.

Re:Dueling Challenges

[The Monster]

The REAL problem is that spammers will start collection email-pairs instead of simple emails and simply circumvent any whitelist. They already do collect addresses out of chain-emails and mailing lists, they just need to collect the FROM address as a second field to the TO address and go on happily

...which is only a problem if a spammer can forge a FROM address in the first place. SPF closes that loophole, allowing white/blacklists to work again. If there's one thing in the article I agree with completely, it's that there is no Magic Bullet -- we need

• SPF to deal with forged headers
• White- & Blacklists for people we already know.
• Challenge/Response for people we don't know yet.
• Bayesian filters
• Special tokens for web sites that let you send a news item to a friend's email by attaching a brief signed personal message (that includes the date and title of the news article to prevent replay attacks) that grants a one-time pass through the filters and C/R.

These tools can be used in various combinations:

During the 'transitional phase' of SPF, source addresses that lack SPF records in DNS would go through challenge/response as an alternative. The challenge email could even include URLs with FAQs about how to implement SPF, handy for forwarding to your mail administrator.

Those tokens might be treated by the Bayesian filter as just one more hint as to whether something is spam. The preprocessor might replace a validated signature with:

<spampass from="dumbfriend@aol.com" date="Fri Mar 12 07:27:46 2004" title="Spam White Paper" />

which might not boost the rating of the email at all, if prior spampasses from this same friend have generally ended up manually marked 'spam' by the recipient.

Re:Dueling Challenges

[Tony-A]

Non-forgeable From-addresses would be nice, but the most critical emails that I send or receive are when email is broken and/or one of us is not in a position to be able to use the normal channels.

It's like phoning the phone company to report that your phone is out of order.
It's like a backup system that works perfectly as long as you don't need it.

The from-address is where the email claims to be from. It should be easily forgeable. If I am using someone else's computer to send a quick note, I should be able to send it, from me, without messing up the computer's settings.

The headers also include where the email came from, at least the last leg of the trip. The headers should be blatantly obvious when mail is delivered. Otherwise it's like the postman delivering the letter inside and keeping the envelope.

The problem with spam is not that it is unsolicited, nor that it is commercial. The problem is that there is far too much of it, and it is being sneaky about delivering it. Spam is socially unaceptable and the solution will be social not technical. For the technical side, the email client needs to distinguish between what it knows and what the email purports to be. For HTML emails, it would help to see which domains are referenced by the email. The difference between the malware running loose now and the Unix Honor Virus is that with the latter you can see what is going on. Anything that pretends to be other than what it is is up to no good. Anything that encourages this pretense (hint, hint Microsoft) is encouraging the malware. Anything that calls something secure when it has only secured part of it is encouraging the malware. A tar-paper shack with a steel security door is not secure.

Of course there is

[Sycraft-fu]

There are plenty of tasks that you can do that computers find nearly impossible. Facial recognition is a good one. Humans do it easily all the time. Computers are trying, but still screw it up badly. Musical recognition is another one. A human can easily pick out individual instruments in a peice, and can tell that the song is the same even if it is a complete different orchestration and mix (like a remix for example). Computers are confounded by this, even when they break something into component sine waves. Pragmatic language interpreatation is my favourite. Even when people speak non literally and indirectly, you still have no trouble with their meaning. You can also tell which level of meaning they want, and successfully decode the other levels if asked. Computers are lucky if they can get the literal direct meaning out of a sentence, never mind anything else.

So, just because a human can do it, doesn't mean a computer can. I don't know about any of these image schemes, I've never played with it. However if you make it sufficiently hard for it to recognise characters form background, and one character form another, it's screwed. Computers have trouble with fuzzy and incomplete information that humans are so good with.

Also remember it needs to be feasable to do in a reasonable time. Maybe you develop some whiz-bang image recog program that can take amazingly distorted text and figure it out. If it takes 5 minutes to process a box, it does you no good anyways, too much time to be worth it for this use.

Re:Of course there is

[whereiswaldo]

Maybe you develop some whiz-bang image recog program that can take amazingly distorted text and figure it out. If it takes 5 minutes to process a box, it does you no good anyways, too much time to be worth it for this use.

Not really. Since spammers are now into the illegal business of commandeering people's computers using viruses and trojans, it would be an easy step to have them process distorted images and feed the results back to some web site.

It wouldn't even take that many computers to send a lot of spam out even at 5 minutes per. Say you want to send 1 million emails. 1,000,000 / 5 minutes = 138 days. If you have 138 computers, you can send out 1 million spams per day.

Re:Of course there is

[michaeltoe]

A computer program is only as good as the way it's been written.

I am not trying to argue that it's a trivial problem that some fresh-from-the-womb teenage programmer with VB could solve. Then again, you seem to believe that the interpretation of geometric symbols is something achieved by only those imbued with the power of God.

Don't get me wrong, but if your brain isn't executing some form of mathematical logic to solve these problems... then what is it doing? Magic? Is that how you're going to support your argument?

Re:Of course there is

[Red Alastor]

It's no more magic than for humans understanding irony. And I would be surprised that computer are likely to learn it in a near future.

Re:Of course there is

[Wm_K]

how about:

1000000 * 5 / (24 * 60) ~= 3472 Days or 3472 computer to send 1 millions emails. Doable, but much more difficult than the 138 you state

Public key cryptography.

[Gadzinka]

When using certificates, such as X.509 or TLS, some type of certificate authority must be available. Unfortunately, if the certificates are stored in DNS then the private keys must be available for validation. (And if a spammer has access to the private keys, then they can generate valid public keys.)

Someone, either me or the author of the article is on crack. I was under the impression that one does not have to have private key in order to validate the signature.

Lets assume that there are CRT records that store SSL certificate for clients allowed to send mail on the behalf of the domain.

example.com. IN CRT "Certificate goes here"

1. Client connects via SMTP-TLS session signed with Client Certificate.
   
   
2. Client sends SMTP command:

   MAIL From: <example@example.com>

3. Server checks CRT record for sender domain and looks if Client Certificate that signed the session is signed with this domain's certificate.
   
   
4. If not, than reject the offer with:

   550 You don't have valid CERT for sending as @example.com

   end everybody's happy.
   

Now somebody tell me, in which step one needs private key to verify certs?

Robert

Re:Public key cryptography.

[Gadzinka]

Well, the better yet news it that you could use it as replacement for both SPF (i.e. sending directly to recipient) and SMTP-AUTH (i.e. sending via smarthost).

In case when mail server finds out that the session is signed with cert ``blessed'' by its own IN CRT, it could allow the messages send in this session to be relayed anywhere sender wants. It would have to have matching domain still, because server wouldn't have the means to deliver it otherwise.

I was trying to post something about this method to Ask Slashdot about a month ago, but editors keep it ``pending'' indefinitelly... The question was ``what am I missing?'' since this method seems so obvious, elegant and simple, that I am surprised that no one came out with this before.

Robert

Re:Public key cryptography.

[Gadzinka]

(2) The private-key is kept on the mail servers at example.com

No, and that's the beauty of it.

Domain's private key doesn't have to be stored anywhere on the net. On mailserver of this domain is another cert (private+public) signed with IN CRT for example.com. But the real private key signing all those certs is only on the terminal disconnected from the net entirelly, used for batch-signing of client certificates.

This way you cannot crack into the computer to steal private key because it isn't anywhere on the net.

Robert

most effective

[mabu]

Make no mistake...

The most effective spam solution at this time is RBL blacklisting. Bottom line.

When you take into account that the biggest problem of spamming is bandwidth consumption and network resources, there is NO better way than blacklisting spam sources and refusing to communicate with them.

Services like Spamcop's RBL really piss off the spammers. All client-side filtering is counterproductive and ultimately useless as you constantly have to update the systems to catch new efforts on the part of spammers to thwart the filters. At least with RBLs, the spammers' connections are immediately refused as soon as they're ID'd.

If you want to identify what is the most effective solutions, it's simple. Look at what pisses off the sleazebag spam community the most. That's relay blacklisting. They don't DDOS the moronic client-side filtering companies because the spammers know they're useless, and even if they're not, the spammers can't tell. What hurts them are when systems say, 'screw you spammer, (click)' and that's done via relay blacklisting.

Why are spammers increasingly changing mail relays and pursuing open proxies? Because of RBLs. Even AOL uses RBLs (including Spamcop). All the major ISPs look at the RBLs because they are THE most effective way of stopping spam. And they're the only way to actually shut down the spammers.

Forget client or server-side content-based filtering. They will NEVER work. RBLs are responsible for forcing spammers into corners of IP space, forcing them to deploy worms and viruses to infiltrate new IP space (which exposes them to more prosecution). RBLs ** WORK ** !

Re:most effective

[Tripster]

You are so right, I use a few on all my servers and they work, cbl.abuseat.org works wonders at cutting down on the trojan spam.

I've also setup my own private RBL, any spam that makes it thru the public ones has the IP it originated from added with no hope of ever getting off it either since there is no contact info sent so spammers have no clue where the RBL is housed.

Just this morning I was forwarded the dynamic ranged from Shaw Cable here in Canada, we were getting hammered by the infected fools there and I complained to them to at least close port 25, instead they sent me the ranges I can safely block, sweet, now to work on Telus.

Re:most effective

[mabu]

Amen.

Shaw is a spam haven.

Comcast is a spam haven.

Virtually all IP space in Korea.

When you start doing IPLOOKUPs of the spammers you begin to see a pattern of which ISPs don't have their shit together.

Why did Comcast start cracking down on spammers? It was probably because admins like us stopped accepting mail from their business customers because they were embedded in the DSL IP space that spammers have compromised. Do you think Comcast gives a damn about spamming? No. But if you start making their IP space unuseable by legit companies, then their buttom line is hit.

Blacklisting WORKS. Unless you run your own mail server, your opinion doesn't matter. Run your own server, deal with these sleazebags every single day, bombarding your systems with their crap, then talk to me about BS client-side filtering.

Re:most effective

[Tripster]

Here ya go, this will help you keep out Shaw's residential customers ...

24.64.0.0/13
24.76.0.0/14
24.80.0.0/13
24.108 .0.0/16
24.109.0.0/18
24.109.64.0/19
68.144.0.0 /13

Those ranges are safe to block, they have other ranges for the static business clients.

Of course another simply step the ISP can take is to block outgoing SMTP entirely for those ranges except to their own mail servers.

Re:most effective

[Ragica]

Some would say RBLs work "too well". They have a fairly consistant history of accidentally abusing innocent parties. Is it the price to be paid for the overall protection? Depends on your point of view.

We don't have that many clients using our mail server, but one noticed one day that mail to him to friends was bouncing. He reported this and we discovered that we were on SpamCop's RBL list.

I did a quick audit of the mail server, fearing we'd been highjacked, but found no evidence anywhere of spam going out.

Being generally sympathetic to RBLs I was eagre to get to the bottom of this, and cooporate with whatever needed to be done to prove our innocence.

But i found the SpamCop web site to be extremely frustrating to find any information. I found some references stating that to refute being listed you must reply to the email that SpamCop sent you: I searched and searched but we recieved no mail from spamcop.

As I spent a precious day trying to figure out what to do, as mysteriously as we'd been listed, our IP disappeared from spamcop's list.

To this day I don't know what happened; but have a somewhat more bitter taste in my mouth regarding the arbitrary power of RBLs.

(Though I still tend to more blame the system which blindly obeys a single RBL: I think SpamAssassin is more democratic in that it only assigns a probability, and an IP has to be on multiple block lists before it goes over a threshold. This gives spammers more lead time before they are blocked, but also prevents any single RBL from weilding absolute power... a sort of check-and-balance.)

expert my ass

[xenocyst]

while his credentials certainly would put him in a far better position to know these things than i am... i find his death and doom attitude annoying... he doesn't really address the parts of anti-spam that do work.. he glosses over them, and then hypes the parts that are broken.. without any sort of proof if i were to mod the article it would probably get something like +2 informative -2 Overrated -1 Flamebait and -1 Troll

C-W Problem

[xSquaredAdmin]

Consider that both the sender and the recipient have a C-R filter. How will either one get the challenge? Wouldn't it just end up in an infinite loop of challenge e-mails? Or is there something I'm missing?

SPF Anyone?

[ignoramus]

One proposed solution I would love to see getting more attention is SPF ("Sender Policy Framework"), which allows each domain admin to specify their email sending policy using existing infrastructure.

See the SPF site or read this month's Linux Journal to find out more.

Executive summary of SPF: Just use DNS to specify where mail from your domain may originate from. If everyone used this, we could have domain blacklists that actually work.

Do an "nslookup -type=txt psychogenic.com" to see an example entry. And if you manage any domains, please consider doing the same.

challenge-response handling being outsourced

[SuperBanana]

I did hear of a theory where C/R was being cracked by taking the C/R image, posting to a porn session, and letting a seeing person do the work.

I had a chat with a Veep that was hired on to a company I used to work at. Very down to earth guy, very friendly. We got to talking about spams and semi-legitimate emailings to customers, etc.

He had one very interesting tidbit; stick with me for a sec here. Most companies outsource their semi-legit stuff because they get reported as spammers and whatnot, or it bogs down their email server/network, etc. No surprise there- however, the interesting tidbit is that the outsourcing companies turn around and outsource to Indian firms for handling the bounces. There's literally a room full of people in India, sitting there answering those challenge/responses and updating the client's customer email list(unlike spammers, it really is in their best interests to minimize failed deliveries). It sounds "expensive", but it's not, considering how few people use challenge/response systems. Further- a reasonably smart human can get familiar with all the various systems quickly(an hour or two, I'd guess, tops) and probably process close to a message every few seconds with a client program set up to do that limited functionality smoothly. Best part- if your client does several mailings, unless the recipient goes in and removes you, you're clear for future emailings.

Two-pronged combination

[Lenbok]

What is wrong with using a combination of a hashcash type approach in conjunction with cryptographic signing to address the shortcomings of both.

Thus the following rules for the user:

If an incoming email is cryptographically signed by someone on your whitelist, accept it.

If an incoming email has made hashcash payment, accept it. The user then decides whether to accept future signed messages from the sender.

Other incoming mail is returned to sender instructing them to make hashcash payment.

Sign all outgoing messages, and also generate hashcash if you haven't previously sent to the user.

How this affects the downsides:

Mailing lists: Would generate hashcash payment for the subscription process, but regular mail messages are just cryptographically signed (i.e. independent of the number of subscribers).

Unequal taxation: May still be a concern if your machine isn't up to the task of signing the bulk of your outgoing messages.

Robot armies: Users (should) quickly notice if their machine is burning the CPU generating hashcash tokens and address the problem.

Legal robot armies: I don't see what the problem is here -- the sender is still having to pay to generate the tokens, so the economics of spam are changed.

Automated abuse: Hashcash payment is required for all initial messages, so generating countless certs doesn't help.

Usability: Crypto signing is done with self-signed certs (e.g.: PGP) so no central CA is needed.

Re:Nothing really works 100%

[silentbozo]

Well, to prove identity you could cryptographically sign mails. When the recipient gets the signed mail, they do a key lookup and verify that the signed mail was signed with the correct private key.

Now, how do you handle the situation where spammers are generating thousands of keys? Well, the spammers are forced to waste some cpu time, but that's trival for them. They're also polluting key registries with their garbage - that's a big negative.

However, in terms of trustworthiness, the spammer probably hasn't gotten all his keys signed by somebody else who is of a "trusted" ranking. Even more likely, much of the signed mail you do get will either be known to you (ie, you've signed their keys) or will be known to people you know (ie, someone you know has signed somebody else's key.)

Mind you, this is no replacement for other types of filtering (ie, SpamAssassin with Bayes, etc.) but it would make whitelisting useable against spammers who forge e-mails, UNLESS the spammers know the private key of the poor slob that they're impersonating.

Current boycott of Microsoft email caller ID!

[Rayban]

A related note- the current Microsoft anti-spam solution, Email Caller ID is currently being boycotted.

Re:Not for all, but a good start..

[mabu]

From spoofing verification won't make a difference... it'll slow down mail services and won't make a dent in spam.

Spammers are now rotating IP space all over the place... they're also beginning to NOT forge header information, so what are you left with?

Recognizing rogue relays and blacklisting them, even if they have valid header information. Any improvement to SMTP protocol won't make a bit of difference.

Most mail servers and large ISPs are already employing additional methods of header-verification. It hasn't stopped spam.

RBLs ARE working. They're making spammers scramble for un-blacklisted IP space. That's why they're running overseas; that's why they're sending out worms and viruses. Lord help us if IPv6 gets introduced... we'll never be able to stop spam then.

Interview with a spammer

[dbIII]

The truth is 90% of spam comes from open relays, that is SMTP servers that can be tricked

A couple of years ago I didn't have a job - and a government sponsored job database came up with a listing for a job using computers in the "adult" industry. I went along to an the job interview, and found the job would have been setting up a dozen modems on a linux box and writing a program to scan for open relays (he'd apparently paid US$10,000 for a list). All income would have been undeclared, and some dodgy accounting involving payroll in the name of tourists would have gone on. Some background checks on the employer turned up a few interesting things as well as birthplace, education, home address etc. It looked like a may have had a choice between becoming a spammer and never getting paid for it, or losing my unemployment benefits (the consequence of turning down a job offer in my country). Another, actual legitimate job came up for a dying dot-com, so I never had to argue with beuracrats as to why I had turned down a job.

Oddly enough the spammers name was "Fagin", as in the Oliver Twist villain, and he was born with that name.

SPAMfighter works for me...

[Alex_Ionescu]

The big problem with mail filters, as the article mentions, is that they need to be updated when new spam technologies appear... and there's also a lot of false positives... I gave SPAMfighter a try (from www.spamfighter.com) and although it was a bit worse at finding spam (At first), I never got any false positives. The way it works is that the "filters" are actually some kind of hash that users submit whenever they block or unblock an email (it analyses the whole content I think, not just the text). So if a new type of spam technique appears, the users will just block it. And unlike many other client-side plugins, it actually works on Outlook Express.

Another one I recomment is Spambayes...but there's the problem with false positives. All the other ones I've tried are utter crap.

Best regards,
Alex Ionescu
Relsoft Technologies

Re:Nothing really works 100%

[Skim123]

What happens when someone on your whitelist opens an attachment that automatically sends email from their account, signing it? Now you have a spam that has been legitamately sent from your friend's account.

I created a C/R anti-spam system myself, but gave up on it and turned to Spambayes for two main reasons:

1.) I was losing challenges in others' spam filters
2.) I would still get emails from whitelisted folks when they were infected with an email worm.

If you're interested, I blogged about my switch from C/R to Bayesian filtering here.

An idea to make ISPs responsible

[mabu]

I have an interesting idea to force ISPs to crack down on spamming customers...

This basically works only if the spamming ISP is from your country. Which is why blacklisting of foreign IPs is still necessary.

But for domestic ISPs who don't reign in spamming, someone should post the 800 numbers of ISPs that don't crack down on spamming. Put up a web site listing the 800 numbers of the ISPs that are top-ranked in harboring spammers. Most of them have 800 numbers.. if everyone calls these ISPs and complains, or at least takes up air time, it costs them money, and money seems to be the only thing that motivates these companies.

Comment removed

[account_deleted]

Comment removed based on user account deletion

CR deadlock

[Skim123]

Another deadlock case, which happened too many times in my experiences with C/R:

• Alice sends a message to Bob. Alice is not in Bob's whitelist, so Bob's C/R anti-spam system sends a challenge to Alice.
• Alice doesn't use C/R, but rather a filter. Her filter, unfortunately, marks Bob's challenge as spam. Since Alice is only a computer novice, she does not know how to check his Junk Email folder, and therefore never receives Bob's challenge, hence Bob never gets Alice's email. Alice, who is blissfully ignorant of the "behind-the-scenes" happenings, thinks Bob just is trying to ignore her. So she sends another email, which is, of course, not received by Bob. And she sends another. Still, no response from Bob. Alice takes it personally and decides if she does ever hear from Bob again she won't be going on a second date with him no matter what.

Re:CR deadlock

[LostCluster]

Such a deadlock is avoidable either Alice and Bob are both smart enough to see it coming and value the communication enough to head it off.

When Bob gives Alice his e-mail address, he could put Alice on his whitelist immediately, or have given Alice a password that would automatically get her past the screening process on the first try. If Bob was really interested in getting Alice to go out with him again, he could have sent a request through through whatever common friend or dating service first introduced the two of them.

Alice could have not sent a second e-mail, but instead contacted Bob through whatever common friend or dating service first introduced the two of them. Alice could have also asked Bob for a secondary contact means during the first date. She also could have asked for a better communication means than e-mail if she wanted to make sure she could get through.

You should never make any important decisions based on an unencrypted e-mail alone unless you can contact the sender in another medium to confirm that the message you got was one they sent and meant. If you're girlfriend is dumping you by e-mail, veryify that with a phone call or at least confirm that she's refusing to take your calls... you do need make sure it's not somebody else trying to freak you out by making you think your girlfriend dumped you before you take any irreversable actions based on that information.

Re:CR deadlock

[berbo]

" Alice takes it personally and decides if she does ever hear from Bob again she won't be going on a second date with him no matter what." Excellent. Clueless luses like Alice shouldn't be allowed to breed. -berbo

What about typos?

[TekGoNos]

Humans do them lal teh tiem.

So you cant just block someone after one mistake.

You just have to get your computer program better than the average typo occurance.

Oh ... and remember the Slashdot story a few weaks ago where a computer spam filter was MORE accurate than the human testers. (Yeah, it probably was spam filter reads whole message vs. human reads only subject, but still ...)

I think there are many tasks where a well trained computer program will perform even better than the average human.

Re:What about typos?

[LostCluster]

I wouldn't block anybody after their first mistake. However, there comes a point where too many mistakes indicate either a robotic attempt that isn't learning from its errors, or a really stupid human who likely can't compose a useful e-mail either.

Many spammers who are trying to beat a Bayes filter are either using misspellings of their most spammy words, or large lists of random dictionary words to try to lower their score. However, a coutermeasure to that would be to factor in the results of a spell check and grammar check. Some errors can be tolerated, however having too many mispellings and too many word groups that can't possibly be a proper sentance should raise the score enough to counteract the attempts to lower it and then some.

Oh, but the easiest defeat of C/R...

[Ayanami Rei]

most C/R engines use a constant suite of pictures and words because the pictures are too time consuming to create on the fly... so the signup page might take too long to load.

What the spammers do is just download as many challenges as possible, solve them, and store the hashes in a database.

When the harvester goes out, it is likely to encounter many of the challenges a second time, and it already has the answer. :-)

If it doesn't know it, it flags the spammer, who identifies it offline, adding it back in, and the database is that much more useful.

Re:Oh, but the easiest defeat of C/R...

[Tony-A]

Good one that.

If this becomes a race between the "good guys" and the "bad guys", the bad guys have more incentive to get it right. Just like virus writers will buy anti-virus software, spammers will buy the C/R software. You don't attack your enemy's strengths, you attack his weaknesses, preferably ones he doesn't even know about.

The spammers weak spot is the money he makes.

[sbaker]

I think we are attacking Spam from the wrong direction. Attempting to stem the flood of incoming spam is tough - everything about the identity of the incoming spam can be faked. However, we could alternatively attempt to prevent the replies going back the other way.

There are two inevitable facts:

1) In order for spamming to be worth someone's effort, they have to somehow get money from people. If NOBODY replied to them, then spamming would stop overnight.

2) Something in the content of the Spam must be real - a reply address - a web site, a phone number or something. Block traffic to that location and the spammer gets no money and dies.

Hence, I think they may be vulnerable. Educating people not to reply to SPAM would help - it only takes a mere handful of people to respond to a SPAM to make it profitable - but if education could drop that handful to a mere one or two - then we could succeed in putting more spammers out of business simply by cutting their margins to the point where it wasn't worth the hassle.

Where are the TV adverts: "Replying to Spam is Bad!"....we know that the morons who reply to spam are suckers for advertising - they are as likely to believe a well targetted TV advert as a crappy email shot. If Spam is costing the ISP's as much as they say it does - then funding some TV ads might not be impossible.

What if we made it illegal to respond to an emailed advertisement that was not clearly labelled as such, that would help to deter people from responding. Such a law would be next to impossible to enforce - but we are trying to deter the gullible here - so it might not have to be enforcable - just very well advertised.

Since every SPAM has to either advertise a product that you can buy from somewhere - or direct you to a postal address, a phone number or a web site - then that route for getting money back to the spammer could be blocked.

The return route has to be genuine. There is no point in them sending you a fake phone number or faked web address. If the phone companies (who are often also ISP's - or have at least some cause to want to kill spam) were to block calls to and from phone numbers that were seen in Spam - then the reverse route for the money would be curtailed. Whilst you can afford to change the aparrent source of your spam and fake those addresses for each new mail shot, you can't change your phone number for every couple of dozen orders you take. Similar considerations apply to web sites and postal addresses.

If it was required for credit card companies not to transfer money to businesses that employed spammers to push their goods - then that would also help some.

It wouldn't take many people to deliberately reply to spammers - to lead them on into thinking you want their product - to send them fake cheques or bogus credit card numbers. If they only get a handful of positive responses per million spams - then it wouldn't take more than a few determined people per million (eg ISP employees) to clutter up the the spammer's cash collection mechanism to the point where it's too much hassle for him to sort out the real orders from the bogus ones.

I don't pretend to have all of the answers - but there seems to be far too little creative thinking along these lines.

Re:The spammers weak spot is the money he makes.

[sbaker]

1. Most spammers use faked email address, they DO NOT suppose you to answer them. They want you to click the link, they want you to buy something, they want to install some spyware, adware or what-so-ever-ware on your computer!

I agree that the email address they give is likely to be faked - but my point is that in order to make money, SOMETHING in that post has to be real. If not the email address then the postal address, phone number, web site, etc.

2. Who can block the phone call to a certain number, who can block everyone's access to a certain website, and who can block a real physical position (address)?

The government could pass laws requiring phone companies, ISP's and the US mail to block traffic to people who have been logged as advertising illegally via email. It would require an efficient method to collect these addresses and automation to do the banning - but that's within the bounds of technical possibility.

A spammer can change his email address for every spam he sends - but he can't change his web site that often - and he certainly can't keep changing his phone number, physical address or bank account. I read somewhere that 90% of spam comes from just 600 people. It can't be that hard to block the money going back to those 600 people.

Spammers make profit in the hope that 0.000001% of the receivers would click the link, make a phone call, or write a snailmail to that address.

Yes - exactly. But if you can add a couple of zeroes to that 0.0000001% then it won't be worth their while. If every million email spamshot nets them 50 orders (a number I read somewhere as typical) - then they can make just a couple of bucks on each order and they have earned $100 for the time it took to type a single Spam and to run their system to send it. That's good money.

However, if you can get the numbers down to where they have to send several different mailshots to get even one order - then it starts to look like a pretty unprofitable business model and they'll stop doing it.

It seems that you don't understand how spamming works. This is a social problem, and cannot simply be "blocked".

I think I do understand how it works. I absolutely agree that blocking the spam isn't the answer - and that's my entire point. Removing the spammer's motive for sending the spam in the first place is the only answer IMHO.

Yes, of course...

[michaeltoe]

This is similar to the argument that a computer cannot determine when it's in an infinite loop. Humans, however, can... because they are impatient, and given time, will reexamine the code that is executing.

Naturally we may be inclined to believe that this grants us superiority to the computer. That, while stating some arbitrary facts taken from some textbook somewhere, a computer can never accomplish X objective.

Therein lies the fallacy. The computer does not identify that it is in an infinite loop, nor can it, because it is not given the benefit of looking at the actual code. If a compiler were designed to read into code for things like while(true) loops, which naturally could result in infinite loops, then already you would be cutting back on the instances of these problems.

Determining if there is an infinite loop requires a conscious understanding of the code itself, which is no trivial matter. It is not, however, something that could be deemed impossible.

As with all fields of science, there will be those who say "Well, I haven't seen it yet, so it will never happen"... but skeptics are everywhere, and the presence of skepticism is hardly a measure of credibility... rather, a measure of how pious certain peoples assumptions are.

Solutions are always found in math, and never in magic. Don't underestimate the computer, and more importantly, don't underestimate your own brain. You don't perceive things the way you do 'just because'... and that's what's so exciting.

What about a web of signed trust?

[Muerte23]

So what about this:

You start with a central certificate authority. I know, I know, bottlenecks. But you only need them to issue keys to (or sign the keys of) about 100 (or 1000?) servers. The signing authority has to be central, but the *revocation* authority does not. That's the key here.

So those servers can sign the keys of 1000 servers of their own and so on.

So my mail server tries to send your server an email. Your server checks if my key is signed by someone who is signed by someone who is signed by the CA. It also checks against its nightly downloaded revocation list. If everything is good, the mail goes through. Very little processor time, and very little bandwidth.

Suppose someone issues a key to a dishonest server? Well, enough people issue complaints and the issuer's key gets revoked. Or some automatied spamassasin type thing that auto-revokes the key after enough spams get spotted. No more spam from them, and maybe next time the admins are more careful.

This totally eliminates (i think) the threat of zombie SMTP servers on DSL and open relays.

Then the ball is in the park of the ISPs and server hosters (those with their own email keys) to keep spammers out locally. SLL login for SMTP? sure. C/R for each email sent through them? Whatever. Send anything over their open relay? Not for long.

Sounds reasonable to me. It makes it easier for the end user I think, and minimizes spam.

Any suggestions?

Muerte
This totally eliminates zombie SMTP servers on cable lines spewing spam.

Didn't see this solution...

[kwenda]

I didn't see any mention of a pretty good solution that i've run across:

Every time a message hits a server from a sender that it has never met before, it sends a TEMPFAIL back instead of accepting the message. All real MTAs will try again with whatever their retry delay is set to, and usually for about 4 days. If the server gets the same message being delivered again, it accepts it and adds the sender to a whitelist where it never has to 'ask questions' of this sender again.

The reason that this would work, at least for now, is that spammers mostly use badly written MTAs (or something akin to an Expect script posing as an MTA). Their software doesn't know how to deal with a TEMPFAIL and never tries again. All real MTAs will try again within a few minutes. Good times.

Is this really an expert view?

[Tamor]

When I took a look at the first of these two articles which examines end-user anti-spam solutions I had to wonder if the writer had actually tried any of the technology or was relying purely on hearsay. For example:

Spam senders and their bulk-mailing applications are not static -- they rapidly adapt around filters. For example, to counter word lists, spam senders randomize the spelling of words ("viagra", "V1agra", "\/iaagra"). Hash-busters (sequences of random characters that differ in each email) were created for bypassing hash filters. And the currently popular Bayesian filters are being bypassed by the inclusion of random words and sentences. Most spam filters are only effective for a few weeks at best

This is the view of someone who clearly has no experience at all with a high-quality Bayesian classifier like POPFile. I've been using this program for almost a year and it most certainly has not been defeated by random words or spelling. Many of the tokens that trip email as being spam are actually unusual items in the headers or sales terminology. After a very brief training period POPFile has continued to provide me with excellent protection from spam and malicious email, with only a few false negatives to retrain on.

If that's not a good end-user anti-spam solution then I don't know what is.

Re:Is this really an expert view?

[Tamor]

The more I thought about it the more I've come to the conclusion that the sole purpose of the first article was to declare end-user spam solutions dead to set up the need for the second article.

The author has a point when he says that end-user solutions don't stop the spam traversing the network and consuming bandwidth and resources. However, if significant numbers of internet users employed effective end-user anti-spam tools then it would eventually hit the spammers economic return.

Spam becoming a less lucrative quick-buck will probably be the thing that eventually kills it off. That's a long-term goal that can probably be as well achieved by educating the masses as to the wealth of excellent end-user tools available as it can by expensive and unwieldy protocol changes.

Next gen SMTP

[bakreule]

Maybe I'm missing something, but the article never seems to mention anything about changing the SMTP spec itself. It talks about how it's flawed, but then summarizes new ideas to add on top of SMTP (crypto, C/R, etc). It doesn't ever suggest CHANGING the underlying protocol. History, with IPv6, is showing that the industry doesn't change without being prodded. I don't think this would be the case, however, with any new versions of SMTP.

From what I understand, rewritting SMTP to fix most (if not all) of the spam loopholes is no problem (Am I seriously glossing over some big details here?). The trouble is that people want a 100% effective, immediatly pluggable solution. If new email clients support both the old and new smtp protocols, and use the new one as a default, it will be just a matter of time before there's a critical mass of clients and ISPs that are using the new one.

Once this critical mass is reached, boom, everyone is required to use the new protocol, and any email that uses the old one is immediately dumped way upstream, before it can start hogging bandwidth everywhere.

I'm aware that if my idea is so great, how come it hasn't been implemented?? Feel free to pick holes....

hashcash comments

[Adam Back]

I'm the inventor of hashcash. Here are some comments on the article's comments on hashcash, I think the author missed some aspects around how mailing lists work with hashcash, and the economic model. Most of this stuff is covered in the hashcash FAQ

* Mailing lists. [...] if there is a way for legitimate mailing lists to bypass the challenge, then spammers can equally bypass the challenge.

Hashcash is generated for the mailing-list address. The recipient would add the mailing-list to their list of addresses they accept mail as, and a spammer can not send to the list without including hashcash. So the limitation for mailing-lists is that the spammer can send mail to many people (the list subscribers) for the cost of one stamp; if he sends directly he has to send one stamp for each recipient.

* Robot armies [of 0wned machines].

Clearly someone wit lots of owned systems can send lots of mail; but still less mail than they could without hashcash.

* Legal robot armies. [...] Large spam groups can afford purchasing hundreds of systems for distributing an computational cost.

They can do this (and doesn't matter with it's legal or not btw, they'll do it anyway), but it will cost them more per mail which will cost them, so they will send less mails and be economically incentivized to target their mails by buying demographic data etc. (eg. so you would be less likely to receive spams in languages you can't read, or on topics you are not interested in).

Another aspect is that legitimate users do not send mails to lots of new recipients; most email exchanges are conversations over a period of time with sends and receives. Some of the hashcash based systems use hashcash only for introductions, and exempt recipients from hashcash after that based on crypto tokens (or just whitelists) (eg CAMRAM, TMDA do this).

The argument here is that hashcash can be set to higher cost as it is only borne once per new recipient for normal users.

Just how bad is it, really?

[mwood]

Because a 100% UCE-free Internet is going to be darned expensive and rather less usable. At what level of filtration does the next incremental improvement begin to cost more than simply being satisfied with what you've accomplished?

I've tuned up a pretty good stack of procmail recipes, set my MTA to refuse unverifiable senders and obvious forgeries, subscribed to a couple of decent blacklists, and trimmed things down to a level I find tolerable. And thus I'm disinclined to do much more.

Through a bit of mental jiu-jitsu I've come to regard the remaining trickle as a moderately challenging puzzle provided to me for free, and a source of amusement first thing in the morning as I make the initial pass through my inbox to weed out the junk unread. I spend a few moments each week enjoying the logs that Exim and my procmail recipes write to show me what they've strained out. Once you push the S/N ratio high enough to get some work done, it's possible to turn the rest of the N into fun if you have the right attitude.

Oh, there are other things I'd like to do. If most people would crypto-sign their mail, I'd set up recipes to toss unsigned messages, and play around with hacking signature and CA blacklists into my filters to get rid of the more brazen attempts. I'd like to try out some recognizers that would be mighty hard to write as regular expressions. I'd like to tinker with external filters that rip out some of the common obfuscation techniques before procmail even sees the message. But for now I can live without these.

If you're thinking, "but it's costing my company money to deliver this junk," ask yourself how much it's costing your company to have you sitting around trying to find ways to remove the last little morsel of UCE when you could be crafting new competitive advantages for the firm, or at least dealing with the *other* stuff that gets in people's way and which is not actively working against you.

Forget challenges that require human input

[eric76]

What I'd rather see is every e-mail transmitted be digitally signed.

When the e-mail client is set up, it could generate a GPG key set to use for signing the e-mail.

The recipient's computer, if verification is required, could send a standardized e-mail back to the sender's computer asking for the sender's public GPG key. If and when it arrives, check the digital signature and either deliver the e-mail or /dev/null it.

By caching the keys, you really wouldn't even have to have a white list. Or, more accurately, the white list would be by digital signature rather than the Reply-to or From address.

This could even be implemented on the server itself and with better results.

When adding the user, create a GPG key for that user on the server.

Require authorization for each incoming e-mail that is to be relayed. Digitally sign the e-mail with that key if it sender has not already done so on the client side.

The recipient's server or the recipient's client may then request the public key. If the public key used was the server's key used on behalf of the client, then return that. Otherwise, send the request on to the client for his public key.

Of course, this could be abused, but then the e-mail addresses have to be real and could then be used for blocking.

The traffic itself should be relatively small. The data portion of the request would just identify the public key desired based on what was used on the message (sender's key maintained by the server or the sender's key maintained by the client) and the data portion of the response would contain that id and the key.

For those who use multiple e-mail clients, allowing the server to handle the key would be preferable since the multiple clients would generally use different keys.

If the cached public key for that user failed, a request for the public key would be sent in case the public key had been changed. If the new key was different, the cached public key could be expired after a set period of time (in case there were any yet to be delivered e-mails from the old key around) and the new public key added to the cache.

You'd have the benefits of challenge-response systems without the users being annoyed.

One problem with challenge response systems is with mailing lists. With this method, there would be no problem since the mailing list's server would react to requests for the public key by providing it.

This would also take care of the automated e-mail case, say when you place an order and the sender sends an e-mail telling you the order has been fulfilled.

Sorry Won't Work

[Battle_Ratt]

Two words, Joe job.

Any one of these "solutions" can be exploited to hurt legitimate business. Simply send out a spam campaign on behalf of XYZ company with legitimate credentials, and watch the chaos and disaster at the company as phone lines are cut, merchant accounts cancelled, etc.

Spammers have already done all sorts of illegal activity to continue their frauds, what's one more to cut the knees out on the competition, or the competition of their customers.