Spam Solutions from an Expert

Mod N writes "SecurityFocus has posted a nice survey of anti-spam technologies by spam expert Neal Krawetz, in which he delves deeply into the specifics and pitfalls of the numerous proposed solutions. Krawetz makes it obvious that securing the email infrastructure is a very complex problem that many of the current (simple) solutions can't solve alone."

Comments

Proof?

[monstroyer]

The marketing myth emphasizes two misconceptions: (1) a human must perform the challenge, and (2) these problems are too complex for automated solutions. In truth, most spam senders ignore these CR systems because they do not account for a large recipient base, not because the challenge is difficult. Many spam senders use valid email addresses for their scams or for validating mailing lists. When CR systems begin to interfere with spam operations, spammers will automate the responses to these challenges.

Excuse me, what? Where's the proof? That's quite a brave statement to be making considering i've never seen this cracked, ever.

I challenge someone to find an automated response to C/R.

I did hear of a theory where C/R was being cracked by taking the C/R image, posting to a porn session, and letting a seeing person do the work. However, i've yet to witness this in practice. Show me the automated response to C/R that exists beyond a blog theory, and i'll believe. Until them, i hardly consider it "marketing hype".

Re:Proof?

[michaeltoe]

If a human can interpret an image and type in some dumb pieces of text, there's little reason to believe a computer program cannot do the same.

Re:Proof?

[LostCluster]

That's like saying a all theoretical attacks is not worth securing against somebody's fallen victim to it. Sure, there's some way-out ideas that can be dismissed that way, but this one seems so simple I'm pretty sure somebody who runs both spam and a porn site could pull it off...

Re:Proof?

[Anonymous Coward]

All right, the first comment is constructive!

Re:Proof?

[ender-iii]

Is this a joke? He just asked for proof and you got modded up by offering none?

Re:Proof?

[Anonymous Coward]

You've never heard of Optical Character Recognition?

Just because you've never seen the system cracked might merely mean that no one wants to talk to Simon Pooch, not that the system is uncrackable.

Re:Proof?

[ookabooka]

I cant even get my scanner to correctly identify a regular text document, it gets most of it, but it still misses a lot of letters. A computer program could do this, but you would need either a very large database of the letter pictures (most places use all different kinds of text pictures, and add in a degree of randomness). Or you would need a very developed algorithm to detect the letters (in which case you would be making oodles of money from the scanner industry. . . spam would be the least of your worries.
In the end i think it is inevitable that software will eventually break this system, but as soon as it does, there will be another system in place. . . .

Re:Proof?

[michaeltoe]

Yeah, true... I didn't really provide proof... but proof would require me to go out and program something fairly complicated, and I'm in the middle of midterm exams.

The point I was making is that, while noone has done it yet, there's no theoretical reason why it shouldn't be possible.

It's like saying "Oh, that mountain's to big, no one will ever climb it." -- If people are motivated enough, they can accomplish just about anything... and spammers seem clearly motivated.

Re:Proof?

[LostCluster]

Yes, but such a human-check is unlikely to be beaten by a computer 100% of the time. If a log of the failed challenge attempts is kept, the source of repeated failed challenges can be ruled out from getting any more challege attempts, or even just one failed challenge with hundreds of successful ones coming from the same IP space... then the hacker source cna be flagged and ruled out.

The best defenses involve several lines so that when the first gets beaten, another one tightens up against whatever the first line learned from its defeat...

Re:Proof?

[Anonymous Coward]

If a human can play Go there's little reason to believe a computer program cannot do the same.

Re:Proof?

[michaeltoe]

True, or you could just come up with an authentication method that doesn't involve visually identifying numbers and letters... like, showing a picture of an apple, and having people type in the name of the fruit.

Then again, that becomes less reliable and more ambiguous. You could keep on pushing for more difficult to interpret puzzles, and the technology to interpret it can just push back. People will just end up getting annoyed by it.

Sooner or later that idea runs out of gas... it's only a temporary solution.

Re:Proof?

[Anonymous Coward]

I hate to bust your bubble really, and this definately doesn't count for all C/R... but the CR you reference... take a sample of a few of the images.

Run a colour inversion on them (if you don't want to write the code, just use photoshop)... then equalize the image.

In every case i tried it on, the text came out pure white (FFFFFF), while no other piece of the image was white.

Its not much of a further step to abstract the white, place in on a black background and run it thru an OCR.

Re:Proof?

[SYFer]

Mod parent up.

His key point is that, if CR was good enough, we'd all be carrying pocket scanners and libraries would really be going online.

There are bigger "killer apps" for foolproof CR than spam-mailing. And I say fool-proof because to defeat this, its gotta be good.

I chuckled about that overrated post above where the guy essentially said "I'm too busy with mid-terms to write an app to accurately read the original "challenge" example. Dude, if you can do that, to hell with midterms--do it.

Re:Proof?

[silentbozo]

If a log of the failed challenge attempts is kept, the source of repeated failed challenges can be ruled out from getting any more challege attempts, or even just one failed challenge with hundreds of successful ones coming from the same IP space... then the hacker source cna be flagged and ruled out.

Unfortunately, this is one area in which the spam gangs already have a leg up on the rest of us. Trojaned machines provide them with a distributed set of machines (and hence, distributed set of IPs) from which to launch their attacks. While you may be able to block some zombies machines, there are many more from which the spammers can continue launching attacks, many of which overlap with IP space of actual (non-spam) users.

Unless you're being extremely unforgiving (in which case, you WILL get false positives), all the spammers will have to do is continue rotating machines to prevent exposing an IP long enough to get it blacklisted.

Re:Proof?

[jazman_777]

The point I was making is that, while noone has done it yet, there's no theoretical reason why it shouldn't be possible.

I think you have a future in marketing.

Re:Proof?

[Elwood P Dowd]

Challenge / response systems are broken anyway, even if spammers can't break it.

Why? Because from: is forgeable, and viruses use other people's real addresses constantly. Every day, one of my 40 spam emails is a C/R email from someone that I've never heard of. Am I going to click the link and authorize my email address? Fuck no. But I'll never be able to send email to that person. I realize that's a *tiny* incidental, but it's still broken by design.

If your C/R system includes a solicitation to purchase said C/R system, you're a fucking spammer. Fuck you.

Re:Proof?

[Anonymous Coward]

Thanks, that's a good one.

"I'd solve this open problem in AI (myself, apparently) but I'm in the middle of exams."

I hate people like this. "I can make something complicated enough to solve whatever you want." OK... Here: Beat Deep Blue at chess.

Some numbskull actually tried to argue with me that he could do this, by enumerating the game-state space. Pointing out that this would require more atoms than the universe probably contains, he revised his cunning plan. First, he would use randomness to play, until the state space became tractable.

No theoretical reason why it shouldn't be possible.

Maybe. And maybe not.

Re:Proof?

[michaeltoe]

As I stated in the beginning, if the human mind is capable of doing it, so should a computer...

We're not talking about astronomically difficult calculations beyond the grasp of any mathematical deduction... and we can infer this with relative confidence, given that your brain is doing these calculations even now, as you read this text.

I'll admit I was simply being humorous in implying that I, myself, if not burdened by exams, could slap something together. It would require a great deal of work to get this operate properly... but it is by no means an unreasonable goal. Science has tackled far more difficult problems than this.

Re:Proof?

[jmv]

This ought to be the most ridiculous C/R image I've seen. I've converted it to black and white simply by putting a threshold around the light gray (can be coded with 2 lines of C). From there, since all letters are identical, you can use simple pattern matching and get 100% recognition. The image is even saved as png to make things even simpler (jpeg would at least have introduced some quality loss).

Re:Proof?

[LostCluster]

If you're under attack from all corners of the web, then the only logical thing to do (and maybe you should have a doomsday script to recognize that) then the only logical thing to do is to do what the world would want you to do, block everyone from 0.0.0.0 to 255.255.255.255. "Sorry, DDOSers, we're closed. We'll be back when you all get patched."

Re:Proof?

[JavaGeek7654]

Simple convert to a two colour image and use any basic OCR system.

Re:Proof?

[thogard]

If I was going crack the example you gave is trivial. You start by reducing the image to grayscale, adjust hte contrast till the background stuff is gone and then you search out the outside path and record its top left point. You end up with a list of (x,y) and sizes, you sort by x and look up the path length in a table and the result is a letter. How hard is that?

Re:Proof?

[calambrac]

Actually, there are theoretical reasons why it shouldn't be possible for a computer to break these things, at least quickly enough to be useful to spammers. That's why these things are being used.

It's not just a matter of taking the time to pop out the code, either. Non-industrial grade commercial OCR software right now pretty well sucks. It can look at images rendered in black and white above 300dpi and give you back about 90% readable text, provided you don't care about formatting and there are no other foreign entities in the scan field. And it's not like these systems are weekend warrior projects. It's an active field of research.

It takes a while for a computer to recognize visual patterns. OCR uses shortcuts. It makes assumptions about fonts used, about letter positions, about possible words, basically, it makes the assumption that the text is a real message laid out in a way where it wants to be read. So when the image is being purposefully formed to trick the system...

Example: the link in the parent. You can't hone in on the shapes of the letters by finding boundaries between colors, because the background has all the colors of the letters, and all the letters are different colors, so you have to spend time branching off course, realizing you made a mistake, and backtracking... you can't predict the positions of the letters because they're all staggered randomly, so you have to spend time parsing the image to find concentrations of color... you can't use dictionaries to make predictions, because they aren't words, they're just random sequences.

It definitely possible to eventually get a proper translation of the image, but after how long? How does that help spammers, which is the point?

The porn site workaround is genius, though...

Re:Proof?

[calambrac]

Dude, your brain can figure out if a program will halt on a given input (obviously not all brains for all problems, but point still stands.) A computer can't, and will never be able to with our current models. It's called the halting problem, look it up. There are many problems like this... computers are great tools but they aren't the holy grail of problem solving. Sorry.

Re:Proof?

[chill]

How about legal challenge? If I remember, a couple of big ISPs are going through court on these -- violation of the American's with Disability Act. Specifically, the visually impaired have problems with this sort of thing.

Once enough people start using something like this, someone will successfully automate it.

In your example's case:
1. Convert to greyscale
2. Adjust threshold to 170/255 (66%)
3. Convert to 1-bit b&w
4. Remove letters left to right
5. OCR

I automated the first 3 steps in about 5 minutes, and it worked on 5 consecutive graphics from your CR. The last two require more effort than I want to invest right now. Considering all the letters in your CR are CAPS and the same font, you can replace step 4 with a script to move them to the same baseline. A good OCR package may handle it as is.

-Charles

Re:Proof?

[NarrMaster]

Humans can't solve the halting problem for EVERY case either. Halting Problem

Regardless, your point still stands. Pattern recognition for objects as rudimentary as letters is still a very hard AI problem.

Re:Proof?

[scheme]

Dude, your brain can figure out if a program will halt on a given input (obviously not all brains for all problems, but point still stands.) A computer can't, and will never be able to with our current models. It's called the halting problem, look it up. There are many problems like this... computers are great tools but they aren't the holy grail of problem solving. Sorry.

That's just so wrong. No one has shown that there exists a person that can figure out whether a program will halt a given input. People may be able to do subsets of these but no one can has been able to show that a person's brain can figure this out in all situations.

Incidentally, you can write programs that will analyze certain programs and figure out whether it will halt on certain inputs. Just because the general case hasn't been solved, it isn't necessarily true that specific cases are also unsolvable. A trivial example would be to check a program for a loop with an exit conditiion that can't be solved.

Re:Proof?

[TekGoNos]

like, showing a picture of an apple

Doesnt work either.

Except the problem that the whole world doesnt speak english / may be analphabet, there is a fundamental flaw.

As I said in another message, as the computer can not take the word apple and generate the image of an apple, you need a database of images classified by humans.

Nothing prevents the spammer from ripping all (or almost all) examples and then engaging some indians to classify them.

To prevent this, you would need the help of volunteers on the net that would at pictures at infinitum, so that your number of pictures isn't finite.

But then, nothing prevents spammers from filling the database with crap, rendering it unusable.

And if, instead of volunteers, you engage a lot of indians to add new images, the spammers just have to engage as many indians. Or, they really do the porn trick cited in the ancestor post : every time a new image is encountered by the bot, they put in on a porn page, requesting the visitor to solve the problem to get his free access pass.

But as the cost of taking the pictures, and then putting them into the computer, then classifying them is higher as the cost for the spammers to just classify them, I guess the big spammers would win this battle. (It might kill the basement spammers, but they aren't the mass of the problem.)

Re:Proof?

[calambrac]

I said, "not all brains for all problems", meaning the point of impossibility could lie in the particular brain or in the particular problem. A better way to make the point would have been:

There exist programs such that a human can say the program would halt on some input, where a computer would not be able to determine whether the program would halt under some input.

But despite the awkward wording, the point, that there are problems humans can solve which computers are proven incapable of solving, still stands.

Re:Proof?

[michaeltoe]

But despite the awkward wording, the point, that there are problems humans can solve which computers are proven incapable of solving, still stands.

I'm not entirely sure that it does... read the post about infinite loops above.

Re:Proof?

[Anonymous Coward]

Your Vegan cook books link is broken. I don't see any recipes to cook vegans in it. Those vegans are mighty tasty too, being corn fed and all. The problem is there's no meat on them because they can't eat anything but dirt and leaves.

Re:Proof?

[Anonymous Coward]

You miss the point.
- several of those example letters are unreadable to about 10-20% of male human eyes. They hit the range of color blindness.

- C/R is reactive and will never be popular in comparison to less restrictive technologies like standard SMTP. People just won't use them. Look at Earthlink, people don't turn the C/R system on because it interferes with mailing lists and there is too much false positive chance.

Re:Proof?

[cmallinson]

There is a difference here. OCR software, at least my OCR software, is better than 99.9% accurate for good text, and better than 95% accurate for really dodgy, faded text. It has to be, or nobody would use it. I have successfully OCRed text faxed from Chile on a fax machine from the middle ages - text that I could not read with my eyes.

Spammers don't need to get it right every time. Even if the the software can detect the letters in a C/R system 1% of the time, it would rake in the cash.

Re:Proof?

[Saltcreek]

I challenge someone to find an automated response to C/R.

Students at Berkeley have already beaten the C/R system setup by Yahoo! and with a selection of 191 different version of text obfuscation they were able to return a 92% success rate. In much more detailed images, with random background textures and overlaying text they were only able to achieve a 33% success rate but I am sure with time they would be able to do better.

In a paper published by Greg Mori and Jitendra Malik they explain the methods used to defeat the system. For the full write up you can visit their site on Breaking a Visual CAPTCHA

Re:Proof?

[eddeye]

Where's the proof?

The last time I checked Yahoo's account creation challenge-response system was vulnerable to a trivial attack. From my experiments, the challenge field suffers from these problems:

• No association with the http page request which generated it
• No usage frequency limits
• No expiration time (at least, not shorter than 1 month)

This allows one to build a dictionary of valid challenge/response pairs which can be reused indefinitely. I did this successfully with a perl script two years ago (I study security). I haven't tested the script since then but the system doesn't appear to have changed.

Granted this is an implementation flaw which can be corrected. But how many challenge-response systems out there suffer from similar problems? I'd wager quite a few.

Re:Proof?

[hermooz]

Show me the automated response to C/R that exists beyond a blog theory, and i'll believe.

This "strenghtness" of C/R technique is also their biggest weaknessness. This is a principle the writer of the article stresses, but apparently does not got on him that C/R systems, just like many other suggested theories, is more detrimental to legitimate user than to spammers. Ever thought of visual impaired people, or with other disabilities? They RELY on systems that do automatic reading of digital documents, and often they're not even so good. If a C/R defeats automatic reading, it locks out those users.

Sure, numerically these are almost unsignificant cases. But in principle? We are talking of systems that *maybe* could stop an illegitimate use of a media, but *surely* damage persons that already have their share of discriminations, difficulties, and so on.

Under a marketing point of view, one could accept to lose an handful of user if this permit him a monetary benefit: so if Hotmail uses a C/R system to register accounts, nobody can condamn it. But are we talking in introducing a C/R system in the protocols for sending mail? This would render the protocols themselves not usable by certain category of legitimate users.

Definitely NOT a good thing.

Re:Proof?

[chrisbtoo]

Well, this is by no means a proof, but maybe a method.

1) Get image. I followed your link and got given this image.

2) Pre-process. I loaded it into the GIMP and did Image->Mode->Greyscale, which yielded this image. Then I did Layer->Colours->Threshold, which yielded this image.

3) Match characters. At this point, you have a monochrome image, in what appears to be a known font. The chars don't even appear to overlap, so a simple 1-for-1 match is achievable. Scan left-right, top-bottom until you see a 10x10 (or whatever) section with a black pixel. Scan down and right from that pixel until you see a character.

I don't have the time to code it up right now, but if someone wanted to pay me to do it, I'm pretty sure it's acheievable - not least because a whole bunch of the more difficult code is available for me to use under the GPL.

Re:Proof?

[chrisbtoo]

I don't have the time to code it up right now, but if someone wanted to pay me to do it, I'm pretty sure it's acheievable[sic]

Of course, I don't mean I want paying to do it. I mean that someone who wanted it cracked might choose to pay someone to do it.

I might actually have a go later if I get the time.

Re:Proof?

[Anonymous Coward]

1. Yes, but such a human-check is unlikely to be beaten by a computer 100% of the time.

Spammers already play the odds. They will just tweak tactics and increase the spam if they get the same number of suckers.

Re:Proof?

[soupman]

This appears to me to be a sender-created white list. What if a virus arrives from this sender? Do you accept it as a good, not junk mail?

I figure validation should be based on the email address or domain, and the IP address(es) of the sending server. The incoming mail server should have a white list of addresses and domains and there should be a list of IP addresses associated with each address and domain.

That way if Aunt Mary who lives in Smalltown, USA, sends an email from somewhere in Europe (no offense meant to our European neighbors), you'd figure something is not right. Also if an unknown person is sends an email from Aunt Mary's computer, you'd figure something again is not right.

Granted some large ISPs have many sending IPs. Maybe they should post a list of those IPs, so that email admins can copy/paste. Maybe email admins could tell their mail server to fetch a validation list from the MX server(s) of a domain. This being done without changing the SMTP spec.

Of course, this is all talk about connection filtering. Content filtering is handled well IMO by SpamBayes.

Re:Proof?

[EvilAlien]

Actually, that would be an incorrect assumption. Its a fairly common misconception that the everyday tasks our brains perform are "easy", whereas the tasks that computers perform are "difficult". The normal situation is that computers perform very simple computational tasks extremely quickly, but complex cognitive, perceptual, and interpretive tasks performed by the human brain and very difficult to get a computer to do.

The vision challenge in robot design is perhaps the best example of something we take for granted by is very difficult to accomplish with silicon.

Re:Proof?

[drooling-dog]

Having done a lot of image analysis work in the past, I have to say that this doesn't look very difficult. The "noise" in this image consists of vertical and horizontal lines of various colors that traverse its entire extent; removal of these is trivial. The characters themselves are very clean and regular. The varying Y-offets add no complexity at all, because it's only the X dimension that matters. This could be cracked in a day by anyone who cares enough to spend a day doing it.

What you need is some non-transparent dependency between the contents of the image and the instructions that go along with it; i.e., the proper interpretation of the image shouldn't be obvious without the instructions. Of course, if it's too complex then some people won't be able to send you mail, but maybe that's not always such a bad thing...

Re:Proof?

[Progman]

Except the problem that the whole world doesnt speak english / may be analphabet, there is a fundamental flaw.

Being analphabet is a serious drawback to reading email, with or without a CR system.

Re:Proof?

[Arslan ibn Da'ud]

> I challenge someone to find an automated response to C/R.

I challenge someone to find an automated response to all types of C/R. That's what spammers would have to deal with.

Alice's C/R system makes you enter text hidden in an image. Bill's C/R system makes you click on a keyhole. Charle's C/R system simply asks you what's 2+2?

Each of these systems can be automated by a determined spammer. But is there any spammer that can automate all three? How about one that can automate all of the possible challenges? That will foil the spammers.

Once again, we will find salvation in diversity.

Re:Proof?

[BobTheLawyer]

don't challenge systems also force spammers to provide their real e-mail address? Isn't this an additional advantage?

Re:Proof?

[scrytch]

> I challenge someone to find an automated response to C/R.

I challenge you to justify using C/R, which abuses everyone who has their From: address forged. It's very simple, either find a way to implement C/R in-protocol, or don't use it at all. Once you have accepted the message, it is too late to go back.

And when I was working at a helpdesk and had to jump through all these little C/R hoops to respond to a customer ... I worked those tickets dead last. Might be days before I get to you. You mail someone, you damn well better pre-emptively whitelist them out of the C/R loop.

Re:Proof?

[Arslan ibn Da'ud]

> Challenge / response systems are broken anyway, even if spammers
> can't break it.
>
> Why? Because from: is forgeable, and viruses use other people's real
> addresses constantly.

So Alice spoofs a mail from Bill to Charlie. Charlie's C/R system
falls for the spoof and challenges Bill's system. Bill's system,
formerly oblivious, doesn't recognize Charlie and issues a challenge
to Charlie's challenge. Whereupon Charlie's system re-challenges
Bill's system, Bill's system re-challenges Charlie's system, and...

Obviously a simple 2-way deadlock.

The way to break this deadlock is for a C/R system (Bill's or
Charlie's, doesn't matter which), to silently ignore identical
mail. That is, if Bill's system gets a 3rd mail from Charlie's that is
identical to the first two, it doesn't issue a challenge, but ignores
it.

Since you never have to worry about 3-way or N-way where N>2, problem
solved.

Re:Proof?

[Elwood P Dowd]

No, you completely misunderstand my point. I do not suggest a deadlock:

Alice spoofs a mail from Bill to Charlie. Charlie's C/R system challenges Bill. Bill has no C/R software.

1) That is an unsolicited commercial email, from Charlie to Bill, and is unnacceptable (especially if the challenge includes instructions on how to purchase the C/R system, as most do).

2) Bill will not respond, and thus, depending on the C/R system's design, may not be able to send legitimate mail to Charlie in the future.

Re:Proof?

[michaeltoe]

I guess the whole reason I even posted the original comment was that my argument is more on a theoretical level than anything else... I felt it was obvious that that's the argument I was making.

Quite frankly I don't care about spam, nor about whether or not spammers find a work around. All the arguments about how that workaround may be so costly as to deter the spammers... well sure, that may be true, and if it is, fine.

What I am opposed to is the assumption that these problems are somehow impossible to tackle, and they aren't. I'm not just speculating on this issue, as there is evidence to that effect all around us. You're doing these calculations right now, just as you're reading this post. You have to be doing it with a relatively high rate of speed as well, otherwise reading would be impossible.

We can discuss the differences between how the brain works and how a computer works for as long as you want, but one fundamental fact remains true; your brain isn't a magic box. To try and argue that a computer cannot perform the same visual recognition (albeit, perhaps slower) is exactly like arguing that your brain is beyond mathematical definition. It's an argument I don't buy, and it's hardly supported by science.

Re:Proof?

[kindbud]

I challenge someone to find an automated response to C/R.


if ( message looks like C/R ) {
discard;
stop;
}

Re:Proof?

[Henry Stern]

Paste into matlab:

I = imread('http://si20.com/img/authimage?seed=63419&u serid=1');
J = double(I(2:31,2:161));
hl = nncopy(J(:,2),1,size(J,2));
vl = nncopy(J(1,:),size(J,1),1);
J(find(J == hl)) = 0;
J(find(J == vl)) = 0;
J(find(J)) = 1;
spy(J);

Tada! Use a simple hamming distance-based nearest neighbour matching algorithm to find the values of the individual letters.

Re:Proof?

[Anonymous Coward]

"Excuse me, what? Where's the proof? That's quite a brave statement to be making considering i've never seen this cracked, ever."

http://www.duo-creative.com/chrisb/authimage/

Re:Proof?

[Anonymous Coward]

Machine attacks are documented, for instance,
Greg Mori of UC Berkeley Computer Vision Group has some pages outlining a
method used to break Gimpy, the CAPTCHA used at Yahoo!
http://www.cs.berkeley.edu/~mori/gimpy/gim py.html

There's a news article here:
http://www.siam.org/siamnews/11-02/gimpy.ht m

I think the bottom line is that "easy" CAPTCHAs are likely to be vulnerable
to machine-based attacks. It's clearly possible that "hard" puzzles may be
susceptible to what Luis von Ahn at CMU has called "Stealing Cycles from Humans" (or in
the case of the pay-with-porn approach, "buying cycles"), however, the
economics of this kind of attack have not been fully explored. We don't
have any firm evidence for an actual exploit "in the wild".

Re:Proof?

[thetaikung]

I hate people like this.

I hate people like you. "You're wrong, the universe isn't infinite so I will now post an anectdote about how numbskulls come up with ideas that are maybe or maybe not possible."

Nothing really works 100%

[Espectr0]

There is no anti spam technology that actually works. Not even whitelisting, because those viruses fake email addresses.

Maybe whitelisting with custom mail headers to prove identity

Re:Nothing really works 100%

[Anonymous Coward]

i'd settle for 99.9% -- i'm flexible!

requiring incoming emails to show authentication -- obviously the header ain't that -- would work, but what a pain. all email is currently built around this low-security trust system that was ok among academics but has long since been exploited to the point that more of email is junk than not.

the solution i'd like is AI that would read and rate my email for me. it might zap some of the stuff from my friends, but maybe that would be doing a favor, too -- who needs another mass-mailed joke list?

Re:Nothing really works 100%

[root-kun]

Thats what alot of theyse bayesian analyzer attempt to do. They statistically learn your patterns by what emails you like and what you dont like, and then try to "intelligently" discard the bad ones for you. I mean obviously the worry exists (mostly for companies) that good email may get stopped, but in my experience its very uncommon, aslong as the user has taught the spam bot/blocker properly.

Re:Nothing really works 100%

[silentbozo]

Well, to prove identity you could cryptographically sign mails. When the recipient gets the signed mail, they do a key lookup and verify that the signed mail was signed with the correct private key.

Now, how do you handle the situation where spammers are generating thousands of keys? Well, the spammers are forced to waste some cpu time, but that's trival for them. They're also polluting key registries with their garbage - that's a big negative.

However, in terms of trustworthiness, the spammer probably hasn't gotten all his keys signed by somebody else who is of a "trusted" ranking. Even more likely, much of the signed mail you do get will either be known to you (ie, you've signed their keys) or will be known to people you know (ie, someone you know has signed somebody else's key.)

Mind you, this is no replacement for other types of filtering (ie, SpamAssassin with Bayes, etc.) but it would make whitelisting useable against spammers who forge e-mails, UNLESS the spammers know the private key of the poor slob that they're impersonating.

Re:Nothing really works 100%

[Roman_(ajvvs)]

...UNLESS the spammers know the private key of the poor slob that they're impersonating.

So you're saying it's not 100% effective? isn't that the whole problem? If that's the loophole that breaks this system, you can be sure a spammer will attempt to use it. They already break several laws and ethical boundaries. What reason would they have not to attempt getting people's private keys?

Re:Nothing really works 100%

[Skim123]

What happens when someone on your whitelist opens an attachment that automatically sends email from their account, signing it? Now you have a spam that has been legitamately sent from your friend's account.

I created a C/R anti-spam system myself, but gave up on it and turned to Spambayes for two main reasons:

1.) I was losing challenges in others' spam filters
2.) I would still get emails from whitelisted folks when they were infected with an email worm.

If you're interested, I blogged about my switch from C/R to Bayesian filtering here.

Re:Nothing really works 100%

[Red Alastor]

I use popfile as a bayesian filter. My mails are classified into 12 categories (spam being one of them) and my accuracy rate is 98%. Most of the time, the misclassifications aren't about mails being wrongly put in spam. Bayesian classifications are *highly* effective. And they also easilly spot mass mailed jokes and the various hoaxes that you got forwared all the time. You can get popfile at this address : http://popfile.sourceforge.net/ If you use a pop3 server, it works no matter what mail software you use.

Re:Nothing really works 100%

[iamcf13]

My program CF13(TM) would render such malware email attachments safe to handle no matter who the sender was. It would also detect virtually all spam and treat it appropriately.

Re:Nothing really works 100%

[0x0d0a]

Compromised hosts are ultimately lost. You can't "fix" a compromised host. Even a C/R system can be beaten (especially if C/R becomes commonplace and there is standardization on one system) by making (usually) an image recognition system or by having bogus "legitimate" emails come in to a user so that he thinks that he's doing C/R on a legitimate email when he's actually doing it on a spam. Say it starts eating all his email and replacing it with ads...

Also, the usability problems inherent in a non-whitelist C/R system are too overwhelming for me to deal with them. I just refuse to jump through a hoop to send each email.

Re:Nothing really works 100%

[hkmwbz]

There are spam solutions that work, for certain values of "works" :)

It doesn't have to be perfect to make spam less intrusive. It would be great to have a device which detected the location of the spammer and sent in a gang of hired goons to off him, but until someone develops such a solution, there are things that are available today that more than takes the load off your mailbox, and reduces the annoyance somewhat.

Re:Nothing really works 100%

[Anonymous Coward]

What reason would they have not to attempt getting people's private keys?

They don't have to "get" anyone's private keys. All they have to do is what they do already - get access to someone's email (MUA) program, then use their address book and send out copies of themselves (the virus/spam) to everyone in the address book because they can use the email program to generate signed emails.

Re:Nothing really works 100%

[Roman_(ajvvs)]

that would be a reason, indeed. :)

I suppose It's using the "path of least resistance" when it comes to getting addresses...

Re:Nothing really works 100%

[Anonymous Coward]

All they have to do is what they do already - get access to someone's email (MUA) program, then use their address book and send out copies of themselves (the virus/spam) to everyone in the address book because they can use the email program to generate signed emails.

True enough. I guess it's too much to expect that people will use crypto unless forced, and even then they'll be lazy enough to just leave the password for their private key embedded in their mail client. :(

Oh Well

[dirkdidit]

With the way the Chinese government keeps making their own versions of everything, maybe they'll have their own version of the Internet. That shoud alleviate a good deal of the spam right there, given that their Internet will probably be incompatible with ours.

Re:Oh Well

[_Sharp'r_]

The Chinese government will probably solve any internal spam problem pretty quickly.

I mean, if you start by shooting all convicted spammers, the profession tends to stop attracting replacement members.

Re:Oh Well

[Anonymous Coward]

You: Comrad, it was a virus.

Them: you got a virus?

You: Yes, yes, it sent the spam!

Them: And what "Authorized" program did you get that from?

You: I...umm...

Them: When did you stop going to porn sites?

You: No, no, you misunderstand.

Them: You never stopped?

You: yes...I mean..No I never went.

Them: I'm wrong?

You: ahhh, umm, it was because I was using that American scum software from Microsoft!

Them: You admit you're violating the quotas and exposing us to spam?

You: Well, it was just for games.

Them: You're saying our software isn't good enough?

Them: Just sign this confession.

You: And if I don't?

Them: You'll be shot.

You, signing: There!

Them: Perhaps if you had been more cooperative, a re-programming center would have been adequate. But, you'll prevent more spam this way.

Re:Oh Well

[shlomo]

better yet, issue hunting licences...I'd sign up

Re:Oh Well

[Anonymous Coward]

I mean, if you start by shooting all convicted spammers, the profession tends to stop attracting replacement members.

That is the solution exactly! - Make the penalty for repeated spamming a death sentence and we'll quickly get rid of the spam manace for good.

It really isn't hard to identify spammers. Just go to SpamHaus.org at take a look. That archive is full of evidence against the few dozen spammers behind almost all spam on the planet. Just pass the death-to-spammers law, round them up and take them to court, convict them using all this evidence and put them (and thus us) out of our misery. Spam problem solved!

Re:Oh Well

[alan_dershowitz]

Maybe not. It's hard to trace email, so its hard to catch spammers. The death penalty in the USA doesn't tend to be a real effective deterrent to crime because people don't think they are going to be caught.

Minor quibble, you probably don't even need to be convicted before you're dispatched in China.

Re:Darth Vader

[Anonymous Coward]

pishhhhh *breathe*
I find your lack of junk mail disturbing.

Don't forget SMTP+AUTH

[RT Alec]

Good overview, all things considered. I would like to add to one of his conclusions (from part 1):

IMAP can be used with SSL and supports secure authentication, but not all servers support this. SMTP also supports SSL or TLS but again, many organization's servers do not support this or use only server-side certificates.

This conclusion is correct, but why is this considered a stopping point? Mail admins-- get off your collective butts and add encryption and authentication to your mail servers! The author also forgot to mention that server side certificates are not necessary for SMTP, SMTP+AUTH addresses this quite nicely.

Note that such measures are not necessary for most users. Home users that use their ISP's mail server don't have to implement any of this, since the ISP can already account for the user. Let us not forget that "most users" do not have the e-mail needs that many Slashdot readers do. For those needing roaming access and multiple addresses, use IMAPS and SMTP+SSL+AUTH.

Re:Don't forget SMTP+AUTH

[KingJoshi]

I know I'm ignorant on this topic, but I don't see how spam has spread to such a problem without acceptance from ISPs and systems/network administrators for companies that provide bandwidth.

TCP/IP is *flawed* in that it allows you to fake your IP address, but why hasn't more (all?) networks not allow for packets that fake outside of their subnet? Why don't mail servers authorize their users?

The net allows for anonymity only if you allow it. You can always check whom you're connect to at your link. If there are bad links, then they specifically should be blacklisted. Like everything else, the problem is lazy/greedy humans. And that hasn't been a problem we've yet to solve.

Re:Don't forget SMTP+AUTH

[xpl_the_myst]

That is a good point - that it wouldnt take much for ISPs and major mail providers to set up a Public Key Infrastructure.

In fact, it is the *only* reasonable method out there because of the one interesting point of the article - Whatever kind of challenge response or computational puzzle techniques are used, it means that mailing lists will have to do the same to send mail to their users. You could add the mailing list to the list of accepted senders, but guess what, there is no guarantee that the from address cannot be faked. The only solution, it appears, is for everyone to use a strong form of identity like public keys.

There was a time when IP's were identity enough ... gone are those days ...

Re:Don't forget SMTP+AUTH

[Anonymous Coward]

it wouldnt take much for ISPs and major mail providers to set up a Public Key Infrastructure...In fact, it is the *only* reasonable method out there

Cryptography has plenty of problems--most of them addressed in the article. I certainly don't want some centralized certificate authority dictating who can or can not send email, or for that matter dictating who can or can not compete in the email software and services market.

Even if you managed to get everyone to use authenticated mail, spammers would continue their old ways of taking over people's machines. Then, spammers would send mail under innocent people's names.

Re:Don't forget SMTP+AUTH

[Anonymous Coward]

The net allows for anonymity only if you allow it.

I want it to. I consider privacy very important. The more anonymity there is, the harder it is for websites, ISPs, the government, etc. to track people.

Anonymous speech is also critical to a free and open democracy. Anonymous speech has played a key role in US history. Some of the articles of confederation (those were a series of newspaper articles written to build up support for the US Constitution) were published anonymously. Deep throat (the anonymous whistleblower in the Nixon scandal) also maintained his anonymity.

Re:Don't forget SMTP+AUTH

[Anonymous Coward]

That is a good point - that it wouldnt take much for ISPs and major mail providers to set up a Public Key Infrastructure.

The article addressed this. Spammers (or script kiddies, or foreign governments, or...) can DDoS the root PKI servers and stop legitimate mail from getting to anyone.

Re:Don't forget SMTP+AUTH

[zcat_NZ]

TCP is NOT flawed. Sure you can spoof a packet or two, but (assuming reasonably strong sequence numbers) you can't fake a whole connection unless you are actually getting the reply packets.

mail is likewise not flawed; It is fairly hard to find an open relay these days; it is all-but-impossible to find one that doesn't put your IP address in the headers. That's your _REAL_ ip address. The one that ends up in RBL's so nobody accepts your mail any more.

The big flaw is home users; they keep getting pwn3d. And you can't even blame Microsoft for this any more. The viruses are arriving as a zipped, passworded attachment FFS. We've long since passed the realm of just clicking on an executable!

Here's how I see it; the antispam community were on the right track from the beginning. Blacklisting has made it impossible for spammers to spam from their OWN connections, even overseas, and pushed them to finding home users (to spam from, or to attack the blacklist sites). Now they're talking about changing the entire mail system, persuade thousands of users to change the way they do email? Hell no, we've almost won. We just need to educate enough END USERS not to get pwn3d, with the result that the DDoS attacks get cut down and the remaining much smaller number of spam sources can be more efficiently blacklisted.

Or we can force one more 'wafer thin' kludge onto the entire mail system, which the spammers will just find a way around next week anyhow.

Re:Don't forget SMTP+AUTH

[Anonymous Coward]

I bet that within a year from the date a yahoo "domain keys" type system is put in place, it will be used for billing senders for email and will, astonishingly to everyone [except yahoo and MS, who know what it is for], still not decrease the flow of spam and viruses.

As discussed here many times (whenever the /. editors want to see us do the "spam solution" dance), domain level keys are not viable for many reasons.

So, I guess I am joining you on the dance floor...

Cut Your Junk Mail By 50% !!!

[Snagle]

Just buy porn in magazine format instead of registering for it online :)

Re:Cut Your Junk Mail By 50% !!!

[redJag]

What is this buy? *squints suspiciously*

Re:Cut Your Junk Mail By 50% !!!

[Anonymous Coward]

If you use your primary email address when you register for anything online (yeah, I know there's some exceptions) then you are just bringing the spam upon yourself.

Re:Cut Your Junk Mail By 50% !!!

[Anonymous Coward]

I think he also misspelled pr0n aswell.

Solution: Stop Spam at the Source

[ElliotLee]

According to the article, there is no good lasting solution to spam. Indeed, there isn't, but we need to consider more the reason behind the spamming.

Why has spam grown to what it is today? It is an undeniably effective means of cheap marketing. What we need to do is come up with a way to stop this not on our end, but by looking at as a social problem or making it non-worthwhile to the spammers. If nobody ever responded to spam, spammer wouldn't bother.

Re:Solution: Stop Spam at the Source

[Texodore]

Why would we do that? We never stopped Columbia from producing cocaine, and we're not stopping Afghanistan from producing heroin. In this country, we attack the end result, not the source.

There's more money to be made by allowing spam to occur via spam fighting tools, bigger Internet pipes, advertising through spam, you name it.

Re:Solution: Stop Spam at the Source

[Vellmont]

Yes, I'm sure the kind of people that respond to penile enlargement, viagra, and porn advertisements will certainly be shamed out of responding to spam email from television ads. I'm sorry, but this has to be about the most ineffective method of stopping spam I've heard of.

Re:Solution: Stop Spam at the Source

[An Anonymous Hero]

What we need to do is come up with a way to stop this not on our end, but by looking at as a social problem or making it non-worthwhile to the spammers. If nobody ever responded to spam, spammer wouldn't bother.

Hey, an automatic 10 years in Guantanamo for anyone convicted of replying to spam. Come to think about it, that would be a solution :-)

One approach I'd like to see discussed is as follows:

1. "Unsolicited bulk email messages" are allowed.
2. However, as a condition, all the addresses contained in such amessage (SMTP From:, Return-path:, From:, Reply-To:, SMTP server used, and all URLs in the body) must be from the same domain .
3. People may redirect any mail they consider non-compliant to (e.g.) abuse@their.isp. ISPs poll the results.
4. Any SMTP server associated with more than N complaints gets blocked.

Re:Solution: Stop Spam at the Source

[kent_eh]

If nobody ever responded to spam, spammer wouldn't bother.
...and if no one clicked on executable attachment viruses/worms would stop dead in their tracks.

User education is obviously the answer to both problems, but for all the information that is thrown at users, these are still running rampant.

If anyone has a solution to get people to actually *pay attention* to the information they are given, then maybe we would have a solution .

Re:Solution: Stop Spam at the Source

[m00nun1t]

"If nobody ever responded to spam, spammer wouldn't bother."

I've heard this argument a few times, I'm not sure that I buy it. Yes, some spammers would stop - consider the article on Balan where he was keeping the money from his spamming activities. No one clicks, he stops, simple. But I suspect a lot of spams are operating as an "agency" - eg, "here's $10,000, please send out a spam promoting my site SexyHorses.com". That spammer doesn't care if no one clicks. True, SexyHorses aren't likely to use his service again, but there's a sucker born every minute.

So, my guess is not clicking will reduce but not eliminate spam.

It Was the Emporer's Death Star, not Darth Vader's

[Anonymous Coward]

n/t

Deterrents

[erroneus]

At this point in the game, I am honestly surprised that we haven't heard of violence resulting from spam affliction.

I don't know about anyone else, but I'm pretty sure I'm not alone in this. I have, at times, felt utterly enraged at all the spam flying about and further all of the innocent and naive people that are being abused by all of this.

I know if I feel violent internally, then surely there are those with less self-control out there who will eventually act on his or her rage... perhaps the parent of a child afflicted with porn spam?

I think if two or three spammers are attacked physically, it might give them pause. Frankly, I'm amazed it hasn't happened.

Re:Deterrents

[LostCluster]

Of course, the worst spammers make it impossible for the average user to ever identify the true source. I guess you are just giving them another reason why they need to do that.

Re:Deterrents

[Anonymous Coward]

Many of the people who are angry at spam are not angry because of how spam affects them but because of how it affects others like children and the elderly. Spam makes people indignant instead of angry; when someone is indignant, they tend to react rationally.

Re:Deterrents

[AndroidCat]

Type "two dead spammers" into Google. You might even get a link back to Slashdot where it was covered. (Stock spammers likely killed by their business .. partners.)

Re:Deterrents

[Killswitch1968]

There was a case where a man verbally abused a spammer. I think he was actually prosecuted for it. Oh I wish I had the article.

Re:Deterrents

[Elwood P Dowd]

At this point in the game, I am honestly surprised that we haven't heard of violence resulting from spam affliction.

I'm surprised you haven't heard about it either. Some senile twit that got defrauded by a Nigerian "409" scam email figured that all Nigerians were in on the scam, or something, and killed a Nigerian diplomat.

Obviously, not what you were talking about: it was fraud more than spam, and the spammer didn't suffer, but... that's certainly violence resulting from spam affliction. (Also, note from this article: According to State Department figures (PDF), 25 murders or disappearances of Americans abroad have been directly linked to 419 fraud.)

Re:Deterrents

[Anonymous Coward]

1. Spam Company Inc sends spam. 2. Visa and Mastercard denies Spam Company Inc's ability to process credit cards on the internet.

Re:Deterrents

[dave420]

If spam makes you feel violent, maybe it's time for some therapy? Seriously... it peeves me off, but it has never caused violent feelings...

Open Relays

[QuePasaCalabaza]

The truth is 90% of spam comes from open relays, that is SMTP servers that can be tricked (a bit like lying to a 5 year old) into accepting and sending out massive ammounts of mail. Simply blocking open relays using The Open Relay Database at http://www.ordb.org/ or other open relay checking utility will save you lots of time if you run your own mailserver. When we can bascially negate the usefulness of open relays to spammers, they will then have to rely on their own bandwidth for the most part providing they cannot comprimise other "closed" relays.

Re:Open Relays

[SSpade]

The year 2000 called, they miss your opinions.

In other words, your data is so out of date as to be positively misleading.

Open relays are dead. Open proxies are so 2003.

All the cool kids are using virus distributed trojans these days, some of 'em proxies, some dedicated spamware.

Re:Open Relays

[Anonymous Coward]

Please. Use it as a target list. We can test next generation cruise missles against it.

Re:Open Relays

[AndroidCat]

Some of the dedicated spamware probably acts like a relay. (Why waste time using an 0w3n3d box like a socks-puppet? Let it do all the work!) Certainly not an open relay, of course. The tiff between the various MyDumb-type trojan groups might be because they're locked out of each other's relay-farms.

Re:Open Relays

[yelvington]

Open relays -- legitimate mail handlers that were misconfigured -- used to be major problem. My mail stats show that's not the case any more. I use ORDB. A year or so ago, it stopped a lot of spam. It hardly ever shows up in my logfiles these days.

Most of the spam I'm seeing now is coming through virus-infected computers. These aren't supposed to be mail relays at all. They're generally Windows PCs on cable modems.

Viruses install trapdoor relays on these machines, accessible only by the virus writers, who are in league with spammers. These virus relays aren't being detected by ORDB checks. They may not even be running SMTP on the receive side, for all I know.

I've also noticed that these virus-spammers tend to not use harvested email lists, but rather employ the equivalent of a dictionary attack -- sending email to randomly chosen common first names: anna, brenda, dave, ted, andrew, sam, joe, and so forth. If I were ambitious enough I'd wire some of these bogus names to firewall triggers and automatically blackhole the senders.

Re:Open Relays

[bishopi]

Why am I cynical that it's still 90%?

My bet is that the bulk of it is now going through worm-riddled zombie mail-relays.

Ian

Re:Open Relays

[Anonymous Coward]

bullshit.

2/3 of spam is sent from virus/trojaned PCs.

Re:Open Relays

[Anonymous Coward]

I hear the cluephone ringing for you.

Re:Open Relays

[iamcf13]

Why not go one step further and treat email from unapproved senders sent through a 3rd party machine as spam. Doing that should drop your spam down to almost zero--consiting of dumb spammers using stolen/throwaway accounts at a domain running their own mailserver(s) or worse, compromised user PCs acting as 'zombie relays'.

But then, for example, my program (see signature) would examine the content of such messages and likely deem them spam anyway.

Re:Open Relays

[DrPepper]

As everyone else says, open relays no longer seem to figure much in the spam we receive here. I thought I'd link through to the stats to show the point, although (to my surprise) they seem to show just as many relays as ever (best graph at the bottom).

I suspect therefore that open relays (and proxies) are still an issue, but the spammers just don't use them as they are all blacklisted anyway.

"0wning" a machine seems to be the current way to do it. It really needs the ISPs to cut off any machine that has been 0wned immediately, and then contact the subscriber for some LART'ing. Unfortunately, at least on my ISP, they leave the user connected and then contact them later on; weeks later, if you are lucky, the user will fix their machine.

Filter them out at the client

[Anonymous Coward]

Client-side filters are getting much easier and better to use. Check out Ella for Spam Control. Unfortunately it only works on Outlook and Outlook Express, but it doesn't require endless configuration and tweaking because it uses adaptive learning technology.

Let's use the Patriot Act for the benefit of good

[mao che minh]

I am in full support of using the broad-powered, freedom crushing Patriot Act in apprehending and imprisoning spammers. We might as well get some good out of it.

Must be adaptive and pre-emptive

[fembots]

According to this site, spammers are constantly changing tactics to deal with anti-spam filter. So the site is trying to use human to catch any changes in spammers' styles.

So, are spam filter changing at least as quickly as spammers?

white list / web of trust similar to PGP?

[newdamage]

A nice fool proof system, while a bit of a hassle, would effectly remove spam. PGP uses a white list of sorts, that only allows people to send you encrypted messages that have your public key. This in a sense could be done with email. Someone wants to send you an email, and has your email address. They send the small request to your mail server (1-2 KB in size) with their name, email address, and name of their mail server. The mail server holds this information and notifies you that a new sender is awaiting access. You then:

1. Verify the identity of the sender, okay then, and the sender is then given the return request, and is notified that they will be allowed to send emails.

2. Deny the sender, and all their emails will be bounced back.

Yes, spoofing problems still exist, but this system could be expanded, and guess what, you only recieve email from people you want to, and the mail server acts at the first point of defense.

This would require more complex and smarter mail servers, but it would make the every day user's life so much more simple.

Re:white list / web of trust similar to PGP?

[Kwil]

Not to mention slow email to a crawl.

Picture, some guy you've never heard from before sends you an email.. is it legit? Check with your boss to see if they're working with anybody new and have referred them to you.. if they have, then check if this guy is the one? Yes? Okay.. accept him then.. now wait for what the email was in the first place.

Or, even worse when you're expecting email from a new group of clients, but don't know their email addresses?

Plus, it does absolutely nothing to deal with someone spoofing someone on your white list.

So it's not "you only receive email from people you want to" it's, "you only receive email from people you already know the email addresses of" very different thing.

Re:white list / web of trust similar to PGP?

[00420]

I really don't know anything about the technicalites of this, so this suggestion could be worthless but...

Couldn't it be set up so that when you know in advance you're going to be doing business with somebody you could pre-authorize them if you know the pertenant information.

I would imagine companies could then have the IT department authorize any new companies/persons who they would need to be doing business with. Then if somebody who is doing business with your company gets referred to you, there would be no need for another authorization.

Obviously this wouldn't work in all situations (how many things in life do work in "all situations"?) but I would imagine it would help in many situations.

I know if I could do that with my personal email I would.

Re:white list / web of trust similar to PGP?

[Anonymous Coward]

Right so now the offer for cheap penis enlargment pills comes in as a 1-2kb request for permission to send you mail!!!!!

How about a password system?

[Anonymous Coward]

Why couldn't you set up some kind of email password system? You would need a password of sorts to send an email to someone, and the passoword would be checked by their mail server. Anything that's good goes right throught, and anything that's bad goes to a separate folder just in case you ever need it.

Re:How about a password system?

[FLEB]

I do this on USENET. I have a spamtrap address set up, with an instruction to put "Hey!" in the subject line. Anything with "Hey!" gets bumped up, anything without gets trashed.

There's one billion people in India...

[3770]

that the challenge/response could be outsourced to.

Only kidding (I think).

Re:There's one billion people in India...

[fembots]

This might be a joke now, but it may well happen in the future if we're really into this C/R thing.

At the moment spammers are already paying people to send emails from home, obviously it is profitable enough to pay someone to do the dirty job for you.

As a result, if recepients are less defensive against spams in a C/R system, those slipped spams might get a greater response rate. And this is good news to spammers, and they might very well be able to afford to outsource to deal with C/R.

Re:There's one billion people in India...

[psichaotic]

yeah but who says they would get it right

Re:There's one billion people in India...

[Kallahar]

The topic of a message is supposed to be the TOPIC, not the first sentence of the body.

Re:There's one billion people in India...

[Anonymous Coward]

they are smarter than you.

Re:There's one billion people in India...

[psichaotic]

at least im smart enough to know how to login to slashdot

Good old fashioned riddles

[KalvinB]

My free anonymous (as in they can only be traced back to a common e-mail account on my server) e-mailer uses a simple quiz to keep spammers out.

The form page records the IP address of the visitor along the with the question number they were given in a file named with the IP address. That number is never sent to the client. When they hit submit the file of their IP is opened, the question number is read in and the answer given by the user is compared to the stored answer. The file is then deleted and if the answer was correct the e-mail is sent. Otherwise it's not.

This forces my custom form to be used to be able to send the e-mails. And it's not possible to simply keep refreshing the submit page to keep sending the message.

And the challenge is in the form of old riddles and a couple new ones like "what's your favorite color?"

Things a bot would never get but that anyone who knows how to use Google can. Someone would have to program a custom bot with the answers in order to even attempt to spam. And even then since everything goes through my mail server nobody is going to sneak garbage past me for long and I know who your ISP is.

I also include a disclaimer with every e-mail. It'd be quite silly for me not to.

Ben

Re:Good old fashioned riddles

[LostCluster]

A further piece of planning would be to have a "three strikes you're out" rule so that any IP address (or maybe even a range) who is proving to be really stupid at answering the questions is told they can't bother trying for a while, and any further attempts to load the page before their time is up will just result in an extention to their penalty.

Re:Good old fashioned riddles

[weeboo0104]

And the challenge is in the form of old riddles and a couple new ones like "what's your favorite color?"

Yellow....NO, BLUE! AAAIIIIIIIIEEEEEE!
(thrown into /dev/null)

Re:Good old fashioned riddles

[dave420]

And that works when the user is behind a proxy/webcache how? Hundreds of thousands of subscribers to freeserve or AOL or whatever appearing from the same IP address, most definitely at the same time.

Matching people to IP addresses when visiting web servers is really tempting, but inherently prohibitively flawed for any global purpose...

He's right, we're doomed

[Zork the Almighty]

I think the author of the article is correct. Having a system whereby anybody can communicate at virtually zero cost without unsolicited commercial messages are mutually exclusive goals. I think that for most people, a simple whitelist is good enough, along with the understanding that there is a small chance that email between new contacts will be blown away.

Re:He's right, we're doomed

[0x0d0a]

Having a system whereby anybody can communicate at virtually zero cost without unsolicited commercial messages are mutually exclusive goals.

It's not quite simple. If the communications system accepts email to anyone from anyone with out restriction, then you might be right.

Antispam systems are designed to poke at exactly that point.

This will never end

[superpulpsicle]

SPAM is like popups. The one day you find a solution to stop it, the next day they find a new solution to send it. It's a never ending cycle get used to it.

Re:This will never end

[straponego]

Hmm, I haven't seen a popup on my machines in months. Not that there aren't techniques which would fool Mozilla (and Konqueror, et al), but they don't seem to be widely deployed. Now if only the Flash blockers worked on both OBJECT and EMBED tags...

Re:This will never end

[The Cookie Monster]

No it's not.
No other medium has this problem (not in my country anyway)

• The telephone does not have a spam problem.
  
• My instant messanger does not have a spam problem (it used to but they fixed it).
  
• SMS does not have a spam problem.
  
• My postal mailbox does not have a spam problem - "No circulars".
  
• The fax does not have a spam problem.
  

email is the only communications medium that has a spam problem, you are suggesting there is something magical about email that makes email and spam a law of nature.

The only thing special about email is it uses a protocol that was designed with different goals to what is needed now (ie security) and switching is hard, so hard that instead we cop out and just bolt more shit onto SMTP.

A secure protocol with existing anti-spam technology in combination with legislation (which mostly exists already) is all that's required.

Hopefully Microsoft (Hotmail+Outlook+OE) will one day join Yahoo and a few others and together they'll have enough momentum to make the jump to a protocol designed for todays environment. Then SMTP email will go the way of usenet - ie you can still use it if you like, but most people won't have a clue what it is.

If the jump isn't made then email will become less and less useful until it is entirely replaced in our lives by a better (and spam free) communications medium. I'm guessing this will be instant messaging (we already use it more than email), and if I had to put money on the future I'd say the gradual death of email and its replacement by another medium is more likely than actually seeing people stop kicking a dead SMTP uphill and adopting a secure protocol.

Re:This will never end

[Anonymous Coward]

Ok, now Cookie Monster just hit the nail right
on the head. Look at his examples.

>No it's not.
>No other medium has this problem (not in my >country anyway)
>
> * The telephone does not have a spam problem.
> * My instant messanger does not have a spam >problem (it used to but they fixed it).
> * SMS does not have a spam problem.
> * My postal mailbox does not have a spam >problem - "No circulars".
> * The fax does not have a spam problem.
>
>email is the only communications medium that has >a spam problem, you are suggesting there is >something magical about email that makes email >and spam a law of nature.

All of those examples have absolutely no security
built in (except possibly IM), but the one thing they all have that email doesn't is laws against spamming! That's it! Eureka! Why don't we just make it illegal to spam like we make it illegal to telemarket and fax-spam? Boom, problem solved with passage of a law. No I don't mean the "You CAN SPAM" law, but a real law that has prison and fines attached to it. Not a law that specifically allows spam, but a law that specifically makes spamming illegal in any form, no matter what. That is the only solution that will ever work.

>The only thing special about email is it uses a >protocol that was designed with different goals >to what is needed now (ie security) and switching >is hard, so hard that instead we cop out and just >bolt more shit onto SMTP.
>
>A secure protocol with existing anti-spam >technology in combination with legislation (which >mostly exists already) is all that's required.

I disagree with the security part. My telephone has no security whatsoever, and my number is even published and distributed for free to millions. And yet I receive zero telemarketing calls since the do not call list was put into effect. I can actually use my telephone now and answer it and it's always somebody I want to hear from. This was a direct result of passing a law that made it illegal to telespam me. Before, my home phone was pretty much useless but now it has its original purpose restored. Laws work. Why not do the same for email spam?

Re:This will never end

[The Cookie Monster]

The security in the telephone is that the phone company can look up who called you, this is missing from SMTP and is needed before spam legislation can be effective.

Re:This will never end

[Desert Raven]

What country do *you* live in????

Until the do not call list got enacted, I'd have gladly eaten twice as much spam just to get control of my phone back. I still get the occasional spam call.

I don't use IM much, so can't comment there.

SMS spam, not very much, but I've gotten text message spams before. It's a legal grey area, and a few folks are taking advantage of it. Getting them at 3:00am is wonderful for raising my blood pressure 50 points or so.

Postal spam? Holy spamola Batman! Over 4/5ths of my daily mail goes straight into the shredder. not only that, some of that stuff is downright dangerous from a privacy standpoint. Credit card offers, "checks" from my credit card company, etc...

Fax? Cripes, I don't even *have* a fax machine, yet I still have problems with fax spam. Nothing like a 2:00am phone call that answers with beeps because some jackass is using a sequential autodialer. Even more annoying, I have two phone lines with numbers just a few hundred apart. If I get one, I know another one is coming in 15-20 minutes.

Truth is, on an individual message basis, email spam is the *least* objectionable when compared to the others. The problem is that email spam is a veritable firehose at full blast. If I got telemarketer calls at the same rate as I get spam, my phone would be unusable 24/7. As it is, with using four blocklists on my server, and using spamassassin on what gets past that, with very high scores (15+) sent automagically to /dev/null, I'm *still* dealing with 100+ spam messages a day.

The point is, other mediums have this problem. The difference is that with other mediums, there's a much greater "bandwidth" limitation, and the cost to the sender is much higher. Thus, they tend to be more self-limiting.

Re:This will never end

[0x0d0a]

No other medium has this problem (not in my country anyway)

* The telephone does not have a spam problem.

I live in the US, and we *do*. Do you never get telemarketers?

My instant messanger does not have a spam problem (it used to but they fixed it).

IM systems do. The only reason that problems aren't worse than one might expect is that it's easier to pick up peple blasting out masses of messages because everything in centralized. Centralized systems have their own associated problems (easy monitoring of everything you say, easy abuse by monopolies, single point of failure).

# SMS does not have a spam problem.

I don't carry a cell, but I've certainly heard about people getting SMS spam.

# My postal mailbox does not have a spam problem - "No circulars".

*I* get junk mail in my postal mailbox. Admittedly, a manageable amount, but the majority of the mail I get is junk.

* The fax does not have a spam problem.

True. Up until not all *that* long ago, it *did*, though, at least in the US.

Re:This will never end

[The Cookie Monster]

I live in the US, and we *do*. Do you never get telemarketers?

I was under the impression that problem had just been solved in the US by leglislation. I live in New Zealand, I've probably had a 3 telemarketing calls in the last few years, and twice that number again in phone surveys. I've nevered asked why it's so low, I've always been listed too.

IM systems do. The only reason that problems aren't worse than one might expect is that it's easier to pick up peple blasting out masses of messages because everything in centralized

No, it's easyier to fix things when once entity gets to determine what protocol everyone is using, with ICQ messages only pass through a central server if the person is offline, otherwise it is peer to peer - and spam has still stopped (if you use ICQ and spam hasn't stopped, upgrading might be the answer). Also, Jabber is not centralised but has been designed not to fall into the same trap SMTP did. Being centralized makes things easier, sure, but it's not required.

I've certainly heard about people getting SMS spam

I imagine it's because the cell provider has a large $$$ incentive in making sure people don't turn off SMS, and people kick up a big stink when they do get SMS spam. I've had a couple of promotional SMSs from my cellphone provider, but that's it.

*I* get junk mail in my postal mailbox

Do you have a "no circulars" sign, and does your country have laws to enforce that? Our laws allow party political spam to delivered to "no circulars" addresses but that's it.
Someone else in this thread mentioned postal spam that is addressed specifically to you (as opposed to the stuff that goes into every letterbox), I might just be lucky in this respect as I've moved every 3 or so years.

** The fax does not have a spam problem.

True. Up until not all *that* long ago, it *did*, though, at least in the US.

Well that is even more significant, it means the spam problem was solved - lucky for us nobody was listening to the guy that said "fax spam will never end, learn to live with it and get on with your life"

Re:This will never end

[shrubya]

* The telephone does not have a spam problem.
I live in the US, and we *do*. Do you never get telemarketers?

Nope. Not any more. Do-Not-Call worked wonders for my phone.

heard about people getting SMS spam

It's rare in most countries, because they use a "sender pays" system. USA, get with the program!

junk mail in my postal mailbox. Admittedly, a manageable amount

Bingo. It's manageable, thanks to the magic of sender pays.

However, email postage is a monstrously bad idea. Free (both Gratis and Libre) mailing lists would die instantly.

Fix SMTP!

[schnarff]

Well, at the risk of sounding like a broken record, SMTP itself is the problem -- it's badly broken, security-wise, and needs to be fixed. It's going to be painful to move to a new mail standard, or to change SMTP so that it's not broken, but that's what needs to happen to stop spam. Thankfully, our friends the Russian Mafia and the ever-growing number of Windows zombie machines are making spam levels so great that, sometime soon, spam will represent such a large percentage of e-mail traffic that fixing SMTP will be necessary, not just something mail admins like myself wish for.

BTW, does anybody have a good figure on what percentage of all e-mail spam represents these days? I'm talking about *all* traffic, too, not just what ends up in peoples' Inboxes after all the filtering going on out there has done its job.

Re:Fix SMTP!

[Anonymous Coward]

With relatively conservative settings (spamassassin/amavisd-new, tag as spam at score 6.0), I'm killing about 40% of the SMTP transactions to a mail server that I run. It's about 30,000 messages per day being blocked. (Small academic user community, who don't like the idea of anything being blocked, so it's only tagging as spam for most users, who then use filters in their MUA or procmail.)

Of course, at home I'm discarding about 83% of the smtp transactions (at score 6.0) and tagging about another 14% as probably-spam and filtering with procmail, using a similar setup.

Re:Fix SMTP!

[mabu]

What "fix" is there to SMTP?

There is no "fix".... there is a "hack" and what does it ultimately involve?

Client authentication.

Ok, kids, what is client authentication? What's another name for it?

Whitelisting.

Now ask yourself. Are there more SMTP servers out there spamming than acting honorably?

No. The spammers are fewer in number than the legit SMTP servers.

So do we need a new SMTP protocol?

No. We need to blacklist rogue ISPs and SMTP relays that don't act responsibly.

You can list 100k rogue mail server IPs, or you can manage billions of legitimate SMTP IPs. Which makes more sense?

The RBL community has already taken this upon themselves to address this problem. If you don't understand the real dynamics at play here, at least just go along with those that do.

Re:Fix SMTP!

[Sjobeck]

Not broken but ancient. Whomever is repsonsible for updating SMTP ought to be flogged.

Re:Fix SMTP!

[TwoBit]

Since when is SMTP the problem. With all the owned machines out there, nothing you could do to change the protocol would fix the problem by itself.

Re:Fix SMTP!

[martin-boundary]

One more time.

What makes you think that a system that's different from SMTP can succeed in stopping spam? All you're saying is that the current system (SMTP) is broken, and a new system (WHAT?) will solve the problems (like WHICH problems, and HOW?).

That's a lot of faith based reasoning, but not a whole lot of proof, or even evidence based on experiments. Let's suppose you have a solution in mind. Have you tried it? Where's the case study, where are the test runs, where are the happy customers? Or are you just taling out of your ass when you say that SMTP must be replaced?

Spam is a huge problem, but saying that the answer is to replace SMTP is like saying that the answer to corporate corruption is to do away with capitalism.

Re:Fix SMTP!

[FLEB]

Why not just dump SMTP, and phase it out slowly via the carrot-on-a-stick of "New, spam-free(er) email!". (Hey, it worked for IPv6... well... is going to work... right?)

Take time to design and work out the replacement, and make it a protocol that's designed for accountability, security, adaptability, and a number of other things ending in "bility" that would be a great thing to have in email. SMTP withers, until it's only a dim acronym that accounts for the blind rambling of old zombies talking to one another. Stop adding layers and patches to an essentially flawed system.

Re:Fix SMTP!

[Anonymous Coward]

I wrote the script for my company that records and graphs spam deleted versus email that passes. We use brightmail and also access list. Brightmail gets about 60% and the access blocking of domains and address blocks gets the rest. Altogether we are now up to about 80% deleted and 20% passes. Of the 20% we estimate that 10 - 15 percent of that is still spam. So the overall percentage of inbound email attempted to our company is about 82 percent spam and 18 percent legitimate email. It was only about 50/50 two years ago, but the spam percentage shows a steady linear percentage increase over time. Looking at the monthly graphs its a negligable increase of legitimate email and a very large increase of spam so the legit line is horizontal and the spam line is about 30 degree incline over time.

Re:Fix SMTP!

[Anonymous Coward]

Well, at the risk of sounding like a broken record, SMTP itself is the problem -- it's badly broken, security-wise, and needs to be fixed.

No, it's not broken. It's a feature that anybody can send email to anybody else. You can't have that feature without the possibility for spam.

If you are talking about the large numbers of servers that are misconfigured as relays, then sure, that's a problem, but it's not a problem with SMTP.

Re:Fix SMTP!

[Anonymous Coward]

Yeah, the IPv6 "rollout" is a paragon of success, to be emulated. Lets try to get everyone to stop using mp3 and start using FLAC cause it is new, loss-free encoding!.

For crying (game) out loud people can't even agree on a little nub on top of SMTP like SPF (because it breaks stuff :0 and a great howling arose... - "it doesn't break anything if you follow these simple 27 steps, grrrr....") how the hell are we going to agree on a whole new protocol?

Oh, I forgot the same way we are getting things like UDDI - thank god people only had 32K of RAM to fuck around with when they built the internet or else it would have turned out like the useless bloatfests that are the current crop of standards.

Wait until they start to talk seriously about implementing the per packet tracking and billing standard that I read about but can't find right now...

More details in Part 1

[fembots]

The linked article is part 2, Part 1 is here.

Re:More details in Part 1

[LostCluster]

Are you telling them to read the whole article? I'd settle for people just skimming it or reading part...

Having experience, I can answer 1.2.1

[snakecoder]

I am not recommending mailblocks, I belive there is a sourceforge project called TMDA which does the same thing. Having said that, my experience comes from using mailblocks:

-cr deadlock: This does not exist because when you e-mail someone in a challenge and response system, it automatically assumes they are friendly. So if they have a challenge and response system, it will make it into your inbox, because you e-mailed them first

-automated systems He is correct here. Personally I hate when friends submit my e-mail to third parties without my consent so I do not mind missing these e-mails. I have caught a few while searching my pending folder, and inform my friends I rather have them e-mail me directly.

-interpretation challenge I believe he is wrong here because of a fundamental issue. When dealing with spam filters, the onus of working out refinements is left to the spamee, to make sure they filter out all spam. If a spammer adds a new technique, they get around the filter. With challenge systems, you have a few methods waiting as backup. When a spammer finally figures out how to read your words through AI, you simply change the challenge system and they are back to square 1 in trying to figure out how to defeat. As long as you have a few methods waiting in the wings, the spammers can easily be defeated, and have huge amounts of work to do.
if you doubt this, write an AI system to defeat hotmails gifs. Now what if the next day instead of showing a word, they show you a picture of 3 fire trucks and 2 police cars and ask you how many police cars are in the picture, etc ...

Re:Having experience, I can answer 1.2.1

[vanyel]

He's also wrong about using certificates:

1. certs don't require a connection to the cert authority. You get their CA cert ahead of time and then trust certs signed by it.

2. Responsible CA's won't grant certs to spammers because people will stop trusting their certs

3. If spam does come in signed, then they are trackable and the backlash will quickly shut them down.

Re:Having experience, I can answer 1.2.1

[dozer]

So if they have a challenge and response system, it will make it into your inbox, because you e-mailed them first

That can't be right... Am I understanding correctly?

• Bob sends you an email.
• Your system sends a challenge message back.
• Bob's system sends you a challenge of his own
• According to you, your system automatically assumes that Bob is friendly and therefore allows his challenge through.

Now, what if Bob is actually a spammer? All he has to do is send you two spams and one will get through. Doesn't sound very effective to me...

Re:Having experience, I can answer 1.2.1

[Sawbones]

Not quite. It would be

1. Bob sends you an email.
2. Your system sends a challenge message back.
3. Bob's system - knowing he's sent you an email - trusts email coming from you automatically so does not send a challenge back.
4. Bob sees your challenge, responds and everything is kosher.

So if bob's a spammer he'd have to reply to your challenge - something he's unlikely to do millions of times over.

Re:Having experience, I can answer 1.2.1

[tupps]

The only problem with the interpretation challenge is they would get around it the same way that porn/spammers get around the authentication checks at some site (eg where they present a skewed picture that is hunman readable but mucked up enough so a machine couldn't read it) is the porn companies set up pages that before you can see the porn you must do a few of these puzzles for the spammer.

This was on /. a couple of weeks ago.

Re:Having experience, I can answer 1.2.1

[mabu]

Certs are useless. Anyone can get a cert. Requiring certs for mailing is the same thing as creating a "license to e-mail" which would be better served as calling it what it is, and that basically becomes a whitelist.

The cert companies will ultimately give certs to anyone who will pay them money. All cert companies have been compromised by the almighty buck already. I got a cert from Verisign without having to VERI'anything.

Re:Having experience, I can answer 1.2.1

[janolder]

You missed the point...

• Bob sends you an email and his MTA remembers that it sent an email to you
• Your system sends a challenge message back.
• Bob's system forwards the challenge to Bob since Bob initiated the contact.
• Bob completes the challenge
• You receive Bob's email

If Bob is a spammer it would not be economical for him to reply to your challenge.

The tricky part is when a spammer forges Bob's email address and sends you spam. That would cause a cr loop unless the MTA's were to remember who it has previously challenged.

Re:Having experience, I can answer 1.2.1

[vanyel]

Those certs will simply not be trusted for purposes of accepting email. Thawte has a very thorough process for getting a cert with your name in it. Even their "Freemail" certs require some level of data input, but it's not verified. It takes enough time to keep it from being a viable option for spammers though.

Requiring certs would spell the end of anonymous mail, but spam has already done that, and the Beagle virus has shown another reason why everyone (ISPs in particular in this case) should digitally sign their email.

Re:Having experience, I can answer 1.2.1

[Nic-o-demus]

I agree. in re: interpretation challenge, I think the biggest leap I see here is that the spammers must become reactive instead of the anti-spammers. Filters for example, by their very nature, are reactive. Filters, no matter how brilliant, and with the exception of white-lists (if you categorize them as filters like the author), are adapting ex post facto to the spam. The spammers by the very nature of the thing are always a step ahead and the best a filter can hope for is to be so super-reactive that it seems almost proactive.

In this case, though, like you pointed out, the spammer must be constantly reactive to the turing challenge. And I must say, the tables will have turned significantly, because the producers of challenge response systems can change them very easily, whereas most spammers rely on black-market software that takes a few months to make the rounds and update

At this year's MIT spamconference there was a presentation I really enjoyed (Spamconference - 1st morning session) that talked about the timeline of feature adoption in spam (for example, when most spam started spelling viagra funny, etc.). You can see in the graph that there are new sets of spam features that occur every few months or so, all of a sudden, all at once. The presenter theorized that the evolutionary jumps in features that are a consequence of a new version of spamming software being released. The point being, spammers proper aren't the smartest guys around and won't be able (IMO) to be very reactive. The only reason they've been able to stay ahead is our favorite solution to date has been reactive. (For the record, though, I think filters are very fun to program and play with).

Re:Having experience, I can answer 1.2.1

[martin-boundary]

Filters don't all work reactively. There are two ways to throw an email out, one is to discover that it contains something bad and throw it out (that's reactive becasue, what's bad is decided by the spammers), and the other way is to discover that it doesn't contain something good and throw it out. The latter is proactive, because the spammers have to figure out what's good and put it in. The advanced bayesian filters straddle both methods, they learn what's bad and also what's good, then use whatever's most applicable for each email.

I found the first conference session to be a waste of time. It's a nice idea in principle, but their email samples (from memory, 7000 ?) are way too small to have any predictive power. And unfortunately, their mathematical methods are too computation intensive for reasonably sized email collections (ie I'm talking 50, 100 million emails is reasonable for large scale trends - remember that AOL and MSN transport several *billion* emails a week).

Re:Having experience, I can answer 1.2.1

[FLEB]

Put enough lag into the process, though, and it becomes easy to poison. This works on site signups, since the spammer gets a quick confirm/deny message, and can deny the precious pornography until the viewer stops clowning and gives the right response.

If you introduce even a five or ten minute lag into the "Your Message Has Been Delivered" on the C/R system, the spammers would have to keep the porn viewers waiting or else accept any old trash they put in as legit. I suppose you could run a "login today, porn tomorrow" service, but it would still be a pain to the spammers.

For extra points, make the "delivery confirmation" in graphic text on a noisy background.

Re:Having experience, I can answer 1.2.1

[jasgo]

So if they have a challenge and response system, it will make it into your inbox, because you e-mailed them first

How does it know if an email it happens to get back from me is a challenge/response email?

I used to have a home-grown procmail script to do this (I removed it because of some of the reasons above, but mainly because half the people who emailed me either "couldn't be bothered" hitting reply and send to the challenge email, or "didn't understand" the instructions in it and just deleted it). But how does your system recognise a challenge email when it sees one? A database or something of common challenges? What happens when it's not a common program? You could end up (like I did a few times) with 2 challenge/response systems each sending challenges back at each other because they don't recognise that the email it's challenging is itself a challenge (I'm confusing myself here).

Apart from that it's great technically but I've actually found it doesn't work because people are stupid and/or lazy (see above)

Re:Having experience, I can answer 1.2.1

[0x0d0a]

This does not exist because when you e-mail someone in a challenge and response system, it automatically assumes they are friendly.

Or you can use syncookies-style crypographic tokens or something.

The problem is that interpretation systems have severe usability issues and screw mailing lists.

Re:Having experience, I can answer 1.2.1

[0x0d0a]

Requiring certs would spell the end of anonymous mail, but spam has already done that, and the Beagle virus has shown another reason why everyone (ISPs in particular in this case) should digitally sign their email.

No, it wouldn't.

Zero-Knowledge sold "nyms" for a while (short for pseudonyms", which basically came down to cryptographic identities. They might in theory be able to map your real identity to your nym and visa versa, but other people couldn't, and I believe they didn't maintain logs. It might be more realistic for someone like Sealand to provide such services.

Aside from the pain it takes to deploy, I really feel that cryptographic solutions are the answer -- at *some* point they have to be deployed. They have the strongest theoretical protections, the lest theoreitical disadvantages to the user, and provide a host of benefits (avoiding users spoofing emails, ensuring that people don't eavesdrop on your email).

Re:Having experience, I can answer 1.2.1

[vanyel]

By "requiring certs" I mean "requiring a cert signed by a trusted entity". Though I suppose they could sell or issue anonymous certs that are certified to not be spammers... then people might trust them...

Re:Having experience, I can answer 1.2.1

[0x0d0a]

That's one solution, though not the only one.

Here's another example. I use a middleman a la ZK to get anonymized email and anonymized e-banking. I then (under my nym) provide a significant deposit to someone to give me a cert. Sure, I can spam, but then I lose my deposit (making spammers not go for this).

Or I can set up a nym, built up trust as that nym (without producing a mapping between the nym and me), and just acquire endorsements from trusted entities the same way I would in real life.

Re:Having experience, I can answer 1.2.1

[Illserve]

Interpretations challenges have limitations. You cannot throw up pictures of fire trucks because then it's suddenly language and culture dependent. Same with facial recognition.

In fact there are damned few interpretation challenges that cross language and cultural barriers. The problem is not so easy as you might think.

Re:Having experience, I can answer 1.2.1

[Anonymous Coward]

Not to mention the accessibility problem. Visual challenge/response problems discriminate against the blind/partially sighted/colour blind etc.

Re:Having experience, I can answer 1.2.1

[snakecoder]

>How does it know if an email it happens to get back from me is a challenge/response email?

It doesn't have to. The idea is, if I e-mail you, you are automatically in my system's whitelist. Anything coming from you is now accepted because I e-mailed you.

Now if you have an authentication system, your automatic authentication letter will make it into my in box as long as it comes from you.

As far as people being lazy, I guess that is a function of supply and demand. If you are in demand, people will do what they need to,to get in touch with you. If you are looking for leads though and these leads are fickle, this type of system is not for you. You'll probably have to just deal with the spam

I managed to appall a colleague today...

[Ungrounded Lightning]

Was out to lunch with three colleagues today and the subject of anti-spam measures came up.

I managed to appall the one from Berkeley by suggesting that the most practical solution was probably a moderate-size bomb.

B-)

But seriously:

In an arms race, weapons eventually defeat armor. Spam will continue until two real-world things are BOTH brought to bear on spammers:

- Economics
- Muscle

If a governmental solution applying both is not forthcoming soon, I predict that there WILL be vigilantism.

In fact we're already seeing it.

For instance: Subscribing the Detroit area spammer and his lawyer to enough real-world junkmail lists to bury his bills and other US Main correspondence in several daily truckloads of catalogues and other solicitations.

Soon to come: Retaliatory information-war software directed at DDoSer / spammer zombi-net machines. (As discussed in a recent Slashdot article.)

Re:I managed to appall a colleague today...

[mabu]

For instance: Subscribing the Detroit area spammer and his lawyer to enough real-world junkmail lists to bury his bills and other US Main correspondence in several daily truckloads of catalogues and other solicitations

With all due respect, get a clue.

You don't fight a noisy neighbor by cranking up your stereo.

Re:I managed to appall a colleague today...

[Anonymous Coward]

Actually, I have done this and it worked on the second try.

It was a matter of borrowing about 800 watts worth of public address systems, pointing them out my windows at his house, and cranking it up whenever I could hear his stereo. He couldn't hear his stereo over the noise, and quickly came to realize that when he turned it up above 7, he no longer got to hear it.

Re:I managed to appall a colleague today...

[mabu]

Good for you. I feel sorry for all your other neighbors who suffered because of your little "arms race."

I'd give even odds that if you try the "get back and them with the same strategy" you can just as easily end up on the receiving end of punishment by the authorities as them, probably sooner.

Re:I managed to appall a colleague today...

[dave420]

Muscle never solved any argument - it just stopped one side from arguing. The only way to win an argument is to win the other person to your side.

Basically, to get the spammer to stop spamming, stop people buying their product. It's legal, ethical and will stop spam in seconds. Instigate laws that outlaws spam as a method of selling products. Any company found trading via spam can be brought before a court. The beauty with that system is the company has to be reachable via the email somehow (otherwise they wouldn't sell anything, so the spammer wouldn't spam for them), whereas the spammer remains hidden. That lack of anonymity the company posesses means you can find the perpetrator, and press charges. Most likely, the company will release the information about the spammer (including financial information, which can be used to persue the actual spammer).

To reach the spammer you have to go through the only route possible - the vendor.

Newest anti spam technology

[cluge]

**note location of tongue**
Of all the odd places to find anti-spam technology, was this killer solution in WalMart. Yep, it turns out they have a remarkable tool that convinces spammers to stop spamming! I was AMAZED. This tool usually only has to be applied once, and the affect lasts for years. It doesn't require updating or re-installation. I was also suprised to find these very same tools in other places, like sears, and even in a "sneaker" store. What is this tool you ask? An aluminem baseball bat. It seems sadly though that there is a law protecting spammers. I believe useing this AWESOME anti spam technology falls under something called assault There is hope that exceptions for spammers could be provided for in a constitutional amendment!
**note location of cheek**

AngryPeopleRule

dont forget ...

[segment]

I don't bother getting too deep into downloading too many 'new improved!...' filters. I block entire damn countries/netblocks. Besides I don't know anyone in korea, brazil, china, nor any other one of the massive spamming countries. I configure postfix to filter out a lot and the minute I receive one spammed message, I always whois -h whois.apnic/arin/ripe/lacnic offender and block their entire range. I also have spam assassin running and I have to admit I get about maybe... maybe... 4 spams a week not kidding. Again though this is my personal machine.

block return-icmp (8) in proto tcp from 24.76.0.0/14 to any port = 25
block return-icmp (3) in proto tcp from 81.208.64.0/18 to any port = 25
block return-icmp (4) in proto tcp from 163.121.163.0/22 to any port = 25
block return-icmp (4) in proto tcp from 82.77.83.0/24 to any port = 25
block return-icmp (4) in proto tcp from 61.247.224.0/19 to any port = 25
block return-icmp (4) in proto tcp from 217.132.0.0/17 to any port = 25
block return-icmp (4) in proto tcp from 62.103.204.32/27 to any port = 25
block return-icmp (4) in proto tcp from 210.111.224.0/17 to any port = 25
block return-icmp (4) in proto tcp from 144.135.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 195.166.224.0/18 to any port = 25
block return-icmp (4) in proto tcp from 61.228.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 207.144.229.0/24 to any port = 25
block return-icmp (4) in proto tcp from 193.252.22.160/28 to any port = 25
block return-icmp (4) in proto tcp from 200.0.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 209.202.192.0/18 to any port = 25
block return-icmp (4) in proto tcp from 83.32.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 68.38.64.0/8 to any port = 25
block return-icmp (4) in proto tcp from 219.240.0.0/10 to any port = 25
block return-icmp (4) in proto tcp from 195.57.218.0/25 to any port = 25
block return-icmp (4) in proto tcp from 129.79.245.98 to any port = 25
block return-icmp (4) in proto tcp from 24.150.0.0/19 to any port = 25
block return-icmp (4) in proto tcp from 24.205.28.0/21 to any port = 25
block return-icmp (4) in proto tcp from 220.116.0.0/8 to any port = 25
block return-icmp (4) in proto tcp from 200.128.0.0/9 to any port = 25
block return-icmp (4) in proto tcp from 212.81.64.0/17 to any port = 25
block return-icmp (4) in proto tcp from 32.10.58.0/19 to any port = 25
block return-icmp (4) in proto tcp from 210.183.110.0/20 to any port = 25
block return-icmp (4) in proto tcp from 134.196.0.0/16 to any port = 25
block return-icmp (4) in proto tcp from 24.60.88.0/23 to any port = 25
block return-icmp (3) in proto tcp from 24.190.8.0/24 to any port = 25
block return-icmp (2) in proto tcp from 24.98.77.0/23 to any port = 25
block return-icmp (2) in proto tcp from 24.173.29.0/23 to any port = 25
block return-icmp (2) in proto tcp from 205.206.176.0/23 to any port = 25
block return-icmp (2) in proto tcp from 172.128.0.0/10 to any port = 25
block return-icmp (2) in proto tcp from 200.171.99.0/24 to any port = 25
block return-icmp (2) in proto tcp from 200.171.97.0/22 to any port = 25
block return-icmp (2) in proto udp from 200.171.97.0/22 to any port = 25
block return-icmp (2) in proto tcp from 68.62.80.128/25 to any port = 25
block return-icmp (2) in proto udp from 68.62.80.128/25 to any port = 25
block return-icmp (2) in proto tcp from 218.76.0.0/17 to any port = 25
block return-icmp (2) in proto udp from 218.76.0.0/17 to any port = 25

Re:dont forget ...

[Anonymous Coward]

Dude, segment, help me. I've been stuck in Korea for the past 3 months, and I've been trying to reach you.

Please give my wife my love, and nothing more.

Re:dont forget ...

[Anonymous Coward]

Don't worry. You dont need to be talking to racists like segment anyway. There are a few of us who are still not as close-minded in this world.

Re:dont forget ...

[NuShrike]

Why not just blacklist all mail servers except the major ones you expect mail from anyways?

Also check logs for attempted connects and add the ones you care about?

That's how my spam filter works, and I get 100% blockage with no overhead, unlike the other spam email filters out there.

Re:dont forget ...

[Anonymous Coward]

Excellent idea! Given the fact that 56% of all spam originates in the US, I already know which country to block off first.

Re:dont forget ...

[Catbert66]

You can make it even simpler. Don't accept mail from likely abuse sources, from dynamic IP addresses, or from known abusers. Those three blocklists get rid of an enormous amount of my spam.

Taken along with a few select country blocklists (I use China, Taiwan, Hong Kong, Korea, Brazil, and Argentina), you can go from a flood to a trickle in no time. China is a Very Special Case -- they're completely filtered at the borders now. If they ever clean up their act, they may get to pass packets again, but I'm not holding my breath. In the meantime, they can enjoy their shrinking view of the Internet.

insightful... insight to violence

[Roger Keith Barrett]

This should be modded flamebait. Talking about violence as a solution to spam is frankly just total bullshit.

If you get that angry because of hitting delete (even if is an excessive number of times) you have an anger management problem.

Re:insightful... insight to violence

[Anonymous Coward]

I wish i could click delete on your post.

Re:insightful... insight to violence

[Anonymous Coward]

At this point in the game, I am honestly surprised that we haven't heard of violence resulting from a bad slashdot post.

I don't know about anyone else, but I'm pretty sure I'm not alone in this. I have, at times, felt utterly enraged at all the goatse links flying about and further all of the innocent and naive people that are being abused by the GNAA.

I know if I feel violent internally, then surely there are those with less self-control out there who will eventually act on his or her rage... perhaps the parent of a child afflicted with a picture of tubgirl?

I think if two or three bad posters are attacked physically, it might give them pause. Frankly, I'm amazed it hasn't happened.

Re:insightful... insight to violence

[Anonymous Coward]

He's not angry about having to hit delete.

He's angry that his 7 year old daughter got a spam about things 7 year olds don't generally talk about; He's angry that his grandfather has been doing business with some guy in Nigeria.

Re:insightful... insight to violence

[Anonymous Coward]

A 7 year old girl with an e-mail account? A 7 year old allowed to access a computer without supervision? Besides the fact this 7 year old daughter probably weighs in around 200 lbs., I think you should take a parenting class or two. I don't even leave my dog unattended. What a jerkstore.

Re:insightful... insight to violence

[Roger Keith Barrett]

1) The seven year old probably didn't understand the message and won't for a couple of years.

2) The grandfather is STUPID. He's old enough to know better. If he is senile why is he on the internet in the first place?

Part 1 is damn close to "what about the childen" argument, which is abused so much as to be worthless. Sorry.... I don't buy either. Neither one is an excuse for violence, either.

Missing from the article?

[Anonymous Coward]

One "solution" which seems to be missing from this article is the "verify each stage" solution. You know, close down all open relays and implement a C-R system between the mail client and the server (password authenticaton to send?) and perhaps between servers too (a public-key challenge before transfers between servers, e-mail transferred in bulk after said challenge for speed reasons). The idea being not so much to make spam disappear, but to make all e-mail clearly and easily traceable so that no spammer would want to keep operating, and allowing any spammer who continues to operate to be tracked down.

Perhaps one of those SMTP fixes or SMTP alternatives mentioned at the end implements this idea? Anyone have more info?

Maintenace the problem

[powerpuffgirls]

As stated in the article's summary, the main problem with most spam-filter is the need for constant maintenance. We need a solution that requires ZERO maintenance by the joe-users, and yet cost-effective enough to implement.

My ISP seems to have a so-called "Watch Dog" spam filter, where they actually hire people to read spams and filter them manually, that's probably the most effective way to filter spam, but I wonder if it is cost-effective though.

Re:Maintenace the problem

[mabu]

You need to get your ISP to use Spamcop's RBL. This is exactly what they do. They employ a real-time spam-reporting system (with checks and balances) that identifies sources of spam in real time. I have no affilliation with the company but I cannot deny that their system blocks at least 16,000 spams a day to my server. It works.

Do not call ...

[Ephboy]

Prior to this October, telemarketing calls were a national scourge. Amazingly, since we signed up for the Do-Not-Call list, we've only received 2 illegal calls. I'm rather surprised, in fact, at the relatively uniform acquiescing to this law. While spam, coming from all corners of the earth and is more anonymous, will be harder to enforce, some law with real teeth may be a good start.

Re:Do not call ...

[btempleton]

A do not spam list would just be a big list for spammers to use as a source for who to mail. You can't hide the list with hashes.

Most spams are already illegal, con games, selling prescription drugs etc. They are not scared of a little do not spam list. The do-not-call list stopped businesses that wanted to stay legit.

Instead, sign up for the do not spam list (I have an infinite number of email addresses so it may take me some time to do so) and you will just get more spam, I am pretty sure.

Re:Do not call ...

[mabu]

There are a few problems with your comparison:

* It's a lot easier to jack into the Internet than it is to get a phone line

* It's more expensive to perform telemarketing than cybermarketing; you have to pay people and you're not nearly as anonymous - there are costs in launching telemarketing efforts, whereas with spamming, all you have to do now is jack into a network or open proxy and unload your spam.

A spam do-not-e-mail list won't work, because at the present time, the spammers can hide much more effectively on the Internet than they can using POTS.

Not to mention that you don't see telemarketers engaging in the fraudulent practices that spammers employ, so that should tell you something.

Re:Do not call ...

[Anonymous Coward]

I also have been pleasantly surprised at how effective the do-not-call list hast been. I went from receiving 10 calls a day to maybe 2 a month.

Re:Do not call ...

[dave420]

The problem is, it's expensive to call from outside the US, and easily traced. Those two problems alone means it's next-to-impossible for a company to make illegal telemarketing calls to the states. As soon as they did, the complaints reaching the telco would make them track down the telemarketers, and at least stop routing their calls. The cost of international calling also means the percentage of callers who purchase their products has to be highter, meaning slimmer profit margins. That must be a very risky game to play.

Unfortunately, with spam, sending a mail to anywhere in the world is free, and very easy to obscure the true origin. As no-one's paying per-email fees for passing the spam along, no-one's that interested if it's spam or not. There's certainly no vested financial interest in stopping it. Just ignoring it is cheaper than actively trying to cut it out.

The real problem with spam is the relative cheapness and anonymity behind it. The only things that stop people spamming via phone/fax/SMS/etc is the fact that the spammer is easily traced. As we all know, with email it's not that simple.

Stop the madness

[Anonymous Coward]

First, Microsoft should bite the bullet and by default not execute executable attachments in email. They should also not obfuscate certain file extensions such as .pif.

Second, companies selling products via spammers should be held equally guilty as the spammers themselves.

SpamBayes

[jonfromspace]

I've been using SpamBayes for about 4 months now, plugged into my outlook (Don't ask) with a very high degree of success... It seems to me that if email is to remain a free(as in beer) and open part of internet infrastructure, and in fact our lives, that in-box solutions are the only way to go.

The many problems with challenge solutions and the like meen any hope of seamless introduction and integration into existing business processes is not likely, and seem to keep pointing us back to the inbox for our filtering. Aside from "e-stamps" (yuck) I just don't see a reliable alternative anywhere in the near future.

Then again, I could be full of crap.

Reputations

[grotgrot]

The only thing that will work in the end is some sort of distributed reputation management system. To a certain extent that is what RBLs do, except they are on or off. SpamAssassin does offer shades of grey to the RBLs (differening weights to each one).

To a certain extent this is what we already do in real life. We 'judge a book by its cover' as a first pass (for example people will often walk past a beggar in the street completely ignoring them) and then include other factors. How polite they appear, where they are from, recommendations from friends etc

All other mechanisms suffer from a determined spammer being able to get around them as the article pointed out. Any mechanism that prevents some spammers makes things more lucrative for the rest.

Re:Reputations

[chriskenrick]

Have you checked out WPBL, as linked in my sig?

It basically attempts to classify IPs as primarily spam senders or not according to the ratio of spam/non spam they send.

The more signed up, the merrier, so feel free to check it out.

Re:Reputations

[.@.]

I agree.

So much so that I've been working on just such a system.

Re:Reputations

[grotgrot]

It isn't really addressing what I am talking about. For example, is Amazon a spammer? The answer to that depends on reputation, which can change person to person. (To me they are as I cancelled my account a few years ago due to the privacy shenanigans. To my housemate they are fine.)

At the moment spammers don't have to be particularly good. They don't really masquerade as anything but as spammers. But what would happen if an online store similar to Amazon had totally legit business, but also tried to sell you Viagra? These are the grey areas.

You can already see it in Google searches where you get those results that look like something useful but aren't. Then there are "news" web sites which are just trying to sell you something, but it isn't too overt.

For everyone legit, reputation matters. To a certain extent the process of branding is about building reputation. If you had reputation based feedback about mail and sites (reputatation of the people reporting repuation etc) you can make a better judgement. Sites that don't take much action against their users or others pretending to be them will lose reputation. Sites that have users who send contentless or annoying email will lose reputation.

And if you suddenly come up with a new domain name, you will find it just as hard as in the real world. Reputation is one of those things that has to be earned.

Re:Reputations

[grotgrot]

I keep forgetting to mention eBay. It can be seen as a testbed. The fundamental problem of spam is dealing with a communication between two random people. Is that email from someone/somewhere you have not dealt with before spam or not?

eBay has to solve exactly the same problem but instead of communication it is enabling a transaction. They delete accounts based on abuse (just like RBLs, spamcop etc) but at the end of the day, just as in real life, the thing that works is reputation.

Re:Reputations

[leviramsey]

I just devised a setup that might be interesting:

• Users (sysadmins) of the blacklist submit two lists of IPs, good (non-spammers) or bad (spammers).
• When a server receives a mail, it checks with the list to see on which lists the IP appears as good and on which it appears as bad.
• The user marks the mail as ham or as spam. A Bayesian algorithm then determines which lists are trustworthy for marking spam hosts.
• Filters could then /dev/null mail based on this bayesian score

The idea is essentially to allow a collaboratively developed decentralized blacklist and whitelist to develop. Spammers will either submit the IPs they use to this list or not submit them; if they do submit them, then a "good" report from them will eventually be taken as a strong sign of spamminess. If they don't, then nothing happens, but presumably "trustworthy" blacklists would list them.

Thus, a user in Brazil, where they would be receiving lots of legit mail from Brazilian IPs would not find a blacklist that listed all of LACNIC to be a strong indicator of spamminess. The effects of blacklisters who maliciously put enemies into their blacklist would also be reduced, if not eliminated.

A suggested implementation detail on the blocking would be to make it random; that is to say that 100% of the mail with a 100% probability of being spam gets dropped, 99% of mail with a 99% probability gets dropped, 97% of mail with a 98% probability gets dropped, 94% of mail with a 97% probability gets dropped, 90% of mail with a 96% probability gets dropped, etc. according to this function:

d(p) = d(p+1)*p/100, where d(100) = 100, and 73<=p<=100

This would allow for a degree of "retraining" in the event of false positives (since a /dev/null'd mail cannot be retrained from!).

Re:Reputations

[grotgrot]

While reputations of an IP address are a good starting point, systems will ultimately need to take into account way more than that, including the email address From and To. For example my address from my ISP is pretty reliable, but one that has never sent a message before isn't. My ISP address but from a webmail provider in Kenya is not reliable (as would a Kenyan from my ISP).

My emails to the mailing lists of open source projects I founded have high reputation. My ones on the art of Persian rug making in the 2nd century have low reputation. Reputation in one area affects another.

Re:Reputations

[Anonymous Coward]

eBay has to solve exactly the same problem

If you have been ripped off on ebay, as many have, I don't think you'd consider it "solved" very effectively - in that way, ebay mirrors real life, people who have put time/effort/$$$ into trying to build up alot of reputation can still turn out to be crooks (Enron, Martha, MS) or hypocrites or liars (OJ, SCO, Rush Limpbalm, steamroom sharing preachers, WMD finger pointers, etc).

We could solve this problem without filters..

[Pidder]

Spammers will not go away until they stop making money. If we could make people stop clicking links in the spam, or buying their products, spam would go away tomorrow without the need for any kind of filter. I'm clueless how that's supposed to happen though. It's seems that your average computer user will click everything that comes into their mailbox.

Re:We could solve this problem without filters..

[Anonymous Coward]

SPAMMERS should learn and send mails in intelligent way so that they target less people and better sucess rate, instead of making world gainst them, they themselves should take steps to reduce spam ..

Easy...

[TekGoNos]

Your example is EXTREMLY easy to break.

You need the same font that is used on their page.
Then you loop through every possible starting position and every possible letter. When the pixels occupied from the letter have all the same color and the pixels surrounding them havent, you have a match.
Afterwards, you just need to order the letters according to the starting pixel.

Heck, if you put some money behind your challenge, I might write one, that breaks your example, over the weekend. If you send me the code you use to generate the image, I'll break it overnight.

For a real challenge, use at least something as the yahoo challenge from the article and add some random noise. This will make it harder, but I bet it would still be possible with a good (and perhaps adapted CR).

Remember, if you can reverse engineer their algo that produces the challenge images (and that is not difficult as you dont have to get exactly the results as long as they are similar), you can generate an infinity of training instances for your CR that can run automaticly.
And a CR trained with billions of instances will become VERY good (possibly even better than humans).

Another partial solution

[PapayaSF]

1) Tap the Slashdot and creative communities to produce a series of anti-spam TV/radio/print ads on the theme of "Spammers are Scammers." Smear all spammers as scam artists who sell fake merchandise and steal credit cards, and their customers as stupid losers.
2) Get media outlets to run them for free as public service ads.

Yes, I know this isn't a 100% solution. However, it is relatively low cost, and requires no new laws, software upgrades, or Internet standards.

Re:Another partial solution

[Skim123]

Spammers are Scammers

Am I the only 20-something who immediately thought, "Spammers are Scammers, and Scammers are Spammers, So Don't Do Spam, Dooooon't do Spam!"

Ok, I feel better now.

Re:Another partial solution

[Red Alastor]

According to a Paul Graham paper, 15 people out of a million answer to spam. Will we reach these 15 morons by publicity ? Will they understand the message... Well... Maybe, we should not forget that they are stupid and will listen to anything publicity tells them to do even if it is in a spam. It would probably be more effictive than can-spam.

What works for me

[gregwbrooks]

I know not everyone can run their own mail server*, but here's what has reduced my inbox spam to about 1 miss out of every 400-500 messages:

• I run SpamAssassin and ClamAV on my server and check all inbound mail against a series of RBL lists; and
• All mail POP'd into my Outlook (yeah, I gotta use it - no flames!) gets checked using the free-and-excellent SpamBayes.

Works in the bakcground with damn-near zero false positives, and doesn't require Microsoft-pushed e-mail postage, changes in the e-mail RFCs or anything else.

The tools are out there. If you use them, spam isn't nearly as much of an issue as the press makes it out to be.

*Well not everyone in the Real World anyway -- here on /. we all run our own boxes, right?

Re:What works for me

[Sjobeck]

Kudos to SpamAssassin & ClamAV, both awesome projects from our friends at SF.net. Wow. We love 'em both here. Couldnt recommend any higher.

Dueling Challenges

[The Monster]

I just copied that challenge into IrfanView and had it reduce the number of colors to 2. It came out quite readable, which suggests that OCR would be able to take it from there nicely. I bet someone could throw together some Script Fu for the GIMP to convert those pictures to text with a reasonable accuracy rate. Bear in mind that the technique doesn't have to be anywhere near 100% accurate to be worth the effort for the spammer, who already has a business model based on a fraction of a percent of his emails actually generating a response.

What I take issue with is this paragraph from the article:

CR deadlock. Alice tells Bill to email her friend Charlie. Bill sends an email to Charlie. Charlie's CR system intercepts the email and sends a challenge to Bill. Unfortunately, Bill's CR system intercepts Charlie's challenge and issues its own challenge. Since neither user actually receives the challenge, neither user will receive the email. And since the emails are unsolicited and unexpected, neither user knows to look for the pending challenge. In essence, if two people both use CR systems, then they will not be able to communicate with each other.

This is leaving out a key feature of any decent challenge system... When Bill tries to send an email to Charlie in the first place, Charlie's email address is automatically added to Bill's whitelist. So Charlie's challenge, showing his address as its source, flies straight to Bill's Inbox without a hitch. If Bill were so arrogant as to think he could send email to someone not on his whitelist, then he deserves not to have his email go through.

Re:Dueling Challenges

[TekGoNos]

If Bill were so arrogant as to think he could send email to someone not on his whitelist, then he deserves not to have his email go through.

While I agree with this statement, somebody will introduce the following bug into his system : the C/R message will not come FROM: Charlie, but FROM: the C/R system.
Of course it's a bug, but it will not manifest itself loudly, so I predict it wont be fixed for a rather long time, with a lot of messages being dropped.

But I agree that this argument is rather weak.

The REAL problem is that spammers will start collection email-pairs instead of simple emails and simply circumvent any whitelist. They already do collect addresses out of chain-emails and mailing lists, they just need to collect the FROM address as a second field to the TO address and go on happily.
The clever ones might already do this and wait to use it till whitelists become widespread enough.

Re:Dueling Challenges

[RollingThunder]

Not so much that it would come from Charlie, but that the C/R would have an In-Reply-To that referenced the unique Message-ID of Bill's mail.

When the mail goes out, Bill's system would record the Message-ID (and probably the recipient, but that could screw up on forwarders if you try for a hard match on the two) and then allow Charlie's C/R because it matches the whitelist.

Re:Dueling Challenges

[Tony-A]

Charlie's email address is automatically added to Bill's whitelist. So Charlie's challenge, showing his address as its source, flies straight to Bill's Inbox without a hitch.

Now all I need to do is know or guess anything on your whitelist (or have some means to automatically add something to your whitelist;).

Methinks all a CR system would do is add hassle to legitimate traffic and give the spammers an even easier time of it.

Re:Dueling Challenges

[CanadianCrackPot]

I could easily see this being a failure through which a spammer could still send spam, although much smaller. Suppose that a spammer just sends out e-mails as normal, challenges would be issued to both the spammer and the reciever. Should the challenge contain some heading information (namely the Subject field, which is logical you'd want to know why someone is e-mailing) then a spammer can send out very very small ads. The easy soulution for this is to not issue challenges to incomming challenges.

Re:Dueling Challenges

[The Monster]

The easy soulution for this is to not issue challenges to incomming challenges.

Sorry, but that would just open up the chance to spam someone by sending a 'challenge', with a graphic that's a clickthrough ad.

Re:Dueling Challenges

[The Monster]

The REAL problem is that spammers will start collection email-pairs instead of simple emails and simply circumvent any whitelist. They already do collect addresses out of chain-emails and mailing lists, they just need to collect the FROM address as a second field to the TO address and go on happily

...which is only a problem if a spammer can forge a FROM address in the first place. SPF closes that loophole, allowing white/blacklists to work again. If there's one thing in the article I agree with completely, it's that there is no Magic Bullet -- we need

• SPF to deal with forged headers
• White- & Blacklists for people we already know.
• Challenge/Response for people we don't know yet.
• Bayesian filters
• Special tokens for web sites that let you send a news item to a friend's email by attaching a brief signed personal message (that includes the date and title of the news article to prevent replay attacks) that grants a one-time pass through the filters and C/R.

These tools can be used in various combinations:

During the 'transitional phase' of SPF, source addresses that lack SPF records in DNS would go through challenge/response as an alternative. The challenge email could even include URLs with FAQs about how to implement SPF, handy for forwarding to your mail administrator.

Those tokens might be treated by the Bayesian filter as just one more hint as to whether something is spam. The preprocessor might replace a validated signature with:

<spampass from="dumbfriend@aol.com" date="Fri Mar 12 07:27:46 2004" title="Spam White Paper" />

which might not boost the rating of the email at all, if prior spampasses from this same friend have generally ended up manually marked 'spam' by the recipient.

Re:Dueling Challenges

[The Monster]

Now all I need to do is know or guess anything on your whitelist

...and be able to impersonate one of those 'things' (email source addresses). see my response to someone else for details on how to deal with such shenanigans.

Re:Dueling Challenges

[Tony-A]

Non-forgeable From-addresses would be nice, but the most critical emails that I send or receive are when email is broken and/or one of us is not in a position to be able to use the normal channels.

It's like phoning the phone company to report that your phone is out of order.
It's like a backup system that works perfectly as long as you don't need it.

The from-address is where the email claims to be from. It should be easily forgeable. If I am using someone else's computer to send a quick note, I should be able to send it, from me, without messing up the computer's settings.

The headers also include where the email came from, at least the last leg of the trip. The headers should be blatantly obvious when mail is delivered. Otherwise it's like the postman delivering the letter inside and keeping the envelope.

The problem with spam is not that it is unsolicited, nor that it is commercial. The problem is that there is far too much of it, and it is being sneaky about delivering it. Spam is socially unaceptable and the solution will be social not technical. For the technical side, the email client needs to distinguish between what it knows and what the email purports to be. For HTML emails, it would help to see which domains are referenced by the email. The difference between the malware running loose now and the Unix Honor Virus is that with the latter you can see what is going on. Anything that pretends to be other than what it is is up to no good. Anything that encourages this pretense (hint, hint Microsoft) is encouraging the malware. Anything that calls something secure when it has only secured part of it is encouraging the malware. A tar-paper shack with a steel security door is not secure.

Re:Dueling Challenges

[RantRant]

Mail is sent in the clear, right?

Spammers can watch for Charlie's C/R, copy the Message-ID info and SPAM Bill.

Re:Dueling Challenges

[CanadianCrackPot]

Right guess there is no spam-proof system.

Re:Dueling Challenges

[Anonymous Coward]

Challenge Email from the recipient could also be used to confirm an active email address (Mail harvesting attack)

Re:Dueling Challenges

[etLux]

Wouldn't it be a whole lot easier for Bill to just call Charlie on the phone?

Re:Dueling Challenges

[instarx]

Not every solution to a technological problem has to be technological as well. Recent /. post told of looters in Chernobyl stealing contaminated TV's and reselling them to unsuspecting customers in flea-markets. The Russian government caught four doing this and shot them. Looting problem dropped to near zero afterwards.

Serious jail time or financial punishment for illegal spamming would be a real deterent to spam. Currently there is almost no punishment for any spammer even if he or she is caught.

Hey, no tips!?

[Anonymous Coward]

And here I thought the article was going to give help on how to increase my profits in sending viagra ads and other spam.

Of course there is

[Sycraft-fu]

There are plenty of tasks that you can do that computers find nearly impossible. Facial recognition is a good one. Humans do it easily all the time. Computers are trying, but still screw it up badly. Musical recognition is another one. A human can easily pick out individual instruments in a peice, and can tell that the song is the same even if it is a complete different orchestration and mix (like a remix for example). Computers are confounded by this, even when they break something into component sine waves. Pragmatic language interpreatation is my favourite. Even when people speak non literally and indirectly, you still have no trouble with their meaning. You can also tell which level of meaning they want, and successfully decode the other levels if asked. Computers are lucky if they can get the literal direct meaning out of a sentence, never mind anything else.

So, just because a human can do it, doesn't mean a computer can. I don't know about any of these image schemes, I've never played with it. However if you make it sufficiently hard for it to recognise characters form background, and one character form another, it's screwed. Computers have trouble with fuzzy and incomplete information that humans are so good with.

Also remember it needs to be feasable to do in a reasonable time. Maybe you develop some whiz-bang image recog program that can take amazingly distorted text and figure it out. If it takes 5 minutes to process a box, it does you no good anyways, too much time to be worth it for this use.

Re:Of course there is

[whereiswaldo]

Maybe you develop some whiz-bang image recog program that can take amazingly distorted text and figure it out. If it takes 5 minutes to process a box, it does you no good anyways, too much time to be worth it for this use.

Not really. Since spammers are now into the illegal business of commandeering people's computers using viruses and trojans, it would be an easy step to have them process distorted images and feed the results back to some web site.

It wouldn't even take that many computers to send a lot of spam out even at 5 minutes per. Say you want to send 1 million emails. 1,000,000 / 5 minutes = 138 days. If you have 138 computers, you can send out 1 million spams per day.

Re:Of course there is

[michaeltoe]

A computer program is only as good as the way it's been written.

I am not trying to argue that it's a trivial problem that some fresh-from-the-womb teenage programmer with VB could solve. Then again, you seem to believe that the interpretation of geometric symbols is something achieved by only those imbued with the power of God.

Don't get me wrong, but if your brain isn't executing some form of mathematical logic to solve these problems... then what is it doing? Magic? Is that how you're going to support your argument?

Re:Of course there is

[Red Alastor]

It's no more magic than for humans understanding irony. And I would be surprised that computer are likely to learn it in a near future.

Re:Of course there is

[stw0ng]

I don't mean to be inflammatory at all, but a few counterpoints:

About the language interpretation bit:
Isn't that mainly a problem with audio recognition? If you type in the sentence, it's impossible for the computer to hear your tone of voice, pitch, and the other nuances, isn't it? And if you speak it into a microphone, problems would be found with the fact that the mike input is not necessarily at the level of human's ears, and that it would first need to be able to break the sentence apart into individual words and have the various properties of each one recorded there.

I doubt that *understanding* the sentence would be a problem, with a program that is good enough. The input would be the problem. A deaf person cannot understand a sentence, right?

For audio recognition, we'd first need microphones as good as human ears. Once we can get rid of the various static and stuff, I'm pretty sure that it would be relatively easy to program an interpreter for the noise. Speaking of which, has anyone done extensive work on audio recognition? An idea would be to use a tracker program to create music that isn't flawed by static, compared to most microphones' input, and then work off of that... Purely speculation.

"If it takes 5 minutes to process a box, it does you no good anyways, too much time to be worth it for this use."
CPUs keep getting faster, right? In the DOS days, pre-emptive development might have been started on GUIs, even though it might take up to a minute to refresh the screen. It just means the project will be that much farther along when it's actually feasible.

"Computers have trouble with fuzzy and incomplete information that humans are so good with."
And at this one, I just ask: Is it the computer that has trouble, or the programmer? Being one myself, I honestly think it's the programmer's problem that he can't help the computer use fuzzy / incomplete information. I haven't taken enough biology to know really how brain cells work, and I doubt that anyone knows for sure, but I believe that the human brain might work much like a computer does--just much, much faster, with lower voltage, and the other goodies natural selections have given us.

The hardware will advance, much like life did, and eventually they may match or even surpass us.

Re:Of course there is

[Wm_K]

how about:

1000000 * 5 / (24 * 60) ~= 3472 Days or 3472 computer to send 1 millions emails. Doable, but much more difficult than the 138 you state

Re:Of course there is

[Anonymous Coward]

I would be surprised that computer are likely to learn it in a near future

I'd be surprised too, since the average American hasn't figured it out yet.

Joke! It's a joke! Jesus wept...

Re:Of course there is

[whereiswaldo]

1000000 * 5 / (24 * 60) ~= 3472 Days or 3472 computer to send 1 millions emails. Doable, but much more difficult than the 138 you state

You're absolutely right - good of you to double check.
But like you say, it is definitely still doable:

Federal officials reportedly have closed in on an 18-year-old man believed to be the author of a variant of the Blaster worm, which affected nearly half a million computers earlier this month.

Re:Of course there is

[dave420]

Computers can't do facial recognition?? They've got CCTV systems that can recognise 30 people a second, from a centralised database. If that's not good facial recognition, I'd like to know what is ;)

Re:Of course there is

[Sycraft-fu]

As for language recognition, the problem is with text, never mind even trying to translate speech. Computers have all hell even parsing through written language and getting the literal, direct meaning. Well the problem is that in our communication we are almost always using some indirection or impliciture. There are few sentences that stand on their own and need no more information than they contain themselves. They rely on general background assumptions, and so on.

It gets worse when people are DELIBERATLY being non-literal or indirect (or both) which they often do. Suppose I want to date a girl named Jane, and wonder if she's available. So I ask you, you respond "Jane has two kids." That, of course, answers my question. But, really it doesn't. Your literal direct meaning has nothing to do with my question. You are answering me indirectly. I, of course, will understand perfectly that Jane having two kids means she had a partner at some time and since you said it in response to my question you mean she has a partner CURRENTLY.

Or how about metaphors and allusions and the like? Suppose you are negoating with someone. During a break I ask you how it's going. You say "I'm talking to a bunch of rocks here." You are speaking non-literally here. You aren't actually talking to rocks, you are talking to people that refuse to budge or listen to what you are saying.

In either case a computer is totally screwed in finding the meaning. This is compounded by the fact that we don't have a perfect, all encompassing theory of Pragmatics to being with. Humans can understand and use all the different levels of meaning, but we can't say HOW, at least not completely.

As for audio recognition, it's EASY to get a mic that exceeds human ears. You can do it for a couple hundred dollars. However you don't need to. Humans don't recognise speech based on our full range of perception. Think: you can easily do it on a telephone, which has only 4khz of bandwidth (your ears have 20khz ideally). You do it by cueing in on things like the first two formant frequencies, and the ratios between them. You also interpolate incomplete data.

The human brain does NOT work like a computer does, and that is quite clear. A computer is a digital, imperitive device. All it's states are discreet and finite. It processes instructions in sequence, and the result of a given instruction is gaurenteed based on given input. This is why computers are so good at math, and numerical analysis. They do good where things are definite.

Humans are the opposite. Our brain is analogue, non-discreet, and based on tons of interconnected neurons. Based off of tests of cognition, you can only perform about 100 steps in the time it takes you to deal with problems like language recognition. So clearly, lots of work is done in parallel. Also, connectionist networks like the brain can't explain why they do something. A computer program is perfectly explicable, each instruction leading to the next. A connectionist network isn't, it just learns a way to get the right result, but the individual parts don't make sense.

Not that's not to say we can't someday design a computer that works like the brain, a connectionist network rather than an imperitive device. Also we currently can make computers simulate connectionist networks, very popular in AI research. However, it's all primitive at this point. It comes nowhere near what the brain is capable of. Since we are talking about what spammers could do NOW to foil those checks, not what AI research may be able to do 30 years in the future, it's all academic.

Oh, and if we had AIs, SPAM wouldn't be a problem. You just have an AI filter your mail for you.

Re:Of course there is

[instarx]

Restricting the challenges letters and numerals gives the spammers a big advantage if they do try to decode the challenges with software since there are only limited numbrs of ASCII characters to pick from. Most people seems to be thinking the CR has to be letters or numbers, but it doesn't. Your example of faces is a perfect example. Facial expressions are universal and it would be very easy to have the responder pick the happy face or the angry face from a selection of other faces. This is trivial for humans and nearly impossible for computers, plus the available faces number in the billions, rather than just 255 (?) or so ASCII characters available.

I suspect the almost universal use of letters and numerals in challenges today resulted simply from 1) the original programmers were just thinking that way, and 2) nothing more complex or creative has really been necessary yet.

If advances in character recognition begin to figure out the current crop of CR it should be trivial to develop methods that will be much more difficult to circumvent. Spammers should realize that spending a lot of effort to overcome the letter/numeral system would be a waste of their time and money because it would be trivial to make the challenges orders of magnitude more difficult for computers. Ironically, that may be a reason why the letters and numerals may be all we ever need.

Public key cryptography.

[Gadzinka]

When using certificates, such as X.509 or TLS, some type of certificate authority must be available. Unfortunately, if the certificates are stored in DNS then the private keys must be available for validation. (And if a spammer has access to the private keys, then they can generate valid public keys.)

Someone, either me or the author of the article is on crack. I was under the impression that one does not have to have private key in order to validate the signature.

Lets assume that there are CRT records that store SSL certificate for clients allowed to send mail on the behalf of the domain.

example.com. IN CRT "Certificate goes here"

1. Client connects via SMTP-TLS session signed with Client Certificate.
   
   
2. Client sends SMTP command:

   MAIL From: <example@example.com>

3. Server checks CRT record for sender domain and looks if Client Certificate that signed the session is signed with this domain's certificate.
   
   
4. If not, than reject the offer with:

   550 You don't have valid CERT for sending as @example.com

   end everybody's happy.
   

Now somebody tell me, in which step one needs private key to verify certs?

Robert

Re:Public key cryptography.

[AnotherFreakboy]

You don't need a private key to verify certs.

Just to simplify it for those not up on thier cryptography:

Encrypt with the public key = can only be unencrypted with the private key, ie only the intended recipient can view the original message.

Encrypt with the private key = can be unencrypted with the public key, ie anyone can read it, but it is garuanteed that the person who claims to have sent it actually is the person who sent it, because they have the private key (authentication)

Re:Public key cryptography.

[Sjobeck]

This is an excellent plan & I do not know why we're not trying it right now. It could have the same sort of soft phase in that SPF is trying to use for their phase in (ie: three stages, as I recall). It would be simple, mostly free, and very effective. Why oh why oh why?

Re:Public key cryptography.

[Anonymous Coward]

You're right. The "expert" is on crack.

(1) The public-key can be stored in DNS.
(2) The private-key is kept on the mail servers at example.com.
(3) This works.

Re:Public key cryptography.

[Gadzinka]

Well, the better yet news it that you could use it as replacement for both SPF (i.e. sending directly to recipient) and SMTP-AUTH (i.e. sending via smarthost).

In case when mail server finds out that the session is signed with cert ``blessed'' by its own IN CRT, it could allow the messages send in this session to be relayed anywhere sender wants. It would have to have matching domain still, because server wouldn't have the means to deliver it otherwise.

I was trying to post something about this method to Ask Slashdot about a month ago, but editors keep it ``pending'' indefinitelly... The question was ``what am I missing?'' since this method seems so obvious, elegant and simple, that I am surprised that no one came out with this before.

Robert

Re:Public key cryptography.

[Gadzinka]

(2) The private-key is kept on the mail servers at example.com

No, and that's the beauty of it.

Domain's private key doesn't have to be stored anywhere on the net. On mailserver of this domain is another cert (private+public) signed with IN CRT for example.com. But the real private key signing all those certs is only on the terminal disconnected from the net entirelly, used for batch-signing of client certificates.

This way you cannot crack into the computer to steal private key because it isn't anywhere on the net.

Robert

Trying to fix the problem from the wrong side!

[Flashbck]

I keep hearing about all these different methods of using authentication to curb spam. However, the problem is a little bigger than the minor inconvience that you recieve from having to delete 10, 100, 1000 e-mail's from your box a day. The real problem is the disgusting amount of traffic dedicated to sending out spam. While it would be great to block spam when it hit's my SMTP server, the traffic is still there and possibly doubled if my server sends a reply. We need to find a way to address this problem from the source before it bogs down pipelines.
Ok so I guess I'm all talk because I can't think of a single possible solution, but I do feel that this is the approach we need to take.

Re:Trying to fix the problem from the wrong side!

[tehdaemon]

If most people start using filters or other spam-blockers, the idea is that it will no longer be effective to send spam, 'cause almost none of it gets to a person. It would be like everyone having caller-id connected phones that only rang or even went to voice-mail if there was valid ID info. Telemarketers would quit calling or use corect ID info. Once this happens spammers will move on.

That is the theory anyhow.

Re:Trying to fix the problem from the wrong side!

[Anonymous Coward]

Hey, you not insensitive clod! That is the "solution" I am working on...

most effective

[mabu]

Make no mistake...

The most effective spam solution at this time is RBL blacklisting. Bottom line.

When you take into account that the biggest problem of spamming is bandwidth consumption and network resources, there is NO better way than blacklisting spam sources and refusing to communicate with them.

Services like Spamcop's RBL really piss off the spammers. All client-side filtering is counterproductive and ultimately useless as you constantly have to update the systems to catch new efforts on the part of spammers to thwart the filters. At least with RBLs, the spammers' connections are immediately refused as soon as they're ID'd.

If you want to identify what is the most effective solutions, it's simple. Look at what pisses off the sleazebag spam community the most. That's relay blacklisting. They don't DDOS the moronic client-side filtering companies because the spammers know they're useless, and even if they're not, the spammers can't tell. What hurts them are when systems say, 'screw you spammer, (click)' and that's done via relay blacklisting.

Why are spammers increasingly changing mail relays and pursuing open proxies? Because of RBLs. Even AOL uses RBLs (including Spamcop). All the major ISPs look at the RBLs because they are THE most effective way of stopping spam. And they're the only way to actually shut down the spammers.

Forget client or server-side content-based filtering. They will NEVER work. RBLs are responsible for forcing spammers into corners of IP space, forcing them to deploy worms and viruses to infiltrate new IP space (which exposes them to more prosecution). RBLs ** WORK ** !

Re:most effective

[Tripster]

You are so right, I use a few on all my servers and they work, cbl.abuseat.org works wonders at cutting down on the trojan spam.

I've also setup my own private RBL, any spam that makes it thru the public ones has the IP it originated from added with no hope of ever getting off it either since there is no contact info sent so spammers have no clue where the RBL is housed.

Just this morning I was forwarded the dynamic ranged from Shaw Cable here in Canada, we were getting hammered by the infected fools there and I complained to them to at least close port 25, instead they sent me the ranges I can safely block, sweet, now to work on Telus.

Re:most effective

[mabu]

Amen.

Shaw is a spam haven.

Comcast is a spam haven.

Virtually all IP space in Korea.

When you start doing IPLOOKUPs of the spammers you begin to see a pattern of which ISPs don't have their shit together.

Why did Comcast start cracking down on spammers? It was probably because admins like us stopped accepting mail from their business customers because they were embedded in the DSL IP space that spammers have compromised. Do you think Comcast gives a damn about spamming? No. But if you start making their IP space unuseable by legit companies, then their buttom line is hit.

Blacklisting WORKS. Unless you run your own mail server, your opinion doesn't matter. Run your own server, deal with these sleazebags every single day, bombarding your systems with their crap, then talk to me about BS client-side filtering.

Re:most effective

[mabu]

Oh let's not forget WANADOO - wanadoo.fr is one of the largest spamming pools of IP space on the planet now. This has been happening for more than a year.

We need all mail admins to BLACKLIST WANADOO. They've had a goddam year to stop their spamming customers. The only way to stop them is to send them a message that we WILL NOT ACCEPT MAIL FROM WANADOO. Then and only then will they get their act together. This is the way it's done.

WANADOO YOU'RE NEXT! Someone post an IP list of all WANADOO IP space so we can shut this lousy ISP off the net until they get responsible!

Re:most effective

[Tripster]

Here ya go, this will help you keep out Shaw's residential customers ...

24.64.0.0/13
24.76.0.0/14
24.80.0.0/13
24.108 .0.0/16
24.109.0.0/18
24.109.64.0/19
68.144.0.0 /13

Those ranges are safe to block, they have other ranges for the static business clients.

Of course another simply step the ISP can take is to block outgoing SMTP entirely for those ranges except to their own mail servers.

Re:most effective

[permanentE]

Forget client or server-side content-based filtering

SpamBayes works REALLY well for me.

Re:most effective

[Ragica]

Some would say RBLs work "too well". They have a fairly consistant history of accidentally abusing innocent parties. Is it the price to be paid for the overall protection? Depends on your point of view.

We don't have that many clients using our mail server, but one noticed one day that mail to him to friends was bouncing. He reported this and we discovered that we were on SpamCop's RBL list.

I did a quick audit of the mail server, fearing we'd been highjacked, but found no evidence anywhere of spam going out.

Being generally sympathetic to RBLs I was eagre to get to the bottom of this, and cooporate with whatever needed to be done to prove our innocence.

But i found the SpamCop web site to be extremely frustrating to find any information. I found some references stating that to refute being listed you must reply to the email that SpamCop sent you: I searched and searched but we recieved no mail from spamcop.

As I spent a precious day trying to figure out what to do, as mysteriously as we'd been listed, our IP disappeared from spamcop's list.

To this day I don't know what happened; but have a somewhat more bitter taste in my mouth regarding the arbitrary power of RBLs.

(Though I still tend to more blame the system which blindly obeys a single RBL: I think SpamAssassin is more democratic in that it only assigns a probability, and an IP has to be on multiple block lists before it goes over a threshold. This gives spammers more lead time before they are blocked, but also prevents any single RBL from weilding absolute power... a sort of check-and-balance.)

Not that easy

[Sycraft-fu]

The problem comes from the fact that any reasonable fix I can think of necessitates breaking compatibility with the current version. Well e-mail is, by it's nature, a tool for collabration. It is useful precisely because I can send it to anyone with an e-mail address. So I'm not going to upgrade my server if that upgrade means I can't communicate with anyone anymore.

No, the REAL solution is prosecution. Not just for the spamming, but for hacking. We know spammers pay to get systems hacked and viruses written. The government needs to track this down, find out who paid for it (not going to be hard, your average virus writer is not going to want to go to prison) and then lawyer the fuck out of them.

It won't stop it, but it will reduce it to managable levels, and I can't think of a workable technical solution that will.

expert my ass

[xenocyst]

while his credentials certainly would put him in a far better position to know these things than i am... i find his death and doom attitude annoying... he doesn't really address the parts of anti-spam that do work.. he glosses over them, and then hypes the parts that are broken.. without any sort of proof if i were to mod the article it would probably get something like +2 informative -2 Overrated -1 Flamebait and -1 Troll

Not bloody likely

[phr1]

Drug dealers and mobsters have been killing each other in bloody shootouts (or swordfights or whatever) for centuries and it hasn't slowed them down one bit. I don't see it being different for spammers.

C-W Problem

[xSquaredAdmin]

Consider that both the sender and the recipient have a C-R filter. How will either one get the challenge? Wouldn't it just end up in an infinite loop of challenge e-mails? Or is there something I'm missing?

Re:C-W Problem

[tehdaemon]

Yup, others have pointed this out.

Simply have the email client keep track of who you send email to, and never issue a challenge to someone you already sent something to.

Example: Bob sends email to Alice, Alice's client sends Bob a challenge. But Bob's client knows he sent an email to Alice, and so that challenge gets to Bob. Bob gives the responce, and Alice gets Bob's email.

Re:C-W Problem

[Silburn_Luke]

Or get a bit fancier (which I suspect you'll have to) and list any email addys you challenge on a provisional whitelist and only move them over to the real whitelist once they've correctly responded.

This allows you to get funky with the provisional list - add time expiry rules, move addys to a blacklist if they screw up the challenge more than a few times, scan the provisional list for 'spammy' IP ranges etc etc

As with any arms race the ideal is to use layered defences and a mix of techniques, so whitelists have a part to play, but so do blacklists, reputational systems, cryptographic signatures and so on.

Regards
Luke

SPF Anyone?

[ignoramus]

One proposed solution I would love to see getting more attention is SPF ("Sender Policy Framework"), which allows each domain admin to specify their email sending policy using existing infrastructure.

See the SPF site or read this month's Linux Journal to find out more.

Executive summary of SPF: Just use DNS to specify where mail from your domain may originate from. If everyone used this, we could have domain blacklists that actually work.

Do an "nslookup -type=txt psychogenic.com" to see an example entry. And if you manage any domains, please consider doing the same.

challenge-response handling being outsourced

[SuperBanana]

I did hear of a theory where C/R was being cracked by taking the C/R image, posting to a porn session, and letting a seeing person do the work.

I had a chat with a Veep that was hired on to a company I used to work at. Very down to earth guy, very friendly. We got to talking about spams and semi-legitimate emailings to customers, etc.

He had one very interesting tidbit; stick with me for a sec here. Most companies outsource their semi-legit stuff because they get reported as spammers and whatnot, or it bogs down their email server/network, etc. No surprise there- however, the interesting tidbit is that the outsourcing companies turn around and outsource to Indian firms for handling the bounces. There's literally a room full of people in India, sitting there answering those challenge/responses and updating the client's customer email list(unlike spammers, it really is in their best interests to minimize failed deliveries). It sounds "expensive", but it's not, considering how few people use challenge/response systems. Further- a reasonably smart human can get familiar with all the various systems quickly(an hour or two, I'd guess, tops) and probably process close to a message every few seconds with a client program set up to do that limited functionality smoothly. Best part- if your client does several mailings, unless the recipient goes in and removes you, you're clear for future emailings.

Re:challenge-response handling being outsourced

[Arslan ibn Da'ud]

> however, the interesting tidbit is that the outsourcing companies turn
> around and outsource to Indian firms for handling the bounces. There's
> literally a room full of people in India, sitting there answering
> those challenge/responses and updating the client's customer email
> list(unlike spammers, it really is in their best interests to minimize
> failed deliveries). It sounds "expensive", but it's not, considering
> how few people use challenge/response systems. Further- a reasonably
> smart human can get familiar with all the various systems quickly(an
> hour or two, I'd guess, tops) and probably process close to a message
> every few seconds with a client program set up to do that limited
> functionality smoothly. Best part- if your client does several
> mailings, unless the recipient goes in and removes you, you're clear
> for future emailings.

I'm not sure this is a problem. Remember spam costs the spammer
zero. But if the spammer has to hire a roomful of people (India or
anywhere) merely to answer C/R's, then that costs $.

So you've effectively turned your C/R system into a pay-for-mail
system, which many of the big boys are in favor of, anyway.

Two-pronged combination

[Lenbok]

What is wrong with using a combination of a hashcash type approach in conjunction with cryptographic signing to address the shortcomings of both.

Thus the following rules for the user:

If an incoming email is cryptographically signed by someone on your whitelist, accept it.

If an incoming email has made hashcash payment, accept it. The user then decides whether to accept future signed messages from the sender.

Other incoming mail is returned to sender instructing them to make hashcash payment.

Sign all outgoing messages, and also generate hashcash if you haven't previously sent to the user.

How this affects the downsides:

Mailing lists: Would generate hashcash payment for the subscription process, but regular mail messages are just cryptographically signed (i.e. independent of the number of subscribers).

Unequal taxation: May still be a concern if your machine isn't up to the task of signing the bulk of your outgoing messages.

Robot armies: Users (should) quickly notice if their machine is burning the CPU generating hashcash tokens and address the problem.

Legal robot armies: I don't see what the problem is here -- the sender is still having to pay to generate the tokens, so the economics of spam are changed.

Automated abuse: Hashcash payment is required for all initial messages, so generating countless certs doesn't help.

Usability: Crypto signing is done with self-signed certs (e.g.: PGP) so no central CA is needed.

The God solution

[Faeton]

"Kill 'em all and let God sort it out"

What we need is a sniper at a Spam convention to thin out the ranks a bit... There's probably hundreds of pyschos out there into all kinds of weird stuff. Why can't a few be interested in offing a few of these spamboys like me? Waitaminute....

Re:The God solution

[Sjobeck]

whooopsy there, problem with that theory is that there is no god. Now what?

Not for all, but a good start..

[SuperQ]

This isn't the solution to all spam, but it solves a lot of the big failures of the SMTP protocol. Sender Policy Framework.

http://www.ietf.org/internet-drafts/draft-mengwo ng -spf-00.txt

SPF removes the posibility of From: spoofing, by publishing a list of valid servers that are allowed to send mail for the From: domain.

try this command: 'host -t txt aol.com'

You will see a list of valid servers, and "?all" says, be warry of other servers. If it said "-all" it would be block alll servers not listed here.

The big huge major drawback to this system is email forwarding and aliases. The example I like to use is.. I subscribe to debian-devel@debian.org to an email account, and then that account forwards to my home server. The mail would bounce because the email came from a server not in the debian.org approved list.

The solution to the forwarding problem is header-rewriting. When a .forward or alias is processed, the From: is changed to the email address that recived the mail. This will require as much work as the work to eliminate open relays has been. and it may never be 100% until the standards are officialy changed, and vendors change the default settings.

Re:Not for all, but a good start..

[mabu]

From spoofing verification won't make a difference... it'll slow down mail services and won't make a dent in spam.

Spammers are now rotating IP space all over the place... they're also beginning to NOT forge header information, so what are you left with?

Recognizing rogue relays and blacklisting them, even if they have valid header information. Any improvement to SMTP protocol won't make a bit of difference.

Most mail servers and large ISPs are already employing additional methods of header-verification. It hasn't stopped spam.

RBLs ARE working. They're making spammers scramble for un-blacklisted IP space. That's why they're running overseas; that's why they're sending out worms and viruses. Lord help us if IPv6 gets introduced... we'll never be able to stop spam then.

Re:Not for all, but a good start..

[mdfst13]

Using both SPF and blacklisting is more effective than either alone. SPF requires the spammer to actually own the domain name in the From address. Also, it slows down spammers ability to make changes (they need to update the SPF records in DNS). Further, now you can blacklist *both* the domain (which costs the spammer money) and the IP (which also costs the spammer money).

Another advantage of SPF is that it prevents joe jobs and virus spams (unless they point their SPF DNS at the virus computer's IP, which would be fraudulent and illegal). RBLs success is pushing them into this. No reason to let them squeeze out.

No one solution will fix spam. It takes a variety of methods. Where one is weak, another takes up the slack.

Re:Not for all, but a good start..

[firewood]

RBLs ARE working. They're making spammers scramble for un-blacklisted IP space. That's why they're running overseas; that's why they're sending out worms and viruses. Lord help us if IPv6 gets introduced... we'll never be able to stop spam then.

IPv6 will just require a change from distributed blacklisting to distributed whitelisting. Make it cost money, or an verified ID check which takes time, to put an IP block on enough master whitelists and that will put additional pressure on spammers hunting for usable IP addresses.

Re:Not for all, but a good start..

[Anonymous Coward]

Sender Policy Framework.

WTF? I guess "Sender Permitted From" wasn't cool enough. Geez, get some press and they have to get a fancy-pants new name "Sender Policy Framework"
Come on!
And why the same 3-4 people keep chiming in (politely for now) about Sender Permitted From everytime there is a /. spam article, I see the next step to be the polite tone to turn into the smug, crusader you-are-a-criminal-if-you-question-the-system tone that the RBL (esp. SPEWS) types now have. As traditional BLs fall away due to dynamic spamming schemes, someone will need to take over...

Current boycott of Microsoft email caller ID!

[Rayban]

A related note- the current Microsoft anti-spam solution, Email Caller ID is currently being boycotted.

Re:Current boycott of Microsoft email caller ID!

[vegetablespork]

That's because (as mentioned in the linked article) it's an inferior ripoff of SPF.

Alternative Spam Solution: The Magic Word

[auburnate]

Has anyone tried the magic word on the spamming offenders?

.... "Please"

It doesnt change the fact...

[Anonymous Coward]

That you have a small dick.

Interview with a spammer

[dbIII]

The truth is 90% of spam comes from open relays, that is SMTP servers that can be tricked

A couple of years ago I didn't have a job - and a government sponsored job database came up with a listing for a job using computers in the "adult" industry. I went along to an the job interview, and found the job would have been setting up a dozen modems on a linux box and writing a program to scan for open relays (he'd apparently paid US$10,000 for a list). All income would have been undeclared, and some dodgy accounting involving payroll in the name of tourists would have gone on. Some background checks on the employer turned up a few interesting things as well as birthplace, education, home address etc. It looked like a may have had a choice between becoming a spammer and never getting paid for it, or losing my unemployment benefits (the consequence of turning down a job offer in my country). Another, actual legitimate job came up for a dying dot-com, so I never had to argue with beuracrats as to why I had turned down a job.

Oddly enough the spammers name was "Fagin", as in the Oliver Twist villain, and he was born with that name.

sophistry

[sweatyboatman]

so you suggest a "hack" to SMTP that wouldn't solve the problem. and then, because your hack is unfeasable you suggest that the problem cannot be solved.

Besides, blacklisting spammers is not a new idea and yet somehow SPAM continues to clog our inboxes. Perhaps it's not the silver bullet you're claiming.

why exactly would it be difficult to have a system that guarantees that the sender's address on the email is the actual sender? and that the date on the email is the date it was sent?

why can anyone with a computer and a connection to the internet send out millions of emails?

what exactly is the advantage provided by SMTP that there's so much resistance to changing it?

Re:sophistry

[mabu]

verifying the sender is the sender doesn't matter... that just means that the spammer has to have DNS properly configured. Big deal. That's an extra 10 seconds they have to spend before they spam.

And that reduces spam... how?

Re:sophistry

[Anonymous Coward]

And that reduces spam... how?

Because then the spam comes from spammersite.com instead of ibm.com, so now you can filter the spam with blacklists.

Re:sophistry

[martin-boundary]

Thank you Mr know-it-all. It's not like the spammers can hack machines in the ibm.com domain and take over users' desktop mail sending software so that the trusted users really *do* send the spam out. Oh wait, viruses already do that today.

Not a bad idea

[Anonymous Coward]

And allow disposable passwords to register for stuff and be able to block the spam afterwards.

You could even give different passwords to different people, so you'd know who's responsible for some spam you're getting because he typed your e-mail + password into a "send an e-mail card" website.

A rotating password on your website could even allow some fans to contact you, while making it impossible to harvest and ship on a CD to spammers (password expires after 12h).

I guess we'd need a new "Password: " header in mails, and some minor changes to e-mail clients to filter incoming mail based on passwords, and to store passwords in the addressbook for outgoing mail.

Way too difficult

[artemis67]

If it's something that you have to explain to your parents how it works, then it's just not a mainstream solution. Besides, you're still talking about text, and if the answer can be found on a Google search, then a script can figure it out, too.

I like the challenge-response system, but instead of text I think they should use pictures. "Type in the name of the object in the picture, the first letter has been provided for you." Provide the first letter because some people will see a tiger and think "Lion" and the first letter will get them back on track. Show all of the images as grayscale so that image scanning software can't tell the difference between the yellow of a sunflower and the gray of a building.

SPAMfighter works for me...

[Alex_Ionescu]

The big problem with mail filters, as the article mentions, is that they need to be updated when new spam technologies appear... and there's also a lot of false positives... I gave SPAMfighter a try (from www.spamfighter.com) and although it was a bit worse at finding spam (At first), I never got any false positives. The way it works is that the "filters" are actually some kind of hash that users submit whenever they block or unblock an email (it analyses the whole content I think, not just the text). So if a new type of spam technique appears, the users will just block it. And unlike many other client-side plugins, it actually works on Outlook Express.

Another one I recomment is Spambayes...but there's the problem with false positives. All the other ones I've tried are utter crap.

Best regards,
Alex Ionescu
Relsoft Technologies

strange timing

[Ozric]

Today, I just set my cron job that does salearn --spam on my missed spam folder from once a day to every 30 minutes.

My Spam Solution

[Starfury_2260]

The best spam filter I've found so far is Spambayes. It's also free. Does a better job than any of the shareware ones i've seen so far.

Speed limit for newbies...

[LostCluster]

For a new account that's never been seen before, or worse yet, a new domain that's never been seen before, the number of knock-knocks (of any kind) they should be allowed to make should be limited.

Afterall, if somebody new out of the blue wants to speak to a large users on your mail server, they deserve some checking into. They're at least a new mailing list who is growing at unheard of rates. Better to blacklist them until they've been proven worthy.

Cumulate stats for new accounts, new domains, and new originating SMTP servers both individually and by IP space. Any that pass their speed limit are shut off. And, if it just so happens that too many new accounts are coming from the range of 0.0.0.0 to 255.255.255.255, well, it's time to shut down for a bit...

An idea to make ISPs responsible

[mabu]

I have an interesting idea to force ISPs to crack down on spamming customers...

This basically works only if the spamming ISP is from your country. Which is why blacklisting of foreign IPs is still necessary.

But for domestic ISPs who don't reign in spamming, someone should post the 800 numbers of ISPs that don't crack down on spamming. Put up a web site listing the 800 numbers of the ISPs that are top-ranked in harboring spammers. Most of them have 800 numbers.. if everyone calls these ISPs and complains, or at least takes up air time, it costs them money, and money seems to be the only thing that motivates these companies.

Use Bayesian filtering

[Skim123]

Yes, spam changes over time, but Bayesian filters naturally adapt over time. Of course, the downside is that as spam mutates you will get it in your inbox and have to manually mark it as spam, but I've found that this is somewhat more of perverse pleasure than a chore. (For more on why filters aren't an ideal solution, check out this article...)

Comment removed

[account_deleted]

Comment removed based on user account deletion

CR deadlock

[Skim123]

Another deadlock case, which happened too many times in my experiences with C/R:

• Alice sends a message to Bob. Alice is not in Bob's whitelist, so Bob's C/R anti-spam system sends a challenge to Alice.
• Alice doesn't use C/R, but rather a filter. Her filter, unfortunately, marks Bob's challenge as spam. Since Alice is only a computer novice, she does not know how to check his Junk Email folder, and therefore never receives Bob's challenge, hence Bob never gets Alice's email. Alice, who is blissfully ignorant of the "behind-the-scenes" happenings, thinks Bob just is trying to ignore her. So she sends another email, which is, of course, not received by Bob. And she sends another. Still, no response from Bob. Alice takes it personally and decides if she does ever hear from Bob again she won't be going on a second date with him no matter what.

Re:CR deadlock

[LostCluster]

Such a deadlock is avoidable either Alice and Bob are both smart enough to see it coming and value the communication enough to head it off.

When Bob gives Alice his e-mail address, he could put Alice on his whitelist immediately, or have given Alice a password that would automatically get her past the screening process on the first try. If Bob was really interested in getting Alice to go out with him again, he could have sent a request through through whatever common friend or dating service first introduced the two of them.

Alice could have not sent a second e-mail, but instead contacted Bob through whatever common friend or dating service first introduced the two of them. Alice could have also asked Bob for a secondary contact means during the first date. She also could have asked for a better communication means than e-mail if she wanted to make sure she could get through.

You should never make any important decisions based on an unencrypted e-mail alone unless you can contact the sender in another medium to confirm that the message you got was one they sent and meant. If you're girlfriend is dumping you by e-mail, veryify that with a phone call or at least confirm that she's refusing to take your calls... you do need make sure it's not somebody else trying to freak you out by making you think your girlfriend dumped you before you take any irreversable actions based on that information.

Re:CR deadlock

[Skim123]

Well, a more realistic example would be where I run a Web site with many readers. Someone wants to send me feedback, so I have no idea that they should be on a whitelist. Or perhaps someone is sending me an email to reply privately to a USENET post I made. Etc., etc. It's naive to think that both parties will always have some alternate means to communicate with one another. Now, you may discern that such email isn't vital, and htat it's ok if it gets lost in the C/R deadly embrace, but Alice and Bob might not feel the same way...........

Re:CR deadlock

[Anonymous Coward]

This is what happens when engineers spend too much time working out the specifics of a communication medium, and not enough time going on dates. :-|

Re:CR deadlock

[Anonymous Coward]

Alice doesn't use C/R, but rather a filter. Her filter, unfortunately, marks Bob's challenge as spam. Since Alice is only a computer novice, she does not know how to check his Junk Email folder, and therefore never receives Bob's challenge, hence Bob never gets Alice's email. Alice, who is blissfully ignorant of the "behind-the-scenes" happenings, thinks Bob just is trying to ignore her. So she sends another email, which is, of course, not received by Bob. And she sends another. Still, no response from Bob. Alice takes it personally and decides if she does ever hear from Bob again she won't be going on a second date with him no matter what.

One question: Do you have Alice's number?

Re:CR deadlock

[berbo]

" Alice takes it personally and decides if she does ever hear from Bob again she won't be going on a second date with him no matter what." Excellent. Clueless luses like Alice shouldn't be allowed to breed. -berbo

Just right

[KalvinB]

"then it's just not a mainstream solution"

This isn't for the mainstream. And it's not difficult for anyone to get the answer. A computer may exist that can find the solution but it's a lot easier and a lot cheaper to bark up someone else's tree.

"If it's something that you have to explain to your parents how it works"

Why would I have to explain to my parents how to answer a question? And why would I force them to go through a script? I can access the common account from any e-mail client since I have the password. All the script is, is a custom e-mail client.

There are a million and one ways to organize a challenge system that will confound any computer that a spammer has access to. If not completely, to the point that it's a waste of time to try to break the system.

You seem to be confusing what computers can do with what is feasible.

"Besides, you're still talking about text, and if the answer can be found on a Google search, then a script can figure it out, too."

Scripts can't understand context. They may find the question but they won't be able to discern what text on the page is the answer. It's one thing to just assume it can be done and quite another to assume someone is going to go to the trouble of trying it.

The only reason to try to break a script set up like mine is purely academic. It takes all of two seconds to break the script so that no answer is correct and then a day or two to swap it out with something else.

The amount of effort needed to break a bot is infinitly less than the effort needed to come up with the bot.

"Show all of the images as grayscale so that image scanning software can't tell the difference between the yellow of a sunflower and the gray of a building."

Why? Anyone with that kind of money to afford such a system that could do those opperations has the money and know how to set up their own anonymous mailer.

Again, you're confusing what's feasible with what's possible.

Ben

Re:Just right

[FLEB]

And, on that note, if you're not in the mainstream, you're probably not going to get spammed. It doesn't make economic sense to think that someone with ANY sort of decent anti-spam technique would be remotely interested in buying a product or falling for a scam. It's just not worth the cycles.

I recall a honeypot-based study that showed that nobody even cared about reconstructing logically-mangled addresses (this at that dot mil), since the brain time and proc time aren't worth diverting from the steady flow of idiots who ARE wide open to the world.

Time ...

[TekGoNos]

Always depends on the amount of money involved.

Neural networks offer a good solution to fuzzy problems. An interesting point of neural networks is that you can train them and once they are stabilized you can burn them into a chip.
Making the chip isnt cheap, but once you have one, you can run it at MHz speed and decode one image every cycle! So a challenge that takes a long time on a normal PC will just take one cycle on dedicated hardware. And as spammers will be likely to get such dedicated hardware, every time advantage is void.

And about your problems that are currently hard for computers : The problem with these is that you either :
- have to classify a limited number of examples by humans. Than spammers could just get all examples and classify them by hand too.
- or have the computer generate an exemple following some rules. In this case you have rules + random noise. But noise reduction has become quite good and I dont know any example of a rule that can easily be reversed by a human but not by a computer (an "one-AI-way" function?)

And on facial recognition : there have been huge advances in facial recognition lately, to a point where there are the first prototypes that use facial recognition as a password.
There is still work to do to recognize a face from any angle, but even there is progress.

Since AI has stopped doing nothing other than trying to pass the turing test, there has been a huge advance on practical applications of it.

Thinking in concepts (needed for natural language) is really hard.
But just recognizing things is much easier and AI is getting better and better at it.

Re:Time ...

[psetzer]

I wouldn't burn any algorithm to a chip in an adversarial system. Sure, you can do it faster with a custom chip, but they just have to change the challenge enough to screw up the chip, and it's several million dollars down the drain in development costs. However, I agree that a Neural Network would have potential as long as you can break it up into single letter chunks. Let's look at some of the characteristics of the challenge shown. 1. The characters chosen are on average darker than the background. 2. The characters are only a subset of the alphabet, reducing the amount of things a computer would need to search for. 3. No letter overlaps another letter. 4. There is always at least one pixel between each of the letters. Given this, I think that the problem is solvable, but I doubt that it's solvable in a practical amount of time.

CR deadlock is a bogeyman

[aminorex]

The CR traffic does not need to be filtered.
It's a UI issue is all.

Consider for example, a system in which I send you
a cold email. Your MUA caches it, and sends me a
challenge token. My MUA recognizes the challenge
token and the fact that it corresponds to a sent
mail. It gets flagged and replies with my response,
a pgp public key. Now you can see my mail, and
decide if you want to blacklist me, or whitelist me.
All future mail is private.

It's just software -- go write it.

awww looking for sympathy?

[Anonymous Coward]

Try looking between shit and syphillis in the dictionary.

This is quite easily defeated using bounded sets.

[Ayanami Rei]

I am noting that each character is a single, solid, filled region of one color, without aliasing. The background is a repeating pattern of different colored lines

You could do a reduction looking for regions that are two are more pixels thick, and leave only the letters... very simple operation. Then all you have to do is use correlation against the constant font, iterating through each letter, and mark the position of highest value, sort by increasing x, then record the letters found sorted by x, and then post into the box.

If I had access to matlab this semester, I could code a solution in about 6 hours that would crack it in less than 5 seconds, for any image.

Even if they used noise, you can still use a least-squares match with the original font set (just a dumb iterated correlation/RMS/sort operation) to find a position 99% of the time. If noise is a problem you can apply a spatial "emboss" type filter and use letter outlines to match instead of letters themselves.

And if the computer make a mistake because it got a pathological one, maybe the next one will work. No big deal. Just detect the error on the resulting page and try again.

What about typos?

[TekGoNos]

Humans do them lal teh tiem.

So you cant just block someone after one mistake.

You just have to get your computer program better than the average typo occurance.

Oh ... and remember the Slashdot story a few weaks ago where a computer spam filter was MORE accurate than the human testers. (Yeah, it probably was spam filter reads whole message vs. human reads only subject, but still ...)

I think there are many tasks where a well trained computer program will perform even better than the average human.

Re:What about typos?

[LostCluster]

I wouldn't block anybody after their first mistake. However, there comes a point where too many mistakes indicate either a robotic attempt that isn't learning from its errors, or a really stupid human who likely can't compose a useful e-mail either.

Many spammers who are trying to beat a Bayes filter are either using misspellings of their most spammy words, or large lists of random dictionary words to try to lower their score. However, a coutermeasure to that would be to factor in the results of a spell check and grammar check. Some errors can be tolerated, however having too many mispellings and too many word groups that can't possibly be a proper sentance should raise the score enough to counteract the attempts to lower it and then some.

Re:What about typos?

[GreeboNZ]

>However, a coutermeasure to that would be to factor in the results of a spell check and grammar check.
>Some errors can be tolerated, however having too many mispellings and too many word groups that can't
>possibly be a proper sentance should raise the score enough to counteract the attempts to lower it and
>then some.

Ah.. finally - an intelligence test is required to use the internet. (Or some fraction thereof.)

Oh, but the easiest defeat of C/R...

[Ayanami Rei]

most C/R engines use a constant suite of pictures and words because the pictures are too time consuming to create on the fly... so the signup page might take too long to load.

What the spammers do is just download as many challenges as possible, solve them, and store the hashes in a database.

When the harvester goes out, it is likely to encounter many of the challenges a second time, and it already has the answer. :-)

If it doesn't know it, it flags the spammer, who identifies it offline, adding it back in, and the database is that much more useful.

Re:Oh, but the easiest defeat of C/R...

[Tony-A]

Good one that.

If this becomes a race between the "good guys" and the "bad guys", the bad guys have more incentive to get it right. Just like virus writers will buy anti-virus software, spammers will buy the C/R software. You don't attack your enemy's strengths, you attack his weaknesses, preferably ones he doesn't even know about.

The spammers weak spot is the money he makes.

[sbaker]

I think we are attacking Spam from the wrong direction. Attempting to stem the flood of incoming spam is tough - everything about the identity of the incoming spam can be faked. However, we could alternatively attempt to prevent the replies going back the other way.

There are two inevitable facts:

1) In order for spamming to be worth someone's effort, they have to somehow get money from people. If NOBODY replied to them, then spamming would stop overnight.

2) Something in the content of the Spam must be real - a reply address - a web site, a phone number or something. Block traffic to that location and the spammer gets no money and dies.

Hence, I think they may be vulnerable. Educating people not to reply to SPAM would help - it only takes a mere handful of people to respond to a SPAM to make it profitable - but if education could drop that handful to a mere one or two - then we could succeed in putting more spammers out of business simply by cutting their margins to the point where it wasn't worth the hassle.

Where are the TV adverts: "Replying to Spam is Bad!"....we know that the morons who reply to spam are suckers for advertising - they are as likely to believe a well targetted TV advert as a crappy email shot. If Spam is costing the ISP's as much as they say it does - then funding some TV ads might not be impossible.

What if we made it illegal to respond to an emailed advertisement that was not clearly labelled as such, that would help to deter people from responding. Such a law would be next to impossible to enforce - but we are trying to deter the gullible here - so it might not have to be enforcable - just very well advertised.

Since every SPAM has to either advertise a product that you can buy from somewhere - or direct you to a postal address, a phone number or a web site - then that route for getting money back to the spammer could be blocked.

The return route has to be genuine. There is no point in them sending you a fake phone number or faked web address. If the phone companies (who are often also ISP's - or have at least some cause to want to kill spam) were to block calls to and from phone numbers that were seen in Spam - then the reverse route for the money would be curtailed. Whilst you can afford to change the aparrent source of your spam and fake those addresses for each new mail shot, you can't change your phone number for every couple of dozen orders you take. Similar considerations apply to web sites and postal addresses.

If it was required for credit card companies not to transfer money to businesses that employed spammers to push their goods - then that would also help some.

It wouldn't take many people to deliberately reply to spammers - to lead them on into thinking you want their product - to send them fake cheques or bogus credit card numbers. If they only get a handful of positive responses per million spams - then it wouldn't take more than a few determined people per million (eg ISP employees) to clutter up the the spammer's cash collection mechanism to the point where it's too much hassle for him to sort out the real orders from the bogus ones.

I don't pretend to have all of the answers - but there seems to be far too little creative thinking along these lines.

Re:The spammers weak spot is the money he makes.

[yudan]

Sorry but your idea does NOT work.

1. Most spammers use faked email address, they DO NOT suppose you to answer them. They want you to click the link, they want you to buy something, they want to install some spyware, adware or what-so-ever-ware on your computer!

2. Who can block the phone call to a certain number, who can block everyone's access to a certain website, and who can block a real physical position (address)? Spammers make profit in the hope that 0.000001% of the receivers would click the link, make a phone call, or write a snailmail to that address.

It seems that you don't understand how spamming works. This is a social problem, and cannot simply be "blocked".

Re:The spammers weak spot is the money he makes.

[MyFourthAccount]

If it was required for credit card companies not to transfer money to businesses that employed spammers to push their goods - then that would also help some.

Here's another one for you: if everyone receiving spam would use their credit card to purchase the product, and subsequently call the credit card company to dispute the charge (honestly, my willy did NOT grow by 2 inches!), then:
a) the credit card company would no longer provide services to that business because of the large number of charge-backs
b) the spammers would hurt badly because of the incurred cost of charge-backs (around $25 or so, from what I understand).

Of course I'm mostly joking, but if done in an organized manner, it would be possible to hurt some of the spammers badly. Possibly enough to make the others think twice about their practice.

Re:The spammers weak spot is the money he makes.

[sbaker]

1. Most spammers use faked email address, they DO NOT suppose you to answer them. They want you to click the link, they want you to buy something, they want to install some spyware, adware or what-so-ever-ware on your computer!

I agree that the email address they give is likely to be faked - but my point is that in order to make money, SOMETHING in that post has to be real. If not the email address then the postal address, phone number, web site, etc.

2. Who can block the phone call to a certain number, who can block everyone's access to a certain website, and who can block a real physical position (address)?

The government could pass laws requiring phone companies, ISP's and the US mail to block traffic to people who have been logged as advertising illegally via email. It would require an efficient method to collect these addresses and automation to do the banning - but that's within the bounds of technical possibility.

A spammer can change his email address for every spam he sends - but he can't change his web site that often - and he certainly can't keep changing his phone number, physical address or bank account. I read somewhere that 90% of spam comes from just 600 people. It can't be that hard to block the money going back to those 600 people.

Spammers make profit in the hope that 0.000001% of the receivers would click the link, make a phone call, or write a snailmail to that address.

Yes - exactly. But if you can add a couple of zeroes to that 0.0000001% then it won't be worth their while. If every million email spamshot nets them 50 orders (a number I read somewhere as typical) - then they can make just a couple of bucks on each order and they have earned $100 for the time it took to type a single Spam and to run their system to send it. That's good money.

However, if you can get the numbers down to where they have to send several different mailshots to get even one order - then it starts to look like a pretty unprofitable business model and they'll stop doing it.

It seems that you don't understand how spamming works. This is a social problem, and cannot simply be "blocked".

I think I do understand how it works. I absolutely agree that blocking the spam isn't the answer - and that's my entire point. Removing the spammer's motive for sending the spam in the first place is the only answer IMHO.

Re:The spammers weak spot is the money he makes.

[sbaker]

Here's another one for you: if everyone receiving spam would use their credit card to purchase the product, and subsequently call the credit card company to dispute the charge (honestly, my willy did NOT grow by 2 inches!), then:
a) the credit card company would no longer provide services to that business because of the large number of charge-backs
b) the spammers would hurt badly because of the incurred cost of charge-backs (around $25 or so, from what I understand).


Right - but if appropriate laws were passed, it would be possible to simply
have the credit card companies block these businesses automatically - or even to automatically refund everyone who'd bought something from them. That kind of
action would hurt quite a bit.

Of course I'm mostly joking, but if done in an organized manner, it would be possible to hurt some of the spammers badly. Possibly enough to make the others think twice about their practice.

I like to think of this as reverse-spamming where we'd inconvenience them by
sending them false orders, false payments, etc - just as they inconvenience us by sending bogus email. However, the number of people doing the reverse spamming would have to *far* exceed the number of people who are genuinely trying to buy the spammers products if we wanted to dissuade the spammers.

The trouble is that there are laws against passing bad checks - and relying on a credit card company to refund me seems a little 'iffy'.

Re:The spammers weak spot is the money he makes.

[hords]

Sorry, but his idea *does* work.

I am the main mail administrator for a medium sized ISP. I came up with the same solution basically. The spammers are making money, usually by a link within the message. I modified qmail and kept track of every message that came in. I then input the urls from these messages into a database. I can whitelist and blacklist the domains. I have qmail scan the message as it comes in. If it senses a blacklisted domain it does not accept the message it instead gives a 554 error message.

I found that the worst of the spammers change their domains often (like once a week.) So after running this thing for like 6 months+ now I decided that I had a huge database of valid vs. non-valid domains. I was a little tired of blocking the same people over and over, it was time to automatically block domains. So a domain that has never been sent to us in an email before automatically gets blocked after so many messages. It works beautifully. I have found very few legit domains that I had to unblock. Mostly from mailing lists like rootsweb, groups.yahoo.com, etc, which I have now stopped automatically blocking theirs.

We are basically blocking ~85% of all email attempting to come into our mail servers now. It was really out of no choice. About a year ago our mail servers were choking so hard that we had some major service interruptions. One time we had over 400,000 messages in our queue. This was just unacceptable! We found that we were getting about 2,000,000 emails a day.

Methods of blockings in order...
1 - RBLs
2 - Recipient Unknown Block 3 - Keyword Blocking
4 - Url Blocking
5 - Attachment Blocking
6 - Virus Quarantining (clam av)
7 - Customer modifiable email white/black lists
8 - Spam Assassin w/Bayes,DCC,Razor2,Pyzor

Example of a five minute period on one of our mail servers. This is both incomming and outgoing combined:
2004-03-02 21:20:01: 5418 out of 6490 (83%) messages blocked.
New Domains: 5
Blocked File Extensions: 12
Blocked Domains within URLs: 258
Blocked IPs within URLs: 11
Blocked URLs (Domain contains % signs): 14
Blocked URLs (Domain contains & signs): 0
Blocked Recipients (User Unknown): 67
Messages Blocked by RBLs: 1131
Messages Blocked by keywords (viagra, etc): 3924

The really cool thing is that if we shut off the RBL blocking, the URL and keyword blocking totally take its place! With the RBL blocking on we still block about 100,000 emails a day with a variation of the word viagra!

I keep hearing people say that they shut off our spam filter (#7 and 8) because spam is so light now. Anyway, with some effort and creativity you *can* make a huge dent in spam. Nothing is perfect, but I would say that we have gotten it down to 95%+ easily.

Sounds like a good thing

[tehdaemon]

In the case where bob's email address is spoofed, wouldn't a cr loop be a Good Thing?

Re:Sounds like a good thing

[janolder]

Afraid not. The loop would be between two "friendlies": Bob and yourself. The spammer would get a great laugh out of it - more wasted bandwidth and headache for you at no cost to him.

Solution is to prevent cr loops by caching previous cr response targets and not responding to them again.

Seems to me..

[Trystansr]

that you would still have to sort thru every new message that came in, even if it's just "yes/no" to the address, in which case you havent really solved anything.

Yes, of course...

[michaeltoe]

This is similar to the argument that a computer cannot determine when it's in an infinite loop. Humans, however, can... because they are impatient, and given time, will reexamine the code that is executing.

Naturally we may be inclined to believe that this grants us superiority to the computer. That, while stating some arbitrary facts taken from some textbook somewhere, a computer can never accomplish X objective.

Therein lies the fallacy. The computer does not identify that it is in an infinite loop, nor can it, because it is not given the benefit of looking at the actual code. If a compiler were designed to read into code for things like while(true) loops, which naturally could result in infinite loops, then already you would be cutting back on the instances of these problems.

Determining if there is an infinite loop requires a conscious understanding of the code itself, which is no trivial matter. It is not, however, something that could be deemed impossible.

As with all fields of science, there will be those who say "Well, I haven't seen it yet, so it will never happen"... but skeptics are everywhere, and the presence of skepticism is hardly a measure of credibility... rather, a measure of how pious certain peoples assumptions are.

Solutions are always found in math, and never in magic. Don't underestimate the computer, and more importantly, don't underestimate your own brain. You don't perceive things the way you do 'just because'... and that's what's so exciting.

Re:Yes, of course...

[calambrac]

The computer isn't given the benefit of looking at the actual code? What are you talking about? If the compiler were designed to read into code for things like while(true) loops, this would be solved?

Allright, tell me how to write the compiler to catch this:

int i=1;
int x=new_x(); //random int x>1
int y=new_y(x); //random int 0<y<x
while(i%x != 0)
{
x=new_x(); //random int x>1
y=new_y(x); //random int 0<y<x
i=(x*i)+(x-y);
}
printf("halt");

Will halt ever print? Should I ask my computer? Maybe in a few years when processors are waaay faster and we have those fancy "while(true)"-proof compilers?

Uhmmm....

[mdfst13]

You do realize that image recognition is one of the *hardest* tasks to do with a computer right? It would probably be cheaper just to hire a person to respond to the challenges than to automate this. Just like it is cheaper to use people to sort parts and insert them into a machine than it is to make a robot to do it.

In and of itself, it would be computationally difficult (if it is even possible) to scan these images consistently. Implementing such a thing would increase the cost of spamming sharply (image recognition is way more resource intensive than sending email).

It's worth noting that TMDA.net includes the ability to include these kinds of challenges in the response but does not use them. Why? They aren't necessary. Spammers don't use valid email addresses to send their emails. Forget the question of whether they could automate responses; they aren't set up to receive responses, much less answer them. This is unlikely to change, as receiving responses would double the work of sending spam. Answering the responses would triple it -- even without complicated challenge systems.

What happens if they do create an automated response? They successfully get through so that ... their newly validated email address can be blacklisted. If the email address is blacklisted quickly and broadly (through spamhaus or similar), then it will miss most of the intended recipients. Again, this increases the costs of sending SPAM.

No one method is not going to eliminate SPAM. Every anti-spam method has a response. However, each response increases the complexity of spamming, which reduces the profitability.

There is no CR deadlock problem. Yes, it is theoretically possible, but every Challenge/Response system has checks to handle this. In particular, if you send someone an email, TMDA.net whitelists that person. Thus, when you get their challenge, you will accept it rather than challenging.

It is true that automated systems will have trouble with challenge/response systems. After all, this is the whole point. Who cares? If you sign up for it, you can whitelist the sender. If someone else requests it for you, then they can answer the challenge (or maybe they don't need to do so, as they are already whitelisted). It would be easy enough to switch the tell a friend systems to launch the email client to send the email.

The author correctly reviews the issues with Computational Challenge systems. Mainly because those systems aren't well thought out (Thank You Microsoft).

I am unsure that public key interfaces like GPG offer much in fighting SPAM (although the encryption aspect can be useful in and of itself). Otoh, they are very good at whitelisting your own circle of friends and catching joe jobs in that circle. I have not done a more in depth study (any more than the author did) to see if they would work well in more general situations. My feeling is that domain verification like SPF is sufficient there.

The author's previous article mentioned the domain verifiers and once again missed the point. His two problems are mobile computing and domains that are host-less or vanity. He starts talking about sending email through ISP mail servers, but everyone already knows that this is incorrect. Instead, one should send through a mail server associated with the domain name by using SMTP AUTH. In general, vanity domains already come with these. Mobile computers should use these for security's sake anyway. Host-less forwarding accounts should demand that they get an SMTP server as well. Most of them already come with webmail (which has its own SMTP server) anyway. This is a downside but only a minor one.

Our security *expert* notices that POP mail authenticates in plain text by default. For some reason, he associates this with sending mail, even though POP is used by the *recipient*. Possibly POP before SMTP is the issue. Minor issue, since SMTP AUTH is strictly preferable to a POP before system.

He does make

Attack spam at the true source...

[TwoBit]

http://ppedriana.homeip.net/blog/SpamScreensaver.h tml

What..?

[Ayanami Rei]

So what, do you tell all your friends the password so they can send mail to you?

You know you could achieve the same effect by telling them to all put [this_is_the_password] in the subject line, and filtering out anything that doesn't have it...

Why change the protocol when the existing software can already do this? But I don't know why you would, it seems like it could be a nightmare: people have enough trouble remembering passwords as it is.

Re:What..?

[Anonymous Coward]

people have enough trouble remembering passwords as it is.

That's why you'd want to make some minor changes to e-mail clients to store passwords in the addressbook for outgoing mail.

Not for long.

[Ayanami Rei]

Just wait till telemarkerters combine open relays with VoIP... then you'll never be able to stop the calls.

Violence may be justified

[DonGar]

I think there is a possible moral case for capital punishment against any form of intrusive marketing used on a large enough scale.

I'll explain by giving an example.

Lets assume a large scale spanner sends out about 6,000,000 spam messages a day (figured based on vague memories of slashdot articles). They does this 365 days a year. Let's assume that, on average, each of those emails wastes 1 second of someones time. This is a rough guess based on the time to recognize the message as trash and hit delete, or to setup and maintian spam filters, support the the extra load on mail servers, to vent anger and calm down enough to be productive, etc.

This means that they are wasting 2,190,000,000 seconds of other peoples lives every year. That means they are wasting a bit over 66 years of other peoples time per year. That means that they are wasting an entire human life a year. Even though they are only doing a little harm to many individuals, I see this as being the same as killing someone.

Yes, they make a lot of money, but in effect, they steal a life a year to do it. These figures are based on a lot of vague guess work, but they aren't THAT far out of line. I'm mean, would it be okay, if they only steal 30 years per year they are in business? 20?

I see legally sanctioned violence (meaning trials and executions or life imprisonments) as a reasonable response from society.

I also see this as a rough formula for measuring the damage done by any form intrusive marketing. How much time is someone stealing? Are we as a society willing to allow them to steal that much?

Re:Violence may be justified

[Anonymous Coward]

So how many lives did Enron Execs steal, if you consider the employee's retirement, unpaid corp taxes and ruining the California economy for 5 years or so?

The PHD boy is on crack...

[Vellmont]

Don't ever say a PHD in Computer Science, and "15 years of experience in computer security" means you can't say something stupid. You're right, you don't need a private key to do authentication, you need the public key. The private key holder is the only one that can sign messages.

The fact that Mr. Krawetz doesn't understand basic public key crypto makes me question his credibility and this entire article. I don't make it a habbit of studying spam solutions, and I understand authentication schemes. This guy writes a damn article about it in Security Focus, and makes a MAJOR mistake about the very basis of public key authentication. Sorry Neal, but you're on my bozo list now.

Another solution

[mdfst13]

Drop SMTP in favor of a system where email is kept on the *sending* server until you request it. Sending an email would involve sending the message envelope (sender name, etc.) along with an access key (to identify you as the actual recipient of the email). If you look at the envelope and decide to open it, then you connect to the sending server and download the rest of the message.

Why this helps:

1. It moves the burden from the receiving server to the sending server. Sending now takes up ISP bandwidth and storage rather than putting that burden on the receiver's ISP.

2. As this is kept on the *original* sender's server, there is no point in open relays. You still have to reveal the *actual* mail server that is sending the email. Same with open proxies.

3. Because of that, blacklists work. You can now blacklist the storage facility IP, which can't be hidden (otherwise you wouldn't be able to connect and get the email).

4. Virus spammers can no longer send directly to the recipient's server (unless it will also serve up the messages). They will have to use an intermediate (sending) server. This server will be able to see that it is sending loads of messages, delete the message, and cancel the envelopes.

You can do sender verification as part of this as well if you like. The public key would go into the envelope.

Re: Finding things

[some guy I know]

Oh I wish I had the article.

Google is your friend.
The first result points to the Wired article.

Re:Let's use the Patriot Act for the benefit of go

[Anonymous Coward]

The instant spammers start pumping out emails questioning Cheney's secret energy meetings, Bush's being AWOL from the National Guard, or what will happen as a result of the U.S. leading the way for nuclear powers to adopt preemptive strike policies, you can bet that the Patriot Act WILL be invoked to shut them down. Heck, for all we know, Guantanamo Bay already has a few spammers in it. Maybe this Orwellian nightmare isn't such a bad thing after all.

Just use 'Parity Even' Spam Elimination!

[Anonymous Coward]

Publish TWO email addresses and let people use either of them.

Spammers will use both and if the addresses are alphabetically close you will receive the same spam in the same order at both addresses and have no problems getting rid of all of it.

With this 'parity even' spam elimination *I*/You really dont need anything but a lil comparatorator to get rid of all your spam.

I junk more than 1k messages this way daily.
About 80% are caught in the first pass (message sequence) the rest after the balance of the messages are compared when sorted by size.

Thanks very much, can we finally move off this silly topic?

Easy solution for windows

[skydancer666]

i'm suprised no one wrote about mailwasher yet ... its pretty simple , it connects and lists whats on the pop server, and lets you select what to trash (with the option to block entires domains) and what to keep (friend list). pretty soon, it has a pretty big blacklist & friend list and does most of the work by itself. It can also send a fake bounceback message, but i don't really trust it... http://www.mailwasher.net for the interested

Negative Feedback

[OGmofo]

Counter Spam Measure: Negative Feedback.

Imagine if all or some very large contingent of email clients allowed you to
"retaliate" against spam messages. Highlight message, select "negative feedback"
option, a daemon is spun that traces back as far as possible the route of the
message and barrages it some fashion. By pings maybe? By directed replies? Imagine
it does this in some scheduled fashion so as to minimize the impact on your local
network. As 1 million disparate sources converge upon the last traceable source of
the route of the offending spammer, some network somewhere will start to feel the
load. Like the spokes of a wheel converging on the hub, the retaliation traffic will
thicken as it closes in on the source. The pain increases. ISPs inundated by
individuals expressing their right to freedom of speech, will feel suddenly inclined
to exercise their right to refuse service to someone.

The "negative feedback" could be dosed in a coordinated fashion if there were some
P2P means of establishing how many individuals had received a particular spam. If a
spammer hits only a hundred people, the dose of retaliatory traffic would have to be
increased to be felt. If the spam hit a million, it would require only a modest
retaliation to utterly swamp the source.

Just thinking out loud. Could this be made to work? No one's free speech is
curtailed, spam is dealt a serious blow.

fight fire with fire.

Not Solutions. Explains some ideas that won't work

[MMHere]

The article and its predecessor (part I and II) do not provide any solutions. Both articles simply run down a laundry list of currently proposed solutions, explaining along the way why none of them will work.

Spam sucks, but this article doesn't help provide a solution. It does help frame the discussion, and provides coherent analysis of problems with currently proposed ideas.

Damn. I was hoping for a fix.

What about a web of signed trust?

[Muerte23]

So what about this:

You start with a central certificate authority. I know, I know, bottlenecks. But you only need them to issue keys to (or sign the keys of) about 100 (or 1000?) servers. The signing authority has to be central, but the *revocation* authority does not. That's the key here.

So those servers can sign the keys of 1000 servers of their own and so on.

So my mail server tries to send your server an email. Your server checks if my key is signed by someone who is signed by someone who is signed by the CA. It also checks against its nightly downloaded revocation list. If everything is good, the mail goes through. Very little processor time, and very little bandwidth.

Suppose someone issues a key to a dishonest server? Well, enough people issue complaints and the issuer's key gets revoked. Or some automatied spamassasin type thing that auto-revokes the key after enough spams get spotted. No more spam from them, and maybe next time the admins are more careful.

This totally eliminates (i think) the threat of zombie SMTP servers on DSL and open relays.

Then the ball is in the park of the ISPs and server hosters (those with their own email keys) to keep spammers out locally. SLL login for SMTP? sure. C/R for each email sent through them? Whatever. Send anything over their open relay? Not for long.

Sounds reasonable to me. It makes it easier for the end user I think, and minimizes spam.

Any suggestions?

Muerte
This totally eliminates zombie SMTP servers on cable lines spewing spam.

Re:What about a web of signed trust?

[WuphonsReach]

Suppose someone issues a key to a dishonest server? Well, enough people issue complaints and the issuer's key gets revoked. Or some automatied spamassasin type thing that auto-revokes the key after enough spams get spotted. No more spam from them, and maybe next time the admins are more careful.

Or spammers attack the system so that legitimate servers get blacklisted/revoked which reduces trust in the system. (Centralized solutions are prone to revenge-attacks and other fun things like censorship.)

Re:What about a web of signed trust?

[vegetablespork]

The only problem is that no one is going to want to pay Verisign, Microsoft, or whoever the trust root would end up being in this scheme to send email.

SPF broken

[0x0d0a]

As I've detailed before every time there's an SPF article on Slashdot, SPF is seriously broken in a number of ways. It was not produced by security people. Microsoft's counterproposal fixes a few of SPF's problems and leaves open a number more.

I thought that the article as a whole was a very realistic and down-to-earth view of what's going on -- it's really nice to see someone in the anti-spam world that knows what they're talking about.

I do have two points -- I really think that despite the drawbacks mentioned, cryptographic systems must ultimately become the final anti-spam solution. There are a number of ways to deal with the issues Neal brought up:

* Automated abuse: this is significant -- it does mean that no ISP can provide an abusable automated system for generating new certs. This is not an insurmountable technical issue, however. Is is an issue for the early transition to Internet-wide email cryptography. One can link trust of multiple identities, so that people can "obtain" up to, say, six new "identity" certs from their ISP per address in an automated fashion, but that if any of these certs are abused, the trust of all of them falls. There needs to be a secure system for granting certs already for webservers or for administering domains. It does not seem unreasonable that piggybacking one of these distribution systems could not be used for handing certs to ISPs/domain owners. If I go to work at a company, the IT people that set up my email client and whatnot just drop my "identity cert" in when setting it up.

* Usability: I see no reason for Neal's assertion that a CA going down would break everything as a problem. At the moment, almost all people rely on MX records to get their email. If the DNS server goes down, they don't get their mail. Is it so much harder to host whatever CA stuff is an issue on a server of DNS reliability? I don't see CA load being an issue -- it's not as if the CA is going to generate a cert per-email in any kind of a sane system. Certs could be cached, cert chains could be attached to emails, etc.

There are three big reasons cryptography is seen as an unpopular solution (and why Microsoft and the SPF people don't like the cryptography path). First, having a cryptosystem that allows signing in place probably means that people are going to include encryption support with it. A lot of governments really, really do not want to lose the ability to read email. I think this can be gotten around by forcing ISPs in such unenglightened countries to use key escrow. Second, a lot of people are concerned about server CPU load. I just don't see this as a problem. I cause more CPU load on systems webbrowsing than I do sending my tiny average number of bytes in email each day. Third, cryptography isn't easy to deploy. You need essentially everyone to be using it for it to be useful (though you can start tying it in to existing antispam filters immediately, treating a signed email as far more trustworthy than an unsigned one -- SpamAssassin can already do this). You need to do PKI, handing out certificates to users, and you need to ensure that these certs are kept reasonably secure.

I don't see Neal's complaint about "nobody should control email" as a problem. Is that an issue? Fine...have, say, several CA roots per country, and allow people to add more if they want. Use a trust system that allows CAs to become untrusted (I've gotten all these spam emails from this domain that this CA alone signed off on...) That should be enough distribution that it's damned hard for anyone to muck around with email as Verisign does with DNS. Also, instead of having CA roots directly sign domains, have CA roots sign "signer servers" that are responsible for handing out certs to domain owners (and have. That way, if I want to set up a root CA tomorrow for my friends to use, all I have to do is kick it on, sign all the "signer servers" for name registrars or whatever that I trust, and then tell my friends to add the root CA as trusted). E

SpamAssassin/ClamAV should be default client-side

[0x0d0a]

IMHO, SA and CAV should be set up on client-side boxes by default in Red Hat and other distros, run through procmail or whatnot.

They are absolutely amazing programs when used in conjuction, and I think that too much focus is put on their server-side use -- I *know* that my anti-spam/anti-virus system is excellent, even if my ISP's isn't...and *I* can whitelist and use my own Baysian filtering on things, unlike server-side SA/CAV users.

It is extremely unfortunate that SA/CAV take effort to set up. I consider them more useful and fundamental than the firewall that Red Hat ships in their basic distro.

Trust systems better with cryptography

[0x0d0a]

This sort of trust system works better with cryptography.

The idea is that people move around IPs, IPs change ownership, people use tunneling, IPv6 comes out, etc.

However, if everyone sends their email with a cryptographic identity and uses these identities instead of IPs to identify people, then you can do the trust system you proposed (or another trust system) and it works reliably. It can take some work to set up, yes...

Spam about Bush! It's legal!

[0x0d0a]

Actually, when the current federal antispam law went in, "political emails" were exempted -- those legislators wanted to be able to get in on this cheap communication, even if companies couldn't.

Of course, that means that spamming people regarding Bush's or Cheney's flaws is entirely legal and legitimate.

Pollution of key registries

[0x0d0a]

They're also polluting key registries with their garbage - that's a big negative.

So do something like having each domain maintain a key registry server, just as they currently do for DNS. It's not that hard -- it's just up until now, PGP needs have been met by a handful of servers, so each server handles "all and sundry" domains.

What about C/R with images?

[zarkzervo]

To train a neural network to recognize letters divided by open space seems quite easy. I've seen school projects recognizing signatures after 'minimal' training.

What if you create image-elements or rather drawing elements put together and the user has to press buttons for all elements in the picture. "a PIG with HORNS, JUMPING over a FENCE". Or let the user write in "PIG".

There Is Only One Solution

[Anonymous Coward]

I've proposed this before and was modded a troll, then modded funny, then modded insightful, but I was being serious!

The only way to stop SPAM is vigilantism. A bounty on the head of every spammer. And not dead or alive. Just dead.

A few dead spammers would stop the problem. It's not a moral dilemna. These scum aren't human and they don't deserve mercy.

Don't tell me you're not nodding your head. You know it makes sense. I'm Sam Kekovich.

Didn't see this solution...

[kwenda]

I didn't see any mention of a pretty good solution that i've run across:

Every time a message hits a server from a sender that it has never met before, it sends a TEMPFAIL back instead of accepting the message. All real MTAs will try again with whatever their retry delay is set to, and usually for about 4 days. If the server gets the same message being delivered again, it accepts it and adds the sender to a whitelist where it never has to 'ask questions' of this sender again.

The reason that this would work, at least for now, is that spammers mostly use badly written MTAs (or something akin to an Expect script posing as an MTA). Their software doesn't know how to deal with a TEMPFAIL and never tries again. All real MTAs will try again within a few minutes. Good times.

Re:Didn't see this solution...

[WuphonsReach]

The reason that this would work, at least for now, is that spammers mostly use badly written MTAs (or something akin to an Expect script posing as an MTA). Their software doesn't know how to deal with a TEMPFAIL and never tries again. All real MTAs will try again within a few minutes. Good times.

Um, any missing functionality in spammer zombie MTAs would quickly be patched by the bad guys to get past that hurdle. So it wouldn't be much of a barrier (maybe a day). It works right now because you're the only one doing things that way (or are part of a very limited group).

(Which gets into the whole monoculture/polyculture security theories... distributed/local decision making on spam/ham is better then centralized/monofocused.)

Re:Didn't see this solution...

[scrytch]

> The reason that this would work, at least for now, is that spammers mostly use badly written MTAs

Part of what I do for a living is looking at spamware, though not to the degree of analyzing it in depth (that's what I'm aiming for, but sadly I won't be able to share the findings). Fact is, most spamware simply re-queues rejects. Hell, lots of them simply run over the same list time and again with different forged senders, regardless of ANY protocol responses.

This scheme would more than double the traffic of legitimate mail, something a large ISP like AOL could ill-afford (well they could *afford* it financially, but it would be painful). It might have marginal benefit, but I do stress "marginal".

Re:Didn't see this solution...

[SuiteSisterMary]

This scheme would more than double the traffic of legitimate mail,

Wuh?

220 WELCOME TO MY MAIL SERVER
EHLO mydomain.com
220 blah blah capabilities
MAIL FROM: somebody@mydomain.com
451 Sorry, I don't know you. Try again, though; I like persistance.
[socket closed]

Automate the /. effect!

[Anonymous Coward]

What about /.'ing the advertisers? You could setup your spamfilter to make a wget -r > /dev/null or the like on all links in incoming spammails. That way their servers will be overloaded, and they cannot get any real traffic/orders, meanwhile the load on you would be minimal. However the ISP's probaly don't like this idea as it will generate enormous amounts of traffic (initially anyway), and there will be lots of innocent bystanders.

Anyway, no one will probably ever read this as I am an Anonymous Coward.

No Mention of Spamhaus???

[OC_Wanderer]

I've been using the sbl-xbl.spamhaus.org combined lists for a few days, and copying the HELO/EHLO hosts to my incoming transaction filter. This method cuts down on spam actually received and saves me an incredible amount of bandwidth.

How to stop spam, TODAY

[Jesus IS the Devil]

*Sarcasm*

Make spamming punishable by death. :)

Is this really an expert view?

[Tamor]

When I took a look at the first of these two articles which examines end-user anti-spam solutions I had to wonder if the writer had actually tried any of the technology or was relying purely on hearsay. For example:

Spam senders and their bulk-mailing applications are not static -- they rapidly adapt around filters. For example, to counter word lists, spam senders randomize the spelling of words ("viagra", "V1agra", "\/iaagra"). Hash-busters (sequences of random characters that differ in each email) were created for bypassing hash filters. And the currently popular Bayesian filters are being bypassed by the inclusion of random words and sentences. Most spam filters are only effective for a few weeks at best

This is the view of someone who clearly has no experience at all with a high-quality Bayesian classifier like POPFile. I've been using this program for almost a year and it most certainly has not been defeated by random words or spelling. Many of the tokens that trip email as being spam are actually unusual items in the headers or sales terminology. After a very brief training period POPFile has continued to provide me with excellent protection from spam and malicious email, with only a few false negatives to retrain on.

If that's not a good end-user anti-spam solution then I don't know what is.

Re:Is this really an expert view?

[cpex]

I agree i got the same impression. I am currently writing a naive bayesian filter for an ai algorithms class. Pretty interesting project. Unfourtnatly we are not tying it into any e-mail client.

Re:Is this really an expert view?

[Tamor]

The more I thought about it the more I've come to the conclusion that the sole purpose of the first article was to declare end-user spam solutions dead to set up the need for the second article.

The author has a point when he says that end-user solutions don't stop the spam traversing the network and consuming bandwidth and resources. However, if significant numbers of internet users employed effective end-user anti-spam tools then it would eventually hit the spammers economic return.

Spam becoming a less lucrative quick-buck will probably be the thing that eventually kills it off. That's a long-term goal that can probably be as well achieved by educating the masses as to the wealth of excellent end-user tools available as it can by expensive and unwieldy protocol changes.

Re:Is this really an expert view?

[cavac]

The author has a point when he says that end-user solutions don't stop the spam traversing the network and consuming bandwidth and resources. However, if significant numbers of internet users employed effective end-user anti-spam tools then it would eventually hit the spammers economic return.

Many of the "big" spammers are known by name. Why not starting to charge them a monthly fee like $5/End-User for using the "extended mail services" aka sending commercial mail to the end-user? They'd start only spamming the dumb-ass idiots that buy products from their spam.

Re:Is this really an expert view?

[Henry Stern]

I have analytically and experimentally proven that over time, those random words will break your spam filter. I hope to publish a paper on the subject this summer at the First Conference on Email and Anti-Spam (CEAS). If you are interested, contact me by e-mail and I can send you a pre-print once the paper is finished and submitted.

Next gen SMTP

[bakreule]

Maybe I'm missing something, but the article never seems to mention anything about changing the SMTP spec itself. It talks about how it's flawed, but then summarizes new ideas to add on top of SMTP (crypto, C/R, etc). It doesn't ever suggest CHANGING the underlying protocol. History, with IPv6, is showing that the industry doesn't change without being prodded. I don't think this would be the case, however, with any new versions of SMTP.

From what I understand, rewritting SMTP to fix most (if not all) of the spam loopholes is no problem (Am I seriously glossing over some big details here?). The trouble is that people want a 100% effective, immediatly pluggable solution. If new email clients support both the old and new smtp protocols, and use the new one as a default, it will be just a matter of time before there's a critical mass of clients and ISPs that are using the new one.

Once this critical mass is reached, boom, everyone is required to use the new protocol, and any email that uses the old one is immediately dumped way upstream, before it can start hogging bandwidth everywhere.

I'm aware that if my idea is so great, how come it hasn't been implemented?? Feel free to pick holes....

Re:Next gen SMTP

[scrytch]

Maybe I'm missing something, but the article never seems to mention anything about changing the SMTP spec itself. It talks about how it's flawed, but then summarizes new ideas to add on top of SMTP (crypto, C/R, etc). It doesn't ever suggest CHANGING the underlying protocol.

I'm not blindly defending the status quo, but properly configured SMTP does NOT HAVE A PROBLEM with spam. Using the mail submission port ensures you can do all the relaying for mobile users in a proper secured manner (just make sure those passwords are good). The only weakness seems to be in having no standard for outbound MX records, but that's not a SMTP problem, that's DNS -- the only connection between the two is that you must look up MX records to deliver, not A records.

SMTP has survived over a decade of information warfare being waged over it. I'd say it's doing just fine thank you.

Rebuttal/Solution/Sales pitch-mod up if merited...

[iamcf13]

My credentials are years of programming experience and months of research invested in CF13(TM), my solution to English-language spam.

Part I of the article:

Identity theft - The ones I got were relayed through a 3rd party machine and deemed spam. Should I ever get a 'real one' that would mean either a spammer is using a stolen/throwaway account at a domain with mailserver(s) and easily traceable, or, worse yet, an 'inside job' by someone unscrupulous at the sender domain. The rule of thumb is to not give out sensitive information via email no matter how convincing said email is.

Viruses - All attachments are treated as 'text files' by my program and are 'harmless' provided a certain registry key affecting Notepad hasn't been changed/hijacked (see my website for more details). Also, all email is downloaded and treated as text files, making HTML related exploits impossible as well.

Sender - Anonymous senders are treated as spam. No exceptions. I've only gotten spam from such senders the rare times I recived them before I wrote my program.

Recipient - No 'BCC:' email if desired. In the past, such email I've recieved were spam.

Word lists - My program uses two of them. One of them is the single word list from Grady Ward's Moby project. The other file contains 'spam words' that appear in the first file. Both lists make 'hashbusting' and 'L33T' spelling, two tried and true spammer tactics, impossible.

Black/White lists - Supported at the email address and email domain level. I decided not to support IP level black/whitelisting since the IP source of spam is irrelevant--it is the content of spam that is relevant. Likely, such spam is deemed spam at the email header level anyway--or at the email message content level if need be.

Hash-tables - Pointlest due to 'hashbusting' and 'L33T' spelling. Unecessary in my program.

AI/Probabilistic systems - I researched the Bayesian approach and decided not to use it in my program. Though effective at first, spammers have thoroughly 'poisoning' this method of spam detection. Also, this method requires additional disk storage space, processor time (to do the math calculations on top of the pattern matching), and training time to be effective.

Bypassing filters - A default install of my program should catch almost all spam. Should any get through, one could read through the spam and identify new 'spam words' to be added to that list.

False-positives - Alas, to avoid deleteing such email at the server level, All such email is downloaded and processed. My program displays the subject lines of email messages it process and logs them to a separate file for further review if needed.

Spam filters do not stop spam - Agreed, but they can be as effective as my program which only has one known form of spam it cannot detect sent by a spammer from a stolen/throwaway account.

Reverse lookup - Not supported in my program to avoid slowing my program down and not overburdening the (likely) overtaxed DNS server system. This should be handled at the mailserver level to head off the sending of spam in the first place.

Part II of the article:

Challenge-Response - I considered using this but decided against it. In doing so I avoid 'mail loops' with another Challenge-Response system and outright rejection by email correspondents who hold a dim view of this antispam system.

Computational challenge: Another idea that fell by the wayside due primarily to the wide disparity in the CPU clock speeds of user's systems.

Cryptography - Not used by my program to process incoming email and thus unecessary. The 'bu

Except

[Kjella]

...a lot of the time people want to get in contact with you, that you don't immediately recognize. Maybe a contact address you've given out somewhere, on a business card, web board, im buddy, all sorts of places. It'd be no problem creating SPAM that seems to come from "normal" people, and many people would have no choice but to accept them all. And the gullible people would still allow the "From: FDG Lottery - you are a winner" spams through.

Kjella

From spoofing verification would knock out 98%

[arete]

From spoofing verification would knock out 98% of my spam. It would also stop one of the most annoying virus attacks - where windows user "Alice" has me and "Bob" in their addressbook and I get blamed when Alice's virus mails Bob with my address.

I propose
1) (seems to exist, as stated) extensions to DNS allowing you to specify domain outgoing servers for "from" addresses irrelevant of your MX records, and to specify whether you should accept from unlisted servers

2) (seems NOT to exist) a new lookup service probably running on your mail server (or a much bigger extension to DNS) allowing you to look up additional acceptable servers (ideally by IP OR name) on a _per username_ basis. Whether or not to check this, and where to check, should be an extension of the listing in 1)

Neither list requires you to have any control over the hosts listed (that's a requirement)

#2 has the disadvantage that lots of users need to configure it - but it's the users who are already configuring vanity domains and forwards that have to do it, not the recipients. For a vanity domain, you configure your vanity domain to include the SPF of your home ISP. For your forward situation, you add your remote address to that domains user-policy. (Of course, you should ALSO be able to whitelist it in, since the recipient is you.)

There's no reason you couldn't additionally have your local client automatically authenticate (using a decent authentication) and update this if you for some reason use a random local SMTP server (presumeably because outside SMTP is blocked on the network)

Here are the big advantages:
1: It would help. It would eliminate from spoofing from any partipating domain to any recipients using the filtering.

2: It wouldn't block anybody legitimate. At all. Or force any upgrades, at all.

3: Senders would want to upgrade so they don't get spoofed. Recipients would want to upgrade so they get less spam.

4: It would knock out the worst kind of spam - the joejob.

As a bonus, if an ISP gets a lot of checks against a given username, they could be suspicious of that username. They shouldn't automatically block them, obviously, since those queries could be made for just that reason. But even if they DID block that lookup for that user, it would only prevent them from sending mail through ANOTHER SMTP server using THAT domain from. So in the most common odd case of my using a vanity domain and sending email through my ISP, my vanity domain host would have to be blocking me, not my ISP. Since my vanity domain host expects me to have a vanity domain, presumeably they'll get this rather correct.

{PS: I'd like to add real legal $ penalties not only for sending spam, but for letting spammers abuse your network, especially after you've been notified. (While I'd like to include trojaned boxes too, what I really mean is ISPs) But that's not really part of the same proposal.}

sorry this was so long.

and the states?

[Anonymous Coward]

You forgot to include your own country, the largest source of SPAM, in your axis of evil.

This Got Modded Interesting?

[Silburn_Luke]

If the parent had RTFA, he/she would've seen that this option was discussed and discounted. The author's conclusion was that if you automate the whitelisting solution in the email infrastructure then spammers can figure out a way to game the automation.

Personally I feel that is overly pessimistic, but I don't design protocols for a living and I haven't really thought hard about the spamming problem at a strategic level so I'm not really qualified to comment.

Regards
Luke

SpammersFriend Technology presents...

[Anonymous Coward]

Even if the the software can detect the letters in a C/R system 1% of the time, it would rake in the cash.

Our patent-pending "It's an A!"(TM) software is almost 4 times more accurate than that. The licensing costs are very reasonable. Our dedicated research team is working on a whole suite of apps, the next ("It's a B!"(TM)) will be released next spring.

At present, though, 3.85% seems to be some sort of theoretical limit to the accuracy that can be obtained.

Re:SpammersFriend Technology presents...

[Anonymous Coward]

1. At present, though, 3.85% seems to be some sort of theoretical limit to the accuracy that can be obtained.

You are misinformed or delusional.

How do you think the numbers and words you scrawl on your checks are processed? Do you think this is new technology, or maybe it has been around for 10+ years?

Re:SpammersFriend Technology presents...

[aWalrus]

It was meant to be funny (and was, actually). The 3.85% is meant to be the frequency of the letter "A" in English, although I don't know where the poster got that info. The actual frequency seems to be closer to 8%

Re:SpammersFriend Technology presents...

[Anonymous Coward]

3.85% = 1 / 26

Re:SpammersFriend Technology presents...

[aWalrus]

HAH! you're right! Thanks.

Re:This is quite easily defeated using bounded set

[interiot]

Not only that, but the background is actually quite repetitive. The vertical lines are the same color all the way up and down, so you could find some bounds of the letters by where there's a different color. And the horizontal lines show through on every other pixel and are the same color all the way across... so there's at least some more information there too.

So, basically it's like everything else in security... if the scheme isn't designed by PhDs or government agencies with very solid security experience, or it isn't at least internally peer-reviewed thoroughly, it's very likely to have holes in it that will be exploited in short order.

Re:This is quite easily defeated using bounded set

[interiot]

Also, they appear to use different colors for the background vs. the text. If you convert the colors to hue/saturation/brightness, the brightness of the letters seem to be in the 40-60% range, while the background seems to be in the 79-96% range.

Think the other way round

[millarm]

How can we use spam to drive progress in AI research? I say let's require every email sender must pass a turing test before sending an email - thus we require spammers to develop software that is indistinguisable from humans in order to send spam. So spam only comes back when they have - and we simply then ask the new AI's for a solution. Evolution in action. ;-)

What about blind people?

[Anonymous Coward]

What about blind people? They need email too!

hashcash comments

[Adam Back]

I'm the inventor of hashcash. Here are some comments on the article's comments on hashcash, I think the author missed some aspects around how mailing lists work with hashcash, and the economic model. Most of this stuff is covered in the hashcash FAQ

* Mailing lists. [...] if there is a way for legitimate mailing lists to bypass the challenge, then spammers can equally bypass the challenge.

Hashcash is generated for the mailing-list address. The recipient would add the mailing-list to their list of addresses they accept mail as, and a spammer can not send to the list without including hashcash. So the limitation for mailing-lists is that the spammer can send mail to many people (the list subscribers) for the cost of one stamp; if he sends directly he has to send one stamp for each recipient.

* Robot armies [of 0wned machines].

Clearly someone wit lots of owned systems can send lots of mail; but still less mail than they could without hashcash.

* Legal robot armies. [...] Large spam groups can afford purchasing hundreds of systems for distributing an computational cost.

They can do this (and doesn't matter with it's legal or not btw, they'll do it anyway), but it will cost them more per mail which will cost them, so they will send less mails and be economically incentivized to target their mails by buying demographic data etc. (eg. so you would be less likely to receive spams in languages you can't read, or on topics you are not interested in).

Another aspect is that legitimate users do not send mails to lots of new recipients; most email exchanges are conversations over a period of time with sends and receives. Some of the hashcash based systems use hashcash only for introductions, and exempt recipients from hashcash after that based on crypto tokens (or just whitelists) (eg CAMRAM, TMDA do this).

The argument here is that hashcash can be set to higher cost as it is only borne once per new recipient for normal users.

Just how bad is it, really?

[mwood]

Because a 100% UCE-free Internet is going to be darned expensive and rather less usable. At what level of filtration does the next incremental improvement begin to cost more than simply being satisfied with what you've accomplished?

I've tuned up a pretty good stack of procmail recipes, set my MTA to refuse unverifiable senders and obvious forgeries, subscribed to a couple of decent blacklists, and trimmed things down to a level I find tolerable. And thus I'm disinclined to do much more.

Through a bit of mental jiu-jitsu I've come to regard the remaining trickle as a moderately challenging puzzle provided to me for free, and a source of amusement first thing in the morning as I make the initial pass through my inbox to weed out the junk unread. I spend a few moments each week enjoying the logs that Exim and my procmail recipes write to show me what they've strained out. Once you push the S/N ratio high enough to get some work done, it's possible to turn the rest of the N into fun if you have the right attitude.

Oh, there are other things I'd like to do. If most people would crypto-sign their mail, I'd set up recipes to toss unsigned messages, and play around with hacking signature and CA blacklists into my filters to get rid of the more brazen attempts. I'd like to try out some recognizers that would be mighty hard to write as regular expressions. I'd like to tinker with external filters that rip out some of the common obfuscation techniques before procmail even sees the message. But for now I can live without these.

If you're thinking, "but it's costing my company money to deliver this junk," ask yourself how much it's costing your company to have you sitting around trying to find ways to remove the last little morsel of UCE when you could be crafting new competitive advantages for the firm, or at least dealing with the *other* stuff that gets in people's way and which is not actively working against you.

Re:This is quite easily defeated using bounded set

[Anonymous Coward]

I have seen much better C/R image systems, e-gold.com although only black and white seems harder to crack via OCR, and one, I don't have the URL for it, but it was great - letters were sized, distorted, streched, oriented in different directions, random colored blobs were on the image, etc. These systems haven't begun to explore the realm of creatively damaged fonts that are available, like:
corrupt
or
punk-ass
there are a wide variety of creative ways to make fonts hard to read for OCR but still recognizable to humans.

Computational Challange should work

[guanxi]

In the article, Krawetz says that Computational Challange won't work. I disagree, and I think it's the most elegant solution. His reasons (paraphrased by me):

* Mailing lists and other legit mass mailers, such as Amazon, will be hit as hard spammers

But the recipient could whitelist these. If they don't, the sender would 'pay' for one message that says: Either whitelist us or use our website.

* Robot armies created by worms will be used by spammers to bypass the costs.

If Computational Challange is implemented, ISPs -- to protect their own budgets -- would immediately cut off any zombie computer. They should be cutting them off, now, anyway.

* Spammers can construct legal robot armies -- essentially buy lots of computing power

This argument misses the point of a small fee (paid in cpu cycles) for each e-mail. It's simple economics: You only have to make it expensive enough that spammers lose money on the deal.

If you can send 1 million emails for free, then you can afford sending ads with only a one in a million shot of selling something. But if you pay a penny per email, it's not worth it. No spammer is buying and operating a server farm to send 1 million e-mails, just to make $1,000 in sales.

* It's reverse taxation: To answer the same Computational Challange, a slower computers will require more CPU time than a faster computer. Thus the poor pay more than the rich

This is true, but it would still be a minimal tax, except for mass mailers.

Consider also,

* The recipient can set the bar as high as he wants. He could crank up it up to problems that consume 30 seconds or a minute, if he really hated unwanted mail.

* The recipient could set different levels for different classes of email, including a whitelist class for those who get through for free. Mailing lists might require subscribers to whitelist them.

* If you look at it one way, it's really the penny per email tax, but it's implemented more efficiently: No micropayment system, banks, accounts -- it just uses existing infrastructure. And nobody gets the penny.

what if

[WormholeFiend]

what if an added feature was added to email systems?

specifically, what if you removed the need for the @ in email addresses (while keeping systems backward compatible)?

wouldnt that complicate things for harvesting programs?

Re:Let's use the Patriot Act for the benefit of go

[dave420]

And when they decide that political emails from the Democrat party are spam, and start raiding people... Seriously, gone from no PATRIOT act to nearly PATRIOT 2 in 3 years. Going from PATRIOT 2 to PATRIOT 2.5 (with DemoClene(tm)) wouldn't take much longer...

Forget challenges that require human input

[eric76]

What I'd rather see is every e-mail transmitted be digitally signed.

When the e-mail client is set up, it could generate a GPG key set to use for signing the e-mail.

The recipient's computer, if verification is required, could send a standardized e-mail back to the sender's computer asking for the sender's public GPG key. If and when it arrives, check the digital signature and either deliver the e-mail or /dev/null it.

By caching the keys, you really wouldn't even have to have a white list. Or, more accurately, the white list would be by digital signature rather than the Reply-to or From address.

This could even be implemented on the server itself and with better results.

When adding the user, create a GPG key for that user on the server.

Require authorization for each incoming e-mail that is to be relayed. Digitally sign the e-mail with that key if it sender has not already done so on the client side.

The recipient's server or the recipient's client may then request the public key. If the public key used was the server's key used on behalf of the client, then return that. Otherwise, send the request on to the client for his public key.

Of course, this could be abused, but then the e-mail addresses have to be real and could then be used for blocking.

The traffic itself should be relatively small. The data portion of the request would just identify the public key desired based on what was used on the message (sender's key maintained by the server or the sender's key maintained by the client) and the data portion of the response would contain that id and the key.

For those who use multiple e-mail clients, allowing the server to handle the key would be preferable since the multiple clients would generally use different keys.

If the cached public key for that user failed, a request for the public key would be sent in case the public key had been changed. If the new key was different, the cached public key could be expired after a set period of time (in case there were any yet to be delivered e-mails from the old key around) and the new public key added to the cache.

You'd have the benefits of challenge-response systems without the users being annoyed.

One problem with challenge response systems is with mailing lists. With this method, there would be no problem since the mailing list's server would react to requests for the public key by providing it.

This would also take care of the automated e-mail case, say when you place an order and the sender sends an e-mail telling you the order has been fulfilled.

My solution !

[Anonymous Coward]

Put in jail every business that buys spam services, put in jail every final-end spammer.

Make both pay 1$ for each spam they sent, and make the money go to affected users.

Ok, that wont stop spamming, but i'd even prefer spam not to stop and to earn thousands of dollars (euros) for receiving it.

That was covered in part 1 of that article

[Anonymous Coward]

See part 1 and look for "Reverse lookup".

Sorry Won't Work

[Battle_Ratt]

Two words, Joe job.

Any one of these "solutions" can be exploited to hurt legitimate business. Simply send out a spam campaign on behalf of XYZ company with legitimate credentials, and watch the chaos and disaster at the company as phone lines are cut, merchant accounts cancelled, etc.

Spammers have already done all sorts of illegal activity to continue their frauds, what's one more to cut the knees out on the competition, or the competition of their customers.

Hardly an Expert

[Nuclear Elephant]

I would hardly consider this guy an expert on spam - in fact nobody I know in the spam filtering community even knows who this guy is. Ph.D from Texas A&M...wow...there's a technical college for you. Unfortunately he's very misled on the whole topic and doesn't seem to understand even the basic caveats of spam filtering. This is the kind of guy who likes to say "everything sucks" to prove his point.

Re:Hardly an Expert

[EpiCat]

Don't cry dspam dude, cuz you're shit ain't popular. When it gets popular, you'll see that your accuracy numbers will lower too, because you underestimate spammers. Thanks again, ego-boy.

Re:Hardly an Expert

[Anonymous Coward]

Agreed. Who was nuclearelephant till he got slashdotted a bit ago...?

Re:Hardly an Expert

[Anonymous Coward]

I'm not an Aggie, but I work with a bunch of them. Did you know that Texas A&M was one of the first universities that offered a course in Computer Security? Or that more A&M alumni attend BlackHat than any other school? I hung out with them last year -- a very impressive bunch.

I'd be careful who you offend. This doesn't make me think highly of whatever product you're pushing.

Re:Hardly an Expert

[Nuclear Elephant]

Actually DSPAM has become quite popular. It's being used on several large systems up to 150,000 mailboxes. Just because YOU haven't heard of it doesn't mean anything. The right people have heard of, are using, and are financially supporting it...and i still maintain my original claim that this "spam expert" is full of crap.

Re:Hardly an Expert

[EpiCat]

Oh I'm sure dspam was heard of, the point is you ain't making waves to spammers, so of course you can claim you can "stop spam" (even though filtering doesn't stop spam, it just categorizes it, aka it's not a solution, just a band-aid). My point is, if you were a threat to Spammers, they would get past your filter, you'd be another spam assassin, and yet another failing filter that doesn't stop spam as perfect as you would like it to. I've heard of dspam, I use it, but I use it, because it's not popular enough for the spammers to care.

Re:Hardly an Expert

[Nuclear Elephant]

Not likely. DSPAM is not like SpamAssassin. You should spend some time researching what makes probabilistic filters different from cheezy filters like SpamAssassin. There are plenty of tricks spammers are using today to try and get past these filters, but are not succeeding. In fact, a majority of the spams I see in quarantine these days are specifically directed at Bayesian filters.

Re:Hardly an Expert

[EpiCat]

Understood, they are targetting Bayesian filters as they know it. If spammers took the time to study dspam (right now it's not a major threat to them, as not enough people use it, so it's not hurting their bottom line), I believe they would and could get around it if it was worth it to them. What I'm getting at, is that you underestimate the persistance and intelligence of spammers, and that your product is only good right now. It is still a patch, it does NOT stop spam, just categorizes email, and it will be targetted if it were to become a threat, and your accuracy would go down because they would find ways around it. That is all. Case closed, you can't really argue with that until it happens, because you refuse to acknowledge it.

Re:Hardly an Expert

[Nuclear Elephant]

From a scientific perspective, anything that filters spam better than a human is successful at stopping spam. This includes nearly all of the popular Bayesian filters. Your argument is the same one they have about Linux - if it was more popular, it would have more exploits, but if you look at the architecture of the Linux operating system the way it has been constructed proves that this is not necessarily the case. You can speculate that if spammers focuses specifically on dspam, they could find a way to bypass it, but it's just speculation - but what I like _most_ about Bayesian spam filters is that there's not a monoculture set up like there is with heuristic filters. There are many different people using many different BCF filters, and there is no real "mainstream" filter. If the anti-spam community continues to work together as it has been, and can maintain this status quo, it won't be worth it for spammers to be able to poke a hole in a particular filter, as they still won't find a wide enough audience to push to. I don't think DSPAM or any other filter ought to be #1 for this reason...but if one should become the leading filter, it's my belief that (with the exception of some bug that could be patched for) the filter's adaptive qualities would make bypassing it computationally infeasable. And that's what we're ramping up to combat with things like Bayesian Noise Reduction and Message Inoculation.

Re:Hardly an Expert

[EpiCat]

You conclude with the statement that bayesian filters are wonderful. You missed the main points. 1. Filters "filter" -- they do not stop spam. (Network volume, resources, etc.) 2. If you know how it works, then you can defeat it. 3. And if it becomes a popular solution and heavily used, then spammers WILL put efforts into defeating it. You're proud of how rarely it needs to be updated. But if someone is consciously going around it, then it better update more often. And since it isn't "100%" perfect, people CAN go around it.

Try reading what I wrote.

[Ungrounded Lightning]

For instance: Subscribing the Detroit area spammer and his lawyer to enough real-world junkmail lists to bury his bills and other US Main correspondence in several daily truckloads of catalogues and other solicitations

With all due respect, get a clue.

You don't fight a noisy neighbor by cranking up your stereo.

With all due respect, try actually reading what I wrote.

I didn't say it was RIGHT to do this. I PREDICTED that it WOULD HAPPEN. Very different thing.

Re:Try reading what I wrote.

[mabu]

Yea, thank you Captain Obvious.

I'm aware that if you don't like someone, splitting their head open with a pickaxe is an option, but talking about it seems pedantic and a waste of energy that could be better spent on suggesting more enlightened ideas.

Microsoft doesn't use Microsoft email caller ID!

[nexus987]

It looks to me like microsoft isn't even using their own caller-id standard? IE: no TXT records published for microsoft.com? Lame. It took me all of about a minute to publish dns spf records for my domains.

Diff Filter w/ Rep Mgt of Auth Sending Service

[jfaughnan]

Differential filtering based on reputation management of authenticated sending services.

You don't need to authenticate the email author (too much overhead, too disruptive). Only the sending service.

Once the sending service is authenticated, you apply differential filtering based on the reputation of the sending service.

Optional: allow authenticated senders to bypass filtering.

Put this in place, then let natural selection work its magic.

I've been writing about this for about 3 years, but I'm sure it's been discussed for many years. I never see it presented though.

It's not that hard.

http://www.faughnan.com/spam.html#CheapFix has roughly what I wrote here, but not much more.

OK, you beat this one...(economics)

[nexus987]

OK, you beat this one. I'm using a different system. And my friends are using yet another C/R system. And my system poses questions like: what's thr33 plu$ 2? Oh yeah, and you have to pay somebody to write the program to crack these C/R's, you have to have a valid domain name, working system to receive my challenge mails, cpu power to do the OCR, possibly pay for the bandwidth to download a boatload of graphics. Pretty soon it becomes un-economical to spam.

Here's a comforting thought...

[Skim123]

19% of opt-in email is blocked by spam filters.

Back in the .com craze there were a number of companies that made money solely by running large, popular, opt-in email lists. While the dot com crash surely killed off the poorly managed ones, the ones that were run modestly by a few folks from a small office in some cheap location still hung on... although I would wager such companies are hardly around anymore with the way spammers have effectively killed off opt-in email....... drag

what the fuck are you smoking?

[Anonymous Coward]

just pile on the BS. Live in your own alternate reality don't belive the "blog theory". Whatever.

what are you smoking

[Anonymous Coward]

fine alternate reality.

Re:Let's use the Patriot Act for the benefit of go

[Grrr]

Slippery slope, there.

<grrr>

So...

[KalvinB]

you append or prepend a random sequence of letters to the IP address and pass that sequence to the client in the form. The client then passes it back to the server so it can open up the file by recreating the file name from it and the IP.

"but inherently prohibitively flawed for any global purpose..."

Only if you're not a problem solver. The IP basically amounts to a private key since it's grabbed from the request header and the sequence is essentially the public key.

And frankly no, it's not a big deal. If you were planning on marketing something like this to a massive number of users then you would need to spend the five minutes needed to rectify the situation.

In my case, I might get around to changing how the file is named to account for same IPs overlapping.

Ben

Not exactly...

[Ayanami Rei]

Yahoo uses one system. Hotmail uses one system. Juno uses one system.

But just breaking _one_ of those C/R systems is useful enough to a spammer for a long while. Once the system is broken, you can get unlimited email addresses from a domain.

You only need to crack the next one once someone blocks every user on a site... (which is draconian, so it's not as commonplace as one would expect).

No, C/R just means you have fewer, more sophisticated spammers. It gets rid of the small fries who can't compete in the "market".

Solution: Stop Spam at the Source

[capojim1]

Here's "THE" solution for spamming:

This requires a new feature to be added to mail servers and clients to implement this functionality, but it should be relatively straightforward and is 100% backwards compatible with non-conforming servers and clients.

Basically how it should work is if johnny@aol.com sends me a message at andy@att.com, the mail server at aol.com (the sending server) will store a list of recently sent emails.

All it stores is the sender email address (johnny@aol.com) and a unique id for the email, maybe a CRC number (see explanation at the very end) derived from the message contents and all attachments.

When the receiving mail server (that's Andy's server at ATT) gets the message, it contacts the server at aol.com (derived from the 'from' field) and queries to see if a message from such a person was actually sent.

It sends the email address (johnny@aol.com) together with its own generated CRC number.

The sending server (which was aol.com) now checks its list of recently sent email and either returns a yes or no based on the test to see if the address/CRC pair is on the list.

Once the user (Andy) downloads the message and removes it from the server the receiving server (Andy's at ATT) sends a message to the originating server (Johnny's AOL) that it's ok to remove the message record from the recently sent email list.

This method makes it impossible to spoof the "from" field--- (I am sure all you who read this are more than familiar with the spoofing done by spammers).

If spammers can't spoof the "from" field they lose their anonymous/fake cover.

It's possible to trace them back to the originating ISP and that ISP will have records of whom that account belongs to or will simply shut down the account if it's a free mail service.

Basically spam can be traced back to its source (and maybe even viruses).

Of course, not all servers will implement such functionality right away.

The end user can set up their mail client to simply filter email from servers that don't support this feature into a special folder that will contain "unverified" email, but this folder will get less and less email as this feature gets implemented more and more.

If the server does support this feature, and the sender is not verified, you KNOW its spam.

If AOL, Hotmail, Yahoo implemented this feature, and you have a client that supports this feature, you KNOW you won't get spam from any of those servers anymore.

------------
CRC

Short for cyclic redundancy check, a common technique for detecting data transmission errors.

Transmitted messages are divided into predetermined lengths that are divided by a fixed divisor.

According to the calculation, the remainder number is appended onto and sent with the message.

When the message is received, the computer recalculates the remainder and compares it to the transmitted remainder. If the numbers do not match, an error is detected.

Not realistic at all

[fingerfucker]

What are web forms for?? You set up a web-based form on the website and upon submissions, the server-side script sends you an email. Think before you write.

Re:Not realistic at all

[Skim123]

What are web forms for?? You set up a web-based form on the website and upon submissions, the server-side script sends you an email.

I addressed two issues: feedback on a Web site and USENET posts. On a newsgroup post, my email is out there, there's no "form" for sending feedback. Also, maybe I don't run the site, maybe I just wrote an article for the site and the site place's an email address as contact information. Or maybe some spammers are smart enough to create a script that will send email via a feedback form. I know I am hit with automated spam messages on my blog system's comments.

Think before you write.

You should follow your own advice.

The Author

[minas-beede]

Kraewetz also wrote an article on anti-honeypot technology for IEEE Security & Privacy (pp 76-79, January-February 2004 issue.) He seems to asume (like too many) that the spamemrs are near God-like in their abilities. He also seems not to have ever run an anti-spam honeypot. If he had he might have, many times, seen spammers behaving in an incredibly stupid manner. The spammers aren't all that sharp and honeypots (at least now) don't have to be very sophisticated.

His survey article (the topic of this Slashdot thread) once again continues the misconception that anti-spam tools all must be at or after the destination server. Let's look at it this way: the spammers quickly figured out (4-5 years ago) that going direct from their server to the destination wasn't working for them, so they went to intermediate severs, starting with open relays. Anti-spammers are still stuck on their servers - if it isn't the final server it doesn't exist (to their closed minds.) The open relay layer is just as available to the anti-spammers as it is to the spammers, but the anti-spammers mostly refuse to go there. Actually the layer is more available to the anti-spammers: any system with no real email function could serve as an open relay honeypot. You don't need to filter or be smart at all with an open relay honeypot: the spammers do all the work for you. They find the honeypot, they think it's an open relay (the honeypot dutifully does deliver the spammers' test messages), they send the spam, you just sit and watch. For open proxies (another layer dominated by spammers) that's even more true: not that many systems need to run any real proxy software at all. If you don't like spammers (pretty likely) think how gleeful you'd be to see a spammer falsely assuming your trap will deliver his spam for him - and then think of the glee as you trap more and more of it. Think of finding the spammer's IP address (if you run a proxypot) and reporting it to the spammer's ISP. Many ISPs actually are ethical enough to rid themselves of a spamming customer, once they learn of him. Honeypots still work: the spammers don't all use spam-server zombies yet.

Retain your blocklists and filters in the meantime: until spam stops flowing you'll need them. If you've got an available IP and an available box you could probably be causing some spammer grief as early as tomorrow - if you'd run a honeypot.