Slashdot Mirror
Spam Solutions from an Expert
Mod N writes "SecurityFocus has posted a nice survey of anti-spam technologies by spam expert Neal Krawetz, in which he delves deeply into the specifics and pitfalls of the numerous proposed solutions. Krawetz makes it obvious that securing the email infrastructure is a very complex problem that many of the current (simple) solutions can't solve alone."
There is no anti spam technology that actually works. Not even whitelisting, because those viruses fake email addresses.
Maybe whitelisting with custom mail headers to prove identity
Open Source Java Web Forum with LDAP authentication
If a human can interpret an image and type in some dumb pieces of text, there's little reason to believe a computer program cannot do the same.
Why has spam grown to what it is today? It is an undeniably effective means of cheap marketing. What we need to do is come up with a way to stop this not on our end, but by looking at as a social problem or making it non-worthwhile to the spammers. If nobody ever responded to spam, spammer wouldn't bother.
At this point in the game, I am honestly surprised that we haven't heard of violence resulting from spam affliction.
I don't know about anyone else, but I'm pretty sure I'm not alone in this. I have, at times, felt utterly enraged at all the spam flying about and further all of the innocent and naive people that are being abused by all of this.
I know if I feel violent internally, then surely there are those with less self-control out there who will eventually act on his or her rage... perhaps the parent of a child afflicted with porn spam?
I think if two or three spammers are attacked physically, it might give them pause. Frankly, I'm amazed it hasn't happened.
That's like saying a all theoretical attacks is not worth securing against somebody's fallen victim to it. Sure, there's some way-out ideas that can be dismissed that way, but this one seems so simple I'm pretty sure somebody who runs both spam and a porn site could pull it off...
I think the author of the article is correct. Having a system whereby anybody can communicate at virtually zero cost without unsolicited commercial messages are mutually exclusive goals. I think that for most people, a simple whitelist is good enough, along with the understanding that there is a small chance that email between new contacts will be blown away.
In Soviet America the banks rob you!
SPAM is like popups. The one day you find a solution to stop it, the next day they find a new solution to send it. It's a never ending cycle get used to it.
The Chinese government will probably solve any internal spam problem pretty quickly.
I mean, if you start by shooting all convicted spammers, the profession tends to stop attracting replacement members.
The party of stupid and the party of evil get together and do something both stupid and evil, then call it bipartisan.
Is this a joke? He just asked for proof and you got modded up by offering none?
ender-iii
I cant even get my scanner to correctly identify a regular text document, it gets most of it, but it still misses a lot of letters. A computer program could do this, but you would need either a very large database of the letter pictures (most places use all different kinds of text pictures, and add in a degree of randomness). Or you would need a very developed algorithm to detect the letters (in which case you would be making oodles of money from the scanner industry. . . spam would be the least of your worries.
In the end i think it is inevitable that software will eventually break this system, but as soon as it does, there will be another system in place. . . .
If you are about to mod me down, keep in mind that this post was most likely sarcastic.
If a human can play Go there's little reason to believe a computer program cannot do the same.
He's not angry about having to hit delete.
He's angry that his 7 year old daughter got a spam about things 7 year olds don't generally talk about; He's angry that his grandfather has been doing business with some guy in Nigeria.
This might be a joke now, but it may well happen in the future if we're really into this C/R thing.
At the moment spammers are already paying people to send emails from home, obviously it is profitable enough to pay someone to do the dirty job for you.
As a result, if recepients are less defensive against spams in a C/R system, those slipped spams might get a greater response rate. And this is good news to spammers, and they might very well be able to afford to outsource to deal with C/R.
Rock that crushes, Paper & Scissors that don't matter.
1) Tap the Slashdot and creative communities to produce a series of anti-spam TV/radio/print ads on the theme of "Spammers are Scammers." Smear all spammers as scam artists who sell fake merchandise and steal credit cards, and their customers as stupid losers.
2) Get media outlets to run them for free as public service ads.
Yes, I know this isn't a 100% solution. However, it is relatively low cost, and requires no new laws, software upgrades, or Internet standards.
Q: What does the "B." in Benoit B. Mandelbrot stand for? A: Benoit B. Mandelbrot
What I take issue with is this paragraph from the article:
This is leaving out a key feature of any decent challenge system... When Bill tries to send an email to Charlie in the first place, Charlie's email address is automatically added to Bill's whitelist. So Charlie's challenge, showing his address as its source, flies straight to Bill's Inbox without a hitch. If Bill were so arrogant as to think he could send email to someone not on his whitelist, then he deserves not to have his email go through.[100% ISO 646 Compliant]
SVM, ERGO MONSTRO.
Someone, either me or the author of the article is on crack. I was under the impression that one does not have to have private key in order to validate the signature.
Lets assume that there are CRT records that store SSL certificate for clients allowed to send mail on the behalf of the domain.
Now somebody tell me, in which step one needs private key to verify certs?
Robert
Bastard Operator From 193.219.28.162
Make no mistake...
The most effective spam solution at this time is RBL blacklisting. Bottom line.
When you take into account that the biggest problem of spamming is bandwidth consumption and network resources, there is NO better way than blacklisting spam sources and refusing to communicate with them.
Services like Spamcop's RBL really piss off the spammers. All client-side filtering is counterproductive and ultimately useless as you constantly have to update the systems to catch new efforts on the part of spammers to thwart the filters. At least with RBLs, the spammers' connections are immediately refused as soon as they're ID'd.
If you want to identify what is the most effective solutions, it's simple. Look at what pisses off the sleazebag spam community the most. That's relay blacklisting. They don't DDOS the moronic client-side filtering companies because the spammers know they're useless, and even if they're not, the spammers can't tell. What hurts them are when systems say, 'screw you spammer, (click)' and that's done via relay blacklisting.
Why are spammers increasingly changing mail relays and pursuing open proxies? Because of RBLs. Even AOL uses RBLs (including Spamcop). All the major ISPs look at the RBLs because they are THE most effective way of stopping spam. And they're the only way to actually shut down the spammers.
Forget client or server-side content-based filtering. They will NEVER work. RBLs are responsible for forcing spammers into corners of IP space, forcing them to deploy worms and viruses to infiltrate new IP space (which exposes them to more prosecution). RBLs ** WORK ** !
Maybe you develop some whiz-bang image recog program that can take amazingly distorted text and figure it out. If it takes 5 minutes to process a box, it does you no good anyways, too much time to be worth it for this use.
Not really. Since spammers are now into the illegal business of commandeering people's computers using viruses and trojans, it would be an easy step to have them process distorted images and feed the results back to some web site.
It wouldn't even take that many computers to send a lot of spam out even at 5 minutes per. Say you want to send 1 million emails. 1,000,000 / 5 minutes = 138 days. If you have 138 computers, you can send out 1 million spams per day.
If a log of the failed challenge attempts is kept, the source of repeated failed challenges can be ruled out from getting any more challege attempts, or even just one failed challenge with hundreds of successful ones coming from the same IP space... then the hacker source cna be flagged and ruled out.
Unfortunately, this is one area in which the spam gangs already have a leg up on the rest of us. Trojaned machines provide them with a distributed set of machines (and hence, distributed set of IPs) from which to launch their attacks. While you may be able to block some zombies machines, there are many more from which the spammers can continue launching attacks, many of which overlap with IP space of actual (non-spam) users.
Unless you're being extremely unforgiving (in which case, you WILL get false positives), all the spammers will have to do is continue rotating machines to prevent exposing an IP long enough to get it blacklisted.
Consider that both the sender and the recipient have a C-R filter. How will either one get the challenge? Wouldn't it just end up in an infinite loop of challenge e-mails? Or is there something I'm missing?
Crushing dreams at the speed of sarcasm
Challenge / response systems are broken anyway, even if spammers can't break it.
Why? Because from: is forgeable, and viruses use other people's real addresses constantly. Every day, one of my 40 spam emails is a C/R email from someone that I've never heard of. Am I going to click the link and authorize my email address? Fuck no. But I'll never be able to send email to that person. I realize that's a *tiny* incidental, but it's still broken by design.
If your C/R system includes a solicitation to purchase said C/R system, you're a fucking spammer. Fuck you.
There are no trails. There are no trees out here.
There are a few problems with your comparison:
* It's a lot easier to jack into the Internet than it is to get a phone line
* It's more expensive to perform telemarketing than cybermarketing; you have to pay people and you're not nearly as anonymous - there are costs in launching telemarketing efforts, whereas with spamming, all you have to do now is jack into a network or open proxy and unload your spam.
A spam do-not-e-mail list won't work, because at the present time, the spammers can hide much more effectively on the Internet than they can using POTS.
Not to mention that you don't see telemarketers engaging in the fraudulent practices that spammers employ, so that should tell you something.
We're not talking about astronomically difficult calculations beyond the grasp of any mathematical deduction... and we can infer this with relative confidence, given that your brain is doing these calculations even now, as you read this text.
I'll admit I was simply being humorous in implying that I, myself, if not burdened by exams, could slap something together. It would require a great deal of work to get this operate properly... but it is by no means an unreasonable goal. Science has tackled far more difficult problems than this.
From spoofing verification won't make a difference... it'll slow down mail services and won't make a dent in spam.
Spammers are now rotating IP space all over the place... they're also beginning to NOT forge header information, so what are you left with?
Recognizing rogue relays and blacklisting them, even if they have valid header information. Any improvement to SMTP protocol won't make a bit of difference.
Most mail servers and large ISPs are already employing additional methods of header-verification. It hasn't stopped spam.
RBLs ARE working. They're making spammers scramble for un-blacklisted IP space. That's why they're running overseas; that's why they're sending out worms and viruses. Lord help us if IPv6 gets introduced... we'll never be able to stop spam then.
Good for you. I feel sorry for all your other neighbors who suffered because of your little "arms race."
I'd give even odds that if you try the "get back and them with the same strategy" you can just as easily end up on the receiving end of punishment by the authorities as them, probably sooner.
It's not just a matter of taking the time to pop out the code, either. Non-industrial grade commercial OCR software right now pretty well sucks. It can look at images rendered in black and white above 300dpi and give you back about 90% readable text, provided you don't care about formatting and there are no other foreign entities in the scan field. And it's not like these systems are weekend warrior projects. It's an active field of research.
It takes a while for a computer to recognize visual patterns. OCR uses shortcuts. It makes assumptions about fonts used, about letter positions, about possible words, basically, it makes the assumption that the text is a real message laid out in a way where it wants to be read. So when the image is being purposefully formed to trick the system...
Example: the link in the parent. You can't hone in on the shapes of the letters by finding boundaries between colors, because the background has all the colors of the letters, and all the letters are different colors, so you have to spend time branching off course, realizing you made a mistake, and backtracking... you can't predict the positions of the letters because they're all staggered randomly, so you have to spend time parsing the image to find concentrations of color... you can't use dictionaries to make predictions, because they aren't words, they're just random sequences.
It definitely possible to eventually get a proper translation of the image, but after how long? How does that help spammers, which is the point?
The porn site workaround is genius, though...
Comment removed based on user account deletion
Those certs will simply not be trusted for purposes of accepting email. Thawte has a very thorough process for getting a cert with your name in it. Even their "Freemail" certs require some level of data input, but it's not verified. It takes enough time to keep it from being a viable option for spammers though.
Requiring certs would spell the end of anonymous mail, but spam has already done that, and the Beagle virus has shown another reason why everyone (ISPs in particular in this case) should digitally sign their email.
most C/R engines use a constant suite of pictures and words because the pictures are too time consuming to create on the fly... so the signup page might take too long to load.
:-)
What the spammers do is just download as many challenges as possible, solve them, and store the hashes in a database.
When the harvester goes out, it is likely to encounter many of the challenges a second time, and it already has the answer.
If it doesn't know it, it flags the spammer, who identifies it offline, adding it back in, and the database is that much more useful.
THIS THING CAN TURN ON A DIME, MACROSSZERO STYLE ALSO FUCK BETA, ~NYORON
I think we are attacking Spam from the wrong direction. Attempting to stem the flood of incoming spam is tough - everything about the identity of the incoming spam can be faked. However, we could alternatively attempt to prevent the replies going back the other way.
There are two inevitable facts:
1) In order for spamming to be worth someone's effort, they have to somehow get money from people. If NOBODY replied to them, then spamming would stop overnight.
2) Something in the content of the Spam must be real - a reply address - a web site, a phone number or something. Block traffic to that location and the spammer gets no money and dies.
Hence, I think they may be vulnerable. Educating people not to reply to SPAM would help - it only takes a mere handful of people to respond to a SPAM to make it profitable - but if education could drop that handful to a mere one or two - then we could succeed in putting more spammers out of business simply by cutting their margins to the point where it wasn't worth the hassle.
Where are the TV adverts: "Replying to Spam is Bad!"....we know that the morons who reply to spam are suckers for advertising - they are as likely to believe a well targetted TV advert as a crappy email shot. If Spam is costing the ISP's as much as they say it does - then funding some TV ads might not be impossible.
What if we made it illegal to respond to an emailed advertisement that was not clearly labelled as such, that would help to deter people from responding. Such a law would be next to impossible to enforce - but we are trying to deter the gullible here - so it might not have to be enforcable - just very well advertised.
Since every SPAM has to either advertise a product that you can buy from somewhere - or direct you to a postal address, a phone number or a web site - then that route for getting money back to the spammer could be blocked.
The return route has to be genuine. There is no point in them sending you a fake phone number or faked web address. If the phone companies (who are often also ISP's - or have at least some cause to want to kill spam) were to block calls to and from phone numbers that were seen in Spam - then the reverse route for the money would be curtailed. Whilst you can afford to change the aparrent source of your spam and fake those addresses for each new mail shot, you can't change your phone number for every couple of dozen orders you take. Similar considerations apply to web sites and postal addresses.
If it was required for credit card companies not to transfer money to businesses that employed spammers to push their goods - then that would also help some.
It wouldn't take many people to deliberately reply to spammers - to lead them on into thinking you want their product - to send them fake cheques or bogus credit card numbers. If they only get a handful of positive responses per million spams - then it wouldn't take more than a few determined people per million (eg ISP employees) to clutter up the the spammer's cash collection mechanism to the point where it's too much hassle for him to sort out the real orders from the bogus ones.
I don't pretend to have all of the answers - but there seems to be far too little creative thinking along these lines.
www.sjbaker.org
Naturally we may be inclined to believe that this grants us superiority to the computer. That, while stating some arbitrary facts taken from some textbook somewhere, a computer can never accomplish X objective.
Therein lies the fallacy. The computer does not identify that it is in an infinite loop, nor can it, because it is not given the benefit of looking at the actual code. If a compiler were designed to read into code for things like while(true) loops, which naturally could result in infinite loops, then already you would be cutting back on the instances of these problems.
Determining if there is an infinite loop requires a conscious understanding of the code itself, which is no trivial matter. It is not, however, something that could be deemed impossible.
As with all fields of science, there will be those who say "Well, I haven't seen it yet, so it will never happen"... but skeptics are everywhere, and the presence of skepticism is hardly a measure of credibility... rather, a measure of how pious certain peoples assumptions are.
Solutions are always found in math, and never in magic. Don't underestimate the computer, and more importantly, don't underestimate your own brain. You don't perceive things the way you do 'just because'... and that's what's so exciting.
TCP is NOT flawed. Sure you can spoof a packet or two, but (assuming reasonably strong sequence numbers) you can't fake a whole connection unless you are actually getting the reply packets.
mail is likewise not flawed; It is fairly hard to find an open relay these days; it is all-but-impossible to find one that doesn't put your IP address in the headers. That's your _REAL_ ip address. The one that ends up in RBL's so nobody accepts your mail any more.
The big flaw is home users; they keep getting pwn3d. And you can't even blame Microsoft for this any more. The viruses are arriving as a zipped, passworded attachment FFS. We've long since passed the realm of just clicking on an executable!
Here's how I see it; the antispam community were on the right track from the beginning. Blacklisting has made it impossible for spammers to spam from their OWN connections, even overseas, and pushed them to finding home users (to spam from, or to attack the blacklist sites). Now they're talking about changing the entire mail system, persuade thousands of users to change the way they do email? Hell no, we've almost won. We just need to educate enough END USERS not to get pwn3d, with the result that the DDoS attacks get cut down and the remaining much smaller number of spam sources can be more efficiently blacklisted.
Or we can force one more 'wafer thin' kludge onto the entire mail system, which the spammers will just find a way around next week anyhow.
455fe10422ca29c4933f95052b792ab2
When I took a look at the first of these two articles which examines end-user anti-spam solutions I had to wonder if the writer had actually tried any of the technology or was relying purely on hearsay. For example:
Spam senders and their bulk-mailing applications are not static -- they rapidly adapt around filters. For example, to counter word lists, spam senders randomize the spelling of words ("viagra", "V1agra", "\/iaagra"). Hash-busters (sequences of random characters that differ in each email) were created for bypassing hash filters. And the currently popular Bayesian filters are being bypassed by the inclusion of random words and sentences. Most spam filters are only effective for a few weeks at best
This is the view of someone who clearly has no experience at all with a high-quality Bayesian classifier like POPFile. I've been using this program for almost a year and it most certainly has not been defeated by random words or spelling. Many of the tokens that trip email as being spam are actually unusual items in the headers or sales terminology. After a very brief training period POPFile has continued to provide me with excellent protection from spam and malicious email, with only a few false negatives to retrain on.
If that's not a good end-user anti-spam solution then I don't know what is.
Because a 100% UCE-free Internet is going to be darned expensive and rather less usable. At what level of filtration does the next incremental improvement begin to cost more than simply being satisfied with what you've accomplished?
I've tuned up a pretty good stack of procmail recipes, set my MTA to refuse unverifiable senders and obvious forgeries, subscribed to a couple of decent blacklists, and trimmed things down to a level I find tolerable. And thus I'm disinclined to do much more.
Through a bit of mental jiu-jitsu I've come to regard the remaining trickle as a moderately challenging puzzle provided to me for free, and a source of amusement first thing in the morning as I make the initial pass through my inbox to weed out the junk unread. I spend a few moments each week enjoying the logs that Exim and my procmail recipes write to show me what they've strained out. Once you push the S/N ratio high enough to get some work done, it's possible to turn the rest of the N into fun if you have the right attitude.
Oh, there are other things I'd like to do. If most people would crypto-sign their mail, I'd set up recipes to toss unsigned messages, and play around with hacking signature and CA blacklists into my filters to get rid of the more brazen attempts. I'd like to try out some recognizers that would be mighty hard to write as regular expressions. I'd like to tinker with external filters that rip out some of the common obfuscation techniques before procmail even sees the message. But for now I can live without these.
If you're thinking, "but it's costing my company money to deliver this junk," ask yourself how much it's costing your company to have you sitting around trying to find ways to remove the last little morsel of UCE when you could be crafting new competitive advantages for the firm, or at least dealing with the *other* stuff that gets in people's way and which is not actively working against you.
Basically, to get the spammer to stop spamming, stop people buying their product. It's legal, ethical and will stop spam in seconds. Instigate laws that outlaws spam as a method of selling products. Any company found trading via spam can be brought before a court. The beauty with that system is the company has to be reachable via the email somehow (otherwise they wouldn't sell anything, so the spammer wouldn't spam for them), whereas the spammer remains hidden. That lack of anonymity the company posesses means you can find the perpetrator, and press charges. Most likely, the company will release the information about the spammer (including financial information, which can be used to persue the actual spammer).
To reach the spammer you have to go through the only route possible - the vendor.
Unfortunately, with spam, sending a mail to anywhere in the world is free, and very easy to obscure the true origin. As no-one's paying per-email fees for passing the spam along, no-one's that interested if it's spam or not. There's certainly no vested financial interest in stopping it. Just ignoring it is cheaper than actively trying to cut it out.
The real problem with spam is the relative cheapness and anonymity behind it. The only things that stop people spamming via phone/fax/SMS/etc is the fact that the spammer is easily traced. As we all know, with email it's not that simple.
Two words, Joe job.
Any one of these "solutions" can be exploited to hurt legitimate business. Simply send out a spam campaign on behalf of XYZ company with legitimate credentials, and watch the chaos and disaster at the company as phone lines are cut, merchant accounts cancelled, etc.
Spammers have already done all sorts of illegal activity to continue their frauds, what's one more to cut the knees out on the competition, or the competition of their customers.