Stopping Spam Before It Hits the Mail Server

Al writes "A team of researchers at the Georgia Institute for Technology say they have developed a way to catch spam before it even arrives on the mail server. Instead of bothering to analyze the contents of a spam message, their software, called SNARE (Spatio-temporal Network-level Automatic Reputation Engine), examines key aspects of individual packets of data to determine whether it might be spam. The team, led by assistant professor Nick Feamster, analyzed 2.5 million emails collected by McAfee in order to determine the key packet characteristics of spam. These include the geodesic proximity of end mail servers and the number of ports open on the sending machine. The approach catches spam 70 percent of the time, with a 0.3 false positive rate. Of course, revealing these characteristics could also allow spammers to fake their packets to avoid filtering."

Comments

It'll work..except when it doesn't.

[MrCrassic]

I'll go first.

All spammers have to do is change the characteristics of the message. It's always going to be a cat and mouse game, just like antivirus and antispyware, so saying that they've found THE solution to blocking spam from hitting the server is slightly irresponsible.

Re:It'll work..except when it doesn't.

[ByOhTek]

Unless they use a truly novel approach of stopping spam before it hits the server.

I suggest an AK-47.

Re:It'll work..except when it doesn't.

[gnick]

I realize that you're kidding, but removing more than expected is something that I consider unacceptable. If it hits the mail server and gets shuffled off into a spam folder with 100 pieces of trash, that's fine. But if it's not even going to make it to the mail server, 0.3% is too high a false positive rate.

RFC 3514

[Anonymous Coward]

Problem already solved back in 2003, I don't get any spam now.

Re:RFC 3514

[darpo]

For those who don't feel inclined to Google for it:

"The evil bit is a fictional IPv4 packet header field proposed in RFC 3514, a humorous April Fools' Day RFC from 2003 authored by Steve Bellovin. The RFC recommended that the last remaining unused bit in the IPv4 packet header be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem."

Spammers evolve from experience

[pearl298]

Just like other criminals, spammers must quickly respond to what actually works. In essence this is the flaw in any "security by obscurity" scheme, the bad guys simply respond to whatever works. If you get to try several billion times a day then you can try a whole lot of combinations.

.3% false positive is pretty high

[Dynedain]

That means that in my office of 50 people, with an average of 50 emails per day (a very very low estimate), we'd get 7-8 false positives daily. I'd hear bloody murder if that was the case.

We get a lot more mail than that per day, and our spamassassin without autolearning (simply flag anything higher than 5.0) does a hell of a lot better job than that... down in the range of 1-2 false positives a month. Assuming a low daily average of emails (like my example), that's .002% false positives.

Re:.3% false positive is pretty high

[Ioldanach]

From the article, "The end result was a system capable of detecting spam 70 percent of the time, with a 0.3 percent false positive rate." The summary dropped an instance of the word "percent". I wasn't sure how to read it either so I specifically looked for the source of the 0.3 in the original.

Re:I don't get it...

[BlueKitties]

Many spam messages are propagated by botnets, spoofed IPs, etc, so that isn't a perfect solution. Really, we need to combine different approaches, instead of trying to find a holy-grail.

False positive rate?

[johndiii]

0.3 would be terrible - three out of ten false positives. 0.3 percent - what the article actually says - is not too bad. But current techniques allow me to check the spam bin for such messages. This technique would pretty much preclude that capability, since the mail would never arrive at the server. I'm not sure that a rate of 0.003 would be acceptable under those circumstances.

Re:False positive rate?

[raju1kabir]

Help me here... Personally I would think that if 10 is 100% 0.3 is less than 1 mail. And not 3 out of 10.

.3 is 300 out of 1000.

.3% is 3 out of 1000.

It's similar to the confusion created when idiots write "It only costs me .25 cents to make a phone call" when they really mean ".25" or "25 cents".

Re:False positive rate?

[vux984]

And when my mail filters blocks spam, it sends out a message with redirections to an alternative gsm-number telling them to call me so I can whitelist the adres.

That's called back scatter and its as bad as spam.

Think about it, my mail servers block about 35,000 spam per day. If they sent a message to each failed recipient with alternative instructions, that would be 35,000 messages I sent out. Some 34,990 of those messages would either be undeliverable or would get delivered to people who had nothing to do with the original message. You are effectively clogging up a bunch of innocent peoples mail systems with your messages.

Put it another way, suppose some spammer sends 1,000,000 messages with your email address spoofed as the sender. If everyone else did what you do, you would then receive 1,000,000 messages back to your inbox giving you alternate instructions to contact these people.

You wouldn't want that. Nobody else does either. So please stop.

Re:False positive rate?

[vux984]

I do get your point really. But my dad (read: the boss) would not be happy if he missed a deal cause a million people who got spoofed got 1 mail from us telling them to call us if their message wasn't spam.

Read that over a few times. You are saying its ok to send out a MILLION unsolicited and annoying email messages (aka SPAM) to people who have never heard of you, so that your father won't miss a single deal?

How is that any different from rationalizing sending out a million direct marketing spam in the hopes of securing a single deal from it?

Bottom line, if your dad absolutely can't afford to "miss a deal", two things:

1) your backscatter system isn't going to necessarily work. Just because it worked 4 times is meaningless, you have no idea how many legitimate emails you lost. There is a high percentage change that your backscatter will be (correctly) identified as spam by other mail servers and discarded, so your notifications won't get delivered. And there is a high percentage that even if someone received your backscatter, they just deleted it. (I receive literally dozens of 'your mail could not be delivered' messages daily - some of them are backscatter, many of them are virus/malware pretending to be backscatter.) Which leads me to my next item:

2) if 'not missing a deal' is that important, then scan your own spam box for false positives. That's the sane way to handle this.

We send on every 1000 mails one message, telling them they got in the spambox and that they should call if it's not spam.

Please clarify this. Are you saying for every 1000 spam, you only send 1 notification? If so how do you choose which 999 spam you ignore vs the 1 you send a notification? Or are you saying only 1 in a thousand messages you receive is spam??

We are not the problem. The spammers are, so please, don't turn it around... I am not the problem

This is like seeing a drive by shooting in progress, whipping out your semi-auto and pumping as many rounds as you can in the general direction of the car. Your bullets are just as likely to hurt innocent bystanders as the criminals. Similiarly your email back scatter is just as harmful as the spam itself. You aren't the only problem, but you aren't part of the solution.

Now, go bug the dudes who don't want to make their precious smpt more secure.

Don't be naive. Secure email is trivial. Convincing everyone from Australia to Zimbabwe to switch to it is hard. And until YOU are willing to miss out on messages from people who haven't switched to your secure solution of choice, the problems will persist. And I don't see that happening anytime soon... you said it yourself... dad doesn't want to miss a single deal... no way in hell he's going to require that everyone who wants to send him messages conform to some new security regime... he'd miss messages left and right.

"IP addresses, he notes, are easy to fake."

[amorsen]

IP addresses, he notes, are easy to fake.

Sure, you can fake your IP address so you get past this filtering, because it just looks at the first packet. It won't help you though, because you can't complete a TCP 3-way handshake from a fake address, and without doing that you can't actually send spam.

Re:"IP addresses, he notes, are easy to fake."

[Anonymous Coward]

oh ye of little knowledge.

Re:"IP addresses, he notes, are easy to fake."

[Minwee]

It's easy, really. All you need to do is use a fake address which happens to be exactly the same as your real address.

It's as simple as closing a user's browser window without using Javascript.

Re:"IP addresses, he notes, are easy to fake."

[girlintraining]

oh ye of little knowledge.

If I compromise any layer 2 device on any network between you and the destination, not only can I fake the address, I can have it doing 480 spins in a pink tutu. Have you read any of the reports from the major network access points around the world? Bogus packets pass through them all the time. They even have a name for them -- martian packets.

Spatio-temporal

[CopaceticOpus]

So this software functions in both space AND time? Fascinating.

It's good that they specified that in the name, to avoid questions such as "Will this software work in the universe which we inhabit?"

Re:Not practical.

[oldspewey]

what happens when someone tries to contact me out of the blue before I have a chance to white list them?

Absolutely nothing happens ... at least from your perspective.

Is that really a practical trade-off?

[damn_registrars]

It sounds like this approach would be fairly CPU intensive; analyzing the characteristics of packets, comparing them to other packets, looking for information on their originating systems, etc... It seems like they are throwing a non-trivial amount of computational time at the problem in order to spare the storage space that would be otherwise taken up by spam.

And of course as others have already pointed out, this just starts another round of whac-a-mole by pursuing this avenue.

Wrong approach

[Animats]

The fundamental property of spam is that it involves many similar messages going to a large number of destinations. That's what to look for. Google can do that, because they manage a very large number of mailboxes with a single system. SpamCop used to do that, but they had to be in the mail-forwarding business to do it and that was too expensive.

Trying to detect spam by looking only at the mail for a single account is inherently a form of guessing. The existing technologies are reasonably good, but not good enough that the spammers give up.

Oblig Checklist

[crymeph0]

Your post advocates a

(x) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
(x) It will stop spam for two weeks and then we'll be stuck with it
(x) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
(x) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
(x) Armies of worm riddled broadband-connected Windows boxes
(x) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( ) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
(x) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
(x) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(x) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!

Obligatory!!

[jammindice]

Your post advocates a

( X ) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( X ) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( X ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
( X ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( X ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( X ) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email ( X ) Killing them that way is not slow and painful enough Furthermore, this is what I think about you: ( X ) Sorry dude, but I don't think it would work. ( ) This is a stupid idea, and you're a stupid person for suggesting it. ( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!

Re:I don't get it...

[Lennie]

Many have found, if your outside the US, blocking US is much more effective then blocking China and Russia.

MS -- Virus -- Bots -- Spam

[cenc]

Why does it seem everyone ignores the real source of the majority of spam: Microsoft windows computers infected by viruses running botnets that send spam. Yes, is generated by other systems, but not nearly the amount that is being generated by MS based botnets.

How about everyone just send their frigen spam bill to MS. How about a class action for everyone to collect for the damage that MS does to networks around the World. Better yet lets just forward all the spam we get to MS. Let them sort it out.

Re:Not practical.

[cybernanga]

From now on, whenever you complain to IT, do it in writing, and send them a telegram first, telling them to expect a letter with your complaint. Hopefully they will soon see sense.

Re:Not practical.

[John Hasler]

And follow up with a phone call to make sure they got the letter.