Stopping Spam Before It Hits the Mail Server

Al writes "A team of researchers at the Georgia Institute for Technology say they have developed a way to catch spam before it even arrives on the mail server. Instead of bothering to analyze the contents of a spam message, their software, called SNARE (Spatio-temporal Network-level Automatic Reputation Engine), examines key aspects of individual packets of data to determine whether it might be spam. The team, led by assistant professor Nick Feamster, analyzed 2.5 million emails collected by McAfee in order to determine the key packet characteristics of spam. These include the geodesic proximity of end mail servers and the number of ports open on the sending machine. The approach catches spam 70 percent of the time, with a 0.3 false positive rate. Of course, revealing these characteristics could also allow spammers to fake their packets to avoid filtering."

Comments

It'll work..except when it doesn't.

[MrCrassic]

I'll go first.

All spammers have to do is change the characteristics of the message. It's always going to be a cat and mouse game, just like antivirus and antispyware, so saying that they've found THE solution to blocking spam from hitting the server is slightly irresponsible.

Re:It'll work..except when it doesn't.

[ByOhTek]

Unless they use a truly novel approach of stopping spam before it hits the server.

I suggest an AK-47.

Re:It'll work..except when it doesn't.

[MrCrassic]

C4 on the outside of the firewall. That might remove more than expected...but it works!

Re:It'll work..except when it doesn't.

[gnick]

I realize that you're kidding, but removing more than expected is something that I consider unacceptable. If it hits the mail server and gets shuffled off into a spam folder with 100 pieces of trash, that's fine. But if it's not even going to make it to the mail server, 0.3% is too high a false positive rate.

Re:It'll work..except when it doesn't.

[OrangeMonkey11]

you've hit the nail on the head also considering they said the app catches 70% of the spam , so what happen to the other 30%

Re:It'll work..except when it doesn't.

[Have Brain Will Rent]

You are right, it is a cat and mouse game. The problem is that the solution being employed is looking at the wrong mouse and the wrong cat.

In this case the mice are the spammers themselves - the people - not the messages. And the cat shouldn't be anti-spam software, at least not in the sense of detect and eliminate software. The cat should be a bounty on the spammers themselves. There are always people who know who the spammers are. Those people are likely just as unscrupulous as the spammers so they can be induced to rat out the spammers for a reasonable reward. Think of the amount of money spent on software to deal with this, the value of ISP resources, the value of individual time dealing with this etc. It should be possible to offer some pretty significant rewards in return for helping with the elimination of the spammers.

The same method would work with those who deliberately spread a virus/worm for any reason.

Re:It'll work..except when it doesn't.

[TheBeardIsRed]

Did you read the article? I mean, sure it is a cat and mouse game, autonomous system manipulation and IP based forensics goes a little beyond the "characteristics of the message".

Re:It'll work..except when it doesn't.

[bschorr]

Well, the other 30% will be subject to the same filtering systems we use now I assume. I certainly wouldn't scrap my existing filters for a system that only catches 70%.

But it seems to me that there are already a variety of devices you can deploy between the firewall and your mail server (or even as part of your firewall) that promise to filter out significant amounts of spam. Barracuda and other such devices. Many firewalls have some spam filtering too. And there are services like Pau Spam or MailFoundry or others that promise to filter out most spam before it even hits your firewall.

So while these folks might have a new algorhythm it doesn't seem to me like there's anything revolutionary about filtering mail before it hits your mail server.

Re:It'll work..except when it doesn't.

[Alarash]

You are, of course, correct. The main flaw of such techniques is that you just need to hold the packets. What matters is that you deliver the packets, not the rate at which you deliver them. Instead of sending all the packets of a spam at the same time, the spammer will send the first packet of the first spam, then the first packet of the second, and so on. When they reached the first packet of the 100th spam, they'll send the second packet of the first spam, and so on.

This technique is used by Layer-7 attacks where DPI devices will 'forget' about an attack pattern if the packets of the said pattern are not all sent within X seconds/minutes.

Re:It'll work..except when it doesn't.

[foxx1337]

the short description: "developed a way to catch spam before it even arrives on the mail server"
>"a way"

tfa title: "A Better Way to Shoot Down Spam"
  >"a [...] way"

your post: "so saying that they've found THE solution to blocking spam"
>"THE solution"

someone's a bit eager to bash here...

Re:It'll work..except when it doesn't.

[Bysmuth]

Not quite - the hit rate is 70 percent, which means that 70 percent of the messages that are spam will never reach your inbox, and the other 30 percent will. The number you're after is the false alarm rate, 0.3%, which tells you that only 3 out of every 1000 non-spam messages will be incorrectly flagged as spam.

My initial reaction to these numbers was to wonder what percentage of e-mail is spam. The article says it's a whoppingly high 90.4%, which I think makes the false-alarm rate more than acceptable, as (by my calculations) only one out of every 229 e-mails flagged as spam will be a real e-mail. In contrast, if, say, only 1% of e-mails were spam (in which case we probably wouldn't need a spam detector at all, but still), 3 out of every 10 e-mails flagged as spam would actually be real e-mails.

Of course, this is just a long-winded way of noting that hit rates and false alarm rates aren't terribly useful without taking the base rate into account.

Re:It'll work..except when it doesn't.

[sqldr]

nah.. I mean, how can anyone defeat ANYTHING called "Spatio-temporal Network-level Automatic Reputation Engine"? I can barely even say it. When the machines finally take over, the human race will probably be wiped out by a Spatio-temporal Network-level Automatic Reputation Engine. It probably has its own gravity field. ZOMG! The Spatio-temporal Network-level Automatic Reputation Engines are coming!

RFC 3514

[Anonymous Coward]

Problem already solved back in 2003, I don't get any spam now.

Re:RFC 3514

[darpo]

For those who don't feel inclined to Google for it:

"The evil bit is a fictional IPv4 packet header field proposed in RFC 3514, a humorous April Fools' Day RFC from 2003 authored by Steve Bellovin. The RFC recommended that the last remaining unused bit in the IPv4 packet header be used to indicate whether a packet had been sent with malicious intent, thus making computer security engineering an easy problem."

Re:RFC 3514

[NevarMore]

Fictional?!?!?!?

Well then I guess its time to go polish up the old resume.

I don't get it...

[KC7GR]

Why do we need a crazily complex scheme like this when a simple entry in your router's 'Deny' list (for the source IP of the spam) has the same end effect?

Given the spew pouring out of the IP space of China, LACNIC, and Russia, blocking in such a manner appears to be near-lossless compression.

Re:I don't get it...

[BlueKitties]

Many spam messages are propagated by botnets, spoofed IPs, etc, so that isn't a perfect solution. Really, we need to combine different approaches, instead of trying to find a holy-grail.

Re:I don't get it...

[JoeBuck]

Spam is almost exclusively produced by botnets. Vulnerable computers exist all over the world, so it shouldn't be surprising that more spam comes from outside your country (wherever you live) than inside. You, personally, have no one in China or Russia that you correspond with, but a debtor nation like the US is in a rather poor position to f*ck with the legitimate mail traffic of its main creditor. The most effective way to kill spam would be to aggressively eliminate botnets, wherever they are. A machine determined to be a member of a botnet could be isolated, blocked from sending email any place other than the support address of its ISP. Access could be restored when the machine is disinfected.

Re:I don't get it...

[Drakin020]

Because many spam emails are generated from open relay servers.

Re:I don't get it...

[Lennie]

Many have found, if your outside the US, blocking US is much more effective then blocking China and Russia.

Re:I don't get it...

[TheRaven64]

Good plan, block the countries sending the most spam. Currently, most spam is sent from the USA. I notice that your mail server is in the USA, so unfortunately this means you won't be able to contact anyone adopting this plan, but I don't think it's too high a price to pay for reducing the total amount of spam.

Re:I don't get it...

[John Hasler]

> Good plan, block the countries sending the most spam.

No, just block countries that send a lot of spam and in which you have no correspondents. Obviously this will work well for some and poorly for others. If it won't work well for you don't do it.

Re:I don't get it...

[element-o.p.]

A machine determined to be a member of a botnet could be isolated, blocked from sending email any place other than the support address of its ISP. Access could be restored when the machine is disinfected.

That's what we did at the ISP I used to work at.We ticked off a few customers but not as many as were ticked off at us before we implemented that policy. I was amazed at how vocal our customers could be when AOL started rejecting our e-mails...

Re:I don't get it...

[element-o.p.]

Not in my experience, for however much or little that's worth.

I found that the bulk of UCE that hit my mail server came from either China or Korea. After I began blocking all IP addresses from either of those two countries, I found the amount of spam hitting my inbox was almost nil.

Of course, it was my personal mail server, and I don't have any contacts in either of those countries, so the SNR from China and Korea was essentially zero (no signal, all noise). YMMV, however.

Re:I don't get it...

[MikeS2k]

Indeed, a lot of spam to my mail server comes from China, Korea, or India.
I see the occasional spam come from the USA, but it's a very small amount. Same with Canada.
The hostnames I often see seem to belong to residential addresses - DSL connections etc. It seems a damn botnet is responsible for sending all the "Acai berry" and Viagra/Cialis spams to my domain.

I don't shitcan the mails from "bad" countries outright, but I do increase the "weight" / probability it might be spam.
I've also whitelisted Britain, so no e-mail coming from UK IP addresses will get filtered (I am based in the UK)

Setting this up on an SMTP proxy box running ASSP took a day, and has reduced spam on my network by 90-95% - it's a no brainer, really.

Spammers evolve from experience

[pearl298]

Just like other criminals, spammers must quickly respond to what actually works. In essence this is the flaw in any "security by obscurity" scheme, the bad guys simply respond to whatever works. If you get to try several billion times a day then you can try a whole lot of combinations.

.3% false positive is pretty high

[Dynedain]

That means that in my office of 50 people, with an average of 50 emails per day (a very very low estimate), we'd get 7-8 false positives daily. I'd hear bloody murder if that was the case.

We get a lot more mail than that per day, and our spamassassin without autolearning (simply flag anything higher than 5.0) does a hell of a lot better job than that... down in the range of 1-2 false positives a month. Assuming a low daily average of emails (like my example), that's .002% false positives.

Re:.3% false positive is pretty high

[Sockatume]

And of course, if you want to actually spot the false positives, you have to let all the spam hit the mail server anyway. Unless you're willing to just ignore all the spam packets and put up with all those false positives being lost to the ether, this won't reduce your mail processing load at all.

Re:.3% false positive is pretty high

[dyingtolive]

It is somewhat ambiguous, but I had read it 0.3%, not 3%, which implies that you'd lose 0-1 emails/day if you were averaging 50 total a day. Still higher that way than your current method, but nowhere near as bad as 7-8 daily.

Re:.3% false positive is pretty high

[Sockatume]

50 a day * 50 people = 2500 messages, 2500 messages * 0.3% = 7.5 emails.

Re:.3% false positive is pretty high

[dyingtolive]

Ah, right. For some reason, I was reading 50 emails a day total. I seem to have taken my stupid pills today.

Re:.3% false positive is pretty high

[scorp1us]

Right, you read it wrong, like you were supposed to. 70% = 0.7, 30%= 0.3. Ergo, if it isn't catching spam correctly, its marking the rest as spam, that way you catch all the spam! I wonder at what point in time it'd be better to reject everything and just deal with escalated messages (to phone calls, txts, tweets, etc). Then you can ignore email all together.

Re:.3% false positive is pretty high

[element-o.p.]

I wonder if you could make this technology a plug-in to SA to simply bump up the spam score by, say, a point and continue with your SA filtering anything higher than 5.0?

Re:.3% false positive is pretty high

[Ioldanach]

From the article, "The end result was a system capable of detecting spam 70 percent of the time, with a 0.3 percent false positive rate." The summary dropped an instance of the word "percent". I wasn't sure how to read it either so I specifically looked for the source of the 0.3 in the original.

Re:.3% false positive is pretty high

[Ioldanach]

50 * 50 * .3% = 2500 * .003 = 7.5, so yes 7-8 emails per day. 3% would be 75 false positives per day. Incidentally, the 70% success rate if every person got one spam for every good email means the company would still receive 750 spams per day, or 15 per person per day.

Re:.3% false positive is pretty high

[scorp1us]

I was hoping to be modded funny. Ir insightful. hard to tell the difference these days...

Re:.3% false positive is pretty high

[ancientt]

what point in time it'd be better to reject everything and just deal with escalated messages

This is in fact, the real solution. Email was not designed for the world we live in and cannot be truly fixed. The convenience and ubiquity of email, however, makes it very difficult to give up. IM, social networking and SMS are taking over the role that email used to reign alone. There will be a natural progression toward obsolescence as email is replaced.* When the consumer does not need email, that will be the tipping point. At that point some of the solutions that cannot hope to be implemented now can begin to be considered. (Crypto-signed emails, validated servers with validated nodes and pay per email spring to mind.)

* - At some point there will be a widely enough supported solution that email will be obsolete enough in businesses that the average consumer will not need it. That's a lot of "that"s. If it were said thus, however, even more readers would suffer permanent eye glaze.

If you going to cut & paste, then do it correc

[davebarnes]

The original is "The end result was a system capable of detecting spam 70 percent of the time, with a 0.3 percent false positive rate."

False positive rate?

[johndiii]

0.3 would be terrible - three out of ten false positives. 0.3 percent - what the article actually says - is not too bad. But current techniques allow me to check the spam bin for such messages. This technique would pretty much preclude that capability, since the mail would never arrive at the server. I'm not sure that a rate of 0.003 would be acceptable under those circumstances.

Re:False positive rate?

[santax]

Help me here... Personally I would think that if 10 is 100% 0.3 is less than 1 mail. And not 3 out of 10. Personally I'd rather deal with spam than mis out one 1 legitimate mail. My own personal anti-spam filter is quite rough but I don't mis that many mails. And when my mail filters blocks spam, it sends out a message with redirections to an alternative gsm-number telling them to call me so I can whitelist the adres. I had about 4 times people calling me. Not one of those was a spamwhore :) So the granted whitelist turned out to work perfect.

Re:False positive rate?

[girlintraining]

0.3 percent false positive

They predicted something around 97 billion e-mails per day sent in 2007. I wouldn't want to guess what it's at today, but it's probably higher. Regardless, 0.3% of the emails equates to about 291 million legitimate emails per day black holing. No errors. No "marked return to sender". It just vanishes, eaten by the shub internet. Oops. And we can be pretty sure those numbers are higher -- this is a back of the envelope analysis.

Re:False positive rate?

[amorsen]

Personally I would think that if 10 is 100%

10 isn't 100%. 1 is 100%. That's how % is defined.

Re:False positive rate?

[amorsen]

No errors. No "marked return to sender".

If the box just dumps the packets on the floor, the sender will eventually get an error message from their mail server. Of course the mail server will have tried uselessly quite a lot of times (for days, usually) before giving up.

Re:False positive rate?

[raju1kabir]

Help me here... Personally I would think that if 10 is 100% 0.3 is less than 1 mail. And not 3 out of 10.

.3 is 300 out of 1000.

.3% is 3 out of 1000.

It's similar to the confusion created when idiots write "It only costs me .25 cents to make a phone call" when they really mean ".25" or "25 cents".

Re:False positive rate?

[vux984]

And when my mail filters blocks spam, it sends out a message with redirections to an alternative gsm-number telling them to call me so I can whitelist the adres.

That's called back scatter and its as bad as spam.

Think about it, my mail servers block about 35,000 spam per day. If they sent a message to each failed recipient with alternative instructions, that would be 35,000 messages I sent out. Some 34,990 of those messages would either be undeliverable or would get delivered to people who had nothing to do with the original message. You are effectively clogging up a bunch of innocent peoples mail systems with your messages.

Put it another way, suppose some spammer sends 1,000,000 messages with your email address spoofed as the sender. If everyone else did what you do, you would then receive 1,000,000 messages back to your inbox giving you alternate instructions to contact these people.

You wouldn't want that. Nobody else does either. So please stop.

Re:False positive rate?

[element-o.p.]

Personally I would think that if 10 is 100%

10 isn't 100%. 1 is 100%. That's how % is defined.

Trying to follow all of the numbers without any context is making my head hurt.

Neither TFS, GPP nor you were very clear on the key aspect of percentages: they are a ratio. You said, " 10 isn't 100%" but that's not necessarily true. Ten out of how many? 10 out of 10 is 100%. However, expressed as a ratio, 10/10 = 1, which is what you said is how 100% is defined (100 per cent, i.e., 100 per 100, or 100/100 = 1 = 100%). Since no one specified ten out of how many total, 10 could be 1%, 100%, 1000% 3.14159265358% or 42%; you can't tell.

</pedantic>

Re:False positive rate?

[element-o.p.]

Mod parent up!

Back scatter spam has been a known issue for a long time now. There's no more excuse for anyone still operating a mail server this way than there is for anyone still knowingly operating an open relay (since for all intents and purposes, that's what this is).

Re:False positive rate?

[mshieh]

.3% false positive rate isn't bad but isn't great. However, you have to think of this approach as a technique rather than a solution. An effective anti-spam solution will combine several techniques, so the false-positive rate of any individual technique won't be enough to reject mail. Also, the penalty for flagging a mail as spam can be scored in a way that mail is not lost. For example, yahoo is notorious for flagging legitimate mail as spam, but generally delays the mail via greylisting instead of rejecting it outright.

When combined with other scoring mechanisms into an overall heuristic, .3% is tolerable as one tool in the box. However, my first impression is that this is at best an incremental improvement over an IP blacklist.

Re:False positive rate?

[gujo-odori]

Actually, that's horrible, especially at such a low catch rate. I work for one of the major anti-spam vendors, and if our FP rate was that high, the only thing that would stop our customers from killing us is the fact that they would all be former customers. People get called in the middle of the night for an FP rate much, much lower than that. And our catch rate is way, way north of 99%.

At a catch rate of only 70%, we could guarantee zero false positives. Ever. Anyone who gets an FP rate that high at a catch rate of only 70% has nothing to brag about. I'd be ashamed to show my face in public, let alone publish my results.

Re:False positive rate?

[MikeS2k]

What's the problem with rejecting the SMTP session, with the error displaying the SMTP error code along with your phone number/error message in it?

Re:False positive rate?

[vux984]

What's the problem with rejecting the SMTP session, with the error displaying the SMTP error code along with your phone number/error message in it?

Nothing, but that's not what was described. What was described was a situation where a 'bounce' message was sent by the recipient.

Additionally, for performance reasons a lot of spam processing usually happens after the message is accepted. In this case you can't reject the SMTP session, because by the time you decide its spam, the session is long over.

Re:False positive rate?

[santax]

And when my mail filters blocks spam, it sends out a message with redirections to an alternative gsm-number telling them to call me so I can whitelist the adres.

That's called back scatter and its as bad as spam.

Think about it, my mail servers block about 35,000 spam per day. If they sent a message to each failed recipient with alternative instructions, that would be 35,000 messages I sent out. Some 34,990 of those messages would either be undeliverable or would get delivered to people who had nothing to do with the original message. You are effectively clogging up a bunch of innocent peoples mail systems with your messages.

Put it another way, suppose some spammer sends 1,000,000 messages with your email address spoofed as the sender. If everyone else did what you do, you would then receive 1,000,000 messages back to your inbox giving you alternate instructions to contact these people.

You wouldn't want that. Nobody else does either. So please stop.

I do get your point really. But my dad (read: the boss) would not be happy if he missed a deal cause a million people who got spoofed got 1 mail from us telling them to call us if their message wasn't spam. It's a flaw in the smtp protocol and we are handeling it is as best as we can. We send on every 1000 mails one message, telling them they got in the spambox and that they should call if it's not spam. We are not the problem. The spammers are, so please, don't turn it around... I am not the problem, I just tackle the problem in the best way for both me and our customers. And it works. Now, go bug the dudes who don't want to make their precious smpt more secure.

Re:False positive rate?

[Leebert]

I do get your point really. But my dad (read: the boss) would not be happy if he missed a deal cause a million people who got spoofed got 1 mail from us telling them to call us if their message wasn't spam.

...

I am not the problem, I just tackle the problem in the best way for both me and our customers.

I really, truly hope you and your dad both are victims of a joe-job some time in the near future. I'm reasonably certain that this is the only way you'll truly understand.

(Not that you'd actually see it with your current setup.)

Re:False positive rate?

[vux984]

I do get your point really. But my dad (read: the boss) would not be happy if he missed a deal cause a million people who got spoofed got 1 mail from us telling them to call us if their message wasn't spam.

Read that over a few times. You are saying its ok to send out a MILLION unsolicited and annoying email messages (aka SPAM) to people who have never heard of you, so that your father won't miss a single deal?

How is that any different from rationalizing sending out a million direct marketing spam in the hopes of securing a single deal from it?

Bottom line, if your dad absolutely can't afford to "miss a deal", two things:

1) your backscatter system isn't going to necessarily work. Just because it worked 4 times is meaningless, you have no idea how many legitimate emails you lost. There is a high percentage change that your backscatter will be (correctly) identified as spam by other mail servers and discarded, so your notifications won't get delivered. And there is a high percentage that even if someone received your backscatter, they just deleted it. (I receive literally dozens of 'your mail could not be delivered' messages daily - some of them are backscatter, many of them are virus/malware pretending to be backscatter.) Which leads me to my next item:

2) if 'not missing a deal' is that important, then scan your own spam box for false positives. That's the sane way to handle this.

We send on every 1000 mails one message, telling them they got in the spambox and that they should call if it's not spam.

Please clarify this. Are you saying for every 1000 spam, you only send 1 notification? If so how do you choose which 999 spam you ignore vs the 1 you send a notification? Or are you saying only 1 in a thousand messages you receive is spam??

We are not the problem. The spammers are, so please, don't turn it around... I am not the problem

This is like seeing a drive by shooting in progress, whipping out your semi-auto and pumping as many rounds as you can in the general direction of the car. Your bullets are just as likely to hurt innocent bystanders as the criminals. Similiarly your email back scatter is just as harmful as the spam itself. You aren't the only problem, but you aren't part of the solution.

Now, go bug the dudes who don't want to make their precious smpt more secure.

Don't be naive. Secure email is trivial. Convincing everyone from Australia to Zimbabwe to switch to it is hard. And until YOU are willing to miss out on messages from people who haven't switched to your secure solution of choice, the problems will persist. And I don't see that happening anytime soon... you said it yourself... dad doesn't want to miss a single deal... no way in hell he's going to require that everyone who wants to send him messages conform to some new security regime... he'd miss messages left and right.

Re:False positive rate?

[lanzz]

What's the problem with rejecting the SMTP session, with the error displaying the SMTP error code along with your phone number/error message in it?

qmail is usually the problem, not supporting recipient verification during smtp out of the box due to its security model of minimum privileges for each subsystem.

Re:False positive rate?

[crimperman]

> What's the problem with rejecting the SMTP session, with the error displaying the SMTP error code along with your phone number/error message in it?

Agreed! Back-scatter in the form of messages generated by the recipient server are worse than the original spam in my opinion because you can't effectively block them without blocking bounces for mail your clients _did_ send. Add to this that spammers usually fake the from and reply-to headers and it's somebody else getting the bounces.

Receiving SMTP servers should not send bounces for unknown recipients (and should _never_ respond to suspect spam messages). Instead they should simply reject the message at SMTP level with a user unknown error. You can add any custom message to that. On a previous server we had "user unknown - please try info@". This works because the relaying server handles the rejection and in turn responds to whomever it permitted to relay through it. I am yet to see a legitimate reason for a receiving SMTP server to generate a bounce message (out off office replies is a different issue!).

Re:False positive rate?

[u38cg]

Thanks. Half the spam I receive is backscatter of various sorts.

Re:False positive rate?

[SlashDev]

I'm sorry, but how did you calculate that 0.3 = 3 out of ten? 0.3 is out of a hundred, that would be 0.03 out of ten. The article says 70% caught and marked as spam, out of those 70%, 0.3% were false positives.

Re:False positive rate?

[johndiii]

0.3% does indeed correspond to a rate of 0.003. That's what the article said, but it's not what the summary said. The summary said "The approach catches spam 70 percent of the time, with a 0.3 false positive rate." Omitting the "percent" changes the actual value by a factor of 100 from what the article says.

"IP addresses, he notes, are easy to fake."

[amorsen]

IP addresses, he notes, are easy to fake.

Sure, you can fake your IP address so you get past this filtering, because it just looks at the first packet. It won't help you though, because you can't complete a TCP 3-way handshake from a fake address, and without doing that you can't actually send spam.

Re:"IP addresses, he notes, are easy to fake."

[Anonymous Coward]

oh ye of little knowledge.

Re:"IP addresses, he notes, are easy to fake."

[Minwee]

It's easy, really. All you need to do is use a fake address which happens to be exactly the same as your real address.

It's as simple as closing a user's browser window without using Javascript.

Re:"IP addresses, he notes, are easy to fake."

[girlintraining]

oh ye of little knowledge.

If I compromise any layer 2 device on any network between you and the destination, not only can I fake the address, I can have it doing 480 spins in a pink tutu. Have you read any of the reports from the major network access points around the world? Bogus packets pass through them all the time. They even have a name for them -- martian packets.

Re:"IP addresses, he notes, are easy to fake."

[element-o.p.]

Research before slamming others, please.

It is possible to spoof an IP address and fake a TCP three-way handshake from a fake address. The trick is that any modern OS randomizes TCP sequence numbers to make it difficult to "complete" the three-way handshake without actually receiving part 2 of the three-way handshake. However, if you have the analytical tools to guess the correct sequence number to send back in part 3 of the three-way handshake, you're golden.

It's not exactly trivial, but girlintraining is correct -- it is possible to complete a three-way handshake from a fake address.

Re:"IP addresses, he notes, are easy to fake."

[ajs]

IP addresses, he notes, are easy to fake.

Sure, you can fake your IP address so you get past this filtering, because it just looks at the first packet. It won't help you though, because you can't complete a TCP 3-way handshake from a fake address, and without doing that you can't actually send spam.

Not true. When we say "fake", we don't always mean "not your valid IP address right now." For example, you might send your spam from a van that drives slowly through a large city, taking advantage of any open corporate wireless networks it finds on the way. That's one way. Another is to simply bribe your way in to a different ISP or corporate network every night (this has been done). You drive your van up to the back door at 3AM, pay the NOC guy on duty $1000 to hand you a live RJ45 jack and you pump out a few billion messages, pack up and go.

Variants of the above exist and are used all over the world. Gone are the days of a few spam-friendly ISPs. Today it's much harder to nail down the bad guys.

Re:"IP addresses, he notes, are easy to fake."

[amorsen]

I really think "breaking into other people's networks" shouldn't be called "faking your IP address". They are very different concepts.

Still reaching its destination

[darpo]

Isn't this just pushing the processing back a level, but still arriving at its destination? I guess you could implement bandwidth-provider-level (i.e. before the customer even gets their packets) spam filtering this way, but I'm sure most organizations would prefer to retain control by doing their own filtering.

Spatio-temporal

[CopaceticOpus]

So this software functions in both space AND time? Fascinating.

It's good that they specified that in the name, to avoid questions such as "Will this software work in the universe which we inhabit?"

Re:Spatio-temporal

[amorsen]

Perhaps they should add some disclaimers, just to be completely sure.

I'd use ...

[neonprimetime]

a baseball glove.

But I'd first have to question why somebody is throwing spam at my mail server in the first place?

Re:Not practical.

[oldspewey]

what happens when someone tries to contact me out of the blue before I have a chance to white list them?

Absolutely nothing happens ... at least from your perspective.

IronPort and the ilk

[snooz_crash]

I've got a device in front the mail server, many people do. These and others work fine. Sorry for folks that don't have one. As long as it is free, it will be abused. Someone already said it was cat and mouse.

Global companies

[mu51c10rd]

I hear this suggestion a lot. However, many of us work for global companies that deal with legitimate email from these countries. We can't just reject IP blocks for countries when we have dealings in them. China and Russia are huge for international companies.

Is that really a practical trade-off?

[damn_registrars]

It sounds like this approach would be fairly CPU intensive; analyzing the characteristics of packets, comparing them to other packets, looking for information on their originating systems, etc... It seems like they are throwing a non-trivial amount of computational time at the problem in order to spare the storage space that would be otherwise taken up by spam.

And of course as others have already pointed out, this just starts another round of whac-a-mole by pursuing this avenue.

Re:Is that really a practical trade-off?

[element-o.p.]

I was thinking the same thing, only along a slightly different line:

These include...the number of ports open on the sending machine.

WTF? Does the filter nmap the sending mail server before accepting the message?!?!

A Spam Filter is like DRM

[olsmeister]

Regardless of how complex you make it, someone will always eventually figure out a way around it.

Wrong approach

[Animats]

The fundamental property of spam is that it involves many similar messages going to a large number of destinations. That's what to look for. Google can do that, because they manage a very large number of mailboxes with a single system. SpamCop used to do that, but they had to be in the mail-forwarding business to do it and that was too expensive.

Trying to detect spam by looking only at the mail for a single account is inherently a form of guessing. The existing technologies are reasonably good, but not good enough that the spammers give up.

Re:Wrong approach

[cpghost]

The fundamental property of spam is that it involves many similar messages going to a large number of destinations.

It won't be long until the zombies create individual spams for each recipient. Just scramble the catch words, add some random stuff to the gifs so they message-digest differently etc..., and there's not enough similarity in the messages anymore to be statistically detectable. If at all, traffic analysis would help, but here too, botnets are extremely flexible and could spread batch runs in IP-space and time domain quite effectively, if need be. It's a never ending arms' race.

Re:Wrong approach

[jonbryce]

Messagelabs are in the mail-forwarding business, and they seem to manage to make money out of it.

Re:Wrong approach

[ajs]

It won't be long until the zombies create individual spams for each recipient. Just scramble the catch words, add some random stuff to the gifs so they message-digest differently etc.

Back when I was in the spam hunting business, we called that 2002. Since then, techniques have become radically more sophisticated.

Re:If you going to cut & paste, then do it cor

[godrik]

Oh yeah. I was thinking a rate of 0.3 was huge. 0.3 percent is much better but still not acceptable.

Re:Just analyze the source....

[jarl1976]

Taiwan is not a third world country, and depending on who you ask not even a country. Anyway blocking out all of asia is probably a bad idea for many businesses.

Oblig Checklist

[crymeph0]

Your post advocates a

(x) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
(x) It will stop spam for two weeks and then we'll be stuck with it
(x) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
(x) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
(x) Armies of worm riddled broadband-connected Windows boxes
(x) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( ) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
(x) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
(x) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email
( ) Killing them that way is not slow and painful enough

Furthermore, this is what I think about you:

(x) Sorry dude, but I don't think it would work.
( ) This is a stupid idea, and you're a stupid person for suggesting it.
( ) Nice try, assh0le! I'm going to find out where you live and burn your
house down!

Re:Oblig Checklist

[dkleinsc]

I think you missed a few:
(X) Bandwidth costs that are unaffected by client filtering

(X) Ideas similar to yours are easy to come up with, yet none have ever been shown practical.

Obligatory!!

[jammindice]

Your post advocates a

( X ) technical ( ) legislative ( ) market-based ( ) vigilante

approach to fighting spam. Your idea will not work. Here is why it won't work. (One or more of the following may apply to your particular idea, and it may have other flaws which used to vary from state to state before a bad federal law was passed.)

( ) Spammers can easily use it to harvest email addresses
( ) Mailing lists and other legitimate email uses would be affected
( ) No one will be able to find the guy or collect the money
( ) It is defenseless against brute force attacks
( X ) It will stop spam for two weeks and then we'll be stuck with it
( ) Users of email will not put up with it
( ) Microsoft will not put up with it
( ) The police will not put up with it
( ) Requires too much cooperation from spammers
( ) Requires immediate total cooperation from everybody at once
( ) Many email users cannot afford to lose business or alienate potential employers
( ) Spammers don't care about invalid addresses in their lists
( ) Anyone could anonymously destroy anyone else's career or business

Specifically, your plan fails to account for

( ) Laws expressly prohibiting it
( ) Lack of centrally controlling authority for email
( ) Open relays in foreign countries
( ) Ease of searching tiny alphanumeric address space of all email addresses
( X ) Asshats
( ) Jurisdictional problems
( ) Unpopularity of weird new taxes
( ) Public reluctance to accept weird new forms of money
( ) Huge existing software investment in SMTP
( ) Susceptibility of protocols other than SMTP to attack
( ) Willingness of users to install OS patches received by email
( ) Armies of worm riddled broadband-connected Windows boxes
( X ) Eternal arms race involved in all filtering approaches
( ) Extreme profitability of spam
( ) Joe jobs and/or identity theft
( ) Technically illiterate politicians
( ) Extreme stupidity on the part of people who do business with spammers
( X ) Dishonesty on the part of spammers themselves
( ) Bandwidth costs that are unaffected by client filtering
( ) Outlook

and the following philosophical objections may also apply:

( X ) Ideas similar to yours are easy to come up with, yet none have ever
been shown practical
( ) Any scheme based on opt-out is unacceptable
( ) SMTP headers should not be the subject of legislation
( ) Blacklists suck
( ) Whitelists suck
( ) We should be able to talk about Viagra without being censored
( ) Countermeasures should not involve wire fraud or credit card fraud
( ) Countermeasures should not involve sabotage of public networks
( ) Countermeasures must work if phased in gradually
( ) Sending email should be free
( ) Why should we have to trust you and your servers?
( ) Incompatiblity with open source or open source licenses
( ) Feel-good measures do nothing to solve the problem
( ) Temporary/one-time email addresses are cumbersome
( ) I don't want the government reading my email ( X ) Killing them that way is not slow and painful enough Furthermore, this is what I think about you: ( X ) Sorry dude, but I don't think it would work. ( ) This is a stupid idea, and you're a stupid person for suggesting it. ( ) Nice try, assh0le! I'm going to find out where you live and burn your house down!

Re:Obligatory!!

[Sir_Lewk]

This one seems to be filled out more correctly.

Re:Obligatory!!

[morcego]

Ideas similar to yours are easy to come up with, yet none have ever
been shown practical

This is partially true. Spamassassin uses a few of the things described on the article already.

Re:Obligatory!!

[Hoi Polloi]

How about requiring that every email be printed out then physically delivered to the recipient? Wait a minute...

Re:Obligatory!!

[thePowerOfGrayskull]

Speaking of Obligatory, one can't make the post above without giving credit to its grandfather.

Re:Obligatory!!

[u38cg]

And does anyone know what some of the questions are leading to the given answers?

Two things:

[Hurricane78]

First: I do not want others to decide what's spam for me.
Second: I got graylisting, amavisd with spamd & co, and more. Why exactly would I put such a system on every other node of the net too? To throw away resources?

Re:Not practical.

[gnick]

Slightly off-topic, sorry, but I think it's abysmal enough to post and interest a few (or just make you thankful you're not here.)

"Absolutely nothing" is my company's solution to filtering out large or suspect attachments. If somebody sends me an attachment and my company's filters don't like it, the e-mail is dropped. I don't get a notice saying, "This e-mail contains suspicious attachments and has been removed." My customer doesn't get a reply saying, "This e-mail could not be delivered to the recipient because it contained suspicious attachments." Nothing - Zip, zero, nothing. My customer thinks it went through and it's invisible to me.

After numerous complaints to IT, the response was that I need to contact each of my customers and any of their contacts that may be sending e-mail that I may be copied on with an attachment and have them call me on the phone any time they send on so that I know to expect it.

Beautiful, huh?

with a 0.3 false positive rate

[QuietLagoon]

What exactly does this mean? A rate is usually a comparison of two values. What two values were compared to get 0.3?

Re:with a 0.3 false positive rate

[wastedlife]

Well, 0.3 usually translates to 30%, so that is how I read it. I see McAfee is involved somehow, so that is likely an improvement for them.

Not that precise

[Big_Monkey_Bird]

Big whoop. All it does is block email with IP addresses from France, Belgium, Russia, Italy, and Argentina.

Sounds like TurnTide

[rysar]

I want to try to keep this as non-spam as possible, but Symantec acquired a company about 5 years ago called TurnTide that did almost *exactly* that. Take the reputation of the sending address, and shape the TCP/IP packets to slow down the rate of mail into the system. Symantec touts a 70% reduction in mail volume and an 80% reduction in the amount of spam that hits a mail server. I've had it in production in one environment where the customer went from approximately 5 million messages/day to 500,000 messages/day.

I hope they don't get rid of my spam folder.

[FlyingSquidStudios]

It's become a source of unending comedy as spammers who aren't very good at English in the first place use a dictionary and thesaurus to get past the filtering software resulting in extremely entertaining subject lines. For example-

YOU REMEMBER WHEN SEX WAS THE LAST TIME? REFRESH THE MEMORY OF VIA GRA!

No more hair Rogaining medicine.

GIRLS DO ANYTHING FOR A BIG HOSE

It boosts your rod!

Make two days nailing marathon

for your delicate advantage

And all that is just from the most recent page in my spam folder.

Re:Don't worry about false positives

[Bondolo]

Except that the sender's source domain is almost always forged in spam. So, as an owner of several domains, I get hundreds of bounce messages per day in response to spams sent with my one of domains as the source domain. The "Undeliverable Mail" messages I receive have become, for me, almost worse than the spam.

Re:LinuxMagic already did this

[babywhiz]

The spam filter we use already does a geography-based check. Even allows you to choose how to classify it based on geography....as well as IP, grey listing, header, recipient, sender, subject, content, bayesian pattern, honeypot.... My complaint anymore is not about the amount of spam making it in (we have had 1 that slipped thru the filters in the last 6 months, but that is because it actually spoofed an email address on our whitelist (we don't whitelist domains at all). It's the amount of traffic created, and how huge our log files get, and even if we set the amount of logging to limit log file size, then we end up with HUNDREDS of smaller log files.

Way too high I think.

[TheLink]

But do those 2500 messages include spam or are they just the mails that get through the existing spam filters?

Otherwise my understanding of the 0.3% false positive is where 100% = the total number of emails.

Which is rather unacceptable given the handling of false positives, and the total number of emails could be very high when you include spam.

Re:Way too high I think.

[Tony Hoyle]

If you assume that 90% of all email is spam, and 2500 is the legit number, that's 250000 emails hitting the mailserver and being rejected.. a .3% false positive means 750 emails a day being dropped.

IMO that's pretty useless.

Re:LinuxMagic already did this

[babywhiz]

Oh, and I don't use LinuxMagic LOL...

Nothing extraordinary...

[nudzo]

Technology which is already here for a long time has now a buzzword - SNARE. For example OpenBSD spamd doing the same based on blacklists, greylists and even on Operating System fingerprints. Wheel is reinvented again... ;-)

rblsmtpd

[Mr_Reaper]

Since 1996...

Even Better

[nixdroid]

A few years ago the company I worked for came under an email DOS attack that bogged down our Exchange server to the point that it took about 10 hours for a legitimate email to get through. The Windows admins tried all 10 spam settings with no affect. I put a Linux box running SpamAssassin in front of the Exchange server and within a couple of hours the delivery time dropped to about 10 seconds. Products like SpamAssassin are essentially dynamic filters that can and do get fresh filter information as often as you like. This case was a dictionary attack and we got rid of the vast majority of the spam by the simple expedient of deleting anything that wasn't addressed to a legitimate account. As another poster noted, most spam filtering methods are just educated guessing. Rely on one that is educable.

Re:If you going to cut & paste, then do it cor

[Tony Hoyle]

0.3% FP on the total mail input, but 90% is spam anyway.. so that means 3% of legit mail is dropped.

3$ is way too high.

MS -- Virus -- Bots -- Spam

[cenc]

Why does it seem everyone ignores the real source of the majority of spam: Microsoft windows computers infected by viruses running botnets that send spam. Yes, is generated by other systems, but not nearly the amount that is being generated by MS based botnets.

How about everyone just send their frigen spam bill to MS. How about a class action for everyone to collect for the damage that MS does to networks around the World. Better yet lets just forward all the spam we get to MS. Let them sort it out.

Re:MS -- Virus -- Bots -- Spam

[justinlee37]

Awesome, then after we drive MS into the ground we can brace ourselves for the new wave of malware that will inevitably strike the new common standard of operating systems!

The real source of the majority of spam is from hackers who develop botnets. They only target windows because it's a widely accepted standard used by people who are not technically proficient. Furthermore you might be able to sue the spammers but you would have hard time suing Microsoft. So you either need to get rid of all the hackers or educate people on how to keep their machines free of infection. Simply getting everyone to change operating systems will not do a thing for anyone.

For the record, I use windows most of the time, and I keep my machine squeaky-clean simply by browsing intelligently, using Firefox, and running a third-party anti-virus program. So you can keep a windows machine clean. Windows isn't the problem ... PEBKAC.

Re:MS -- Virus -- Bots -- Spam

[cenc]

Most of that dead dog you just posted has been beat and beat again. If "the new standard" OS where an issue, we would have real and WIDE SPREAD viruses in the wild long long ago for the millions of unix servers that have been under attack for years. This is a far more a fundamental issue of systems design and responsibility of MS for their product.

 

Re:MS -- Virus -- Bots -- Spam

[justinlee37]

Is there a confounding variable? Are users of unix servers typically more technically proficient? Does the widespread proliferation of PCs and consumer internet change the game?

Re:MS -- Virus -- Bots -- Spam

[NoOneInParticular]

The confounding variable is called diversity. In a mono-culture such as Microsoft, a single exploit immediately makes 90% of the market vulnerable. In a Unix world, a crack for Solaris has no effect on HP-UX, Linux or *BSD. So, the blackhat might be able to take out a particular version of Solaris, but he then needs to figure out where all these boxes are... In the meantime, the rest of the internet keeps chugging along nicely until the pieces are picked up.

Re:MS -- Virus -- Bots -- Spam

[justinlee37]

So, if the mono-culture of Windows goes away, do you really think it will be replaced by diversity? Or will all of the computer illiterate users gravitate towards the same system, creating a similar environment?

0.3 false positives

[yamfry]

So 70% of the time it works every time. Sold.

Spam eradication

[chemeleon]

Wouldn't it be cheaper in the long run to simply design a new mail protocol from the ground up, with security and spam prevention as the main focuses? It seems to me that when you need to implement solutions which are as complex as this one to keep the system running as intended it is more or less a failure.

Re:Spam eradication

[raylu]

That's a great idea. While you're at it, make browsers W3C compliant too.

Re:Spam eradication

[sgt scrub]

Wave. A MIMEXML, XMPP/SMTP hybrid type thing. http://wave.google.com/

Re:Not practical.

[cybernanga]

From now on, whenever you complain to IT, do it in writing, and send them a telegram first, telling them to expect a letter with your complaint. Hopefully they will soon see sense.

Re:Not practical.

[John Hasler]

And follow up with a phone call to make sure they got the letter.

Re:If you going to cut & paste, then do it cor

[ajs]

Not even remotely. At best this system could only be used as input to a secondary system that then uses this information along with other sources. See, e.g., SpamAssassin's scoring approach.

Network-based spam blockers

[Netssansfrontieres]

... like network-based virus blockers bring several good things:
* an entirely different set of algorithms can be used, leveraging data and traffic patterns not specific to the message contents
* a team of engineers not tied to a single enterprise

And, indeed, major network operators like to do stuff like this - takes traffic off the network, and relieves enterprises of evil traffic forms (including DDOS)

BUT then, net neutrality purists, like 4chan, despise this and fight back, as recently when AT&T worked to thwart a large-scale DDOS attack.

Regulate SMTP's and Open Relays

[DJRumpy]

I still don't understand why they don't regulate SMTP servers on the net just like other business areas. These have a real financial impact on other's operating costs. If they required all SMTP servers on the net to be closed and regulated, I think it would be a good start.

I'm talking fines and the ability to cut off any rogue SMTP servers. They also need a better method to validate connecting servers and it needs to be an industry wide adopted standard, whether that is done via certificate authority or some other 'secure IP' method.

Re:Not practical.

[dword]

Actually, here's a good idea: put it in a greylist and let it hit once and see what the user chooses: whitelist or blacklist. And we're back at square one. GP should be modded Insightful because a lost email may be the perfect business opportunity, therefore not all spam must be blocked.

I have a 100% guaranteed way to stop spam from rea

[Dan541]

I have a 100% guaranteed way to stop spam from reaching the mail sever.

Unplug the dam thing!

Re:Not practical.

[Dan541]

That's what happens when companies hire incompetent IT staff.

Re:Not practical.

[WuphonsReach]

It's probably due to how your company's mail filtering is setup.

In an ideal world, you do your filtering during the SMTP session and give either a 4xx or 5xx code to the originating server if you are blocking the message due to filters. This puts the burden of notifying the original sender on the originating server and does not put your SMTP server at the mercy of relying on a (usually) forged return sender address to notify the original sender.

However, a lot of shops do their filtering after accepting a message for local delivery. This causes problems because there's no way of reliably informing the originator of the message that their message was blocked or bounced. Sender addresses are almost always forged in spam runs, and relying on that to notify the sender causes "backscatter" spam. A lot of email anti-virus tools cause this backscatter because they make the naive assumption that the sender address is not forged.

Some systems do a quarantine style setup, where the message is placed in a queue, and the recipient is informed of the sender / subject and allowed to retrieve it.

Unfortunately, filtering during the SMTP transaction (a.k.a. "pre-queue filtering") has its downsides (performance, CPU usage, lack of integration with 3rd party tools). But it's still best to reject as much mail as possible during the SMTP transaction.

A better "THE" fix already exists

[sgt scrub]

Although it is not 100% effective, having a spam filter in front of the email server is the best solution IMHO. Solutions like this let traffic hit the mail server before stopping it as spam. Other than it being annoying to users, the big issue with spam is lots of small connections slowing down the system. Letting a EHLO for each of the spam hits despite filtering it away before completion is not helpful. But then, it might depend on if your an end user that hates getting spam or an admin that hates what spam actually does to your mail server.

Greylisting + SA trumps these rates, moving on

[dageyra]

Are they kidding? 70% and 0.3 % false positives? I employ a simple GreyList which catches 90% of spam and 0 false positives short of a misconfigured sending email server that does not adhere to RFC. Couple this with user-configured Spam Assassin, and my clients see maybe 1 (generally 0) spam email in their inbox a day, with around 10-20 ending up in the spam due to SA. This is down from hundreds in the spam folder and 20-25 in the Inbox before implementing this solution. At least if we're going to pretend something is newsworthy, make it better than what already exists.

Thuggery, anyone?

[sjdude]

Call me old school, but I think the best way to keep spam from getting to the servers would be for there to be a spirited geek vigilante initiative for a couple of years where guys with pocket protectors and baseball bats would show up on the doorsteps of spammers and break their kneecaps. I think there was a Russian spammer who got harsher treatment than this a year or two ago, but I think broken kneecaps would suffice. Just saying...