NYTimes: Tangled Up in Spam

ezekieldas writes "Congratulations to the SpamAssassin developers and community! There's a mention of SA in the NYTMag as "one of the best tools for network administrators..." in an extensive article entitled Tangled Up in Spam. The article is quite substantial and the author, James Gleick, is more technically educated than what we've come to expect from the big press. Central to the story is the complexity in dealing with spam effectively in both technical and legal terms and the confusion it brings upon the neophyte. The conclusion drawn may be oversimplified but nonetheless pragmatic: 1) forged headers should be illegal 2) a specific header entry should identify the email as unsolicited."

Comments

At last

[Mourgos]

now that it has been advertised in NYTmag, more people will become aware that spam is something they can actually stop. Can't wait for the new tricks spammers will use to disable anti-spam programs.

Re:At last

[H310iSe]

Ugh, not spam cops - those guys, I think, have become a little unhinged in their anti-spam hatred and have developed some kind of a demigod complex as a result. I helped run a mailing list generated from submissions to a website - they sent out mailings to people who opted-in for various sex clubs (I know, but sex does not automatically equal spam). We never hid who we were, where we were sending from, we told everyone why they got our mail (because they signed up at the website) and had a valid reply-to address as well as an unsubscribe feature.

Someone sent an email from us to spamcops saying we were spamming - I checked our logs and in one day one person sent us 4 unsubscribe requests - they never got another email but I wonder if it wasn't them. Anyway, we were totally shut down with no warning, two different sites (one hosted the website the other hosting the email program) yanked off the internet when spamcops complained to our ISP.

This is downright stupid. One, anonymous complaint (never did find out who did it so we couldn't very well remove them from our list!) and all our websites, over a dozen, art galleries, political sites, stores, and some 'adult dance club' sites (you do what you can to make clients now...) all went down. No warning. And no apologies from our ISP or spamcops when we pointed out they pulled our service with absolutely NO research, no attempt to contact us, no evidence whatsoever other than a sole complaint which could have been posted by anyone (um, competitors to the adult club jump to mind).

My ISP (Speakeasy) eventually got someone in touch with us who really did nothing more than empathize with how angry we were and promised to try and not do it again. That's it. There's a movement afoot to try and reign in this sort of insane overkill, one story here and an a nascent organization against overzealous antispammers is here.

For the record, we did not have confirmation on our opt-in list so theoretically someone could have signed up another (say a priest or something) for our mailing lists. We never got more than a couple new registrations a day so there was no systematic abuse, still, we fixed this and added confirmation (using mailermailer.com, I'm very impressed with them so far) after the complaint (no need to knock us off the web to get our attention, a simple email would have done) and, as I said, we had valid contact info if they had only bothered to ask...

Anyone else been a 'victim' of crazy blacklist providers?

Illegal?

[waytoomuchcoffee]

The conclusion drawn may be oversimplified but nonetheless pragmatic: 1) forged headers should be illegal 2) a specific header entry should identify the email as unsolicited

Why does everyone in the USA assume that everyone else in the world will somehow obey US law when it is made "illegal"?

Re:Illegal?

[jjo]

People don't assume this. What they do assume is that, by and large, people who try to get money from US residents are actually situated in the USA, regardless of where the e-email might have originated. Even those who are not in the USA will mostly use a US agency to get their money. That is their Achilles heel: Follow The Money.

Stop the flow of money from US residents, and you will be effectively making everyone in the world obey US law, with respect to spamming within and into the USA.

MIT's Post Servers...

[g_arumilli]

now use SpamAssassin. Basically, a set of new headers is attached to the e-mail of the form X-Spam-foo, and if X-Spam-Score is 7.5 or greater (on a scale of 10 I believe), then X-Spam-Flag is yes. It's really useful for sorting out spam quickly, and I haven't gotten a false positive yet...It doesn't get all of the spam, but it gets the vast majority of it...

Always with the legislation...

[Sheetrock]

Spam is a technical problem, so why can't we come up with a technical solution? For example, it should be impossible to forge headers, not illegal. Why rely on a legal solution from many of the people who have brought us such brilliant solutions as the DMCA and the CDA in the past when all that's required is what our community has always been good at: sitting down and thinking things out?

Re:Always with the legislation...

[TGK]

I'd say the best technical solution I've seen to breaking the SPAM system is the use of the internets distributed nature against the spammer.

Consider the following. We all access the internet from a fixed and typicaly small number of physical and virtual locations. Were we to map the internet as a whole, starting from any given location the map would look like an expanding cone.

In short, almost all of the traffic from a given point flows through a very small number of servers and routers at some point close to the source.

Since spam messages are sent by the millions and it is fairly easy to determine what messages are likely to BE spam why not set up a filtering system on the routers that determines the rough content of a message based on both its Spam Precentage and the number of identical messages sent.

I.E. If the router sees 500,000 messages of nearly identical content with a 89% spam rating it blocks all of them. If it sees 44 messages with a 23% spam content it lets them through.

Thoughts anyone? I'm sure this idea has gaping flaws in it... what would have to be chnaged for it to work? What are the critical flaws? Is this a viable model or am I missing something major?

Interesting free speech point

[jenkin sear]

Towards the end of the article, Gleick makes a really interesting point- he says that as commercial speech, spam isn't entitled to any particular first amendment protection:

The Supreme Court has made clear that individuals may preserve a threshold of privacy. ''Nothing in the Constitution compels us to listen to or view any unwanted communication, whatever its merit,'' wrote Chief Justice Warren Burger in a 1970 decision. ''We therefore categorically reject the argument that a vendor has a right under the Constitution or otherwise to send unwanted material into the home of another.''

Looks like we have the supremes on our side; if we could just congress to issue some letters of marque and reprisal on the spamhausen, we'd be getting somewhere...

Techical Solutions Are Required

[esme]

As much as I'd like to see spammers prosecuted for fraud (and think making various deceptive tactics illegal is a good short-term approach), legal and social approaches are doomed to failure. The number of people you can spam is so vast, that even if only one in a million takes the bait, it's still profitable -- that's a powerful economic imbalance that you don't find anywhere else. And it's going to make people forge headers, spam from overseas, etc. to get around any legal and social roadblocks.

I think that breaking that economic model -- ending the reciever-pays system for email -- is the only way to fix spam. If you had to pay some amount of money -- event 1 cent -- for each message that is delivered, spam would stop being economical. And that's the only thing that's going to make it stop.

-Esme

Re:Techical Solutions Are Required

[rthille]

There's no reason to involve money (dollars) to stop spam, make them spend CPU cycles instead. Take a look on google for 'hashcash'. Basically, it involves the sender computing a function that takes a long time to figure out, but is very easy for the receiver to verify. So, if i want to send you mail, I spend ~10 cpu seconds, and you verify that I spent the time, and you accept the mail. If I don't compute the function, you sideline/reject the mail. Whitelists can be used to prevent always needing to compute the function. That way I can accept mail from anyone who might be willing to send me mail, if they are willing to spend the CPU cycles. However, since spammers would need to spend 10 seconds per message, they could only send about 1000 messages per day. That wouldn't be economically viable for them...

The real way to get rid of spam

[KevinIsOwn]

Sure all these programs help, but think about what creates spam in the first place.

There are clearly people out there willing to buy the things offered in spam. Obviously not that many, but enough to make a profit. I think that there should be more of an effort to target these people and tell them not to buy stuff from spam!

There is only so much a program can do to stop spam. As we've seen numerous programs have been made, Spam Assasin being one of the best (I use it), but the spam just keeps coming

Until there is no incentive to send spam in the first place people will do it despite any laws against it.

Re:SpamAssasin in large corporate use?

[Webratta]

I don't work for a large corporation, but a state-wide ISP. I asked my boss, the chief technical officer of the company, why we weren't using Spam Assassin. He replied that while it is a very neat program and does a great job of filtering spam, the performance just isn't quite there yet. He's of the mindset that it needs some tweaking still before it can be a competitor to commercial products like what Brightmail offers.

Personally, I'd like to see more companies using SpamAssassin just to prove that it can stack up against other products, because I think it can work well if it's configured properly and you use spamd. I use it on my mail server at home and at last check it catches 98.2% of all spam message sent to my machine, and I haven't had any false positives since I set up my whitelists.

Another cool anti-spam tool

[yiingineer]

I've been using Cloudmark's SpamNet for the past few months and it's been working quite well.

The smart thing that SpamNet does, is that it relies on its users to determine if something is spam or not. If some email lands in your inbox and a few hundred SpamNet members have proclaimed it spam, it most likely is, and it gets immediatly filtered out. This has the net effect of a few user's needing to filter out a few message ocassionally, while the vast majority of messages are filtered out for all users. Although SpamAssassin seems quite good, it's still based upon filtering rules and spammers are constantly tweaking their emails to try to get around them. Since people are still better at determining what's spam and what's not, I find that its accuracy is generally better.

SpamNet isn't perfect though, as far as I know, it only works with Outlook on Windows and doesn't have a Unix, Linux or Mac version. It also sometimes filters out valid bulk mailings, but overall, I would definitely recommend it.

Re:SpamAssasin in large corporate use?

[winnetou]

I was wondering how many large corporation are using SpamAssasin. And if not, why not?

Reasons for not using SpamAssassin are the CPU and bandwidth costs. Refusing e-mail from known spam sources is cheaper and (more importantly) does not give away information about which addresses are valid.

After checking the source IP address against lists such as Wirehub, Osirusoft (despite its name not only a list of open relays) and/or some other lists, almost no spam will be accepted.

IP space is finite and, even better, allocated in ranges. Continued spam from (or spamvertizing a website on) an IP address is a very good indicator for more spam from the IP range.

Re:Kudos to SA.

[MeanMF]

But how do people get on the spam-lists to begin with? I mean, I have one email address for work and one private. Neither one of these gets more than one spam/month. Ever.

If you email address is simple (e.g. first initial+last name+some number) and your domain name is that of a public ISP, then there's an excellent chance that the spammers will find you regardless of whether or not you ever use the address. Email addresses at work tend to be safer because spammers usually don't bother guessing at addresses in domains with so few valid mailboxes.

I rarely ever get spam.

[cpaluc]

Heres how:
1. Spend 10 bucks, buy a domain name (eg xyz.com).
2. Set up a few email aliases to point to your real email. eg:

joe@xyz.com ---> you@hotmail.com

temp123@xyz.com ---> you@hotmail.com

spam123@xyz.com ---> you@hotmail.com 3. Never give out 'joe@xyz.com' to anyone except friends/family.
4. Use the other emails for signing up for things on the web or in usenet.
5. When you get your first spam addressed to 'temporary21@xyz.com', delete the email address (no more spam from that source!).

I find this method works extremely well. By using aliases in this way you effectively hide your real mailbox. Even if your hotmail account starts receiving spam you can just get a new one and point your aliases at it. Also, if you change ISP you don't need to change your email address.

If you use it to forward to a hotmail account it might be better if the hotmail account name isn't a dictionary word or name (ie. use a random string for an account name that the 'bots won't guess.

You're screwed if your 'trusted' address gets out there but if you're careful you'll at least get much more use out of it before needing to kill it.

Re:I rarely ever get spam.

[LocalH79]

Spamgourmet does exactly what you propose, and is much more effective.

Internet mail architecture sucks

[cdegroot]

Change to something like IM2000 (http://cr.yp.to/im2000.html), spam vanishes in a poof. Keep around with the current broken system, and we'll have ever more draconian laws in ever more futile attempts to suppress it.

Re:Kudos to SA.

[domninus.DDR]

Ive tested something similar to this. Make a hotmail account with jibberish (rand(), 8 char isalnum() strings is what I used) for the name and see how long it takes to get spam. Out of ten tries my average was about 3 days.

Bad idea

[Goonie]

This is near-impossible, technically. By the time the traffic flows through the "core routers", it's just a bunch of IP packets which the system doesn't even try to interpret at a higher level. Reconstructing the messages, running spamassassin on them, and selectively blocking them would put an insane CPU load on the routers. They would effectively be acting as mail relays, not routers.

There are also philosophical problems with such a scheme which others can explain...

A new breed of email is on the horizon

[mcrbids]

If we can pull it off.

With Bind 9, we finally have a decent, working implementation of DNSSEC. This will allow for a new breed of secure, verified websites and email, and (Finally!) makes a RBL actually mean something.

How's that you ask?

Well, one of the biggest problems with SPAM is the forged header, open relay issue. It's a complicated issue, and one that doesn't have an obvious, "in your face" kind of answer.

DNS is designed to tell you where to go, and SSL/Certs make sure that you got there. Why aren't they joined together? The fact that you are the DNS server for a domain makes it clear and obvious that you are an authoritative designator for where you are supposed to go - why have this wholy separate and dis-jointed SSL/Cert that can't even be made to work consistently?

If an ISP can issue DNS-SEC certs with impunity, we might actually see a reason to have encrypted and ISP certified email.

And suddenly, the ISP is back in charge again, able to validate every email going out as coming from one of it's customers. Revoke the cert and their email becomes unreadable.

Now, we have an email system with a powerful mechanism built in that is:

1) Standards compliant
2) Easy to implement
3) Clearly laid out
4) Cheap
5) secure
6) private - using the ISP's cert to identify yourself doesn't mean that the ISP can read your email! (like they can now - the command is "mail -u _username_")

What's not to argue with? The issue of locking down an open relay becomes a non-issue - an ISP could simply identify an "s-mail" server (secure mail) that will only relay for those holding a valid cert at that ISP.

Roaming wouldn't be an issue, nor would open relays or forged headers.

A brave new world? Yep. One I'd like to live in? Yep. One that's coming? We can only hope...

Where spam really comes from

[Cbs228]

Spam isn't a legal problem-- it's a social problem. It is the result uncontrolled avarice, of people wanting to make money at any ethical cost. There will always be these kinds of people who will steal our time (and our bandwidth) regardless of any laws against them. There are also people (Sysadmins of certain Far East networks come to mind) who are willing to look the other way for a few extra dollars.

But most importantly of all, we cannot forget that American consumers are responsible for spam. That's right, spam is OUR fault. It is our fault because no matter how many messages are filtered, and no matter how many websites are closed for spam complaints (or get DDoS'd by rampaging slashdotters), they still make money. They make money because of that infinitesimally small group of consumers who buy stuff from spammers. That small percent is what makes it all worth it to them.

The day that spammers' profit margins drop to nil because consumers refuse to buy from spammers is the day that spam vanishes from our inboxes forever. No laws, no filters, no problems.

Unfortunately, as P.T. Barnum would put it, "There's a sucker born every minute..."