Slashdot Mirror
NYTimes: Tangled Up in Spam
ezekieldas writes "Congratulations to the SpamAssassin developers and community! There's a mention of SA in the NYTMag as "one of the best tools for network administrators..." in an extensive article entitled
Tangled Up in Spam.
The article is quite substantial and the author, James Gleick, is more technically educated than what we've come to expect from the big press. Central to the story is the complexity in dealing with spam effectively in both technical and legal terms and the confusion it brings upon the neophyte. The conclusion drawn may be oversimplified but nonetheless pragmatic: 1) forged headers should be illegal 2) a specific header entry should identify the email as unsolicited."
I been using Spam assassin for a while now, it is sad to say, but email would be almost unusable with out it.
now that it has been advertised in NYTmag, more people will become aware that spam is something they can actually stop. Can't wait for the new tricks spammers will use to disable anti-spam programs.
By simply filtering out all e-mails that have the word "Nigeria" in them.
Work sucked, until it became unemployment, when it became slightly more tolerable. -Tet
>>> 2) a specific header entry should identify the email as unsolicited." NO NO NO There is no excuse for sending spam. I fail to see how marking it as junk makes it any better. So I can sort it from the mail I actually want? NO. Just stop people sending me crap I don't want.
Sig is taking a break!
I was wondering how many large corporation are using SpamAssasin. And if not, why not?
Consensus is good, but informed dictatorship is better
The conclusion drawn may be oversimplified but nonetheless pragmatic: 1) forged headers should be illegal 2) a specific header entry should identify the email as unsolicited
Why does everyone in the USA assume that everyone else in the world will somehow obey US law when it is made "illegal"?
So how much spam am I likely to get if I give in and register with NYTimes so I can read the article?
now use SpamAssassin. Basically, a set of new headers is attached to the e-mail of the form X-Spam-foo, and if X-Spam-Score is 7.5 or greater (on a scale of 10 I believe), then X-Spam-Flag is yes. It's really useful for sorting out spam quickly, and I haven't gotten a false positive yet...It doesn't get all of the spam, but it gets the vast majority of it...
Spam is a technical problem, so why can't we come up with a technical solution? For example, it should be impossible to forge headers, not illegal. Why rely on a legal solution from many of the people who have brought us such brilliant solutions as the DMCA and the CDA in the past when all that's required is what our community has always been good at: sitting down and thinking things out?
Try not. Do or do not, there is no try.
-- Dr. Spock, stardate 2822-3.
Filter any e-mails containign the phrase, "this is not an unsolicited message".
"Sic Semper Tyrannosaurus Rex."
Looks like we have the supremes on our side; if we could just congress to issue some letters of marque and reprisal on the spamhausen, we'd be getting somewhere...
What a strange bird is the pelican, his beak can hold more than his belly can.
I think that breaking that economic model -- ending the reciever-pays system for email -- is the only way to fix spam. If you had to pay some amount of money -- event 1 cent -- for each message that is delivered, spam would stop being economical. And that's the only thing that's going to make it stop.
-Esme
Sure all these programs help, but think about what creates spam in the first place.
There are clearly people out there willing to buy the things offered in spam. Obviously not that many, but enough to make a profit. I think that there should be more of an effort to target these people and tell them not to buy stuff from spam!
There is only so much a program can do to stop spam. As we've seen numerous programs have been made, Spam Assasin being one of the best (I use it), but the spam just keeps coming
Until there is no incentive to send spam in the first place people will do it despite any laws against it.
The one big feature missing for me in evolution is a spam filter. Fortunately, spamassassin works great even if you have to run it locally. Here are some instructions for evolution users who need to run it locally or are lucky enough to have spamassassin installed on their mail server.
Be careful what you outlaw. If the law is too broad, it could easily be used to prohibit not only headers in email messages, but in connecting to a web server. How would you like to have it be illegal to lie about what browser you're using? Or refuse to send a referer?
The conclusion drawn may be oversimplified but nonetheless pragmatic: 1) forged headers should be illegal 2) a specific header entry should identify the email as unsolicited.
I don't know what is meant by unsolicited -- and I doubt that there are good definitions that are practical. Nor do I want any single e-mail ever to be treated as spam because some unsophisticate forgot to (or didn't have the software) to make the e-mail unsolicited.
I *DO* want the anti-spam laws to have teeth and very few exceptions -- for that, the criteria for spam should be sufficient to permit adequate filtering (to be useful), not be content-based (to be constitutional), and should be relatively objective (to be practically enforeceable).
Thus, in lieu of forcing headers to identify whether an e-mail is solicited, i would punish falsely identifying an e-mail as non-broadcast. That is to say, an e-mail is not broadcast if it was sent to, say, fewer than 200 different addresses that had not specifically opted-in by affirmative request to receive it.*
Then, we simply get most e-mails clients to flag routine e-mails as non-broadcast, and you have a decent result.
*the only tricks here are (1) subtle and non-substantive changes in each e-mail making them different and (2) sending e-mails on behalf of many different sources (from 1000's of different e-mail accounts). The solutions can be readily addressed by (1) referring to the e-mail and "substantially similar" e-mails (the copyright standard); and (2) referring to e-mails sent by or on behalf of a particular individual. Thus, the person commissioning the spam is always liable for the crime -- regardless how many different persons send the spam on her behalf.
when people say SpamAssassin is good - they should really be talking about 2.5
that is the version with the Bayes fully in it and it is head and shoulders above the previous versions IMO
There are some odd things afoot now, in the Villa Straylight.
Or you can register normally and help the NYT pay James Gleick's salary as well as their bandwidth bill, by allowing the NYT to get a better grasp of who their readers are.
But this Slashdot, where information wants to be free unless it's your own.
The uneducated guy that send this story in, need to know that was instrumental in taking Chaos theory from an obscure science in Santa Fe into something that almost every scientific discipline benefits from. Incl CS. .
Help fight continental drift.
illegal is great in theory, but there is no possible way to enforce that on a world wide basis.
It's impossible to enforce almost any laws with 100% effectiveness, but that does mean that we should ignore the problem. If some sleazeball in Florida hires a firm in Korea to spam me, put his ass in jail.
white lists are the only way to stop spam.
I'm amazed by this user-hostile suggestion every time I hear it. Suppose you post your resumé on Monster.com. Who are you going to whitelist? Suppose your friend changes ISPs and then tries to e-mail you his new address? It won't be whitelisted, so it will bounce. Suppose to fill out a tech support request form. You don't know the address of the person that will contact you (or even if they will be the same domain as the web site).
I've been using Cloudmark's SpamNet for the past few months and it's been working quite well.
The smart thing that SpamNet does, is that it relies on its users to determine if something is spam or not. If some email lands in your inbox and a few hundred SpamNet members have proclaimed it spam, it most likely is, and it gets immediatly filtered out. This has the net effect of a few user's needing to filter out a few message ocassionally, while the vast majority of messages are filtered out for all users. Although SpamAssassin seems quite good, it's still based upon filtering rules and spammers are constantly tweaking their emails to try to get around them. Since people are still better at determining what's spam and what's not, I find that its accuracy is generally better.
SpamNet isn't perfect though, as far as I know, it only works with Outlook on Windows and doesn't have a Unix, Linux or Mac version. It also sometimes filters out valid bulk mailings, but overall, I would definitely recommend it.
What's the use of having an email address if you don't give it out to any of your friends? It's like asking a hot date to call you, but you won't give her your unlisted telephone number.
boldly going forward, 'cause we can't find reverse
I should be able to ask Hotmail (or whoever) "I have message #xyz from your domain. Does it originate from a user in good standing?" If the ISP gets too many queries for an individual account, it will stop vouching for it.
Likewise, you need a database of "ISP's in good standing". I.e., who is known to play by the rules with MSSMTP?
Verification would serious server resources, but better that than spam.
-mse
Who steals my .sig, steals trash.
Fiat Lux.
SpamAssassin's a great idea, but for the non-technically minded user, POPFile's the best choice. Bayesian filters, learning, kickass UI, and a Windows installer (and Perl for other platforms.)
>>1) forged headers should be illegal 2) a specific header entry should identify the email as unsolicited
Don't we ever learn from the past? We've all seen the unintended consequences of poorly-crafted legislation (e.g. DMCA), so why run to the shelter of more restrictions which, in the end, will only cause us more problems? Like the criminals trying to scam your mom with the Nigerian-hold-my-money-for-a-day scam are going to suddenly begin obeying the law... yeah, right. Which begs another question: what law, in what jurisdiction? Even if the US were to pass this law and ruthlessly enforce it (domestically), all scammers would simple flood us from offshore servers.
The solution is not legislation, it is the creative use of technology. Build software that "learns" what is spam and what isn't, then evolves to keep up with the changing tactics of the spammers. Something like PopFile
The only headers that should be preserved are perhaps the Received: lines which show that route that the message has taken. Still, I can think of a legitimate reason to muck with these - if a company network has a sufficiently complicated internal structure, these headers might reveal some information that they don't want widely available.
Go figure.
Is this thing on? Hello?
1. Spend 10 bucks, buy a domain name (eg xyz.com).
2. Set up a few email aliases to point to your real email. eg:
joe@xyz.com ---> you@hotmail.com
temp123@xyz.com ---> you@hotmail.com
spam123@xyz.com ---> you@hotmail.com 3. Never give out 'joe@xyz.com' to anyone except friends/family.
4. Use the other emails for signing up for things on the web or in usenet.
5. When you get your first spam addressed to 'temporary21@xyz.com', delete the email address (no more spam from that source!).
I find this method works extremely well. By using aliases in this way you effectively hide your real mailbox. Even if your hotmail account starts receiving spam you can just get a new one and point your aliases at it. Also, if you change ISP you don't need to change your email address.
If you use it to forward to a hotmail account it might be better if the hotmail account name isn't a dictionary word or name (ie. use a random string for an account name that the 'bots won't guess.
You're screwed if your 'trusted' address gets out there but if you're careful you'll at least get much more use out of it before needing to kill it.
The most important Q, if gov't help is going to mean anything.
Enforcement is currently a state problem, for the dozen or so states that have antispam laws. Even if they can establish jurisdiction, they have to locate the offender. An asst. attorney general I chatted with in Washington state described an almost comic crusade to get ONE spammer who set up under a different corporate name every week. They used three private investigators to track him (successfully), suggesting to me their investigatory resources were limited. Anyway, they couldn't afford to do this with everyone, and this one example was located in-state!
I was surprised the author didn't really talk about state laws at all. They're kind of the laboratories for the eventual federal effort, and state law/enforcement will be complementary.
Once there is a law on the books the "cyber" aspect of it is only as issue for tracking. Postal mail and telephone calls have "no physical boundaries," too, and actually it is the crossing of state lines taht is an obvious source of federal jurisdiction. The rest is standard law enforcement. The FTC, which the author briefly visited, was busy enough with outright fraud, where it already has jurisdiction, just as it does over fraudulent TV ads and newspaper ads and product labeling and so on. I can say that I've seen some very good work by the FTC, even leading to jail terms for the guys who just won't give up. (The jail term I saw was for criminal contempt of court.)
I think they're going to need to provide a private enforcement action, as with the fax law. The gov't resources would still be needed to track down and prosecute the really tough ones, such as the WA case I described. We already have some relevant experience from the anti-junk fax law.
Recognizing spam -- good Q. I don't have any trouble recognizing 99% of it. For teh false positives, it should be possibly to allow the merchant to provide evidence of opt-in, and if enough complaints are tallied there would be further action.
The big problem I have now, new in the last two months or so, is that many of the spams are now uuencoded text bodies... so the filters don't work on them. They are reconstituted by the client (Eudora in my case), after passing through the filters.
Unfortunately the filters (e.g. Spam Weasel, Eudora,etc.) don't have an "automatically reject if no text components" option.
Change to something like IM2000 (http://cr.yp.to/im2000.html), spam vanishes in a poof. Keep around with the current broken system, and we'll have ever more draconian laws in ever more futile attempts to suppress it.
If you just want a fake email address that is "valid", use whatever@example.com
example.com is an official internet blackhole, sanctioned by RFC. It is what everyone is supposed to use in books, demonstrations, etc, similar to 555-XXXX phone numbers on TV.
Better strategy.... But requires having control of your own mail server...
.forward-amazon and have it put mail in /dev/null. Alternatively you could use procmail or maildrop in the dot-forward file to perform per-extension filtering or bounce messages to explain why the mail will never be read, in case legitimate mail tries to come into that box, perhaps with a random, unique extension provided for them to try a legitimate box. Not only do you have an effective mechanism for filtering out unwanted mail by source and outdated email, you also have a way to track how your email gets out. It has worked quite well. Last week I got three spams, and blocked that address. Aside from that and a couple of other incidents in the past year (about 8 or 9 spam mails total), the signal to noise ratio in that mailbox is excellent.
I run my own mail server. I have Postfix configured to forward username-@the.server to username@the.server by default. So, for example, I registered with amazon username-amazon, and it gets to me. If this email is ever put on a list, I'll complain to amazon, and then create a
XML is like violence. If it doesn't solve the problem, use more.
There are also philosophical problems with such a scheme which others can explain...
Any sufficiently advanced technology is indistinguishable from a rigged demo
--Andy Finkel (J. Klass?)
For what it's worth, an ever-so-slightly longer version, lacking a few bits of Times editing, is posted here, at my own site. And may I say how helpful and fascinating the many Slashdot discussions of this subject have been?
If we can pull it off.
With Bind 9, we finally have a decent, working implementation of DNSSEC. This will allow for a new breed of secure, verified websites and email, and (Finally!) makes a RBL actually mean something.
How's that you ask?
Well, one of the biggest problems with SPAM is the forged header, open relay issue. It's a complicated issue, and one that doesn't have an obvious, "in your face" kind of answer.
DNS is designed to tell you where to go, and SSL/Certs make sure that you got there. Why aren't they joined together? The fact that you are the DNS server for a domain makes it clear and obvious that you are an authoritative designator for where you are supposed to go - why have this wholy separate and dis-jointed SSL/Cert that can't even be made to work consistently?
If an ISP can issue DNS-SEC certs with impunity, we might actually see a reason to have encrypted and ISP certified email.
And suddenly, the ISP is back in charge again, able to validate every email going out as coming from one of it's customers. Revoke the cert and their email becomes unreadable.
Now, we have an email system with a powerful mechanism built in that is:
1) Standards compliant
2) Easy to implement
3) Clearly laid out
4) Cheap
5) secure
6) private - using the ISP's cert to identify yourself doesn't mean that the ISP can read your email! (like they can now - the command is "mail -u _username_")
What's not to argue with? The issue of locking down an open relay becomes a non-issue - an ISP could simply identify an "s-mail" server (secure mail) that will only relay for those holding a valid cert at that ISP.
Roaming wouldn't be an issue, nor would open relays or forged headers.
A brave new world? Yep. One I'd like to live in? Yep. One that's coming? We can only hope...
I have no problem with your religion until you decide it's reason to deprive others of the truth.
1) use a "throw-away" email address when including them in your resume.
/.ers were nerds and knew how to write programs.
Most people can't even deal with a single address.
2) develop a more friendly "white list" system that makes it easy for you to "open it up" for your potentual employers. So when I send mail out to someone important, I'm just one click away from adding them to my "white list".
Listen Miss Cleo, you have no way of knowing who will respond to your resumé. It might be a company that you send it to. It might be someone at that company working from home. It might be someone at another division that you did not know about. If your resumé was posted on a web site, it might be anyone responding.
Come on guys, I thought
My mail server and e-mail processing software implement filtering that would probably make your head spin. Despite having dozens of e-mail accounts and three different domains, I probably see less the one percent of the spam that's sent to my domains. I have autoresponders for retired addresses, auto-complaints for mail from Brazil (to mail-abuse@nic.br), and I use multiple blacklists. Some of my e-mail addresses accept blind copies from untrusted senders and some do not.
But the spam problem needs to be solved for everyone, not just computer geeks that hang out on Slashdot. When the risk of fines and jail time make it unattractive, then we will have really solved the problem.
If you use MS Outlook (we are forced to at work), try out Spammunition. It's a free Bayesian spam filter that's integrated right into Outlook. Works really well. No spam problems any more. This bayesian approach really works.
But most importantly of all, we cannot forget that American consumers are responsible for spam. That's right, spam is OUR fault. It is our fault because no matter how many messages are filtered, and no matter how many websites are closed for spam complaints (or get DDoS'd by rampaging slashdotters), they still make money. They make money because of that infinitesimally small group of consumers who buy stuff from spammers. That small percent is what makes it all worth it to them.
The day that spammers' profit margins drop to nil because consumers refuse to buy from spammers is the day that spam vanishes from our inboxes forever. No laws, no filters, no problems.
Unfortunately, as P.T. Barnum would put it, "There's a sucker born every minute..."
At our school, we don't earn a degree when we graduate—we earn pi/180 radians
There are many perfectly reasonable reasons why you would want to provide an alternative to the default value for many SMTP headers. It's when you lie and mislead by using values that *other* ISP's use in their own headers that you are said to have "forged" them. Bogus "Received" headers can be considered "forged headers" as well, as they are not added by the MTA per the SMTP specification, they are crafted by hand to make it *look* like they were added by an MTA.
These are forgeries. Providing alternative (but still "correct") values for some SMTP headers are not.
(Technically, instead of mucking with the From header, you might want to consider adding a Reply-To and/or Errors-To header instead.)
Hi John,
...
I got this from my friend who works at the mall - check this girl, she's hot!
Spam is not a technical problem.
It is generated by the most complex processing system known (The Human brain) and obeys to one of the simplest known principle (or absence thereof: greed).
That's a pretty potent combination.
Certainly not one for a machine to match.
No AI based solution will ever be able to reliably block spam, it's like handwriting recognition: I can't even read my own handwriting sometimes!
Spam is a human problem that has two sides:
- Some nutters will stop at nothing to sell you something (expecially if the numbers look good).
- Some idiots will genuinely think a girl called Sangria has the hots for them - type in your credit card here darling.
Don't worry: if you've read that far, then you're probably not that dumb.
Of course the solution is legal.
Here in the UK, I used to receive a fair amount of junk mail. There is however an opt-out list which I subscribed to and all I get is a few of them a year for the guy who used to live here before me.
So, yes, forged headers should be illegal.
And no, an 'Unsollicited mail' one is not a solution:
Why?
Because of this:
"Hi Tee, I am your long lost cousin in Australia - I found your e-mail on your web page, So good to be in touch again..."
A header that says whether or not the email is advertising is a better idea. If the values of this field follow an agreed classification, you could actually filter IN *voluntarily* things you are genuinely interested in.
The inforcement problem about spam will eventually be resolved. Europe is getting bigger and more integrated, the USA are a big chunk too. Now if these two and, say Japan or Taiwan agreed to block any other network that does not adhere to the guidelines, there will be a lot of pressure from inside those banned countries to make them adopt compatible legislation.
Of course it takes guts (something politicians rarely have), technical awareness (ditto) and time (Well fortunately we have plenty of that - it's only our patience that's running out.)
Check this site it's hot: http://www.aptilis.com/
(Sorry couldn't help...)
Teebo.
"the author, James Gleick, is more technically educated than what we've come to expect from the big press."
Maybe because after many years as a reporter, he founded Pipeline, one of the first big ISPs.
I think it would be great if you could actually prosecute someone for forging headers. Unfortunately you don't know who that person is, now do you?
But how would you ever determine is something is unsolicited? After all, there are a lot of registration websites that have a tendency to quietly flag you as willing to accept spam from them. If I missed it, does that still make it UCE? If it does, how do I now remove myself from all the lists that I am now on...
Spam has a solution and it doesn't have to be so drastic as to put in this kind of legislation or use whitelist only maling lists. We just haven't figured it out yet.
I think that the router should not use this information to shut anybody off. Rather, it should use this information to reorder its routing priority tables. Thus the router will serve its most spam-free peers first, handling the heavy spam forwarders only when it has time. Eventually consumers will leave ISPs with poor throughput, so ISPs will have a much stronger incentive to track down and terminate their members who spam.
So what was the e-mail with a score of 27?
"Hello, I am a Nigerian prince who is selling XXX-brand diet pills that also have the side effect of enlarging your penis. Also if you forward this email to five other people and tell them to each send you a dollar you can make money fast."
*ducks*
Check out where Gleick quotes Feynman on the inherent risk of Shuttle flights. Prescient, that Feynman.
Spam is not about content. Not everyone even agrees what constitutes spam when they are evaluating it based on content, so how can a program or a recipient community do this? What makes mail spam is stuff like sending it unsolicited and in bulk. It won't matter what the content is.
I have signed up with some companies for announcements about their products. While that company may not be spamming, their content could have a lot of the same wording as another company selling similar products, but is sending it to harvested addresses. The latter is spam, but the former is not. How do you tell based on the content?
Tools that evaluate a message based on content are probably going to classify both messages the same way. If they are both classified as spam, then one of them will be "collateral damage". If they are both not classified as spam, then the other will be "leaky pinky". So I still prefer to block spam on the basis of the behaviour of the sender.
now we need to go OSS in diesel cars
OK, don't shoot them, but maybe conduct a poll. Find out why they are stupid enough to purchase anything offered through an unsolicited commercial e-mail. Find out if they actually believe that anything purchased through an e-mail will increase their penis/breast size, allow them to lose a ridiculous amount of weight, make an impossible amount of money or get the best mortgage rate around.
And then shoot them. A lot.
Please don't humanize the morons around me. It makes me very uncomfortable.
Sounds great... but don't you think the spammers might catch on eventually and just send to:
...
username-amazo@the.server
username-amaz@the.server
username-ama@the.server
u@the.server
figuring that somewhere in there they'll hit the real address? (And they'll figure it out even quicker once they notice they have both username-amazon@the.server and username-yahooGroups@the.server in their mail-lists)
Any technological solution (widely employed) will eventually be caught up to by the spammers, perpetuating the SPAM arms race, and bringing us down to their level (as the article alludes to).