Slashdot Mirror
Spam Catchers Block Latest Crypto-Gram
An anonymous reader writes "Bruce Schneier sent out a note about SpamAssassin and possibly other spam filters blocking his excellent Crypto-Gram newsletter. Fortunately you can get it here (early no less!)." Schneier's email reads, in part "Tomorrow I will be sending out the February CRYPTO-GRAM, as I do on the 15th of every month. In the process of creating this month's Crypto-Gram, I discovered that SpamAssassin thinks that this issue is spam, probably because of certain links and descriptions of scams in the text. I have anecdotal evidence that other spam filters block Crypto-Gram as well. ... I'd apologize for the inconvenience, but I'm not sure what I could do to make it less so -- I don't intend to alter my content to accommodate spam filters."
block that important e-mail I was waiting for on enlarging my....never mind, I have to check my e-mail now.
but why not distro the newsletter encrypted? then the spam filters wouldnt have anything to trigger the filters, and id say the target audience have the knowledge to unencrypt it when it gets there..
So he sends out the Crypto-Gram newsletter, then he sends out a note about the Crypto-Gram newsletter. 2 emails to cover what should've been sent as 1. Seems like the spam filter is doing just fine ...
That's easy to fix, add the crytogram address to a whitelist. Every spam
filtering software I've ever run, including spamassasin (which I like a great
deal) has a whitelist option. If you're running some kind of filtering
software, it behooves you to keep an eye on what it's blocking, hence, I am
sure that people are aware of it and have adjusted their software accordingly.
SealBeater
-- Its survival of the fittest...and we got the fucking guns!!!
That's why most good spam blockers (especially OS X's Mail.app) use their filters but compare the senders to a whitelist so that your friends can send you whatever they want to. If you've been receiving CRYPTO-GRAM for a while, it should be on your whitelist, and the blocker should just let it by.
But you don't always want to get everything people send you (everybody has those people who send you things they think are funny but you just can't stand). So there should be levels of "friendship" in the whitelist, so that some senders can be considered dubious (their mail shouldn't be deleted like spam, but perhaps placed in a different "Uninteresting" folder).
Lack of eloquence does not denote lack of intelligence, though they often coincide.
SpamAssassinAssassin could look at the folder where you put your filtered mail and learn what to pull back out, and flush the rest to /dev/null.
I'm sure Paul Graham will be glad to write it in lisp.
Or, of course, we could just do what the obvious solution is: get in a P.O. Box, send out spam for herbal viagra and penis enlargement, and when you get the checks in the mail HUNT THE CUSTOMERS DOWN AND KILL THEM.
It's simple, really.
False-Positives should be a non-issue. Either you choose to run a spam filtering software and live with thoose limitations or don't run a spam filtering program and deal with the extra emails about enlarging various organs that you will receieve every day.
I do tech support for a webhosting company and people call us every day complaining about their spam but as soon as we offer blocking software based on lists, etc all we get is complaints that some more-valuable-than-gold email is going to get lost and ruin their entire business.
This is a simple choice and people have to learn they can't have their cake and eat it too.
If religous zealots don't believe in Evolution, then why are they so worried about bird flu?
obPlug: This is why I created Trustic.
This is exactly the problem with most content filtering approaches.
It is very hard to discern the difference between talk about sex, spam, viruses, etc and talk from sex, spam, viruses, etc. Newsletter authors go as far as writing "v*rus" and "sl*mmer" so that pitiful content filtering blocks don't trash them.
It gets even worse for email lists that use inline text ads. The ads alone would constitute spam, but they're nestled within several paragraphs of high-quality discussion.
The problem is that content filtering approaches usually only analyze the "spamminess" of a piece. They usually don't analyze the "goodness" of a piece. So if I put "hot teens go crazy for debt-free viagra while earning $$$ from home" in the middle of some fine Shakespeare, that will get flagged as spam.
The new "bayesian" approaches are finally dealing with this problem -- something can look an awful lot like spam, but it will be saved if it looks even more like legitimate email.
In this case, spam doesn't generally run for 21 pages with words like "cryptography," and "full disclosure."
It all goes downhill from first post
At least he is only on Spamassassin which tends to be run on the client-side, so statistically less people would not see the newsletter. If he were on the SPEWS's blocklist, he'd never get out!
http://www.antispews.org/ the SPEWS fansite (not!)
Personally I see less problem with client-side blocking, as there is less chance that any 2 people would use exactly the same combination of blocklisting/priorities/etc. Plus, programs like Spamassassin use quite a lot of processing power, so large mail servers (eg. for an ISP) would need significant additional resources to handle this. Thus it is best to move such individualized and resource-intensive applications to the client-side anyway.
YMMV.
**FREE** Track and view your phone's via CellID and/or WIFI and/or GPS
Sure. Assuming Schneier has the public keys of all his subscribers, AND the processing power to encrypt everything in a reasonable span of time. That second is a big if, considering the number of subscribers. It would be possible to use a symmetric algorithm and include the key in the message, but while most readers would have the knowledge to decrypt it, they would likely not have the software to do so easily, and so it would be much more convenient for them to just get the announcement and go check the website, as opposed to spending half and hour trying to find and configure software.
When you run SpamAssassin in test mode, it tells you what rules got hit. You can also look at the headers in "Spam-Tagged" email to see what rules got hit. I looked for "Spam Testing" pages on the 'net, but had no luck.
Could someone run the Crypto newsletter through SA to find out what cased its evaluation?
As an aside, Counterpane could have done this to find out what the problem was, too. Not that they should have to, but they could have.
It all goes downhill from first post
I have
Const maxspamsize = 42695
in my spam filter - I've only receive one piece of spam larger than than in the last 12 months (a giant promotion for a Korean trade show). It speeds up my spam filter processing and lets large newsletters (with false triggers like this) through without a problem.
Recycle PCs and build a wireless community network www.hillsborough.org.nz
So blocking untrusted servers doesn't make email unreliable? I find that very hard to believe. Considering that most of the time it is Net blocks that are blocked, not just individual IP addresses.
blocking IP addresses is also open to abuse... If I had a grudge against an ISP, I could fake some SPAM headers and send it to any of the IP blockers. Maybe send several copies from different accounts. Getting an IP listed is usually easier than getting it removed, so in the mean time many legitimate emails are being blocked...
I believe you have to attack the root of the problem, and that is stopping the SPAM at the origin. This is probably the more difficult approach, but it is the only one that will avoid dropping legitimate mail.
The first inclination one has would be to suggest that everyone close their open relays. But this depends on people doing the right thing all the time, and has proven ineffective.
Fortunately, there's another way.
Right now, everyone who receives mail has to listen to everyone who tries to connect. The problem is how do you separate the wheat from the chaff?
The solution is to take advantage of the information SMTP and TCP/IP give you when a connection is established. The fact that you're receiving a connection gives you the address of the sender. And during an SMTP transaction, one of the SMTP commands (the MAIL FROM command) gives you the domain of the email's sender, e.g. "MAIL FROM slashdot@sysexperts.com".
When you're sending email to someone else, you do so by looking up the MX records for their domain, which tells you which systems are responsible for receiving email for that domain. This gives us a possible answer to the spam problem.
Suppose instead of blindly accepting email from everyone, you were to take the domain given to you by the MAIL FROM command, look up the MXes for that domain, and reject the email connection if the IP address of the sender doesn't match one of the domain's MXes?
Now, suddenly, you would end up rejecting email sent from every unauthorized relay, because the owner of the domain can make any system that is allowed to send email on behalf of his domain into an MX (and, if he doesn't want that system to be used for delivering email, then he simply makes such systems the lowest priority MXes in the list and blocks outside port 25 connections to them ... something he's probably doing anyway).
Suddenly, the only systems that spammers can send email from are systems that they legitimately control and that are defined as MXes for a domain they control. Suddenly, spammers have to set up and maintain their own domains and their own boxes. The costs have just become a lot higher, which will get rid of most of the spammers.
And suddenly, blocking spam becomes orders of magnitude easier -- you only have to deal with spammers who have decided to pay the (now much higher) price for sending spam and who cannot use someone else's system to do their dirty work without permission.
Use 'slashdot stuff' in the subject line in any email you send me if you want to get past the spam filter.
I use procmail with SpamAssassin in this manner:
It only takes a little bit of thought and minimal configuration to keep your mail from incorrectly being flagged as SPAM. For me, using this method has led to zero (0) false positives on messages from known sources, for two years. Every once in a while a SPAM message sneaks into my INBOX (a couple a year), but then I submit it to a SPAM database used in SA's checks (like Razor), or adjust any particularly annoying rules' scores, and it doesn't make a repeat appearance for me.
If your find that any particular newsletter is being treated as SPAM by your mail filters, there's probably a very simple way for you to make sure it isn't filtered out. Use the tools you have wisely, and you won't be disappointed.
~Chris
Unfortunately, I have executed a virus
We often see viruses and spam being send with spoofed sender address, and some spammers are clever enough to even use sender addresses from the same domain, which would be more likely to be on the whitelist. It would be nice to combine the whitelist with signature checking, if you know the senders public key, you simply filter anything unsigned.
Do you care about the security of your wireless mouse?
As a lot of people will probably whitelist cryptogram, if one wishes to spam technical people, he just needs to set From to Bruce.
An employer of mine sent out a very important e-mail with "IMPORTANT - MUST READ" in the title, and guess how many people got it? All thanks to wonderful e-mail filters...
"If anything can go wrong, it will." - Murphy
Am I the only one that has all of the mailing lists I subscribe to bypass SpamAssassin?
For each mailing list I subscribe to, I use a special address suffix just for that list, that bypasses all of my spam checks (including SpamAssassin ), and just goes right into the mailbox that I use for that mailing list.
No problems with false positives, and it saves me the overhead or running SpamAssassin on every incoming message from a busy list.
it just seems like common sense, no one should have a problem with SpamAssassin misclassifying incoming newsletters if they just think about how they organize their email.
Aside from the spot-on comments that people have made regarding adding a whitelist entry Crypto-Gram (an obvious candidate for whitelisting if there ever was one, given that it frequently discusses spam, scams, and probably even includes text straight out of some spams), here is my initial analysis and response to him.
Oh, first one other comment: SpamAssassin does not block content. SpamAssassin only flags probable spam. What the site or user does with that flag is their own business. Some mail administrators misuse SpamAssassin to block email, but we do not recommend blocking email. Really.
------
[...] One false positive (or a related set of false positives) is not really a statistically useful sample size. To get to a high rate of filtering, most filters do have some false positives. You can get fewer false positives with customization of one form or another (personalized Bayes training, whitelists, rules, automatic learning algorithms). Our goal (everyone's goal, I think) is to get the best ratio of false positives to false negatives. It's a difficult balance sometimes and some legitimate content has a harder time.
On to the data:
I checked your newsletter with two versions of SpamAssassin: the current stable version (2.44) and the very-soon-to-be-released development version (2.50).
A score of 5.0 is the default threshold to be flagged as spam.
In SA 2.44, your mail receives a score of 3.20 (2.40 as I received it, but I believe the score would be about 3.20 for most people). That's on the high side, but has bit to go before being flagged as spam. The score is the same with network tests (DNS blacklist tests and Razor).
In SA 2.50, your message would probably receive a score of 1.90 without network tests and 1.00 with network tests. Note that the test scores may change a bit before the final release of 2.50, but those are better scores, more what we like to see for non-spam content. They would be even lower when using Bayes (part of SA 2.50). Those lower scores are not unexpected because... well, 2.50 is better. :-)
Based on these results, it's not clear to me why yesterday's newsletter was flagged as spam. Some possibilities:
Can you give me more information about the false positive that you experienced or was reported to you?
Thanks.
Dan
------
If I find out more of interest before the thread is closed to comments, I'll try to post a follow-up to my post.
This simply shows that newsletters and similar are not really sent by the right medium right now. EMail hasn't kept up with the times and as a result we see this endless amount of spam.
;)
What is needed is a foolproof way of saying "I want this, please send it to me" and then being able to reject it safly without needing the other party to do it for you. For example:
I send a message to cryto-gram, including a key. This key can then be used to send it to me, and I accept it (key in combination with who send it and so on, I am sure someone with even more experience can figure out a fool proof way). Good stuff. But then I realise that I don't want this anymore, and I simply remove the acceptance of this key in my own software (and send a message that I don't want it anymore, no harm being nice to the nice), and it will be filtered away.
Or something along those lines, I can asure you that I haven't fixed up a foolproof and perfect system yet
Comment removed based on user account deletion
..."Ancient Gurus srb and guenther say, 'Sort your mailing lists to the folders before you filter your spam.'"
Crypto-Gram isn't the only mailing list that gets hit by misunderstandings - all automatic mail handling is always confused about automailers and mailing lists. And even due to usability factors, it makes sense to sort mailing lists to folders anyway, and use a client that supports multiple specific folders.
My primary mailbox is with a small, local ISP. I can't buy broadband from them, so I get my connectivity via cablemodem. I do have a mailbox in the cablemodem company domain -- that's the one I give out when I expect abuse. (I do it this way because I expect to be dealing with that ISP long after the cable vendor has either ceased to exist or has treated me badly enough that I left.)
So I want my outbound mail to appear to have come from the ISP. Setting Reply-To is usually adequate, but not always -- when a human is looking for the address, they could easily grab the wrong one. And it creates potential confusion I don't want to create. So I set my from address to name@isp.com.
I can't relay through the ISP's relays, because I'm outside of their IP range. (If they did some form of authenticated SMTP, such as SMTP-after-POP, they could let me.) And the cable vendor's mail relays won't send mail out with some other domain name on it. So I send everything out directly, no relays.
If you look at many headers, I suspect you'll find that I'm not the only one forging my From: address for legit reasons. The presence of the X-Authentication-Warning header some MTAs add correlates fairly weakly with spam. (Some details of it -- e.g. no valid reverse DNS for the sending machine's IP -- could be useful indicators.)
Note that SpamAssassin isn't on my whitelist or anything like that -- it just worked.
False alarm?
DO NOT LEAVE IT IS NOT REAL
I just got the email today and it failed. I'm running 2.44 from Debian and haven't yet looked at tweaking any of the rules.
Here's the verbose banner that SA put on my copy:
It looks like some dumbass has entered it into Razor. Unfortunately, some people (and yes I did this originally) had their procmail setup to enter an email into razor if it is deemed "spam" by SA or something else. Those 3.9 points are what puts it over the threshold.
It has some other advantages too:
I learned this from the electronic greeting industry. Similar to Usenet 2 and Internet Mail 2000, messages semaphores will become the future of e-mail. People will create web content as easy as they create e-mail messages now and semaphore the recipients (using IM or email) to look at their content. Recipients who are interested will click on the URL in the semaphore. Recipients who want mail from Bruce, will open it. Bruce might even (G)PG(P)-sign the announcement notice so that spammers can't pretend to be him.
Then again, why should Bruce have to mail anyone at all? If his newsletter is so good, his readers will bookmark his page and read it every now and then, just like I do with DaemonNews or ArsTechnica.
The Internet is evolving, and Bruce is whining along the way. Mass-mailed newsletters are going the way of the dino-WAIS-server (just like FTP
-ez