Slashdot Mirror
Are Spam Blockers Too Strict?
Myrte writes "Wired.com has a long piece on whether spam blockers are blocking wanted messages." From the article: "For years, e-mail users complained that torrents of unwanted messages clogged their inboxes and crimped their productivity. Now, e-mail users, marketers and mailing list operators are more worried that spam filters are blocking out too many wanted messages. AOL isn't the only company to face charges that it improperly blocks legitimate messages. But, as the world's largest ISP for years, it has long borne the brunt of complaints from mass e-mailers over the problem."
Thanks to my damn spam blocker, I've missed out on hundreds of opportunities to accept millions of dollars from Nigerian royalty.
Slashdot Burying Stories About Slashdot Media Owned
The absolute biggest piece of hilarity is Norton Antispam. People rush out and buy it, and install it on their computers. Usually they never do anything in the way of setting it up (just expect it to work magically), but that makes no difference because it continually reconfigures itself on its own whims.
:rolleyes:
And then they call and abuse their ISP support personnel for days on end of "I'm not getting any of my damned email!!"
And it's all right there in their 'Deleted Items' folder.
do() || do_not();
Stop using email. It's 100% effective at blocking email spam.
Um, error exists in both directions. Limiting error in one without concern for the other usually increases the other. (Instead of limiting the error you usually shift the range.) This is known.
What's news here?
'Sensible' is a curse word.
I can't send email from my work place to my free register.com hosted account because I had emailed myself some links to look at while at home. Apparently the spam bot assumed messages with just a subject and links and flagged my work address as spam.
I couldn't get them to undo the change... But it is a free service and I figured I won't get anywhere if I push it and these days I just send any emails with links to my hotmail account.
"I am the king of the Romans, and am superior to rules of grammar!"
-Sigismund, Holy Roman Emperor (1368-1437)
I'd like it if my spam filter could "mod up" non english email.
:) (this probably wouldn't fly at work, but for his personal email it's fine)
most of my email correspondance isn't in english, while most of my spam is in english... I've instructed my dad to delete ANY mail with an english subject if he doesn't know the sender before opening it, and that seems to work out fine, english is his 3rd/4th language and only has 2 contacts using it. If something is important enough, he'll get at call about it
Obviously spammers are trying to get through filters by making their email appear legitimate. The closer spam looks like legitimate email traffic the harder it is to block them without also blocking some legitimate email. It's kind of a stupid question with a "WELL DUH!" answer.
Not trying to put out a flame but really guys...
The race isn't always to the swift... but that's the way to bet!
It's more that SMTP is too broken. The model we use to communicate with each other is sadly too open, given the potential of the technology for automation. The real solution is to extend or replace SMTP completely.
Luck favors the prepared, darling.
It's like inviting someone to a party & you agree that they can bring their "affiliates" along. Your invitee shows up with 20 strangers & whoever you have working the door says "I don't know all these people, they aren't allowed in."
The solution isn't to cry about the "gray" area, it's to explicitly tell people who the fark these affiliates are & what they'll be sending.
[Fuck Beta]
o0t!
When I get a message with a moderate probability of being spam, my spam blocker sends a message back requesting that the sender confirm the message. Works great. Those few legitimate senders stuck on a problematic server can still get their messages to me and so far no spammer has attempted to bypass it.
The only time it doesn't work is when the sender's spam blocker dumps the confirmation request or when the sender doesn't understand what to do.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Just like door to door salesmen and tele-marketers, mass e-mailers have ruined their reputation as a group and are no longer effective at what they are trying to do. If you want to keep your customers updated, offer an RSS feed, personalized with their user id if necessary. Times change, deal with it.
If a user has signed up for a mailing list, and doesn't get what they asked for, then that's a false positive, no matter how commercial the mailing list. And this does happen. So in that respect, spam blockers are too strict.
But on the other hand, I fish out a few false positives from my spam dump every month and look to see why they were blocked. In most of the cases, it's because the mailing list operator is doing something dumb. For instance, the last false positive I received - for a legitimate, informative mailing list I deliberately signed up for - triggered my spam filter because of forged headers, two counts of malformed headers, and every other line was in all caps.
The reason why they were caught out was because they used what appears to be a mass mailer designed for sleazy purposes, and they didn't bother with any QA.
Anybody who is running a mailing list should follow a few simple rules:
That's what I consider to be common sense, but apparently common sense is hard to come by these days.
Bogtha Bogtha Bogtha
Sounds like it was working fine to me ;-)
+----------------- | What is the question!
My experience, though, is that it isn't the spam catching software that works with typical desktop email applications like Apple's Mail, Entourage, Thunderbird or Outlook that's too strict (sometimes far from it, especially w/regards to Entourage); it's the spam catching software used by Webmail providers like Hotmail and Yahoo's Mail.
I know it's in their best interest to flag as much stuff as Bulk Mail as possible (which can then be filtered into a bulk mailbox, and removed automatically after 30 days), but until I recently switched hosts, everything I was sending to Yahoo or Hotmail was going into the Bulk Folder. Now, I think this may have been due to my hosting provider, but all the tests I ran seemed to indicate that they weren't on any blacklists, or anything like that.
I even took the time to implement SPF records for my domains. This had a noticeable effect in GMail, which actually adds a header to incoming mail stating whether an SPF record was found and followed; it had no effect in Hotmail, however, which is maddening, since it's Microsoft's stupid initiative!
I don't know what the answer is, but we're not there yet.
concrete5: a cms made for marketing, but strong enough for geeks.
This is what happens when you don't think forward on protocols. The cure, in the form of hundreds of attempts at everything from Baysien filters to source-IP blockers, seem to always fail. Why? Because SMTP, our mail protocol, is based on telnet, 7-bit ASCII, and easily fudged authentication. Worse, 'thinking' filtration systems use a rules basis that appears to work, but can never work because the rules can change, as any successful spammer knows.
Then, we get a bunch of techno-idiots like the US Congress to legislate email relationships, miserably, contributing further to the problem.
The real solution? Simple blockage. Route the bastards to 127.0.0.1. Force authentication of the address and its owner before it can go out of the blocked ACLs. And if it happens again, shunt the address to a different CIDR block. Or re-write SMTP. That's all that's going to work. Nothing is foolproof because fools are so ingenious. Never underestimate the power of a hacker, and locks keep your friends out, your enemies have pick tools.
---- Teach Peace. It's Cheaper Than War.
I use spam assassin, and I found it only blocked stuff that was actually spam. I set it to 4, and it still let things like marketing emails from Nintendo and Sony though (I like being on the mailing list), and other newsletters I subscribed to. It rarely if ever blocks anything that I want to see. It's very good at blocking stuff that I didn't want to see. I don't really see a problem with spam blockers. And I had mine set pretty low.
Anthropic principle: We see the universe the way it is because if it were different we would not be here to see it.
The real problem is that people are typically assuming that email is a reliable and secure technology, when it is not at all. People just need to learn about using 'return receipts'. The alternative is to use an entirely different communications protocol for messaging.
--jeffk++
ipv6 is my vpn
I used to work for a company that sent emails to medical professionals regarding ongoing clinical drug studies.
These emails absolutely took "opt-in" to the next level.
Not only did the doctors opt-in to receive these emails, they had to go through a fairly rigorous screening process to be eligible to receive them. On top of that, it actually would have been highly illegal for us to send these emails to others!
So, needless to say, the emails weren't spam and were going to modestly-sized email lists of 100-1,000 total recipients, approx 25% of which were AOL users.
And still, we had countless problems with AOL blocking them. AOL never listened nor responded.
OtakuBooty.com: Smart, funny, sexy nerds.
They say, "List operators, marketers, and email users complain spam filters are too strict." I'll bet 99% of marketers, 90% of list operators (not the 10% that are legitimate), and 1% of users think it's too strict.
Listen, when you go to your snail-mailbox and get the mail, you can pretty much tell which mail is good and which is junk, right? I mean, it's easy to tell letters and cards from family members and friends from bills and unsolicited junk. It's easy because there's a physical form of recognition taking place.
Email is tougher, because in most cases all you have to go by is a sender's email address/identifier and the subject line. Now I don't knwo if you've looked at those two things closely, but it's usually easy to tell when the email is spam (how many freinds do have named Lemon T. Viceroy?). Now, as reported, phishers are getting more sophisticated and they are making much more convincing emails that are tricking people into believing the email is from their bank. They's be able to save themselves some time and frustration by checking the email address vs. a legit email they've received from the bank.
I think blocking has to start at the user end. You have to put up a wall and say that only these addresses are legit and anything else is suspect. You dump suspect emails into a separate folder and peruse it for emails that are actually legitimate, and add a pass-through for them to your wall. It requires maintenance and vigilance, and cooperation from banks, credit card companies, etc., who have to make sure you know what legitimate addresses they will send emails to you with. Any left over emails you fire back to the senders and alert your ISP
Putting the responsibility for screening mail on the user is problematic, but it's certainly a lot more efficient than having to listen to complaints about legitimate mail getting blocked constantly. I do this very thing constantly with my personal account and by using my ISP's spam filter, I'm doing a pretty good job of screening out the crap. By alerting my ISP of definite frauds, I'm hopefully making things easier for others. Of course, you have to make this system easy to use, or users will get frustrated and it won't work properly.
Maybe snail mail isn't dead yet for a reason.
GetOuttaMySpace - The Anti-Social Network
This should be a given. If you try to block spam, you are going to block some legitimate messages. Hopefully, your ratio of blocking spam messages against legitimate messages is good, but it will never be perfect. This is due partly because spam itself is subjective. A lot of spam messages can be picked out and determined to be a spam message by 10 out of every 10 people. But for some messages, its not that simple. It's just real subjective. Then you're asking an algorithm to use subjective logic to determine whether a message is spam or not and problems just occur. Like I said, for the most part these filters work pretty good, but its not going to be perfect and anyone that thinks so, is just not thinking straight.
I am not opposed to some degree of flagging an alleged spam message, but to discard it without the end user knowing about it is where issues begin to arise. By flagging a message, the end user is able to use their own discretion to determine whether a message is a spam message and they can do whatever they want with those messages.
This isn't to say that RBLs and spamlists are a bad idea, just if you implement one of these, then be prepared for some type of backlash. Perhaps in some cases an RBL is necessary, but to think that using an RBL you are going to stop all spam and all of your clients are going to be happy, that's just wrong.
I know this isn't the final answer, but to me it is by far the most responsible and far reaching.
Will spammers register real domains, yes. Will they send emails with a fake from address that has at least a valid domain, yes. It makes it just that much harder, and makes it harder to use farms. If the SPF record has a huge subnet then the spam blockers can ignore it, and then put it on a watch list. At least we are adding some level of authentication to the process.
The cost of SPF is so little, I don't understand why their is not more push for it, and why we can't just give it a shot. I'd rather do that then go thru some authentication process with a company and then pay for some type of certicificate. Lastly, as a programmer I hate when all of the suden we have to do quadruple opt-outs, when the real problem is people sending gobs of rolex adds from their dorm room with or without their knowledge.
Force authentication of the address and its owner before it can go out of the blocked ACLs.
This would be so trivial to bust thru and automate it isn't funny. What happens to zombie machines? They can authenticate fine, so slip right by this problem. Instead of sending thousands of messages as fast as possible, use thousands of zombies and send just and handful messages each. You'll never trip the thresholds for volume and the spam will be buried in among the legitimate e-mail sent by that user.
Authentication is not a solution.
Learning HOW to think is more important than learning WHAT to think.
So 25% of doctors are AOL users. Now I'm really afraid to go in for my next checkup.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
WHITELIST. If you want it, whitelist it. If you don't have it whitelisted, then the SPAM filter can classify it... If it does it improperly, then tell the filter that it is/isn't spam (as the case may be).
Teach the users how to do this, and let the whiners kill themselves with angst.
If I want something, I'll go seek it out for myself. Leave me the hell alone. It's not your place to constantly bother me.
In general, if people want something, they will seek it out for themselves.
Look, I'm with you. I hate this stuff as much as you. It's usually even a nice safe rant for a few insightful mods, but yours is practically a troll.
I can assure you that there are quite a few hundred thousand consumers out there who do not share our outlook on this subject, who become very hostile when you fail to keep them informed of important information, and who couldn't set up an RSS reader if their lives depended on it.
Sorry, I'd love to live in that fantasy world, but you have to face that it's just not reflective of reality.
The solution to all of this, is dspam, of course.
We were previously running SpamAssassin for about 4 years with 13 RBLs and blackholes.us, and we were at 90% accuracy or so, and still seeing 10-20 spams slip through per-day.
I gave dspam a test, and after 3 days, we were already up to 95% accuracy, with ZERO spams slipping through.
Today, about 3 years later, we're now at 99.726% overall accuracy, again, with ZERO spams slipping through to any user's mailbox. For false-positives, the users can go to the web interface, check the "legit" emails getting incorrectly marked as spam, and have those sent to their mailbox, retrained as HAM. After a user receives 'n' number of messages from a specific address, they're auto-whitelisted.
dspam blows away anything I've ever used, ever. We're not seeing a single spam in any user's mailbox in 3 years, and we're at about 85% incoming spam per-day with 1 RBL.
This is one of the things SPF (http://www.openspf.org/) is meant to end - false positives. One of the problems with SMTP is that you can't build up a reputation by domain because anyone can claim to be you.
If a verified sender is sending [lots of] unwanted email, they are a spammer and should be blacklisted. Otherwise, verified senders should probably be trusted.
The closer spam looks like legitimate email traffic the harder it is to block them without also blocking some legitimate email.
/. a top-tier spammer was aggravated by their efforts and managed to get a list of addresses for those who signed onto bluesecurity. I just checked the "junk box" on my email server and have found that in the past 12 hours there have been about 50 emails entitled "bluesecurity.com" with a body containing the WHOIS record for their domain. Apparently, the spammers are already striking back with a vengeance.
Your argument makes sense but there is more to it than that. Spammers are starting to catch on that their techniques to thwart mail filters can be used to manipulate those filters to block other people's emails. THAT is still pretty inceniary. Let me explain what I mean:
Some time ago I signed onto the "bluesecurity" website as I was intereste in their counter-spam efforts. As we all know here on
Besides annoying the heck out of those unfortunate enough to be on the target list, the thought came to me that this could be a crude attempt to train email filters to block out any (legitimate) correspondence affiliated with bluesecurity.com. I think we're going to see a lot more of this in the future: Spammers for whatever reason select a victim (anti-spam organisations, Microsoft, Symantec, etc) and start sending out massive spams that either repeatedly mention the victim's name, website address domain, etc, or are crafted to look like legitimate correspondence from the victim. The scummy vermin that send out the spam are the same types that go on phishing expeditions so they've had practice imitating others.
Since so many people run email filters, once these filters intercept and mark those messages as spam then legitimate email from their victims are more likely to be blocked as spam. That's all I need is for a spammer to send a few dozen emails that look like Microsoft correspondence, only to have the email filter get trained to filter out REAL email from Microsoft about my MSDN subscription for example.
AOL is rumored to do most of its spam-blocking without notification to the sender or recipient, and that's a big problem and they're hardly alone in this behaviour.
If there's anything broken about SMTP's handling of spam, it's that you sometimes don't decide that a message is spam until after you've accepted it, so it's hard to provide synchronous notification in case it wasn't spam. (SMTP milters let you look at the message body and run it through spam filters before accepting the message if you want to do that, but a message might already be sitting in the recipient's mailbox before you figure out that 1000 of your users have received identical mail and 99 of the first 100 users that read it marked it as spam.)
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
You meant to say SPF and DKIM.
"senderID" was an unsuccessful non-standard created by Microsoft hijacking SPFv2 with submarine patents and other deceits. Read up on MARID and see what I mean. senderID is dead, do not try to implement it, do SPFv1 or domainkeys if you want the current gold standard.
DKIM is the successor to domainkeys, and it's looking pretty good.
There is no "easy" involved in crypto, however. If you want "easy" do SPFv1... spoofing prevention with 5 minutes of work by any competent DNS administrator.
> When I get a message with a moderate probability of being spam, my
> spam blocker sends a message back requesting that the sender confirm the
> message. Works great. Those few legitimate senders stuck on a
> problematic server can still get their messages to me and so far no
> spammer has attempted to bypass it.
Well thank you so much!
Since the lowlifes started forging "from" addresses using my domain, I am getting several such "confirmation" messages every day. And while my spam filter is doing its job pretty well, I have not found a way to filter out your smug verifications without getting rid of the legitimate ones.
So, thanks to people like you, I get 5 times more verification requests than actual spam.
You better hope that there is no higher power because if there is, and it decides to grant my wishes just when I get yet another verification, you'll have a bit of a problem removing that sequoia from your rear orifice.