Slashdot Mirror
Handling Anti-Spam Systems When You Aren't Spamming?
"Many large ISPs are implementing anti-spam filters based on how many emails they receive from a single sender to many of their clients (thinking that if they get over five mails in a few seconds, they must be bulk-mail spammers, and therefore block the rest of them), but this is hurting the delivery of services like ours. Worse still is that there is typically no error message returned to us - the emails simply get dropped, much like a standard packet-filter firewall works. Then we have clients wondering why they didn't get their expected message.
Sometimes, ISPs will add us to their "white" lists (as opposed to "black" lists of known spammers), which fixes the problem, but only for that one ISP.
(I find it ironic that the email system was designed to be quite reliable, so that you could send a message and have reasonable confidence that it got to its intended recipient, and yet we're now moving away from this in the effort to fight spam.)
Now I know we don't want to tell spammers how they can get around the anti-spam filters, but I'm wondering how have others fought the anti-spam problem with their mailing lists?"
Configure your mailer not to send more than 5 messages along the same connection, or whatever is needed to get through. If it's too much, notify your audience that due to unreasonable policy on behalf of their ISP, you can't deliver to their inbox.
I don't know how you are managing your newsletter, but eGroups doesn't seem to have too many problems with that; Either they know how to get through (more probable), or everyone makes an allowance for an egroups address (less probable). Either way, if all else fails consider using egroups or a professional service that works (Never tried myself and am not affiliated with, but I hear whatcounts is good.)
Maybe ISPs could utilize a system that could scan outgoing email for mailing list joins and then add those addresses to the "white" list for a specific user.
That could probably go down as the most stupid idea I've heard so far this year. All this 'monitoring' is sounding way too authoritarian to me.
In the majority of cases, it should be the individual's responsibility to sort mail, not the ISPs. Would you like it if USPS decided to go through your mail throwing away whatever it thought was 'unsolicited'? You bet your ass you wouldn't. How about if they suggested 'looking through your outgoing mail' to find out what you were expecting to receive? If people like you were taken seriously, it'd be like the Third Reich.
I do not want anyone reading or filtering my mail except myself! If you want to be nannied, that's your choice, and you can go use AOL or whatever, but we don't want the majority of ISPs controlling mail delivery in this way. Even if their intentions are good, 'proper' e-mail could easily get thrown away, and worse.. if laws were passed that allowed governments to control ISPs in some way, they'd have a system already in place to 'control' mail delivery. No thanks!
The answer to this question is that any freedom loving citizen should be filtering their own mail and not relying on a nanny state to sort it out for them.
mogorific carpentry experiments
We're in the same boat. We're a small ISP and we run a list server for our clients. Some of the stuff they send out is so amusing, even I sign up for it.
. ht ml
/dev/null by the big guys.
What we've been doing is verifying our email lists (this goes a long way to avoiding getting flagged as a bad guy) and sending messages out one per connection. It's fabiously inefficient and it takes 4 hours to send out 12,000 emails (our biggest customer) but we've only managed to tick off about 3-4 other ISPs.
There's two things that I see as being issues that we're going to have to deal with soon in a real way:
1) Little Napolean wannabe sysadmins at other small ISPs that belive anything sent to more than one recipient is spam. These guys really irk me. Its one thing if their customer complains about mail from our domain and they evaluate the situation and block it but it's another for them to see a message destined for more than one mailbox on their domain and arbitrarly decide to reject all mail from our mail server (not just the domain that sent it mind you; ALL the domains we host.) Heart's in the right place but they left the lens cap on thier mind. I've tried talking with them but that just seems to iritate them more.
2) Big email hosting companies (Yahoo, AOL, MSN, Hotmail) looking to make yet another buck. Take a peak at these headers on a bulk email I got from Yahoo:
X-YahooFilteredBulk: 209.164.21.221
And this page from the Yahoo help desk:
http://help.yahoo.com/help/us/mail/spam/spam-17
Now don't get me wrong, I love (well, like) the bulk mail folder on my Yahoo account. I'm just waiting for these companies to decide to offer "Prefered Sender" subscriptions that will garante delivery to thier user's Inbox or maybe Prefered Partners Inbox or something. What are we (small ISP's) going to do then? We're not going to buy a subscription from every Yahoo/MSN/AOL out there and we can't serve our customers well if all thier lists get piped to
I'm one of the SpamAssassin developers and I find their technique odd.
;-)
Wouldn't this have a horrendously high false positive ratio for things like mailing lists?
Anyway, tell them to use SpamAssassin - it kicks ass. And I'm not biased, honest
Matt. Want XML + Apache + Stylesheets? Get AxKit.
More importantly it's a largely waste of time, because we have bounced precisely *zero* emails because of this filter. Obviously the spammers have gotten wise to this filtration method and have worked around it (it's really old after all), which rather makes the whole point of this discussion redundant, doesn't it? ;)
UNIX? They're not even circumcised! Savages!
Email is never going to get fixed. The fundamental concept is flawed. You can't allow arbitrary messages from arbitrary anonymous sources without getting spam. Probably well over 99% of solicited mail is non-anonymous anyway, so the solution is simple, in theory.
Until anonymous email is deprecated the spam problem will not be solved, plain and simple.
I know it might border on heresy, but why not have the ISP actively manage the mailing lists? Here's an example:
Suppose I publish Gland Nut Weekly, and I use fatboys.net as my ISP. I register myself with the ISP, giving them the name of my mailing list, and the names/email addresses of the allowed publishers. When I have an issue ready to publish, I send it to fatboys.net, who then sends it to the current subscribers on the list.
Other ISPs can 'trust' that the email sent by fatboys.net isn't spam, since fatboys handles the mailing list, fatboys.net can be sure they're not a source of spam (and look like one of the good guys) since they're handling the mailing list, and the publisher benefits from having the ISP send the actual mail at high speed and without having to employ tricks to get around outbound spam filters. Whaddya think?
Tiller's Rule: Never use a word in written form that you've only heard and never read. You will end up looking foolish.
Yahoo, to pick one example of an email provider, if not an ISP, exactly. If a server sends more than a certain number of emails to yahoo addresses within a certain period of time (I don't know what the specific values are), yahoo will automatically stop accepting mail from that server.
Like some ISPs, yahoo maintains a "white list" of servers that will be excepted from this rule. For an email provider the size of yahoo, this actually makes a lot of sense: there are only a small number of people who will fail the "too much mail too quickly" test for legitimate reasons (other big email providers, for example), so it's easier to work with the small number of exceptions.
I have worked for an email list management company that sends out several million messages per day; yahoo took a look at the company's subscription processes and the messages being sent, decided that their mail was okay, and added them to the white list. No one at the company really minded having to make the effort to get on the yahoo white list, since it benefits everyone involved for yahoo to filter as much spam out as possible.
* * *
It is a dada story -- it has no moral.
I suggested something like this a while ago. Server side filters accessable by ordinary users. People here said they have those, but misunderstand. Most server side mail filters apply to ALL accounts and are not accessable by users who have pop accounts. In fact I have not heard of an ISP implementing such an idea and I claim this as prior art for such an idea so don't even think of patenting it I'll sue.
It's simple, a users logs into their isp with a web based app that allows them to say filter out this that and blah. I'd use mail headers, and filter out korean character sets as that is where most of my spam lately comes from. Funny I can't even read it but the charset says korean.
I am leaning alot about smtp / pop and basically the only requirements are HELO, MAIL FROM, RCPT TO, DATA, QUIT, USER, PASS, etc. The protocols themselves are too stupid for most else. Filters on the server could also interfear with privacy. In order for them to filter mail they would have to have a mail scanning program. If they log this data then it becomes an provacy issue.
The real solution is better mail filters in the pop mail cleints. For a delete filter it may be better if the pop client were to call TOP and get the message header and then delete the message appropriately. I am working on a java implementation of this. My POP3 bean can do this, I just need to scan the headers.
Only 'flamers' flame!
"just" require all SMTP traffic to use TLS, and have them all under one CA, so everone can test the authentication of the sender .. of course .. this is only a pipe dream ;)
Come the revolution, they'll be the first up against the wall -- someday Denial of Service will be illegal, and then they'll get theirs.
If all this should have a reason, we would be the last to know.
Nope. Not all. Perhaps it is supposed to, but not all does. Especially at an ISP. I've sent mail from one of my email accounts (that I pay for) to another (that I also pay for), and the second location just drops them off to the bit-bucket.
Remember, if someone falsifies mail origins, kicking back won't help as much. Or the filtering might kick in a little later in the ISP's server chaining. Or the ISP might feel that would be like supporting the VRFY command, which most do not nowadays just for spamming reasons.
The sysadmin running the mail server can have it do other things, like put likely spam into a different spam mail account that the user can check periodically.
Give a man a fire, and he'll be warm for a day, but set him on fire, and he'll be warm for the rest of his life.
From 'spamassins web site'
The 'user agent' it the users mail program. This means that the users is not filtering out the data on the server. The server is only 'tagging mail'. The user still has to download the whole mail. Obviously your to stupid to understand a thing I am talking about. I am talking about a filter on the mail server that I set up that delete the mail from my inbox and I never ever see it. So in my case I would create a filter that says 'delete mail where charset like "korean"', then all mail that is coming from korea is deleted form the web server when it arrives at the pop mail account on the mail server.
My ISP uses the spaminator which reduces my spam by over 50%, but it is still not a filter that I set up for my account on their servers.
Its obvious from your post that it doesn't require brains to post on slashdot.
Only 'flamers' flame!
it is possible to execute the TOP command and download the headers of mail and from the mail headers have it delete mail based on that. TOP 1 0, gives me just the mail headers. If I have 20 spam messages and I just get the headers of them I can delete all the spam and not download the whole message. I do this through my web based application that I have where I display the inbox I only get the headers. Maybe the soultion is to leave the mail on the server and only get the headers in the mail app and then select which messages I want to download after that. I could also set up filters based on these headers so that I never see the messages in my inbox that have lets say a character set that is in another language other then my own preference.
headers are usually less than 1k, but html spam is usually several k. This would cut down on my download time.
Only 'flamers' flame!
Ahh. But a large issue is that if a spammer issues a bunch of mail into your server, and some of them are accepted and some of them return errors... then suddenly the spammer has a way to check if addresses are live or not, and has a replacement for the VRFY command.
I'm not saying that this ISP behavior has good reasons, just that it has some reasons. And for some ISPs, that's reason enough. Really sucks for legitimate users, though.