Handling Anti-Spam Systems When You Aren't Spamming?

nautical9 asks: "Spam is a huge, annoying, and costly problem, there's no question. But what about those of us who run a valuable service, such as a newsletter, that users willingly sign up for and actually DO want to receive in their inbox every day? It's really too bad a few bad apples (ok, thousands of bad apples) are ruining the email system for the rest of us. Not all bulk-mailers are spammers, and large service providers do have a legitimate need to communicate reliably with their customers. But with everyone focusing on blocking commercial and unsolicited mail, no one seems to remember that there are valid reasons for having large-scale mailing lists." Maybe ISPs could utilize a system that could scan outgoing email for mailing list joins and then add those addresses to the "white" list for a specific user. Actually, why haven't ISPs adopted some form of user-level filtering system for email yet? It would seem like this would be the next sensible step in the fight against SPAM.

"Many large ISPs are implementing anti-spam filters based on how many emails they receive from a single sender to many of their clients (thinking that if they get over five mails in a few seconds, they must be bulk-mail spammers, and therefore block the rest of them), but this is hurting the delivery of services like ours. Worse still is that there is typically no error message returned to us - the emails simply get dropped, much like a standard packet-filter firewall works. Then we have clients wondering why they didn't get their expected message.

Sometimes, ISPs will add us to their "white" lists (as opposed to "black" lists of known spammers), which fixes the problem, but only for that one ISP.

(I find it ironic that the email system was designed to be quite reliable, so that you could send a message and have reasonable confidence that it got to its intended recipient, and yet we're now moving away from this in the effort to fight spam.)

Now I know we don't want to tell spammers how they can get around the anti-spam filters, but I'm wondering how have others fought the anti-spam problem with their mailing lists?"

Comments

Make /var/log/mail public (sort of)

[Anonymous Coward]

First, I have to state that anyone suggesting you throttle/limit outgoing emails isn't thinking of the very large numbers involved here. If I have 100,000 subscribers, then I have to send more than one email per second, for 24 hours, to send all the messages. 60*60*24 = 86,400. If it's a daily newsletter, then I need to take less than 24 hours to send each newsletter.

Upon opt-in, issue each user a user identity (some random alphanumeric widget). Have a web page on your site that allows a member to enter their identity, and then a little CGI program parses /var/log/mail (which you're now storing in a database) to see what happened to the email. If the system says "delivered", then it's a problem for the user to take up with their ISP.

This is going to take a LOT of user education, but it's going to solve problems slowly over time. The emails that get dropped, if the user notices, will at least give your level one support something to go by. "Yes, our logs show that our mail server has delivered the newsletter to you on these days.... You didn't get it? Could you contact your ISP, and ask if they are filtering inbound email? Here, we'll email you the logs to pass along to your ISP, or you can get it from the web site."

To be polite, you could make the mail logs even more public, allowing the ISP to look up things, but you'd have to "sed" out email addresses, or at least obfuscate them (like everything left of the @ gets replaced by X's).

At the very least, it moves the technical problem from something vague behind the scenes to something more easily described, and seen, and comprehended, by the user. And it allows you to point the blame finger at the guilty party.

Finally, during the sign-up page, and on the troubleshooting pages you give to users, mention that if the newsletter doesn't arrive, a likely cause is their ISP. Give a top 10 list, based upon the problem frequency reports. (User changed email address, local mail filtering, ISP mail filtering, network outage....)

My worries

[nottestuser]

We're in the same boat. We're a small ISP and we run a list server for our clients. Some of the stuff they send out is so amusing, even I sign up for it.

What we've been doing is verifying our email lists (this goes a long way to avoiding getting flagged as a bad guy) and sending messages out one per connection. It's fabiously inefficient and it takes 4 hours to send out 12,000 emails (our biggest customer) but we've only managed to tick off about 3-4 other ISPs.

There's two things that I see as being issues that we're going to have to deal with soon in a real way:

1) Little Napolean wannabe sysadmins at other small ISPs that belive anything sent to more than one recipient is spam. These guys really irk me. Its one thing if their customer complains about mail from our domain and they evaluate the situation and block it but it's another for them to see a message destined for more than one mailbox on their domain and arbitrarly decide to reject all mail from our mail server (not just the domain that sent it mind you; ALL the domains we host.) Heart's in the right place but they left the lens cap on thier mind. I've tried talking with them but that just seems to iritate them more.

2) Big email hosting companies (Yahoo, AOL, MSN, Hotmail) looking to make yet another buck. Take a peak at these headers on a bulk email I got from Yahoo:

X-YahooFilteredBulk: 209.164.21.221

And this page from the Yahoo help desk:

http://help.yahoo.com/help/us/mail/spam/spam-17. ht ml

Now don't get me wrong, I love (well, like) the bulk mail folder on my Yahoo account. I'm just waiting for these companies to decide to offer "Prefered Sender" subscriptions that will garante delivery to thier user's Inbox or maybe Prefered Partners Inbox or something. What are we (small ISP's) going to do then? We're not going to buy a subscription from every Yahoo/MSN/AOL out there and we can't serve our customers well if all thier lists get piped to /dev/null by the big guys.

Re:Who?

[Zocalo]

Depends on what, precisely, you understand this to mean. Quite a few ISPs will reject emails that are sent to more than a certain number of recipients in an attempt to combat SPAM. Off the top of my head I think ours are configured at 50, although I know from experience that this is actually 150 because To: CC: and BCC: are counted seperately and not totalled.

More importantly it's a largely waste of time, because we have bounced precisely *zero* emails because of this filter. Obviously the spammers have gotten wise to this filtration method and have worked around it (it's really old after all), which rather makes the whole point of this discussion redundant, doesn't it? ;)