Slashdot Mirror
Handling Anti-Spam Systems When You Aren't Spamming?
"Many large ISPs are implementing anti-spam filters based on how many emails they receive from a single sender to many of their clients (thinking that if they get over five mails in a few seconds, they must be bulk-mail spammers, and therefore block the rest of them), but this is hurting the delivery of services like ours. Worse still is that there is typically no error message returned to us - the emails simply get dropped, much like a standard packet-filter firewall works. Then we have clients wondering why they didn't get their expected message.
Sometimes, ISPs will add us to their "white" lists (as opposed to "black" lists of known spammers), which fixes the problem, but only for that one ISP.
(I find it ironic that the email system was designed to be quite reliable, so that you could send a message and have reasonable confidence that it got to its intended recipient, and yet we're now moving away from this in the effort to fight spam.)
Now I know we don't want to tell spammers how they can get around the anti-spam filters, but I'm wondering how have others fought the anti-spam problem with their mailing lists?"
Then who's complaining?
Opt out? Fuck off.
'Tis better to kill a thousand innocent men than to let a guilty man escape justice.
Configure your mailer not to send more than 5 messages along the same connection, or whatever is needed to get through. If it's too much, notify your audience that due to unreasonable policy on behalf of their ISP, you can't deliver to their inbox.
I don't know how you are managing your newsletter, but eGroups doesn't seem to have too many problems with that; Either they know how to get through (more probable), or everyone makes an allowance for an egroups address (less probable). Either way, if all else fails consider using egroups or a professional service that works (Never tried myself and am not affiliated with, but I hear whatcounts is good.)
Maybe ISPs could utilize a system that could scan outgoing email for mailing list joins and then add those addresses to the "white" list for a specific user.
That could probably go down as the most stupid idea I've heard so far this year. All this 'monitoring' is sounding way too authoritarian to me.
In the majority of cases, it should be the individual's responsibility to sort mail, not the ISPs. Would you like it if USPS decided to go through your mail throwing away whatever it thought was 'unsolicited'? You bet your ass you wouldn't. How about if they suggested 'looking through your outgoing mail' to find out what you were expecting to receive? If people like you were taken seriously, it'd be like the Third Reich.
I do not want anyone reading or filtering my mail except myself! If you want to be nannied, that's your choice, and you can go use AOL or whatever, but we don't want the majority of ISPs controlling mail delivery in this way. Even if their intentions are good, 'proper' e-mail could easily get thrown away, and worse.. if laws were passed that allowed governments to control ISPs in some way, they'd have a system already in place to 'control' mail delivery. No thanks!
The answer to this question is that any freedom loving citizen should be filtering their own mail and not relying on a nanny state to sort it out for them.
mogorific carpentry experiments
If you want your email to go through then stop using open relays.
Can you be specific about what ISPs are doing this?
I am not aware of any ISP that filter SPAM based upon multiple emails from the same source. That seems pretty stupid to me. Are you sure you're not using an open relay that has been black listed?
First, I have to state that anyone suggesting you throttle/limit outgoing emails isn't thinking of the very large numbers involved here. If I have 100,000 subscribers, then I have to send more than one email per second, for 24 hours, to send all the messages. 60*60*24 = 86,400. If it's a daily newsletter, then I need to take less than 24 hours to send each newsletter.
/var/log/mail (which you're now storing in a database) to see what happened to the email. If the system says "delivered", then it's a problem for the user to take up with their ISP.
Upon opt-in, issue each user a user identity (some random alphanumeric widget). Have a web page on your site that allows a member to enter their identity, and then a little CGI program parses
This is going to take a LOT of user education, but it's going to solve problems slowly over time. The emails that get dropped, if the user notices, will at least give your level one support something to go by. "Yes, our logs show that our mail server has delivered the newsletter to you on these days.... You didn't get it? Could you contact your ISP, and ask if they are filtering inbound email? Here, we'll email you the logs to pass along to your ISP, or you can get it from the web site."
To be polite, you could make the mail logs even more public, allowing the ISP to look up things, but you'd have to "sed" out email addresses, or at least obfuscate them (like everything left of the @ gets replaced by X's).
At the very least, it moves the technical problem from something vague behind the scenes to something more easily described, and seen, and comprehended, by the user. And it allows you to point the blame finger at the guilty party.
Finally, during the sign-up page, and on the troubleshooting pages you give to users, mention that if the newsletter doesn't arrive, a likely cause is their ISP. Give a top 10 list, based upon the problem frequency reports. (User changed email address, local mail filtering, ISP mail filtering, network outage....)
We're in the same boat. We're a small ISP and we run a list server for our clients. Some of the stuff they send out is so amusing, even I sign up for it.
. ht ml
/dev/null by the big guys.
What we've been doing is verifying our email lists (this goes a long way to avoiding getting flagged as a bad guy) and sending messages out one per connection. It's fabiously inefficient and it takes 4 hours to send out 12,000 emails (our biggest customer) but we've only managed to tick off about 3-4 other ISPs.
There's two things that I see as being issues that we're going to have to deal with soon in a real way:
1) Little Napolean wannabe sysadmins at other small ISPs that belive anything sent to more than one recipient is spam. These guys really irk me. Its one thing if their customer complains about mail from our domain and they evaluate the situation and block it but it's another for them to see a message destined for more than one mailbox on their domain and arbitrarly decide to reject all mail from our mail server (not just the domain that sent it mind you; ALL the domains we host.) Heart's in the right place but they left the lens cap on thier mind. I've tried talking with them but that just seems to iritate them more.
2) Big email hosting companies (Yahoo, AOL, MSN, Hotmail) looking to make yet another buck. Take a peak at these headers on a bulk email I got from Yahoo:
X-YahooFilteredBulk: 209.164.21.221
And this page from the Yahoo help desk:
http://help.yahoo.com/help/us/mail/spam/spam-17
Now don't get me wrong, I love (well, like) the bulk mail folder on my Yahoo account. I'm just waiting for these companies to decide to offer "Prefered Sender" subscriptions that will garante delivery to thier user's Inbox or maybe Prefered Partners Inbox or something. What are we (small ISP's) going to do then? We're not going to buy a subscription from every Yahoo/MSN/AOL out there and we can't serve our customers well if all thier lists get piped to
SpamAssassin
I'm not involved with this group, but from what I hear of other ISPs implementing this, it works well. It allows you to set headers based on it's own message rating system, sends checksums of messages that it thinks are spam to a clearing house (DCC), and uses checksums that match 'mass' email that have been rated as spam to mark messages that have been sent to a lot of people. This lets the user filter the garbage to a folder in their MUA if they want. It can also delete them server side.
Someone that uses this please correct me if I'm wrong.
[%- PROCESS life -%]
I'm one of the SpamAssassin developers and I find their technique odd.
;-)
Wouldn't this have a horrendously high false positive ratio for things like mailing lists?
Anyway, tell them to use SpamAssassin - it kicks ass. And I'm not biased, honest
Matt. Want XML + Apache + Stylesheets? Get AxKit.
I'm sending to a single valid email address which I have sent emails to successfully before and since. When I try to contact Yahoo! via the postmaster account, I get a form letter thanking me for telling them about spam!
Email is never going to get fixed. The fundamental concept is flawed. You can't allow arbitrary messages from arbitrary anonymous sources without getting spam. Probably well over 99% of solicited mail is non-anonymous anyway, so the solution is simple, in theory.
Until anonymous email is deprecated the spam problem will not be solved, plain and simple.
I know it might border on heresy, but why not have the ISP actively manage the mailing lists? Here's an example:
Suppose I publish Gland Nut Weekly, and I use fatboys.net as my ISP. I register myself with the ISP, giving them the name of my mailing list, and the names/email addresses of the allowed publishers. When I have an issue ready to publish, I send it to fatboys.net, who then sends it to the current subscribers on the list.
Other ISPs can 'trust' that the email sent by fatboys.net isn't spam, since fatboys handles the mailing list, fatboys.net can be sure they're not a source of spam (and look like one of the good guys) since they're handling the mailing list, and the publisher benefits from having the ISP send the actual mail at high speed and without having to employ tricks to get around outbound spam filters. Whaddya think?
Tiller's Rule: Never use a word in written form that you've only heard and never read. You will end up looking foolish.
Many large ISPs are implementing anti-spam filters based on how many emails they receive from a single sender
Can you qualify this please? How many is "Many"? Two? Four? A hundred?
Worse still is that there is typically no error message returned to us - the emails simply get dropped
If this is true, then their mail servers are misconfigured, or your return address is wrong.
Are you sure you're not screwing up? Can you post your mail server logs showing that delivery has taken place?
If you're not getting bounces, then the ISP's are really accepting your email - which pretty much defeats the anti-spam logic (the whole point of anti-spam is to prevent mail transfer - which according to you, they're not doing.)
I'd guess that it's a problem with your equipment, or your mailing list software. Either your return address is wrong, or your mail server is dropping the mail instead of delivering it.
Sometimes, ISPs will add us to their "white" lists
OK, so you've contacted multiple ISPs, who all have their mail servers misconfigured in the same way, and you're convinced there are still more out there..
I think maybe the problem is at your end.
I think that while it is valiant of ISP's to try to block spam as early as possible, it goes against their duty to provide an unfiltered connection. Furthermore, clients might actually _like_ to receive spam (like the guy who actually replies), or be friends with lots of spammers, or just generally not have an objection to spam.
That said, I also think that all emails should be PGP signed, and all emails that fail in THAT regard should be summarily filtered... (of course the process to get there could be as gradual as having the email client flag unsigned messages as "suspicious", yadayadayada... so as not to shock the masses with a sudden change... blah blah)
Becuase, of course, spammers are too stupid to download PGP and make a key.
Why on earth does this pop up in any anti-spam discussion? PGP signing simply means the sender can prove it was from him. It doesn't mean you know who the sender is.
If you want to set up some sort of whitelist, it makes just as much sense, and takes much less space, to say 'I will accept email from blah@mail.dom, and only if it arrives via mail.dom or dialup.dom.'.
If you want to do something useful with PGP, you could make something where you auto-whitelist anyone who has a key signed by someone you trust. That's about the only way PGP can help fight spam.
If corporations are people, aren't stockholders guilty of slavery?
I suggested something like this a while ago. Server side filters accessable by ordinary users. People here said they have those, but misunderstand. Most server side mail filters apply to ALL accounts and are not accessable by users who have pop accounts. In fact I have not heard of an ISP implementing such an idea and I claim this as prior art for such an idea so don't even think of patenting it I'll sue.
It's simple, a users logs into their isp with a web based app that allows them to say filter out this that and blah. I'd use mail headers, and filter out korean character sets as that is where most of my spam lately comes from. Funny I can't even read it but the charset says korean.
I am leaning alot about smtp / pop and basically the only requirements are HELO, MAIL FROM, RCPT TO, DATA, QUIT, USER, PASS, etc. The protocols themselves are too stupid for most else. Filters on the server could also interfear with privacy. In order for them to filter mail they would have to have a mail scanning program. If they log this data then it becomes an provacy issue.
The real solution is better mail filters in the pop mail cleints. For a delete filter it may be better if the pop client were to call TOP and get the message header and then delete the message appropriately. I am working on a java implementation of this. My POP3 bean can do this, I just need to scan the headers.
Only 'flamers' flame!
"just" require all SMTP traffic to use TLS, and have them all under one CA, so everone can test the authentication of the sender .. of course .. this is only a pipe dream ;)
However, you luckily aren't on any blackhole lists. Yet.
And it's a problem with your mailer. All anti-spam software returns errors to your mailer when you connect, or bounces the email. It wouldn't drop them on the floor, that's not discouraging you at all, you'll still keep sucking up their bandwidth, as you can't possibly know they're being dropped.
Ergo, your mailer does not understand the 5xx reply they are sending. You need to report it as a bug.
If corporations are people, aren't stockholders guilty of slavery?
It's simple. Users have whitelist, ignorelist, and blacklist. Anyone on a list gets the appropriate response. If you're not on a list, you get a confirmation email before your message gets to my inbox. This kills virtually all spam.
o rge.net/projects/a-s-k/
http://www.paganini.net/ask
or
http://sourcef
I am a freedom loving citizen who has freely chosen to drop my prior ISP because they did not filter and who has freely chosen to move to a new ISP because they do filter. The fact that you don't like it is irrelevant; I am exercising my liberty by selecting an ISP willing to act as my agent. Were your friends the spammers to make blocking by the ISP illegal, that would be an infringement on my liberty.
What duty? Their duty is whatever they promise in their contract and UAP. If they promise to filter, then their duty is to filter.
If you have a permanent connection, are allowed to run your own mail server and can afford the time to administer it, that might be your best solution.
btw, you're a fucking retard.
Ahh.. but I said earlier that spam was really a non-issue for me. The reason is that I get so much legitimate mail that spam becomes white noise which is easy to filter & delete without thought, and it is quite easy to set up individual rules in kmail (or whatever you use) to filter out, say, anything from the .cn domain if you know that no chinese mail to you is going to be legitimate (eg. if you don't speak chinese).
And you are exactly right about keeping a list of valid PGP signatures, since the one thing I don't want a spammer (or con artist) to be able to do is fake being someone I know and trust.
DISCLAIMER: THEORETICAL EXPLAINATION, I HAVE NO CLUE OF LEGAL ISSUES.
... this is guaranteed to get the attention of the administrator. ... whatever number of SPAM you receive, you dish more out ... essentially, the admins of the spammer gets as much spam as the total amount of spam received by your list, guarnateeing their attention.
Actually, I have a very good idea.
I'm not too sure if it's technically legal, so I'll discuss it in theory.
Facts:
(1) The mailing list has numerous individuals (hopefully 1000+).
(2) SPAM was sent to the majority of the individuals on the mailling list.
(3) The spammer's address can be found, (or if faked, traced).
The idea, of course, is similar to fillibuster's in Congress, or massively sending petitions to the government; it operates as follows.
(1) When 1/3 or more ppl get spam, Moderator takes action.
(2) Moderator writes a long complaint to the administrator of the spamming server.
(3) Moderator distributes the email on the mailing list, now everyone on the mailing list "sends" the email, with their own name instead, to the administrator of the spamming server.
This is effective since
(1) The emails sent to the given administrator comes from different domains, so there's no easy filter (if the filter subject, you can tell everyone to change the subject a bit)
(2) This creates a slash-dot like effect; slashdot is mostly legal DOS; this would be possibly legal "mailbombing?" (word choice?)
(3) You pass the damage on
Hope this helps, again, this is all theoretical
I'm assuming that the spammers want to sell you something. If they're just out to create trouble, I doubt this'll work. On the other hand, if they're out selling something, they must include (1) web address (2) phone number (3) fax number or (4) email address Have a policy on your mailing list that whenever the user receives any crap, they phone the company, ask as much BS as possible, and they quit; print out the email, write "STOP SPAMMING ME" on it, and fax it; and foward the spam to the given email address (to buy the stuff). This guarantess that (1) the emails are coming from different places, so there's no easy way to block it (2) the company gets as much crap as it dished out, and probably won't want to do it in the future. The key is that people take an agressive approach and fight back, instead of just deleting the emails.
Plus, that's easily solved, if they actually start doing that, by saying 'I will only accept mail from whoever@server.dom, and the only machine that can send me that mail is server.dom.'. If someone has a weird situation where email doesn't arrive from the machine server.dom, you simply give them an exception.
PGP signing is so that you can prove later they sent it, not so you 'know who it's from', it's trivially easy to figure out if an email is from someone you know just by looking at the headers. If a friend always PGP signs his email, sure, accept that as proof it's from him. But don't make everyone start signing things, being from the right server with the right email address is proof enough it's not a spammer.
If corporations are people, aren't stockholders guilty of slavery?
it's trivially easy to figure out if an email is from someone you know just by looking at the headers
ermm.. not really. maybe i'm just paranoid, but AFAIK the best headers can do for you (without disruptively contacting system administrators to discover mac addresses) is narrow down the subnet that the message came from. Most ISP's that i am aware of have open smtp relays within their subnets.. i.e. anyone within the subnet could pretend to be anyone else within the subnet and nobody could know the difference.
Granted that most Outlook-using users and spammers wouldn't have a clue how to do this, but anyone who can understand the command-line syntax for sendmail can do pretty much whatever they please.
From 'spamassins web site'
The 'user agent' it the users mail program. This means that the users is not filtering out the data on the server. The server is only 'tagging mail'. The user still has to download the whole mail. Obviously your to stupid to understand a thing I am talking about. I am talking about a filter on the mail server that I set up that delete the mail from my inbox and I never ever see it. So in my case I would create a filter that says 'delete mail where charset like "korean"', then all mail that is coming from korea is deleted form the web server when it arrives at the pop mail account on the mail server.
My ISP uses the spaminator which reduces my spam by over 50%, but it is still not a filter that I set up for my account on their servers.
Its obvious from your post that it doesn't require brains to post on slashdot.
Only 'flamers' flame!
it is possible to execute the TOP command and download the headers of mail and from the mail headers have it delete mail based on that. TOP 1 0, gives me just the mail headers. If I have 20 spam messages and I just get the headers of them I can delete all the spam and not download the whole message. I do this through my web based application that I have where I display the inbox I only get the headers. Maybe the soultion is to leave the mail on the server and only get the headers in the mail app and then select which messages I want to download after that. I could also set up filters based on these headers so that I never see the messages in my inbox that have lets say a character set that is in another language other then my own preference.
headers are usually less than 1k, but html spam is usually several k. This would cut down on my download time.
Only 'flamers' flame!
This is so far from reality I don't know where to start. Spammers run software that looks for things like blah@example.com. This is the entire extent of their 'finding email addresses'. They not only don't do any of these complicated things you're talking about to figure how to get in past one address, they don't even filter out obviously wrong addresses. Spammers sometimes send to Usenet Message-IDs, which only look like email address if you're just globbing *@*.???, and don't bother to look and see it's jf3224-usieof.disuwod@example.com.
If it takes a spammer an hour to send a message to a person, they've lost and we've won. Hell, if it takes a spammer one minute to send a message to someone, we've won. Spammers are sending out something like a million messages each time, and each run needs to be done in a few hours.
If corporations are people, aren't stockholders guilty of slavery?
(warning, I reordered your message a little bit)
If it takes a spammer an hour to send a message to a person, they've lost and we've won. Hell, if it takes a spammer one minute to send a message to someone, we've won. Spammers are sending out something like a million messages each time, and each run needs to be done in a few hours.
I agree with you at least this much.
So, you're assuming spammers are sniffing your email and finding out not only the names and address of your friends, but what headers they send with their message, and searching until they find an open relay within the right subnet so they can send using the same SMTP server as your friend?
ArggggghhhhH!!! NO! I said already (several times) that I come on the side of not particularly caring if I get spam. Bandwidth isn't even an issue for me since newer clients (like the newest kmail) can filter based on subject and sender while the email is _still on the server_.
All I want from my email is to know (beyond a reasonable doubt) that the person who sent it to me is the person I think it is. I also want to know (beyond a reasonable doubt) that it would be impossible for another person to forge an email from me to someone else without that email being red-flagged as suspicious.
However, if the above properties were true of email, it would be very hard for spammers who send gazillions of anonymous emails to get any attention, since those emails could be sent into an "anonymous" pile which rarely gets looked at (since it's full of spam).
The other emails are PGP verified in a way that should not reveal the email address doing the verifying, eg. the final server could verify the authenticity of each incoming email, valid or invalid, and modify the headers to reflect the authenticity of lack thereof.
Once a client receives an email, one of the things it would be able to do is look at the headers to see if the email is valid or invalid, and react accordingly by sorting or doing whatever user-defined action it is supposed to do. Older clients can hopefully just ignore the strange new header. If a person reading an email is particularly interested in knowing if an email is valid or invalid (i.e. if they think the server might have made a mistake, or they don't trust the server), the person can click on a button that checks the authenticity of that message manually. The other thing that the person can do now (which they couldn't do before because of anonymous emails) is COMPLAIN about the unsolicited email, and have a solid line of accountability leading straight back to the spammer's server.
At the very least, the problem for spammers would have moved from finding open smtp relays to finding open httpd servers (much harder to find)....