Your network is probably still providing some service to a spammer in some way. The requirement of SPEWS, other than for first time spammers (i.e. this means any services to any repeat spammers), is that absolutely every service be terminated with no exceptions. This not only includes IP access through which they may spam, but also web hosting, DNS hosting, phone service, office space rental,... everything... period. Now if you really have done all that, and posted a description of exactly everything that was terminated (don't just say you did, admit to what services you provided and when that service was terminated), it should get read by one of the SPEWS team, who can check the database.
But you do need to realize that SPEWS does have a punitive element. If you kept providing services to a known spammer for N months, expect SPEWS to delay your deletion for N months.
Also, many people have mis-interpreted the SPEWS listings. Level 1 means listed, and level 2 means probationary. If you are on probation, it is because you delayed long enough to get your network listed (you should have disconnected the spammer before that happened). Level 2 is not listing to be blocked. A few networks choose to block based on level 2 for extended punitive purposes. You know who they are (from your mail server logs), so complain to them for mis-using SPEWS.
Provide some specifics, like which network this is, or which SPEWS record number, and I can look up some of it (my archives of the public data from SPEWS cover 7 July 2002 to 15 August 2003).
Yes, many have the entirety of Brazil blocked. And for good reason, too. Doing so cuts out a huge chunk of spam and reduces the costs on the receiving mail servers and networks noticeably. It works.
The problem is that most of Brazil is served by one big telco monopoly that is operated entirely incompetently. That doesn't necessarily mean each person in that company is incompetent, but those that are not are surely aware of their inability to do the right thing and stop the spam.
Some people even blocked all of 200/8.
Now I don't actually agree with the actions those people did. What I did was scan those networks for patterns and figured out specific domains to block. I'm getting most of the effectiveness without the false positives. I do have almost all the cable modem and dynamic DSL lines blocked as best as I can.
But the real goal is to get spammers disconnected so they can't even send a SYN packet, much less make an SMTP connection. You have a better idea that meets those goals that what is being done now? If so, post it.
The DNSBL blacklists have been immensely effective against spam, spammers, the ISPs that harbor them, and the customers of those ISPs that subsidize them.
The only enemies these blacklists have made are those they are intending to block. For example, if you are a customer of an ISP that harbors spammers, then you are subsidizing spammers, and blocking you is intended. So of course I'd expect you to hate them.
The effectiveness varies from one ISP to another. While there are many ISPs which continue to harbor spammers despite being listed, many others have decided it is bad business to do so, and have ejected their spammers, in many cases at significant loss of revenue and significant risk of legal action by those ejected spammers. But they did the right thing by preventing those spammers from stealing resources like my network bandwidth and by mail server processing capability.
Those large ISPs may very well be participating in these attacks, or are being accomplices in it by refusing to backtrace to where the attacks are coming from. Expect more people, not fewer, to be blocking those ISPs in the future. It is the very network itself that wants spammers, spammer harboring ISPs, and their subsidizing customers, to be destroyed.
An opportunity to fix this will be coming. At some point (really in the past 5 years) we need to deploy a last-mile fiber infrastructure. This infrastructure should be designated to handle everything to everyone, with several dark strands to every house or apartment, allowing the ends to be hooked up as desired... including cable TV (this would sooooo solve my 2m and 70cm RFI problems). Then many businesses can vie for servicing customers via the central endpoint, and homes and businesses can sign up for more than one, or even rent dark fiber interconnects for their own private usage at the same per-hop price as anyone else pays (plus the setup costs).
And for God's sake, let's keep that broadband trash off the power lines.
You should check into the issue with the California PUC (or whatever it is called). These remote vaults, if they are end points for the pairs, should be available for all competitors.
Consider my opinion on how the breakup of the phone companies should have been done. The break point should have been between the infrastructure (e.g. all the wiring, punchdown panels, and the buildings and vaults the wiring goes into), and the "dialtone" (e.g. switches or any other equipment that activates the wiring). Too bad that in the 1980's when the breakup happened, people were not clear about how local and long distance access were really essentially the same service. And the use of things like DSL by the masses for internet was unknown then. Now even though my idea of a correct breakup didn't happen, I think people can still look at that to understand the two roles the incumbent telco has. Their "infrastructure role" is what they really need to share. And anything that lets them tap into a pair anywhere along its path to amplify it either has to be part of the infrastructure (and thus shared), or at the very least has to be doable by any competitive carrier (e.g. they get to drop in their own equipment anywhere along the lines to boost the range). What I see would be an issue there is that the incumbent would not like for anyone to be tapping in on the line (you've seen the 1000+ pair bundles, I'm sure... you wouldn't want to let just anyone touch those if you ran the telco). So in the end I think any distance boosting equipment is going to be part of the infrastructure. We need to make sure it gets shared (at least on a whole pair basis, even if we can't always get the incumbent to split the line out between DSL to a competitor and force the incumbent to do the POTS).
How many years ago was this? I'd like to see specific data. I've seen cases where SPEWS listed networks the admins thought should have been delisted, and it turned out there was a problem remaining (such as DNS hosting for a spammer still operating which would still get that ISP classified as a spammer haven). Many times ISPs think that because the SMTP traffic doesn't come through their network that they can't be harboring spammers. But SPEWS goes after any services provided to habitual spammers. I even read a case where a commercial real estate company was listed because they rented office space to a spammer. I don't have access to current SPEWS data, but I do have the last year of listings archived.
Many people consider spamming a form of theft already covered by existing law (and hence, feel that no new laws are needed, and only knowledgeable enforcement is what will help clean up the net), which just happens to not be enforced by the legal authorities because they don't understand where and how the theft is taking place. As far as I could tell, SPEWS considered it this way as well. And providing any resources whatsoever to a spammer whom you know is a spammer (as determined by whether that theft has taken place and been repeated) would be cause for being part of the SPEWS "boycott" (which is really more like a partitioning of the network between the spammy part and the clean part). What made it difficult to get unlisted is that it required posting in a very noisy USENET forum (NANAE), and required the poster to provide information they often didn't know was needed and had to read around the noise and find out what more needed to be posted to clear the matter up. Unfortunately, many didn't even bother to read any of the followups, and many situations were "dropped".
If you have all your userbase known to the secondary MX machine(s), such as it being small and replicated (probably your case at home), or via LDAP or other database, then it's actually easy to do a secondary MX and have it not be accepting mail it won't be able to deliver to the primary. Just be sure to set its queue limit very very long (like 1461 days) so it never bounces anything.
Are you referring to a business that has a monopoly because of regulation? Or is there competition in the DSL market there? There are things to do about it but what that is depends on the particular circumstances. Is SpeakEasy available there for DSL or T1? I've heard a number of people have gotten static IP with reverse DNS from them (but I don't know whether that is delegated or hosted).
You still have to achieve nothing less than a 100% impact to make it work. As long as there is some way to slip through, spammers will do so. They will be more clever in hiding, and will be in foreign jurisdictions. And if you go after the people that pay them to spam, the ones that are advertised, the spammers will do promotions of some businesses that never did anything (except maybe piss off a spammer) just to get you to attack them. As long as spammers know how you will react to what they do, then they control you.
... SPEWS, the (in)famous blacklist that got spammers mad enough to launch a massive attack, will new be distributed to end users via P2P file sharing networks.
What makes you say they were doomed to suck? I found SPEWS to work quite well. Oh, and when it listed an ISP that harbored spammers, I cheered. And when some whiney land shark comes along and doesn't understand that it's a boycott against his ISP for harboring the people that keep on stealing my network and server resources, I laughed. And I tried to explain it. And if they didn't move on to another ISP (and I don't want to hear excuses about that not being an option, because it was an option in 99.9% of the cases), I laughed even more.
I found SPEWS to work quite well. What was your problem? Were you using, and getting IP space from, an ISP that was harboring the spammers that keep attacking my network?
That was apparently the goal of the spammers who launched the attack. Take down the DNS blacklist by DDoS-ing it so much that it has to get shut down (not just unavailable, but literally switched off, as the case is now).
There are actually two different anti-spam goals. A few people have both of these goals, but quite many people have only one or the other:
Prevent the spam from entering my mailbox.
Prevent the spam from using my resources (or my company's, or my ISPs).
The first goal includes such things as making sure children and sensitive adults don't see porn spam. But lots of people are simply offended by the spam, especially porn or body part enlarging spam. And others are simply offended by someone assuming they were interested in a great money saving offer for something they have no need for. This first goal seems to be what most people have, and what the current political rumblings are about.
The second goal is one a lot of people are not aware of, or don't understand. yet it is as serious a goal, if not more so, by certain groups of people. This involves reducing the network bandwidth and server processing resources used by the spam, or stopping it entirely. These things cost money, and it costs about 10 to 40 times as much money to receive (delivered) spam as to send it. It still costs 5 to 10 times as much just to take the SMTP connection, carry out the talk, discover it's a spammer, and refuse the spam.
In other words: the spam problem is not solved by blocking spammers... just reduced in cost a good bit.
Solutions that involve scanning spam content for the nature of what spam looks like does not help reduce the costs at all. In fact it increases it because all this extra processing is now done by the server, and the network bandwidth is used to send the content that might otherwise not have been sent.
To those, like myself, whose goal is to reduce costs, SPEWS was a great tool. It was very effective in blocking spammers, plus it forced quite a number of ISPs to terminate the spamming scumbags that slipped into their networks under the guise of legitimate customers. In that way, it worked; it did what it was supposed to do. Too bad a few other ISPs were too stubborn to deal with the problem, and too many customers of spammer harboring ISPs whined more about why SPEWS was targeting them, and making excuses why they could not switch to a decent ISP (excuses that didn't apply in 99.9% of cases). Unfortunately, quite a lot of people simply never "got it" as to what the purpose of SPEWS was. The SPEWS web site was more geek/admin talk, and not well enough written for the average person to understand. I was starting to work on my own "how to get out of SPEWS" document, but I just haven't had time to put in on it.
There are a lot of things people say as to how to stop spam. The one I hear most often is that if people would just delete the spam, or if network admins would just block only spammers and no one else, then spammers would cease making money and would stop. This is simply not the case. First, not everyone will do this. We see from these recent worms and virii that way too many people don't patch their computers anyway. There will always be gullible people who respond, and there will always be spammers to take their money.
The real way, and I think possibly the only way, to stop spam, is to treat all spammers as equivalent to cyberspace terrorists. Take no prisoners, and take no excuses.
Remember, spammers don't care what people who will never respond do with the spam they send. They don't care if you press delete, or filter it out with SpamAssassin, or even block them. They don't care because you aren't going to make any difference to them anyway. And if you do block it, you won't be complaining to the spammer's ISP, and hence, they get to spam even more. To a spammer, someone who blocks their mail is better than someone who gets their ISP account terminated. This is part of why just blocking spammers is actually making the problem worse.
Spews was an excellent solution. It wasn't perfect and a few mistakes were made. The fact that the real operators had to remain secret due to all the lawsuit threats did make it difficult to provide feedback to make corrections. I predict SPEWS will be back, but in a different form, possibly as a distributed file of sites to block... which will make it even harder to get removed since it will then not be operating as a live database.
Much of the problem was because a lot of people didn't understand that the purpose of SPEWS went beyond just blocking spammers (which will not accomplish stopping spam), but actually blocking the ISPs that allow spammers to continue to operate and continue steal resources from networks and mail servers. This was in effect a boycott of that ISP, and it was intended to drive customers from that ISP to other ISPs that do not harbor spammers. In many ways it was working because it clearly got a lot of spammers upset, and a lot of ISPs upset as well. I even believe it is possible that the DDoS attack on OSIRUSOFT was caused by many of these ISPs.
My question to you is, did you understand that SPEWS was blocking whole ISPs, not just spammers? You don't have to agree with that method or principle... just understand that others do think it is right, and understand why they do.
That's one reason I quit running backup MX. Why should I have to reject mail being delivered from my own (backup) mail server. By not having any secondary MX at all, if I do go down, the sending MTA just queues it for a few days. But spamware generally won't queue it, so that's not even a worry at all.
The bandwidth may be getting cheaper, but it hasn't been keeping up with the spam volume. Cost is a real factor when running internet services in a cut-throat market. So I don't think the issue is just eyeball protection. In fact I get a chuckle out of some of the spam that does leak through.
As for the reverse DNS, that's a problem between your ISP and you. I do not accept the excuses. Their service might be cheaper because they don't want to hire the staff to run it right. Reverse DNS is trivial once you have a static IP. While it is possible to do without a static IP by using a dynamic MX, you're in for problems that way. I hope at the very least you are constantly complaining to the appropriate government agency out there about how it is you still can't get decent internet service at a decent price on DSL because of the monopoly SBC holds, combined with their incompetency.
Of course we know that as long as bulk mailers are making money (I have my doubts about the people who are paying them to do the mailings), many ISPs do want a piece of the action. Those are the ISPs we need to boycott.
I don't know what (or if) your role at Rutgers is. But I can say this. If the departments won't report MTAs, then things are out of control. The central networking policy can in fact fix that with proper advance notification. They can always "smart host" forward to the central campus mail server. If they want to be able to connect SMTP direct, then access-list permit them to do so (of course that means they have to tell you, or whoever handles that, what the address is). If you don't want the access-list cluttered with a bunch of individual addresses, then set up a virtual subnet and access-list permit that, and route all those addresses around campus as/32's. Everything else will just have port 25 blocked (but not port 587, so you can still allow use of outside mail services when those services have deployed the message submission protocol as defined in RFC2476, which is AUTH-SMTP compatible).
I'd even be willing to let the ISP block mail traffic at the IP level as long as the users have the opportunity to override it. So instead of just dropping all packets from a given IP address, the SMTP server allows the transfer to proceed until all the recipients are given. If all of the message recipients have previously agreed to block the IP address in question, then the transfer can be terminated at that point without transferring the message body. This could be done by answering the DATA command with a permanent error code, or just resetting the TCP connection. Otherwise the message is accepted and delivered only to those that haven't requested that it be blocked.
Actually, you can perform the test during the RCPT command for the tuple of recipient and what IPs they want blocked. Then you can 550 the RCPT command if that user wants that IP blocked. There's no need to have all users agree.
Rejecting as much mail as possible during the SMTP session allows 5XX responses in place of queueing a bounce message (which in most spam cases can't be delivered anywhere anyway, so it sits in the queue for whatever the duration is set for).
My eventual goal is to set up a control panel for end users to define their own reject/permit policy, which will be done during the SMTP reception session. They will be able to choose their own DNSBLs or none at all. They will be able to choose defined sets of blacklists, or create their own. They will be able to whitelist, as well, of course. Or they can simply delegate the policy to someone else (there will be a default delegation). I just need to figure out a good schema to store all this policy preference data (needs to be easy to access from both the SMTP daemon and the web control panel logic), and do some SMTP daemon hacking to make it play.
How is a Bayesian classifier going to help me reduce the amount of bandwidth used by the spam? How is a Bayesian classifier going to help me delay the point in time when I have to upgrade my mail server to keep up with the load?
Actually, it's going to cost me more to deploy such methods. I presently block a huge amount of spam by a combination of private blacklists and DNS based blacklists that are applied before the DATA phase of the SMTP session. To switch to a content based test means I have to incur all that data transmission for all the cases where the promotion of that method says that my existing blocking is inappropriate (some say I should do it for all email, and some say I should do it for some subset, such as mail coming from cable modems and such).
The issue really comes down to what one's goal is with regard to spam blocking. Some people have the goal of keeping filth out of their mailbox so they (or their children) don't have to see it. Some people just hate getting email offers from someone who had no reason in the world to believe it might be wanted. But my goal is to minimize my costs by keeping down the bandwidth usage and the server resources needed.
It's been said many times before: spam is about conSent, not conTent.
So I also believe that the testing for spam should relate to what it is that makes it spam in the first place. So I don't test on content. I test on who it is that is sending to the best extent I can, which means it is based on sender address (for those few spammers who keep using blocked sender addresses) or client address (in addition to blocking by DNSBL and IP ranges, I also block by reverse domain names, and by default refuse from any client with no verified reverse DNS).
Of course there are false negatives. If a sender is unwilling to go to the effort to distinguish themselves from spammers, then I really don't worry about it. I've examined many of these false negative cases, and I've found that they are all some form of the sender not willing to make the effort to send email in a way that lets me know during the SMTP session that it really is from them. Since the costs of email are so heavily slanted towards the receiving end, I think it is only prudent that senders make some effort. Simple things like proper reverse DNS on their mail server can help a lot (and whining about their ISP refusing to do it for them only evokes my rant about changing to a better ISP that isn't being subsidized by spammers).
Many people have spoken about whether or not we should improve SMTP. Maybe we should. Or maybe SMTP is good enough as is. I've found it adequate, but would welcome some improvement. But the improvement that I would see as working would simply be something to more reliably identify who is sending. Forward verified reverse DNS at least lets me pin it down to a registered domain name. Then at least I can whitelist specific domain names, such as ka9q.net (it is whitelisted so if your reverse DNS is set up right for your own mail server, it should be able to deliver here just fine... I trust you won't run open relays or open proxies).
My ability to whitelist shows I can in fact opt-out of any blacklisting I have in place, including slicing parts of my own blacklists. Even if your upstream IP address source (ISP) is entirely blacklisted by every DNSBL, as long as your reverse DNS is correct and names ka9q.net, your mail from there can get through. Spammers can't forge that without hijacking your domain since I test the address with a forward lookup. So I meet your requirement to be able to opt-out of others' listings.
As for letting users have control, I do that, too. My user base is still small enough that it is done manually. There are a few whitelistings my users have requested. One user has elected to opt-out entirely from all anti-spam measures.
The rates at such ISPs often are lower because the ISP is getting other revenue sources from spammers. And of course, when lots of people ignore that and choose that ISP, they are also helping to lower the costs to the spammers, too.
I do agree that AOL should do a better job of letting you show them that you have terminated that customer that sent those 3000+ unsolicited emails. You did termate that customer, right? If you want to understand why they blocked everyone, the reason is the technology is more efficient to route connections by their source address to specific mail servers that filter content. Servers not in their blacklist get routed to streamlined mail servers that don't do that filtering. I don't know what they actually do, but that is plausible, and I do know that some kinds of tests on email do cost more resources to do than others.
AOL's practice can certainly cause problems when there are thousands of AOL customers subscribing to certain mailing lists. When a mailing goes out, AOL gets briefly slammed with all those copies (although 3000 is certainly only a drop in the bucket considering their total overall volume).
It is your responsibility to make your mail server distinguishable from the unwashed masses of cable modems, DSLs, and dialups with dynamic IP allocations. If you are dynamically allocated, you shouldn't be running a mail server. If you have static IP, you need to have your own domain name in reverse DNS (no excuses accepted because reverse DNS is easy to do).
I don't know to what extent AOL blocks by IP or by domain name. I for one block by domain name. And I have most ISP dynamic/generic addresses listed. So if your reverse DNS name looks like a numbered sequence like all the others, then welcome to my blacklist (I can whitelist you if you tell me your IP address). Blocking by domain name actually works better than blocking by IP address because if the ISP changes the purpose of an IP address, all they have to do is make the DNS name match, and voila, not in the blacklist. Get your reverse DNS name as your domain name, and unless you really were spamming from it at some point, you should be in the clear.
Slashdot Mirror
Your network is probably still providing some service to a spammer in some way. The requirement of SPEWS, other than for first time spammers (i.e. this means any services to any repeat spammers), is that absolutely every service be terminated with no exceptions. This not only includes IP access through which they may spam, but also web hosting, DNS hosting, phone service, office space rental, ... everything ... period. Now if you really have done all that, and posted a description of exactly everything that was terminated (don't just say you did, admit to what services you provided and when that service was terminated), it should get read by one of the SPEWS team, who can check the database.
But you do need to realize that SPEWS does have a punitive element. If you kept providing services to a known spammer for N months, expect SPEWS to delay your deletion for N months.
Also, many people have mis-interpreted the SPEWS listings. Level 1 means listed, and level 2 means probationary. If you are on probation, it is because you delayed long enough to get your network listed (you should have disconnected the spammer before that happened). Level 2 is not listing to be blocked. A few networks choose to block based on level 2 for extended punitive purposes. You know who they are (from your mail server logs), so complain to them for mis-using SPEWS.
Provide some specifics, like which network this is, or which SPEWS record number, and I can look up some of it (my archives of the public data from SPEWS cover 7 July 2002 to 15 August 2003).
Yes, many have the entirety of Brazil blocked. And for good reason, too. Doing so cuts out a huge chunk of spam and reduces the costs on the receiving mail servers and networks noticeably. It works.
The problem is that most of Brazil is served by one big telco monopoly that is operated entirely incompetently. That doesn't necessarily mean each person in that company is incompetent, but those that are not are surely aware of their inability to do the right thing and stop the spam.
Some people even blocked all of 200/8.
Now I don't actually agree with the actions those people did. What I did was scan those networks for patterns and figured out specific domains to block. I'm getting most of the effectiveness without the false positives. I do have almost all the cable modem and dynamic DSL lines blocked as best as I can.
But the real goal is to get spammers disconnected so they can't even send a SYN packet, much less make an SMTP connection. You have a better idea that meets those goals that what is being done now? If so, post it.
You are absolutely and totally wrong.
The DNSBL blacklists have been immensely effective against spam, spammers, the ISPs that harbor them, and the customers of those ISPs that subsidize them.
The only enemies these blacklists have made are those they are intending to block. For example, if you are a customer of an ISP that harbors spammers, then you are subsidizing spammers, and blocking you is intended. So of course I'd expect you to hate them.
The effectiveness varies from one ISP to another. While there are many ISPs which continue to harbor spammers despite being listed, many others have decided it is bad business to do so, and have ejected their spammers, in many cases at significant loss of revenue and significant risk of legal action by those ejected spammers. But they did the right thing by preventing those spammers from stealing resources like my network bandwidth and by mail server processing capability.
Those large ISPs may very well be participating in these attacks, or are being accomplices in it by refusing to backtrace to where the attacks are coming from. Expect more people, not fewer, to be blocking those ISPs in the future. It is the very network itself that wants spammers, spammer harboring ISPs, and their subsidizing customers, to be destroyed.
Can you suggest a good time-management trouble-ticket system that runs on FreeBSD, Linux, OS X, and Solaris?
An opportunity to fix this will be coming. At some point (really in the past 5 years) we need to deploy a last-mile fiber infrastructure. This infrastructure should be designated to handle everything to everyone, with several dark strands to every house or apartment, allowing the ends to be hooked up as desired ... including cable TV (this would sooooo solve my 2m and 70cm RFI problems). Then many businesses can vie for servicing customers via the central endpoint, and homes and businesses can sign up for more than one, or even rent dark fiber interconnects for their own private usage at the same per-hop price as anyone else pays (plus the setup costs).
And for God's sake, let's keep that broadband trash off the power lines.
You should check into the issue with the California PUC (or whatever it is called). These remote vaults, if they are end points for the pairs, should be available for all competitors.
Consider my opinion on how the breakup of the phone companies should have been done. The break point should have been between the infrastructure (e.g. all the wiring, punchdown panels, and the buildings and vaults the wiring goes into), and the "dialtone" (e.g. switches or any other equipment that activates the wiring). Too bad that in the 1980's when the breakup happened, people were not clear about how local and long distance access were really essentially the same service. And the use of things like DSL by the masses for internet was unknown then. Now even though my idea of a correct breakup didn't happen, I think people can still look at that to understand the two roles the incumbent telco has. Their "infrastructure role" is what they really need to share. And anything that lets them tap into a pair anywhere along its path to amplify it either has to be part of the infrastructure (and thus shared), or at the very least has to be doable by any competitive carrier (e.g. they get to drop in their own equipment anywhere along the lines to boost the range). What I see would be an issue there is that the incumbent would not like for anyone to be tapping in on the line (you've seen the 1000+ pair bundles, I'm sure ... you wouldn't want to let just anyone touch those if you ran the telco). So in the end I think any distance boosting equipment is going to be part of the infrastructure. We need to make sure it gets shared (at least on a whole pair basis, even if we can't always get the incumbent to split the line out between DSL to a competitor and force the incumbent to do the POTS).
How many years ago was this? I'd like to see specific data. I've seen cases where SPEWS listed networks the admins thought should have been delisted, and it turned out there was a problem remaining (such as DNS hosting for a spammer still operating which would still get that ISP classified as a spammer haven). Many times ISPs think that because the SMTP traffic doesn't come through their network that they can't be harboring spammers. But SPEWS goes after any services provided to habitual spammers. I even read a case where a commercial real estate company was listed because they rented office space to a spammer. I don't have access to current SPEWS data, but I do have the last year of listings archived.
Many people consider spamming a form of theft already covered by existing law (and hence, feel that no new laws are needed, and only knowledgeable enforcement is what will help clean up the net), which just happens to not be enforced by the legal authorities because they don't understand where and how the theft is taking place. As far as I could tell, SPEWS considered it this way as well. And providing any resources whatsoever to a spammer whom you know is a spammer (as determined by whether that theft has taken place and been repeated) would be cause for being part of the SPEWS "boycott" (which is really more like a partitioning of the network between the spammy part and the clean part). What made it difficult to get unlisted is that it required posting in a very noisy USENET forum (NANAE), and required the poster to provide information they often didn't know was needed and had to read around the noise and find out what more needed to be posted to clear the matter up. Unfortunately, many didn't even bother to read any of the followups, and many situations were "dropped".
If you have all your userbase known to the secondary MX machine(s), such as it being small and replicated (probably your case at home), or via LDAP or other database, then it's actually easy to do a secondary MX and have it not be accepting mail it won't be able to deliver to the primary. Just be sure to set its queue limit very very long (like 1461 days) so it never bounces anything.
Are you referring to a business that has a monopoly because of regulation? Or is there competition in the DSL market there? There are things to do about it but what that is depends on the particular circumstances. Is SpeakEasy available there for DSL or T1? I've heard a number of people have gotten static IP with reverse DNS from them (but I don't know whether that is delegated or hosted).
You still have to achieve nothing less than a 100% impact to make it work. As long as there is some way to slip through, spammers will do so. They will be more clever in hiding, and will be in foreign jurisdictions. And if you go after the people that pay them to spam, the ones that are advertised, the spammers will do promotions of some businesses that never did anything (except maybe piss off a spammer) just to get you to attack them. As long as spammers know how you will react to what they do, then they control you.
... SPEWS, the (in)famous blacklist that got spammers mad enough to launch a massive attack, will new be distributed to end users via P2P file sharing networks.
That's nice. Now explain how to make it actually happen, forkboy.
What makes you say they were doomed to suck? I found SPEWS to work quite well. Oh, and when it listed an ISP that harbored spammers, I cheered. And when some whiney land shark comes along and doesn't understand that it's a boycott against his ISP for harboring the people that keep on stealing my network and server resources, I laughed. And I tried to explain it. And if they didn't move on to another ISP (and I don't want to hear excuses about that not being an option, because it was an option in 99.9% of the cases), I laughed even more.
I found SPEWS to work quite well. What was your problem? Were you using, and getting IP space from, an ISP that was harboring the spammers that keep attacking my network?
That was apparently the goal of the spammers who launched the attack. Take down the DNS blacklist by DDoS-ing it so much that it has to get shut down (not just unavailable, but literally switched off, as the case is now).
There are actually two different anti-spam goals. A few people have both of these goals, but quite many people have only one or the other:
The first goal includes such things as making sure children and sensitive adults don't see porn spam. But lots of people are simply offended by the spam, especially porn or body part enlarging spam. And others are simply offended by someone assuming they were interested in a great money saving offer for something they have no need for. This first goal seems to be what most people have, and what the current political rumblings are about.
The second goal is one a lot of people are not aware of, or don't understand. yet it is as serious a goal, if not more so, by certain groups of people. This involves reducing the network bandwidth and server processing resources used by the spam, or stopping it entirely. These things cost money, and it costs about 10 to 40 times as much money to receive (delivered) spam as to send it. It still costs 5 to 10 times as much just to take the SMTP connection, carry out the talk, discover it's a spammer, and refuse the spam.
In other words: the spam problem is not solved by blocking spammers ... just reduced in cost a good bit.
Solutions that involve scanning spam content for the nature of what spam looks like does not help reduce the costs at all. In fact it increases it because all this extra processing is now done by the server, and the network bandwidth is used to send the content that might otherwise not have been sent.
To those, like myself, whose goal is to reduce costs, SPEWS was a great tool. It was very effective in blocking spammers, plus it forced quite a number of ISPs to terminate the spamming scumbags that slipped into their networks under the guise of legitimate customers. In that way, it worked; it did what it was supposed to do. Too bad a few other ISPs were too stubborn to deal with the problem, and too many customers of spammer harboring ISPs whined more about why SPEWS was targeting them, and making excuses why they could not switch to a decent ISP (excuses that didn't apply in 99.9% of cases). Unfortunately, quite a lot of people simply never "got it" as to what the purpose of SPEWS was. The SPEWS web site was more geek/admin talk, and not well enough written for the average person to understand. I was starting to work on my own "how to get out of SPEWS" document, but I just haven't had time to put in on it.
There are a lot of things people say as to how to stop spam. The one I hear most often is that if people would just delete the spam, or if network admins would just block only spammers and no one else, then spammers would cease making money and would stop. This is simply not the case. First, not everyone will do this. We see from these recent worms and virii that way too many people don't patch their computers anyway. There will always be gullible people who respond, and there will always be spammers to take their money.
The real way, and I think possibly the only way, to stop spam, is to treat all spammers as equivalent to cyberspace terrorists. Take no prisoners, and take no excuses.
Remember, spammers don't care what people who will never respond do with the spam they send. They don't care if you press delete, or filter it out with SpamAssassin, or even block them. They don't care because you aren't going to make any difference to them anyway. And if you do block it, you won't be complaining to the spammer's ISP, and hence, they get to spam even more. To a spammer, someone who blocks their mail is better than someone who gets their ISP account terminated. This is part of why just blocking spammers is actually making the problem worse.
Spews was an excellent solution. It wasn't perfect and a few mistakes were made. The fact that the real operators had to remain secret due to all the lawsuit threats did make it difficult to provide feedback to make corrections. I predict SPEWS will be back, but in a different form, possibly as a distributed file of sites to block ... which will make it even harder to get removed since it will then not be operating as a live database.
Much of the problem was because a lot of people didn't understand that the purpose of SPEWS went beyond just blocking spammers (which will not accomplish stopping spam), but actually blocking the ISPs that allow spammers to continue to operate and continue steal resources from networks and mail servers. This was in effect a boycott of that ISP, and it was intended to drive customers from that ISP to other ISPs that do not harbor spammers. In many ways it was working because it clearly got a lot of spammers upset, and a lot of ISPs upset as well. I even believe it is possible that the DDoS attack on OSIRUSOFT was caused by many of these ISPs.
My question to you is, did you understand that SPEWS was blocking whole ISPs, not just spammers? You don't have to agree with that method or principle ... just understand that others do think it is right, and understand why they do.
That's one reason I quit running backup MX. Why should I have to reject mail being delivered from my own (backup) mail server. By not having any secondary MX at all, if I do go down, the sending MTA just queues it for a few days. But spamware generally won't queue it, so that's not even a worry at all.
The bandwidth may be getting cheaper, but it hasn't been keeping up with the spam volume. Cost is a real factor when running internet services in a cut-throat market. So I don't think the issue is just eyeball protection. In fact I get a chuckle out of some of the spam that does leak through.
As for the reverse DNS, that's a problem between your ISP and you. I do not accept the excuses. Their service might be cheaper because they don't want to hire the staff to run it right. Reverse DNS is trivial once you have a static IP. While it is possible to do without a static IP by using a dynamic MX, you're in for problems that way. I hope at the very least you are constantly complaining to the appropriate government agency out there about how it is you still can't get decent internet service at a decent price on DSL because of the monopoly SBC holds, combined with their incompetency.
Of course we know that as long as bulk mailers are making money (I have my doubts about the people who are paying them to do the mailings), many ISPs do want a piece of the action. Those are the ISPs we need to boycott.
I don't know what (or if) your role at Rutgers is. But I can say this. If the departments won't report MTAs, then things are out of control. The central networking policy can in fact fix that with proper advance notification. They can always "smart host" forward to the central campus mail server. If they want to be able to connect SMTP direct, then access-list permit them to do so (of course that means they have to tell you, or whoever handles that, what the address is). If you don't want the access-list cluttered with a bunch of individual addresses, then set up a virtual subnet and access-list permit that, and route all those addresses around campus as /32's. Everything else will just have port 25 blocked (but not port 587, so you can still allow use of outside mail services when those services have deployed the message submission protocol as defined in RFC2476, which is AUTH-SMTP compatible).
Actually, you can perform the test during the RCPT command for the tuple of recipient and what IPs they want blocked. Then you can 550 the RCPT command if that user wants that IP blocked. There's no need to have all users agree.
Rejecting as much mail as possible during the SMTP session allows 5XX responses in place of queueing a bounce message (which in most spam cases can't be delivered anywhere anyway, so it sits in the queue for whatever the duration is set for).
My eventual goal is to set up a control panel for end users to define their own reject/permit policy, which will be done during the SMTP reception session. They will be able to choose their own DNSBLs or none at all. They will be able to choose defined sets of blacklists, or create their own. They will be able to whitelist, as well, of course. Or they can simply delegate the policy to someone else (there will be a default delegation). I just need to figure out a good schema to store all this policy preference data (needs to be easy to access from both the SMTP daemon and the web control panel logic), and do some SMTP daemon hacking to make it play.
How is a Bayesian classifier going to help me reduce the amount of bandwidth used by the spam? How is a Bayesian classifier going to help me delay the point in time when I have to upgrade my mail server to keep up with the load?
Actually, it's going to cost me more to deploy such methods. I presently block a huge amount of spam by a combination of private blacklists and DNS based blacklists that are applied before the DATA phase of the SMTP session. To switch to a content based test means I have to incur all that data transmission for all the cases where the promotion of that method says that my existing blocking is inappropriate (some say I should do it for all email, and some say I should do it for some subset, such as mail coming from cable modems and such).
The issue really comes down to what one's goal is with regard to spam blocking. Some people have the goal of keeping filth out of their mailbox so they (or their children) don't have to see it. Some people just hate getting email offers from someone who had no reason in the world to believe it might be wanted. But my goal is to minimize my costs by keeping down the bandwidth usage and the server resources needed.
It's been said many times before: spam is about conSent, not conTent.
So I also believe that the testing for spam should relate to what it is that makes it spam in the first place. So I don't test on content. I test on who it is that is sending to the best extent I can, which means it is based on sender address (for those few spammers who keep using blocked sender addresses) or client address (in addition to blocking by DNSBL and IP ranges, I also block by reverse domain names, and by default refuse from any client with no verified reverse DNS).
Of course there are false negatives. If a sender is unwilling to go to the effort to distinguish themselves from spammers, then I really don't worry about it. I've examined many of these false negative cases, and I've found that they are all some form of the sender not willing to make the effort to send email in a way that lets me know during the SMTP session that it really is from them. Since the costs of email are so heavily slanted towards the receiving end, I think it is only prudent that senders make some effort. Simple things like proper reverse DNS on their mail server can help a lot (and whining about their ISP refusing to do it for them only evokes my rant about changing to a better ISP that isn't being subsidized by spammers).
Many people have spoken about whether or not we should improve SMTP. Maybe we should. Or maybe SMTP is good enough as is. I've found it adequate, but would welcome some improvement. But the improvement that I would see as working would simply be something to more reliably identify who is sending. Forward verified reverse DNS at least lets me pin it down to a registered domain name. Then at least I can whitelist specific domain names, such as ka9q.net (it is whitelisted so if your reverse DNS is set up right for your own mail server, it should be able to deliver here just fine ... I trust you won't run open relays or open proxies).
My ability to whitelist shows I can in fact opt-out of any blacklisting I have in place, including slicing parts of my own blacklists. Even if your upstream IP address source (ISP) is entirely blacklisted by every DNSBL, as long as your reverse DNS is correct and names ka9q.net, your mail from there can get through. Spammers can't forge that without hijacking your domain since I test the address with a forward lookup. So I meet your requirement to be able to opt-out of others' listings.
As for letting users have control, I do that, too. My user base is still small enough that it is done manually. There are a few whitelistings my users have requested. One user has elected to opt-out entirely from all anti-spam measures.
The rates at such ISPs often are lower because the ISP is getting other revenue sources from spammers. And of course, when lots of people ignore that and choose that ISP, they are also helping to lower the costs to the spammers, too.
I do agree that AOL should do a better job of letting you show them that you have terminated that customer that sent those 3000+ unsolicited emails. You did termate that customer, right? If you want to understand why they blocked everyone, the reason is the technology is more efficient to route connections by their source address to specific mail servers that filter content. Servers not in their blacklist get routed to streamlined mail servers that don't do that filtering. I don't know what they actually do, but that is plausible, and I do know that some kinds of tests on email do cost more resources to do than others.
AOL's practice can certainly cause problems when there are thousands of AOL customers subscribing to certain mailing lists. When a mailing goes out, AOL gets briefly slammed with all those copies (although 3000 is certainly only a drop in the bucket considering their total overall volume).
It is your responsibility to make your mail server distinguishable from the unwashed masses of cable modems, DSLs, and dialups with dynamic IP allocations. If you are dynamically allocated, you shouldn't be running a mail server. If you have static IP, you need to have your own domain name in reverse DNS (no excuses accepted because reverse DNS is easy to do).
I don't know to what extent AOL blocks by IP or by domain name. I for one block by domain name. And I have most ISP dynamic/generic addresses listed. So if your reverse DNS name looks like a numbered sequence like all the others, then welcome to my blacklist (I can whitelist you if you tell me your IP address). Blocking by domain name actually works better than blocking by IP address because if the ISP changes the purpose of an IP address, all they have to do is make the DNS name match, and voila, not in the blacklist. Get your reverse DNS name as your domain name, and unless you really were spamming from it at some point, you should be in the clear.