Slashdot Mirror


O'Reilly Article on Spam Defense

Dru writes "Here's an article talking about the details of implementing a network level spam defense with Qmail. It also talks a little about a new site called Trustic which uses a trust system (like Advogato) for nominating spammer/hostile IP's."

19 of 189 comments (clear)

  1. I love qmail. by BoomerSooner · · Score: 4, Informative

    I suggest buying the book if you plan on implementing it. The online version isn't enough (and covers about 1/3 what the printed version does).

    Make sure you follow the relay-ctrl section very close. You could be a source of spam if you do it wrong!

  2. Not too impressed by augustz · · Score: 3, Informative

    Please remember that the service is beta and will start charging for advanced features once it is out of beta. As usual, worth waiting to see if it goes totally commercial. Looks like they plan to charge to allow listing multiple "trusted" servers.

    A fair number of the spams I submitted came from servers that had already been voted on as TRUSTED by other users. In other words, my credability went down by reporting them as spammers.

    http://www.trustic.com/ip/219.94.114.6 for example and I've got a fair number of others. Folks are either polluting the space intentionally or being very very sloppy in reporting trusted servers.

    Groups like spews have a very nice evidence file, and it gets reviewed by a person. I've generally been impressed with the real community blacklist sites.

    Technically the site works great and is super fast. But wouldn't follow the O'Reilly recommendation and pick it as my primary blacklist just yet (even through the guy doing the site worked with the author of the article to make changes.)

    My two cents.

  3. That depends upon their methodology. by Population · · Score: 3, Informative

    I use SpamAssassin with Bayesian filtering.

    Your forged headers are noted and factored in when determining whether you are spam or not. But by themselves they are not sufficient to mark you as spam.

    Your e-zine will tend to have the same format and similar content from issue to issue. The Bayesian filter can detect this and let it through.

    I'm running this setup at work for our offices and it works very well. The only real problems we've had is monster.com's resumes. But even that seems to be working now.

  4. Relying on RBLs by GC · · Score: 5, Informative

    There are many problems with using RBLs to block connections. A very good description can be found here:
    I've found SpamAssassin a fairly good, rather than block messages from RBLs it analysis message content, adds points to messages in RBLs and checks known Spam databases such as Razor and Pyzor. Rules matches are given a score, and messages with a total aggregate score are tagged in the message headers, allowing users to filter these if they want to.
    A main advantage of this method is that no single rule can flag a message as spam, hence legitimate mail sourcing from the badly configured mail relay has a chance of getting through, and in my mind it's probably a particularly bad idea to block any email unless it's actually addressed to you.

  5. Re:Here's my question. by gfody · · Score: 2, Informative

    have you considered using an asp marketing service? you could upload your templates/e-zines and your mailing list and schedule the deployment. depending on how much you send its probably cheaper than hosting your own server, plus you dont have to worry about being filtered or black listed. check out www.dynamicsdirect.com

    --

    bite my glorious golden ass.
  6. Or you could use a better mailer... by SuperBanana · · Score: 5, Informative
    Here's an article talking about the details of implementing a network level spam defense with Qmail

    Or, you could just use Postfix, which:

    • is almost entirely compatible with sendmail. It's pretty much drop-in-and-go.
    • adheres to RFCs(and there's a warning for any configuration option which would violate said RFCs)
    • has builtin anti-spam tools- you can turn on, individually, any of a dozen-plus different checks, such as making sure the claimed hostname in the HELO matches the IP the connection is coming from(you can do this several ways), or that the claimed hostname matches the mail-from user@hostname(ie, if you're coming from spammer.com, you're not gonna be able to claim to be joe@yahoo.com), etc. It's also one builtin command to check an RBL.
    • has a really sharp cookie of an author(the guy wrote tcpwrapper), who isn't widely regarded as an obnoxious twit
    • is completely free

    Personally, I refuse to use any software written by DJB as a matter of principle. The guy flagrantly ignores RFCs because he simply feels like it and arrogantly thinks he knows better(and further that there is benefit to ignoring said RFCs).

  7. RFC violations by SuperBanana · · Score: 1, Informative
    You're aware that rejecting mail based on HELO violates RFC 1123?

    ...which is why a)it's not turned on by default and b)the docs(including the docs in the config file) warn you as to such. The docs are very specific about WHICH of the checks violate the RFCs and which don't.

    What RFCs does qmail not comply with?

    Based on a very quick google search(so thus some of this might be outdated or simply wrong), pipelining, for one. RFC-2821 for another. RFC 2821 and RFC 1123 for two more.

    The difference is that while Postfix CAN reject based on HELO etc...qmail seems to do so by DEFAULT.

    Also, take a look at djbdns some time- it violates RFC's left and right.

  8. Read Your TOS. by Electrawn · · Score: 3, Informative

    5. Acceptable Use Policy; Prohibited Uses of the Service.

    b. Prohibited Uses of the Service: Use of the Comcast Equipment or the Service for transmission or storage of any information, data or material in violation of any federal, state or local law or regulation is prohibited. In addition, unless you are subject to a Service plan that expressly permits otherwise, the Service is to be used, and you expressly agree to use it, solely in a private residence, living quarters in a hotel, hospital, dormitory, sorority or fraternity house, or boarding house, or the residential portion of a premises which is used for both business and residential purposes. Without limiting the generality of the foregoing, the Service is for personal and non-commercial use only and you agree not to use the Service for operation as an Internet service provider, a server site for ftp, telnet, rlogin, e-mail hosting, "web hosting" or other similar applications, for any business enterprise, or as an end-point on a non-Comcast local area network or wide area network.

    I'll keep my toungue in cheek for any other comments.

  9. Other choice than Trustic - SPAMCOP by Swift+Gilmer · · Score: 2, Informative

    I have been using SPAMCop for the past 5 months at my work. I am also using QMAIL as my mail server and it took me about 10 minutes to get it hooked into the Spam Cop Database. The best part it is free and it it blocks about %80 of SPAM that gets delivered - I will just have to live with the other %20. Has anyone heard of other Spam IP Databases that are available for public use?

  10. Re:Hurrah for blacklists by qtp · · Score: 4, Informative

    You're probably right, they will eventually want to charge money, and, IMHO, thier solution looks overly complicated and manipulable (spammers pay for "trusted" members to list them as "trusted").

    It would be better if ISPs participated in services like the ORDB, SORBS and Monkeys that have simple network testable criteria for listing open relays. Spews, Spamhaus, and DSBL have reputable lists of usernames and addresses that send spam. If ISPs and admins would participate in projects like these, the spam problem would be greatly reduced. And it seems that these projects are mostly run by admins who are interested in blocking spam, not selling a service.

    By the way, MAPS is currently free for individual use (look at the bottom of the page).

    --
    Read, L
  11. Using Trustic with SpamPal by NaDrew · · Score: 2, Informative
    I use SpamPal with the Bayesian filter as my client-side spam filter on Win2K. It works well enough but I'm always looking to improve things, so this article gave me the impetus to see if SpamPal could be made to use Trustic's DNSBL in addition to its preconfigured lists. The answer, at least for SpamPal Beta 1.295, is yes--using the "Extra DNSBL Definitions" section of the Options dialog. Here are the steps I used to add Trustic to the DNSBLs used by SpamPal:
    1. Create a Trustic account
    2. Once you've verified your registration, go to Trustic's DNS Query Information page for your account and note the second DNS query address.
    3. In SpamPal, open the Options dialog and drill down to the "Extra DNSBL Definitions" section. Click the "Extra DNSBL Instructions" button for information on adding a DNSBL to SpamPal. Read this text and then close the file.
    4. Click the "Extra DNSBL Definitions" button. This opens "extra_dnsbl.txt". Add a new DNSBL entry as follows:
      LIST Trustic
      NAME Trustic DNSBL
      TYPE STANDARD
      WEBSITE http://www.trustic.com/
      ZONE queryaddress
      DESCRIPTION Trustic is a community-based block list that prevents untrusted servers from sending spam. It is a new approach to the spam problem, and it is better than existing solutions.
      Substitute the personalized query address you saw in step 2 above for queryaddress.
    5. Save and close "extra_dnsbl.txt", then exit SpamPal and relaunch it.
    6. Open SpamPal's Options dialog and drill down to Spam-Detection, Blacklists, Public Blacklists. Trustic should now appear on the list. Select it and click Apply, OK.
    That's it--SpamPal should now be checking Trustic's DNSBL for your incoming mail. Trustic may require additional RESULT_CODE settings--I'm waiting for a response from Trustic and will follow up if needed.
    --
    Vista:XPSP2::ME:98SE
  12. Re:Spews by CryBaby · · Score: 3, Informative

    I initially thought spews was completely over the top. My first encounter with them was due to a client's server getting blocked when he inherited some new ip's that had previously belonged to a spammer. I couldn't believe that these people had so much control over so many networks (almost zero mail got out) and that there was NO official or standardized removal procedure. But after reading the FAQ, joining the newsgroup and asking a few polite questions - my client was delisted within a day. Also, the evidence file on these ip's was truly damning.

    Their heavy-handed approach seems to be the only way to make a dent in the spam onslaught. I watched employees of major ISP's post to the newsgroup humbly asking for removal only to be told "kick your spammers off and you will be delisted, when we feel like it. You took too long to respond to our notices" As the spews philosophy goes, these people will only pay attention to the problem when it hits their bottom line - i.e. floods of customer complaints and cancelled accounts because no one can send mail from their entire polluted network.

    Back to the topic, I have a lot more faith in the hard-headed anti-spam warriors at spews than I do in some touchy-feely "trust network". It sounds far too vulnerable to manipulation and, based solely on some of the comments here, potential market pressure in the future.

    Thanks for listening...

  13. Re:IP banning by NerveGas · · Score: 2, Informative

    I've turned down some attractive deals with large, key providers for *years* precisely because they blatantly tolerated spammers.

    As far as I'm concerned, if the netblock in question was blacklisted with the RBLS that are taken more seriously, it was precisely because the provider didn't take any sort of action to contain or prevent spamming. And if you sign up with a provider like that, well, don't come crying to me when you're affected, too.

    All the time that went by before they knew they were on the blackhole list nearly led them to bankrupty.

    It takes less than five minutes to see if you're on the major blacklists, and any administrator who doesn't do it on a regular basis simply isn't worth his pay. I certainly don't have any sympathy for them.

    I also have a hard time believing that they simply went about their business for that long without realizing what was going on. How brain-dead do you have to be to realize that a particular person never responds to your email? How long does it take you to realize that SEVERAL people never respond to your email? And for the email problem to truly cause bankruptcy, you're talking about some very important email: The kind that you don't just send and forget. If my users think that someone isn't getting their email, you bet I'll hear about it. And you bet I'll track down the reason.

    Really, your description of them makes them sound completely incompetent. For the sake of those involved (and the rest of the world), I hope that's not so.

    --
    Oh, you're not stuck, you're just unable to let go of the onion rings.
  14. Re:IP banning is bad by Animats · · Score: 2, Informative
    That specific example won't normally happen...

    If only it were so. It happens to me about once every two months, when somebody using SBC DSL sends out spam and I get their IP address later. There are a a few ISPs that set temporary IP blocks within their own network, and these persist for a day or so. I then get mail bounces for a few people I really need to reach, which is a pain.

  15. Re:Blackists by archbish99 · · Score: 2, Informative

    ORDB offers such a service, actually -- they run quite a variety of tests against servers to see if they fall for any of a number of forms of relaying tricks. I, thankfully, fell into the opposite hole -- I couldn't relay from anywhere when I set up qmail, so I had to go back and figure out how to enable relaying for localhost and the local network. ;-) I ran the ORDB test set against my server once I thought it was up, and again a month or so ago when I had a scare which *looked* as if someone had sent a mail through my server. (Turned out it was a different mistake, and not a relay issue at all.)

  16. Re:Blackists by Phroggy · · Score: 2, Informative

    however I feel that the "people like you" comment is a little unfair.

    A little perhaps, and it's great that you're no longer causing a problem, but the fact remains that for a brief period of time, you were part of the problem. Spam came through your server. There are many others like you - good intentioned, but making an honest mistake once, quite by accident, and then fixing the problem and never doing it again - and these people collectively make up a very significant source of spam. That's why AOL blocks you.

    That said, I'm glad you've learned enough about it now to be a responsible Internet citizen, and I certainly don't want to discourage you from continuing on that path. Something you may want to look into is forwarding all mail destined for @aol.com to your ISP's SMTP server; they should be able to relay it to AOL (and since you're using one of your ISP's IP addresses, they should allow relaying from you).

    --
    $x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
    $x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
  17. WARNING: Not the same SpamCop by Phroggy · · Score: 2, Informative

    Be aware that SpamCop.com is not the same as SpamCop.net - I'm not sure who SpamCop.com is, but having worked in the abuse department at an ISP, as well as having been a paying subscriber for a couple of years now, I can say that SpamCop.net is absolutely wonderful. They're best known for automating spam reporting - you paste in your message with full headers, and they figure out where it came from and prepare an e-mail to be sent to the administrators of those networks. Upon your approval, the complaints are sent from a unique SpamCop.net e-mail address, so your own e-mail address is not revealed (in case the complaint is forwarded to the spammers), yet you still receive any replies (SpamCop forwards them back to you).

    On top of that, they also offer a service for $3/month that includes just about everything you could look for in an e-mail provider - pop3, imap, webmail, the ability to retrieve mail from other POP3 (and recently AOL and Hotmail) accounts, e-mail forwarding, easier spam reporting, and of course, spam filtering using a variety of blacklists (including SpamCop's own automated RBL) and recently SpamAssassin. It's all fully configurable so you can use it however you'd like.

    Again, I have no connection to them, but SpamCop's reporting really does great things towards reducing the total volume of spam going around (by informing network administrators of the problem in a clear and consistent format so it's easy to deal with). I've only seen a couple of abuse reports from SpamCop.com, compared to thousands from SpamCop.net.

    --
    $x='S24;r)>63/* h@<5+oZ)32"5cz';$me='phroggy'x$];
    $x=~y+ -xz+\0-Tx+;print$_^chop$me for split'',$x;
  18. Automatic Reporting by Specialist2k · · Score: 2, Informative
    Im currently pushing all the spam SpamAssassin finds to my Trustic account with procmail, to register my negative recommendations.

    IMHO, automatic reporting is a bad idea. SpamAssassin isn't perfect and might flag legitimate mail as spam. It happens rarely, but it does happen. If you submit manually, you'll (hopefully) notice this, but automatic submission will report the IP of an innocent party as untrusted...

  19. TMDA by TheSync · · Score: 2, Informative

    My favorite solution is still TMDA, a free challenge-response auto-whitelist and complex filtering system for Linux. I realize you anti-challenege / response people won't hit the "R" key for me, but I consider that a useful filter...