Slashdot Mirror

there are commercial firewall appliances that do, indeed, watch HTTP streams and scan them.

You can accomplish the same task for free with the Squid proxy and one of the plugins that adds virus scanning with ClamAV. Do a Google search for "squid clamav" for some pointers.

I usually set up a transparent Squid proxy for my clients on the firewall. This enables us to block the types of garbage the article discusses. For instance, I usually have an access control rule that blocks downloads of files ending in .exe, with a prior rule permitting the local admin to do so for updates, etc. I don't usually bother setting up ClamAV with Squid. I do use ClamAV (with MailScanner and SpamAssassin) for e-mail, though.

SpamAssassin normally consults online databases of dangerous URLs when scoring messages. I'm imagine that those databases have some bad FTP URLs along with the evil HTTP ones.

I implemented SURBL recently, and it's helped a lot. Your filter extracts url's from the *body* of the e-mail, and checks them against SURBL's blacklist. The idea is that most spam is trying to get you to click on a link, and although they can forge the From: line, they're still constrained to give the address they want you to click on. This has been amazingly effective for me, and it's really nice because there are essentially no false positives. It won't necessarily work with pump-and-dump scams, though, since it's possible for them to say "buy SCOX," without giving a URL.

On my mailserver, if Spamhaus SBL-XBL fails to return an answer, SpamCop's BL will still be up and working. I've got my MTA configured to check for both. In the past day, Spamhaus SBL-XBL has rejected 25,791 emails and after checking through that list SpamCop has found an additional 4,256 emails to block. If I switch which service is checked first, the numbers are roughly the same (SpamCop will catch ~80% and Spamhaus will catch the other 20%). I'm sure any ISP mail admin knows at least this much.

However, that other 20% that Spamhaus SBL-XBL would have blocked will then get through, and SpamAssassin will start checking against SURBL.org for spam-vertized domains in the email content and catch the rest. This is at a much larger CPU cost to use SpamAssassin than using Spamhaus SBL-XBL on the MTA before it even accepts the email.

BTW, SpamAssassin with SURBL and a number of other filters (pyzor, dcc) still tagged another 5,135 emails as spam (I don't auto-delete, just add headers).

That's pretty scary to me that my system, which houses a few domains for friends and family, has blocked/tagged 35182 spam messages in the last 24 hours.

I personally like the SURBLs. They list spamvertised web sites, not the originating hosts of spam messages. If you block those then you're one step closer to cutting down on their profits.

But the spammer doesn't care. They don't check if you're using nofollow, they just vandalise your comments and run. Thinking nofollow will stop this type of spam is akin to thinking spam assassin or dnsBLs stop spammers. It hasn't, it just means the crud doesn't end up in your inbox.

I've ended up having a little database which holds both referral spammers and comment spammer URLs, so anyone who either tries to send an http request with a site listed as the HTTP referrer or post a comment with those sites in gets redirected to a permission denied page.

But I could do that because I'm vain enough to roll my own code (and embarassing it is too). Most bloggers will have to wait for their blog software authors to add something like that and then for their hosts to update.

Now what we really need is something akin to the SURBL where blog spam and referral spam urls end up, then plugins for every major blog engine out there to use it.

I believe ebay has know about this for a while but sat on it for some unknown reason: SURBL List gave first warning. Took them almost a month, not bad.

Its been exploited in phishing attempts since at least Feb 16th: http://lists.surbl.org/pipermail/discuss/2005-Febr uary/004192.html

Quite why they thought running an open redirector was a good idea is anyones guess.

One solution is SURBLs (Spam URI Real-time Block List, I think). This is a list of web addresses contained in spam. An anti-spam filter parses an email, then checks any URIs against various SURBLs. They are pretty damned effective. Any URI in spam gets blocklisted pretty soon, and filters can act accordingly and block spam.

These are up and working, and have been for at least a year. The latest SpamAssassin has support for them out of the box, I haven't checked but it may use around 5 different lists.

There is a small network delay and very little processing overhead on the spam filter. So email may be delayed for 15 seconds, but spam will be filtered to a far greater extent.

Visit www.surbl.org for more info, and don't forget to check out SpamAssassin as well. Anyone running a modern Linux can filter their own email, even if they pick email up from a pop3 server. I'd recommend a fetchmail, postfix, procmail and spamassassin combination, but there are many, many ways to do this.

Since it works just like a DNSBL, you would need your plugin that grabs the URL, does a quick SURBL lookup and open a standard error page if it gets a 127.0.0.x response to the lookup. The option to continue anyway needs to be something that a network administrator can override, naturally. Best of all (and I can't believe I'm typing this), owing to the high level of integration of IE into Windows it might actually stop people from opening HTML spams in Outlook, inadvertantly or otherwise, as well.

Thinking about it, why stop at IE? Anyone care to write a Mozilla Extension?

It don't understand the problem with that. Most blacklists based on spamvertized links (like surbl) are text-based, not IP-based. They should need any DNS lookups.

I am entirely unfamiliar with the issue of spam as it pertains to blogs. Are spammers placing ads (as in, posting their URLs) to random peoples' blogs? Or is the problem that they are just polluting the comment list with random garbage?

If the issue is posting of URLs, then it should be a simple matter of the blog site checking any URLs against SURBL, a spam URL blocklist.

What am I missing here? When did this become such a huge issue?

There are some URI domain blacklists that cover URIs that often appear in spam messages, but that's a little different. Symantec Mail Security for Microsoft Exchange's spam filtering consisted of several blacklisted words, and blacklists from any mail from @hotmail.com, @yahoo, etc. This may have changed in later versions, however. The snakeoil anti-spam systems many vendors release has got to be the worst, though.

Remember that the Phishing spam arrives in email - so rather than building the checker into browsers for people who've already clicked on it, you can solve the problem by junking the email before it's read, and reduce the spam problem as well as the phishing risks. I suppose that implementing it in the browser could help people whose email programs use their browsers to fetch URLs, but remember that most of the target population uses IE, not Firefox.

Of the million or so emails I process per day, 80% are marked as spam. Of those approximately 75% are caught by the RBLs before it even reaches the spamassassin engine.

I highly recommend RBLs to anyone. Not only are they fast and usually pretty accurate, but they are very fast learners usually. :)

One of my favorites is the SURBL which seems to catch a good chunk of it. Bayes filters are always gonna be thrown off by the dummy words thrown in there but the minute they try to link the person to their site BAM the surbl gets them.

Content RBLs have been working fairly well for me

Well, SpamAssassin just failed on me two weeks ago - suddenly its effectivity dropped from a fair 95%-98% to less than 50% and spam started pouring into my mailbox.

Two things that I have done that keep SA hitting hard on the Spam problem, is to install the additional rules_du_jour and my_rules_du_jour
and secondly the SURBL real-time system.. And SpamAssassin works like a charm again ..

A thread on our discussion list archive has an outline of our next engine with prior IPs used to lower the domain inclusion thresholds.

One major reason to avoid name resolution on the incoming messages is to cut out the timeouts needed for the DNS queries to complete. We figure the domain name is nearly as useful to block on, modulo the improvement we expect to make mentioned in the thread.

Differences between the two lists and more topics about our project are described on the SURBL site. If anyone has a question or comment about SURBL you can write to me directly at jeffc at surbl dot org, or much better ask the growing SURBL community on our discussion list. The separate announcement list is a good way to keep up with news about the project. Archives of the lists are available on the list site.

Folks who have tried SURBL have generally been pleased with the results, and we expect to improve the ~60% spam detection rate and lower the already low (<0.1%) false positive rate further in the next version of the data engine behind sc.surbl.org. As has been previously mentioned, Devin Carraway has written an MTA use of SURBL domain data to check message body URIs in his qpsmtpd plugin called uribl. This is the first MTA use for SURBL that I've heard of, though I don't necessarily claim to have heard of all uses of SURBL so far. Some other folks are thinking about implementing a sendmail milter for which will probably use sc.surbl.org since they are also a major ISP source of SpamCop's data.

Since I probably won't be posting here too often, I'd like to thank some of the people who have been quietly very supportive and responsible for the success of this effort so far including Eric Kolve, Justin Mason, Daniel Quinlan, Raymond Dijkxhoorn, Julian Haight, Kelsey Cummings and others who already know how greatly they have helped to make SURBL possible.

One thing we could use some help with now are more BIND-compatible secondary DNS servers and rsynced rbldnsd servers. Please see the SURBL site for details, and lend a hand if you can. DNS traffic may get kind of heavy when SA 3.0 comes out since it has SURBL support in URIDNSBL (URIBL), so we definitely need some help with name service.

Differences between the two lists and more topics about our project are described on the SURBL site. If anyone has a question or comment about SURBL you can write to me directly at jeffc at surbl dot org, or much better ask the growing SURBL community on our discussion list. The separate announcement list is a good way to keep up with news about the project. Archives of the lists are available on the list site.

Folks who have tried SURBL have generally been pleased with the results, and we expect to improve the ~60% spam detection rate and lower the already low (<0.1%) false positive rate further in the next version of the data engine behind sc.surbl.org. As has been previously mentioned, Devin Carraway has written an MTA use of SURBL domain data to check message body URIs in his qpsmtpd plugin called uribl. This is the first MTA use for SURBL that I've heard of, though I don't necessarily claim to have heard of all uses of SURBL so far. Some other folks are thinking about implementing a sendmail milter for which will probably use sc.surbl.org since they are also a major ISP source of SpamCop's data.

Since I probably won't be posting here too often, I'd like to thank some of the people who have been quietly very supportive and responsible for the success of this effort so far including Eric Kolve, Justin Mason, Daniel Quinlan, Raymond Dijkxhoorn, Julian Haight, Kelsey Cummings and others who already know how greatly they have helped to make SURBL possible.

One thing we could use some help with now are more BIND-compatible secondary DNS servers and rsynced rbldnsd servers. Please see the SURBL site for details, and lend a hand if you can. DNS traffic may get kind of heavy when SA 3.0 comes out since it has SURBL support in URIDNSBL (URIBL), so we definitely need some help with name service.

Differences between the two lists and more topics about our project are described on the SURBL site. If anyone has a question or comment about SURBL you can write to me directly at jeffc at surbl dot org, or much better ask the growing SURBL community on our discussion list. The separate announcement list is a good way to keep up with news about the project. Archives of the lists are available on the list site.

Folks who have tried SURBL have generally been pleased with the results, and we expect to improve the ~60% spam detection rate and lower the already low (<0.1%) false positive rate further in the next version of the data engine behind sc.surbl.org. As has been previously mentioned, Devin Carraway has written an MTA use of SURBL domain data to check message body URIs in his qpsmtpd plugin called uribl. This is the first MTA use for SURBL that I've heard of, though I don't necessarily claim to have heard of all uses of SURBL so far. Some other folks are thinking about implementing a sendmail milter for which will probably use sc.surbl.org since they are also a major ISP source of SpamCop's data.

Since I probably won't be posting here too often, I'd like to thank some of the people who have been quietly very supportive and responsible for the success of this effort so far including Eric Kolve, Justin Mason, Daniel Quinlan, Raymond Dijkxhoorn, Julian Haight, Kelsey Cummings and others who already know how greatly they have helped to make SURBL possible.

One thing we could use some help with now are more BIND-compatible secondary DNS servers and rsynced rbldnsd servers. Please see the SURBL site for details, and lend a hand if you can. DNS traffic may get kind of heavy when SA 3.0 comes out since it has SURBL support in URIDNSBL (URIBL), so we definitely need some help with name service.