Slashdot Mirror

We've recently set up a Spam Troll-box using Vipul's Razor on our new Tux4Kids dev server (you can find our troll box here).

A troll-box gives Spam-bots a place to send their spam. When this box intercepts the spam, it reports it to the Vipul's Razor network, and everyone else on this network becomes aware of that spam (if they are also using Vipul's Razor to filter, which, chances are they are, it will filter that spam if they get it).

If Vipul's Razor isn't enough, one can even use something like SpamAssassin in conjunction with Vipul's Razor to get even better results.

Of course, this isn't cutting off Spam-bots at their source... but if enough sites were to cut them off at their source, then I'd imagine the Spam-bot authors would get wise to this and devise a way around it. Whereas with something like a SPam Troll-box, the Spam-bots seem to still be working to those running the Spam bots ;-)

X-Spam-Report: 28.5 hits, 6 required; * 1.0 -- Subject contains lots of white space * 4.0 -- Invalid Date: header (timezone does not exist) * 2.8 -- BODY: Uses a dotted-decimal IP address in URL * 1.2 -- BODY: Tells you how to stop further SPAM * 3.2 -- BODY: URL of page called "remove" * 1.3 -- BODY: "if you do not wish to receive any more" * -2.0 -- BODY: Contains a claim of copyright * 4.2 -- BODY: Asks you to click below * 2.0 -- BODY: Link to a URL containing "unsubscribe" * 1.1 -- BODY: Saved web page * 2.0 -- BODY: Image tag with an ID code to identify you * 1.6 -- BODY: Link to a URL containing "remove" * 2.0 -- Received via a relay in relays.osirusoft.com [RBL check: found 254.1.233.63.relays.osirusoft.com.] * 2.0 -- Received via a relay in relays.ordb.org [RBL check: found 2.48.70.194.relays.ordb.org.] * 2.1 -- Subject contains a unique ID number

I use it on my main email address, and it only generates a few false-positives, mostly from the college students (email addresses ending in numbers, nad sent to >10 addresses in the same domain) but that has been changed in the rules. It works a charm most of the time.

spammassassin will catch 99% of your spam. Due to the massivley non-brilliant decision to write it in perl, its a resource hog. But it does the job.

This is part of the insane attitude that one's workers are one's worst enemies. Letting people do these little things is far from bad for business. It is most likely actually good as it creates an environment where people feel invested and where they have the wild concept that maybe their employer sees them as more than "production units".

As I read the article, the point isn't "Joe smith just spent 10.3 minutes reading slashdot when he could have been working".. It has more to do with "Joe Smith just downloaded a pirated version of Photoshop to run on a company owned PC". Your doing some online shopping or checking your Hotmail (possibly) hurts your productivity, but NOT the productivity of others. Now imagine you're pulling up porn in your cube and Cindy M. Biblethumper happens to walk by... Or when you open your outlook and unleash the latest win32 virus on the network. This cost the company serious money above providing net access.

We're reached this point at my company. As the network admin I've taken to explicitly blocking any e-mail with a .exe, .vbs, or any one of a 100 different virus-carrying file-types across. I still allow .gif's, .zip's, .doc's, etc, but scan them before delivery. If they get upset because they can't receive dancingbaby.exe from their cousin in Toronto, that's too bad.. Let them download it home their home computer and infect it.

The same thing is happening with spam. For 5 years now our policy has been "we can't do anything about it", because we didn't want to be responsible for attemping to filter the incoming e-mail stream. It has reached the point that our CEO is receiving 15 - 30 porn spams a day and has had enough. We have to pay the costs while he's travelling in europe and dialed in to our 800 number at 28.8 downloading this shit. We're about to deploy spamassassin site-wide, and if it happens to catch someone's birthday card from his step-mother, that's too bad.

Shayne

I guess I have to throw in my $0.02 here. Instead of relying on a single services or technique for stopping SPAM, try something heuristic that combines the best of multiple worlds: SpamAssassin [spamassassin.org], for example.

Just for laughes, here's the record SpamAssassin score in one of my spam's:

SPAM: --- Start SpamAssassin results ---
SPAM: This mail is probably spam. The original message has been altered
SPAM: so you can recognise or block similar unwanted mail in future.
SPAM: See http://spamassassin.org/tag/ for more details.
SPAM:
SPAM: Content analysis details: (31.38 hits, 5 required)
SPAM: Hit! (1 point) From: contains numbers mixed in with letters
SPAM: Hit! (1.2 points) From: does not include a real name
SPAM: Hit! (2.37 points) Message-Id generated by a spam tool
SPAM: Hit! (1.94 points) From: ends in numbers
SPAM: Hit! (0.9 points) Message-Id is not valid, according to RFC-2822
SPAM: Hit! (0.01 points) BODY: Asks you to click below
SPAM: Hit! (1.32 points) BODY: Contains word 'guarantee' in all-caps
SPAM: Hit! (1.93 points) BODY: Contains a 1-800- number
SPAM: Hit! (1.2 points) BODY: HTML mail with non-white background
SPAM: Hit! (4 points) BODY: Uses control sequences inside a URL's hostname
SPAM: Hit! (1 point) BODY: Link to a URL containing "opt-in" or "opt-out"
SPAM: Hit! (1.82 points) BODY: Link to a URL containing "remove"
SPAM: Hit! (1 point) BODY: Image tag with an ID code to identify you
SPAM: Hit! (1.56 points) Contains phrases frequently found in spam
SPAM: [score: 20, hits: click here, email address,]
SPAM: [from future, future mailings, here for,]
SPAM: [including shipping, offer order, this email,]
SPAM: [with our, with this, you not, your]
SPAM: [email]
SPAM: Hit! (3 points) Listed in Razor, see http://razor.sourceforge.net/
SPAM: Hit! (1 point) spam-phrase score is over 20
SPAM: Hit! (3.33 points) HTML-only mail, with no text version
SPAM: Hit! (1.8 points) No MX records for the From: domain
SPAM: Hit! (1 point) Received via a relay in orbs.dorkslayers.com
SPAM: [RBL check: found 11.124.183.200.orbs.dorkslayers.com.]
SPAM:
SPAM: --- End of SpamAssassin results ---

Now I've turned spam into something of a game. I have procmail rules tell me when a new record has come in so I can laugh at how cliché the message is. It's fun. Really.

The sad thing is that spammers are most likely already using these rules to try and author messages that will sneak in "under the radar" so to speak. I wouldn't be suprised if I start getting messages in pig-latin one day.

-AP

I cannot speak to what approach DCC uses, but razor only picks pieces of a message it believes to be static when computing its SHA1 hash. In the very near future, razor is going to implement Nilsimsa hashes which are 'fuzzy' and should be able to detect everything from spam with minor differentials to mutating e-mail viruses.

Combined with the new razor trust system, razor is going to be quite the tool; and when used in conjunction with SpamAssassin we'll have quite the arsenal to battle unwanted spam.

The other possibility is a net-block equivalent of ORBS. Some on the Sec-Focus Incidents list (and other fora, over the years) have bounced around the idea of blocking netoblocks who'#s POCs don't work, or who don't have or respond to mail to the RFC-mandated abuse@, security@, hostmaster@,.. standard mail accounts. I'm all in favour. Automate probes, the way ORBS did for anonymous relays. I think this would be a Good Thing. People do have a legitimate need to communicate between Asia, America and Europe: simply dropping everything from .kr is evil and wrong, IMHO.

Finally - y'all know that anonymous HTTP proxies are just as bad, if not worse, than traditional open mail relays? Just testing ;)

Do you have an account where you can get a shell on the mail system, and install a filter for yourself? If so, SpamAssassin is an excellent choice for fighting spam. It has a very broad rulebase, which catches all kinds of typical spam behaviors, has whitelisting support and adjustable thresholding, and other nice features. I just began employing it on my mailserver at work, and it does a great job.

No, I'm not a developer, just a very satisfied user.

So, there's the ROKSO list of spammers, plus the usual MAPS and so on. Of course, there's also hieristic software such as Spam Assassin...

However, does anyone know of any web hosting providers that actually use these tools? I'm particularly interested in any that use SpamAssassin, as that appears to be very effective.

I discovered SpamAssassin a couple months ago and I must say I am very pleased.

It has successfully stopped around 84% of incoming spams and no false-positives (marking non-spam as spam) thus far.

You can hook it into blacklists, but I never used that feature. I doubt it'd really help much, anyways and would probably end up doing more harm than good.

Yes, 16% of spam still gets through, but that's not nearly as annoying as having mailboxes fill up with spam and eat away the spool partition at an alarming rate. Not to mention I could probably stop 90%+ of spam, but that increases the change of incorrectly tagging non-spam as spam.

Best of all, I control the rules and the scoreboard... I don't rely on a third party to deem who sends spam.

spam is a serious waste of energy. We're filtering roughly 3000 messages per week. It's unfortunate that you've ended up on those lists. It seems any successful hosting company will end up being fingered as a spammer at some point. In the case of the companies I've worked for, we've been innocent. One, a mainstream hosting company, was stuck on a blackhole list, which we never managed to get removed from. One customers ISP used that blackhole list, and he was very upset that he couldn't have his domains Email forwarded from us.

There are better solutions than using the "blackhole" lists to block someone, like yourself. Recently, we've stared using MailScanner, which uses SpamAssassin for spam identification by pattern recognition, blackhole lists, and Razor for spam identification through cataloging. MailScanner and SpamAssassin are very nice in that they don't just "black hole" you, they simply tag the message as possible spam. That's what any responsible ISP should do, rather than blocking all transactions based on a 3rd parties list. We get the occasional Email sent through a mail server which would have been black holed, and it is a legitimate Email which should be delivered.

Running a mail server, it's not my job to block mail based on where it came from. I can provide the service to my users by adding flags for potential unsolicited bulk messages, but it's up to them to decide if they did or didn't want it. You never know, they might have been interested in going to a hardcord teenage beastality site. Who am I to say that's wrong. :)

Configuration is simple and straight forward and it integrates nicely with any email system. Personally I'm using exim to pipe all received email through spamc/spamd and then the mail is received by exim after the spam check. There is only one check for spam per email entering my system.

Spamassassin only flags the email as spam, but it's up to the MUA to actually delete it.

This is log output from exim+SA: 2002-02-12 17:21:35 From: Subject: *****SPAM***** save money for dank X-Spam-Status: Yes, hits=14.1 required=5.0 tests=NO_REAL_NAME, FROM_ENDS_IN_NUMS, INVALID_DATE_NO_TZ, REPLY_REMOVE_SUBJECT, EXCUSE_3,REMOVE_SUBJ, TO_BE_REMOVED_REPLY, SUPERLONG_LINE, FREQ_SPAM_PHRASE, FORGED_YAHOO_RCVD version=2.01 Sender: owner-freebsd-questions@freebsd.org

I'll soon move all my email users' to email filtered by spamassassin. This is just too damn simple.

• Remember when some large company (I think it might have been ebay) reset all the user preferenes for "send me newletters" and "share my info with spammers^wpartner companies", claiming that there was some problem and they were resetting the user preferences because the users didn't understand? This is very similar to that. Suddenly all the nice, mostly working spam filters on places like hotmail, yahoo mail, or pretty much any large free email service that has spam filters will stop filtering these emails. Result, now you get just as much spam, but now a chunk of that will go into your inbox instead of your spam folder.
  
  Users then get to go through their spam, clicking on the 'click here to be removed' and wasting their time and bandwidth, until the next bout of spam comes through.
  
  
• People will get just as much spam as before, just now some will be digitally signed. Chances are you will NEVER get off all the "certified spammers" lists, so you'll still get spam in your inbox, and have extra hassle as now users feel they have to go through the removal process for them. I'd much rather have a "never have any certified spammer send me any mail" service, which goes and removes you from all the certified spammers' databases. The services is to try to give the user control right? So give us the control to not get spam that we don't want!
  
• How long do you think it'll take for these groups to really get it right? There are always glitches that show up in new systems and I'm anticipating that there'll be more than a few people who are spammed multiple times from companies that are not only certified, but the user has said "I don't want spam from you anymore!" Just a start up glitch or two, yea, that's it....
  
• How long before someone figures out a way to beat the system? Sure, I know that it's a signed cert, but think of the potential for a non-certified spamming bastart to manage to spoof the 'seal of approval' and be assured that their spam gets into everyone's inbox. Not only that, but when people email them back with the 'remove' emails, they get a nice list of 'live ones' that they can spam merrily along using perhaps a different company name, from address or approach as not to make the user suspicious.
  
• Along those lines, what stops companies from not spamming multiple times for different products, or from different spinoffs. Use the database of 'removes' to feed into a list of emails to send out for their next product, promotion or whatever... hell, just sell the list to non-legit spammers!
  

...than email can be filtered server-side to cull it out.

I can do that already far more effectively using tools like procmail and SpamAssassin. SpamAssassin in turn can use various RBLs and Vipul's Razor (recently mentioned here), if you choose to.

That combination has saved me from recieving and processing about 20 messages in my personal mail today alone, not to mention the other benefits of auto-filing/trashing/redirecting that using procmail gives me.

Dear Internet user,

We have recently patented a sure fire way to rid your Inbox of spam, FOREVER.

Our new Genetic Algorithms have been researched for nearly ten years and have increased our already successful 99% kill rate to an incredible >99% kill rate. That's right an unbelieveable less than one percentage point increase. This means that if you upgrade NOW you will REDUCE your spam intake by up to 1 unsolicited email per day *

So don't delay, download Spam Assassin today and wave that spam goodbye forever.

* based on an estimated 100 unsolicited emails per day

--
No animals were harmed during the typing of this post

This is what SpamAssassin is doing, and it's becoming incredibly accurate (it was already 99% accurate before they used GAs).

I've been using Spamcop for the last 9 months as a reporting tool, but for filtering, I'm a huge fan of SpamAssassin. It's a bit of a bitch to build and install (leaving a vital patch file out of the distribution tar probably isn't the smartest thing to do), but dear god, it does the job right.

Plus, you can configure it to use Spamcop's black-list so you get the benefit of Spamcop's filters too.

Sysadmins/users with an ounce of savvy should check this bad boy out.

rOD.

It uses genetic algorithms to assign scores to it's ruleset, it supports RBL and Razor, is highly cusomizable (you can add your own rules, change the current ones, set how sensitive you want it to be and how it should tag messages), and it comes with a daemon for high volume environments.

I've just tried SpamAssassin this WE and it works great :

• higly configurable Spam Scoring Filter according to predefined rules (each set of rules adds some pts as it matches, and it is "declared" spam when the result is highter than a specified value)
• can rely on RBLs
• is able to report spam to Vipul's Razor (distributed, collaborative, spam detection and filtering network)
• personal black and white lists
• can be tuned for particular filtering (changing scores etc.)
• can be used for a whole domain/network

...the best thing is that you don't have to perpetually update black lists of well know spammers
it is just based on content detection of spams (subject in CAPITALS; lots of exclamation marks, sp sammer X-Mailer etc.)

and it really works well

I've just tried SpamAssassin this WE and it works great :

• higly configurable Spam Scoring Filter according to predefined rules (each set of rules adds some pts as it matches, and it is "declared" spam when the result is highter than a specified value)
• can rely on RBLs
• is able to report spam to Vipul's Razor (distributed, collaborative, spam detection and filtering network)
• personal black and white lists
• can be tuned for particular filtering (changing scores etc.)
• can be used for a whole domain/network

...the best thing is that you don't have to perpetually update black lists of well know spammers
it is just based on content detection of spams (subject in CAPITALS; lots of exclamation marks, sp sammer X-Mailer etc.)

and it really works well

Sounds a bit like SpamAssassin, if I say so myself ;)

SA analyses mail headers, body, and uses RBL and Razor to come up with an aggregate spam/non-spam score, then filters appropriately. Most of its smarts is encapsulated in a Perl module, which means it can be run from virtually anywhere; a procmail filter, a spam-protection SMTP proxy server, a system-wide checking system, etc. (all 3 of those have been implemented). Its scores are generated using a GA and a large corpus of test mail, too. Hit rates nowadays are fantastic ;)

Disclaimer: I'm the maintainer.

SpamAssassin is a mail filter to identify spam.

Using its rule base, it uses a wide range of heuristic tests on mail headers and body text to identify "spam", also known as unsolicited commercial email.

The spam-identification tactics used include:

• header analysis: spammers use a number of tricks to mask their identities, fool you into thinking they've sent a valid mail, or fool you into thinking you must have subscribed at some stage. SpamAssassin tries to spot these.
• text analysis: again, spam mails often have a characteristic style (to put it politely), and some characteristic disclaimers and CYA text. SpamAssassin can spot these, too.
• blacklists: SpamAssassin supports many useful existing blacklists, such as mail-abuse.org, ordb.org or others.
• Razor: Vipul's Razor is a collaborative spam-tracking database, which works by taking a signature of spam messages. Since spam typically operates by sending an identical message to hundreds of people, Razor short-circuits this by allowing the first person to receive a spam to add it to the database -- at which point everyone else will automatically block it.

Once identified, the mail can then be optionally tagged as spam for later filtering using the user's own mail user-agent application.

SpamAssassin requires very little configuration; you do not need to continually update it with details of your mail accounts, mailing list memberships, etc. It accomplishes filtering without this knowledge, as much as possible.

SpamAssassin is a mail filter to identify spam.

Using its rule base, it uses a wide range of heuristic tests on mail headers and body text to identify "spam", also known as unsolicited commercial email.

The spam-identification tactics used include:

• header analysis: spammers use a number of tricks to mask their identities, fool you into thinking they've sent a valid mail, or fool you into thinking you must have subscribed at some stage. SpamAssassin tries to spot these.
• text analysis: again, spam mails often have a characteristic style (to put it politely), and some characteristic disclaimers and CYA text. SpamAssassin can spot these, too.
• blacklists: SpamAssassin supports many useful existing blacklists, such as mail-abuse.org, ordb.org or others.
• Razor: Vipul's Razor is a collaborative spam-tracking database, which works by taking a signature of spam messages. Since spam typically operates by sending an identical message to hundreds of people, Razor short-circuits this by allowing the first person to receive a spam to add it to the database -- at which point everyone else will automatically block it.

Once identified, the mail can then be optionally tagged as spam for later filtering using the user's own mail user-agent application.

SpamAssassin requires very little configuration; you do not need to continually update it with details of your mail accounts, mailing list memberships, etc. It accomplishes filtering without this knowledge, as much as possible.