Spam Filtering For Small/Medium Business?

or_is_it writes "The company I work for has been growing dramatically and I've been charged with the task of being the gatekeeper for our GFI Spam filters. This involves manually inspecting the subject line/to/from for all caught messages in each filter rule folder. For a company of about 50 people, in one day the number of spam messages can exceed 2,000. Neglect it for a day and you end up with quite a task on your hands. I've made the rules lax enough so important messages can go through, along with a few stray spams, for which I get bitched at. Tighten the rules up and then maybe an important time-sensitive email never gets to its intended recipient, and I get bitched at. Manually reading through all those subject lines is supposed to prevent that, but I'm only human and genuine messages can easily get overlooked. How do larger organizations deal with the spam issue? I can't imagine having one centralized person manually inspecting everyone's junk-mail header is the optimal solution. Purchasing a different commercial mail filter product is a possibility, but I'd like to hear some anecdotal evidence before jumping ship."

Comments

Client-based?

[Gaxx]

To be honest, for somewhere of that size I'd be tempted to use some sort of client-based filtering (along the lines of spambayes [http://spambayes.sourceforge.net/]) which would put the power and responsibility in the hands of your users.

Re:Client-based?

[holophrastic]

Pardon me, but I just don't see the "size". I personally (and professionally) receive well over 3'000 spam e-mails each and every day. I take about three to five minutes to run through them. For 6'000 in two days, I take four to seven minutes.

I do it without a spam filter of any kind. I have only two technique.

First, simple rule-based filters throw clients and friends into their own folders by from: line alone. That covers everyone I know in advance.

The second set of rules simply looks for my full name, my company name, my e-mail signature, my telephone number, or my mailing address. These into the "it's damn likely a legitimate e-mail" folder. This folder gets about 2 spam e-mails per week.

The remaining I simply run through, in outlook express of all clients. Sorting wins the day. The greatest trick? Sort by the to: field. It doesn't take long to see that 75 messages went to moocow@mydomain.com, 75sevens@mydomain.com, or some other horribly malformed address to that doesn't exist. Sorting by subject does similar things -- like give you "70% off . . ." which get selected and deleted in a block of one hundred at a time.

Your spam has very simple patterns to look for. Sort by them, click the first, shift-click the last, and hit delete.

Last year, I was contracted by Viagra's H.R. department to do some quick work, I made it through unscathed.

Barracuda SPAM filter

[spacepimp]

I purchased a Barracuda for my organization of about 120 employees, and it has been fantastic. I fine tuned a few options on the config and it has blocked about 200,000 emails in the almost two months i have deployed it. There are very few false positives, and very few that get through its filters. I actually get calls of gratitude from the end users about how happy they were not receiving any more SPAM messages. The hardest part was informing them the user base on the difference between the mailing lists they were on and SPAM. Barracudas support has been good as well.

Re:Barracuda SPAM filter

[Lershac]

Gah they are so expensive. And to keep them up to date is ridiculously expensive. I prefer free with ASSP.

Additionally I have a serious problem with the backscatter they cause. They should reject mail at SMTP time and not bounce them.

But Barracuda support is very very good. Very responsive and timely and overall a good people orgaization which can make the difference for wanting to deal with them.

Re:Barracuda SPAM filter

[Arrogant-Bastard]

There are multiple, very serious problems with Barracuda appliances. I've already commented on their propensity to generate backscatter elsewhere in this thread. They're also poorly supported, have systemic security issues, may have privacy implications (since Barracuda personnel have unauditable access to your mail stream), are expensive, use community resources such as DNSBLs in ways contrary to those resources' policies, and do not use current best practices in spam control. (This last is unsurprising given that Barracuda personnel do not participate in the discussions and consensus-building which generates those BCPs.)

Consider as well that the Barracuda appliances consist of (a) an open-source operating system (b) an open-source MTA (c) an open-source web server (d) an open-source spam scanner (e) an open-source virus scanner (f) other pieces of open-source software and (g) use community-mintained DNSBLs and RHSBLs. This is all held together with proprietary (closed-source) code, mostly for the purpose of providing a poorly-designed GUI interface. Any competent email system administrator should be able to create their own near-equivalent in an afternoon; it's not difficult. Such homebrewed creations have repeatedly been shown to vastly outperform Barracudas on multiple metrics, including cost, scalability, customization, security, and perhaps most importantly -- adaptability to new spammer techniques. (Barracuda is years behind the times and falling further back.)

It's very tempting to "just buy an appliance" and consider the problem solved, but it doesn't work. There's no substitute for expertise -- and given that much of that expertise is available for free, for the asking, on lists such as spam-l and spamtools and so on, it's difficult to understand why anyone would choose not to avail themselves of it.

email != IM

[Viraptor]

> maybe an important time-sensitive email never gets to its intended recipient

When will users learn...
Email is not instant messaging - with bad greylisting / random connection reset / busy server, you can get >=2 hours delay. And it's normal.

Re:email != IM

[cfulmer]

Your assessment of the current state of email is correct. But, blaming users for using it to fill a need when there is no realistic alternative is silly.

email is ubiquitous and easy. 99.5% of the time, it's nearly instantaneous. Should I really have to get an IM account on google, yahoo, aim, microsoft, etc.... so I can deal with time-critical messages? And, for that matter, should everybody else?

Power to the people :)

[grantdh]

Whatever solution you get, the simple answer is:

1) Set up the system to put junk mails in a folder the user can see

2) Train the end user to check their junk mails

3) Show the user how to set the spam triggers high or low and what the implications are

If user says they're too busy/important, advise them that due to your workload, their email box will be added to the "manually checked list" which gets done once per week. Point out the impact of losing a time-critical email wrongly flagged.

Most times they do it themselves. For those who are dead set on having someone else do it, hire a temp or arrange for an office junior to do it.

If you're in IT, you have better & more important things to do than check for real mail in a junk mail box...

OpenBSD spamd

[DaMattster]

I've had excellent results with this particular product. Spamd uses blacklisting, greylisting, and tarpitting. It really is delightfully evil and still makes me smile because it includes a fake smtp daemon which sets the tcp rcv window to 1. This is a kick in the nuts to the spammer. I've used it with resounding success at a client who was recieving 2000 spam emails a day. Prior to implementing spamd, we were using just a Barracuda. When I combined spamd and the Barracuda, spamd caught about 1975 of the spam messages and the barracuda took over from there. No false positives and we've been running for three months. This link details how to set it up, http://www.linux.com/feature/61103.

This is largely a known-solved problem

[Arrogant-Bastard]

The place to ask this question isn't here, it's on the "spam-l" mailing list, which arguably has the highest concentration of the world's most experienced anti-spam researchers and developers. Simple techniques for tackling this have been repeatedly covered there over a period of many years, and their behavior is well-understood and predictable, making them viable choices for production systems. So I would suggest that you subscribe to that list (via listserv@peach.ease.lsoft.com) and repeat your question there, along with some indication of your MTA environment.

Meanwhile, here is some general guidance. First, do not waste your money on commercial products -- they're expensive, poorly-maintained, and in many cases (e.g. Barracuda) actually make the spam problem worse via backscatter. (There are now several thousand Barracudas on a communally-maintained blacklist, making it obvious to everyone working in this field that Barracuda is completely incompetent.) Second, do invest your money and time in open-source solutions: it is easy for anyone who possesses baseline competence in mail to craft their own, superior spam handling system using postfix or sendmail or another open-source MTA, DNSBLs, RHSBLs, judicious configuration, and other tools such as rbldnsd, mimedefang, SpamAssassin, ClamAV, and so on. Third, a little googling will reveal near-cookbook procedures for combining these pieces of software together into a useful system; which cookbook procedure is appropriate for you depends on your environment -- which brings me to the fourth point, which is that you need to perform log analysis in order to understand your particular mix of spam/not-spam. Everyone's is different, which is why one-size-fits-all solutions usually fail. Only after you have some clue about the size and shape of your problem will you be able to determine which approach(es) are likely to minimize both false negatives (FN) and false positives (FP).

As an aside, one set of highly effective anti-spam tactics involves enforcing RFC requirements that have been in place for many years: for example, all mail servers must have rDNS; that rDNS must resolve to a host which in turn resolves back to the IP; the domain of the host must exist; the host must HELO as a valid FQDN or bracketed-quad IP; the envelope-sender's domain must exist; the host must not HELO as you; the host must wait for the SMTP greeting before HELO'ing; the host must handle a multi-line SMTP greeting; the MX records for the host must point to valid IP space; and so on. Enforcement of these requirements yields differing rates of spam control (which is again why log analysis is crucial) but has the very valuable property that it can be done at low computational and bandwidth cost. Substantial experience with these suggests that enabling them and augmenting them with a few DNSBLs (especially the Spamhaus Zen zone) is enough to deal with the overwhelming majority of the spam problem at most sites, reducing what's left to a much smaller issue to be dealt with.

Re:Combined effort is necessary

[entrigant]

The use of dnsbl/enhdnsbl also does bounce back to the sender with a reasonable message for the cases where a message is denied so the sender shall be informed about any messages that are denied. Of course - it isn't fool-proof, but it works for me.

Do you generate a bounce, or do you reject with a 500 error and a proper message at spam time? You should not generate a bounce to remote mail. Ever. This is the cause of e-mail backscatter and is a significant problem. Always reject at SMTP time with a 500 error.