Slashdot Mirror
Spam Filtering For Small/Medium Business?
or_is_it writes "The company I work for has been growing dramatically and I've been charged with the task of being the gatekeeper for our GFI Spam filters. This involves manually inspecting the subject line/to/from for all caught messages in each filter rule folder. For a company of about 50 people, in one day the number of spam messages can exceed 2,000. Neglect it for a day and you end up with quite a task on your hands. I've made the rules lax enough so important messages can go through, along with a few stray spams, for which I get bitched at. Tighten the rules up and then maybe an important time-sensitive email never gets to its intended recipient, and I get bitched at. Manually reading through all those subject lines is supposed to prevent that, but I'm only human and genuine messages can easily get overlooked. How do larger organizations deal with the spam issue? I can't imagine having one centralized person manually inspecting everyone's junk-mail header is the optimal solution. Purchasing a different commercial mail filter product is a possibility, but I'd like to hear some anecdotal evidence before jumping ship."
To be honest, for somewhere of that size I'd be tempted to use some sort of client-based filtering (along the lines of spambayes [http://spambayes.sourceforge.net/]) which would put the power and responsibility in the hands of your users.
-- Gaxx
I purchased a Barracuda for my organization of about 120 employees, and it has been fantastic. I fine tuned a few options on the config and it has blocked about 200,000 emails in the almost two months i have deployed it. There are very few false positives, and very few that get through its filters. I actually get calls of gratitude from the end users about how happy they were not receiving any more SPAM messages. The hardest part was informing them the user base on the difference between the mailing lists they were on and SPAM. Barracudas support has been good as well.
> maybe an important time-sensitive email never gets to its intended recipient
When will users learn...
Email is not instant messaging - with bad greylisting / random connection reset / busy server, you can get >=2 hours delay. And it's normal.
I've had to send emails to recipients within the Australian Defence Force (specifically, the Army), and every email sent from a civilian must include a keyword within the subject line. The keyword is to do with whether or not the information is classified or unclassified. Sure, getting all the clients to send all their emails with [companyname] in the subject line is a little annoying, and may not be possible depending on your circumstances, but the chances of spam having that keyword within it is virtually impossible.
Set up an automated filter whereby anything that doesn't have the keyword in the subject gets dumped into a spam box to be sorted later. If the senders do the right thing, it assures their emails will be directed to the correct person.
This is just one example of active spam filtering as opposed to the passive spam filtering used in IT today.
You moved your mouse. Please restart Windows for changes to take effect.
Whatever solution you get, the simple answer is:
1) Set up the system to put junk mails in a folder the user can see
2) Train the end user to check their junk mails
3) Show the user how to set the spam triggers high or low and what the implications are
If user says they're too busy/important, advise them that due to your workload, their email box will be added to the "manually checked list" which gets done once per week. Point out the impact of losing a time-critical email wrongly flagged.
Most times they do it themselves. For those who are dead set on having someone else do it, hire a temp or arrange for an office junior to do it.
If you're in IT, you have better & more important things to do than check for real mail in a junk mail box...
I left my body to science, but I'm afraid they've turned it down...
How do larger organizations deal with the spam issue?
I used to work for a mining company you've heard of. Our department had responsibility for managing the email vendor, who used Spamshark to filter spam coming into the organisation. From my limited knowledge of the setup, Spamshark does basic blacklisting etc. but also does selective blacklisting on specific IPs when an email is flagged by a user. So Alice flags a message as spam, Spamshark figures out the message id, grabs the IP address it came from (it knows because it previously handled the email), and then blacklists that IP for a certain amount of time. Now this internal blacklist is then shared to all the other customers who use Spamshark, so they are now protected too; resulting in a 5 nines hit rate on spam.
Like I said we just handled vendor relations, and the above description might not be totally accurate, but this is what I gathered when we dealt with them. I also remember getting about 10 complaints of spam a month for an organisation with 10's of thousands of email addresses - so it was very effective.
"And then I visited Wikipedia
I've had excellent results with this particular product. Spamd uses blacklisting, greylisting, and tarpitting. It really is delightfully evil and still makes me smile because it includes a fake smtp daemon which sets the tcp rcv window to 1. This is a kick in the nuts to the spammer. I've used it with resounding success at a client who was recieving 2000 spam emails a day. Prior to implementing spamd, we were using just a Barracuda. When I combined spamd and the Barracuda, spamd caught about 1975 of the spam messages and the barracuda took over from there. No false positives and we've been running for three months. This link details how to set it up, http://www.linux.com/feature/61103.
Meanwhile, here is some general guidance. First, do not waste your money on commercial products -- they're expensive, poorly-maintained, and in many cases (e.g. Barracuda) actually make the spam problem worse via backscatter. (There are now several thousand Barracudas on a communally-maintained blacklist, making it obvious to everyone working in this field that Barracuda is completely incompetent.) Second, do invest your money and time in open-source solutions: it is easy for anyone who possesses baseline competence in mail to craft their own, superior spam handling system using postfix or sendmail or another open-source MTA, DNSBLs, RHSBLs, judicious configuration, and other tools such as rbldnsd, mimedefang, SpamAssassin, ClamAV, and so on. Third, a little googling will reveal near-cookbook procedures for combining these pieces of software together into a useful system; which cookbook procedure is appropriate for you depends on your environment -- which brings me to the fourth point, which is that you need to perform log analysis in order to understand your particular mix of spam/not-spam. Everyone's is different, which is why one-size-fits-all solutions usually fail. Only after you have some clue about the size and shape of your problem will you be able to determine which approach(es) are likely to minimize both false negatives (FN) and false positives (FP).
As an aside, one set of highly effective anti-spam tactics involves enforcing RFC requirements that have been in place for many years: for example, all mail servers must have rDNS; that rDNS must resolve to a host which in turn resolves back to the IP; the domain of the host must exist; the host must HELO as a valid FQDN or bracketed-quad IP; the envelope-sender's domain must exist; the host must not HELO as you; the host must wait for the SMTP greeting before HELO'ing; the host must handle a multi-line SMTP greeting; the MX records for the host must point to valid IP space; and so on. Enforcement of these requirements yields differing rates of spam control (which is again why log analysis is crucial) but has the very valuable property that it can be done at low computational and bandwidth cost. Substantial experience with these suggests that enabling them and augmenting them with a few DNSBLs (especially the Spamhaus Zen zone) is enough to deal with the overwhelming majority of the spam problem at most sites, reducing what's left to a much smaller issue to be dealt with.
dnsbl/enhdnsbl is enabled for zen.spamhaus.org, bl.spamcop.net, combined.njabl.org, list.dsbl.org, dnsbl-1.uceprotect.net, dnsbl-2.uceprotect.net, dnsbl-3.uceprotect.net and sbl-xbl.spamhaus.org. With all these enabled there are very few spam messages falling through.
Adding to this I am using Mozilla Thunderbird which has a very good intelligent junk mail filter. The only disadvantage is that the junk mail filter has to learn what's junk or not.
The use of dnsbl/enhdnsbl also does bounce back to the sender with a reasonable message for the cases where a message is denied so the sender shall be informed about any messages that are denied. Of course - it isn't fool-proof, but it works for me.
If builders built buildings the way programmers wrote programs, then the first woodpecker would destroy civilization.
Why do people keep suggesting gmail as a viable option?
It's really not that good.
An SQL query goes to a bar, walks up to a table and asks, "Mind if I join you?"
I cast my vote for SpamAssassin.
When set-up with good rules and RBLs it blocks at least 99% spam with very low false positives (I've never had a false positive).
Send anything tagged as spam to another account such as spam@domain (I do this) then you can manually check for false positives to further reduce the chance of losing legit email. (or if a user complains that an email they expected never arrived)
An SQL query goes to a bar, walks up to a table and asks, "Mind if I join you?"
I can't imagine having one centralized person manually inspecting everyone's junk-mail header is the optimal solution
Actually, that strikes me as a good solution; it's certainly better than having other employees dealing with spam as part of their daily routine and losing 30 minutes/day for everybody in the company. And by centralizing it, you have the ability to pick the tools to make your work more efficient, as opposed to having 50 employees each fiddle with their own spam filters.
This is just a simple guide compiled from my experience:
:)
1. Do what you can on the server. I like to use SpamAssassin to add spam scores to beginning of subject lines, so they sort by score in my inbox (I use "/*_SCORE(0)_*/"). I also automatically delete anything over a score of 11, since the highest I've ever seen a legitimate email score has been "10.something". Realistically, anything above an 8 is the sender's fault and they need to do something about it and anything above an 11 you can safely blame the sender (you won't be the only spam filter deleting their emails).
2. Provide the tools on the client. ThunderBird's "spam marker" is a must, and because it learns from what you mark, you aren't just marking them in vain. Also, to deal with spam in real-time, instead of using the junk folder, I like using the "delete junk!" button from the "Buttons!" add-on. Incoming junk gets marked and marked as read, and after marking the spam the filter missed, I hit "delete junk". Very easy and quick. Pre-configure Thunderbird for everyone.
3. Educate and support. If you have 1 and 2 in place, then make sure everyone knows what you are doing and why you chose to do it. Write a short manual or something. Educate them about their tools. They also need to know NOT to publish their addresses.
The idea is to make spam highly visible, and to make it *quick and easy* to deal with. Knowing you've facilitated these two goals should be enough to impress your employer and earn the respect you deserve from everyone you serve
I spent a few days migrating 100,000 emails from Windows Mail, because it was horrible. Thunderbird is a godsend and the add-ons make all the difference. If there is something you dislike or want, chances are someone made an add-on for it.
btw 2000 messages is *not* a lot of spam. It will get far worse with time.
I manage self-hosted email for several small-medium companies. ASSP is platform independent, low resource, and does a VERY good job. VERY very configurable, and free, open source, easy to modify, easy upkeep (almost zero action required beyond checking the logs to keep an eye on things) and free software.
In a company of about 75 email accounts it has blocked 4 million spams in a little over a year.
The false negative rate is so low it might as well be zero, and the false positive rate as well.
It uses among many other things whitelists,so your people never miss an email from an established contact, redlists, so a known spammer cannot ever be accidentally added to the whitelist, does spf checking, checks headers against spoofing, has an antivirus component, can forward a copy of all spam to a spamlover address and much much more.
and its free.
For a single sbs server, you can install it on the same box and zero out of pocket costs except for your time to install it (I would personally budget 20 hours for R&D for a first time administrator to install it).
Please email me if you want more detailed information on how it works for my clients. I can also put you in contact with end users at the executive level of these companies to ask how they like it (the final litmus test)
Good luck
Chuck
Well, it's a two-edged sword.
I run email for several of my domains through Google Apps for Your Domain - essentially, Gmail. On my largest account, I get several hundred legit emails and 200-1000 spam messages each day. The problem isn't Gmail's filtering of this - it's actually damn good, with maybe 2-3 false negatives a week and maybe one false positive. Better than almost anything else I've seen.
The problem is that Gmail gives me NO options - as a user or domain administrator - to sift through the spam box automagically, looking for those false positives. You CANNOT access the spam box in any way other than their web interface, looking manually through your spam, hoping to see the occasional legit message that confused the filters and was labeled spam. (Okay, if you go the full IMAP route, you can apparently see it, but that's cumbersome in the extreme if your users aren't doing IMAP in the normal course of things.)
This borders on perverse. How hard would it be to allow POP to the spam box, so that I could suck down the messages and run my own filters on them? And what's with the lack of user filtering options? "Um, Google, here's a hint: I don't read Chinese or Russian. If mail comes into the spam folder in one of those languages, you just delete it and not bother me with it, OK?".
Dunno, it feels like a case where someone's high up in Gmail's design group has a religious or aesthetic conviction about how spam should be handled ("no filters...no settings...no controls...no access") that blinds them to how badly this works for users and administrators in the real world.