Slashdot Mirror

I'd love to know what kind of levels of success you're seeing in capture and rejection of SPAM. But I guess that information isn't publicly available?

For what its worth I just published this document describing how my defunct anti-spam proxy service worked. It'd be great to see what other people do - sharing these kind of details would help everybody involved.

(I fight spam commercially.)

Much of the spam I see today is both sent from American machines and advertising American companies.

Some of this spam is sent from shady hosters, and the rest from zombie machines (or at least I assume zombie machines I see lots of spam sent from home ISPs such as ".cpe.net.cable.rogers.com").

But it is unfair to pick on the USA, as local spam from home broadband in the UK is just as prolific. (e.g. *..craw.blueyonder.co.uk)

That solution is quite interesting, because its the sort of thing that a lot of people were expecting to happen - virtual images being used as black-box applications.

I love the idea of being able to download a webserver in a box, or a caching HTTP proxy server. There are many other applications which would be nice to see provided like this. Of course in my case I would be hosting them on Linux, but I guess whatever host machine you use is irrelevant so long as you understand it and can support it.

Of course I'm a little biased when it comes to spam filtering, but I hope the idea of custom VM images catchs on more generally.

There are downsides such as the overhead of emulating a whole machine for a single service, but I'm sure the benefits outweigh them if you have spare host capacity (*2 for redundancy)

I'm curious though, did you configure the guest yourself, or find it as a pre-rolled virtual machine image?

Indeed there are many companies like mine selling spam filtering which works via SMTP-proxies.

It is a neat architecture that allows you to off-load spam/virus/junk filtering from your mail server and scale horizontally.

(I run a commercial mail filtering system - using these techniques and more I filter about 5 million spam messages a month.)

There are many more things you can do, from examining the "HELO" presented, looking for matching DNS, dropping mails without a "DATE" header (mandated by RFC).

Beyond that you can detect spoofed mails which have "From: you@you.com" + "oo: you@you.com" being sent by server1.foo.ru.

Similarly you can run greylisting (don't like that myself, but some people do), and be very strict on your domain recipients (ie. no wildcard addresses).

One final thing that is useful is to drop connections from machines that try to blurt an entire SMTP transaction without waiting for the opening SMTP banner to be sent

Indeed - I run a spam filtering proxy service - and I have fight daily with incoming spam.

The end user doesn't see it, but it is still clogging up the network, and causing some of us real headaches.

When it comes to my personal mail I receive maybe 4-10 thousand spam mails a day. I see a couple of those, but the ones I don't? They still cost me bandwidth and processor overhead to filter.

It isn't clear from your guide why you've thrown Archiveopteryx into the mix.

Postfix can be configured to send/receive mail directly - and handle all the authentication.

Just on the basis that you've downloaded it outside the Debian package tree suggests it might get out of date in the future.

If you need it to do IMAP/POP3 handling, etc, then you might want to look at dovecot.

Also you say "postfix handles more mail"; that might or might not be true. But if you're handling such a high volume of mail that exim4 can't keep up your best bet is to look at adding a second MX machine - and as soon as you do that the fun really starts.

ObBias: I run a mail filtering service...

The best solution in this case it to actually disable the wildcard forwarding - and only accept messages for the actual users which should exist upon the domain.

That won't cut down spam, but will instantly remove the problem of dictionary attacks.

Pretty much any mailserver should be able handle this kind of restriction, and if not you can outsource it to people like me!

I think most decent competent SPAM filtering companies allow you to see a quarantine of "rejected" messages - I know that my service does.

That way if they make a mistake (and lets face it sooner or later they will) you have the chance to find it.

Still given a large enough quarantine checking through it manually becomes almost as much of a chore as doing so locally would have been.

My mail filtering service is currently hovering around 2.3 million mails - which is a little down from its peak.

Still these things tend to even out over time; a few days/weeks of lower-than-average SPAM totals then a few more of higher than average.

With only a couple of domains, anecdotally at that, I'd be inclined to assume nothing has changed significantly.

Indeed those automated bounce messages are something we should have moved away from a long time ago.

(Ditto challenge-response systems; but thats a whole other subject.)

My own service gets that right at least!

As you say though if you've got the time, patience, and ability then most of the commercial systems may be bettered for your own setup - its just a matter of deciding whether you want the hassle, or whether you want to outsource it to somebody.

Indeed. I've setup a small mail filtering / anti-spam service, and the privacy is a big part of that.

Some people just don't, won't, and shouldn't trust google...

There is a fair amount of truth in your comment - speaking as somebody who has taken a bunch of open tools and made them play nice though it is harder than you'd expect.

My own solution is built upon the top of QPSMTPD exim4, and a Debian operating system.

Whilst you can come up with something similar there is a lot of benefit to scaling up and handling messages en masse. That's why Google's gmail has such a good reputation (until recently?).

If you have 1 million spam messages passing through your system at any given time and 50 people mark them as spam you've managed to collectively filter out the messages with no real overheard. (Sure you need to make sure they're not malicious reports; but basically getting a lot of users to report the 1/2/10 messages they see as spam helps out all your other users.)

Also, and I might be overemphasizing this because I find it hard personally, designing a good user interface, can be the difference between a "meh" service and a real winner. So it's not a good idea to write off that amount of effort!

They use me!

More seriously there are many approaches that you can take, from the DNS-blased blacklists, bayasian filtering at SMTP time, and then any local content-filtering rules.

Spam is constantly evolving though, so you might find it more productive to just outsource it as others have suggested. (I couldn't recommend gmail though!)

Companies such MessageLabs, etc, exist and do a good job. There is even my own service which uses a nice configurable combination of DNS blacklists, bayasian filtering, valid user detection and more - the advantage to my service/system is that each rejected message is quarantined for a month so you can easily catch false positives.

This company allows you to outsource spam filtering. The founder is a well-known OS developer, so it may be worth a try.

I am an off-site mail filterer, and our stats show that 99% of incoming mail is spam.

Indeed. The standard "reject" or "accept" solutions are very brittle - because the any false positives are disasters as far as your users are concerned.

my solution involves rejecting mail which is determined to be SPAM at SMTP-time but also keeping a copy if it in a quarantine where it may be searched/viewed/re-delivered.

It is still brittle - but the sender knows that their message was not delivered, due to the bounce, and the recipient does have a copy in their quarantine area, if they look for it.

Although recently a Debian Developer was critical of slicehost, and seemingly in a valid way.

Personally I host a reasonably high-traffic antispam service and I think Amazon's offering looks good, but as mentioned a little pricy.

I love the idea of adding extra nodes on-demand, but I think I'm not yet at the level where it would be a worthwhile use of my time or budget.