Slashdot Mirror
Ask Slashdot: Speeding Up Personal Anti-Spam Filters?
New submitter hmilz writes "I've been using procmail for years to filter my incoming mail, and over time a long list of spam patterns was created. The good thing about the patterns is, there are practically no false positives, and practically no false negatives, i.e. I see each new spam exactly once, and lose no legit mail. This works by using an external spam-patterns file, containing one pattern per line, and running an 'egrep -F' against it. As simple as this is, with a long pattern list this becomes rather slow and CPU consuming. An average mail currently needs about 15 seconds to be grepped. In other words, this has become quite clumsy over time, and I would like to replace it by a more (CPU, hence energy) efficient method. I was thinking about a small indexed database or something. What would you recommend and use if you were me? Is sqlite something to look at?"
have you tried spamassassin?
Look up CRM114.
Write something that uses a regular expression library (RE2 would be ideal, if your expressions are actually regular), and keeps the compiled patterns resident. Most of your time is likely spent parsing the patterns.
What would the database achieve? I'm not sure what is the exact nature of the patterns (an example would really help here), but perhaps writing a compiler from the patterns into some decision procedure in something reasonably efficient yet featuring quick start, such as SBCL or Gambit, could help.
Ezekiel 23:20
http://bogofilter.sourceforge.net/
I haven't timed it to see how well its been doing in the 6 years I've had it though.
X(7): A program for managing terminal windows. See also screen(1).
Sorry, couldn't resist the pun.
Your problem (besides not using existing Bayesian tools...) is that every single egrep is a fork. As others have pointed out, you should rewrite your script in something like Python and use the native regex libraries. Even if you have to read and 'compile' the regex list every time, you're saving a *massive* amount of OS-level overhead.
It seems you could easily distribute the load on multiple machines, each doing a subset of the regex.
Try compiling your patterns using Ragel: http://www.complang.org/ragel/
Union them all together and you'll see orders of magnitude improvement in performance (e.g. 10x - 100x) over other regular expression engines, although GNU grep is using Aho–Corasick with the -F switch, so you're likely to see less of an improvement.
Many people use re2c, but it has nowhere near the performance or capabilities of Ragel. Ragel has a steep learning curve, but it's well worth the effort to master. It's well maintained, and has been for years.
http://www.gigamonkeys.com/book/practical-a-spam-filter.html has the nuts and bolts. CL-PPCRE does perl regex matching faster than perl.
Don't complain about syntax, grammar, or spelling. There is no.hell like input on android.
Does your process require that all of the regexes are tried in turn or is it the case that if it hits one of your patterns that it's marked as spam? If the latter, are you able to rank the patterns from most likely to least likely to be matched? And, if so, can you stop your process once a match is made? If all of those things are true, then you should be able to cut the time/CPU/energy required to do the filtering
GMail's false positive rate is absurdly high.
Consider using a proper learning filter, like dspam. You can pipe it through procmail just as easily, and you can feed your corpus of spam into it. You won't get 100%, but it'll recognize spam you haven't seen. :0f /usr/bin/dspam --deliver=stdout
*
|
I've heard, but never timed it myself, that perl is faster for regexp-type stuff than even the specialized tools, just from the massive amount of optimization it has accrued over the years; here is a completely unbiased source. Use a perl or python script, and consider using Storable (perl) or pickle (python) to serialize the data structure, I guess, but just having the whole list in memory will help.
According to this, perl regexps are (unsurprisingly) a superset of egrep's.
I don't see how introducing SQL could do much to help speed, or anything else, in this application.
"They were pure niggers." – Noam Chomsky
Which pretty much defeats the whole point of hosting your own email...
Well, the OP wasn't exactly clear if it was just his personal account or whether it's a corporate server. My "least amount of work" thing is to forward every email address I have to gmail, and pull mail from there via imap. I get a few hundred spams a day just on one mail account, and I haven't lost any real mail due to Gmail's filtering.
>postini
You do realize that is being EOLed, yes?
http://postini-transition.googleapps.com/
>why gmail filters better than postini
Probably separate spam databases. Stuff like that happens. Gmail probably gets orders of magnitude more spam to "teach" the system.
YMMV.
Using grep and procmail is the stone-knives-and-bearskins approach to filtering. There are a lot of other filtering systems that will be much more efficient on Unix systems. He can begin by using greylisting to filter out the non-compliant "fire and forget" spambots and then filter the winnowed pile o' crap. At least greylisting's not server intensive (it throws the load back to the sender) since 5xx and 4xx errors are cheap.
Also blocking mail from dynamic IPs is a good idea.
At this point he can then run the mail through a series of weighted RBLs. Reach a certain score and it's tossed. That's the processor intensive bit, but it's at the end and non-intensive filtering has already happened.
--
BMO
Many years ago I worked with a Unix development tool called LEX that could handle matching multiple patterns simultaneously. Perhaps there is an updated tool that would do the same thing. Java has a 3rd party library called ANTLR that might do the trick. It would involved re-compiling every time a new pattern is added but it should be extremely fast.
Sqlite, or anything that uses an index, will be screaming fast.
Your statement of your current solution makes me wonder, though.. are you using "egrep -F -f pattern_file e_mail_message"? Or are you running egrep many times, once per line of the pattern file, or once per line of the message? I would think that given a pattern file egrep would be smart enough to do something better than repeatedly scanning the input, but based on the time it's taking, it sounds like that's happening.
Note to ACs: I usually delete AC replies without reading them. If you want to talk to me, log in.
... I just gave up on email. Even w/o spam it's more hassle than I like.
now we need to go OSS in diesel cars
Use Perl. Its regex engine is highly optimized and very fast. It should really fly on fixed strings.
There's a Stack Overflow question that addresses this very thing with some Perl code you can try.
Just route everything from Facebook, LinkedIn, my dad, Apple and "i*" to the spam folder, and most of it is covered.
This issue is a bit more complicated than you think.
Is it? I've never had a false positive in all the years I've been using GMail.
I second grey listing and using spamhause filter lists.
It is well worth it. I had a 30 user environment receiving about 100 to 300 spam emails a day per user go to approximately 10 a piece making it through. Then when I activated the spam filtering it went to about 10 a week for about 1/3 of the users. The biggest problem is third party user machines being compromised and the spam being sent through their internet's email servers (grey listing doesn't stop legitimate servers and most big ISP's don't make it on the spamhaus lists). Usually this contains a virus attached to it and the antivirus on the mail server catches it.
PS.. the reason the spam was so high is because they sign up for all sorts of crap from their work computers and use their work email almost as a personal email. The partner's don't mind as long as their work is getting done. The biggest offenders were the partners (read owners) themselves.
Is it? I've never had a false positive in all the years I've been using GMail.
That you noticed. There's a fairly high bias inherent there; it just has to not have hit something that was both noticeable and that you knew was incoming.
The problem is that you're using egrep in the first place. Here's the thing -- the overwhelming majority of your cycles are getting sucked loading, initializing, executing, then unloading, that thread. It's not that using regular expressions is processor-intensive... it's that repeatedly launching the same executable is.
Use something that can load once, read in the patterns, check all the e-mails that are queued, sort them, then exit. Your execution time will go from 15 seconds to 150 milliseconds.
#fuckbeta #iamslashdot #dicemustdie
If spam has made it far enough that it's actually reached your personal instance of procmail, then there's been a problem earlier in the chain. Procmail rulesets should be a last resort, and they should only be asked to deal with minor issues that aren't dealt with via earlier rulesets.
The first line of defense are your perimeter routers. They should implement BCP 38, they should block bogons, and they should bidirectionally deny all traffic to/from the Spamhaus DROP list. In addition, they should block inbound port 25 traffic from everywhere on the planet that you don't need email from. In other words; the fact that someone in country X wants to email you is unimportant unless you actually wish to receive mail from them. Yes, this is a reversal of default-permit, for a simple reason: default-permit for SMTP stopped being reasonable around 2000. Use http://www.ipdeny.com/ to pick up the ranges per-country and only permit what you need. (Obviously a major research university can't do this. But Joe's Furniture, which does not have customers in Peru or Pakistan or Greece, can.)
Then use blacklists, the best defense against spam we've ever developed. (Source: 30+ years of email experience) Spamhaus's Zen blacklist is a good one with a low FP rate and a tolerable FN rate. Augment these with local blacklists based on domains and network allocations. Augment those with as much blocking of generic hostnames and dynamic IP space as possible: real mail servers have real hostnames and are on static addresses.
Then enforce RFC requirements: sending host must have rDNS, that PTR must resolve, what it resolves to should be the sending host's IP. Sending host must HELO as FQDN or bracketed dotted-quad; if FQDN, must resolve. Sending host must not send traffic pre-greeting. And so on. Enforcing these DOES mean occasionally you block mail sent by non-spamming entities: but since they are incompetent non-spamming entities, why would you want mail from them?
Add greylisting. It'll handle a lot of annoying hosts that haven't learned to retry yet.
Rate-limit based on normative values for your site. For example: if analysis of a year's worth of mail logs shows that during that time you never received more than 10 messages a day from ANY host, then rate-limit at 30 or 40. You'll never hit in normal practice; but if you get hammered by a fast-sending host, you'll blunt the attack. Note that these don't have to be perfect to work: provided you send deferrals (SMTP response codes 4xx) instead of refusals (5xx) the worst that happens is that you will mistakenly impose a delay.
There's more -- it's possible to get quite crafty about this. But note that NONE of these measures pay any attention to content. There's a reason for that: spammers can defeat content-based measures at will. They won't have it so easy with these.
Deployed in production in various setups ranging from a dozen to eight million users, these steps yield a FP rate of about 10e-6 to 10e-7 and a FN rate around 10e-5 to 10e-6. Tuning helps, of course: initial rates can be higher but log analysis (which all sensible postmasters do) readily brings them down. If you have the luxury of running your own mail server just for yourself, then you can REALLY tune this setup: you should be able to get the FN rate down to 10e-7 after a few months.
Email from people at one site I look after used to vanish into a black hole at gmail until I convinced them to replace the GIF of their corporate logo attached to all their emails with a PNG version. That's some real mail lost due to gmail's filtering.
IMHO it's better to do the filtering somewhere where you have access to the stuff that is discarded. False positives may be rare now but they still happen. That's why I like stuff such as MailScanner (open source wrapper for spamassassin+your choice of commercial antivirus and/or clamav+other open source stuff+distributed updating rulesets) run on site. There's plenty of others that give you this function including some of the commercial "appliances" and outsourced email filtering.
It used to be the case that one IP address I have a mail server on would get blocked for a couple of days every year because some idiot at a blacklist would load in an obsolete list of dynamic IP addresses from what is now a decade ago. As IPv4 addresses diminish expect the lists of dynamic addresses to become outdated very quickly.
When my ISP discontinued the use of procmail filters, I moved it to my home computer and configured two filters in Evolution: the first one to auto-remove mail marked by my ISP as suspected spam, and the next to pipe the mail through bmf and remove it if it tested positive for spam. When I say "auto-remove", I mean it's moved to a spam folder where I can double-check it in case false positives get through.
http://sourceforge.net/projects/bmf/
And allow Google and the US government to scan all of my mail? No thanks. The same goes for Hotmail, Yahoo and any other commercial email service provider (and certainly those based in the US). These days it makes more sense than ever to maintain your own MTA.
I've been using popfile for years. Works great! Try it.
i'd be interested to see what happens if you run those regex's through this:
http://bisqwit.iki.fi/source/regexopt.html
btw can we please get a copy of the patterns you're using? i think they might prove useful for other people. also i'd like to test them myself against regexopt.
oh - to the other person who suggested spamassassin? i tried that, i set it up to run at MTA-time. it often took THIRTY SECONDS to process a message. in fact it was so bad that i was forced to set a limit of 100k on incoming messages, as a lot of virus-ridden word documents (etc) were typically over 100k. that cut down the amount of CPU cycles but it was still far far too much memory and far too CPU intensive.
the one thing that did work well is greylisting, however the problem with greylisting i find is that if you happen not to be at the computer or have direct access to the server and people on the phone say "i'm sending you a message now, have you got it?" you *know* it's going to be at least an hour before it'll arrive. so, unless you can whitelist them in advance (which you can't always do) greylisting does actually interfere with legitimate business.
anyway: in the end i gave up and went to gmail, but with gmail fucking up how they're doing things i have to revisit this and set up a mail server again. thus we come full circle...
Reducing the time in greylisting does reduce the potential damage but then it's a balance between the patience of the people that expect instant communications and the patience of spammers. While most spambots only tried once in 2005 things have moved on since - greylisting is well behind the spam arms race.
It stops them long enough for it to be a problem in enough cases that I kept getting a lot of "why doesn't X have my email yet" phone calls when greylisting first became popular. I then ran it myself for a while to see what was going on and to see where some of those who were using it were applying frankly insane settings, and how even less tight settings were problematic on occasion.
Sometimes it's better to look at entire systems to resolve problems instead of a tightly focused technical only approach. If you guys are going to call yourselves "engineers" you should act like them and consider entire systems instead of single bolts or what the manual tells you to do. Cute tricks that fuck around with communication policy shouldn't be used unless you can take the consequences of changing communication policy. If it's going to put your boss on the carpet in front of the CEO you have a duty to your boss of explaining to them why you are doing it.
Unless it's a fun hobby for you, it makes much more sense to just pay for email and let somebody else to it. Personal email can be gotten for about $2/month.
I don't respond to AC's.
The canonical spam solution checklist.
I'm going with Specificaly, your plan fails to account for: (x) Users of email will not put up with it.
Help stamp out iliturcy.
Provided your pattern file is under 340K, 'agrep -f' is about twice as fast.
The default resend interval on most mail systems is between 2 and 15 minutes that I know of. If you are using it, it doesn't impact anyone you are sending to. If who you are sending it to is using it, it is their problem not yours.
I'm also surprised by the comment of " never had email from them before". First, are you confusing grey listing with white listing and a challenge response? Second, I'm not sure I would be sending something I worked on all night to someone I never communicated with before by email. It might be possible that the specific person is a different person, but the email should work domain wide (if I emailed your secretary I should be able to email you without the grey listing).
If grey listing stops the email from being received for hours, there is something wrong with the server sending it. Most default times are minutes. I'm not sure I know of a scenario that would require specific minute by minute communications that wouldn't warrant a phone call or something more instantaneous. Even without grey listing, the emails can be delayed for several minutes. Most email clients do not update more then every couple minutes anyways.
The longest I have seen an email delayed from grey listing is about 5 minutes. Anyways, I'm not sure we are talking about the same things. Grey listing simply drops the first connection attempt by a server not in a white list and requires the smtp server to retry at their set interval. There aren't a whole lot of setting you can do besides specifically allowing domains or servers and specifically denying them. The person sending the email might have their servers jacked around but that is their problem, not yours. If a route goes down somehow and the communication is interrupted, it will retry the communications anyways- this is no different except it's on purpose.
I'm not convinced we are talking about the same things here. Like I said, the longest I have seen an email delayed is about 5 minutes. I say about because the log measures seconds between connection attempts when I check them and I have never seen them go over 300-350 seconds unless something was wrong with the route (traceroute fails to complete).
there are million dollar companies that can detect it faster and even better than your OSS bullshit half assed script for free
The NSA, for example. Use a US server as your email service provider, and you get filtering for free!
Ask me about repetitive DNA
And allow Google and the US government to scan all of my mail?
Well, routing it outside of gmail keep gmails hands off it, well the half that doesn't originate with them to start with. But i think you'll need to do quite a bit more to keep the us govt out of it.
In years of using the service I've never once been told that I'd missed an email by anyone I know. I'll take that as a sufficient confirmation that I haven't missed anything important. Things that I have missed which people haven't followed up on or notified me of are just about as good as junk mail anyway.
Gmail's spam detection is spectacular.
Are you new here? Its false positives are equally spectacular. Some days I could swear there's some mean bored Google sysadmin who happens to be a less-than-friendly coworker from my past who's just sitting there randomly applying the Spam label to messages in my Inbox.
At this point he can then run the mail through a series of weighted RBLs.
Fuck you and your RBLs. RBLs are a draconian solution that do immeasurable damage to those of us who (1) aren't spammers, and (2) choose to run our own mailservers on business-class IPs. I can't tell you how many times various IPs I use for outbound mail (I run several mailing lists) end up on an RBL for absolutely no fucking reason.
Oh, because someone in the same /24 block sent spam? Really? That's a good reason to block an entire /24 subnet?
RBLs are a solution in search of a problem. Some of them are nothing more than moneymakers for the people that run them: In order to get off their list, they blackmail you into paying money.
Want to do the world a favor? Don't use RBLs. You'll just end up finding yourself blacklisted at some point anyway.
depending on where your time is going, consider splitting the file up into pieces and run each piece in a different thread.
vice chair orange county java users group (ocjug.org).
A long time ago I benchmarked perl's regex engine against about 5 others. At the time, it was 10x faster than the nearest competitor for the same regex/data.
Also, you can use perl's "study". Or, split the regexes across threads.
Also, with perl you can do some hierarchical saviings. For example:
/Ffoo/ ...
/Fbar/ ...
/Fbaz/ ...
Could be redone as:
... if (/Ffoo/)
... if (/Fbar/
... if (/Fbaz/)
if (/F/) {
}
The above is trivial example, but you get the idea.
Also, how much time is spent compiling (vs. executing) the regexes in egrep? I imagine a lot and you have to do this for each incoming message.
Note that spamassassin (and hence perl) can be set up as a daemon where the regexes are compiled once. The messages are passed through a socket to the daemon. This means that the only CPU time spent is on executing the regexes--a considerable savings.
Additionally, perl regexes have [considerably] more functionality/utility than egrep ones. You might be able to recode/consolidate yours and get the same [or better] bang for less buck.
Like a good neighbor, fsck is there
Here's several things you can do to make this faster.
1) first don't keep invoking egrep. this has to parse the command line and then re-load the egrep command itself every time. Instead do this from within a loaded program. Perl is a very good choice for this
2) the perl command can pre-compile the regular expression. So you can leave the perl program running as a process then simply feed it new data to analyse.
3) given you are searching for words, you probably want to split the incoming stream on white space one-time not every time.
4) even better than that, take the e-mail, parse it to words, then parse each word into all 3,4,5,6,7,8 consecutive strings. Then just look these up in a hash table.
5) if you are only trying to match from the start of the word, (not interior word strings) then this hashing becomes trivial.
Some drink at the fountain of knowledge. Others just gargle.
Last night I sent my Gmail account an email from my ISP email system, then waited for it to show up. Nothing. So I resent it. Second time nothing.
The email contained two screen captures I needed at the office. The subject line was "Steve on telework". Nothing obvious that would trip Postini's spam filter. It is now 24 hours later and neither has shown up. I wonder how many other emails I don't get.
~~
Please stop roleplaying someone stupid with your current game of presenting the incorrect suggestion that the greylisting time set on the recieving server doesn't matter. You're misleading the newbies that haven't worked out from your handle and posting history that you like to pretend to be dumb as shit and offer utterly stupid suggestions as if they are viable. People playing games like yours make it difficult to have an honest and factual discussion in this place.
All your insulting bets are wrong, all that happened is some IP addresses got reassigned quite a few years ago. What used to be dynamic with one ISP became static with the company that bought them out.
I don't even use spam blockers. Instead I've purchased a domain, which is quite affordable nowadays. I have a catch-all redirect, so I any mail addressed to *@mydomain.com.
Then, I give a unique username to each organisation. e.g. slashdot@mydomain.com. If I receive spam at this address, I inform them, then kill the username. I can also just create slashdot2@mydomain.com if I want to keep dealing with their company.
Now, I receive only a few spam emails each year, so I need to do zero automated filtering. I also don't have to deal with the worry of false positives at all.
http://en.wikipedia.org/wiki/Greylisting
Fuck you and your RBLs. RBLs are a draconian solution that do immeasurable damage to those of us who (1) aren't spammers, and (2) choose to run our own mailservers on business-class IPs. [...] Oh, because someone in the same /24 block sent spam? Really? That's a good reason to block an entire /24 subnet?
That sucks. And more generally, I believe the anti-spam fundamentalists have caused as much damage to the use of reliable mail, as the actual spammers. They are much too willing to accept collateral damage. Sad, when you consider how much effort went into designing (a) guaranteed delivery or (b) guaranteed notification of non-delivery.
For this particular situation: I accepted early on that many misconfigured mail servers "work" like you describe, so I make sure that my ISPs lets me relay via them.
A project I worked on many years ago re-wrote a monitoring system in Java.
It was Perl, running a rather large list of regex's over syslog files.
The process of converting it to Java resulted in a 100x speed up - despite Perl possibly having a faster regex implementation. The regular expressions are compiled once on start-up. Regular expressions can be very fast - they're just slow to parse and compile.
How about instead of sitting there watching it process you just block your own access to viewing this 15 second delay and ignore it. Just don't care about it. Pretend it doesn't happen and your mail just arrived in your inbox.
I can see no situation where email being delays by 15 seconds is going to cause a an issue.
I send many millions of emails a week for mailing lists. I never get blacklistings 'for nothing'. What is annoying about blacklists is that they see so many spammers that they fail to see that normal people make mistakes (a lot). Since it is 'one strike and you are out' it is difficult to fix a situation for a legitimate situation.
Example: A medium size company mails its 2000 customers, this time in Finland, and we end up with a blacklisting. This customer comes up with a new list: The opt-outs and bounces (he is a sucker). However, we see 6% more permanent/hard-bounces than in the list. This can be because the spamfilters identified the spam and replied with hardbounces or it can be because the hardbounce list is bad (and there are more spamtraps). I already identified the list earlier as 'bad'. If no blacklist would have bothered, I would have allowed an opt-in mailing, asking recipients (again) for permission. Now I have to tell them to go to another ESP because I do not want to risk be a 'repeating offender' in their eyes. For the recipient, the first would be better. This is not a question you can ask a blacklist guy.
I guess the bottom line is, it is hard to communicate with blacklist people to solve legitimate situations. How can you fix that (without offering bad guys a vector for DOS-sing them)?
nosig today
There is a comparison of blacklists: http://dnsbl.inps.de/analyse.cgi?type=monthly&lang=en
nosig today
Lol.. Even if the particular grey listing you are using allows you to set the interval between reconnection attempts, you do not need to do so for grey listing to work. 90% or better spam email sent will never reconnect after being dropped once. The legitimate SMTPs will reconnect in a couple minutes if it is legit and the email will be received then subjected to other spam filtering processes if present on the servers. There is no pretending involved other then maybe your supposed experience with it.
http://projects.puremagic.com/greylisting/
You get so much mail so furiously that you can't suffer a 15 second delay? I presume you're talking about a personal mail server... if you're hosting mail for a 1000 people then yeah that's a problem.
Yes Francis, the world has gone crazy.
Just forward your mail through gmail. That way all the spam disappears and the NSA can get their data without trouble.
Excuse me, but please get off my Pennisetum Clandestinum, eh!
If you require using sophisticated procmail filters on your personal account then it seems like your setup is wrong from the get-go. Your incoming mail server should be taking the brunt of the work and using a progressive and efficient filtering before any filtering by content.
I use a spamdyke based front end that has a whole arsenal of whilte, black, and gray filtering of emails using RBLs RBLHS, reverse lookups, etc. It also can do header "pattern" filtering as well, but I currently don't use that feature. This blocks almost all spam quickly and efficiently. The last stage is to run it through spamassassin for those things that are in the gray (not a simple reject/accept, but a cumulative scoring) area. Worst case mail delays are on the order of few seconds through the whole chain. Spamassassin only gets a small number of incoming emails to work on. The stragglers usually come via accounts at yahoo, live, etc.
The nice thing about spamdyke and other systems like it is that it does it's job very fast. For example, the blacklists and whitelists in spamdyke can be setup as directory tree structure so it is a very quick lookup to determine whether to accept or reject the specified domain or ip address.
I also use systems like honeypots and hunter-seekers. The latter looks at what is graylisted or accepted by spamdyke and does http checks on the domain to see if it should be blacklisted. It also may decide to do tests in ip address neighbors to see if more should be blacklisted.
Like all systems, you must be proactive at identifying mail that shouldn't have been rejected. It is a rare situation, but there are a few companies with badly configured mail servers (like no reverse dns entries). However, after many years of operation my whitelist contains only a handful of domains. The automated blacklist process sends me email when it adds a domain, just in case.
Well, routing it outside of gmail keep gmails hands off it, well the half that doesn't originate with them to start with. But i think you'll need to do quite a bit more to keep the us govt out of it.
Of course you're right about that, especially if I correspond with someone with e.g. a gmail account. But, why make it any easier for the NSA? It gets a harder for them to listen in when I correspond with people who also maintain their own email servers, and whole a lot harder when those servers can also do automatic encryption. If what they're doing is unconstitutional and a violation of your privacy, why make life easy for them and knowingly play into their hands?
Bullshit.
Thank you for posting that checklist, that's a vital document for any spam planning.
SpamAssassin, executed through procmail on the mail client's email, is indeed resource intensive and does not scale well for an organization. Other people have mentioned other upstream filtering techniques, such as grey listing and DNS blacklists, but those are limited because of the large numbers of zombied Windows clients around the world, which have their resources rented as botnets to send spam from legitimate environments around the world, partly to evade these filters.
My experience is that spam requires management, not silver bullets. Layers of defense such as supporting SPF, which filters very early and cheaply based on DNS records, helps eliminate most forged gmail.com and hotmail.com and other large domain phishing. More powerful, more expensive filters such as SpamAssassin can be applied on the vastly reduced volume of email that gets past the earlier filters. Unfortunately, if you're processing with a local "procmail" by pulling the email from the mail server to your local machine, it's already too late to activate DNS blacklists or SPF, so the increasing burden on SpamAssassin is predictable.
I'm afraid I don't have a great solution for the original poster except tp push the filtering upstream, to the mail server itself, to reduce the load with those lightweight filters such as SPF or blacklists.
Please stop roleplaying someone stupid with your current game of presenting the incorrect suggestion that the greylisting time set on the recieving server doesn't matter.
Unless you are insanely stupid and use a greylist wait of a couple of hours, it really doesn't matter. Typical values for the wait are in the sub-5 minute range. Since it's resource intensive to retry at intervals smaller than about 5 minutes, good client configurations aren't going to even get to the first retry until after the greylist wait has expired. But, based on my own logs, you could set the wait to as little as 10 seconds and still have an extremely effective front line defense against spam.
Once an IP passes the wait successfully, e-mail from that address isn't ever delayed again (although I personally set a 40 day timeout on the whitelist to account for dynamic IP addresses). Add in the ability to pre-seed the list with known-to-retry systems (gmail, Yahoo, Amazon, etc.) and known common senders to your domain, and most people really will never notice any delay. And, if you run for about a month in "log only" mode, you'll get a great starting list of IPs you might need to whitelist. And, you need to think about server farms, so you might want to accept a retry from the same /24 subnet (which is what I do).
Also, if you were dumb enough to greylist mail that came from your internal network (which is the only way that a greylisting config would cause your boss to ask "why doesn't X have my email yet"), you don't understand greylisting enough to be in a discussion about it. For external senders (who should never be asking you that question, anyway), you can always explain the reason the e-mail is delayed: whoever configured their e-mail server didn't feel that a quick retry after a temporary failure was required.
So, this might be flamebait, but what it really comes down to is that your issues with greylisting were likely because you didn't do your homework and create a config that was as non-intrusive as possible.
RBLs block around 95% of my incoming mail with very minimal false positives. Sorry if you don't like them, but people use RBLs because they work.
I got a lot of them. Mostly emails from things like newspapers that I specifically asked them to send me.
Now that's an interesting reading comprehension failure and blaming victims instead of perpetrators. The problems with greylisting impact the sender more than those that have implemented greylisting. An effective setup requires someone who at least knows as much that could be picked up from a quick read of the wikipedia article instead of the misleading rubbish a few posts above. Quick retries don't save you from idiots that tune their settings to reduce spam but forget the entire point of email, it just means you have a lot more retries before they let you in.
No - I did my homework to find out exactly how somebody managed to fuck up communication and greatly delay messages from one end, and found that the answer was greylisting implemented very poorly at the remote end. My comment above is because I "did my homework" and observed the downsides. Those downsides are now listed in the wikipedia article.
For the record of yourself and the other idiot making noise about MS Exchange, I had not configured either of the two servers and instead came in after the problem came to light. So it's not just "flamebait" it's also a stupid jump to a conclusion just because I'm critical of yet another flawed anti-spam stopgap that can backfire if care is not taken. Spammers are channelling stuff via real mail servers now or getting their bots to resend so greylisting is losing what effectiveness it had anyway.
We see people complaining about this problem a lot, and yet for some reason they are afraid to actually put energy into a real solution. Repeat after me : filters can never end spam. That's right, never. All your filters (same can be said for every filter, everywhere) do is encourage the spammers to make their spam more obfuscated to improve their odds of passing future filters. It is a huge waste of time and resources and it's an arms race that the spammers will win.
If you want to actually end spam, you need to collaborate with other people who want to end spam. The way to end spam is not through technology but through economics; as their is only one reason why spam is sent - it is profitable. If you can interrupt the flow of money to the spammer they will move on to a different venture. Until then you're only spinning your wheels and wasting time, storage, and CPU cycles.
Damn_registrars has no butt-hole. Damn_registrars has no use for a butt-hole.
Fuck you and your RBLs.
Not every RBL is created equal. Anyone with half a brain knows which ones are good and which ones are so-so. The trick is to /weight/ them. Give a smaller score to the RBL you think makes mistakes.
Go take your blind hate elsewhere.
--
BMO
And allow Google and the US government to scan all of my mail?
Where have you been for the past (looks at calendar) 25 years?
The government archives every bit of email (and generic net traffic, for that matter) it can get its hands on. Remember Echelon? Remember Total Information Awareness? Snowden's "revelation" was something that everyone, who's paid attention for a few decades, already knew or assumed. The cost of a gigabyte is 3.5 cents retail these days. That's a lot of mail archiving for not a lot of money.
Unless you encrypt your mail locally before sending (and not rely on MTAs that use SSL) and your friends encrypt /their/ mail, then some computer along the way is going to read it.
It depends on how much you want to work at this. Most people don't care. There are people who do care, but don't know what to do, and this is a problem, because there are no turnkey/idiot-proof encrypted messaging systems out there that can be installed by Joe-User. It would be helpful if, by default, Thunderbird came bundled with gpg, for example, but configuring gpg isn't Joe-User proof.
Maintaining your own MTA is not enough. It has to happen client-side, before the mail even hits the MTA.
--
BMO
To give you an idea of how long I've been doing this shit, I remember the line eater.
I apologize for the personal attack. Not sure why I did that. Guess I let my emotions get the better of me.
Still, fuck RBLs. Sadly, many who should know better do not weight RBLs, and instead outright reject any mail that scores a hit. These operators are slowly destroying the email infrastructure by not only fragmenting and marginalizing the smaller email providers (including individuals who choose to responsibly run their own SMTP service), but by implicitly forcing individuals to seek mail services through corporate providers (think "do no evil"). I have gotten to the point where I simply tell subscribers to the lists I admin that they will have to use another ISP if they want to subscribe because their email provider blindly defers to one or more RBLs, most of which are dodgy to begin with (think pay to play, or let's ban entire subnets because we aren't technologically adept enough to filter on just one IP address).
Then the problem isn't with the RBLs, but bad admins that never even skimmed news.admin.net-abuse.email. And yes, there are more bad admins and RBLs out there than you can shake a stick at.
But then there are excellent services out there like Spamhaus.
--
BMO - Lumber Cartel #2501
To follow up, I have to say something -
Even with greylisting on an account that I've had since last century, it gets a few dozen spams. Without greylisting, it got hundreds a day - literally hundreds.
And technically, greylisting "breaks" email. If you have a non-compliant server that doesn't re-send 15 minutes later (as is the default in Sendmail, I believe), whoever you're trying to reach is never going to see your mail.
But without filtering, email is utterly useless. There used to be the idea that advertising through email might be useful, but that well has been poisoned and dead goats thrown in it, nearly poisoning the entire water table.
--
BMO
Note first, I am _not_ saying to replace your call to grep with a call to perl. Perl _is_ fast on assembling strings into a great matching system, but it still takes a _very_ long time to parse, say, 65000 separate strings.
So combine them all into one. Use Regexp::Assemble. With a little bit of fidgetting, it works with GNU grep, as well. Here's an example script, that I've named regex-opt:
!BEGIN regex-opt.pl!
#!/usr/bin/perl
use strict;
use Regexp::Assemble;
my $gnu = 0;
if ((defined $ARGV[0]) && $ARGV[0] eq '-gnu') {
shift;
$gnu = 1;
}
my $ra = Regexp::Assemble->new;
while () {
$ra->add($_);
}
my $string = $ra->as_string();
if ($gnu) {
$string =~ s/\\d/[0-9]/g;
$string =~ s/\(\?:/\(/g;
$string =~ s/([()?|]{})/\\$1/g;
}
print $string;
!END!
So, you have a file with your tens of thousands of lines of patterns to match. Ok, ./regex-opt < patterns.txt > matchpattern.re. This may work with egrep, but it's perl regex syntax, so maybe not completely -- procmail | egrep -f matchpattern.re
With 65000 lines, GNU grep takes about half an hour for the tasks I give it. After assembling all 65000 lines into one expression, even when that expression is _megabytes_ in size, it loads quickly and has the speed of a decision tree.
So, as you accumulate new patterns, output them to a file. Also, _always_ keep your list of separate match patterns -- I'm not sure how well this package can handle reparsing a regex back into itself. Do matches like so:
egrep -f <(cat matchpattern.re newpatterns.txt)
and once a week,
cat allpatterns.txt newpatterns.txt | regex-opt > matchpattern.re; sort -u allpatterns.txt newpatterns.txt > temp.txt && mv temp.txt allpatterns.txt && rm newpatterns.txt
> SpamAssassin, executed through procmail on the mail client's
> email, is indeed resource intensive and does not scale well for
> an organization.
it does scale much better if run through amavis as a persistent process, rather than forked from procmail for each incoming message - much of the CPU usage is from compiling (and re-compiling) the regular expressions over and over again.
pre-processing your regexp lists to consolidate them into far fewer but much longer regexps also gives huge benefits - e.g. instead of 1000 RE rules of 1 line each, join them with '|' and reduce them to 10 or 50. it's far less computational work to match against 50 long and slightly complicated REs than against 1000 simple REs.
in practice, this means generating your spamassasin local.cf file with a script, from one or more "source" files.
even without amavis, SA comes with spamd which provides the same benefit of avoiding RE-recompile - but IMO is a lot more work to configure and maintain than using amavis
even so, i try to reject as much spam as possible in the MTA before the mail gets passed to amavis & SA for final checking.
> My experience is that spam requires management,
> not silver bullets. Layers of defense [...]
yes! SPF, greylisting (even a 5 or 10 second greylisting delay is enough to filter out a huge amount of spam), careful use of RBLs (spamhaus are ethical and have reasonable policies), RHSBLs, DULs, MX-record checks (e.g. reject mail if MX record points to 127.0.0.1), HELO/EHLO checks (block mail claiming to be from my domains or IP address), blocking mail from specific senders and sender domains, tarpitting spammers, and more.
another useful technique is to use well-crafted fail2ban rules to monitor /var/log/mail.log and create temporary iptables rules to block persistent spam sources.
on my home mail server, i also block all mail from specific countries, using IP address and TLD blocking lists - but that's not a good option when spam-filtering mail for a company or organisation.
That seems a very sophisticated, enlightened, multi-layered approach. It can be very difficult to implement so broadly if your mail services are in the hands of another corporate group. MS Exchange managers, for example, can become quite concerned and upset if you want to implement greylisting and SPF blacklists before it even reaches their mail servers, but that's where it's most effective.
Merging the SpamAssassin checks into larger but more efficient regexp statements is a useful technique that I'd encourage you to publish, especially if you publish the tools to build those new rules and move aside the old ones.
@ 1000 emails/day, that's 15000 seconds processing time, or over 4 hours. That seems a bit excessive, but being having your email delayed a cumulative 4+ hours/day.
i thought merging REs was standard practice by now. i've been doing it since long before I started using SpamAssassin, when I was still mostly using postfix body_checks and header_checks.
here's some of my anti-spam stuff.
the scripts are old, but pretty close to what i actually still use today to generate postfix body/header checks and spamassassin rules.
they're not packaged software you can just install and use - think of them as examples of a particular approach to managing anti-spam rulesets.
BTW, note that with SpamAssassin, fewer and larger rules require less CPU time to run, but reduce the likelihood of multiple matches if there are multiple spammy phrases in an email - max one match per rule. this is why the scripts are configured to generate max of 500-character rule lines, when SA can easily handle 5000 or more characters per line. also, shorter lines are easier to read when debugging problems, and each rule is generated with a unique identifier so I can see which rules are matching for each msg
Use CRIU (Checkpoint Restore in Userspace) to checkpoint a hot version of grep that has been started and given a couple seconds to load in the dictionary and build it's pattern matcher and is thus just awaiting stdin (which you haven't given it). Restore a fresh instance for every new email, and pass the new email into the just-opened stdin for that restored, hot, waiting to go instance.
Instead of launching a fresh grep and initializing it with your corpus, this will create a grep that you can online which will be ready to go, awaiting input.
Ma-fucking-gic.
Traditionally one could achieve this effect by forking child workers, but that's a fucking huge pain in the ass as far as program design goes, making things really complicated- instead of a single program doing a single thing, it couples many uses of a program into a single programs lifestyle. Daemonized apps require system level management and have to be running. Service apps require complex interfaces to handle the different servicings they are performing. Decouple concerns (stay unix'y: stdin->program->stdout), and CRIU the bitch. Just use a hot program, rather than a cold one.
If the problem persists: fuck grep, it's pattern matching is rubbish and it's worthless. Please let us know. You might also consider 'head' 'ing the first 64k or some such of your email to avoid pattern matching the entire doc.
It's possible to use a hottened egrep by booting up one egrep, checkpointing it, then restoring that checkpoint again and again whenever you need an instance.
http://ask.slashdot.org/comments.pl?sid=4150171&cid=44759217
The problem is not using egrep, the problem is not using an existing already launched copy of egrep. Which, you CAN do. And I'd even recommend doing so, because it's manageable and uses sane well known and unfancy tools that are decoupled from each other.
Thanks for writing GIT. So many in this thread immediately jump into alternative options without discussing what's really at the heart of this problem. Grep is fine software and is known to do it's job well. As you say, the problem is simply that grep has startup costs, but those can be near totally ameliorated out.