Stopping SpamBots With Apache Part II

primetyme writes: "To address some of the concerns brought up in the first article about stopping email harvesting spambots with Apache, I've written a follow-up article that details even more methods to keep email-sucking bots off your Apache based site.
Stopping Spambots II - The Admin Strikes Back continues the epic saga that pits Spambot vs. Administrator."

Comments

Re:Restarting the server?

[epsalon]

A simple improvement will be to send SIGHUP to the webserver to make it reload the config without restarting. This still can be used for DoS, but less efficiently.
A better way to do it is by writing (using?) an Apache module that does the logging in memory with no costy reloads or restarts.
However, this still does not prevent the proxy and dialup problems illustrated above. Also, you won't catch spambots that don't use robots.txt to find addresses.
Another improvemnt will be to deny addresses the moment they ask for robots.txt while identified as "Mozilla" user-agent, and to detect clients that do a websuck without requesting robots.txt first and deny them as well. You can detect a websuck by posting a "hidden" link in a place normal users won't see and stop any IP that requests it.