Stopping SpamBots With Apache

primetyme writes: "Sick of email harvesting spam robots cruising your Apache based site? Here's an in depth article that shows one way you can configure a base Apache installation to keep those nasty bots of your site - and the spam out of your Inbox." Anything that helps annoy spammers is a good thing.

Comments

One Way...

[ekrout]

is to give your site a terrible color scheme, like purple & brown. ;-)

It won't work long

[anothernobody]

Checking the user agent won't work for long - how hard will it be for the spammers to change the user agent to "Mozilla..."

Using some client side Javascript would be harder for them to deal with (although if your browser can view it they will be able to also).

I guess graphics would be next...

Also useful for...

[dpete4552]

I have been using this method for a long time, I don't know how new that article is, but I used it a long time ago to not only block all the spambots I could find, but all of the software for mirroring my webpage as well.

Here is a longer list of common spam bots and mirror bots that I have been able to find:

SetEnvIfNoCase User-Agent "EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "EmailWolf" bad_bot
SetEnvIfNoCase User-Agent "CherryPickerSE" bad_bot
SetEnvIfNoCase User-Agent "CherryPickerElite" bad_bot
SetEnvIfNoCase User-Agent "Crescent" bad_bot
SetEnvIfNoCase User-Agent "EmailCollector" bad_bot
SetEnvIfNoCase User-Agent "EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "MCspider" bad_bot
SetEnvIfNoCase User-Agent "bew" bad_bot
SetEnvIfNoCase User-Agent "Deweb" bad_bot
SetEnvIfNoCase User-Agent "FEZhead" bad_bot
SetEnvIfNoCase User-Agent "Fetcher" bad_bot
SetEnvIfNoCase User-Agent "Getleft" bad_bot
SetEnvIfNoCase User-Agent "GetURL" bad_bot
SetEnvIfNoCase User-Agent "HTTrack" bad_bot
SetEnvIfNoCase User-Agent "IBM_Planetwide" bad_bot
SetEnvIfNoCase User-Agent "KWebGet" bad_bot
SetEnvIfNoCase User-Agent "Monster" bad_bot
SetEnvIfNoCase User-Agent "Mirror" bad_bot
SetEnvIfNoCase User-Agent "NetCarta" bad_bot
SetEnvIfNoCase User-Agent "OpaL" bad_bot
SetEnvIfNoCase User-Agent "PackRat" bad_bot
SetEnvIfNoCase User-Agent "pavuk" bad_bot
SetEnvIfNoCase User-Agent "PushSite" bad_bot
SetEnvIfNoCase User-Agent "Rsync" bad_bot
SetEnvIfNoCase User-Agent "Shai" bad_bot
SetEnvIfNoCase User-Agent "Spegla" bad_bot
SetEnvIfNoCase User-Agent "SpiderBot" bad_bot
SetEnvIfNoCase User-Agent "SuperBot" bad_bot
SetEnvIfNoCase User-Agent "tarspider" bad_bot
SetEnvIfNoCase User-Agent "Templeton" bad_bot
SetEnvIfNoCase User-Agent "WebCopy" bad_bot
SetEnvIfNoCase User-Agent "WebFetcher" bad_bot
SetEnvIfNoCase User-Agent "WebMiner" bad_bot
SetEnvIfNoCase User-Agent "webvac" bad_bot
SetEnvIfNoCase User-Agent "webwalk" bad_bot
SetEnvIfNoCase User-Agent "w3mir" bad_bot
SetEnvIfNoCase User-Agent "XGET" bad_bot
SetEnvIfNoCase User-Agent "Wget" bad_bot
SetEnvIfNoCase User-Agent "WebReaper" bad_bot
SetEnvIfNoCase User-Agent "WUMPUS" bad_bot
SetEnvIfNoCase User-Agent "FAST-WebCrawler" bad_bot

You can't win an arms race

[CmdrTroll]

The premise behind this article is patently ridiculous. Spambots are voluntarily identifying themselves, and any spambot author with an ounce of common sense will simply change their user-agent string to the standard "Mozilla 4.0 (Microsoft Internet Explorer 5.5)" string that every Windows client uses. A well-designed spambot is indistinguishable from a valid user, or Google, or ht://dig.

On the other hand, there are ways to fight spambots; they just don't rely on trusting the user. Here's one way:

• Buy a domain.
• Set up a cgi that generates a unique email address @ that domain for every visitor. Log the address used, the date/time of visit, the visitor's IP, and other characteristics (user-agent?) of the visitor.
• Use the logged data to block the user when spam mail gets sent to one of the random accounts.
• Use the logged data as evidence to present to the offender's ISP, to get their fast connection pulled.
• Find a way to automate this on a large scale, then get a bunch of sysadmins together to sue and prosecute the spammer for abuse of resources.

There are good ways to deal with spammers but this isn't one of them. It *might* work on a small scale and it definitely won't work on a medium or large scale. It's about as useful as the Sendmail "MX/domain validation" trick that Eric Raymond and the rest of the Sendmail team thought would stop spammers dead in its tracks. (It didn't.) Instead he was "surprised by spam."

-CT

Re:You can't win an arms race

[primetyme]

Thats pretty much what I do in the Hook, line, and sinker section of the article.. By capturing the user-agent's and IP's of the Spiderts that *blatently* disregard the robots.txt file, its like shootin fish in a barrel..

In the next installment of this article, I'm working on a script that grabs the NetBlock of a bot that goes against the robots.txt file, does a ARIN lookup on that block, and emails the administrator of that block with the prob.. Comments have been made that any bot can switch their user-agent string, which is true. If a Spidert does that though, they're more than likely also going to run through the parts of a site that you *specifically* tell them they can't go in the robots.txt file. When they do that, its a lot easier to block their user-agent, email the admin of thier netblock, or block their class c IP block alltogether.

It's like a honeypot for black-hats if you think about it.. And thats one of the *best* ways to find the problem Spiderts and block them out, without blocking any good natured bot :)

Wget is not a spider!

[Anonymous Coward]

"Here are a couple of the User-Agents that fell for our trap that I pulled out of last months access_log for lists.evolt.org:

Wget/1.6"

Email spider, my ass! Wget is a damn useful HTTP downloader utility which is great for obtaining large files as it can resume interrupted transfers. It can also mirror web sites, which I assume is why it fell into the honeypot. Oh, and you can also change what it says it is on the command line.

And to add my 2 cents to the email problems, one other solution I've seen is to translate email addresses into an image and drop that onto the page. It's not a fantastic solution for those still using Lynx, and you can no longer just click to send mail to somebody, but at least it doesn't go the Javascript route and should be a sufficient technical hurdle to stop automated harvesters for a couple of years at least.

- Anonymous and happy.

Re:Good for the goose

[Snootch]

One big difference - MSN discriminated against valid browsers that were just people trying to view their website. The user agent IDs here (with a coupla exceptions - *cough* wget *cough*) are all things that are only ever used for spam purposes. There is a difference between blocking people because they don't use your software and blocking spam robots.