Who Isn't Paying Attention to ROBOTS.TXT?

Kickstart asks: "After wading through the Apache logs, after being hit hard for three hours by a very unfriendly spider, I see that there appear to be real, legitimate, search engines that do not follow robots.txt rules. Looking around, I see that some specialized search engines make no mention of their policy on this or say what servers their spiders come from. Does anyone have information on who follow this standard and who doesn't?"

Comments

Spammers are bad (of course)

[grub]

Does anyone have information on who follow this standard and who doesn't?

Most crawlers will obey. Spambot email harvesters will usually not. Generate a huge page of crap with loads of fake email addresses and put that in your robots.txt as uncrawlable and watch the spammers grab it.

Re:zerg

[Eric Giguere]

Start returning 500 errors... Or 302s that redirect them back to themselves...

Eric
PS: Is there some kind of bot storm going on, I'm getting all kinds of weird accesses to my site today, they're all fetching just the home page and leaving, and the referrer tag is null for everyone... They may be committing click fraud through my site, which makes me mad...

Here is your problem:

[Neil Blender]

All spiders are going to ignore your ROBOTS.TXT file. Instead, they look for a file called robots.txt.

Big name != "real"

[droleary]

I see that there appear to be real, legitimate, search engines that do not follow robots.txt rules.

No, you rather see some well-known search engines that generate illegitimate traffic instead of behaving properly. I note a number of them in this highly-documented robots.txt file. I'm personally most offended by idiots running this shit, since there is no single IP block to blacklist.

Re:Hey I've got an idea

[jbplou]

well you got a poor app if a spider can run right through it without authenicating and inserting/updating/deleting your data.

whitehouse.gov/robots.txt

[CommandoB]

The whitehouse seems to take a "pre-emptive" approach. Just in case they ever put stuff on the internet that they might someday not want you to see (or that they might not want archived by google), they seem to cover all the bases in their 92KB robots.txt file.

My personal favorites:
Disallow: /911/iraq
Disallow: /911/patriotism/iraq
Disallow: /911/patriotism2/iraq
Disallow: /911/sept112002/iraq [sic.]

There's a theme here. Can you spot it? I'd like to think it's intentional, but at 2255 lines, it may just be that all permutations of Republican buzzwords have been covered.