Who Isn't Paying Attention to ROBOTS.TXT?

Kickstart asks: "After wading through the Apache logs, after being hit hard for three hours by a very unfriendly spider, I see that there appear to be real, legitimate, search engines that do not follow robots.txt rules. Looking around, I see that some specialized search engines make no mention of their policy on this or say what servers their spiders come from. Does anyone have information on who follow this standard and who doesn't?"

Comments

Re:zerg

[Intron]

zerg?

Big name != "real"

[droleary]

I see that there appear to be real, legitimate, search engines that do not follow robots.txt rules.

No, you rather see some well-known search engines that generate illegitimate traffic instead of behaving properly. I note a number of them in this highly-documented robots.txt file. I'm personally most offended by idiots running this shit, since there is no single IP block to blacklist.

Re:zerg

[BrynM]

From the WebPoison site:

"WebPoison.org is an open source project... (at the bottom of the page) *Technically speaking, webpoison.org is not "open source" because the source code may never be made public- doing so would undermine the project's central goal.

Sorry, but it rubs me wrong when a project claims to be OSS on the first line of their about page only to tell me they lied in the fine print at the bottom. They may be doing a good thing, but they should be blunt and honest about it.

Known Bad Bots

[stoborrobots]

Oh, yeah, and to actually answer the OPs question, there are lists of known bad bots out there...