Slashdot Mirror
Domain: robotstxt.org
Stories and comments across the archive that link to robotstxt.org.
Stories · 1
-
Who Isn't Paying Attention to ROBOTS.TXT?
Kickstart asks: "After wading through the Apache logs, after being hit hard for three hours by a very unfriendly spider, I see that there appear to be real, legitimate, search engines that do not follow robots.txt rules. Looking around, I see that some specialized search engines make no mention of their policy on this or say what servers their spiders come from. Does anyone have information on who follow this standard and who doesn't?"