← Back to Users
Bloggerheads's activity in the archive.
All implications and conspiracy theories aside... The new standard for robots.txt files (in place for about 2+ years now, IIRC) allows you to disallow the lot and then specify exceptions. Example: http://www.tesco.com/robots.txt Benefit? You don't end up giving people a handy shopping list of places you don't want them poking around. So what strikes me most about this little discovery is that whoever wrote this robots.txt file is a complete doofus. Oh, and Oceania is at war with Eastasia. Oceania has always been at war with Eastasia. Tim http://www.bloggerheads.com/
Slashdot Mirror
All implications and conspiracy theories aside...
The new standard for robots.txt files (in place for about 2+ years now, IIRC) allows you to disallow the lot and then specify exceptions.
Example:
http://www.tesco.com/robots.txt
Benefit? You don't end up giving people a handy shopping list of places you don't want them poking around.
So what strikes me most about this little discovery is that whoever wrote this robots.txt file is a complete doofus.
Oh, and Oceania is at war with Eastasia. Oceania has always been at war with Eastasia.
Tim
http://www.bloggerheads.com/