Slashdot Mirror

Yes, but if only there was some open standard protocol he could use to tell Google how often to index which parts of his site, which Google implemented...

I don't see a way to use robots.txt to limit the number of crawler hits per interval other than just denying it. So you can block it, but that's undesirable if you want people to find it. It's also undesirable to have a robot hit your site every two seconds if ShieldW0lf is saying the truth, but robots.txt only address it in a simplistic allow / disallow.

I'm not sure if any of the other providers implement this, but Google does. SiteMaps

Lets you specify how often to update certain content, what URLs to block. It's a more advanced robots.txt.

They're trying to sell the sitemap.xml idea, yet they themselves have no sitemap.xml file.

Call me an XML nazi but the example usage has unclosed tags:

http://www.sitemaps.org/protocol.html

Wow, I'm awesome. The broken link is my own 1337 skills. Let's try this again: http://www.sitemaps.org/protocol.html Mmm, closing tags.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9 ">
<url>
<loc>http://www.example.com/</loc>
<lastmod>2005-01-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</urlset>