geekfiend writes "Today Google updated their website to indicate over eight billion pages crawled, cached and indexed. They've also added an entry to their blog explaining that they still have tons of work to do."
Re:Google thieves my bandwidth
by
Anonymous Coward
·
· Score: 5, Informative
Google respects the robots.txt file. Use it.
Re:Google thieves my bandwidth
by
jvj24601
·
· Score: 5, Informative
Well, if you know that Google is indexing your site and "stealing" your bandwidth, then you must have looked at the server logs, right? You'd see the name of the search bot is googlebot. Search for it, and you'll find that the first relevant link explains how to prevent googlebot from accessing your site.
The logs would probably also show failed attempts to find the file/robots.txt. Similar info is gained from searching on that term as well.
Slashdot Mirror
Google respects the robots.txt file. Use it.
Well, if you know that Google is indexing your site and "stealing" your bandwidth, then you must have looked at the server logs, right? You'd see the name of the search bot is googlebot. Search for it, and you'll find that the first relevant link explains how to prevent googlebot from accessing your site.
/robots.txt. Similar info is gained from searching on that term as well.
The logs would probably also show failed attempts to find the file