Just a note on your comment about "it only crawled each site on average every 4 seconds" They actuall address this: "To eliminate any possibility that a particular site will be hit more then once every two seconds, our crawler records the most recent two minutes of IP addresses and calculates the frequency of a particular IP within that two minute time frame. If the IP has been hit above a programmable threshold set at two seconds, the page is sent back to the cache and not crawled until the frequency is greater then two seconds." Last time I checked, I can surf faster then that.
Slashdot Mirror
Just a note on your comment about "it only crawled each site on average every 4 seconds" They actuall address this: "To eliminate any possibility that a particular site will be hit more then once every two seconds, our crawler records the most recent two minutes of IP addresses and calculates the frequency of a particular IP within that two minute time frame. If the IP has been hit above a programmable threshold set at two seconds, the page is sent back to the cache and not crawled until the frequency is greater then two seconds." Last time I checked, I can surf faster then that.