I've seen that for a while now, beyond news. Google requests (not only popular) feeds every 15 minutes, often several fetches per second come from the same IP (probably another instance).
It seems that Ms. Googlebot now actively collects feed URIs within her regular crawling, harvests feeds from personalized home pages etc. Once a feed is known, it gets fetched way too often.
Although Google has implemented pinging (sitemap resubmission), it does not make use of it for feeds. http://feeds.google.com/ping?feedURI is still wishful thinking.
Hopefully Google is working on a submission based solution, frequent spidering of feeds based on guessing or time schedules is pretty much inefficient on the long haul.
The number of 8 billion searchable pages on Google's home page wasn't touched for a long time. Usually they do an update when another engine claims to have a bigger index. Also, this number does not include images etc., Yahoo's number does. I agree that Google's sitemap helpers will dig out a lot of stuff from the hidden Web. Most probably Google's index contains way more than 8 billion pages, perhaps even more than 20 billion objects.
Slashdot Mirror
I've seen that for a while now, beyond news. Google requests (not only popular) feeds every 15 minutes, often several fetches per second come from the same IP (probably another instance). It seems that Ms. Googlebot now actively collects feed URIs within her regular crawling, harvests feeds from personalized home pages etc. Once a feed is known, it gets fetched way too often. Although Google has implemented pinging (sitemap resubmission), it does not make use of it for feeds. http://feeds.google.com/ping?feedURI is still wishful thinking. Hopefully Google is working on a submission based solution, frequent spidering of feeds based on guessing or time schedules is pretty much inefficient on the long haul.
The number of 8 billion searchable pages on Google's home page wasn't touched for a long time. Usually they do an update when another engine claims to have a bigger index. Also, this number does not include images etc., Yahoo's number does. I agree that Google's sitemap helpers will dig out a lot of stuff from the hidden Web. Most probably Google's index contains way more than 8 billion pages, perhaps even more than 20 billion objects.