Slashdot Mirror

have you ever seen a new york times article appear in google?

Yes, because the Googlebot is a special exception on the NYTime's website (and a handful of other news-related subscription websites) and can go through anytime.

Drop the Googlebot's User Agent String [ Googlebot/2.1 (+http://www.googlebot.com/bot.html) ] into your browser if you're using Opera, or Firefox with Chris Pederick's User Agent Switcher extension, Safari and Konqueror probably provide User Agent faking as well. Then try to reach a "subscription only" NYTimes page, switch your UAS to Googlebot's, boom you're allowed in.

I've seen googlebot two hours ago with: "Googlebot/2.1 (+http://www.googlebot.com/bot.html)", this came from 66.249.71.69, which actually belongs to google. At one time, I've seen two googlebots spider our sites, one with the googlebot.com URL, the other with google.com, and otherwise identical UA-strings. Does anyone around here know what this difference is about?

why do that?

if you have a user agent spoofer, spoof this user agent:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

if you use user agent switcher for firefox, here are my settings:

Description: Googlebot
User Agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html)
App Name: Googlebot
App Version: 2.1 (+http://www.googlebot.com/bot.html)
Platform: Linux
Vender: Google

why do that?

if you have a user agent spoofer, spoof this user agent:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

if you use user agent switcher for firefox, here are my settings:

Description: Googlebot
User Agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html)
App Name: Googlebot
App Version: 2.1 (+http://www.googlebot.com/bot.html)
Platform: Linux
Vender: Google

why do that?

if you have a user agent spoofer, spoof this user agent:

Googlebot/2.1 (+http://www.googlebot.com/bot.html)

if you use user agent switcher for firefox, here are my settings:

Description: Googlebot
User Agent: Googlebot/2.1 (+http://www.googlebot.com/bot.html)
App Name: Googlebot
App Version: 2.1 (+http://www.googlebot.com/bot.html)
Platform: Linux
Vender: Google

If you have Firefox (duh) get the User Agent Switcher extension.

Add a google useragent string like "Googlebot/2.1 (+http://www.googlebot.com/bot.html)" to it, switch to the google ua and voila, you're in.

This also works for a surprisingly large number of registration only sites as well.

No.

The breakdown of the top 15 is:

1 82.63% Mozilla
2 14.70% Microsoft Internet Explorer
3 0.46% Opera/8.0 (Windows NT 5.1; U; en)
4 0.25% msnbot/1.0 (+http://search.msn.com/msnbot.htm)
5 0.25% Googlebot/2.1 (+http://www.google.com/bot.html)
6 0.21% Mediapartners-Google/2.1
7 0.18% Microsoft URL Control - 6.01.9782
8 0.16% Opera/8.0 (X11; Linux i686; U; en)
9 0.10% Opera/8.0 (Windows NT 5.0; U; en)
10 0.07% Opera/7.54 (Windows NT 5.1; U) [en]
11 0.07% Opera/7.54 (X11; Linux i686; U) [en]
12 0.02% Avant Browser (http://www.avantbrowser.com/
13 0.02% Opera/7.20 (Windows NT 5.0; U) [en]
14 0.02% Googlebot/2.1 (+http://www.googlebot.com/bot.html)
15 0.02% Links (2.1pre17; Linux 2.6.10-gentoo-r1 i686; x)

Informative, thanks!

I find this in my log report for Slurp, is that the one for Yahoo?:

2889: Mozilla/5.0 (Slurp/cat; slurp@inktomi.com; http://www.inktomi.com/slurp.html)

Still, far less than the other guys:

34267: msnbot/0.3 (+http://search.msn.com/msnbot.htm)

28927: Googlebot/2.1 (+http://www.googlebot.com/bot.html)

--