Slashdot Mirror
Google Crawls The Deep Web
mikkl666 writes "In their official blog, Google announces that they are experimenting with technologies to index the Deep Web, i.e. the sites hidden behind forms, in order to be 'the gateway to large volumes of data beyond the normal scope of search engines'. For that purpose, the engine tries to automatically get past the forms: 'For text boxes, our computers automatically choose words from the site that has the form; for select menus, check boxes, and radio buttons on the form, we choose from among the values of the HTML'. Nevertheless, directions like 'nofollow' and 'noindex' are still respected, so sites can still be excluded from this type of search.'"
They'll have to be careful how they go about this. If they start filling in forms with bogus data on blogs, forums etc., there are going to be a lot of pissed off website owners out there. Just imagine the number of admins who'll have to update their robots.txt for this. Just my 2c.
Like these people: http://thedailywtf.com/Articles/The_Spider_of_Doom.aspx