White House Website Limits Iraq-Related Crawling

oscarcar writes "Dan Gillmor is reporting on the White House website's use of its robots.txt file to disable search engines from crawling certain material. Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."

Comments

Funny

[sulli]

whitehouse.com doesn't have that problem.

Re:Funny

[sulli]

Also, TRUE God loving Americans would absoltely love to see that filthy, degrading website taken down because of the damage it causes to children who go there on accident.

I feel the same way about whitehouse.gov. Couldn't have said it better myself.

Drawing farfetched conclusions

[Armethius]

possibly to prevent people from finding changes to past statements and information when archived elsewhere

Or possibly not...

upside

[514x0r]

it's good to see the whitehouse embracing technology so much.

Interesting line

[Lizard_King]

Disallow: /president/spongebobsquarepants_archive

I didn't know gee-dub likes SpongeBob too! My nephew is gonna flip out when he hears this.

Truly Frightening.

[Dlugar]

Obviously, they're keeping people from accessing the top-secret teeball Iraq files ! Besides:

Disallow: /teeball/iraq/

check out these other frightening examples of censorship:

Disallow: /kids/spotty/iraq
Disallow: /kids/eggroll/iraq
Disallow: /kids/barney/iraq
Disallow: /easter/iraq
Disallow: /mrscheney/iraq
Disallow: /national-anthem/iraq

Truly frightening.

I, for one...

[wardomon]

welcome our White House Robot Overlords. It would be funnier if it weren't true.

Re:Other, arguably more reasonable explanations

[RobertB-DC]

Maybe, but I would think they might also be looking for "shady" spiders that ignored robots.txt. I wouldn't be surprised if there aren't a few honeypot pages in there too.

Oh, crap. I just plugged in /firstlady/images/iraq, and now you tell me I'd better watch out. Damn this static IP address!

Quick, Slashdot that link before the Agents get to my cube!

Not just Bush

[dekashizl]

Before you get carried away with the Iraq issue, realize that most of these are just leftover from previous administrations:

Disallow: /~billc/pics/nudity/hillary
Disallow: /~billc/mysexpics/oral/monica
Disallow: /~billc/mysexpics/mf
Disallow: /~billc/mysexpics/mff
Disallow: /~billc/mysexpics/mfff
Disallow: /~billc/mysexpics/mm
Disallow: /~billc/mysexpics/gorebjs
Disallow: /~billc/mysexpics/goats

Barney, agent provacateur of the CIA? You Decide

[mykepredko]

Downloading the "robot.txt" file and doing a quick ctrl-f on different words, I discovered that there are six instances of "Barney" coming up in the robot.txt:

Disallow: /holiday/2002/barney/iraq
Disallow: /holiday/2002/barney/text
Disallow: /kids/barney/iraq
Disallow: /kids/barney/text
Disallow: /kids/photoessays/barney/iraq
Disallow: /kids/photoessays/barney/text

Which is the same number as "cheney", "powell" had 4, "saddam" didn't have any and "bush" only comes up with "bushpets".

Clearly, there is something to do with Barney and Iraq that The White House doesn't want you to know about.

myke

1984: simple answer

[flossie]

There is a very simple explanation for this, as anyone who has read 1984 should know. In order for the glorious government to effectively serve the greater good, they need to be able to communicate changes of policy quickly and effectively. If, for instance, the enemy in a war changes, it is necessary to quickly update all documents that describe how evil the enemy are. Rather than manually editing all the documents, it is much easier to have one generic word, say "text", which can then be altered as appropriate:

sed 's/text/iraq/g' sed 's/text/iran/g' sed 's/text/cuba/g' sed 's/text/belgium/g'

etc.

Obviously robots.txt just happened to be in the path!

Stupidity riegns supreme

[MoneyT]

So, am I to understand that the same administration that was smart enough to rig an election, Smart enough to cause 9/11, Smart enough to forge evidence and go to war is the same administration that came up with the brilliant plan of HIDING information by putting it in a PUBLICALY availible file?

Re:Other, arguably more reasonable explanations

[greenhide]

How can they be Iraq related if they didn't exsist to begin with?

A question that GW gets asked all the time. :-)