White House Website Limits Iraq-Related Crawling

oscarcar writes "Dan Gillmor is reporting on the White House website's use of its robots.txt file to disable search engines from crawling certain material. Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."

Comments

Funny

[sulli]

whitehouse.com doesn't have that problem.

Re:Funny

[Anonymous Coward]

neither does gwbush.com

We have to give him credit for believing in the U.S. values enough not to shut the site down.

Re:Funny

[sulli]

Also, TRUE God loving Americans would absoltely love to see that filthy, degrading website taken down because of the damage it causes to children who go there on accident.

I feel the same way about whitehouse.gov. Couldn't have said it better myself.

Drawing farfetched conclusions

[Armethius]

possibly to prevent people from finding changes to past statements and information when archived elsewhere

Or possibly not...

upside

[514x0r]

it's good to see the whitehouse embracing technology so much.

Careful

[BamaSlam]

Or you'll tear his tinfoil hat and then the black helicopters will be able to find him again.

Nugs

Interesting line

[Lizard_King]

Disallow: /president/spongebobsquarepants_archive

I didn't know gee-dub likes SpongeBob too! My nephew is gonna flip out when he hears this.

Truly Frightening.

[Dlugar]

Obviously, they're keeping people from accessing the top-secret teeball Iraq files ! Besides:

Disallow: /teeball/iraq/

check out these other frightening examples of censorship:

Disallow: /kids/spotty/iraq
Disallow: /kids/eggroll/iraq
Disallow: /kids/barney/iraq
Disallow: /easter/iraq
Disallow: /mrscheney/iraq
Disallow: /national-anthem/iraq

Truly frightening.

Re:Truly Frightening.

[smackjer]

Sssshhh! The terrorists will see this!

I, for one...

[wardomon]

welcome our White House Robot Overlords. It would be funnier if it weren't true.

Re:Oh please

[DrEldarion]

If you think that's the most paranoid comment, you must not read any RFID tag threads here yet.

Re:Other, arguably more reasonable explanations

[RobertB-DC]

Maybe, but I would think they might also be looking for "shady" spiders that ignored robots.txt. I wouldn't be surprised if there aren't a few honeypot pages in there too.

Oh, crap. I just plugged in /firstlady/images/iraq, and now you tell me I'd better watch out. Damn this static IP address!

Quick, Slashdot that link before the Agents get to my cube!

Didn't work.

[Quasar1999]

You found it didn't you? It failed... congratulations, you have somehow circumvented the government's website security system, prepare for the wrath of the DMCA, backed by none other than Bush himself!

Well either that, or it's simply preventing search engines from indexing honeypot type pages used for mis-information... Either or... but I like the first version... since it's more paranoid, and I have plenty of tinfoil ready to be shaped into hats... ;)

SHHH!!!

[jpmahala]

Goodness knows we can't have googlebots archiving all of those top-secret/confidential web pages at the whitehouse. I guess we'll just have to live with the top-secret info that has already been archived.

What's that? Oh, all of the real top-secret stuff is at the NSA website?

Never mind then.

New entries added ...

[obsidianpreacher]

As of just a few minutes ago, these entries were seen added to the robots.txt file:

Disallow: /news/slashdot
Disallow: /news/tinfoilhat
Disallow: /allyouriraq/are/belongto/US ...

Come on. This is extremely paranoid and far-fetched, even for /. ... if they're so worried about people finding out their insidious plots, they'll just flip the switch on all their mind-controlling ...

MUST DESTROY SLASHDOT ... MUST DESTROY WEBLOGS ... TRUTH GETTING OUT ... DUBYA IS MY FRIEND ... MUST DESTROY SLASHDOT ... MUST DESTROY WEBLOGS ...

Not just Bush

[dekashizl]

Before you get carried away with the Iraq issue, realize that most of these are just leftover from previous administrations:

Disallow: /~billc/pics/nudity/hillary
Disallow: /~billc/mysexpics/oral/monica
Disallow: /~billc/mysexpics/mf
Disallow: /~billc/mysexpics/mff
Disallow: /~billc/mysexpics/mfff
Disallow: /~billc/mysexpics/mm
Disallow: /~billc/mysexpics/gorebjs
Disallow: /~billc/mysexpics/goats

Re:Take a look for yourself

[Anixamander]

Looks like someone just added IRAQ to all of the exsiting links. It's obviously some sort of search/replace/copy function. Go look for yourself, I found this one:

Disallow: /firstlady/recipes/iraq



Soylent Green is Iraqi people!

WMD's found!

[missing000]

I knew it!

Barney, agent provacateur of the CIA? You Decide

[mykepredko]

Downloading the "robot.txt" file and doing a quick ctrl-f on different words, I discovered that there are six instances of "Barney" coming up in the robot.txt:

Disallow: /holiday/2002/barney/iraq
Disallow: /holiday/2002/barney/text
Disallow: /kids/barney/iraq
Disallow: /kids/barney/text
Disallow: /kids/photoessays/barney/iraq
Disallow: /kids/photoessays/barney/text

Which is the same number as "cheney", "powell" had 4, "saddam" didn't have any and "bush" only comes up with "bushpets".

Clearly, there is something to do with Barney and Iraq that The White House doesn't want you to know about.

myke

1984: simple answer

[flossie]

There is a very simple explanation for this, as anyone who has read 1984 should know. In order for the glorious government to effectively serve the greater good, they need to be able to communicate changes of policy quickly and effectively. If, for instance, the enemy in a war changes, it is necessary to quickly update all documents that describe how evil the enemy are. Rather than manually editing all the documents, it is much easier to have one generic word, say "text", which can then be altered as appropriate:

sed 's/text/iraq/g' sed 's/text/iran/g' sed 's/text/cuba/g' sed 's/text/belgium/g'

etc.

Obviously robots.txt just happened to be in the path!

Stupidity riegns supreme

[MoneyT]

So, am I to understand that the same administration that was smart enough to rig an election, Smart enough to cause 9/11, Smart enough to forge evidence and go to war is the same administration that came up with the brilliant plan of HIDING information by putting it in a PUBLICALY availible file?

Not a suprise

[VivianC]

This really shouldn't shock anyone. It has been going on at the White House for ages. Look at this clip from the robots.txt file from 1998:
Disallow: /history/photoessays/blueroom/blowjobs
Disallow: /history/photoessays/blueroom/text
Disallow: /history/photoessays/cabinetroom/blowjobs
Disallo w: /history/photoessays/cabinetroom/text
Disallow: /history/photoessays/crosshalls/blowjobs
Disallow : /history/photoessays/crosshalls/temp/blowjobs
Dis allow: /history/photoessays/crosshalls/temp/text
Disallo w: /history/photoessays/crosshalls/text
Disallow: /history/photoessays/diplomaticroom/blowjobs
Disa llow: /history/photoessays/diplomaticroom/text
Disallow : /history/photoessays/downstairscorridor/blowjobs
Disallow: /history/photoessays/downstairscorridor/text
Disa llow: /history/photoessays/easter/2002/blowjobs
Disallo w: /history/photoessays/easter/2002/text
Disallow: /history/photoessays/easter/2003/defenselink/blowj obs
Disallow: /history/photoessays/easter/2003/defenselink/text
Disallow: /history/photoessays/easter/2003/blowjobs
Disallo w: /history/photoessays/easter/2003/text
Disallow: /history/photoessays/easter/one/blowjobs
Disallow : /history/photoessays/easter/one/text
Disallow: /history/photoessays/easter/three/blowjobs

Re:More American Cencorship

[cquark]

Not a single sole in the United States could give a flying rats ass about what some Kiwi newspaper has to say about 9/11

It's not just the sole. Most salmon in the US don't pay attention to newspapers in NZ either. The kind of attitudes today's schools are producing are just shameful.

Re:Other, arguably more reasonable explanations

[greenhide]

How can they be Iraq related if they didn't exsist to begin with?

A question that GW gets asked all the time. :-)

Re:If they want to crawl them...

[Le Marteau]

...they will. Who pays attention to robots.txt anyway?

Umm... it's on the tip of my tongue... what was that name... begins with a 'G'... oh, Google, that's right, Google. Supposed to be some kind of search engine or something.