White House Website Limits Iraq-Related Crawling

oscarcar writes "Dan Gillmor is reporting on the White House website's use of its robots.txt file to disable search engines from crawling certain material. Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."

Comments

Other, arguably more reasonable explanations

[rot26]

Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."

Maybe, but I would think they might also be looking for "shady" spiders that ignored robots.txt. I wouldn't be surprised if there aren't a few honeypot pages in there too.

Re:Other, arguably more reasonable explanations

[sketerpot]

Honeypot or not, look at robots.txt. It's creepy: just about every entry is an Iraq-related page, and there are a lot of entries. If they wanted to just have a few honeypots, that shouldn't involve that many entries, or so many with the common theme of Iraq.

Re:Oh please

[cgranade]

This gets modded up as Insightful? I mean, the White House is routinely editing their trascripts, and if bots like Google and Wayback can go and find that no, Bush said that we found weapons, not a weapons program, then there goes Bush's latest FUD... *thud*. Just because it's a tinfoil hat worthy theory doesn't mean it isn't true... most aren't, but therein lies the issue: most.

related links

[js7a]

A couple of web sites that (1) have in the past done a great job of catching these kind of things, and (2) have mailing lists you can subscribe to:

• The Memory Hole

• Fairness and Accuracy in Reporting

Here's a minor example of something those two sites didn't catch: Remember Iraq's so-called "mobile biological weapons factories"? A month after the story broke that they were for weather balloons, the CIA moved their report's URL.

An intriguing fact about this whitehouse.gov/*/iraq thing is that they do in fact cover some of the important statements which are apparently not duplicated in the press release, conference, and briefing directories. Perhaps there was a "unique urgency" to cover up some poor choices of words?

Re:Funny

[jovlinger]

true. true. Apparently some poor fool made similar remarks on k5 a while back, and did indeed receive a personal visit from the SS. No charges filed, but 'tis a rude awakening indeed when your online words come and knock on your door.

Re: and your ...

[Zeinfeld]

every major journalist worth his/her salt would be all over it within hours. so it wouldn't succeed in obscuring information.

Where have you been living the past five years? Journalists don't criticize Bush.

They still have not published the fact that he deserted from the national guard during Vietnam and they practically ignored his DUI conviction.

The GOP has the media cowed with their constant 'liberal media' babble. There number of journalists who are prepared to hold Bush to account is tiny - Krugman, Conanston, Irvins, Alterman. After that its Al Franken, Jon Stewart and David Letterman.

it would create an incredible backlash as soon as detected. what purpose would this serve?

The chances that the mainstream media will pick this one up are very small. Just think how they would have reacted if it was Clinton!

Re: and your ...

[Zeinfeld]

The crux of this argument is that Bush missed some drills in 1972 while he was working on a political campaign in Alabama.

The crux of the matter is that he refused to have his pilots medical just after the Pentagon added a check for illegal drug use.

You can try to spin this whichever way that Karl Rove tells you but the facts are against you. The fact is that your great leader is a coward who ducked the draft and then deserted to avoid a drug test.