Slashdot Mirror
White House Website Limits Iraq-Related Crawling
oscarcar writes "Dan Gillmor is reporting on the White House website's use of its robots.txt file to disable search engines from crawling certain material. Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."
whitehouse.com doesn't have that problem.
sulli
RTFJ.
it's good to see the whitehouse embracing technology so much.
!(^((ri)|(mp))aa$)
Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."
Maybe, but I would think they might also be looking for "shady" spiders that ignored robots.txt. I wouldn't be surprised if there aren't a few honeypot pages in there too.
To ensure perfect aim, shoot first and call whatever you hit the target
Queue somebody to take a crawler (hell, even a bash script using wget) to specifically archive these pages. Hell, they could even use a user-agent which doesn't look like a bot.
Of course, people would be less likely to trust random-Joe from the Internet than, say, The Wayback Machine, but I expect this is what will happen...
It looks like 99% of the stuff related to Iraq is filtered out in robots.txt.
/infocus/iraq directory (which is dissallowed in robots.txt)
But not a problem, on google.com I just specify the site by saying 'Iraq site:whitehouse.gov' and it had 14,000 hits... the first one is the root of
I have to admit, when I first read the story I thought someone was being paranoid. But you really should RTF robots.txt file before you accuse the poster of being paranoid. The disallowed files are extraordinarily specific. I really can't come up with a plausible explanation beyond simoniker's.
Obviously, they're keeping people from accessing the top-secret teeball Iraq files ! Besides:
check out these other frightening examples of censorship:Truly frightening.
Computer Go: Writing Software to Play the Ancient Game of Go
welcome our White House Robot Overlords. It would be funnier if it weren't true.
- - - If the sun is a star, why can't I see it at night?
This gets modded up as Insightful? I mean, the White House is routinely editing their trascripts, and if bots like Google and Wayback can go and find that no, Bush said that we found weapons, not a weapons program, then there goes Bush's latest FUD... *thud*. Just because it's a tinfoil hat worthy theory doesn't mean it isn't true... most aren't, but therein lies the issue: most.
#define DRM chmod 000
Here's a minor example of something those two sites didn't catch: Remember Iraq's so-called "mobile biological weapons factories"? A month after the story broke that they were for weather balloons, the CIA moved their report's URL.
An intriguing fact about this whitehouse.gov/*/iraq thing is that they do in fact cover some of the important statements which are apparently not duplicated in the press release, conference, and briefing directories. Perhaps there was a "unique urgency" to cover up some poor choices of words?
If you try actually *loading* the directories listed in the robots.txt, they don't exist. Not one. Not by going to their index.html or trying to find them through the site navigation. While they could still be accused of deleting them, many of the links are unlikely to have existed in the first place (http://www.whitehouse.gov/president/heartland-tou r-gallery/iraq? /president/holiday/decorations/iraq? /president/tee-ball-01/iraq? ) This may be just some IT grunt running a bad script on robots.txt.
Keep telling yourself that.
And 70% of the people in this country STILL think that Saddam played some part in 9/11. What was your point again?
Looks like they removed a bunch of files where they were making claims that Saddam was behind 9/11. One could be lead to suspect that now that Bush got his war his doesn't need that lie anymore, and wants to erase all history of it since it undermines his authority.
Peace, or Not?
Better explanation: Someone screwed up a search-and-replace in a major way. Many (most?) of those pages with "iraq" in them don't exist.
It looks like someone blocked off parts of the site to web-crawlers; I don't know for sure why all those blah/bloo/iraq entries are in there but they sure as hell don't lead to anything.
Censorship: 0
Screwups: 100
what's that old saying? "never attribute to malice that which can be attributed to stupidity" or something like that?
let's not get reactionary here, folks. it wouldn't make sense to do what's being alleged:
1. every major journalist worth his/her salt would be all over it within hours. so it wouldn't succeed in obscuring information.
2. it would create an incredible backlash as soon as detected. what purpose would this serve?
ed
The complaint is they've done it before - "combat operations are done" became "major combat operations are done" when the fighting didn't stop. You can check here.
Compare the screenshots of what used to be on the white house website vs what's currently on the website.
Yes, I know, "how do we know this blogger didn't alter the screenshots?" You don't.
Downloading the "robot.txt" file and doing a quick ctrl-f on different words, I discovered that there are six instances of "Barney" coming up in the robot.txt:
/holiday/2002/barney/iraq /holiday/2002/barney/text /kids/barney/iraq /kids/barney/text /kids/photoessays/barney/iraq /kids/photoessays/barney/text
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Which is the same number as "cheney", "powell" had 4, "saddam" didn't have any and "bush" only comes up with "bushpets".
Clearly, there is something to do with Barney and Iraq that The White House doesn't want you to know about.
myke
Mimetics Inc. Twitter
It really doesn't look like it. It looks like someone screwed up, because none of those directories appear to exist at all. I mean really, what are the chances of /firstlady/photos/2003/01/iraq actually having at some time contained real data?
It looks like someone did a
find . -type d|perl -e 'while(<>){print "${_}/iraq\n"; print "${_}/text\n";}' > robots.txt
I have no idea what the purpose would be, but it seems like a funny thing to do if you were trying to hide something.
By the way, who is going around looking at people's robots.txt files?
Engineering and the Ultimate
Paranoia aside, I object to these restrictions as a matter of principle. They're making it more difficult to access publically available information. It's not classified, and it never was. I, as a citizen of the U.S.A., have a right to know what my leaders have said and done.
Let's assume the whitehouse.gov search engine is completely honest, and faithfully returns a complete listing of all materials on the site having to do with Iraq. If that's so, then there should be no reason to disable other search engines, since their results would just confirm the internal results.
But the restrictions are in place, meaning that someone thought there was a good reason to do so. Restricting access makes it more difficult for people to research information pertaining to Iraq on the whitehouse.gov web site. Who are the people most likely to be doing that? Answer: journalists, activists, and concerned citizens. Obviously these restrictions aren't enough by themselves to dissuade a determined researcher; but it might slow them down. And it might actually stop a diffident researcher completely.
I'm not even going to go into scenarios where the whitehouse.gov search engine is not trustworthy, because serving up "doctored" speeches or information is highly unlikely. There are too many other archives to compare against, and it would be a major scandal if the administration was found to be altering records on its website. They'd have to be really, really dumb to do that.
The whole thing still leaves a bad taste in my mouth, though.
Obviously robots.txt just happened to be in the path!
flossie
Write now. Defend liberty
"1. every major journalist worth his/her salt would be all over it within hours."
Don't be naive. How long do you think that any mainstream journalist who made a story of this would have a job for? The answer - not long. The US media in particular, although the UK is getting as bad, is little more than a relay system for government propaganda and real, detailed, complete examination of government behaviour, with equal air time to truly dissenting opinions (how many times has Chomsky been on CNN in the past 4 months?) is out of the question. What the government does is Good and Right and Should Not Be Questioned.
Media by the elite, serving the elite.
The other rule for transparency is that all material information be made available, kept, or destroyed in accordance to public regulation and individual policy. Individual policy must be consistent and decisions must be defensible based on policy.
The fact that people do not understand these two aspectsof transparency are what allow situations like Enron to develop. The later is what caused the destruction of Arthur Anderson. They have done nothing wrong, but they did not follow their own policy on document destruction, which made then look like at best idiots and at worst criminals.
We may compare this to other ventures to suggest policy. The NYT does not want google to cache articles because the NYT sells those articles after a certain time. Many other companies do not want deep linking because it reduces ad revenue. A fascist government may want to insure all users enter their site from a top page to make sure all users must go through the daily propaganda. A library tries hard to not track patrons so that no is afraid of using the library. The rational of the White House is beyond me.
The White House is not hiding documents. However, they are reducing the transparency of the government by limiting the avenues by which the public may access documents. Since the White House has stated many times that it believes in transparency, and in fact requires transparency when dealing with other governments, one can stipulate that transparency is the appropriate standard. So, until someone comes up with a policy that was developed and vetted through the normal processes used in the U.S., one has every reason to suspect nefarious motives.
And, if I may modify a statement that conservatives like to make, if you do not like transparency, go move to Iraq.
"She's a scientist and a lesbian. She's not going to let it slide." Orphan Black
An honourable country would not keep people imprisoned in Guantanamo Bay without either giving them PoW status or charging them with a specific offence and giving them the right to a fair trial, including free, unhindered and unmonitored access to legal counsel.
flossie
Write now. Defend liberty
Where have you been living the past five years? Journalists don't criticize Bush.
They still have not published the fact that he deserted from the national guard during Vietnam and they practically ignored his DUI conviction.
The GOP has the media cowed with their constant 'liberal media' babble. There number of journalists who are prepared to hold Bush to account is tiny - Krugman, Conanston, Irvins, Alterman. After that its Al Franken, Jon Stewart and David Letterman.
it would create an incredible backlash as soon as detected. what purpose would this serve?
The chances that the mainstream media will pick this one up are very small. Just think how they would have reacted if it was Clinton!
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
Nobody thinks Bush and Cheney are updating the website. Jeeze. But the folks that are running the website (and I would bet this extends down to the actual webmaster/tech guy) are political appointees who are there to make the president look good. That is their job. Their actions are all filtered through this political role.
Let's present an alternate scenario - since you have no evidence for yours, I don't have to present any evidence for mine.
It's May - Pres. makes his speech on the Carrier, the assumption by those-in-charge are that Chalabi's government will have control of the country within a couple of weeks and the US troops will be heading on home. The web folks (who want to make B & C look good) declare "combat's done! the troops are coming home! re-elect Bush!"
A few months later, that rosy scenario hasn't quite panned out. The aircraft carrier speech is becoming a liability for Bush - people started counting the number of dead troops in Iraq since he gave the speech, and it keeps going up. The web folks (who want to make B & C look good) say to themselves "this is a potential embarrassment to the president - let's see how we can make it less embarrassing."
And there you have it.
The crux of the matter is that he refused to have his pilots medical just after the Pentagon added a check for illegal drug use.
You can try to spin this whichever way that Karl Rove tells you but the facts are against you. The fact is that your great leader is a coward who ducked the draft and then deserted to avoid a drug test.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
Well terrorists have been attacking us since we have been in Iraq till this point in time, but i guess that doesnt mean there is any link..... naaaah
Native people fighting against an occupying force are known as freedom fighters, not terrorists.
ry again sparky.