Slashdot Mirror
White House Website Limits Iraq-Related Crawling
oscarcar writes "Dan Gillmor is reporting on the White House website's use of its robots.txt file to disable search engines from crawling certain material. Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."
whitehouse.com doesn't have that problem.
sulli
RTFJ.
it's good to see the whitehouse embracing technology so much.
!(^((ri)|(mp))aa$)
Many excluded items in the robots.txt file involve mentions of Iraq, possibly to prevent people from finding changes to past statements and information when archived elsewhere."
Maybe, but I would think they might also be looking for "shady" spiders that ignored robots.txt. I wouldn't be surprised if there aren't a few honeypot pages in there too.
To ensure perfect aim, shoot first and call whatever you hit the target
Queue somebody to take a crawler (hell, even a bash script using wget) to specifically archive these pages. Hell, they could even use a user-agent which doesn't look like a bot.
Of course, people would be less likely to trust random-Joe from the Internet than, say, The Wayback Machine, but I expect this is what will happen...
I help pay to run that website! /sarcasm
the preceding comment is my own and in no way reflects the opinion of the Joint Chiefs of Staff
Or you'll tear his tinfoil hat and then the black helicopters will be able to find him again.
Nugs
...if true. I'd like some proof that the WH is trying to cover-up before we put it in a posted news story though.
If true however, this would be mainstream newsworthy, IMO.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
If this was some crazy government conspiracy and they were trying to hide the information, why would they put it on their website? Could be any number of reasons they have done this perhaps they were getting loads of hits from google about iraq related things but if anyone really wants the information surely they can just visit it.
--
On Slashdot I'm a lawyer.
Disallow: /president/spongebobsquarepants_archive
I didn't know gee-dub likes SpongeBob too! My nephew is gonna flip out when he hears this.
"My mother never saw the irony in calling me a son-of-a-bitch." - Jack Nicholson
crackers... lol...
Perhaps their goal is simply so that when people google or whatnot for information on the Bush Administration and Iraq, they will be likely to find the Bush Administration's current views on and actions in Iraq, rather than outdated material?
Completely ignoring for the moment the fact that these views and actions are really somewhat embarrasing for the Bush administration, this really makes sense from a practical viewpoint. Few things are as annoying as searching for something news-ish and finding primarily material from two years ago. And after all, if they ONLY were interested in people forgetting the old materials, they could have just removed those materials from the site totally. (Though perhaps they were aware removing the materials completely would cause mirrors, which would be fully searchable, to spring up.)
Irritable, left-wing and possibly humorous bumper stickers and t-shirts
If you're surprised by this, THAT's the news, not what the White House is doing with this information control. Click here for a list of the White House's policies with restricting FOI and other related requests since Sept 11th.
This isn't partisan politics, either. The Republican party has been trying to keep Bush from violating the Presidential Records Act.
Yes, yes, the country's at war. Makes you wonder why Bush doesn't want anybody to know about communications between Reagan and his advisors.
--------
Bleah! Heh heh heh... BLEAH BLEAH!!! Ha ha ha ha...
American people should have some say in a situation like went on in Iraq.
They do, it's called voting, not to mention public opinion polls, which were near 70% for the invasion when the US invaded.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
Robots accounted for well over 10% of all web traffic at a Huge E-commerce company I worked at a few years ago...
Those robots consumed Many millions in system capacity.
Of course this is completely different as our freedom is at stake.
Congratulations to shrikel, who doesn't understand what that "quotation marks" are not just for fun.
While anything is possible in politics, is it possible that the web admin is trying to limit the amount of traffic on the site? Is it possible that his analysis of the weblogs show a lot of traffic from robots looking for Iraqi-related info?
If you persist in contemplating a world where whatever statements that the WH puts out, no matter how they might seem to contradict previous statements, are not totally true and correct, then a relocation expert from Guantanamo will be by in a few minutes. Just step away from the computer.
simple try to go to one of those links from whitehouse.gov they're broken. further check the http return code its 200. This looks more like an incorrect usage of http response codes and someone was nice enough to let the webcrawlers no about it. see!! HTTP/1.0 302 Moved Temporarily Server: Apache Last-Modified: Mon, 27 Oct 2003 21:29:15 GMT Content-Length: 153 Content-Type: text/html Location: http://www.whitehouse.gov/911/iraq/ ETag: "7fbd6f-1d5-3e8b0d9f" Cache-Control: max-age=1970 Date: Mon, 27 Oct 2003 21:29:15 GMT Connection: close and following the location: HTTP/1.0 302 Moved Temporarily Server: AkamaiGHost Content-Length: 0 Location: http://www.whitehouse.gov/error-404.html Date: Mon, 27 Oct 2003 21:29:46 GMT Connection: close and a little further we get the 200, which should be 404 ;)
HTTP/1.0 200 OK
Server: Apache
Last-Modified: Sat, 26 Jul 2003 07:29:19 GMT
ETag: "7432cf-4347-3f222dcf"
Accept-Ranges: bytes
Content-Length: 17223
Content-Type: text/html
Date: Mon, 27 Oct 2003 21:30:06 GMT
Connection: close
so at least there is some admin nice enough to spare the crawlers and search engines the burden of loggging all these mis represented 404 pages, while also saving them the trouble of three redirects.
this is a good robots.txt to fix a bad http server
All of the now non-spidered pages can be located in Room 101.
It looks like 99% of the stuff related to Iraq is filtered out in robots.txt.
/infocus/iraq directory (which is dissallowed in robots.txt)
But not a problem, on google.com I just specify the site by saying 'Iraq site:whitehouse.gov' and it had 14,000 hits... the first one is the root of
Nothing's hidden, it's all there, it's all searchable from the white house website, just not from search engines.
I have to admit, when I first read the story I thought someone was being paranoid. But you really should RTF robots.txt file before you accuse the poster of being paranoid. The disallowed files are extraordinarily specific. I really can't come up with a plausible explanation beyond simoniker's.
possibly to prevent people from finding changes to past statements and information when archived elsewhere.
could you even possibly consider maybe the server was getting slammed all to hell? remember bandwidth isn't free for anyone....i'm sure there could be a thousand other reasons also...
now before the men in black & the majestic 12 find you go grab your nice tin foil cap that keeps them from reading your mind and hides you from the aliens....
Obviously, they're keeping people from accessing the top-secret teeball Iraq files ! Besides:
check out these other frightening examples of censorship:Truly frightening.
Computer Go: Writing Software to Play the Ancient Game of Go
aren't there laws that stop the government from blocking that sort of thing? I could be wrong, but that could be illegal.
-Seriv
The use of the robots.txt file by crawlers isn't madatory, at no point is it ever enforced, it's merely a curtesy.
All you'd have to do to continue indexing their site is to write a crawler that ignores robots.txt.
Consider the fact that GW Bush has banned media (hello?? freedom of the press? 1st Amendment??) coverage of returning killed soldiers. Why? Because seeing dead soldiers makes people realize that the war is real and people are dieing.
The current administration is trying its damndest to control infomation that it doesn't like
welcome our White House Robot Overlords. It would be funnier if it weren't true.
- - - If the sun is a star, why can't I see it at night?
...it was posted on Slashdot!
God invented whiskey so the Irish would not rule the world.
The tinfoil hat, here comes the conspiracy.
Winston's greatest pleasure in life was in his work. Most of it was a tedious routine, but included in it there were also jobs so difficult and intricate that you could lose yourself in them as in the depths of a mathematical problem -- delicate pieces of forgery in which you had nothing to guide you except your knowledge of the principles of Ingsoc and your estimate of what the Party wanted you to say. Winston was good at this kind of thing. On occasion he had even been entrusted with the rectification of the Times leading articles, which were written entirely in Newspeak. He unrolled the message that he had set aside earlier. It ran:
times 3.12.83 reporting bb dayorder doubleplusungood refs unpersons rewrite fullwise upsub antefiling
In Oldspeak (or standard English) this might be rendered:
The reporting of Big Brother's Order for the Day in the Times of December 3rd 1983 is extremely unsatisfactory and makes references to non-existent persons. Rewrite it in full and submit your draft to higher authority before filing.
<a href="http://www.joblessjimmy.com">Work is dumb and so is Jobless Jimmy.</a>
It could be something innocent but really, why would anyone want to keep search engines out of a publicly funded website? People have been accusing the poster of "baseless accusations" but the guy does have a point. I've seen a couple of GW's speeches and afterwards the transcripts of those speeches and noted that gramatical errors were corrected. While this is only a minor offence in editing history it does make you wonder what other opinions and information may have appeared and then later have been edited. Seriously, these are our government officials here, we deserve to have an unedited record of what they say and to hold them to it. A little bit of speculation on the reasons for excluding various terms is far from paranoia.
Chris
This gets modded up as Insightful? I mean, the White House is routinely editing their trascripts, and if bots like Google and Wayback can go and find that no, Bush said that we found weapons, not a weapons program, then there goes Bush's latest FUD... *thud*. Just because it's a tinfoil hat worthy theory doesn't mean it isn't true... most aren't, but therein lies the issue: most.
#define DRM chmod 000
that's strange... cause if they wanted to hide some information, they'd found a much more effective way to do that. As they are possibly doing with some really important information.
We could have saved sixpence. We have saved fivepence.
From the robots.txt file:
/easter/iraq
Disallow:
Does this mean they're going to ban Christmas in Iraq too?
Ruby on Rails Screencast
Here's a minor example of something those two sites didn't catch: Remember Iraq's so-called "mobile biological weapons factories"? A month after the story broke that they were for weather balloons, the CIA moved their report's URL.
An intriguing fact about this whitehouse.gov/*/iraq thing is that they do in fact cover some of the important statements which are apparently not duplicated in the press release, conference, and briefing directories. Perhaps there was a "unique urgency" to cover up some poor choices of words?
If you think that's the most paranoid comment, you must not read any RFID tag threads here yet.
Gore isnt running, did you even realize that there was a debate on with 9 candidates in Detroit. None named Al Gore.
I have a Cig, but do you have a light?
by sipping some Victory! gin and smoking some Victory! cigarettes.
wow, a webmaster changed his robots.txt. i'm amazed.
vodka, straight up, thank you!
"possibly to prevent people from finding changes to past statements and information when archived elsewhere."
Yeah, that's not a baseless accusation at all...
It isn't so much an accusation, it sounds more like they were just unpacking their Reynolds Wrap
From the robots.txt file:
/kids/barney/iraq
Disallow:
Thank goodness they're limiting the export of that blasted purple dinosaur!
Ruby on Rails Screencast
Look at the robots.txt file. It's pretty amazing: it excludes the normal stuff for a robots file: dynamically-generated pages, and pages like FAQs. It also excludes a whole mass of specific pages and groups of pages: ones relating to Iraq. Something's up.
"They redundantly repeated themselves over and over again incessantly without end ad infinitum" -- ibid.
You found it didn't you? It failed... congratulations, you have somehow circumvented the government's website security system, prepare for the wrath of the DMCA, backed by none other than Bush himself!
;)
Well either that, or it's simply preventing search engines from indexing honeypot type pages used for mis-information... Either or... but I like the first version... since it's more paranoid, and I have plenty of tinfoil ready to be shaped into hats...
---
Programming is like sex... Make one mistake and support it the rest of your life.
Or maybe, just maybe, they're doing it to save their server from being constantly crawled by paranoid conspiracy-theorists looking for changed statements and information.
Vintage computer games and RPG books available. Email me if you're interested.
better explanation would be?
Goodness knows we can't have googlebots archiving all of those top-secret/confidential web pages at the whitehouse. I guess we'll just have to live with the top-secret info that has already been archived.
What's that? Oh, all of the real top-secret stuff is at the NSA website?
Never mind then.
Disallow: /kids/baseball/teeball-20020923/iraq
The technology they introduced in 1968 allows them to rewrite the books and the newspapers. The next step is to make sure that the old copies get burned reliably. America was always in war with Iraq, you know...
this is a funny site.
You're right, screw complaining, let's go for impeachment...
There are plenty of reasons to use robots.txt besides copyright. The most common is to prevent cgi-feedback loops where a generated page links to a copy of itself at a different url. (Or just to prevent the spiders from running the resource-hungry cgi's in the first place.)
;-)
Heh, that could actually be applied here: the information in the pages changes so often the spiders would be constantly reloading the pages, and therefore overloading the server...
'Sensible' is a curse word.
If you try actually *loading* the directories listed in the robots.txt, they don't exist. Not one. Not by going to their index.html or trying to find them through the site navigation. While they could still be accused of deleting them, many of the links are unlikely to have existed in the first place (http://www.whitehouse.gov/president/heartland-tou r-gallery/iraq? /president/holiday/decorations/iraq? /president/tee-ball-01/iraq? ) This may be just some IT grunt running a bad script on robots.txt.
I can't see this as a conspiracy .. it's just too silly.
Why on Earth wouldn't they just EDIT the bleedin' files? They wouldn't have to delete them or set up robots.txt, they would just change them to reflect the "message of the moment". They probably do that anyway, same as a lot of other sites.
Do they really think people would be blocked by robots.txt?? Nobody's that dumb (yeah they could be Windows MSCE droids but c'mon).
I think they did it for some other reason like keeping traffic down.
Another possibility: a hacker got in there and did this because a) he only had write access to robots.txt for some reason or b) he wanted to play a subtle joke. But I doubt that too.
Anyway this is strange, but pointless, so I wouldn't bother with it unless you're a democrat looking for something else to whine about...
Thanks.
This is a test. This is a test of the emergency sig system. This has been only a test.
I checked out the files at www.whitehouse.gov/robots.txt and found that there are over 1600 disallow lines and almost half of them had the word Iraq in the URI.
This could lead one to suspect that this website is trying to control the distribution of information about those articles. I mean, that is what a spider is for, right (among other things...)
Honestly, when I saw the robots.txt file, even I thought of Orwell and how they changed old stories in newspapers to continually change history to reflect whatever the goverment wanted people to believe at that moment.
Peace, Or What?
Peace, or Not?
Look at the robots.txt file, it is filled almost exclusively with Iraqi related links. This is not just a generalized disallow list, it is very specific and should definitely be of note. It really makes you wonder what the government is up to and what it has to hide. 1984 anyone?
so the theory is that these pages, which contain things spoken in public like transcripts and stuff, are published on the web (not only by the white house I'd imagine, probably by a lot of groups) - but really the pages are hidden because, you know, they're in the robots.txt
That's the theory? or did I miss something?
"publish this stuff on the web site, but uh, make sure people can't find it"
"in that case, how about if I just don't publish it?"
"no no, we want people to find it."
"so why hide it?"
"my god man, do you know what would happen if people find this?"
Why would a webcrawler follow the robots.txt file? Is it governed by law or is it just standard practice? I guess your calling them "shady" spiders must mean its illegal... wierd.
Thank you for pointing out this out to us. As a token of our appreciation for your views, you have just won a free all expenses paid trip to lovely Guantanamo Bay in sunny Cuba! A team of men in black suits will be at your door shortly to help you with your trip. Sincerely, John Ashcroft The Minisitry of Information, Love, and Fluffy Kittens
As of just a few minutes ago, these entries were seen added to the robots.txt file:
/news/slashdot /news/tinfoilhat /allyouriraq/are/belongto/US ...
/. ... if they're so worried about people finding out their insidious plots, they'll just flip the switch on all their mind-controlling ...
... MUST DESTROY WEBLOGS ... TRUTH GETTING OUT ... DUBYA IS MY FRIEND ... MUST DESTROY SLASHDOT ... MUST DESTROY WEBLOGS ...
Disallow:
Disallow:
Disallow:
Come on. This is extremely paranoid and far-fetched, even for
MUST DESTROY SLASHDOT
topreacher@signature.slashdot.org 1% rm -rf sig
Most of the pages in the robots.txt are actually 404's and dont exist anymore. Its that simple. Keeps the robots from constantly requesting content that doesn't exist anymore. A few are blocked because they are bandwidth intensive videos and things, and some others are blocked for more mundane reasons I assume.
Or search engines that abide by the rules of robots.txt. Anybody could write a search engine that archives specifically what is in robots.txt.
Anyway, maybe the Whitehouse doens't want to be the number 1 hit when searching for "iraq" in google. Kinda hints at that whole "occupier" vs. "rebuilder" dilema if you search for iraq and get all sites from whitehouse.gov.
Or maybe the content changes so frequently that any archives would be almost immediately out-of-date. Of course it could be something more nefarious, but it just sounds like some below-the-radar PR to me.
Why, o why must the sky fall when I've learned to fly?
robots.txt is fine its the server that is screwed up those are dead links that their server responses with 200
Ditto.
This is a test. This is a test of the emergency sig system. This has been only a test.
Nosirree, no legitimate webmaster would ever use robots.txt to gently guide visiting bots to the appropriate parts of the site and to keep them from trying to do silly things. The only possible use is to trample your rights while installing the new corporate-owned government.
Geez, people. Honestly.
Dewey, what part of this looks like authorities should be involved?
Have you tried going to the locations mentioned? I tried a few and got invalid pages. Of course, that doesn't say there were subdirectories that I can't see... maybe it's a just a way to quickly prune old stuff from search engines without having them spider the site or check old links and generate 400 codes.
Seems odd and pointless to me. I'd like a statement explaining it. A lot like the "Disallow: /hidden/passwd" kind of entries.
Looks like someone just added IRAQ to all of the exsiting links. It's obviously some sort of search/replace/copy function. Go look for yourself, I found this one:
/firstlady/recipes/iraq
Disallow:
Now, how many pages would this possibly block?
M@
Krispy Cream is people
The American people basically have a say, it's called polls. Do you think we would have gone to war if the support was, say, 10%? I don't think so. When we went to war, there was overwhelming support (70% in a poll is a HUGE margin). Politicans aren't stupid, if there isn't support for it, they won't do it.
The sheep are being led. One can only wonder which direction this time, and when someone will finally get to finding those weapons of mass destruction. Or was it removing an evil dictator? Or was it a terrorist government? I forget. And what's worse, it looks like now I won't be able to go back and figure out which one it was...
See, now that's the problem right there. It's not that we care so much about what's on the White House website, since obviously none of it is going to be unbiased, objective reporting in the slightest and thus it has no significance to any of us who are doing real research rather than kissing ass to the President and Dale Earnhardt (because if you don't respect #3, you're unpatriotic and un-American). What we are concerned with is the ability for them to change their wording of things to save themselves from public scrutiny, and deny us the ability to say "Hey, what's going on? I could have sworn that was different a few days ago" and go check Google's cache or the Wayback Machine to determine that, yes, that quote is entirely different than the one that was initially posted.
Keep telling yourself that.
And 70% of the people in this country STILL think that Saddam played some part in 9/11. What was your point again?
With everyone paying attention to the pages listed in robots.txt, maybe they won't see the true secrets hidden in plain sight...
Right now if you google for "iraq" and "whitehouse", http://www.whitehouse.gov/infocus/iraq/ is the top link. Second link is also on whitehouse.gov Amusingly, http://www.whitehouse.org (an anti-bush site) comes up second in the list of sites. If whitehouse.gov manages to push itself out of the robots lists, then guess which site people will see most often?
Looks like they removed a bunch of files where they were making claims that Saddam was behind 9/11. One could be lead to suspect that now that Bush got his war his doesn't need that lie anymore, and wants to erase all history of it since it undermines his authority.
Peace, or Not?
The majority of American People did not vote for this administration. The American People, my friend elected Al Gore. This administration was put in place by the Supreme Court. Has your brain been washed so quickly you have already forgotten? Wake up people these guys don't give a shit about you or anyone you know unless they have a net worth greater than 10 million. Look at the facts, overall our economy is in the toilet with the vast majority of citizens considerably worse off than they were 4 years ago. Of course, the extremely rich are doing kust fine, getting extremely richer.
every time a republican dies a queer angel gets his wings
One would hope that google cache and/or the wayback machine would not be responsible for archiving government websites/propoganda anyway.
The US National Archives *should* be keeping backups of the predidential web sites. I poked around for a bit, and couldn't find any archives of the clinton administration's web site, so maybe they don't.
If they do not, it would be very irresponsible...
Except that, ya know, they *have* been caught lying about Iraq, repeatedly, in the last few months, and are currently doing another lie/spin campaign trying to get everyone to agree that their previous lies never happened, and anyone pointing out that they did lie are just the evil liberal media (TM) attacking them.
Has Microsoft excluded Linux yet?
Has /. excluded goatse posts yet?
This movement has a long way to go.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
robots.txt gives instructions to well behaved search engines. If you all are particularly paranoid about it (and I'm not saying you shouldn't be), go crawl it yourself and *gasp* ignore robots.txt
Sleep is just a poor substitute for caffeine, anyway. -Bob Lehmann
It looks like they just appended iraq and text on to the end of all their web folders. What's the big panic about?
Why should a government-authored site (which, under the Constitution, by definition is public domain text) be exlcuded from non-government electronic publishing sites?
By the way, show me where in that Robots.txt file there's a command that would block http://www.whitehouse.gov/holiday/2002/art/01.html from Google? If you're right, there should be a line
disallow /holiday/2002/art/ . I don't see one. So, yeah, it's explicitly Iraq-related stuff that they're trying to block. Either 1. they're afraid that sensitive information might end up on the site by accident and want to make sure that it isn't archived if it is - in which case, they've got a lot more serious problems than political connivance - or 2. the theory is correct, and they're trying to set up a memory hole. Given Karl Rove's history, which do YOU think it is?
I honestly think this is stuff that goes on beneath GWB's notice. I'm with Molly Ivins on him: he's not evil, mean, or stupid, just wrong.
So, oh non-me entity, why do you think whitehouse.gov is now excluding spiders from a bunch of pages having to do with Iraq?
.htaccess to protect nuclear launch codes?
isn't that the American people didn't vote for the current administration (they did)
50,456,169 - bush
50,996,116 - gore
it seems like about 539947 more americans voted for the other admin.
yes, i know that's how the rules are set up, but it's a pretty hard sell to prove that a system in which a candidate can lose by half a million votes and still win is a 'democracy.'
swear to god, it's not a troll.
!(^((ri)|(mp))aa$)
Silly me, that's why Google sends so few results "mass destruction irak", for a moment I thought they didn't thought any.
I feel better now.
%bash bush
I thought Gore was the Robot.
"It's the height of ridiculousness to say for those 9 lines you get hundreds of millions."
> a directory called /president/holiday/deck-halls/iraq
It's about an initiative to bring XMas to the heathens of the Middle East!
or not...
It seems like every single directory has had the word "iraq" appended to the end. Do you think that this might have been a knee-jerk reaction by some admin who didn't really know what they were doing? I can't really imagine there are legitimate iraq dirs under easter and teeball directories.
try google searching the following: "site:whitehouse.gov iraq"
It appears that google is not paying attention to the nice requests of the robots.txt
Don't waste time... procrastinate now!
It appears that this robots.txt file was probably auto-generated. It looks like someone used a script to crawl the sites entire directory structure appending /iraq and /text to every directory. In the process they seem to have created a pretty complete map of the sites underlying directory structure -- not necessarily a good thing.
.html if they're actual pages.
Having said that, I'm not even sure that this robots.txt file would work the way it's supposed to. Seems like these iraq references should all have a trailing slash or a
Someone clearly doesn't want Google caching Whitehouse content on Iraq. The question is why? And how come they're so lame about it?
There hasn't been a real declared war since WWII. You can't "declare war on terrorists" and be done with it either, wars are supposed to be declared on countries when you go to fight them. It was what an honorable nation would do before hostilities.
This is /. and you expect things to be THAT easy?
'Standards' in computing only impress those who are impressed by things like 'standards'.
If you look at the robots.txt file near the top you will see they removed a bunch of files with the URI of Iraq and 9/11 in them. I can't seem to find them on Google, but one could reasonably suspect that these are now potentially embarrising statements about how Saddam was the real brainchild behind 9/11, since that was the prime argument to convince Americans to go to war. I mean, I don't see them disallow any links to stuff about student loans in that robots.txt file :)
If anyone can find archives of that stuff that is not what I am guessing above, I'll happily eat crow.
Peace, or....
Peace, or Not?
It's all double plus good.
at least when they censored information before they tried to cover their tracks. Talk about sloppy...
I knew it!
- Grep the errors log for 404's from search engines.
- Parse out the directory paths.
- Add those to robots.txt.
Which might explain why at least one of the directories -I have to agree that it's more strange than sinister. Besides, I'm not sure that the web site is the official archive for white house statements.
What about the poor puppies!
/holiday/2002/bushpets/iraq
Disallow:
what the bushpets did not get to go to Iraq in 2002? Poor things! Someone call the SPCA!
Going on means going far
Going far means returning
Anybody who's been running a website knows that there are more search engine crawlers out there than there are grains of sand on the beach, and the amount of crawling constitutes (even for a low-ranked blog like mine) an amazing level of traffic.
My guess is, the White House is just keeping robots off the pages that get the most hits, in order to make better use of their bandwidth. After all, folks, if they wanted to make that stuff unavailable, they'd take it offline.
Is anybody keeping a mirror of whitehouse.gov and additions, deletions, and changes?
I can't check archive.org (&*^@!*&$^ censorware) or whitehouse.com (:-) from work, but I'm assuming that it's polite about obeying robots.txt. However, the White House is impolite about changing data, so somebody ought to run a robot that ignores it.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Seems plausible. However, since the President is usually the number one government source for just about everything, why wouldn't they use robots.txt to limit server loads on the following searches:
Unemployment
The Economy
The United Nations
Tech job outsourcing
Tax Cuts
Budget Deficits
The most recent Congressional vote on Widget manufacturer subsidies
....you should get the point by now
Entered 'less robots.txt | grep iraq > iraq.txt', then ran 'vim iraq.txt' and came up with 768 lines! WTF?
# robots.txt for http://www.ingsoc.gov/
/cgi-bin /search /query.html /help /appointments/eurasia /appointments/eastasia /ask/images/eurasia /ask/images/eastasia /deptofhomeland/analysis/eurasia /deptofhomeland/analysis/eastasia /deptofhomeland/eurasia /deptofhomeland/eastasia /economy/eurasia /economy/eastasia /goodbye/eurasia /goodbye/eastasia /government/handbook/eurasia /government/handbook/eastasia /government/images/eurasia /government/images/eastasia /government/eurasia /government/eastasia
User-agent: *
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
And now, an offering for the lameness filter...
Oceania was at war with Eastasia: Oceania has always been at war with Eastasia. A large part of the political literature of five years was now completely obsolete. Reports and records of all kinds, newspapers, books, pamphlets, films, sound tracks, photographs- all had to be rectified at lightning speed. Although no directive was ever issued, it was known that the chiefs of the Department intended that within one week no reference to the war with Eurasia, or the alliance with Eastasia, should remain in existence anywhere. The work was overwhelming, all the more so because the processes that it involved could not be called by their true names. Everyone in the Records Department worked eighteen hours in the twenty-four, with two three-hour snatches of sleep. Mattresses were brought up from the cellars and pitched all over the corridors; meals consisted of sandwiches and Victory Coffee wheeled round on trolleys by attendants from the canteen. Each time that Winston broke off for one of his spells of sleep he tried to leave his desk clear of work, and each time that he crawled back sticky-eyed and aching, it was to find that another shower of paper cylinders had covered the desk like a snowdrift, half burying the speakwrite and overflowing onto the floor, so that the first job was always to stack them into a neat-enough pile to give him room to work. What was worst of all was that the work was by no means purely mechanical. Often it was enough merely to substitute one name for another, but any detailed report of events demanded care and imagination. Even the geographical knowledge that one needed in transferring the war from one part of the world to another was considerable.
This was written in 1948. Things have really progressed!
I'm playing devil's advocate more than anything. But aren't you scared in the least about the recent Diebold follies? It may deliver you another 4 years of Bush next November, but what if some commie pinko steals it next time?
There's still a few more months left for the media to come around and start looking into anyways. Look at how the mainstream media has only recently taken Bush to task over Iraq.
When I use the whitehouse.gov website, I am a citizen. There's a difference.
Disallow: /climatechangefactsheet/iraq
/climatechangefactsheet/text
Disallow:
Now why would they want to stop these being crawled?
Paul.
Absolutely, this is a case of some IT grunt running a bad script. However, as with most things in Washington I would guess the grunt didn't do this on his own initiative. So the question would be, who's idea was this and what was their motiviation?
You know, I was thinking there was probably some innocent technical explanation for this. But RTFR.TXT.
I can't think of any honest reason to do that.
All's true that is mistrusted
Downloading the "robot.txt" file and doing a quick ctrl-f on different words, I discovered that there are six instances of "Barney" coming up in the robot.txt:
/holiday/2002/barney/iraq /holiday/2002/barney/text /kids/barney/iraq /kids/barney/text /kids/photoessays/barney/iraq /kids/photoessays/barney/text
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Disallow:
Which is the same number as "cheney", "powell" had 4, "saddam" didn't have any and "bush" only comes up with "bushpets".
Clearly, there is something to do with Barney and Iraq that The White House doesn't want you to know about.
myke
Mimetics Inc. Twitter
It's true that you can't load the directories, but it's not true that they don't exist. Of course you can't load the directories, because they have directory listing turned off.
/infocus/iraq" and the first line in a search of Iraq on whitehouse.gov is /infocus/iraq/websites.html, which is clearly in the directory /infocus/iraq.
However, one of the entries is "Disallow:
=Brian
There is nothing so good that someone, somewhere, will not hate it.
The White House web site, while important for the Administration's efforts to put out press releases informing or convincing the public about whatever its agenda is this week, is also important for historical work and analysis. How it reacted to previous events, and to earlier phases of an ongoing situation, are quite important in understanding what they're doing today, and having a Ministry of Truth to make previous statements into unhistory is _not_ a good thing.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
It really doesn't look like it. It looks like someone screwed up, because none of those directories appear to exist at all. I mean really, what are the chances of /firstlady/photos/2003/01/iraq actually having at some time contained real data?
It looks like someone did a
find . -type d|perl -e 'while(<>){print "${_}/iraq\n"; print "${_}/text\n";}' > robots.txt
I have no idea what the purpose would be, but it seems like a funny thing to do if you were trying to hide something.
By the way, who is going around looking at people's robots.txt files?
Engineering and the Ultimate
I see no reason why any search engine or crawler should respect a robots.txt on any .gov site.
Then let's back up the site. How big is the WH site?
Democracy itself is largely a myth... the politicitians will say *anything* to get into power, and, once there, will say *anything* to stay in power.
However, what they actually do is what they damned well please.
Remember the education level of the populace really isn't that high - the average slashdotter is *much* more educated than joe sixpack (60% of people in the US have never read a book, for example). This is why a tall candidate with good hair is a usually a sure win, and 'we're going to raise taxes to so that we can improve transport/healthcare/whatever' is a sure loser, and 'we're going to cut taxes even though the last guy did the same and went bankcrupt. by the way have you seen my latest movie?' is a sure winner.
"possibly to prevent people from finding changes to past statements and information when archived elsewhere"
Ahem. Not even the most clueless of clueless ones would try that!
EMACS: could you prepend the following to each line: "wget --random-wait -r --page-requsisites "
Paranoia aside, I object to these restrictions as a matter of principle. They're making it more difficult to access publically available information. It's not classified, and it never was. I, as a citizen of the U.S.A., have a right to know what my leaders have said and done.
Let's assume the whitehouse.gov search engine is completely honest, and faithfully returns a complete listing of all materials on the site having to do with Iraq. If that's so, then there should be no reason to disable other search engines, since their results would just confirm the internal results.
But the restrictions are in place, meaning that someone thought there was a good reason to do so. Restricting access makes it more difficult for people to research information pertaining to Iraq on the whitehouse.gov web site. Who are the people most likely to be doing that? Answer: journalists, activists, and concerned citizens. Obviously these restrictions aren't enough by themselves to dissuade a determined researcher; but it might slow them down. And it might actually stop a diffident researcher completely.
I'm not even going to go into scenarios where the whitehouse.gov search engine is not trustworthy, because serving up "doctored" speeches or information is highly unlikely. There are too many other archives to compare against, and it would be a major scandal if the administration was found to be altering records on its website. They'd have to be really, really dumb to do that.
The whole thing still leaves a bad taste in my mouth, though.
Obviously robots.txt just happened to be in the path!
flossie
Write now. Defend liberty
> If anyone can find archives of that stuff that is not what I am guessing above, I'll happily eat crow.
Since it seems these urls never existed!
So the incriminating evidence is not in a Lexis Nexis stuff, but the Whitehouse put the worst stuff on the website, so only the mind controlled supporters will get the message, while the dimwitted opposers wouldn't look for the information there.
Yeah, this makes a lot of sense.
- sigs are for wimps.
well on osama's last audiotape, he credited saddam and his sons for all their help. close enough for me. :)
This post cannot be re-broadcast without the express written consent of Major League Baseball.
Last year the Washington Post ran a story on who would benefit from a war in Iraq. It mentioned Haliburton and the $50+ million in stock options Dick Cheney recieved from that company as an employee. It also mentioned the fact that Sadam had cancelled the oil contracts of several American companies. I tried to find it again to show a friend but the story had disappeared from both the Washington Post search routine and Google. (The WP could have had it removed from Google, I doubt Google itself had anything to do with the stories disappearance.)
So what? The White House has a robots.txt file. Check it out. It lists a ton of stuff, not just Iraq-related stuff.
But, hey, it's fun to give the liberals something to chew on to attack Bush on a slow news day.
He invented robots.txt though.
United States of America, good ol' backers of world peace.
Not only paranoid, unrealistic.
First of all, in order to prevent people from finding changes to past statements means ensuring that no previous copies of a given document can be found anywhere. Not exactly a high probability anymore, KWIM?
Second, there exists the potential for legitimate reasons behind the contents of the robots.txt file. (Like limiting the number of documents web-bots look through for performance reasons... or getting /.ed in order to provide impetus to legislation to shut down /. just to prevent laughable accusations of cover-ups.)
Third, no law exists TIAO which stipulates that the white house can't control what content is searchable by web-bots.
Fourth, this really doesn't inspire tinfoil haberdashery. Now, if the files had been removed, and were denied by the White House as having ever existed, well then I might start shopping for an aluminum fedora, except for...
Fifth, there is no guarantee regarding the truth of any document on any webserver. Just 'cause it comes through your browser doesn't mean it's true.
and
Sixth, even if the documents are being blocked from bots for some nefarious purpose, whoever crafted that plan has set a new low even for the toadying political appointees. Even making the observation lacks a certain fundamental understanding of the checks and balances provided by the instituion of a Free Press... Which, incidentally is protected by the constitution, try and find a similar guarantee in that hallowed document for web content...
"Talk minus action equals nothing" - Joey Shithead, D.O.A.
"Talk minus action equals
Right and since the white house has no control over their own search engine i'm sure it will map and archive everything so that discrepencies can be found.
"I can not bring myself to believe that if knowledge presents danger, the solution is ignorance" - Isaac Asimov
So, am I to understand that the same administration that was smart enough to rig an election, Smart enough to cause 9/11, Smart enough to forge evidence and go to war is the same administration that came up with the brilliant plan of HIDING information by putting it in a PUBLICALY availible file?
T Money
World Domination with a plastic spoon since 1984
The other rule for transparency is that all material information be made available, kept, or destroyed in accordance to public regulation and individual policy. Individual policy must be consistent and decisions must be defensible based on policy.
The fact that people do not understand these two aspectsof transparency are what allow situations like Enron to develop. The later is what caused the destruction of Arthur Anderson. They have done nothing wrong, but they did not follow their own policy on document destruction, which made then look like at best idiots and at worst criminals.
We may compare this to other ventures to suggest policy. The NYT does not want google to cache articles because the NYT sells those articles after a certain time. Many other companies do not want deep linking because it reduces ad revenue. A fascist government may want to insure all users enter their site from a top page to make sure all users must go through the daily propaganda. A library tries hard to not track patrons so that no is afraid of using the library. The rational of the White House is beyond me.
The White House is not hiding documents. However, they are reducing the transparency of the government by limiting the avenues by which the public may access documents. Since the White House has stated many times that it believes in transparency, and in fact requires transparency when dealing with other governments, one can stipulate that transparency is the appropriate standard. So, until someone comes up with a policy that was developed and vetted through the normal processes used in the U.S., one has every reason to suspect nefarious motives.
And, if I may modify a statement that conservatives like to make, if you do not like transparency, go move to Iraq.
"She's a scientist and a lesbian. She's not going to let it slide." Orphan Black
some of the URLs
and by some, you mean one right?
Perhaps that was the test case the script was SUPPOSED to fix?
Never confuse volume with power.
Oh well I think I'll worry about more important stuff.
Like how to convince my goverment to cut aid to Isreal by 2X the ammount it costs to have "settlers" on Palestinan land and all the Isreali troops used to protect those "settlers" and build that wall on Palestinian land instead of Isreali land. That is something worth worrying about since it actually might impact your life the next time some islamic facist decides to fly a plane into a building or starts a forrest fire. It sure as hell effects your pocket book by billions. Calling and writing my Senator doesn't seem to work. She is more interested in Isreals saftey then mine I guess.
Maybe we should just vote all the assholes out from A to Z dog catcher to president.
If you don't like what I write don't be a CS and mod it down. Refute it.
Yea I can't spell. So what is your point?
I didn't say "a majority of US voters voted for Bush," I said the American people elected him, under the Constitutionally-defined process. In other words, he got a majority of the Electoral College. Wacky? Yup. Evidence that we should scrap the Electoral College? Yup. The correct result under the rules? Yup.
Note also that the Gore campaign had circulated talking points just prior to the 2000 election so their operatives would be prepared to explain why even though GORE lost the popular vote and won the Electoral College, that was a legitimate result. They thought that it would go that way, not the way it turned out.
This really shouldn't shock anyone. It has been going on at the White House for ages. Look at this clip from the robots.txt file from 1998: /history/photoessays/blueroom/blowjobs /history/photoessays/blueroom/text /history/photoessays/cabinetroom/blowjobso w: /history/photoessays/cabinetroom/text /history/photoessays/crosshalls/blowjobsw : /history/photoessays/crosshalls/temp/blowjobss allow: /history/photoessays/crosshalls/temp/texto w: /history/photoessays/crosshalls/text /history/photoessays/diplomaticroom/blowjobsa llow: /history/photoessays/diplomaticroom/textw : /history/photoessays/downstairscorridor/blowjobs
Disallow: /history/photoessays/downstairscorridor/texta llow: /history/photoessays/easter/2002/blowjobso w: /history/photoessays/easter/2002/text /history/photoessays/easter/2003/defenselink/blowj obs /history/photoessays/easter/2003/defenselink/text /history/photoessays/easter/2003/blowjobso w: /history/photoessays/easter/2003/text /history/photoessays/easter/one/blowjobsw : /history/photoessays/easter/one/text /history/photoessays/easter/three/blowjobs
Disallow:
Disallow:
Disallow:
Disall
Disallow:
Disallo
Di
Disall
Disallow:
Dis
Disallo
Dis
Disall
Disallow:
Disallow:
Disallow:
Disall
Disallow:
Disallo
Disallow:
Viv
Gmail invites for ip
"Simply stated, there is no doubt that Saddam Hussein now has weapons of mass destruction. There is no doubt that he is amassing them to use against our friends, against our allies, and against us. And there is no doubt that his aggressive regional ambitions will lead him into future confrontations with his neighbors -- confrontations that will involve both the weapons he has today, and the ones he will continue to develop with his oil wealth."
I can't possibly imagine why the Bush administration would want to keep these kinds of quotes out of search engines...
This admin has a long history of attempting to rewrite history and expecting (and finding) that journalists are too cowed to call them on it. Extensive past behavior is a strong indicator of current motives.
And it's isn't a conspiracy every time W speaks. It's a lie. You can tell because his lips are flapping.
Like Bush and Cheney are personally updating the WH Website. I'm sure some Junior Web Monkey got the headline wrong (trying to be brief maybe?), someone Sr. Web Monkey eventually caught it, and his boss (like a Deputy Assistant Coffee Monkey) ordred him to fix it.
If at all unsure, chalk it up to a conspiracy.
- differentstrings.info
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
Yes. It was clearly automatically generated.
Now, why would the whitehouse want to automatically generate a robots.txt file that eliminated all references to Iraq?
That Jesus Christ guy is getting some terrible lag... it took him 3 days to respawn! -NJ CoolBreeze
Shut your hole, you fucking troll.
If _ANYONE_ considered what the media had to say, then Arnold Shwartzeneggar would've lost by a landslide.
For the record, I voted for Gore. I'm glad Bush won. I'll vote for Bush again. Why? Because he had the gutts to obliterate enemies in Afghanistan and Iraq, but I cannot say I have the same faith in Gore if put in that situation.
Has anyone noticed that troops in Afghanistan haven't been killed lately? No. Why? Because the media only cares about making things into a train wreck.
Go ahead, mod me down you shitheads.
As others have observed, some of the excluded directories do exist.
So, someone finds a problem with blocking search engine bots.
1) First, a lot of these docs involve Iraq. So, wihtout real factual information, it's assumed they're trying to do something fishy regarding Iraq info
2) Using that assumption, the next assumption is that they're purposely trying to keep people from trying to find contradictory statements.
This could all be true, or it couldn't be. Either way, by making two assumptions without any real facts is just pathetic yellow journalism.
That Bush gave THREE reasons for invading in his SOTU speech: The NON-imminent threat of WMD (pre-emption); links to terror (see: ansar al-islam); to foster democracy in a dangerous, arse-backward region.
The last one is good enough for me, and if successful, Bush should get the Nobel. Then again, maybe they'll give it to that great peacemaker, Yasser Arafat, LOL.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
It looks to me more like just a general list of items to discourage the robots from indexing everything.
I don't see anything about the list that would make me think there was any malicious intent.
Wouldn't the wayback machine already have it for a few years of changes? (Yeah, I'm too lazy to look. It's not my White House.)
One line blog. I hear that they're called Twitters now.
He who controls the present controls the past.
"Learning is not compulsory... neither is survival."
--Dr.W.Edwards Deming
Sorry, I'm with Al Franken on him. (though Ivins is great!)
"I think he's mean. I think we're all too ready to blame Karl Rove, or Dick Cheney, or Ari Fleischer, or Gale Norton, or Donald Rumsfeld, or John Ashcroft when this administration does something despicable. When South Carolinians get push polls saying John McCain fathered an illegitimate black child, you know Karl Rove had something to do with it. But it's really Bush. When our energy policy is set by cronies from the oil, coal, and automobile industries, you can shake your fist at Dick Cheney. But it's Bush. When Ari Fleischer feeds rumors that the Clinton people vandalized the White House, doing $200,000 worth of damage, but month later a GAO report say that ain't true, you can say that Ari Fleisher is a chimp. And he is. But it's Bush."
...
"And I'm through with him."
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
"This admin has a long history of attempting to rewrite history"
Please provide examples.
Engineering and the Ultimate
1. Bush lies about Iraq to get us into a war with them
2. Bush continues to lie about Iraq
3. Bush edits Whitehouse.gov web bage so his lies are concealed (ie: can't be proven any more)
And now we're just supposed to TRUST the filthy slime at his word??
Day by day and almost minute by minute the past was brought up to date. In this way every prediction made by the Party could be shown by documentary evidence to have been correct, nor was any item of news, or any expression of opinion, which conflicted with the needs of the moment, ever allowed to remain on record. All history was a palimpsest, scraped clean and reinscribed exactly as often as was necessary. In no case would it have been possible, once the deed was done, to prove that any falsification had taken place.
When you are sure of something, you probably are wrong (search for "Unskilled and Unaware of It").
Not only does http://www.whitehouse.gov/infocus/iraq/ exist, it is also currently indexed by google.
I guess the googlebot doesn't visit the page, but knows of its existence from other pages??? Either that, or the googlebot is a bad boy that ignores robots.txt.
Your highlighting of the most absurd-looking decontextualized details overlooks the possibility that the webmonkey was told "Make sure that no search engines archive any page on the site called 'iraq'."
"Patriotism is your conviction that this country is superior to all other countries because you were born in it." -- GBS
The plurality of voting Americans, by a margin of a few hundred thousand, voted for Gore. Note that this was not the majority. Al Gore received 48% of the popular vote. It would be fair, if misleading, to say that the majority of America voted for anyone other than Al Gore. The same could be said of George W Bush.
According to records: 50,996,116 people voted for Gore. 50,456,169 voted for Bush. And 3,874,040 voted for "somebody else" of which a good portion was Nader, but a fair amount also was Buchannan, so while there MAY have been a spoiler factor introduced by Nader (and Naderites still dispute this) it's effects are too small to really shift the "mandate of the people" to anything truly significant.
These are just facts. Another set of facts to include is that the Constitution rather explicity does not have majority vote electing the president, but instead has the electoral college, and thus under the current rule of law, the plurality vote has no legal weight behind it to say who "should be" President. Historically, you can make several arguements as to why this was done, but in my opinion, certainly a large part of it had to do with retaining a distinctly state level character to the election, and to allow lower population states to contribute to the electoral decision, in similar fashion as to why we have a Senate as the upper house in our legislature.
And now, the opinion part of my post: Ultimately, my analysis is that nobody had a real mandate coming either into or out of the 2000 election, and thus any attributions of the "will of the American People(TM)" are of little real worth. I also, largely, don't believe that the economy would be substantially different today if Al Gore was in office. You can interpret this as praise/damnation/fatalism of the effects of the current administration as you see fit, but the cards for our current economic situation were in place long before the election. Note that I don't really think it was the "Clinton's fault" either.
These tax cuts are to slow down the booming economy (pre-election).
These tax cuts are to pick up the sagging economy (post-election).
Carbon dioxide is a gas that needs controlling.
We don't need to control carbon dioxide.
Some would say this would only have been true if Al Gore were president.
No one could reasonably assume that, as it was never said. The main reason said for going to war with Iraq was WMDs (of course, that is a bit embarrasing now, but that isn't the subject of your comment). However, the claim was never made by anyone in high-position that Saddam was definatly the brainchild behind 9/11. It was always said that OBL was the brainchild behind 9/11.
Bin Laden's dislike of Saddam is well known (just read the 2000 book "Osama bin Laden: The Man Who Declared War on America"). There was some early speculation about Saddam and OBL working together (there is evidence of a meeting between low-level people, but I doubt that OBL and Saddam were personally involved).
Again: the main (inaccurate) reason for going to war was WMD, not Saddam's involvement in 9/11.
You know, people that dislike anything from the GWB administration "just because" are giving a bad name to people who truely have problems with certain policies and actions of the administration. It is just like the people that hated everything Clinton did "just because." Both of those Presidents have done some incredibly good, and pretty bad, things.
Sarcasm and hyperbole are the final refuges for weak minds
some one went in and added 'iraq' to every directory just on principle, or out of laziness. or just in case. or just because. I dont see any 'afganistan' in there. Maybe they aren't as embarrassed about that one.
TallGreen CMS hosting
Good job being open minded! Let's just assume that they're hiding something, because they're doing something that might possibly be used to hide evidence! .... what does that sound like?
~/ssh slashdot.org ssh: connect to host slashdot.org port 22: too many beers
With what goal?
So... the White House publishes a ton of information on Iraq and a dozen other topics on their website. The information is available to anyone that goes to the website. And they think by disallowing it to robots (which may or may not pay attention to the robots.txt file) they're going to hide that which is already public by following a few links?
In the worst case--that what you said is true--they're not even deleting it or making it inaccessible to crawlers (which they could do with cookies to prevent deep linking), it's still a non-issue.
I'm sorry... that dog just don't hunt.
Hey, I'm a leftist bleeding-heart liberial staunch Democrat-type who almost cried when Gore conceeded, but even I'd be the first to admit, that we don't live in a Democracy. We live in a Republic.
I want peace on earth and goodwill toward man.
We are the United States Government! We don't do that sort of thing.
By the way, who is going around looking at people's robots.txt files?
why you're absolutely right! how silly of us. what a waste of time. maybe we should be looking for crusty stains on the intern's dress instead. that would be a better use of resources, don't you think?
Could somebody please cache this, and maybe put it online, if needed at a European site. This is just too freaky. Thank you /. for reporting.
--- Sigmentation Fault - Comments Dumped
Many people wonder why they did not just remove the sites from the webserver. The answer is not hard to come by if you think about it for a second.
... and last time i checked the archive only indexes sites by url, so searching is much harder than it is on google.
Just imagine what would happen if they removed the pages and a similar story comes up on slashdot. Well, everybody will of course go to the google caches of the removed pages and say "aha you are caugh red handed!!!".
But google caches last only a limited amount of time. And once a page is placed in the robots.txt file, google stops caching it. Well guess what, i bet that right about the time the google caches expire, the webpages will disappear from the website as well.
Now it will be much harder for the ordinary person to check exactly which webpages are missing. Of course there is the internet archive, but not many people know about the internet archive
> > American people should have some say in a situation like went on in Iraq.
> They do, it's called voting, not to mention public opinion polls, which were near 70% for the invasion when the US invaded.
As I recall it, the last poll before the shooting started showed 60% support with a UN resolution in support of the invasion, but only 40% otherwise.
When the shooting started the "support our troops" meme merged with the "my country right or wrong" meme, and then you got overwhelming popular support.
But why would Goober care what the polls showed? We already know that his staff "helpfully" screens the news for him, and the Secret Service show similar enthusiasm to ensure that no protesters are in sight when the presidential motorcade passes by.
Sheesh, evil *and* a jerk. -- Jade
Umm... it's on the tip of my tongue... what was that name... begins with a 'G'... oh, Google, that's right, Google. Supposed to be some kind of search engine or something.
Mod down people who tell people how to mod in their sigs
Last year - Iraq going to acquire WMD imminently
This year - We never said that
Election - Pledge that there will not be a deficit
After huge deficit - Bush claims that the pledge was conditional, only he never said it.
Claim - SEC cleared Bush of corruption at Harken
Fact - It didn't, the letter in question said it didn't
And so on. Even if we accept the Bush claim that he did actually speak to the unnamed journalist about conditions on the no-deficit pledge it hardly excuses him. He knew the pledge was reported unconditionally. In any case the claim is ridiculous, all candidate contact with the media is monitored and minuted. In most cases recorded too.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
Please explain to me why Casper the friendly ghost did not play a role in 9/11? Is there evidence that says he did not?
I think you are all overly paranoid. .I'm sure its got more to do with reducing traffic on commonly searched words then anything else.
If they WERE trying to hide things, it sure as hell wouldn't be out in the open...
---- Booth was a patriot ----
Yes, that was the point of the allusion. Maybe I should have mentioned Jimmuh Carter too, or linked to Yasser's award page.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
I wonder if slashdotting the whitehouse.gov robots.txt is considered a terrorist act. At least we'll all get to meet each other at the US Government sponsored "First Annual Slashdot Terrorism Conference" get together at guantanamo as we all are held as enemy combatants..
Invalid Checksum. Retrying.
The day before California elections. CNN news. Arnold all over them. Sound bytes, clips ("Terminate Gray Davis!"), commentaries. I waited for the part where they show sound bytes and clips from the other contenders. There were zero. If you only watched CNN, you would think Arnold was running fucking solo.
"Only the small secrets need to be protected. The big ones are kept secret by public incredulity." - Marshall McLuhan
> If there was any real evidence of wrongdoing in the 2000 election, you can bet your life that there would be quite a few public investigations of the fact at the time, especially by the Democrat party (and in Congress too).
Nobody denies that the State of Florida counted lots of votes from overseas military stations which were not received by the legal deadline and/or which did not meet various other legal requirements.
The fact that the Democrats were too gutless to contest those votes does not mean that nothing was amiss.
> Of course it would have been too late to change the final results, but if there was a shred of truth to that, the media would eat it up and the Bush whitehouse would be even more disliked than the Nixon administration in its late months.
After 9/11 the media gave Goober a free pass on damn near everything, right up until the Iraq adventure. For the most part, he's still getting a free pass.
Sheesh, evil *and* a jerk. -- Jade
> For the record, I voted for Gore. I'm glad Bush won. I'll vote for Bush again. Why? Because he had the gutts to obliterate enemies in Afghanistan and Iraq, but I cannot say I have the same faith in Gore if put in that situation.
Bush has made us more enemies than he has obliterated.
> Has anyone noticed that troops in Afghanistan haven't been killed lately? No. Why? Because the media only cares about making things into a train wreck.
Have you noticed that the Taliban is getting re-organized in Afghanistan, and that there are actually more pitched battles there now than there were a year ago?
Sheesh, evil *and* a jerk. -- Jade
What "incredibly good" has Bush done? For the record. And assuming that in order to be "incredibly good", it has to positively affect a large part of the population in a measurable way.
"Only the small secrets need to be protected. The big ones are kept secret by public incredulity." - Marshall McLuhan
Correct me if I am wrong but the data is still there right? Also, wasn't the purpose of robots.txt(that honor it) to stop crawlers from incessantly crawlign the page sapping your bandwidth? I just don't feel that this is a big issue. If they made it not searchable from the main whitehouse page, thats when I would have issues. They are just trying to save themselves bandwidth. Pages like these Iraq pages are peobably updated often. They'd be getting crawled constantly.
Gorkman
I really do wonder what brings people to zealously defend actions like this. Sure, it could be a mix up, but a really ill conceived one. It's obvious that you don't have all the answers, just like others here.
My guess is that the poster feels that Slashdot posters are simply leaping to unjustified paranoid conclusions, and the depth of this faith (or so he pictures it) outrages him (or her).
The intensity of the poster's reaction is simply a reflection of his or her perception of Slashdot readers' zeal.
There are many possible explanations which do not involve conspiracy to hide information. For example, this could just be the work of some low-level IT guy who wanted to filter out one URL which happened to contain 'iraq' because the search-engine robots were burdensome to the webserver. I, for one, prefer to remain suspicious.
There are a lot of missing dates, but it looks to me like whitehouse.gov had a major site redesign sometime between Jul 13 and Sep 13 2001, and that when the new site was released they started putting in lots of the disallow statments for certain paths.
From Jul 13:
7-13 Whitehouse.gov
7-13 Robots.txt
From Sep 13:
9-13 Whitehouse.gov
9-13 Robots.txt
It seems to me like the simplest explanation is just that their redesigned site has multiple paths to the same information, and for some reason they felt that their search engine rankings would improve if they eliminated superfluous paths. Although I'll admit it's suspicious that their old robots.txt from 2 years ago had 151 Disallows, and the one from today has 1552 Disallows, while the site uses basically the same navigation structure.
Not true. Some of them do exist, like this one: /climatechangefactsheet/text
"Only the small secrets need to be protected. The big ones are kept secret by public incredulity." - Marshall McLuhan
Other posters have claimed it's more than one. I haven't checked, so I don't know. However, even if it is just infocus/iraq, that's still a hell of a lot.
e pt26.html
That subdirectory seems to contain all or most of the transcripts of Ari Fleischer's and Bush's interviews and press conferences leading up to the war and after. An example is this:
http://www.whitehouse.gov/infocus/iraq/excerpts_s
They don't seem to be blocking archive.org.
Bullshit.
The Iraq entries could only have got there if someone was told to go and stop stories appearing in the Google cache.
The person who got the job appears to have done it in a pretty clumsy way, that is pretty much par for the course for this type of work. Nixon did not expect Gordon Liddy and his pals to get caught in a third rate burgalry either.
It looks to me like someone was told to block out the Iraq files and simply did a directory listing on the web server and then appended /iraq to everything.
If you want to find out for sure file some FOIAs.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
Me neither. Most of those links don't exist, or even make sense. Looks like a script gone ary?
"Ignorance more frequently begets confidence than does knowledge"
- Charles Darwin
I think outrage is strong. I just looked through the thread, as he suggested, and only saw one that actually existed.
Never confuse volume with power.
Is their obfuscating content in any way on a public government website illegal?
Dude, where's my packet?
whitehouse.com > whitehouse.gov
Read Molly Ivins book, _Bushwhacked_.
Warning: If you're a Bush fan, be prepared for horror and disbelief.
Your media are broken.
According to the PBS TV show NOW, your Air Force was supposed to check on aircraft off of their flight paths, and that in 2000 they had in fact escorted several planes back to where they belong. (See here.
BREITWEISER: On the morning of September 11th we had four planes drastically off their flight path transponders disconnected and the FAA procedure and protocol to notify NORAD and for NORAD to scramble fighter jets were not followed. And it wasn't like they all happened in the course of an hour. What I think is very frustrating is looking back when I speak to people they say, "Well it happened in such a short span of time."
It did not happen. It happened over the course of two hours. You're telling me over the course of two hours Andrews Air Force Base in the Washington, DC area which houses F-16s which fly cover for Air Force One could not get a plane up in the air to cover the Pentagon?
Why hasn't this hit your mass media? It's incomprehensible.
Just like journalists were all over the Valerie Plame leak, oh, several months after the independent media was up in arms over it.
The LA Weekly has story on the deliberate "spin" this administration is using. Amusing, but scary.
Can I bum a sig?
my first thoughts exactly...
Someone mirror the website at daily intervals. Have the results indexed on a separate website. This is a matter of public information policy. It is imperative that the public have access to these materials.
archive.org retroactively honours robots.txt exclusions. Could they perhaps be redacting old stuff from there?
Slow news day? This must be a troll, but I'll bite.
You don't consider this enough to "chew on to attack Bush"
except:
"At least 27 were killed in the police station bombings, including one US soldier, and 12 were killed at the Red Cross. Many of the dead were bystanders."
Is that you Mr. O'Reilly?
ymmv
This is all totally offtopic but, you are right that the electoral college is who ultimately decides who is elected president, but the parent poster makes the mistake of referring to the decision of the electoral college as "the Will of the American People" or some BS like that. When, in fact, that is not the case. The points about the economy though, while I admit many of the factors that contributed to our economic downturn were outside of the political realm of the presidency, all of the factors that were within that realm are directly related to Bush Administration decisions. Their ridiculous tax cuts for the super rich were the largest in the history of our nation. And this happened while we were having a rather significant little budget deficit. And what net gain did we all get out it? Well, some of us got a few dollars back on our taxes, at the cost of a booming economy and a whole lot of services that the current administration doesn't think are important like you know, unemployment. And please don't start with some welfare bit, because we spend exponentially more money on corporate welfare than we do on unwed single mothers trying to feed their fucking kids. Obviously if corporations can only afford to pay their CEO's a 5-25 million dollar salaries, they must need some federal money to help out:) No, the Bush administration sold our "poor" asses down the river in exchange for a fattening of their investments. Name one person in the cabinet who is not a multimillionaire. How many slashdotters are multimillionaires? How is this a government of the people by the people?
every time a republican dies a queer angel gets his wings
Might it have something to do with people abusing google images to come up with pictures of the Iraqi information minister, Arafat, Osama's sons, et. al to pass around in stupid email messages and politically minded weblogs?
Dead links in search engine? eetc.etc. etc.
Sounds like a reaction to an onslaught of abuse of the site to me (whether unintentional, purposeful, DdoS, or systemic)
Fuck Beta. Fuck Dice
story at washingtontimes.com
woo hoo! Anyone else have this problem? I think it's funny because I have such a common name.
we are all one consciousness experiencing itself subjectively - bill hicks
Or sarcasm, but I forgot that goes right over the head of people here at Slashdot.
So most are 404s, some are videos, and you assume others have mundane reasons. What about the ones with real content? Likee xt/20030501-15.html
r aq/20030501-15.html
/kids/eggroll/barney/iraq/DoNtInDeX/oldspeak/nosex withthatwoman.txt "Technically", it is still on the publically addressable web page, anyone could look at it, if they knew the obfuscated secret.
http://www.whitehouse.gov/news/releases/2003/05/t
which differs from http://www.whitehouse.gov/news/releases/2003/05/i
In the text version, the pages says 'President Bush Announces Combat Operations in Iraq Have Ended' while in the robot accessible version, it is ''President Bush Announces Major Combat Operations in Iraq Have Ended'.
There are perfectly good error codes for Gone (410), moved temporarily (302), moved permanently (301), and a host of other codes for more mundane reasons.
The question that the tin-foil-hat crowd wants answered is where does the content go that doesn't exist anymore? Did they ship it over to Ashcroft's boys and delete it off the server? Or move it off under
Making the robots.txt file 'accidently' inhibit robots makes the data more inconvenient to access, not impossible. So "Technically", it is still accessible, but instead of using google, you'd have to use the white house search tool instead.
If you trusted them before, you will probably keep trusting them. If you were suspicious, this is another 'mistakes were made' brick in the wall to wonder about.
As for me, the one-word difference in the two headlines above makes me suspicious.
The quote is a lie, a bunch of trash pulled out by someone not educated in our sphere of law. "Guilty until proven innocent" is an exact translation of this quote's intent.
OK. TO correct your correction. The reason for going to war was to enforce UN Resolution 1441 which called for Saddam to declare and dismantle his WMD program. The Security council unanimously approved 1441 and unanimously declared Saddam in violation of 1441. The US, Britain, Poland, Austrialia and a few other Eastern European nations were the only countries with the backbone to actually enforce the resolution.
BTW, the Kay report shows that Saddam had a massive program in place. They discovered that he could have ramped the nation up to the production of several tons of biochemical agents per month a few months after the word was given.
Weapons have not been discovered as of yet, but there definitely was an extensive program to manufacture. In clear violation of UN resolution 1441.
You can tell a great deal about the character of a man by observing those who hate him.
See:e xt/20030501-15.html
r aq/20030501-15.html
http://www.whitehouse.gov/news/releases/2003/05/t
which differs from
http://www.whitehouse.gov/news/releases/2003/05/i
In the text version, the pages says 'President Bush Announces Combat Operations in Iraq Have Ended' while in the robot accessible version, it is ''President Bush Announces Major Combat Operations in Iraq Have Ended'.
Get your own screenshots.
I'm so sorry I expended my mod points earlier in the day. What a bunch of flamebait bullshit this line of crap is. "Dictatorship?" Get fucking real. Let me ask this in non-partisan terms:
If the fiasco that was the 2000 presidential election went in Gore's favor, would you care to label his administration a dictatorship?
Has martial law been declared?
Are SS agents en route to your residence right now to conduct a little Q&A over this post?
Snap the fuck out of it. While I completely disagree with this appraisal of the Bush administration, I can (barely) live with you posting it. Just don't such nonsense to go unanswered and undebunked by me.
Um, robots.
/firstlady/iraq thing.
And boy are they pissed off about this
My amazing wife - Artist, Author, Philosopher - Laurie M
Would the White House sue for violation of the robots.txt file? Under what laws could they sue? Is robots.txt an implicit grant of permission to view copyrighted content? Would GWB press the Congress for a new bill, to mandate legal enforcement of the robots.txt?
That's probably not going to happen anytime soon, but it raises an interesting question. Is robots.txt legally enforceable? And if it was, would that be a good thing or a bad thing?
Your thoughts?
So often, I read, see, or hear this line of thinking that Bush and his administration, namely Attorney General Ashcroft, are dictators, fascists, or the second coming of Nazism.
My problem with this is that labels are not enough. These are serious remarks. Serious enough to give a kuro5hin poster some quality time with Secret Service agents. The point: if you're going to call someone a Nazi or dictator or what have you, you had better be prepared to go all the way. Produce a swastika armband with GWB initials on it. Write an annotated essay that leaves no doubt to your bold, if not irrational conjectures.
This is why you and I do something else for a living. We know shit as it relates to politics. Say it with me. IANAP. I Am Not A Politician. If Bush were a dictator, there'd be a hellstorm from conservatives as well as liberals, or there'd be no hellstorm at all. I see from your post and mine, that this is not the case, Bush is not outlining his plans for the Fourth Reich, and the sun will rise tomorrow. Please get a grip and stop intellectualizing our scheduled re-education and the reincarnation of George Orwell. Stop.
It's a weak administration thats afraid of it's own words.
-------- -------- Support Wesley Clark for president!!!
See:e xt/20030501-15.html
r aq/20030501-15.html
http://www.whitehouse.gov/news/releases/2003/05/t
which differs from
http://www.whitehouse.gov/news/releases/2003/05/i
In the text version, the pages says 'President Bush Announces Combat Operations in Iraq Have Ended' while in the robot accessible version, it is ''President Bush Announces Major Combat Operations in Iraq Have Ended'.
Get your own screenshots.
Check the link in my journal entry, the junk filter won't let me post it here.
Here
http://apple.slashdot.org/~zoloto/journal/50343
No, they're going to obscure future changes from all but the most sophisticated (and generally media-inaccessible) users.
Everyone else will just see today's approved version of the message.
"Patriotism is your conviction that this country is superior to all other countries because you were born in it." -- GBS
Wow, a comparison between 1984 and the present-day administration whic you don't like! That truly is insightful; mods, give this man more points!
If it ain't broke, you need more software.
When two senators attempted to boo him from the gallery, they were requested to leave by the speaker of the house!
From what I hear, the entire visit got a brief mention on Fox news. The silly bugger even brought his own non-alchoholic US brand of beer with him!
By the way, did you US guys know that you won against Japan in the World Cup Rugby?
thank you. i knew there was something wrong with that quote but yours is the most sussinct distillation of all the arguments i've heard.
It's actually very simple, they didn't want the whitehouse to start showing up on google under the search term Iraq...
Because that would lead people to believe that the whitehouse was the official website for the Iraqi government, and that George Bush did indeed conquer that country.
It all boils down to an image thing... the whitehouse is usually really good at hiding their conspiracies.
Pardon me, but some of them do lead to interesting things. /news/releases/2003/05/iraq/ exists, and even contains different data than
e xt/20030501-15.html versus http://www.whitehouse.gov/news/releases/2003/05/ir aq/20030501-15.html and http://www.whitehouse.gov/robots.txt has /news/releases/2003/05/iraq/ in it.
news/releases/2003/05/text/ or news/releases/2003/05/
See for yourself:
http://www.whitehouse.gov/news/releases/2003/05/t
Compare the headlines.
American people should have some say in a situation like went on in Iraq.
Maybe you don't like what the people said, why they said it, or that they aren't as smart as you liberal elites, but they overwhelmingly supported the war and the troops. Use whatever CBS News push-poll you want. After Bush's SOTU, the pro-war numbers - asked straight out, "do you support the war" - according to most polls I read were in the high-60's.
Calling a large majority a "large plurality" seems little disingenous. Since when is a number over half a plurality?
And it seems to me the US will be paying an extremely "large plurality," to use your words, of the bill.
Like 95%.
If 'war is not the answer', I'd sure as hell like to hear what the answer is.
Slashdot "libertarians": Small government for me, big government for those I disagree with. -1, I disagree with you
Slashdot's "anonymous" IP hashes can be undone in 2^32 steps.
autopr0n is like, down and stuff.
http://www.whitehouse.gov/infocus/iraq
Not any more.
Although the current Google cache lists
[snip 22 lines]
the current robots.txt leaps from
to
Conspiracy theory over...
Referring to a website critical of him (but correct in every detail)
And why do you think there is a DISALLOW on /iraq
in the first place? They have nothing to hide/distort, right? I am sure that you nor anyone else will fall for the talkingrealfastattheendofthecaradvertisementsoyoud on'treallycatchthegotchas and go away remembering "Wow! Only $199/mo!" Technically they are speaking the truth, but they are not really motivated for your benefit, eh?
Cheers,
e.
In any case, here's a plausible explanation. First thing to do is note that almost all the entries are duplicates, ending in either /text or /iraq; that many of the /iraq entries are 404s and would seem to be ridiculous anyway, but that most or all of the /text entries work, and lead to the text-only version of the site.
Reason: So that only one version of the site gets indexed, and the text-only version doesn't compete with the full-graphics version for ranking. Perfectly legit.
Stupid? Yes.
Nefarious? Probably not.
wow... that's a MAJOR DISCOVERY... I know the titles are only off by one word but it is significant. The press releases aren't supposed to be different. The difference is especially important given the controversy swirling the end of "major combat operations" and the casulties.
Does anyone know how to automatically compare HTML and text to see the differences (no, I don't want to program or code such a thing)? Someone should attempt to see if there is any other difference.
BTW, nice discovery...
Sivaram Velauthapillai
Sivaram Velauthapillai
Seeking the meaning of life... @slashdot of all places
you're full of shit, you're a liberal
If Google spiders WhiteHouse.gov and then--under your theory--they decide to change something, Google will just recrawl the new version thus making the new version part of Google's copy of the page. If they disallow it with robots.txt either Google drops it completely or the old version remains in their database. Either way, disallowing pages that have already been cached is counterproductive.
Plus Google doesn't build a history of all changes to documents. It just keeps the latest version. If what you're worried about is them making changes after they are published then Google isn't going to help you other than during the few days between the website changing and Google recrawling.
If you are suspicious of the White House changing information previously published on their website--which is silly in itself since you can certainly find copies of everything they publish elsewhere, including the Federal Register--just monitor it with a robot that ignores robots.txt. It's that easy.
To suggest this is some intentional conspiracy or cover-up when the means to "circumvent" it are so trivial is absurd.
Tell you what... If you find evidence that the White House is intentionally changing material previously posted on their website with the intent to mislead someone, bring that to Slashdot and we'll talk about it. Bring me the Disallows of a robots.txt and I'll continue to consider you comic relief in the absence of George Carlin.
1. Google doesn't spider every day. There can be a substantial gap between when a site is changed and when Google's cache is updated.
2. www.archive.org, not google.
"Patriotism is your conviction that this country is superior to all other countries because you were born in it." -- GBS
Or the robots.txt file was updated since the last time google crawled the web.
"We have got to make Stan understand the importance of voting, because he'll definitely vote for our guy." - South Park
So? All the more reason to ALLOW Google to spider those directories so that they get "changed" to reflect the "new" version ASAP.
2. www.archive.org, not google.
Again, so? If their function is vigilance of U.S. Government websites to verify that the Administration isn't modifying their previous statements then by all means they should ignore robots.txt.
It simply is a non-issue. Robots.txt is not required by law and many spiders don't even pay attention to it. So to act like this is some kind of force shield that prevents companies or individuals from downloading the entire whitehouse.gov each night and check for differences is silly.
Believe me, everything the Administration has said is duly noted by news media around the world. You are all fooling yourself if you think the Administration can rewrite history by updating a webpage.
Uhm... There ISN'T a disallow on /iraq at all. None. Zip. Check the robots.txt file yourself.
They have nothing to hide/distort, right?
How would I know? Perhaps they do. But if they have something to hide they are NOT going to hide it with robots.txt. That's just silly.
This is specious. Their function is to provide a record of what web sites have said at various points in the past, and people therefore rely on it for that purpose. Until an institution comes into existence to specifically monitor the course of changes to the whitehouse.gov web site, that's all we've got. Archive.org has voluntarily agreed to respect robots.txt, and this can be taken advantage of for duplicitous purposes. I'm not saying it has, only that it can. This is a speculative discussion.
They don't have to. They just have to make it hard enough for the lazy people to find the truth, and they will have done enough to make it worth their trouble.
It would be just like what Arafat does when he says one thing in Arabic and another contradictory thing in English. Any journalist who cared enough could get a translator or learn Arabic and sort this out. But almost all the time they're too lazy to bother, and each language's press only reports on the message delivered in that language.
"Patriotism is your conviction that this country is superior to all other countries because you were born in it." -- GBS
I'm going to use a bit of real irony here.
When's the last time the media has questioned the government and/or the establishment? Ever since the war in Iraq has started, every report on the country has been glowing. You'd think that the whole country rolled out red carpets to greet the soldiers, who have encountered little to no resistance in a country where they have always loved freedom and the American way.
Let's turn our attention to the latest political topic du jour -- the partial birth abortion ban. Why, once Bush penned his signature on that beauty, all the media reviews and all public opinion has been absolutely glowing, has it not? It is great to have a government that seems to always do things that satisfy everyone simultaneously.
And thinking of which, the last time I watched Hannity & Colmes, I couldn't help but notice how boring the show is because everyone is always agreeing with each other. I mean, take this direct quote from last night's show:
Hannity: I sure loved what the President did yesterday! He is the greatest guy since Jesus! HEIL BUSH!
Colmes: I'll have to disagree with you there, Sean. I think Bush is even greater than Jesus! LONG LIVE THE PRESIDENT!
Howard Dean (a guest on the show): I'm running for president, but I have to agree with you folks. HALLELUJAH that we have Bush for president!
We live in a wonderful world where everyone agrees with each other every day, and every newspaper article, TV news report, and radio discussion is completely positive concerning the current establishment and their decisions.
That's real irony for you.
Note to people who do not know what real irony is, here is a quick definition.
A sort of humor, ridicule, or light sarcasm, which adopts
a mode of speech the meaning of which is contrary to the
literal sense of the words.
The radical sect of Islam would either see you dead or "reverted" to Islam.
Unfortunately, I'd be surprised if they're not doing a lot of log analysis to find out who's reading what parts of the site, to look for political opponents of various sorts and other patterns that could be useful.
;)
Political opponents wouldn't be visiting the white house website--ever. That's almost the definition of a political opponent, isn't it?
Sivaram Velauthapillai
Sivaram Velauthapillai
Seeking the meaning of life... @slashdot of all places
Finally the sordid truth of secret service work is revealed.
"It is a solemn thought: dead, the noblest man's meat is inferior to pork."
Hm, so this is
a) An attempt to hide public information
b) Implemented in the most naive possible way
c) Totally useless at (a)
Something tells me Dubya doubles as the White House sysadmin . . .
How many uncommunicative incompetents does it take to run an administration into the ground?
e xt/20030501-15.html versus http://www.whitehouse.gov/news/releases/2003/05/ir aq/20030501-15.html and robots.txt has /news/releases/2003/05/text/ in it.
With your theory, them web folks did a bad job -- On the pages released by the office of the Press Secretary May 1, 2003 they failed to change both of the pages. Of course the embarassing one of them is hidden from the polite search engines through the very robots.txt file we are all talking about.
See for yourself:
http://www.whitehouse.gov/news/releases/2003/05/t
Compare the headlines.
So tell me, how many mistakes were made, and by who? Do the bucks stop everywhere they get a chance in this administration?
Correction - It's his grandfather's and Hitler's money.
Prescot S Bush has his assets frozen under Trading with the Enemy legislation.
Get the Hell off my planet, you slimy mobster Bush!
http://www.whitehouse.gov/kids/india/
Maybe he isn't all bad after all.
That's a pretty kewl looking cat.
This signature used to contain a cute kitty virus with ansii art. Please set the slashdot editors on fire. Thank you
You know, Rod Roddy from The Price is Right is also dead, at 66.
We at 2600 actually called the White House and did a bit of resource. The article detailing our findings is online.
A few weeks ago I heard that both our Fearless Leader and Condoleezza Rice came out to the press saying that there are no connections between Hussein and 911, and that they have never implied otherwise.
I tried to find some sort of article on this but too much other stuff came up when I tried to search for 911, Saddam, not connected whatever. (Gee, I wonder why?)
Anyway, does anyone know of any mainstream articles on this announcement?
(The Memory Hole looks very interesting, I need to check it out.)
This signature used to contain a cute kitty virus with ansii art. Please set the slashdot editors on fire. Thank you
windside writes "Rex Murphy is reporting on the Canadian Prime Minister's website's use of its robots.txt file to disable search engines from crawling certain material. Many excluded items in the robots.txt file involve mentions of America, possibly to prevent people from finding out that taxes are much lower, that money is spent on government programs instead of on kick-ass jets for parliamentarians and that their senate actually does stuff." It seems Canadian officials could not be reached for because they were all busy taking bribes from their favourite soul-devouring oil company.
Note: Remember, Canadians may look nice, but we're mostly just as corrupt and evil as the Americans.
...Whether my Maker is prepared for the great ordeal of meeting me is another matter.
Churchill
We live in a wonderful world where everyone agrees with each other every day, and every newspaper article, TV news report, and radio discussion is completely positive concerning the current establishment and their decisions.
SHUTTUP YOU STUPID LIBRAL!!!!!!!
GO BACK TO YOUR CAVE IN AFGHANISTA!!!!!!!!
YOU'RE EITHER WITH THE PRESIDENT, OR WITH THE TERRORISTS HIS FATHER HELPED SETTUP!!!!!!!!
Either that, or they do nothing but yell rhetoric and cliches at you. I'm not actually sure which I prefer. The later at least is more entertaining, but not by my much.
This signature used to contain a cute kitty virus with ansii art. Please set the slashdot editors on fire. Thank you
Frankly anyone bringing up 1441 is an ass.
OMG - The text one was last updated on September 11, 2003.
The iraq one: May 2, 2003.
Get the Hell off my planet, you slimy mobster Bush!
you're full of shit, you're a liberal
Yea! Anyone who we disagree with is obviously a Liberal! Grrrrr!!!!!! Fscking Liberals!
Even my father who is a lifetime Republican, and a Marine Corps Officer is now painted a Liberal because he dares to question our fearless leader.
The USA is the greatest country in the world because our politicians are incapable of making mistakes when it comes to foreign policy.
This signature used to contain a cute kitty virus with ansii art. Please set the slashdot editors on fire. Thank you
It is frightening that a joke about Bill C's peccadillos should get modded up so quickly and another more relevant comment about 1984 not.
See my journal, I write things there
All of a sudden this is no longer a place for nice, compassionate nerds; no - now this is a place where all the rats come out of their holes to bring hate and anger. And they are not asking for discussion; as I said; you are with them or you are a lousy stinkin' something. Of course, that is the real life ... but is what we need
here?
The (aggresive) liberal view is easily found at places like www.bushflash.com; the opposite might be found at places like littlegreenfootballs.com/weblog/ - really no need in my eyes to bring slashdot into this league of extraordinary hotheads ...
There's a million posts I could have placed this response to. Whatever.
/text/ are disallowed. These are the print-versions of other pages and ought to be disallowed to allow good searching.
/text/ with /iraq/. These pages were not hidden from the whitehouse search engine or from humans. In fact, most of these pages didn't exist.
It's NOT a conspiacy. It's a mistake. Many pages that end in
The robots.txt (now corrected), had basically duplicated the list and then replaced
Look, I vote democrat and lean green. But this FUD is ridiculous.
"real, detailed, complete examination . . . is out of the question."
...
May I add, "on any complex or technical issue"? Come on, my local paper can't even publish a mostly accurate introduction to digital cameras.
You expect them to give a "real, detailed, complete examination" of the possible causes and results of some changes to a file for which the vast majority of the audience doesn't know the purpose of and didn't even know existed. Hah! You're the one being naive. It's not about politics; The Nation is (most likely) not going to pick this up either.
The most pernicious media bias is neither liberal nor conservative. It is the tendency to misrepresent reality so as to boost ratings and make their jobs easier. Thus, news coverage is disproportionately about bad news because good news is more complex and less dramatic. Also, in the US, there is more television news coverage of kittens and puppies than the majority of countries. (I didn't look up the kitten/puppy coverage versus coverage of other countries, but that's my impression.)
Media by the shallow, pandering to the shallow.
By the way, I think a large part of why Chomsky hasn't been on CNN is his work is not very suited to sound-bites.
Or maybe that's just what the vast right-wing conspiracy wants you to think
The link to the president's letter is also in an earlier post.
What I think is reallly weird is that according to the archive (http://www.whitehouse.gov/infocus/iraq/iraq_archi ve.html) Bush didn't make any 'presidential remarks' from august 2001 till januari 2002.
If you look at the State of The Union page of Jan 29 2002 you can see that one of the links in the robots.txt is actually used their and is indeed 404-ed
These guys are counting.
my password really is 'stinkypants'
what he said is pretty much a copy & paste from damn near any liberal group out there, with a very few changes made look around you'll see it.... did i say there were no mistakes? nope not at all, do i think things can be done better? of course...
Yeah, today we know that they are hiding something. I guess tomorrow they hope for us to "know" that this never happened and that we've always been allied with the Iraqi government and people. Doublethink?
Theres is a pretty good attempt at Doublespeak right?
Most of the attacks to date have been either
1. Remnants of the Fedayeen Saddam who have everything to (re)gain if Saddam got back in power, or
2. FOREIGN fighters trying to destabilize Iraq for their OWN selfish interests.
Most of the normal Iraqi citizens, especially the oppressed Shiite MAJORITY, appreciate being freed from the tyranny of Saddam. If anything, the Americans are the freedom fighters.
There is a difference between inability to question, and unwillingness to question.
Unwillingness becomes inability if the rules are "if you question, you're fired and you have to go live on the streets."
Will I retire or break 10K?
I guess the commie pinko Socialist comment was asking for it...
My OP was about Diebold's voting machines. I would hope that anyone who believes in democracy - Republocrat or Demopublican - would want a system that isn't just a "black box".
And if you read anything about Afghanistan, you'd know it has reverted to being ruled by warlords, and that opium production has gone up ten fold since the days of the Taliban. Women aren't any safer to walk the streets; if anything they are less safe.
The media only cares about MONEY. Duh. Any liberal calling the press a government tool during the march to war is an idiot. Any conservative calling the press liberal since is an idiot.
IMHO, if you think the media just wants to paint things as a train wreck that is because the TRUTH stands in stark contrast to the shit spewing from the White House.
You haven't been modded down yet, so someone agrees with you.
Syrians are native to Iraq. I did not know that. You stupid [private part].
In fact, the dominant monotheistic religions (Christianity and Islam) teach that the human race began in the Garden of Eden. The Bible places Eden between the Tigris river and the Euphrates river, that is, in what is now Iraq.
Will I retire or break 10K?
Slashdot... where truth and logic never get in the way of liberal preconceptions...
No, look again. All three are the same.
The only difference is that one is encapsulated in the "Operation Iraqi Freedom" style, and the other has the standard "I'm the whitehouse.gov site" style. The information contained (except for the links in the container setting) are identical.
Am I the only one who bothered to actually look at those links, rather than just quickly click on them?
At the moment Google still finds to the page you linked to. While I despise those plutocrats as much as the next guy, be very careful before accusing them of something just because they have a history of being vile. As critics of Bush we must be thorough, reasonable, and sceptical.
To stem any confusion: Barney is the name of the Bush dog. This may be widely known, but I don't keep up with the media and I thought you were talking about the purple dinosaur.
Just ignore it. That's a little bit of civil disobediance I can live with.
User-agent: * /
Disallow:
So much for open government.
I suppose they just carry around Syrian passports in case they want to go vacationing?
--- I wish I could hear the soundtrack to my life. That way I'd know when to duck.
What the hell are you talking about? American soldiers are dying there almost everyday. If you're an american, at least show some respect for that. If you're not, then you should stay out of the conversation because you aren't in america and then only viewpoint you probably see is from watching the war on TV.
All implications and conspiracy theories aside...
The new standard for robots.txt files (in place for about 2+ years now, IIRC) allows you to disallow the lot and then specify exceptions.
Example:
http://www.tesco.com/robots.txt
Benefit? You don't end up giving people a handy shopping list of places you don't want them poking around.
So what strikes me most about this little discovery is that whoever wrote this robots.txt file is a complete doofus.
Oh, and Oceania is at war with Eastasia. Oceania has always been at war with Eastasia.
Tim
http://www.bloggerheads.com/
What do YOU call that system of government, then? Just wondering, because "dictatorship" seems the most historically applicable.
Leaders who block access to information which was created by them, need to be replaced. Come on, if they can't even trust the information they are distributing, then they are useless. Or hiding something even more scary then just the rubish they are spouting.
A real patriot is the fellow who gets a parking ticket and rejoices that the system works. -Bill Vaughan
Better then, to simply hold the documents in obscurity from the prying eyes of the curious using search engines to seek out this documentation by using a robots txt file
They could simply remove the web page I guess? It's even more effective you know...
I'd rather be sailing...
And back to you....
e xt/20030501-15.html missing the word "Major" in the title, metadata and the headline, which are precisely what the search engines are most interested in.
I did look again.
The headlines still differ, with
http://www.whitehouse.gov/news/releases/2003/05/t
The president's speech, however, is exactly the same on both pages, so, maybe "technically" they are the same.
Crimethink, Goldstein. Crimethink.
You know, they're called the Red *CRESCENT* over there. The ICRC is actually quite cognizant of the religious sensitivities in the area and they paint a large Red CRESCENT on their vehicles there.
(Even the terrorists knew this as they ALSO painted a large Red CRESCENT on the bomb delivery truck.)
--- I wish I could hear the soundtrack to my life. That way I'd know when to duck.