Britain's Conservatives Scrub Speeches from the Internet

An anonymous reader writes news of an attempt to erase a bit of history. From the article: "The Conservative Party have attempted to delete all their speeches and press releases online from the past 10 years, including one in which David Cameron promises to use the Internet to make politicians 'more accountable'. The Tory party have deleted the backlog of speeches from the main website and the Internet Archive — which aims to make a permanent record of websites and their content — between 2000 and May 2010."

Comments

Where's the torrent file?

[Anonymous Coward]

Where's the torrent file?

Re:Where's the torrent file?

[Joce640k]

I dunno, but I'm guessing none of these politicians have ever heard of the Streisand Effect.

Re:Where's the torrent file?

[Jeremiah Cornelius]

I dunno, but I'm guessing none of these politicians have ever heard of the Streisand Effect.

I dunno, but I'm guessing none of these politicians have ever heard of 1984.

Re:Where's the torrent file?

[HiThere]

They have. That's why they're all doing it at once. That way nobody in particular is noticed. And, yes, copies will be made and kept, but they won't be found by search engines, and they can be plausibly denied.

Re:Where's the torrent file?

[d3m0nCr4t]

I dunno, but I'm guessing none of these politicians have ever heard of the Streisand Effect.

I dunno, but I'm guessing none of these politicians have ever heard of 1984.

Oh they have, but instead of feeling appalled, they just get a hard-on.

Re:Where's the torrent file?

[erikkemperman]

I dunno, but I'm guessing none of these politicians have ever heard of the Streisand Effect.

I dunno, but I'm guessing none of these politicians have ever heard of 1984.

I am pretty sure they have heard of it. In fact it's almost as if they use it for inspiration. That, and all Kafka, of course.

Re:Where's the torrent file?

[LifesABeach]

I thought that America's Tea Party were over the edge, I think they've been one upped by the Torrie's. This is a time when we need WikiLeaks, could someone forward a message to Julian, to "get back to work."

As a side note. Maybe someone at the NSA could send the data over to Snowden who could then send it over to Julian; that would be epic.

Re:Where's the torrent file?

[Anonymous Coward]

Make sure you mean what you say. Tea party != libertarians for years now, so if you meant the libertarians, you're wrong because they would never support government coverups. Unfortunately the leftist lamestream media has the tendency to paint the tea party and libertarians with the same brush as an ad hominem attack on the latter. Gotta keep that control over the narrative after all.

No one said anything about libertarians, but your defensiveness and usage of the phrase "leftist lamestream media" does tell me that you're a fucking moron. Like all libertarians, Libertarians, and Tea Partiers.

Re:Where's the torrent file?

[real gumby]

I dunno, but I'm guessing none of these politicians have ever heard of 1984.

On the contrary it is the party manifesto.

Re:Where's the torrent file?

[bfandreas]

The UK Tories under Cameron are indeed appalling. It is hard to decide if they are merely incompetent or malicious. Their actions of late point to the latter. Indeed one could only speculate how bad it would have been without the LibDems.

The UK political scene has always been a bit foreign to my German tastes. A backbench MP suggesting that feckless fathers should be dragged to work in chains in defense of the badly executed bedroom-tax would have been forced to apologize in German politics. And he would have lost his seat come the next election. The comically idiotic ads targeting "illegal" immigrants to turn themselves in are both malicious and incompetent. And even now there is another push to introduce the "snooper's charta" which in the light of the recent revelations about the GCHQ isn't even needed for them to do what they do.

The other paries in the UK look good in comparison because of the unmitigated disaster that is the current Tory crop. Thatcher was bad but potentially a necessary evil due to the unmaintainability of the Postwar Dream. But think as I may I can't begin to fathom where to start to look for a justification for that cabinet, that PM and that party. They do not even have the use of a compass needle that permanently points to the south. You can't say "let's do the opposite of what they are suggesting" due to the utter confusion that is their politics.

Re:Where's the torrent file?

[dr_blurb]

I'm surprised there isn't already a copy out there somewhere.

Freedom of Information request first?

Re:Where's the torrent file?

[PsyMan]

If I could mod this up I would. Shame the general public don't have a clue or even care any more about what is going on around them. If you ask the majority if they are voting they instantly think X-Factor or Strictly come whatever it is now. Maybe if the Government added TV style £1.50 per minute mobile phone voting with stupid questions tagged on to each party they may get a better turn out at the next elections.

Re:Where's the torrent file?

[N1AK]

This story has to be one of the strangest freedom-zealot / anti-tory circle jerks I've seen in a while. Someone changed the robots.txt on their website so that it didn't include a folder. Maybe it's just me but that doesn't sound like an amazing story. Has there been any reporting so far that they wanted to, or even knew, that the internet archive would delete the content because it retrospectively respects robots.txt?

Personally I find it disappointing that a political party is making it harder to find speeches they have given in the past but it's hardly criminal to slightly amend a website. If anything this just highlights that the internet archive handles robots.txt is a pretty stupid way; or that if we want to stop people vanishing stuff from the internet then we shouldn't be relying on them to keep the information up themselves.

Can anyone find a copy of Tony Blair's speech justifying the Iraq war on the Labour website via searching on google? Can anyone find the transcript of Gordon Brown calling a voter a bigot (I'd also accept the audio) on the Labour website?..

Re:Where's the torrent file?

[dkf]

It is hard to decide if they are merely incompetent or malicious.

"Or"? I think you'll find that "and" works just as well in this case, and it becomes not hard to decide at all. (Though with a boolean "or" as opposed to the customary english usage, it's also easy to decide: "true").

Re:Where's the torrent file?

[AdamColley]

Yeah, I hear the BNP and UKIP are hanging up their axe handles and putting away their pamphlets as they now have the Tories in power to do their work for them. I used to be a Lib Dem member, no more, they might have moderated the tories a bit but without their leaping into bed with them at a chance of a bit of power the tories may not have got in at all, as another election would have been called. It's labour from now on, they might be bastards but at least they're bastard in a fairly even handed way and haven't essentially declared war on the poor, sick and unemployed.

Re:Where's the torrent file?

[Anonymous Coward]

Signed,
The only member of the Political Party of Common Sense

Well, the reason nobody else joined it is that they noticed beforehand that the name of the party doesn't correctly describe the current member.

Re:Where's the torrent file?

[bfandreas]

Let's not forget the recent findings that gov outsourcing isn't too hot either.
Seems like they outsourced entire bureaucracies to G4S and their ilk(most of the contracts go to 4 megacorps) without any kind of oversight. And then, suddenly, olympics and the armed forces have to step in due to G4S not being able to provide security. Also the embarrassment of the outsourced fit-for-work assessment centres produce lawsuit after lawsuit. The total saving seem to eat up the entire budget of a couple of ministries.

Jim Hacker would be proud.

Re:Where's the torrent file?

[Anonymous Coward]

Signed,
The only member of the Political Party of Common Sense

Given half a chance, I'd kill you and your children

Ah, the voice of sanity and reason!

Re:Where's the torrent file?

[doccus]

Frankly, I've been appalled at what's been going on in the UK for years. Not really that different from Canada, another British system. Where, just like y'all on the island, we never had any pesky "Freedom and liberty for all" types to ruin a good socialist dictatorship. Except the British governments have taken totalitarianism to the nth degree. When it comes to internet freedom, which is far more important than "guns" for protecting liberties and freedoms, everybody's capitulated to the big media companies, who have clearly been infiltrated by your MI6, and the CIA in the States, as the best way to shut down the net. Americans ignore the net and instead cry "guns" claiming the constitution, but when that law was drafted, to protect against an overreaching tyrannical government, they didn't have RPGs and these monster tanks, drones,and the rest of the deadly arsenal the military now have. Try fighting THAT off with yer pistol.. Sure.. Anyone who studies history though, knows that any resistance reqires well planned co-ordination, and if a totalitarian state rears it's ugly head, they're going to first purge the military (done as I write this), and then take over the media, by any means possible. In the case of the net, they're only having limited success in the US, so maybe they'll just pull the plug, with an "unexpected" planned CME. China, Europe, Australia, and especially The UK, are having lots ore success though, as they have far more control over the ISPs, and, also, especially as the UK has more electronic stool pigeons than the Soviets could have ever dreamt of. For a long time Canada was still free, but here as of only a very short time ago, it's becoming increasingly controlled now. Of course, as is typical with Canadians, it's been done so quietly nobody's noticed. Except when receiving an inordinate number of "not available in your country" notices. Heck I can't even see a Saturday night live *promo clip* on YT! Also lots of 403 notices, and such. But nothing like Jolly Old Britain's new face.. "Are there no prisons?.. Are there no workhouses?"... Looks like somebody took good ol' Scrooge to heart as a role model, same as many did with the fictional character Gordon Gekko..

Re:Where's the torrent file?

[raorajesh]

I dunno, but I'm guessing none of these politicians have ever heard of the Streisand Effect.

Maybe they have...

Re:Where's the torrent file?

[bfandreas]

Heh. Everybody involved in THAT movie was a bit appalled that the yuppies started wearing suspenders to work. A typical situation of "you didn't quite get the point".

Some awefully clever chap in The Grauniad speculated the lack of noise from the UK might come from misplaced pride stemming from Bletchley Park and 007. That or the dearly beloved island monkeys wear so much wool it's easy to pull over their eyes. God bless their tiny little toes. Fee-fie-foe-fum, I smell a bloodless Englishman.

I shouldn't rock the boat too much since the German authorities seem not so much miffed at the spying but that they weren't playing any significant part in it. Although there was some GCHQ praise for the BND capabilities. The official response to that seems to have been "Gosh! Jolly! That's an awefully nice thing to say, isn't it?". Mutti is rather upset she hadn't been properly and officially notified the Yanks had been naughty nosy little boys.

All in all it seems nobody refrained from spying on everybody. Interestingly only the Mossad remains with not more than the usual egg on face. It's so beautiful I might break into song at any given minute. Something about model generals should be appropriate.

Re:Where's the torrent file?

[BasilBrush]

This story has to be one of the strangest freedom-zealot / anti-tory circle jerks I've seen in a while. Someone changed the robots.txt on their website so that it didn't include a folder. Maybe it's just me but that doesn't sound like an amazing story. Has there been any reporting so far that they wanted to, or even knew, that the internet archive would delete the content because it retrospectively respects robots.txt?

There's a good chance that they didn't know. But the primary result of robots.txt, that they will have known about, would be to stop some or all of the site appearing on Google. Why would be the justification for that?

Can anyone find a copy of Tony Blair's speech justifying the Iraq war on the Labour website via searching on google?

Of course not. All political parties will only have things that they currently still believe in on their current websites. But that's not the issue. This is about robot.txt and it's result of whitewashing the internet archive. I don't know which of the several Blair speeches on Iraq that you mean, but there are plenty of snapshots of the Labour website from that era, so you should be able to find the one you mean.
http://web.archive.org/web/20030615000000*/http://labour.org.uk

Can anyone find the transcript of Gordon Brown calling a voter a bigot (I'd also accept the audio) on the Labour website?..

That wasn't a speech, it was a gaff, recorded by chance, and so of course would never have made it onto a party's website. Just as Tory gaffs understandably never make it onto their web site. You've now made it absolutely clear that YOU are the party political zealot here.

Re:Where's the torrent file?

[Intrepid imaginaut]

Don't they have a law in Germany whereby all else being equal, women are given jobs first?

Re:Where's the torrent file?

[robhammond]

There's a dropbox link in this post to a zip file containing all of the speeches (also turned into an elasticsearch database for easy information retrieval): http://robhammond.co/blog/search-engines-weapon/

Archive.org should not respect robots.txt

[Anonymous Coward]

People have used robots.txt to buy up domains they want to censor.

For example, this happened with partyvan.

Re:Archive.org should not respect robots.txt

[Anonymous Coward]

People have used robots.txt to buy up domains they want to censor.

How do you use robots.txt to buy domains?

Re:Archive.org should not respect robots.txt

[Anonymous Coward]

He misspoke. He meant to say they bought up domains and then used robots.txt to subsequently censor the site (including all older content)

Re:Archive.org should not respect robots.txt

[Obfuscant]

Of course, as a robot, archive.org should respect robots.txt. I have a website with millions of files of data that archive.org has no reason to keep for me, all behind a robots.txt that bars such nonsense.

I also have a link to a realtime predicted tide generator which takes about 30 seconds to calculate the information it sends back. Before I hacked in a robots.txt to cover it (it's on a different port than the normal web server and thus, according to the robot operators, a completely different website than the one that already had a robots.txt to stop them) one "helpful" robot indexer latched onto it and was sending ten requests per minute. Nice of them to throttle themselves, yeah, when they were running my apache server up to the connection limit (keeping other people from using the site) and driving the load up so the site was useless for anyone local.

So any suggestion that any robot operator ignore robots.txt should be shouted down as the complete nonsense it is.

People have used robots.txt to buy up domains they want to censor.

You can't buy a domain with a robots.txt. Once you own the domain, you have the right to "censor" it all you want, including the use of a robots.txt that bars all robots. But if your goal was to "censor" a website, just stop running an HTTP server. That's much better than any robots.txt in keeping everyone from getting your stuff.

Re:Archive.org should not respect robots.txt

[lgw]

As I understand it, Archive.org uses robots.txt to censor old, already captured data. That's a serious flaw in an archive IMO.

Re:Archive.org should not respect robots.txt

[fustakrakich]

When robots.txt is used for censorship, it no longer deserves any respect. I hope more people decide to ignore them. We should never let other people decide what we can see and hear. For the time being we can store stuff locally and employ P2P.

Re:Archive.org should not respect robots.txt

[morgauxo]

The problem is that people are buying up the domain names of old websites which no longer exist just to publish a robots.txt file. Then archive.org automatically deletes, or at least blocks access to the entire history of everything that ever happened at that domain including the past website which the new owner has nothing to do with.

I suppose they are just trying to honor site owner's wishes even when they may have initially forgotten about robots.txt and added it later. The robot doesn't know that the old content belonged to someone else who DID NOT wish to block it. Maybe a good solution is that when they notice a new robots.txt everything for the last 'X' months get deleted. (go ahead and debate values of X) Data from prior to that should be left alone. Even if it was posted by the same site owner who is posting the robots.txt today. Tough cookies! If you want to control how your data is used I don't see a problem with requiring you actually take the time to learn about things like robots.txt before you publish. It's really no different than releasing source code under the GPL and then later turning it into a closed source product. All your new work belongs to you but you don't get to force everyone to delete ever copy they might have of the old code and you can't stop them from forking it.

-- I would totally consider an 'X value' of zero as being on the table btw

Re:Archive.org should not respect robots.txt

[Bardez]

Robots.txt should be respected at the time of retrieval. It should not be retroactively respected to censor or remove old data. That is a shame. I've used the Archive before on a site of a gaming company that I loved, which nearly went bankrupt (or perhaps did) but managed to eke its way through. Part of their relaunch nuked the Internet Archive's archives and I definitely felt a sense of loss.

Re:Archive.org should not respect robots.txt

[RedBear]

Robots.txt should be respected at the time of retrieval. It should not be retroactively respected to censor or remove old data. That is a shame. I've used the Archive before on a site of a gaming company that I loved, which nearly went bankrupt (or perhaps did) but managed to eke its way through. Part of their relaunch nuked the Internet Archive's archives and I definitely felt a sense of loss.

Yeah, I had the silly impression all this time that the entire purpose of the Internet Archive was to archive the goddamn Internet precisely so that people couldn't pull this kind of retroactive erasure "cleansing of history" bullshit and get away with it.

What a dope I am. It's amazing how inadequately we are protecting our freedoms and our history these days. If we don't do something much more drastic our grandchildren will end up being slaves to some theocratic corporatocracy and they'll have no idea that the world was ever any different.

Lately I think Orwell was overly optimistic.

Re:Archive.org should not respect robots.txt

[mikael]

They buy up a domain when it becomes available, set the robots.txt file to "do not archive", then the google-bot spider will send the instruction to delete all
past archives.

You used to be able to visit old web-pages through the google-cache. Remember when google would always have a cached copy of what you wanted to read. Nowadays they just seem to be happy to be a proxy server which records everything you download from the target webpage.

Re:Archive.org should not respect robots.txt

[Anonymous Coward]

It is. The "problem" you have is that robots.txt is retrieved anew upon every request.

The fact of the matter though is any argument against this is irrelevant. There is no such thing a as a truly public web page. The public might be able to access it, but at the end of the day it is owned by one or more people. Those people can decide what is and is not accessible on their own web page. Restricting that ability will not stop people from bending or breaking the rules. It will ONLY affect all the "little guys" trying to speak the truth.

If you haven't learned this yet, then you haven't been paying attention to history for the last forever.

Re:Archive.org should not respect robots.txt

[DriveDog]

That's what had me wondering WTF. I would not have guessed that current settings could affect previously archived information.

Re:Archive.org should not respect robots.txt

[Anonymous Coward]

This is the way I will treat robots from now on and a copy of this as the explanation why.
Abused by politicians.

Re:Archive.org should not respect robots.txt

[Obfuscant]

When robots.txt is used for censorship, it no longer deserves any respect.

It's not censorship when I tell robot data scrapers to bugger off and not abuse the website I run by copying every image I have and looping through the multiple links that take people there, or to invoke a program that generates data on they fly tens of thousands of times a day to the detriment of real users who actually have an interest in the information and can't get it because some robot is using all the available server processes.

I hope more people decide to ignore them.

The day that the first scraper starts ignoring mine, his IP is going into the firewall. If he tries to be a sneaky shit and use multiple IPs, then the site where YOU could come get data for free may very well go away, and you wind up with nothing. Neither I nor my employer have the spare bandwidth and cpu cycles to have every robot come download the Tb of data I have on the web. If free public access becomes an abuse of the server, the free public access goes away.

We should never let other people decide what we can see and hear.

When you are talking about my data, I have every right to decide whether you can see or hear it. It is your attitude of entitlement that makes me always have second thoughts about putting anything on the web. Most people are reasonable, decent people who appreciate the service. Some think they have a right to demand it.

Re:Archive.org should not respect robots.txt

[Anonymous Coward]

Lets hope they never come across the file http://robots.txt

Re:Archive.org should not respect robots.txt

[Obfuscant]

I suppose they are just trying to honor site owner's wishes even when they may have initially forgotten about robots.txt and added it later. The robot doesn't know that the old content belonged to someone else who DID NOT wish to block it.

That's probably why they do it that way. They could have picked either side and been wrong for some group. The website operator who didn't know about robots.txt to start with and found some of his material on a robot indexer shouldn't have to track down every robot who has ever visited to be able to rectify the mistake.

The other issue with keeping data after a robots.txt is published by a new owner of a domain is that the archive will contain data that claims to have come from that website but in fact did not. This can create issues for both the new and old owners. Someone who finds defamatory material in the archive may decide to focus his anger on the new website owner who is completely innocent and forced to prove a negative ("I never published that page.") The old owner may find out that his copyright material now has a presumed ownership by the new domain name owner. ("This image came from the domain example.com which is owned by Bill Smith ...") If the material was popular, the new owner may be deluged with requests for updates to material he never knew existed and doesn't have time to deal with.

The other side of the choice means that someone who buys an abandoned domain name can get all the previous content from that domain dumped from the archives. The question that could be asked, if the original domain owner cared about having his information archived, why did he abandon the domain? Is abandoning his domain also abandoning the data it served?

Tough cookies! If you want to control how your data is used I don't see a problem with requiring you actually take the time to learn about things like robots.txt before you publish.

Yes, "screw every website owner who didn't fall off the turnip truck knowing everything there was to know about websites" is one opinion. Being a bit more considerate and not making every mistake a permanent one is another opinion.

It's really no different than releasing source code under the GPL and then later turning it into a closed source product.

Ahh, yes, it is different. Putting information on a website so people can come look at it is not releasing it under GPL.

A reasonable solution might be for the archives to remove everything they have with the same link as is now protected by the robots.txt, but keep anything that isn't the same. This runs into the problem that to know if the links are the same the robot has to scrape the site.

It's not a simple issue.

As someone who has been running websites since around 1992 or so, long before robots.txt was necessary, and before almost anyone imagined trying to keep a full archive of everything that ever appeared on the web, I am actually amazed at the attitude that everything that has ever appeared on the web must be available for anyone who wants it no matter what the original author desires. I know there are too many things that I've had to correct to ever want all previous versions to be wandering about to confuse people. Too many things I've changed my mind on, too.

Re:Archive.org should not respect robots.txt

[epyT-R]

When you are talking about my data, I have every right to decide whether you can see or hear it. It is your attitude of entitlement that makes me always have second thoughts about putting anything on the web. Most people are reasonable, decent people who appreciate the service. Some think they have a right to demand it.

The problem is that people such as yourself often think that the presence of your data on someone elses machine somehow gives you the right to install invasive DRM software in an attempt to get their machine to do your bidding instead of the owner's. That is also an unacceptable sense of entitlement.

I'm sorry, but the reality is that people are going to have to accept that data cannot be controlled remotely forever. Once the data is recorded and someone else gets a copy, it's only a matter of time before it gets decrypted/distributed. The more interesting the data, the shorter the time involved. The only solution that offers certainty of control is non-distribution. If you can't make money this way, it's time to find a new job.

Re:Archive.org should not respect robots.txt

[SmilingBoy]

I also have a link to a realtime predicted tide generator which takes about 30 seconds to calculate the information it sends back.

Off-topic but could you share the link to this? I was looking for something like it and couldn't find a good one anywhere.

Re:Archive.org should not respect robots.txt

[Obfuscant]

The problem is that people such as yourself often think that the presence of your data on someone elses machine somehow gives you the right to install invasive DRM software in an attempt to get their machine to do your bidding instead of the owner's.

I don't know what the fuck you are talking about. Robots.txt is not DRM, and I'm not trying to get some data scraper's system to "do my bidding". They can do whatever the fuck they want as long as they don't use my system to do it.

Once the data is recorded and someone else gets a copy, it's only a matter of time before it gets decrypted/distributed.

Still unclear on what you think you are contributing to this discussion. You want to look at my data, you can come do it all day, every day. Yeah, if you're a moron who tries to update a static image once a minute 24/7, I'll shut you off like the idiot abuser you are, but other than that, knock yourself out.

What robots.txt stops is people who think they can index my system better than I can and want to make money doing it, or who think that they need to archive tide predictions for every day from year 1900 through 2100 at every site where tidal constituents are available, they need my system to generate those predictions for them, and in doing so they keep legitimate users from being able to access my site.

The only solution that offers certainty of control is non-distribution. If you can't make money this way, it's time to find a new job.

Thanks. Your vote for me to take the free information I provide off-web has been recorded.

As for "mak[ing] money this way", I don't get paid to give out data for free, I get paid for collecting the data. That's whether or not I let people like you abuse my systems by trying to index them for me.

Re:Archive.org should not respect robots.txt

[gl4ss]

yeah call me when they have a law about robots.txt... ..and about hiding right click functionality as if it changed anything.

have a captcha. scrapers gonna scrape anyways.

Re:Archive.org should not respect robots.txt

[fustakrakich]

Hey, don't worry about your bandwidth man. Your site only needs to be scraped once. Then we can distribute for you. Once the stuff is in our machines, it's ours.

Re:Archive.org should not respect robots.txt

[Obfuscant]

Robots.txt is essentially a 'you may trawl, link to, and archive/copy this digital information' or 'you may not trawl, link to, and/or archive/copy this digital information'

Robots.txt has absolutely nothing to do with you linking to my data. What it says is you may not ABUSE MY SERVERS by DOWNLOADING EVERY FILE ON THE OFF CHANCE THAT SOMEDAY SOMEONE MIGHT WANT TO LOOK AT IT ON YOUR ARCHIVE. You can do whatever the hell you want with YOUR server, but MY SERVER is under MY CONTROL and I have the right to say what you may or may not do on it.

Jesus Christ you are arrogant.

Re:Archive.org should not respect robots.txt

[Obfuscant]

IPv6 users are laughing at your dumb ass right now. Can your idiot ass guess why?

Because it is easy for them to be sneaky shits and use mullions of different IP addresses. Like I said, when the sneaky shits overwhelm the services they are being given for free, the services go away. I've dealt with people like you before. I'm still here. So are my websites.

Fuck, my company would have been dead long ago with an idiot like you behind the wheel.

Yeah, it's a horrible thing to try to make sure that company resources are available for the intended company use and not overwhelmed by leeches sucking up every cycle on a service that they aren't paying for but feel entitled to suck dry. Yeah, the company would simply love being told that they need to buy a rack full of servers and more bandwidth because you want to copy every bit of data on the website.

You're the kind of moron that thinks a TI 99/4A would make for a good switch or router.

It's such a pleasure talking to you. Do you do anything constructive or just toss insults at people you disagree with?

Re:Archive.org should not respect robots.txt

[Anonymous Coward]

Archive.org is just Alexa, which is just Amazon. I expect at some point, the Internet Archive will be monetized, just like Wikipedia keeps looking for donation handouts.

Anyway, the loophole of the robots.txt file is it must be at the root of the domain host.

What happens to users who have websites as part of an ISP's shared domain? Archive.org already seems to ignore those robots.txt files.

So, suggesting that Archive.org not respect robots.txt (or any other entity ignore robots.txt) also has a corolary suggestion that websites be built to crash the bots that ignore robots.txt domain files.

Re:Archive.org should not respect robots.txt

[kermidge]

I'm a big fan of being able to get information. Since you already allow that, I have no complaints. If idiots in the form of a human or a human who specifies what a particular spider does is being a prick, then fuck 'em - it ain't their data, and it's already there for the looking on a per-case basis. So, point, set, match to you.

So long as I get to look at what time to set sail so's I can clear the sand bar at harbor's mouth, I'm good (and hoping that last storm hasn't shifted it... but that's what we have fathometers for - and a dinghy and kedge anchor just in case.)

Re:Archive.org should not respect robots.txt

[kermidge]

Oh, yeah, and just to respond to the real topic - seems to me the Brit Conservatives have their heads where the Sun doesn't - any speech they've given is already public domain and they have absolutely no right to try to remove same.

I note that we've lost a whole bunch of info that was out in the open but got yanked or walled after 9/11 - and some of it started way before then. In my opinion of course, most all of this behaviour is down to a whiny bunch of craven power-crazed individuals possessed of egos far exceeding their intellect. What the Tories are doing is more of the same, and rather typical. Aaargh.

Re:Archive.org should not respect robots.txt

[dargaud]

Or when you archive a website, you also archive the whois reference with it. If the whois info changes, then don't retroactively apply robots.txt.

Re:Archive.org should not respect robots.txt

[Anonymous Coward]

The problem is when it is not your data which is affected, but the data of whoever happened to own the domain before you bought it. You should not have the power to delete data which you don't own, just because you currently own the domain name which previously was used for hosting the data.

Re:Archive.org should not respect robots.txt

[wonkey_monkey]

Robots.txt is essentially a 'you may trawl, link to, and archive/copy this digital information' or 'please don't trawl, link to, and/or archive/copy this digital information'

FTFY.

Re:Archive.org should not respect robots.txt

[Intrepid imaginaut]

Google only stored cached pages for a few days, as far as I recall.

Doesn't that kinda defeat the point of the archive

[spiritplumber]

How did they delete them from archive.org? Did they hack it?

Just taking after Obama

[Anonymous Coward]

http://www.huffingtonpost.com/2013/07/26/obama-whistleblower-website_n_3658815.html

Lol! Good luck with that

[DontBlameCanada]

Google cache etc will ensure every public speech made since the late '90s is kept forever and many made before that will also be indelibly etched into history.

Re:Lol! Good luck with that

[x0ra]

The White House and its UK equivalent has unfortunately enough power to order Google to "forget" about this as well.

Re:Lol! Good luck with that

[PPH]

No problem. Just look right here.

Re:Lol! Good luck with that

[game kid]

Lol indeed. When Google aren't being ordered by the NSA (and by extension GCHQ and their political friends) to work for them, they volunteer outright. Enjoy the cache while it lasts and while they allow it, because they'll consider either an oversight.

Re:Lol! Good luck with that

[wbr1]

Until google cancels the public facing aspect of the cache (ie reader, iGoogle, etc).

Re:Lol! Good luck with that

[Anonymous Coward]

Google deletes their cache periodically.

The Internet is for ...

[Anonymous Coward]

The Internet is for Orwell...

Appearently porn will have to take second place to political power...

Re:The Internet is for ...

[Anonymous Coward]

Mmm... Orwell porn...

And thus invoking the . . .

[Mitchell314]

strei . . . you know what, screw it. Let them shoot themselves in the foot.

Re:And thus invoking the . . .

[Anonymous Coward]

Yes, because they are so clueless that it would have taken *your* 15 word post on a site they don't know exists for them to realize they're making a mistake.

Re:And thus invoking the . . .

[Anonymous Coward]

If one deliberately tries to invoke the Streisand Effect, it shouldn't kick-in.

Who cares what the conservatives said, say or will say? Nice try dudes.

Re:And thus invoking the . . .

[compro01]

Them shooting themselves in the foot isn't all that amusing when the foot in question is on your neck.

Re:And thus invoking the . . .

[Mitchell314]

Did I hit a nerve? I think I hit a nerve.

Re:And thus invoking the . . .

[Evil Pete]

OTOH, this means that whenever reference is made to one of their speeches people can just insert scandalous bits. Objections by the Tories would be countered by pointing out that because they removed all copies from the Internet then anything they publish has been modified and is therefore not to be trusted. It should be easy to cultivate an aura of mistrust in anything that they say after that. Well, that is what I would do if I was Machiavelli. Or true to my username. :)

Insert references to an Orwell work here

[themushroom]

Because that's what they did in that book.

Re: Insert references to an Orwell work here

[Mabhatter]

The main character's job was "correcting" stored historical documents to match what was being said "right now".

The reasoning why their government must keep EVERYTHING on private people, but can obstruct and hide PUBLICLY OFFERED documents has to be really really funny!

Re: Insert references to an Orwell work here

[Shatrat]

This is just basic smart politics. An old speech can never help you in an election, but can be mined for quotes to be used against you. Journalists should keep their own records anyway, or establish some 3rd party trusted repository.

Re: Insert references to an Orwell work here

[TapeCutter]

Also politicians don't want you to hear every speech they make, someone might point out they are making contradictory statements to different demographics..

Internet Archive's Wayback Machine

[FriendlyLurker]

Lucky they now have secret blacklists at every major UK ISP to block these. Think of the children that would be harmed by reading these speeches!

FTFA:

In a remarkable step the party has also blocked access to the Internet Archive's Wayback Machine, a San-Francisco-based library which captures webpages for future generations, using a software robot that directs search engines not to access the pages.

Re:Internet Archive's Wayback Machine

[Impy the Impiuos Imp]

I thought the point was to stop crawlers from getting stuck following generated links to infinity, creating large numbers of virtual pages that would rarely if ever be instantiated, wasting the web site's resources.

Clearly using it to retroactively delete should now be revisited, at least by archive sites.

BTW, I am sure he NSA's archive crawler does not honor the robots hing.

Re:Internet Archive's Wayback Machine

[Obfuscant]

BTW, I am sure he NSA's archive crawler does not honor the robots hing.

As a website operator who carefully watches connection counts and has a robots.txt to exclude most of the site content, I am pretty sure there is no "NSA's archive crawler" visiting, at least none with any frequency that it matters.

Re:Internet Archive's Wayback Machine

[Anonymous Coward]

You didn't know...??? They got your data even before it hit your server. You never knew it happened.

Re:Internet Archive's Wayback Machine

[MadCow42]

Well, maybe it's not a "crawler", but if it copies all outbound traffic it does essentially the same thing while leaving no footprints.

Chew on that for a while. :)

Re:Internet Archive's Wayback Machine

[Obfuscant]

Well, maybe it's not a "crawler", but if it copies all outbound traffic it does essentially the same thing while leaving no footprints. Chew on that for a while. :)

Hmm. (Thinking) Oh my! NSA is sniffing all outbound traffic from my webserver, where people get publicly available information for free. Oh noes! They might see something they could have seen for themselves!

Re:Internet Archive's Wayback Machine

[jrumney]

using a software robot that directs search engines not to access the pages.

Don't tell me the Tories have perfected software robot technology. We'd better hope its not the powerful txt variant, or it will be the clearest sign yet that the reptilian shapeshifters have been successful in conquering our planet's political sphere.

Re:Internet Archive's Wayback Machine

[Khyber]

"I am pretty sure there is no "NSA's archive crawler" visiting, at least none with any frequency that it matters."

Idiot, they had your data the second you put it out there.

If you're in any sort of IT position, your boss should fire your dumb ass. You're a major liability to the company right now.

Re:Internet Archive's Wayback Machine

[Obfuscant]

Wow. I've picked up a stalker. Your mother must be so proud.

And let's not forget why:

[Joining Yet Again]

because they broke almost all of their pre-election promises.

The most important thing to learn about the Tory party in the UK is that, contrary to popular opinion, it is not the party for the responsible, the capitalists, nor the hard-working (except in the sense that they want most people to work hard for them). It is a party representing a few wealthy individuals, and their mission is not small government, but privatised government, where nothing happens without their masters getting a cut.

Sorta like a mafia.

Re:And let's not forget why:

[mpe]

because they broke almost all of their pre-election promises.

When was the last time a political party (or even an individual politician) did anything else?

Re:And let's not forget why:

[Joining Yet Again]

There have been more ideologically-oriented governments, from post-War Labour to Thatcher.

They might not keep all their promises, and all ideologically is strongly diluted with practicality, but they're not the vacuous bunch of cunts we have in Britain today. (They're not that different from Blair, of course, but Blair had a more representative set of people to steer him.)

Re:And let's not forget why:

[Anonymous Coward]

Conservatives are all the same - they're all telling Grandma " boo! a terrorist doctor might make you pay to give treatment to a black person for syphilis!"

Re:And let's not forget why:

[neo-mkrey]

sorta (a lot) like the Republicans in the U.S.

Re:And let's not forget why:

[roninmagus]

The main issue that conservatives (at least in the US) have in their thought process (trust me, I am one) is that they believe "responsible," "capitalist," and "hard-working" actually leads one to become one of those few wealthy individuals.

Unfortunately this is usually not the case at all; the responsible, capitalist and hard-working ones only lead those wealthy few to become more wealthy.

This is a truth I think conservatives should realize and embrace, so that we can actually come up with real solutions to problems.

Re:And let's not forget why:

[Anonymous Coward]

If that is so, why even go through the bother of scrubbing their speeches from the internet? You see, they obviously want to claim they're different. And if the facts are too inconvenient to their cause, they'll create new facts and hope to recruit enough gullible, hard-working people to their cause. As I think has been stated before, political ideology is sort of like a mobius strip: the extreme left and right tend to meet. Of course, perhaps it's more like a klein bottle given the spectrum of individuality vs group authoritarianism, but the comments about 1984 or Animal Farm seem to well apply.

So, the real answer to your question is, they're different because they're the ones trying to scrub the past.

Re:And let's not forget why:

[Anubis IV]

The most important thing to learn about the $political_party_name party in the $local_country is that, contrary to popular opinion, it is not the party for $positively_framed_groups_and_associations. It is a party representing a few wealthy individuals, and their mission is not $claimed_mission, but !$claimed_mission, where nothing happens without their masters getting a cut.

Sorta like a mafia.

Sounds like pretty much every political party I know of once I FTFY.

Re:And let's not forget why:

[Blue Stone]

because they broke almost all of their pre-election promises.

Here's a nice little summary of all those broken promises, pledges and outright deceit.

Re:And let's not forget why:

[Anonymous Coward]

When was the last time a political party (or even an individual politician) did anything else?

When was the last time a politican prefaced every promise with "I will try to bring up" or some other such qualifier?

Re:And let's not forget why:

[Anonymous Coward]

because they broke almost all of their pre-election promises.

This part has always baffled me, we have the same thing in my country: before an election the candidates come out with all kinds of promises, people vote them in based on those promises, and then nothing... As far as I can tell the promises are 100% non-binding, there's no contract, no non-performance clause, nothing. The only resort available is to not re-elect the same candidates next election, 4 long years later, whoopee friggin do!

If we don't want to become slaves to the governments in the 21st century, there's one fundamental change we must make asap: make all election candidates post bonds and sign binding contracts for their promises with penalties for non-performance.

Re:And let's not forget why:

[Anonymous Coward]

Which ones? The RINO's (Republicans in Name Only) routinely run on right leaning platforms but they really are center to left. But that is only *part* of the people running with "R" after their name.

Case in point is Ted Cruz. He ran on a Tea Party platform, crushed his opponent and then proceeded to actually do what he said he would as best he could.

So it's not ALL republicans...

Of course, there are a majority of Democrats that promise one thing and actually do another, especially at the national level. "If you like your plan, you can keep your plan." "If you like your doctor, you can keep your doctor." and soon "There are no death panels.. "

Re:And let's not forget why:

[MysteriousPreacher]

That'd be a good start. Have the parties' promises displayed on a big board outside the House of Commons, with promised deadlines and the completion status of each pledge.

Also, let's bring some class back to the house. For every £1000 earned outside of their day job, give them a foot of cloak to wear, with the names of their employers pasted on the cloaks. Good luck to the ones wandering around with cloaks longer than a bus. Include sponsored junkets in the mix, and require they wear their cloaks whenever they're working or travelling on anything but trips paid for themselves or the state.

Forbid them from employing family members who have time commitments that would indicate their inability to do the job for which they are being paid.

Fix their housing allowances to cover only modest renting in London for their time spent there. Base this on the modal income of their constituency, adjusted to be at least as high as the salary of an entry level police sergeant. Right now these fuckers can claim more in housing than most people earn before tax.

Re:And let's not forget why:

[Anonymous Coward]

and (a lot) like the Democrats in the U.S.

Where is this hope and change we were promised?

Re:And let's not forget why:

[Archangel Michael]

I would like to add to each piece of legislation, the exact result desired, and automatically repeal any legislation that does not have the desired results. Why should we "fix" legislation that doesn't work? Why should we trust those that can't figure out how to right legislation correctly to go back and "fix" it when it breaks things more than it fixes?

Re:And let's not forget why:

[AK Marc]

There were always death panels. They just worked for the companies that profited from your death. Why do you fear a death panel run by the government (in which you have a say) and not the one run by the private corporation in which you have no say, and has a financial incentive to kill you?

Re:And let's not forget why:

[MysteriousPreacher]

It could be interesting as well to outline things that shouldn't happen. E.g. The ability to suspend websites under the Stop Happiness In Terrorism Act will not be employed to take down a torrent website with no proven connection to terrorists.

Re:And let's not forget why:

[Anonymous Coward]

Speaking as someone who believes in the principle of harm reduction, shouldn't they cure that black person's syphilis to obviate transmission to the white wimmins he done raeps?

Oh wait, I know the answer to this one - white women can simply exercise that obscure biological function that prevents them from getting syphilis.

Hahaha, silly me. What the fuck was I thinking?

Re:And let's not forget why:

[Joining Yet Again]

This would never work.

1) As any mathematician kno, statistics can be abused to prove or disprove anything;

2) If you define the precise tests used in advance, the system will be gamed extremely well to satisfy the statistic but not the principle.

Politics is a game of values, not targets.

Re:And let's not forget why:

[fnj]

I'll tackle that one, and very succinctly. A panel you can fire is preferable to a panel you can't. Any day. And they both have precisely the same motivation. To save money, the better to spend it somewhere else.

Re:And let's not forget why:

[AK Marc]

In the US, it's harder to fire the private death panel. In countries with socialized medicine, it's as easy to fire the government death panel as any private one.

Only in the US is there a concept of a single-payer system outlawing health insurance or nationalizing all medical care. The entire argument against socialized medicine is on the shoulders of lies and strawmen. I left the US for a place where I'm paid better, and pay less tax and get more services for that lower tax.

Re:And let's not forget why:

[shutdown -p now]

So, as a conservative, what is your take on the real solutions?

Re:And let's not forget why:

[Anonymous Coward]

I thought that a good use for the Olympics accommodation after the games was a secured hostel for all MPs -- in one place so security would cost less, no messing about with second home expenses...

Couple this with a mandate that they have the MP job ONLY -- any other income (including donations taxed at 100%) so they focus on what they're being paid to do. Personal wealth/spouse's income... should be in a frozen account whilst they're in power with interest paid at the prevailing base rate.

Their income to be set to the median** income of the whole population - so they can get a realistic idea of how their policies and actions affect the people they pretend to represent. Senior ministers maybe median + 5%.

This would remove the whiff of scandal that they're profiteering at the taxpayer expense, make them more representative and ensure that it attracts those who genuinely want to help others rather than just get a cushy, well paid job.

** median rather than mean as the income distribution is heavily skewed in the UK

Re:And let's not forget why:

[BigZee]

You can always tell when a politician is lying, his/her lips move. They would have said anything to get elected. Whilst I accept that the world is not static and that pledges such as these cannot always be honored, you do feel that there was little intention to keep to them. The problem is that they're all at it. No wonder we distrust politicians so much, they really are lying to us about these things all the time.

Re:And let's not forget why:

[MysteriousPreacher]

Yeah, removing the second jobs and one off fees would make sense. British MPs are not under-paid by any stretch of the imagination. It seems the average is £66,396 ($106,812), which is comfortably above the median salary for the UK. They also have a raft of expenses they can pull.

Anyone wanting to get rich by sitting on numerous boards and committees should be actively discouraged from seeking office. I'd rather see people in this to do the job, and with the salary as it stands I don't see issues in finding suitably qualified candidates. I'd understand as well that the government may well employ advisors on higher salaries, due to a need for specific expertise, but MPs do not require high expertise in any discipline that should be commanding salaries higher than the ones they already have. Obviously salary would vary based on their position - such as the Home Secretary probably pulling in more than a vanilla MP.

Some interesting sources:

http://www.parliament.uk/about/faqs/house-of-commons-faqs/members-faq-page2/

http://www.thisismoney.co.uk/money/article-2269520/Best-paid-jobs-2012-Official-figures-national-average-UK-salaries-400-occupations.html

Re:And let's not forget why:

[Reziac]

You forgot "..and not regulated out of profitability, maybe out of existence, by various special interests." Overregulation, more than anything else, is what prevents small businesses from succeeding.

1984

[MyLongNickName]

“He who controls the past controls the future. He who controls the present controls the past.” George Orwell, 1984

Re:1984

[Anonymous Coward]

He who controls the spice controls the universe.

Re:1984

[Anonymous Coward]

The spice is vital to space travel.... travel *without* moving! I OBE's baby.

Re:1984

[Anonymous Coward]

The entire comments section should be removed from this space and a copy of 1984 put in its place.

Re:1984

[Anonymous Coward]

Who controls the past now controls the future
Who controls the present now controls the past
Who controls the past now controls the future
Who controls the present now?

Rage Against the Machine, 1999

Re:1984

[spartacus_prime]

NOW TESTIFY!

Wrong

[symes]

This is not accurate. Speeches made in Parliament are archived in Hansard for a start. And there is no changing that.

Re:Wrong

[game kid]

I like your optimism.

They'll find a way to close that to public access (except "on a need-to-know basis" and to Royal family members, staff, and "security" officials) too, as soon as they see how embarrassing (or criminal) parts of the archive may be. Clearly, they always find a way, however brutish.

Re:Wrong

[EasyTarget]

Sigh.. 'Wrong in what way?

This was the archive of speeches, not just the parliamentary ones; but all the ones at election rallies and conferences too.

For instance; ToryBoy recently sat in a big gold chair and ate a 4 course meal along with all his rich chums in the Guildhall, London. He then stood in front of an gilded podium and made a speech in which he told all the little people that they had not worked hard enough and that austerity is now here to stay.

This speech is exactly the sort of one that will never appear on Hansard, and in a few years may well be the sort of thing Tory spinsters will hope to make 'disappear'.

Re:Doesn't that kinda defeat the point of the arch

[uncle slacky]

No, but the Wayback Machine always respects takedown requests. Note that the British Library maintains an archive of UK sites, and still has the speeches in question (from April 2008 onwards):http://www.webarchive.org.uk/wayback/archive/20080410100951/http://www.conservatives.com/tile.do?def=news.speeches.page

History will be lost

[Anonymous Coward]

There's a theory out there that states that because most of what we do in the so-called Information Age is stored is somewhat fragile digital storage systems (as opposed to, for example, parchment) historians in the future will have very little to base their research on about our age, as most of the info will be permanently lost.
Well, hundreds of thousands of posts on BBS systems from the 80's and 90's are already gone, delete the Internet Archive and the Web is gone too, any thoughts?

Re:History will be lost

[Karl Cocknozzle]

There's a theory out there that states that because most of what we do in the so-called Information Age is stored is somewhat fragile digital storage systems (as opposed to, for example, parchment) historians in the future will have very little to base their research on about our age, as most of the info will be permanently lost.
Well, hundreds of thousands of posts on BBS systems from the 80's and 90's are already gone, delete the Internet Archive and the Web is gone too, any thoughts?

An archive of the archive, operated in near-secret and kept in a Datacenter built into the side of a hollowed-out, dormant volcano... Or maybe TWO dormant volcanoes... You know, for redundancy.

Sharks with lasers on their heads optional, but recommended. Once "the last place" for evidence to be found becomes this place a great many people (including likely several large, powerful governments) will want to take control of it.

Re:History will be lost

[Sperbels]

historians in the future will have very little to base their research on about our age

They'll base their research on what they've always based it on: The "official" records of the victors.

Re:History will be lost

[mjr167]

Funny story...

Re:History will be lost

[Anonymous Coward]

I had a great thought but ... I forgot it.

Oh hey look, twitter!

Re:History will be lost

[Zeromous]

They are not gone, there at least dozens of CDROM backups of my FIDONET archives to be dug up from the landfills millions of years from now you insensitive clod!

Re:History will be lost

[Anonymous Coward]

In Alexandria, Egypt.

Re:History will be lost

[Anonymous Coward]

Well, hundreds of thousands of posts on BBS systems from the 80's and 90's are already gone, delete the Internet Archive and the Web is gone too, any thoughts?

The NSA will have a backup, problem is you'd have to wait til they declassify them..

Re:History will be lost

[Anonymous Coward]

An archive of the archive, operated in near-secret and kept in a Datacenter built into the side of a hollowed-out, dormant volcano... Or maybe TWO dormant volcanoes... You know, for redundancy

Mount Vesuvius?

Re:History will be lost

[fa2k]

I use the Shelve firefox plugin to save every single website I visit on the desktop. up to 65 GB (deduplicated) now. A distributed system like this makes sense, but i think we should also expect the people publishing information to keep their own copies (wouldn't solve the issue in TFA of course).

Deleted from the Internet Archive?

[Arancaytar]

The Tory party have deleted the backlog of speeches from the main website and the Internet Archive — which aims to make a permanent record of websites and their content — between 2000 and May 2010"

How'd they do that? Do they make a copyright claim on the record of speeches they made in public?

Re:Deleted from the Internet Archive?

[flimflammer]

No, they put robots.txt on their website and the Internet Archive respects robots.txt retroactively. If they had 20 years worth of data archived from one domain, and someone puts a robots.txt on the domain, all 20 years worth of data is removed from the archive. Whether it's actually deleted or hidden is unknown, but I hope it isn't deleted.

Re:Doesn't that kinda defeat the point of the arch

[LocalH]

It's not even a takedown request. IA will honor robots.txt totally and retroactively - if they have 10-15 years of archived data at a specific domain (or subdirectory on that domain), and someone puts up a robots.txt disallowing them access, not only will they refuse to archive it going forward, but they will remove all previously archived material from being viewable (I hope they don't actively remove it from their archive, but merely stop making it available).

robots.txt

[Anonymous Coward]

Archive.org will retroactively enforce a new robots.txt to all previously archived content.

This has been used by people who buy up domains targeted for censorship. This has happened to partyvan, for example, when it expired and bought up and turned into a honeypot for watching 4channers and Anons. When this happened a new robots.txt was put in place and all of partyvan's history was deleted from archive.org.

Someone else put a backup of partyvan up on github in response.

Re:robots.txt

[Joining Yet Again]

If A clearly and deliberately causes B, then there's nothing misleading about saying that someone chose B to happen when they elect to effect A.

Re:Robots.txt

[WhiteDragon]

The Internet Archive says that it subscribes to the The Oakland Archive Policy which for |requests by governments" says:

Archivists will exercise best-efforts compliance with applicable court orders Beyond that, as noted in the Library Bill of Rights, 'Libraries should challenge censorship in the fulfillment of their responsibility to provide information and enlightenment.'

Seems like this may just have slipped past them. Let's make sure they know they need to sort it out... Surely they only removed it from the Wayback Machine, not from the archive itself.

That's actually a really good point. I wonder if there's any justification in the Policy for retroactively removing content based on current robots.txt

There's only one possible explanation

[AstroSurf]

They're going to sell them and turn them into an asset.

they did it because...

[FudRucker]

they dont want to be called out on their broken promises and outright lies
call me Mister Obvious

Is this legal?

[Anonymous Coward]

Are they legally allowed to remove things from archives like that without any prior notification or permission?

Re:Is this legal?

[JockTroll]

Laws are for the rabble. Lords own the law.

Re:Is this legal?

[flimflammer]

They didn't personally remove it or request it be removed. They simply added a robots.txt to their domain, and the Internet Archive retroactively removes content from the domain when it encounters robots.txt.

Re:Is this legal?

[Anonymous Coward]

I didn't murder him, I simply pulled the trigger and the bullet flew out from the end of the tube and struck him.

They put a robots.txt on their site that blocked Internet Archive. That is a request for it to be removed.

Re:Is this legal?

[Anonymous Coward]

Just adding a robots.txt file doesn't delete anything. Adding a robots.txt file that specifically says to not archive any of these files or directories is what causes the deletion. It's not like this was an innocent effect.

Re:Doesn't that kinda defeat the point of the arch

[Arthur Dent '99]

The post is misleading. The Conservative website now has a "robots.txt" file which is designed to prevent search engines like the Internet Archive from archiving current and future content. They did not delete previously archived content from the Internet Archive.

Basically, the robots.txt convention is based on politeness. It merely lists directories and files which "honest" search engines agree to not search through. There's nothing actually stopping anyone from ignoring these requests and searching those "disallowed" directories anyway.

100 Years

[BringsApples]

We as humans are not able to "remember" back further than 100 years. I mean that you cannot get any information from anyone that would give you a clear, practical understanding of the mindset from 100 years ago. You can go ask your grandparent(s) things about the past, but the vocabulary that they use more than likely won't fit your vocabulary and therefor you will not be able to get the understanding that they're trying for. Maybe 100 years is to small, but it can't be far from the real number, plus it's nice and round ;)

In this way, our society(s) are going through life sorta like that movie Memento. All that has to happen is a slight variation of the real story, that would produce the same basic result, but with a new context - Christopher Columbus "discovered" America comes to mind. Perhaps the powers that be depend on this, and are looking to make that number (100 here) smaller.

Re:100 Years

[Intrepid imaginaut]

You can go ask your grandparent(s) things about the past, but the vocabulary that they use more than likely won't fit your vocabulary and therefor you will not be able to get the understanding that they're trying for

What?

Re:100 Years

[BringsApples]

Really my point was that the context cannot be handed down without some distortion in understanding, but I'll try to paint the picture better.

I was born in 1976, so I remember having a "phone line" to the house, and all phones in the house were devices that were there in order to use that one line. This means that if the phone rang, one person of the household answered it, and then routed the call to whoever it was for. The context behind leaving someone alone to talk on the phone to their boyfriend/girlfriend/boss/whatever, not allowing others to use the phone as you wait for an important call, not having call-waiting, or caller-ID, can be totally lost to today's kids of today. 'How we spend time in today's world' has a contextual change to it from generation to generation. This context is embedded into the fabric of the language of the day, and so it's hard to "convert" into a context that is 100 years older or younger.

However, on the other hand, in Ireland a lot of their history is preserved in songs. Since songs have a way of leaving you with a feeling, they also have a way of preserving some relevant context from the writer of the song, from the time of writing - see how music changes over time.

Anyway, I hope that cleared it up a little bit.

Re:100 Years

[Intrepid imaginaut]

I was born in 1976, so I remember having a "phone line" to the house, and all phones in the house were devices that were there in order to use that one line. This means that if the phone rang, one person of the household answered it, and then routed the call to whoever it was for. The context behind leaving someone alone to talk on the phone to their boyfriend/girlfriend/boss/whatever, not allowing others to use the phone as you wait for an important call, not having call-waiting, or caller-ID, can be totally lost to today's kids of today. 'How we spend time in today's world' has a contextual change to it from generation to generation. This context is embedded into the fabric of the language of the day, and so it's hard to "convert" into a context that is 100 years older or younger.

By that logic people wouldn't understand anything written from 1913. The society of your parents and grandparents (and great great great grandparents) wasn't nearly as alien as you seem to think. Seriously, go read some HP Lovecraft, it's perfectly clear.

However, on the other hand, in Ireland a lot of their history is preserved in songs. Since songs have a way of leaving you with a feeling, they also have a way of preserving some relevant context from the writer of the song, from the time of writing - see how music changes over time.

Again, what? History in Ireland is preserved in books, just like anywhere else. Or do you mean people randomly burst into song to convey the historical context of their comments?

Re:100 Years

[BringsApples]

I'll give it another go...

Back in the day, people used to burn women alive, for being a witch, and the entire town used to watch. We know that is true because of what history books tell us it is. However we're not able to actually understand the point of view that would allow for such an event in today's world. We dismiss what they did as stupidity and leave it at that, and in that same way, we're not able to predict the future in such a way that we feel now, what will later be dismissed as stupidity.

Don't forget that this all has to do with burning (removing, deleting...) books (manuscripts, computers/data...), and nothing to do with actual time-travel or anything like that. In pointing out that we read the past, you only drive my point further, because if they are trying to control the data, which that's what this is about, then they're doing so for the future.

Every bloody time.

[Anonymous Coward]

Every bloody time I see some inane web related thing being done by obvious technically impaired judgement here in Britain, it's associated with the same name. Point in case: David Cameron is a bloody idiot.

Not in the USA!

[edibobb]

In the U.S., politicians post speeches full of lies online, and nobody cares. I'm not sure if this is because everybody believes the lies, or because nobody believes the politicians.

http://www.seattlepi.com/national/article/Rumsfeld-denies-making-claims-Iraq-had-WMDs-1202942.php

http://www.youtube.com/watch?v=CU0m6Rxm9vU

Re:Not in the USA!

[Anonymous Coward]

It is because people can only talk about what is in the news, and they are the news and the news caster.

It is very hard to gauge how strong a political belief is in the US. A lot of land and a lot of people spread out across it. If people can find others that share the same voice before disillusionment kicks in, then we too can have larger political movements. There are so many small interest parties, that they drown out each other and cannibalize their efforts.

Re:Not in the USA!

[Anonymous Coward]

What a fucking sack of shit! He ought to be jailed as a war criminal! How do we get started? I've already written a letter to my senator and sent it through snail mail, and notified my friends and family about the issue. Anything else? What have you done?

Re:Not in the USA!

[hey!]

In the U.S., politicians post speeches full of lies online, and nobody cares. I'm not sure if this is because everybody believes the lies, or because nobody believes the politicians.

You can't describe political behavior with two-state logic. For example one of the most desirable states to entrain your followers into might be described thus: "I don't believe what the politicians say, but I'll act as if I do believe them." This is ideal because it relieves the politicians of the drudgery of dreaming up *plausible* lies.

One of the surest signs of this intermediate belief state is when you hear somebody, when a politician he supports is caught, respond with "they all do it." This only *appears* to be a condemnation of the politician in question, but the person doesn't mean "They all do it, therefore I'll withdraw support from both sides and look for someone less despicable." It means "They all do it, therefore I will pretend none of this has happened."

A related state of intermediate belief is "Everyone says...". This is where the person in question believes that all politicians are liars, but steadfastly refuses to acknowledge the political origin of political lies. Also known as "echo chamber epistemology".

Newspaper print can't be deleted

[Anonymous Coward]

It's amazing that politicians want to delete their broken promises from the web so they can lie more effectively. Good thing there are libraries that keep old newspapers, and hopefully always will.

Re:Newspaper print can't be deleted

[Anonymous Coward]

They'll burn all those books and newspapers.

Re:Doesn't that kinda defeat the point of the arch

[pixelpusher220]

Indeed this is ridiculous that the IA would retroactively remove stuff though as you say hopefully just disable access instead. Even then, why would they keep stuff they aren't displaying? It's an 'archive' and should reflect how stuff 'was' at the time; legalities of that obviously being quite murky and hard to defend against expensive lawsuits, but still.

The problem is career politicians

[Anonymous Coward]

The problem is career politicians who accept campaign contributions and favors. Many people call them bribes. The problem is it corrupts politics and ultimately serves themselves, not the people.

We need term limits for US congress!

The only possible way to start getting out of this mess are Liberty Amendments!!! Check it out.

Re:The problem is career politicians

[Anonymous Coward]

You realize that this is not a US story, right?

Re:The problem is career politicians

[easyTree]

What is this 'non-US' concept of which you speak?

Re:The problem is career politicians

[Anonymous Coward]

No term limits! We have them in California, and they suck. All you end up with are the political parties directly controlling legislation. The parties are then the ones taking the bribes and special interest money.

Look at the Senate. 6 year terms. That's _good_. There are more independent minds in the Senate then just about anywhere else, except perhaps the House of Lords in England. But they're _too_ level headed, so the political parties in the UK want to get rid of the Lords entirely.

Re:The problem is career politicians

[niks42]

Small problem with your sig there ..

Firefox can't establish a connection to the server at isohunt.com

Re:Doesn't that kinda defeat the point of the arch

[pixelpusher220]

Actually no, those speeches don't seem to exist on the party website now either.

we have always been at war with eastasia

[Nadaka]

get with the pogrom.

Re:Doesn't that kinda defeat the point of the arch

[Bohnanza]

Sounds like somebody needs to archive the archive.

Re:Doesn't that kinda defeat the point of the arch

[Arthur Dent '99]

I apologize for my mistake. Until just a few minutes ago, I was unaware that the Internet Archive agrees to RETROACTIVELY honor a robots.txt file. So once a robots.txt file restricts access to content, they voluntarily remove access to previously archived content from the archive. Here's the related item from their FAQ:

Some sites are not available because of robots.txt or other exclusions. What does that mean?

The Internet Archive follows the Oakland Archive Policy for Managing Removal Requests And Preserving Archival Integrity

The Standard for Robot Exclusion (SRE) is a means by which web site owners can instruct automated systems not to crawl their sites. Web site owners can specify files or directories that are disallowed from a crawl, and they can even create specific rules for different automated crawlers. All of this information is contained in a file called robots.txt. While robots.txt has been adopted as the universal standard for robot exclusion, compliance with robots.txt is strictly voluntary. In fact most web sites do not have a robots.txt file, and many web crawlers are not programmed to obey the instructions anyway. However, Alexa Internet, the company that crawls the web for the Internet Archive, does respect robots.txt instructions, and even does so retroactively. If a web site owner decides he / she prefers not to have a web crawler visiting his / her files and sets up robots.txt on the site, the Alexa crawlers will stop visiting those files and will make unavailable all files previously gathered from that site. This means that sometimes, while using the Internet Archive Wayback Machine, you may find a site that is unavailable due to robots.txt (you will see a "robots.txt query exclusion error" message). Sometimes a web site owner will contact us directly and ask us to stop crawling or archiving a site, and we endeavor to comply with these requests. When you come accross a "blocked site error" message, that means that a siteowner has made such a request and it has been honored.

Currently there is no way to exclude only a portion of a site, or to exclude archiving a site for a particular time period only.

When a URL has been excluded at direct owner request from being archived, that exclusion is retroactive and permanent.

Re:Doesn't that kinda defeat the point of the arch

[pixelpusher220]

couple that with the google cached copy of the site has a 'search for speeches' section which now is, interestingly enough, missing as well.

Re:Doesn't that kinda defeat the point of the arch

[asmkm22]

So there's no actual internet archive? How was this not planned for years ago?

Re:Doesn't that kinda defeat the point of the arch

[Zedrick]

That really sucks. And explains why I've not been able to find older versions of my own websites.

standing by their words

[bzipitidoo]

Students usualy want to hide F's. Don't want to look stupid.

Wonder what these conservatives are trying to hide? Not much point trying to hide their stupidity. Everyone already knows that about them.

Re:Taking a Page from Obama?

[Anonymous Coward]

Is that actually supposed to be something easy to find as it is? I'd love to read some of my old papers from my school years! Is there some kind of big archive somewhere??

Re:Doesn't that kinda defeat the point of the arch

[Anonymous Coward]

Sounds like somebody needs to archive the archive.

Ah yes, but then will need an archive to archive all the archives that are not archived in the archive. Are you starting to see the problem?

Ask the NSA

[Anonymous Coward]

Surely, they have a copy somewhere.

Re:Doesn't that kinda defeat the point of the arch

[LocalH]

It fully explains it. Someone bought up the domain that you were hosted on previously, added a blanket disallow in robots.txt, and suddenly all your old stuff is gone.

Re:Doesn't that kinda defeat the point of the arch

[LocalH]

I'm pretty sure I read somewhere that they have a lot of stuff that isn't publicly available on their website for one reason or another. Don't have a citation though.

Only partially. (Also a wishlist.)

[Ungrounded Lightning]

Indeed this is ridiculous that the IA would retroactively remove stuff though as you say hopefully just disable access instead.

I think the archive actually does just suppress access rather than purge the actual data, so they can again display it once copyright runs out (if it ever does...).

I also think the point is that newbies may not know about robots.txt and that even an experienced webmaster might accidentally allow access to something private long enough for it to get archived, or receive and honor a takedown notice, so this allows the correction of the error.

It's an 'archive' and should reflect how stuff 'was' at the time; legalities of that obviously being quite murky and hard to defend against expensive lawsuits, but still.

That's why. They have limited funds and need them to buy more disks and stuff, not fight lawsuits. If the choice is not display some stuff or go broke and not display anything, the choice is also obvious.

I wish, though, that they were able to detect when a domain changed hands and not honor robots.txt requests retroactively past the boundary. IMHO a new owner is a new web site that happens to have the same name.

Especially: I wish domain name parking sites didn't put up robots.txt files that cause the archive to immediately purge/hide the previous owners' content. I've lost access to a lot of content from dead sites that way. (It also keeps the owners from rescuing their old content if they don't have personal backups.)

If you're not doing anything wrong

[hduff]

what do you have to fear? 8)

Re:Doesn't that kinda defeat the point of the arch

[Zedrick]

No, I added the robots.txt myself :-\

The domains are still mine, just took them with me to the different webhosts I've been working for.

OTOH, nothing of value has been lost, just wanted to know exactly what I wrote about Seven of Nine 13 years ago.

This should have been predicted.

[idontgno]

In fact, it was predicted. It was a particularly sharp observer of English politics who coined the phrase "memory hole".

1984

[Anonymous Coward]

Speeches deemed double plus un-good must be corrected.

He who controls the present

[Anonymous Coward]

controls the past. He who controls the past controls the future.

Captcha: Ruthless

Re: Doesn't that kinda defeat the point of the arc

[iamhassi]

Not much of an archive if they delete the past because someone says it should be deleted. Even Wikipedia allows you to go back and see all changes to an article.

The Press

[Anonymous Coward]

This can only completed with the complicity of a corrupt Press. Which is not to say that all publishers have an activated altruism circuit. It only acknowledges the idealism promoted by those who believe the Press as a whole comprises a valuable institution.

Personally, I believe the future of history is imperiled more by the fragility and corruption in human political philosophies than by the impermanence of digital storage or the evolution of electronic storage media and formats.

How about Hansard?

[T.E.D.]

Interesting. Are they going back and censoring Hansard too?

Ummmm... So have the Labour Party

[Anonymous Coward]

This isn't news or at least not in the way that the author of the original article means. The main opposition party, Labour, had already done the same thing.

AC

Re:Doesn't that kinda defeat the point of the arch

[mrclisdue]

Ah yes, but then will need an archive to archive all the archives that are not archived in the archive. Are you starting to see the problem?

That should work, as long as there is a standard.

cheers,

Rubish.

[Anonymous Coward]

Another flamer/fool/idiot that sees a conspiracy in every hooker that he buys.

not deleted

[Anonymous Coward]

They don't delete it, merely delist it from the Wayback Machine. Ask them nicely and I bet they'd turn it into a special collection on the archive proper.

hyperbole.txt

[necro351]

Apparently using the common practice of putting up a robots.txt to ask crawlers to stay away for better political messaging control is actually an Orwellian thought-control power grab. Obviously Cameron was talking about other news/archive sites keeping a permanent record of his speeches, since that is the only way it could work for any party in power. Do we really expect a politically motivated website, of any party, to keep an honest and complete record of its positions and speeches for indefinite periods of time for public scrutiny?

The way this headline read, I half expected to hear about Cameron's administration sending take-down notices to bloggers for dredging up campaign promises, but no, his party's website just updated its robots.txt, sheesh!

It gets worse

[clickclickdrone]

I just tried to complain to my MP about this but it seems he's blocked me on Twitter. I guess that's it then, we are living in a fascist state.

Re:It gets worse

[niks42]

http://www.theyworkforyou.com/ will find you a page where you can mail your MP and they will answer. I complained to my MP about the police use of Terror laws to detain David Miranda, and I know it got to him as he replied. He did reply saying it was a police matter and nothing to do with Parliament, but hell, it struck home! Power to the People ..

Re:Doesn't that kinda defeat the point of the arch

[nitehawk214]

No, I added the robots.txt myself :-\

The domains are still mine, just took them with me to the different webhosts I've been working for.

OTOH, nothing of value has been lost, just wanted to know exactly what I wrote about Seven of Nine 13 years ago.

Well that is the thing... sometimes are better off lost. Apparently the Internet Archive is testing the "cannot be unseen" principle.

Winston Smith's job / 1984

[volvox_voxel]

It makes Winston Smith's job at the Ministry of Truth more difficult if there are old archives available..

Re:Winston Smith's job / 1984

[Anonymous Coward]

Wrong - you need all the old data so in the future you can effectively create fake information from that era.

Amateurs...NEVER throw away data.

Re:Winston Smith's job / 1984

[volvox_voxel]

This naturally implies that the Ministry keeps the data, but conveniently expunges public records. There can only be one Truth; that which is officially sanctioned by the Ministry.

Re:Doesn't that kinda defeat the point of the arch

[nitehawk214]

So there's no actual internet archive? How was this not planned for years ago?

People mistakenly thought the Internet Archive was an actual archive of the internet, instead of the "Internet Archive of Uncensored Things". (until today i was one of these people)

Perhaps now this will either make IA do the right thing, or perhaps someone will step up to the plate.

CDROM Lifetime

[Roger W Moore]

Try a few decades in the future. After that the data will be gone - the dye in writable CDs and DVDs does not last beyond that and even then when stored in ideal conditions.

Re:CDROM Lifetime

[Zeromous]

I have been a recovery specialist.... i was only joking :)

Re:Doesn't that kinda defeat the point of the arch

[Garridan]

Suppose they archive some kiddie porn. And then they let people download it. And those people make donations to them. And there's more than one person involved in administering the archive. That's conspiracy to distribute child porn for financial gain. No shit they censor the archive, dumbass. For self-preservation, if nothing else.

Like the BBC destroying most of the 1960s archive

[Anonymous Coward]

You KNOW of the BBC's destruction of most of its 1960s video archive through the stories of the 'LOST' Doctor Who episodes, but you do not know the reason behind the destruction. The usual filthy shills spread NONSENSE about 're-use' of video tapes, even though the archival shows that were destroyed were either on un-reusable FILM, or on long obsolete BLACK-AND-WHITE video tape standards (the destruction occurred long after the BBC had fully converted to colour production).

The actual reason for trashing the archive at the time was given as "cost"- apparently the Earth's most cash rich broadcaster just couldn't afford the 'massive cost' of storage, and the handful of people employed to look after the archive. Also the entertainment unions actively encouraged the destruction of material that could otherwise be repeated in place of 'new' output. The BBC, at the time of the destruction, had a '3 times only' rule for the maximum number of broadcasts of any given episode of a show. The BBC argued to the public that the archival footage was pointless, since it could never be rebroadcast anyway, and everyone hated old fashioned black-and-white footage.

The REAL reason for destroying the BBC archives was the SAME as the reason that lies behind UK politicians attempting to push previous speeches and manifesto material down the MEMORY HOLE. Remember where Winston worked in Orwell's 1984? Orwell (with the unfortunate real name of Blair) was accused of writing his books as metaphors about communist Russia, but actually Orwell was writing about British Fabian 'socialism'. Stalinist communism was war too crude an ideology to need sophisticated deconstruction. But Orwell was intimated aware of the far more insidious movements within British politics and the BBC.

Orwell's theme is that the sheeple (well, actually Orwell dismissed the working classes as irrelevant and without influence, so his 'sheeple' were entirely the middle-classes) have the memory of a goldfish, so monsters MUST take full control of all media outlets to ensure the sheeple have a CONSTRUCTED view of 'history' at all times. He predicted that we would move from "History is written by the victors" to "History is created entirely as a fiction to suit the current needs of the ruling monsters".

Look, for years the 'Liberal' party in the UK stated in manifesto after manifesto that if elected (and this party has been a distant third for most of the last five decades plus) it would have only ONE major policy, to spend whatever it took to give absolute priority to education. When the Liberals gained real power a few years back, in alliance with the Conservative party, the first act of the Liberals was to raise the cost of university education in England to almost the highest level in the world. Of course politicians have no desire for you to be reminded of what they said and promised only a few years back, let alone decades ago.

To get elected, the Liberals gave a party political broadcast where their leader PROMISED on screen to not raise university fees. How many UK sheeple now remember this broken promise. I'd bet the figure is far less than ten percent and falling.

It so happens that the leader of the Liberals, a vile war monger named Clegg, is actually a Tony Blair loyalist, and while he has the formal description of 'deputy' Prime Minister, he is actually the de facto Prime Minister, while the laughable joke Cameron, the guy who carries the title of Prime Minister, is just one of Blair's dingleberries. Clegg has overseen the most radical attacks on the rights and freedoms of UK citizens in the last 50+ years, and has particularly focused on attacking the young and the poor. You Americans might not see this as odd, since you do not know that every year, at the party conference, the Liberals had seemingly been supporting the exact OPPOSITE agendas. Once in power, the Liberals switched from being more LEFT-wing than New Labour to more RIGHT-wing than the Conservatives.

Without a memory hole constantly eating inconvenient facts, e

That would let me know who not to vote for.

[ralphaostrander]

That is right up there with the scummiest political moves I have heard of.

Interesting Timing.

[gallondr00nk]

They recently made an announcement that contrary to their election pledge (again) they would look to make economic austerity permanent, instead of scaling back on cuts once the economy recovered.

I would have a bet that in a couple of days if the pressure is still on that they'll either claim it was a mistake, or hackers. Who knows, it might even be true.

I could easily believe that they are stupid enough to think that deleting a few pages erases the past.

Off the internet?

[Anonymous Coward]

You can't get something off the internet. That's like getting pee out of a pool, once its in there, its in there.

Don't approve their own messaging?

[HalAtWork]

So they don't even approve of their own messaging? Seems very unconfident. Why should anyone believe in them if they don't even believe in themselves?

They don't crawl your pages

[Anonymous Coward]

They just slurp them up from an unencrypted http connection when some visitor requests them.

If its a visitor they're particularly interested in, they might go to the trouble to compromise that visitor's PC and/or MITM the SSL connection their PC forms to your server. So even encrypting your traffic won't necessarily stop them from collecting all of your content, just in case it turns out several years from now that your content has some nefarious connection to terrorists or drug dealers, or perhaps some politician they want to blackmail, or (eventually) to some regular citizen with "unpopular" political views who they want to round up or add to secret watch lists for extra harassment/scrutiny.

Robots.txt

[LeadSongDog]

The Internet Archive says that it subscribes to the The Oakland Archive Policy which for |requests by governments" says:

Archivists will exercise best-efforts compliance with applicable court orders Beyond that, as noted in the Library Bill of Rights, 'Libraries should challenge censorship in the fulfillment of their responsibility to provide information and enlightenment.'

Seems like this may just have slipped past them. Let's make sure they know they need to sort it out... Surely they only removed it from the Wayback Machine, not from the archive itself.

What the hell is it with Conservatives?

[msobkow]

Here in Canada, Conservative PM Harper has taken heat lately for breaking all the links on our government's historical archive of the legislation that's been posted for the past decade or two. It's just... gone. The entire archive, except for maybe the past 5 years worth.

That archive is public government information, not Conservative property.

Re:What the hell is it with Conservatives?

[Anonymous Coward]

Plus you would think that "Conservative Data Policy" would mean keeping data around!

captcha: misuse

Re:Like the BBC destroying most of the 1960s archi

[volvox_voxel]

I wonder what chance that some of this material is available from other sources? It's a pity to read about destroyed archives. I've wanted to see some of David Attenborough's older material from the 50ies and 60ies that I read was very well made. The BBC had such great material, like the "Voyage of the Beagle", etc. PBS also had a lot of great programming they're also not releasing; stuff I'm sure they have, but won't distribute.

Re:Doesn't that kinda defeat the point of the arch

[fnj]

Nice try, but I'm not buying it. I'd like to think as a non producer or non creator of content their liability would be covered under the same principle as ISP exemption. Do the jackboots come for Comcast because someone peer-to-peers or receives via email a picture of Candy Sweet allegedly taken one day before her 18th birthday? I don't think so. Why should the Internet Archive be treated any differently?

Perhaps I am naive, but one thing I am not. I am not wrong on the principle. That is the way it SHOULD be.

Robots.txt should be subject to exactly the same principle as the US Congress. No retroactive censorship, just the way ex post facto laws are unconstitutional.

Come to that, I am not clear on what the legal basis of robots.txt is. To the best of my knowledge there is no legal basis whatsoever for prosecuting non-compliance; i.e., it is a gentlemen's agreement; no more. Am I mistaken?

Re:Only partially. (Also a wishlist.)

[Trogre]

Especially: I wish domain name parking sites didn't put up robots.txt files that cause the archive to immediately purge/hide the previous owners' content. I've lost access to a lot of content from dead sites that way. (It also keeps the owners from rescuing their old content if they don't have personal backups.)

This.

Today I have lost a lot of respect for the Internet Archive.

Internet Archive deletion?

[nurb432]

How does that work?

Re:Internet Archive deletion?

[Anonymous Coward]

The internet archive adheres to robots.txt, even retroactively.

This is why

[oldfogie]

more people should frequent reddit.com/r/datahoarder

Torrents only work if the data is available somewhere.

Re: Doesn't that kinda defeat the point of the arc

[WhiteDragon]

Not much of an archive if they delete the past because someone says it should be deleted. Even Wikipedia allows you to go back and see all changes to an article.

except for page deletion. In that case, only certain people can view the history.

Re:Doesn't that kinda defeat the point of the arch

[Garridan]

Personally, if I ever found myself in possession of child porn, I'd get rid of it as fast as possible. I suggest the same to you. The jackboots come after whoever the fuck they want to, and find evidence of wrongdoing after the fact... if nothing else, "lying to the police during an investigation" will always stick no matter how small the lie. The less child porn you (knowingly or not) distribute, the fewer pretexts the jackboots have to knock down your door and break your server hardware in an "attempt to gather evidence" and shoot the family dog in front of your kids.

You are not mistaken about the legal power of a robots.txt: it's a gentlebot/gentleadmin agreement. But you are mistaken about the implied legal threat behind one. If you want things done the way you want, do it yourself. The only better thing than an archive is two archives maintained by different people in different political climates.

Winston has seen this before.

[Anonymous Coward]

As short a time ago as February, the Ministry of Plenty had issued a promise (a "categorical pledge" were the official words) that there would be no reduction of the chocolate ration during 1984. Actually, as Winston was aware, the chocolate ration was to be reduced from thirty grams to twenty at the end of the present week. All that was needed was to substitute for the original promise a warning that it would probably be necessary to reduce the ration at some time in April. -- 1984

Re:Doesn't that kinda defeat the point of the arch

[fnj]

This is a kind of ridiculous excursion off point. Nobody is "distributing" anything in the context of the discussion.

What I find perplexing is why the Internet Archive chooses, on their own initiative, to make an ex post facto censorship based on robots.txt. Robots.txt tells you "be nice and don't process the site". It's enough to stop processing the site when you see the request. It's an absurdity to go back retroactively and undo what was done in the past, before there was a robots.txt request. The Internet Archive's policy is absurd. It's batty.

I certainly couldn't agree more, there ought to be a dozen or a hundred internet archives, geographically distributed. It ought to be impossible for anyone to make data that was once free go away.

Actions like this Built in by Definition:

[SinisterRainbow]

I always thought this was built into the Conservative philosophy by definition, at least the far Right. You have principles you try to uphold for as long as possible, by any means necessary.

too bad nobody made a print record

[capaslash]

like in a print newspaper or magazine or something.

Funding opportunity

[jago25_98]

Many might think "problem"...
    I think "opportunity"!

I've written before how the archive can do a lot more than it presently does. For a start, it could be used as evidence. So how do we make money out of this? Well, the archive only goes so far in what it stores. If there is a page that we want to be stored for longer and more securely then we should be able to pay for that. More securely? How's about paying for it to be backed up to a p2p network, similar to SatoshiProof. You then charge a small fee extra for the service.

Of course, you could do the same for following or not following robots.txt but this is closer to messing with the integrity of archive.org

Just make sure this upper tier in the 2 tier system is genuine new functionality.

Scrape a copy from another archive...

[Tim12s]

Its probably worth keeping a copy for historical purposes.

Re:Doesn't that kinda defeat the point of the arch

[dargaud]

I agree that it's ridiculous. The access should be granted with the robot.txt that was valid _at the time_. Just archive it together with the website.

"responsible," "capitalist," and "hard-working"

[Any Web Loco]

Pick any one.

Re:Doesn't that kinda defeat the point of the arch

[Anonymous Coward]

Until someone tries to decide if the archive of unarchived archives archives itself or not.

Better than the U.S.

[Xzing_Quippo]

At least in Britain they have to make an attempt to change history. Here in the U.S.A. our attention spans are so short that it doesn't matter what stupid, appalling, ridiculous thing any politician says it will be forgotten by the masses and largely ignored by the media shortly after it comes out. If it's not making the media money, or if it's not a video news clip, or something with small, simple words thats no bigger than a postage stamp to read, no one will even know it ever existed.