Slashdot Mirror
When the Internet Archive Forgets (gizmodo.com)
A reminder that Internet Archive's Wayback Machine, which many people assume keeps a permanent trail and origin of web-content, has little feasible choice but to comply with DMCA takedown notices. As a result of which, a portion of the archive of things people submit to the website continues to quietly fade away. Gizmodo: Over the last few years, there has been a change in how the Wayback Machine is viewed, one inspired by the general political mood. What had long been a useful tool when you came across broken links online is now, more than ever before, seen as an arbiter of the truth and a bulwark against erasing history. That archive sites are trusted to show the digital trail and origin of content is not just a must-use tool for journalists, but effective for just about anyone trying to track down vanishing web pages. With that in mind, that the Internet Archive doesn't really fight takedown requests becomes a problem. That's not the only recourse: When a site admin elects to block the Wayback crawler using a robots.txt file, the crawling doesn't just stop. Instead, the Wayback Machine's entire history of a given site is removed from public view.
In other words, if you deal in a certain bottom-dwelling brand of controversial content and want to avoid accountability, there are at least two different, standardized ways of erasing it from the most reliable third-party web archive on the public internet. For the Internet Archive, like with quickly complying with takedown notices challenging their seemingly fair use archive copies of old websites, the robots.txt strategy, in practice, does little more than mitigating their risk while going against the spirit of the protocol. And if someone were to sue over non-compliance with a DMCA takedown request, even with a ready-made, valid defense in the Archive's pocket, copyright litigation is still incredibly expensive. It doesn't matter that the use is not really a violation by any metric. If a rightsholder makes the effort, you still have to defend the lawsuit.
In other words, if you deal in a certain bottom-dwelling brand of controversial content and want to avoid accountability, there are at least two different, standardized ways of erasing it from the most reliable third-party web archive on the public internet. For the Internet Archive, like with quickly complying with takedown notices challenging their seemingly fair use archive copies of old websites, the robots.txt strategy, in practice, does little more than mitigating their risk while going against the spirit of the protocol. And if someone were to sue over non-compliance with a DMCA takedown request, even with a ready-made, valid defense in the Archive's pocket, copyright litigation is still incredibly expensive. It doesn't matter that the use is not really a violation by any metric. If a rightsholder makes the effort, you still have to defend the lawsuit.
They should move to Canada as we have an exemption for archives which would allow the content to remain.
... when the Wayback Machine itself has been dropped into a memory hole.
Check your premises.
Get a charter from the Library of Congress, which can essentially bypass DMCA restrictions by fiat. The LoC usually seems pretty progressive about these things.
My Other Computer Is A Data General Nova III.
...science finds out that this reality has a log that can not be erased :O
We will have an Internet Archive server in Asgardia, and just screw every DCMA request in the world !!!!
DOOM ? : https://archive.org/details/do...
Apple II : https://archive.org/details/ap...
Arcade: https://archive.org/details/in...
DOS GAmes: https://archive.org/details/so...
if you deal in a certain bottom-dwelling brand of controversial content
I like how this insinuates that it's the "dark web" trying not to be blocked when it's political leaders, actors and other public personae (people very much out front and wanting to be seen) that go out of their way to delete their internet history when it contradicts with whatever they're pushing today so they can say "this has always been who I am!"
Archiving and to a greater point JOURNALISM (not "reporting" but actually chronicling and journaling the days' notable events in an objective manner) is an indispensable requirement for any person to become educated on a topic and to make an informed decision.
Eventually these things become history and are lost to current though until somebody digs through the archives to rediscover the truth. Except now we can make it go away with a keypress and, poof, we've always been at war with Eurasia.
Puzzle of the Pendulum at Archive.org
What is Winter Sunlight?
Working of Error
https://www.washingtonpost.com/politics/2018/11/29/key-takeaways-michael-cohens-new-plea-deal
1. There are conspicuous mentions of Trump and his family.
2. Putin’s spokesman appears to have helped cover this up.
3. This ties the Trump family’s efforts to the Russian government
4) The deal apparently died the day The Post broke a story about Russian hacking
https://www.huffingtonpost.com/entry/deutsche-bank-offices-raided_us_5c00331de4b027f1097bc8aa
https://www.nydailynews.com/news/politics/ny-pol-manafort-confidential-mueller-trump-giuliani-20181129-story.html
notice the obvious issues related to abuse. Imagine having a personal or non personal website - lets say slashcomma.org. Doesn't matter what you do with it - you store and share with the world publicly accessible information. You do not block webarchive and all this information gets archived from time to time in webarchive. You know you and public gets free backups. Medicine from "memento" type issues. Now imagine you go missing, disappeared, get hit by the bus, etc - you no longer run website. Domain gets expired. It gets registered by another party and they explicitly tell through robots.txt that new site is not subject to be browsed by such spiders. So all the backups and your hard work gets wasted into oblivion. In other words in that sense the new owner of domain is inheriting your intellectual property even though they shouldn't have. It's an exploit of changeling. Some sort of intelligent design by some sort of intelligence community of tar budgets. You know it's their world. The great digital divide that is. In other words they want you to believe that internet is fair and balanced, but in reality it's just a fair of series of tubes filled with balanced diet of carnival food such as yellow cakes.
He was never in office to begin with
Retard alert
So, someone requested, you remove a page — and you decide to comply. By replacing it with something like "Content removed by on date on request from such and such."
Requesting removals of evidence suddenly becomes less effective — an explicit record of removal may appear even more sinister, than whatever was there before...
In Soviet Washington the swamp drains you.
This is stupid, if there is a 'spirit' to the protocol, formalize it and enforce it.
Otherwise all you're saying is that it's as useless as do-not-track/do-not-call and that people are free to ignore it as they choose.
The internet has shown us time and time again, if you are relying on people voluntarily following the 'spirit' of something, it's 100% guaranteed they won't.
My take is that adding a robots.txt cannot possibly be retroactive ... as of the time it was crawled, you didn't have this in place ... therefore, you can't claim adding it after several years goes back in time and revokes what was public access to content. You've already published it for everyone to see, and the stuff which was already crawled was out there for with public access.
Fuck the 'spirit' of the protocol, do what the corporations do, follow the letter of the law and no more. Trusting to the good behavior and intentions of the internet is stupid.
What is it that separates some of us, who believe that a proper, immutable archive is more important to our species than copyright restriction, from those who feel otherwise?
Is it just money? Is that all it is? Or is it something deeper?
A government is a body of people notably ungoverned - AC
>"the robots.txt strategy, in practice, does little more than mitigating their risk while going against the spirit of the protocol."
Spirit of the protocol? I kinda disagree with that. If a site admin put up a robots.txt file, then they are clearly signaling they do not want the specified parts of the site crawled/archived/copied. It isn't just a directive to be convenient signaling to the crawler about what is a waste of time/load/bandwidth, but also a choice the admin made saying "these things should not be crawled" and for whatever reason the admin wants.
To me, the only controversial part would be- does having a robots.txt excluding something NOW mean that it should exclude things that had been "OK" in the past (because there was no exclusion back then). Personally, I tend to go with the interpretation that it means "now or in the past" (perhaps they changed their mind or forgot to put up a robots.txt initially). But that is certainly murky.
I think it is very hostile, and very much against the "spirit of the protocol" to ignore a robots.txt file. I could see where it might even have legal ramifications later (similar to a "no photography" sign in a store).
Some things do indeed need erasing though. I've yet to publish the paper because disclosure was only earlier this month. I found an entire ISP leaking publicly the physical addresses of virtual every single customer on their network (essentially EVERYONE was doxxed at once). This information is mirrored in the Wayback Machine (along with a few other archives). Part of the reason my paper isn't published yet is due to the ISP currently working with these archives to remove the sensitive information the ISP should never have published in the first place.
If the DMCA stymies free speech in practice, then it could be considered a violation of the 1st Amendment. Form a coalition to sue all the way up to the Supreme Court.
A similar situation has arisen for the recent "Online Sex Trafficking Act of 2017", which is so vague that it makes hosting any kind of online romantic discussion
or message group too risky. One could end up in jail because they don't police content tightly enough.
(Craigslist removed the "personals" discussion group because that Act. Ads for shady services now spill over into other discussion groups, often ruining them. Craig may end up in jail anyhow for not scrubbing hard enough.)
Both laws have "excessive side-effects" on legitimate free speech.
Table-ized A.I.
Eventually these things become history and are lost to current though until somebody digs through the archives to rediscover the truth. Except now we can make it go away with a keypress and, poof, we've always been at war with Eurasia.
There is more of an immediate need, since the ability to stealth edit a story after publishing it is too great a temptation to resist. There's been too many examples of 'reputable' news sources getting caught red handed doing this.
Anyway, an archive source that is subject to the hideously malformed DMCA is hardly an archive source at all.
Why did archive.org ever choose to respect robots.txt anyways? Archive.is doesn't. Sites shouldn't put content up if they don't want someone to see it.
You will find out, what those agreements were made for.
This happens all over the world.
Scientist already call it the "digital dark ages".
Fun fact:
There was a time, when Germany didn't have any such laws, but the UK did.
The UK's creative scene suffocated. While Germany's creative scene flourished so much, that that is, where it got its title "the land of poets and thinkers" from.
I recommend you look it up, and don't just believe an AC on the Internet.
And all, because some cokeheads didn't want to work for their money, but leech on artists, and then demand real money, not copies of money, but real money, that people actually had to work for, ... in return for mere copies.
Imagine us doing that: We'd be like: "Hey! I worked HARD for those $100 money! So you better accept 100 copies of my $100 bill of "labor property" as payment for this meal, or I'll go WAAAHHH, buy a law and force you to give me a TRILLION meals, while I get to call you a seafaring rapist thug!"
They are leeches and thieves. Nothing else.
I pay for concerts. For actual work. Because guess what: I had to work for it too!
I don't pay for copies. Unless I can pay with copies too. Period.
One which doesn't have a DMCA?
Seems like if you go to the archive for a forgotten page, they may not be able to retrieve it, but sure could have it in the archive.
And they could tell you why you can't see then you could challenge it if it is worth it to you.
That make the archive still serve it's primary mission which is archiving. (Retrieve is secondary. You can have one without the other, but not the reverse.)
That puts the cost burden for challenges in a more scalable place.
The courts can still provide specific retrievals over DCMA.
Truly 'free' international archive?
Because at this point in time it would require founding a new nation, replacing an existing small one in a coup, then getting internet access and funds to it to maintain the archive and expand its hardware base.
IA requires a *LOT* of hard disks. Basically Backblaze style storage pods. And unless you have access to a steady supply of those, plus internet, plus finances, plus a country with laws tailored to allow you to archive, you are FUBAR.
Why not store it, but not publish?
In mid-2017, Digital Trends reported that "Internet Archive will ignore robots.txt files to keep historical record accurate". At the time of the article, Archive.org already disregarded robots.txt files on U.S. government and military sites, but confirmed that they planned to "do this [ignore robots.txt] more broadly".
Did the expansion of this policy actually happen? Archive.org doesn't seem to have commented further.
DT Link
Let's not forget that there are other internet archives out there. I'm thinking specifically of archive.is, which has less willingness to acquiesce to takedown requests and robots.txt exclusions, but there are others out there that could surely use our support. A patchwork approach of several archives with differing approaches and goals (general, academic, scientific; text, image, video, etc.) could provide a more robust backup of the web. A single archive is a single point of failure.
UK websites are covered by the British Library
https://www.bl.uk/collection-g...
http://data.webarchive.org.uk/...