Caching Content and the Shrinking Web?

kill-hup asks: "I know the issue of caching linked pages has been discussed many times here on Slashdot, but the majority of those discussions centered around the 'Slashdot Effect' knocking remote content servers off-line. How does the ethic/legality issue change, if any, when we're talking about information that once was available but now has moved or disappeared from the provider's site?"

"I run a small discussion-oriented site patterned after Slashdot; small story blurbs and discussion center around links to external content. From time to time we post our own content, but the vast majority involves links to articles on other sites. This structure obviously relies heavily on the external pages being available for our visitors so they can understand the issue or viewpoint being highlighted.

Just before the new year, I took a look back at story entries that had been posted throughout 2002 and found it interesting to note that a large portion of the linked content was no longer available/had moved/etc. In the short term, this is not an issue; most outside material tends to remain available for the length of an active discussion. The problem I see is visitors coming to the site by way of search engines to stories whose linked content no longer exists. Without the background provided by the referenced story link, the discussion or quick blurb may not make sense or may not fulfill the request that brought the visitor to us.

I know I am not alone in this quandary and that others must have run into this before. While I respect the copyright of the external content providers and do not wish to get into the whole issue of lost advertising revenue for them if I were to cache a local copy, I'm curious what other users are doing to mitigate this problem."

Comments

Knowledge is power

[quintessent]

Ethically, we need to keep the channels of knowledge open. If it was public knowledge at one time, it must remain so. Otherwise, we begin to foster an Orwellian world where any number of Ministries of Truth can hide history and rewrite it as needed. A web page is a record of the world at a given time. Just as libraries keep old journals for reference, we need to be able to reference the web of the past.

Legally, I fear that litigation like Scientology vs. the Wayback Machine will begin to erode this protection. Having a monopoly on knowledge gives an entity the power to bring the masses into submission. We must let truth prevail.