Caching Content and the Shrinking Web?

kill-hup asks: "I know the issue of caching linked pages has been discussed many times here on Slashdot, but the majority of those discussions centered around the 'Slashdot Effect' knocking remote content servers off-line. How does the ethic/legality issue change, if any, when we're talking about information that once was available but now has moved or disappeared from the provider's site?"

"I run a small discussion-oriented site patterned after Slashdot; small story blurbs and discussion center around links to external content. From time to time we post our own content, but the vast majority involves links to articles on other sites. This structure obviously relies heavily on the external pages being available for our visitors so they can understand the issue or viewpoint being highlighted.

Just before the new year, I took a look back at story entries that had been posted throughout 2002 and found it interesting to note that a large portion of the linked content was no longer available/had moved/etc. In the short term, this is not an issue; most outside material tends to remain available for the length of an active discussion. The problem I see is visitors coming to the site by way of search engines to stories whose linked content no longer exists. Without the background provided by the referenced story link, the discussion or quick blurb may not make sense or may not fulfill the request that brought the visitor to us.

I know I am not alone in this quandary and that others must have run into this before. While I respect the copyright of the external content providers and do not wish to get into the whole issue of lost advertising revenue for them if I were to cache a local copy, I'm curious what other users are doing to mitigate this problem."

Comments

Re:Mirror of your mirror, just in case...

[clambake]

Well, technically wouldn't the mirror of the mirror look more like this:

[ sihT ot ylpeR ]
".melborp siht etagitim ot gniod era sresu rehto tahw suoiruc m'I ,ypoc lacol a ehcac ot erew I fi meht rof eunever gnisitrevda tsol fo eussi elohw eht otni teg ot hsiw ton od dna sredivorp tnetnoc lanretxe eht fo thgirypoc eht tcepser I elihW .erofeb siht otni nur evah tsum srehto taht dna yradnauq siht ni enola ton ma I wonk I

.su ot rotisiv eht thguorb taht tseuqer eht llifluf ton yam ro esnes ekam ton yam brulb kciuq ro noissucsid eht ,knil yrots decnerefer eht yb dedivorp dnuorgkcab eht tuohtiW .stsixe regnol on tnetnoc deknil esohw seirots ot senigne hcraes fo yaw yb etis eht ot gnimoc srotisiv si ees I melborp ehT .noissucsid evitca na fo htgnel eht rof elbaliava niamer ot sdnet lairetam edistuo tsom ;eussi na ton si siht ,mret trohs eht nI .cte/devom dah/elbaliava regnol on saw tnetnoc deknil eht fo noitrop egral a taht eton ot gnitseretni ti dnuof dna 2002 tuohguorht detsop neeb dah taht seirtne yrots ta kcab kool a koot I ,raey wen eht erofeb tsuJ

.dethgilhgih gnieb tniopweiv ro eussi eht dnatsrednu nac yeht os srotisiv ruo rof elbaliava gnieb segap lanretxe eht no ylivaeh seiler ylsuoivbo erutcurts sihT .setis rehto no selcitra ot sknil sevlovni ytirojam tsav eht tub ,tnetnoc nwo ruo tsop ew emit ot emit morF .tnetnoc lanretxe ot sknil dnuora retnec noissucsid dna sbrulb yrots llams ;todhsalS retfa denrettap etis detneiro-noissucsid llams a nur I"

"?etis s'redivorp eht morf deraeppasid ro devom sah won tub elbaliava saw ecno taht noitamrofni tuoba gniklat er'ew nehw ,yna fi ,egnahc eussi ytilagel/cihte eht seod woH .enil-ffo srevres tnetnoc etomer gnikconk 'tceffE todhsalS' eht dnuora deretnec snoissucsid esoht fo ytirojam eht tub ,todhsalS no ereh semit ynam dessucsid neeb sah segap deknil gnihcac fo eussi eht wonk I" :sksa puh-llik

.tped tcatni-txetnoc-eht-gnipeek eht morf
3002 41 hcraM yadirF -- MA 55:20 no ffilC yb detsoP
?beW gniknirhS eht dna tnetnoC gnihcaC

.msiragialp rof em eus dna ssa ym ot pu pmuj nac sremalf erohw-amrak-atem esoht lla woN .rohtua lanigiro eht morf noissimrep eht tuohtiw derorrim ylsselemahs I tnetnoc eht si ereH

(/gro.todhsals//:ptth)
(4789055#) MA90:21@ ,41 hcraM yadirF no (421452) essj yb
(1:erocS) devomer dna dettodhsals s'ti esac ni rorriM
devomer dna dettodhsals s'ti esac ni rorriM