Using Google to Calculate Web Decay

scottennis writes: "Google has yet another application: measuring the rate of decay of information on the web. By plotting the number of results at 3,6, and 12 months for a series of phrases, this study claims to have uncovered a corresponding 60-70-80 percent decay rate. Essentially, 60% of the web changes every 3 months." You may be amused by some of the phrases he notes as exceptional, too.

Comments

Google's collection of the data

[Fucky the troll]

Are google claiming that they can check through the entire internet inside a timescale of 3 months, ready to check through again at the start of the next quarter?

Surely this can't be true. Check Google's cached pages - see the dates on there?

Google is turning into another history book.

Not exactly decay...

[QuantumFTL]

It seems to me that in a way, the web is like an organism, whose smaller constituents are constantly (or not so constantly, depending on the webmaster) renewing themselves. It's a truely adaptive medium, and thus drastic change in short times like this as interest shifts should be quite expected.

That said, this is one of the many ways in which Google is an invaluable tool for research. Not just finding information, but generating it. Thanks Google!

bill gates sucks...

[jnana]

For once, that is on topic. I'm glad to see that the phrase 'bill gates sucks' had the lowest decay rate of the phrases that the guy tested for.

Re:bill gates sucks...

[prizzznecious]

All this means, actually, is that the sites that would include the information "Bill Gates Sucks" are not being updated very often, or have little else to say.

It's an indicator of the dubious kind of context in which one finds such rash statements.

Re:bill gates sucks...

[Kierthos]

Actually (and unfortunately for any haters of the Evil that lies in the lands of Redmond) Headline News had this lovely little chart on recently, which showed public approval of several companies. Enron and Arthur Anderson had 9 and 11% approval ratings, respectively, while the big "winner" was Microsoft, with something like a 79% approval rating.

Let's face facts here. We might hate Microsoft, but the vast majority of people do not. Good? Bad? Indifferent?

Kierthos

blessed

[thanjee]

How long until all the cheesemakers have fully decayed and are no longer blessed?

I don't look forward to that day.

Long live cheese and cheese makers!

Web Death

[svwolfpack]

It would also be interesting to see how much of the web no longer exists... like at what rate the web is dying. God knows there's enough dead links out there...

Re:Web Death

[DarklordJonnyDigital]

Oh, most of the web is still around... it just looks like pages are decaying because every link you click has already been Slashdotted ;)

Re:Web Death

[Enocasiones]

Educators and "link rot".

In a paper to be published in the June issue of the Journal of Science Education and Technology, Brooks and Markwell likened the rate of link rot to the type of "extinction equation" commonly used to describe natural processes such as radioactive decay. They wrote that the hyperlinks in their study had an expected "half-life" of 55 months."

Also this, which is just a link from the previous article.

Easy! :)

(web's half-life -game -unreal -counter -gamers)

Re:Web Death

[digitalsushi]

we're an isp.. i remember the first time someone contacted me about this horrible thing.. they wanted us to redirect all our 404 traffic to a page that would spawn popup spam. seems like thats what half of my web browsing is these days. find a page with links. click a link. a window pops up, and one under. close both, and the main page says the page doesnt exist. *sigh* the next one will work, though.. although it too will spawn a few windows. it's disenchanting to work on these systems when most people are spoiling the experience with their spammy goo. (and no we never sold our 404 traffic). its kinda sad.. when i get to a plain old apache default error message these days, i get all teary eyed and remember the good old days.

now its all about finding open relays to megaphone your get rich quick idea that you copied from some other guy to 30 million people, praying that you get at least 40 back. course if you decide to bite just to mess with them, you find that they dont even check the box. whats the point? arggvhhh its just frustrating. its completely trashed the fun of having email. and the web.

and the ghost of the old web, the one with low noise, is not viewed as dead, merely its soul is an HTTP redirect to someone's digital billboard, completely unrelated and unwanted.

Re:Web Death

[4of12]

I suppose the rate at which new links are created is roughly a positive coefficient that outweighs the negative coefficient associated with death of a link.

Reminds me of calculations for population growth with k_growth and k_death.

So, two questions:

1. What about deliberately short-lived links like the kind of md5-flavored arguments I get from my favorite news sites hoping to track my usage? How do those affect link life statistics?
2. What's the oldest link on the web?

One of the flaws

[CmdrTaco (editor)]

I think one of the flaws in any analysis of the decay on the web is the fact that most news sites keep an infinite archive of almost everything they have ever published online. The specific phrases probably don't represent a large enough sample size to properly reflect all sites. Sure, he says he used many phrases, but all he gives us is "bill gates sucks", "life's short play hard", "blessed are the cheesemakers", and "late at night". To properly do the study, he should've used a random word letter generator or word generator and test the decay of that.

But, it is interesting to see his results. I can only imagine that if Archive.org did a study like this, they would be able to make a more legitimate conclusion. Perhaps some collaboration is in order?

Obligatory Full Text

[rosewood]

I only do this since I know an angelfire page will get /. and reach bandwidth limits fast! However, there is a pretty excel chart on there so bookmark and come back much later.

Web Decay
by Scott Ennis
4/26/2002
Knowing how anxious most companies are to keep their web content "fresh," I was curious how "fresh" the web itself was.

In order to come up with a freshness rating for the web you need to sample a very large number of pages. Not wanting to do this, I opted to use the Google search engine as a method for reviewing the web as a whole.

My hypothesis is this: By searching Google using some common english phrases and returning results at various time points, a baseline can be reached for the common rate of freshness of overall web content.

I took the total number of pages found for each given phrase at 3, 6, and 12 months. I calculated a percentage for each of these points based on the total number of results found with no date specified.

For example: Phrase 3 mos. 6 mos. 12 mos. Total

buy low sell high 4700 5470 6200 7830
60% 70% 79% 100%

Note:
This method excludes any pages which are not text and more specifically, not English text.
This method relies on a random sampling of phrases.
Using this methodology I determined that the average rate of decay of the web follows a 60-70-80 percent decline at 3, 6, and 12 months.

Therefore, If a company wants to maintain a freshness rate on par with the web as a whole, their site content should be updated at the inverse rate. In other words:
60% of the site should change every 3 months
70% of the site should change every 6 months
80% of the site should change every 12 months
The only way to do this effectively is to either have a very small site, or have a site with dynamically generated information.

The following graph shows the decay rate for a few phrases. I selected these phrase to display because of their unique characteristics.
bill gates sucks--This phrase had the lowest decay rate of any phrases I searched.
life's short play hard--This phrase had the greatest decay rate of any I searched (note: this search was also very small).
blessed are the cheesemakers--This phrase was relatively small, but demonstrates that quantity of pages may not be important in determining decay rate.
late at night--This phrase returned the highest number of results of any I searched and yet it also adheres closely to the 60-70-80 rule.

Conclusion:

Web content decays at a uniform, determinable rate. Sites wanting to optimize their content freshness need to maintain a rate of freshness that corresponds to the rate of web decay.

Study: World Wide Web sites and page persistence

[Seth Finkelstein]

For a more extensive (although older) study, take a look at

Digital libraries and World Wide Web sites and page persistence

That said, the Web and its component parts are dynamic. Web documents undergo two kinds of change. The first type, the type addressed in this paper, is "persistence" or the existence or disappearance of Web pages and sites, or in a word the lifecycle of Web documents. "Intermittence" is a variant of persistence, and is defined as the disappearance but reappearance of Web documents. At any given time, about five percent of Web pages are intermittent, which is to say they are gone but will return. Over time a Web collection erodes. Based on a 120-week longitudinal study of a sample of Web documents, it appears that the half-life of a Web page is somewhat less than two years and the half-life of a Web site is somewhat more than two years. That is to say, an unweeded Web document collection created two years ago would contain the same number of URLs, but only half of those URLs point to content. The second type of change Web documents experience is change in Web page or Web site content. Again based on the Web document samples, very nearly all Web pages and sites undergo some form of content within the period of a year. Some change content very rapidly while others do so infrequently (Koehler, 1999a). This paper examines how Web documents can be efficiently and effectively incorporated into library collections. This paper focuses on Web document lifecycles: persistence, attrition, and intermittence.

Sig: What Happened To The Censorware Project (censorware.org)

Credibility?

[Gossy]

Is it me, or does this 'research' simply look like something a bored guy has just thrown together from a few minutes work, then submitted to Slashdot to see if it gets posted?

From the evidence, he searched for very few phrases. The sample size is way too low to be representive of the web - which some estimates put at several billion more pages than there are people on the planet! There are no signs of more than about 5 different phrases being searched for here..

Can a few simple searches on Google really generate a large enough sample to draw such large conclusions?

The report is one page long, hosted on Angelfire. There is no substantial data to back up his claims. Is this report reliable in any way?

I'm amazed this got posted on the front page of Slashdot..

Re:Credibility?

[Lizard_King]

Agreed. I'm a bit in the dark on *how* this guy came up with his numbers.

I calculated a percentage for each of these points based on the total number of results found with no date specified.

IMHO, This is a bit vague to be called anything but conjecture.

Re:Credibility?

[Stephen]

Well, statistically, if you can develop a good enough test criteria, you could determine the rate with a very, very small sample. This is how some of the more reputable firms can survey 250 voting American adults and usually be within 3% of what the American public will do during the upcoming election.

No, this is because of the (surprising) fact that the accuracy of the survey is dependent only on the sample size, not on the population size. 250 is not a very small sample. The fact that it's 250 out of 100 million or whatever is irrelevant.

archive.org

[mmThe1]

This makes the job of Archive.org - like sites damn tough.

P.S. Are we losing information at a comparable rate to generation....?

interesting but...

[lowLark]

He creates a problem for himself by not providing us with his raw data, making any subsequent verification of the trend difficult. In fact, the one data set he gives us:
Phrase 3 mos 6 mos 12 mos. Total
buy low sell high 4700 5470 6200 7830
60% 70% 79% 100%
seems to demonstrate the opposite of the trend that he describes. Indeed, a current search on google shows about 1,270,000 results (makes you wonder when he did his searches that the current number of results is so many orders of magnitude in difference). The methodology also fails to take in to account any growth in the size of the web, which could mask the effects of decay.

The Web is decaying

[Anonymous Coward]

It is now official - Netcraft has confirmed: The web is decaying

Yet another crippling bombshell hit the beleaguered web community when recently IDC confirmed that the web accounts for less than a fraction of 1 percent of all server usage. Coming on the heels of the latest Netcraft survey which plainly states that the web has lost more market share, this news serves to reinforce what we've known all along. The web is collapsing in complete disarray, as further exemplified by failing dead last in the recent Sys Admin comprehensive networking usage test.

You don't need to be a Kreskin to predict the web's future. The hand writing is on the wall: the web faces a bleak future. In fact there won't be any future at all for the web because the web is decaying. Things are looking very bad for the web. As many of us are already aware, the web continues to lose market share. Red ink flows like a river of blood. Dot-coms are the most endangered of them all, having lost 93% of their core developers.

Let's keep to the facts and look at the numbers.

The web leader Theo states that there are 7000 users of the web. How many users of other protocols are there? Let's see. The number of the web versus other protocols posts on Usenet is roughly in ratio of 5 to 1. Therefore there are about 7000/5 = 1400 other protocols users. Web posts on Usenet are about half of the volume of other protocols posts. Therefore there are about 700 users of the web. A recent article put the web at about 80 percent of the HTTP market. Therefore there are (7000+1400+700)*4 = 36400 web users. This is consistent with the number of Usenet posts about the web.

Due to the troubles of Walnut Creek, abysmal sales and so on, the web went out of business and was taken over by Slashdot who sell another troubled web service. Now Slashdot is also dead, its corpse turned over to yet another charnel house.

All major surveys show that the web has steadily declined in market share. The web is very sick and its long term survival prospects are very dim. If the web is to survive at all it will be among hobbyist dabblers. The web continues to decay. Nothing short of a miracle could save it at this point in time. For all practical purposes, the web is dead.

Fact: the web is dead.

The guy who posted this may have made a mistake.

[eugene ts wong]

Essentially, 60% of the web changes every 3 months.

I think that is incorrect, according the "researcher". He should have said, "Essentially, 60% of the web is getting older every 3 months.".

Better article needed

[Raedwald]

I'm not impressed. The article does not define what he means by decay, or how he measured it, except in the vaguest of terms. The analysis of the data is poor; anyone interested in decay would suspect some kind of exponential decay. They would therefore plot the data logarithmically, and perhaps calcualte a half life. Piss poor.

we've lost the ability to rely on hyperlinks

[thegoldenear]

Tim Berners-Lee wrote :"There are no reasons at all in theory for people to change URIs (or stop maintaining documents), but millions of reasons in practice.": http://www.w3.org/Provider/Style/URI and advocated creating a web where documents could last, say, 20 years and more

Re:we've lost the ability to rely on hyperlinks

[Kallahar]

URL's: people change hosts, usually due to money concerns

filenames: people change languages (php, perl, asp, etc), site layouts, functionality.

filenames: intentionally changed to prevent deep linking (heh heh)

When I change a URI it is usually because I'm changing the logical structure of that program. However I also usually check the referrer logs, and if there has been an outside referral then I will put in a redirect for the old file, and contact the site that had the link to ask them to change it.

There is no excuse for having broken links on your own site though, though it does happen to the best of us :)

Travis

Re:we've lost the ability to rely on hyperlinks

[Reziac]

My main site is going on 4 years old and still has the same core pages and link structure as when it was first uploaded. The main page, and certain subpages, are linked from around 100 other sites (the list has grown every time I check it); I rely on these referrals for traffic, but bedamned if I'm gonna chase 'em all down and get everyone to fix their links. Easier to retain the existing structure, or have a duplicate page (old name, same content) if needed to maintain link integrity.

And this costs me nothing but "oh yeah, must remember to upload index.html as well as index.htm".

I expect that my sites will be just as valid 20 years from now, assuming Earthlink is still in business and still hosting it. (Yeah, I should get my own domain names, but..)

If you have dynamic content, your needs might differ. But for informational sites, change for the sake of change is usually a Bad Thing.

Stop Web Decay Today

[weave]

Do your part to stop web decay. Include this in a cron job. For best results, be sure to brush, I mean touch, three times a day...

find /var/www/html -name '*' -exec /bin/touch {} \;

Study?

[Anonymous Coward]

Wow! What a wonderful, in-depth, study! Is there any link to a scientific paper on that page that I am missing or is that everything? I mean, how can someone claim something just showing us a few numbers and an excel graph.

I appreciate the topic very much, but some more material on it is needed. This study wouldn't be complete enough even for high-school homework...

And look at his homepage (just remove the last part of the url). The most pages are more than two years old... that's decay! :)

Seriously speaking, just look for a few more sources before you accept a story.

Google/CowboyNeal Study

[BoBaBrain]

On a similar note, I was curious to see what the CowboyNeal content of the web is. As luck would have it, a precise answer can be found easily.

Google gives us the following interesting results:

3,840,000 sites contain the word Cheese.

1,640 sites contain the words CowboyNeal and Cheese.

Therefore, 4.27083333333333333333333333333e-2% of cheese related sites contain a reference to CowboyNeal.

As cheese is a randomly chosen word with no special connection to CowboyNeal it is reasonable to assume that 4.27083333333333333333333333333e-2% of all sites contain a reference to The Cowboy (Assuming the number of sites dedicated to CowboyNeal equals the number dedicated to ignoring him).

So there we have it. The web is 99.957291666666666666666666666667% CowboyNeal free. :)


I said the results were "precise", not "accurate". :P

Re:Google/CowboyNeal Study

[TheTomcat]

Do you sell insurance?

S

Re:Google/CowboyNeal Study

[Rupert]

What you got here is one of them reductio ad absurdum thingamajigs.

Your assumptions seem reasonable. I'm a reasonable man, and I don't know nothing about no CowboyNeal and his cheese fetish. But when you get to an absurd result like .04% of the web are belong to CowboyNeal, then clearly your assumptions are flawed.

Ergo, CowboyNeal has a "special connection" to cheese. Quod Erat Doodah.

Not that accurate

[Jormundgard]

I can't even find my page on google anymore. I don't know if it's just because my site's unpopular, or because it has the same name as an online retailer. In any case, it's not searchable anymore, and my guess is that it was removed as "dead".

Jakob Nielsen: Web Pages Must Live Forever

[jukal]

Once you have put a page on the Web, you need to keep it there indefinitely. Read more. Slow news day, eh?

even Bill thinks he sucks more than ever

[maccallr]

I don't claim this is the authoritative answer, or an in-depth study, but the raw data comes from Bill's very own MSN search: bill gates sucks, check it out...

Google SOAP thing for compare-stuff is in the pipeline...

History book? Not as far as I can tell . . .

[phobonetik]

Our weblogs show that google visits our site (www.up.org.nz) atleast monthly, and it is by no means a huge traffic drawing site in the global senee. Its' last visit was on 13th April, drawing 1888 hits...

Heh.. Talk about web decay.

[Bowie J. Poag]

Looks like 100% of the link mentioned in this article decayed in a little under 5 minutes! ;)
Cheers,

heh

[inKubus]

Yeah, in bytes. I wonder how many digits that would be?

Googlebot Visits Monthly

[BigBlockMopar]

Are google claiming that they can check through the entire internet inside a timescale of 3 months, ready to check through again at the start of the next quarter?

I don't know if that's all that far-fetched. I know Googlebot last hit my site on April 7th, crawled every page in my domain over the course of 12 hours, and current searches of their cache show content I'd updated at that time. They seem to visit every month or so.

Perhaps it's based on the traffic they detect to a given site through their CGI redirects... but I'm not a large site, my primary webserver is a Pentium 90. :)

crawl4.googlebot.com - - [07/Apr/2002:13:36:32 -0400] "GET /broken_microsoft_products/ HTTP/1.0" 200 128854 "-" "Googlebot/2.1 (+http://www.googlebot.com/bot.html)"

Re:Googlebot Visits Monthly

[BigBlockMopar]

wow, not only are you running your domain off a pentium 90, but you also have reverse DNS lookup turned on in the logs... that's gotta be giving you a decent preformance hit, no?

Well, it doesn't actually handle DNS; that's felix, an old 486DX-33 running FreeBSD, port-forwarded behind my gateway (I've only got the one IP address). But yeah, I'm sure each logger thread gets held up waiting for resolution.

More impressively, dynamic content. (Most of the pages are generated dynamically as shtml through the x-bit hack; nothing sophisticated, mostly just inserting templates and stuff for color scheme because I'm too lazy to type long BODY tags) And anywhere from 2,000 to 5,000 hits per day. And only 48 megs of RAM. And it's a popular Linux distro's default kernel, not recompiled for that machine. Even so, it hardly ever breaks a sweat.

As you can tell, it's like, zero performance tuning. But it still cranks out a SETI@Home unit every day or two.

As for reverse DNS itself, yeah, I like it. :) It's a nice luxury.

Google "pages found" data

[Per Abrahamsen]

I have maintained a number of google celebrity lists, where celebrities in various categories are ranked based on the number of page hits by google.

While the numbers clearly aren't totally random, they are very fragile indeed. Some people have had a change of two orders of magnitude, within a week. And in these cases, there have usually been no real world events that could explain such a change. I guess the google page hits numbers depend as much on the internal google structure, as on the number of actual pages on the web.

So I doubt google page hits statistics is a useful research tool. Nonetheless, it can be fun. Here are some google hall of fame lists:

• A list of the most famous Danes according to google.
• A list of free software celebrities according to google.
• A list of Emacs contributors sorted according to google hits.
• A list of sequential artists sorted according to google hits.
• A list of OS (Kernel) Mindshare sorted according to google hits.

PS: Mail me to suggest new entries to the lists.

average and real life persistance of documents

[kipple]

.. I noticed that a paper I wrote a LOT of years ago can still be found online somewhere.. so I suppose that although -in the average- web pages do disappear, if those pages contain documents, they will survive the death of their original webpage.

not that it was an interesting document - just a little paper about nothing important. But still, it's out there.

My thoughts? I think that as long as a website can be "saved" in some form, its content will be available in other forms for a long amount of time.

this should make people think, especially those who put copyrights on their webpages, or don't want some information to spread around.

could we say that information want to be free as long as it's downloadable?

hmm..

Blessed are the cheese makers

[kzinti]

What's so special about the cheese makers?

It's not meant to be taken literally. It refers to any manufacturers of dairy products.

-- Monty Python, Life of Brian

Wide jump from findings to conclusion

[gpmart]

In fact, I would argue that good content need not change. Aside from the obvious issues with the small sampling of phrases, the web is, thankfully, not just a series of catch-phrases. In fact, it was designed to carry complex information such that it could not be reduced.

What scares me here is the conclusion that web sites need to change their content 60% every 3 months. This is not freshness, this is reorganizing to re-organize. If you are considering doing this, you had better seriously re-consider your future. Its an interesting study but a good meme doesn't die simply because the catch-phrases are tired.

At faculty meetings at our school I sit with a bingo card. On it are a series of catch-phrases. We listen for the catch-phrases and shout out when we have finished our cards. B***SH*T is the game and to reduce your content to a series of reorganized catch-phrases is like having a marketing guy develop foreign policy.

Anyone willing to write the perl module that searches for the latest catch-phrases and inserts them randomly into your web content. Yeesh!

Google measurements--varied and fun!

[lute3]

Using Google to calculate Tooth Decay.

Intermittence

[markmoss]

At any given time, about five percent of Web pages are intermittent, which is to say they are gone but will return.

For example, most web pages linked to in slashdot articles.

Google Study in Another Place

[scottennis]

The study I posted on Angelfire appears to have reached a bandwidth threshhold. I've made the same study available here:

http://helen.lifeseller.com/webdecay.html

I've also included a link to the raw data I used.

Thought and mod_rewrite are the key

[Fweeky]

The key to making links that don't rot is to design a URI schema that's both independent of any redesigns of your site and independent of any particular way of doing things.

Let's look at a few examples.

The URI to this page is http://slashdot.org/comments.pl?sid=31884&op=Reply &threshold=3&commentsort=3&tid=95&mode=nested&pid= 3434535 - what is it telling you that it doesn't need to?

Well, for a start, that .pl is a bad idea. What happens in 4 years time when SlashDot is running on PHP, or Java, or Perl 7, or a Perl Server Page, or ASP? Then there's the difficult-to-decode query string that tells you nothing about the link other than "this is the information the server needs to locate your page at the moment", and doesn't give you much faith in it living forever.

Now let's look at an equivilent Kuro5hin URI.

http://www.kuro5hin.org/comments/2002/4/29/22137/6 511/51/post#here is a URI to reply to a random comment on k5.

For a start, you can't tell what application or script is serving you the page, and you can't see what type of file it's linking to; both these things can and will change over time.

Second, there's a date embedded in there; you can see the developers, if they ever decide to change the meaning of '/comments', using that date as a reference; if the URI is before the change, they can map it onto the new schema or pass it onto legacy code.

Having the date in the URI is good because it allows you to determine when the link was issued, and map it onto any changes or pass it off to a legacy system as required.

Now let's take an apparantly good link on my now horribly out of date site, aagh.net.

http://www.aagh.net/php/style/ links to an article on PHP coding style.

Certainly, hiding the fact that I'm using PHP to serve this document is good, and shortening the URI to remove the useless querystring is good (you can't see one? Good, that's the point), however, this URI may well stop working in a few weeks; I'm planning a redesign and the old schema may well not fit in well with it.

A short yyyymm in there could have made all the difference; a simple if check on the URI's issue date would keep it working.

The moral of the story: Think about your URI's when you're designing a site. Try to remove as much data as you can without painting yourself into a corner.

Re:Thought and mod_rewrite are the key

[Jahf]

Someone with mod points and a clue about how to organize a web site, please mod the parent up as insightful (or informative).

If I had mod points today, I would do it ... the web needs more thought put into it's architecture and less put into it's look and feel.

Re:Thought and mod_rewrite are the key

[Fweeky]

> By simply adding &threshold=-1 to the end of that, I can see all the replies [slashdot.org] at -1 easily and painlessly.

The argument wasn't "query strings are bad, m'kay"; look at the URI and see what information's in there. Does .pl serve any purpose? Does sid=3188 gain anything, aside from make the page very difficult to serve statically? Does tid=95 and pid=3436807 tell you anything?

The URI's would work just as well using something like http://slashdot.org/stories/31884/comments/3436807 / 5/?mode=nested&threshold=-1; even if /stories/31884 were a static file, the URI would still work and point roughly to the right place. And it's not exposing the internals of how the comments system works, and it's keeping the more readily tunable query strings clear, without making the exact resource you're pointing to difficult to find.

> Do you know how to make k5's comments nested instead of threaded purely using the URI?

No. Actually, I wasn't really pointing out k5 as being the perfect example; Scoop actually tends to really suck in this respect (like setting the URI to '/' when you change comment modes). However, I might be tempted to ask you which URI is likely to live the longest, certainly back when SlashDot used to archive articles after a couple of weeks.

> The point is, wether or not it takes the optimum number of bytes isn't always the priority

I never once said the size of the URI was important. I said it contained a lot of extranious information that changed a lot while meaning little (i.e. the URI's changed from the dynamic query string to an .shtml file when a story was archived).

> in the case of /., its designed to be easy to use for the (savvy) user, not easy on the server

What's easier for the "savvy" user? A URI that will work for the rest of SlashDot's life, or one that'll last until the story is archived, or the underlying architecture changes, and which contains a lot of randomly ordered and mostly meaningless information?

A well designed URI scheme will actually give the savvy user a lot more control; say, you include the date of an article, ala http://slashdot.org/stories/2002/05/30/; you can imagine going to such a URI and getting all of the stories on that day, month or year, and instantly being able to identify how old a linked to article is. You can also imagine an archived URI and a live, dynamic URI both using the same schema.

You can also imagine giving a URI of an interesting article to a friend without first having to decode the query string; just strip off anything after /comments and they get the story.

Note: This applies to any site, those particular SlashDot and k5 URI's were just examples.

guidelines for content change?

[call -151]

From the article:

Therefore, If a company wants to maintain a freshness rate on par with the web as a whole, their site content should be updated at the inverse rate. In other words:
60% of the site should change every 3 months
70% of the site should change every 6 months
80% of the site should change every 12 months
The only way to do this effectively is to either have a very small site, or have a site with dynamically generated information.

This seems so totally- "if everyone else is
jumping off the Brooklyn Bridge, then we
should to" by itself that it discredits what
sliver of credibility the article had. Using
a web-wide average as a guideline for what
a particular web site "should do" is
meaningless. Web sites should present timely,
appropriate information that is useful to
those who visit. Some sites deal with
material that changes frequently (stock quotes
and sports sites should be presumably updated
regularly) and some sites deal with material
that does not change frequently (no need to
redo your tech support documents for long-
out of production products every week.)
This notion of `freshness' is ill-defined,
poorly measured and of dubious value.

Correct links

[Per Abrahamsen]

All the links were wrong. Hopefully, these are better:

• A list of the most famous danes according to google.
• A list of free software celebrities according to google.
• A list of Emacs contributors sorted according to google hits.
• A list of sequential artists sorted according to google hits.
• A list of OS (Kernel) Mindshare sorted according to google hits.

How do you account for domain changes?

[yerricde]

The key to making links that don't rot is to design a URI schema that's both independent of any redesigns of your site and independent of any particular way of doing things.

You can't mod_rewrite a domain name that you have lost control over. If you have a popular site hosted on a university's server, and then you graduate, what do you do? If you put up a site, some Yakkestonian trademark holder takes it from you in WIPO court, and you're forced to go to Gandi.net to get a new domain, what do you do?

Re:How do you account for domain changes?

[Fweeky]

> You can't mod_rewrite a domain name that you have lost control over.

Nope, that's too bad. You can mod_rewrite a domain name you do have control over, though. You can also see if you can get the new owners to redirect to your new domain.

If, say, all your URI's start with a date, you ask the new owners to redirect any URI containing '/yyyy/mm/dd' less than the date you lost the domain to your new site. You may not get it all or even most of the time, but the option is still there.

Alas, this is one of those cases URN's would come in handy.

Mirror Site

[scottennis]

http://helen.lifeseller.com/webdecay.html

You seem to have some good ideas

[drew_kime]

The analysis of the data is poor; anyone interested in decay would suspect some kind of exponential decay. They would therefore plot the data logarithmically, and perhaps calcualte a half life. Piss poor.

So when can we expect to see your rigorous analysis? Or were you just bitching?

See this letter I wrote to TimBL

[yerricde]

Once you have put a page on the Web, you need to keep it there indefinitely.

How is this possible if you happen to lose control of the domain? I wrote a letter to Tim Berners-Lee about this issue.

In "Cool URIs don't change" you wrote that URIs SHOULD never change. However, you left some questions unanswered:

In theory, the domain name space owner owns the domain name space and therefore all URIs in it.

However, what happens when ownership of the domain name is suddenly removed from under a user's feet?

Except insolvency, nothing prevents the domain name owner from keeping the name.

Wrong. A trademark owner in Yakkestonia can drag a domain name owner into WIPO court and have the domain forcibly transferred. Under ICANN's dispute resolution policy, the plaintiff gets to pick the court, and WIPO has shown itself to find for the plaintiff in an overwhelming majority of cases. This is sometimes called "reverse domain name hijacking."

And in theory the URI space under your domain name is totally under your control, so you can make it as stable as you like.

Not if a hosting provider provides only subdomains (or worse yet, subdirectories) and does not offer an affordable hosting package that lets a client use his or her own domain name. Would it be reasonable to construe the "Cool URIs don't change" article as a warning against using such providers?

John doesn't maintain that file any more, Jane does. Whatever was that URI doing with John's name in it? It was in his directory? I see.

What is the alternative to this situation for documents that began their lives hosted on an ISP's or university's server space?

Pretty much the only good reason for a document to disappear from the Web is that the company which owned the domain name went out of business or can no longer afford to keep the server running.

Or that the hosting provider pulled the document under a strained interpretation of its Terms of Service because the company didn't like the document's content.

Is there an official W3C answer to these questions?

Exaclty who should be proving your thesis

[bihoy]

It may be a valid thesis that you are putting forth. It does occur to me, however, that you seem more interested in having someone else prove it for you. Your rather cursory investigation and lack of basis neither lends credence to your theory nor compels one to take it seriously. If you desire the respect of the scientific community then I suggest you put a little more work and effort into it.

Free hosting is a bad bargain

[Frank T. Lofaro Jr.]

Why do so many people use crap like Angelfire, Tripod, Homestead with all their bandwidth limits, restrictions, ads and blocking of remote image loads?

Not to mention that well over 50% of the time any search engine result that points to Angelfire in particular points to a 404 Not Found. This is much more than what I experience with other sites. Do their users get kicked off often, or just go away, or what? I don't even bother clicking on those results unless it looks like the content is truly compelling. And thank God for Google's cache.

I can understand if some truly can't afford hosting, but even for these people, even Geocities is much better!

Somehow I doubt the majority of those people using Angelfire, Tripod, etc can't afford hosting.

Well, after the dot-com world gets a little more squeezed, those sites may no longer exist. Too bad that many people won't bother rehosting their content and will just drop off the web.

olm.net offers Linux based hosting for under $9/month. No I don't work for them, but I am a (satisfied) customer.

$9 a month - and you won't piss off your users.

(Yes I know their other packages are more - but the $9 a month package is better than any of the free services)

Don't EVEN get me started on organizations and commercial BUSINESSES (ack!) that use free hosting - that is so unprofessional. I don't think I'd want to do business with a company (even a local store) that wouldn't/couldn't pay $9 a month to have a less annoying and more reliable website.

Of course, some of the content out on the Web isn't even worth $9/month, heck some of it has NEGATIVE worth. ;) Of course, then it isn't worth looking at, so who cares if it is even hosted.

Re:Free hosting is a bad bargain

[56ker]

"Somehow I doubt the majority of those people using Angelfire, Tripod, etc can't afford hosting." - but for most of the sites - like blogs, pictures of my family and pets - people don't think its worth paying! Also once you change address you lose your search engine rankings.

Irony!

[gnovos]

I love how this very page seems to have died... The web is a massive irony generator.