WWW Surpasses One Billion Documents

Gary William Flake writes "A new study by Inktomi and NEC Research Institute show that there is at least one billion unique indexable Web pages on the internet. The details are pretty interesting; for example, Apache dominates the server market. "

Comments

the best part

[Capt Dan]

Longest domain name:
http://www.tax.taxadvice.taxation.irs.taxservices. taxrepresentation.
taxpayerhelp.internalrevenueservice.audit.taxes.co m

gee. A tax site with a long, unintelligble, confusing domain name. Go figure.

"You want to kiss the sky? Better learn how to kneel." - U2

And at least one of them already comments on that

[dsplat]

Yes, and the Jargon File already has a comment on that, originally from Theodore Sturgeon:

Sturgeon's Law prov.

"Ninety percent of everything is crap". Derived from a quote by science fiction author Theodore Sturgeon, who once said, "Sure, 90% of science fiction is crud. That's because 90% of everything is crud." Oddly, when Sturgeon's Law is cited, the final word is almost invariably changed to `crap'. Compare Hanlon's Razor, Ninety-Ninety Rule. Though this maxim originated in SF fandom, most hackers recognize it and are all too aware of its truth.

Meaningless Statistics

[(void*)]

That's what I hate about such "statistics". No information or context is given. One is not told how this estimate of "one billion" is gotten. No details about how the research methodology was forthcoming. Instead one is only supposed to stare slack-jawed in amazing at the touted figure of one billion an be impressed. That anyone can impress oneself that THIS is an achievement is amazing.

For all you know - the web has surpassed at least 1 webpage count. Big Fscking Deal!!!

Why?

[dsplat]

Why is one of them Hamster Dance? Don't go there with an 18 month old child on your lap. For an adult, this is funny once. For a toddler, it is funny every time the computer is on.

technically inacurate statistics

[TheCodeMaster]

dynamic content makes the technical quantity of distinct "pages" far greater than a billion.

Inktomi, publicity, and mod_perl

[billh]

Well, as any of us geeks know, this isn't really news. I'm sure we passed the billion mark a long, long time ago. Inktomi just wants the publicity, and some news service will probably pick this up, most likely CNN.

One thing of interest, though. If you look under the "Web server market share", Red Hat and mod_perl are apparently web servers now.

Thaaat's great...

[Greyfox]

Now INDEX it.

Finding information on the web is going to increasingly be like trying to find hay in a needle stack. Already the current indexing engines can't keep up, and you have unscrupulous web authors putting bunches of keywords unrelated to their site in their meta tags to insure that they get mentioned in every single search. Some indexing engines already ignore meta tags for that reason. And how many times have you tried Altavista, Excite or Google only to find that the page you're trying to get to has expired or is 8 years old and hasn't been changed in 7?

This issue is going to have to be addressed, because the web is going to continue growing.

unique?

[Signal 11]

Well, yahoo has hundreds, nay thousands, nay hundreds of thousands of "uniquely indexable pages" in their database. It's a web of links. How does one define unique?

Really, this article says nothing. Unless it states (and it does not) *exactly* how they mean "unique" I'm not going to take this seriously. A more interesting statistic (and one I haven't seen updated in awhile) would be what the information conversion ratio is between the "RealWorld" and the web - ie: how much information that you can find in a library can you also find online in it's entirety. That is a more accurate measure of growth than raw page numbers.

1 Billion useless pages.

[Dast]

49.5% Broken links to mp3s
49.5% pr0n pages with javascript popups
1% other

We humans should be so proud of ourselves.

:)

Use Google

[JoeBuck]

Google is one of the best search engines available for most purposes, because it ignores meta tags, and scores pages higher based on links to the site from other high-scoring pages (this is a recursive definition but the recursion bottoms out).

The result of this is that it gives useful results even when very common words are used. Try searching for Linux on Google. The first ten results are

• linux.org
• linux.com
• www.debian.org
• www.linuxworld.com
• linux.davecentral.com
• www.varesearch.com (VA Linux)
• linux.corel.com
• www.li.org (Linux International)
• lwn.net (Linux Weekly News)
• www.linuxhq.com

While a human being might be able to come up with a better list, a machine came up with that list, based solely on the structure of the web. (I wonder why linux.davecentral.com rates so high -- possibly because it's attached to a high-ranking site, davecentral.com).

ObAdvocacy: and Google runs on Linux.

Impressive Marketing statistics

[henley]

Well, my take from the site that what they're actually saying is "Look at our lovely indexing cluster. It can index 1 billion web thingies! Shouldn't you be buying an search engine product that powerfull?

Or, in other words, it's another example of meaningless statistics spewed in the name of marketing, vaguely covered-up as serious research.

References: Car MPG & top speed figures vs actual usage, Processor MHz as function of system throughput, quoted battery life as function of laptop utilisation, quaketest FPS compared to average internet multiplayer experience etc etc etc...