Slashdot Mirror
300 Years to Index the World's Information
Kasracer writes "At the Association of National Advertisers annual conference, Google's CEO, Eric Schmidt suggested that it would take 300 years for them to index all of the world's information. From the article: 'We did a math exercise and the answer was 300 years,' Schmidt said in response to an audience question asking for a projection of how long the company's mission will take. 'The answer is it's going to be a very long time.'"
I always thought 42 years ought to be enough.
The hardest part will be developing the hardware that is able to recursively index the Google data itself an infinite number of times.
I Am My Own Worst Enemy
Did they take into account the information that is being created as they are indexing? Do they plan on live indexing everything that's being made. Information doesn't stop getting created just because they've stored everything that's already been done.
"Plans are for fools! Oglethorpe, the plutonian (Aqua Teen Hunger Force)
I agree. At Google's scale and beyond, the concept of 'information' is such a wooly one.
How the hell did they come to that figure of 300 years?
300 years? I'd have thought their other plan would have been a lot quicker.
How long until Google decides that your house is information? Just imagine an army of small robot spiders invading your home every night, registering the position, name and contents of every single object you own, making it searchable from house.google.com. Unless you nail a robots.txt to your front door, that is...
Please stop creating new information and let Google catch up! You can resume later.
It's going to take them a hell of a lot longer than that, considering my car keys are always moving.
Googlesphere anyone?
She's built like a steak house, but she handles like a bistro....
I wonder how many man-years it would take to listen to all the music and video that could be indexed. Be interesting at least to find out what the order of magnitute would be - millions, or perhaps billions or trillions of man-years of unique recorded audio and video? It would have to be a game of gross estimation - but it would at least put into perspective how much material is out there, even if most of it is boring "security" footage, compared to the scope of our lives.
It'd be interesting, if, perhaps in a couple generations, we could have a cheap media volume that contained "recorded media, prehistory - to - 2050ad"... if the media that exists today even survives a couple generations, and copyrights aren't extended indefinetly. The idea of an indexing system that can even put all that information into a meaningful context would be fascinating to consider though, if it could be possible.
Ryan Fenton
How is 'information' defined in this context? Is a thirteen-year-old girl's blog considered information?
Obviously they're not feeding those pigeons enough. Time to buy some quality feeds Google. Maybe even slip in some uppers every now and then. If all else fails, maybe it's time to consider the parrot upgrade. They're a lot more expensive but their index/poop ratio is much better.
EvilCON - Made Famous by
I'd like to see their definition of information. Certainly, a lot of things that are already of common interest are on the net. Occasionally, I find things that aren't available online but the greatest majority of the time google is able to find what I want.
To further the example: at work we have several filing cabinets that haven't been opened in years. There are lots of papers and stuff in there, I can vouch for that. Some might consider it "information." But in reality all that stuff could be burned and I doubt it would make the slightest difference in the way the future rolls out. None of it is stuff that would ever be needed by an IRS audit or anything like that either. Does google consider this kind of stuff as part of their efforts? Because I think they can safely ignore it.
It's going to take 300 years to index the grammer and spelling mistakes on Slashdot alone.
No, the proper model is not Moore's law but Bono's law. If it takes 300 years now, then it'll take 320 years in 20 years, and most of the time will be spent waiting for exclusive rights to expire (if they ever do). For instance, indexing a literary work that's out of print and not widely available at libraries requires getting a new copy, and those aren't available until the copyright runs out.
We did a math exercise? What exercise?
To estimate the time involved, you surely need to know the size of the information involved (don't quote me that bunkum about 170 terabytes in TFA - yes I did read it), and to know the size you need to know what all the information is, which you can't (and surely new information is created all the time?).
This translates as "I pulled my finger out my ass, waved it in the air and came up with 300 years."
Because once it's all there, you don't have to look for it anymore.
Well, it's just Zeno's paradox. Let's say it takes them 300 years to index all today's information, then another 150 to index all the new infomation, then another 75... By 2605, all information to that point will have been indexed by Google. Then they can start indexing the FUTURE.
I hate the one hundred and twenty character limit for signatures with an all-enveloping, all-destroying, incredible pass
The point is that many current systems spew a huge volume of low value (but nonzero value) data (multiple MB or GB/day/device). The lack of storage means most of this is not captured and is thus never indexed.
Even massive companies can't keep all their data. Wal-Mart stores on the order of 460 TB in their data warehouse, but only has room for the last 13 months of data or so. At 138 million customers per week, they only have room for a paltry 59kB per customer per week.
Two wrongs don't make a right, but three lefts do.
Then:
I tried to find the graph of speed over time because I have seen itseveral times. It shows the exponential increase in the speed of the project. Apparently there are many scientists that believe with techniques as they are now we could repeat the project in 2 years if we started over. The indexing of information could have a very similar timeline. Very slowly at first and then as technology and specific methodology develop off you go. So the truth is... this is a guess. I wouldn't put too much faith in it.
What question is that? What happens inside a woman's head?
Mother, do you think they'll like this sig?
"We did a math exercise and the answer was 300 years," Schmidt said in response to an audience question asking for a projection of how long the company's mission will take. "The answer is it's going to be a very long time."
Since this was in response to an audience member's question, does anyone else think he was joking? Because it is such an outlandish question from an information theory and modeling point of view, perhaps he was mocking it? "Ah yes, we just came up with an equation and it should take 294.59 years." I think this also makes sense in light of his next comment, which was made on a more serious note. I interpret it, "We really didn't use an equation, it will obviously take a long time though." This is how I understod his comments, and I may be wrong, but it wouldn't surprise me if some reporter picked up on this "joke" and put it up as "news".
...Google indexed it all in 6 days, and took a rest in the 7th...
gtkaml.org
question is asked, and they seem to miss that the answer is that it is it's own index.
Also, can someone explain to me how you even approach something like this from a mathematical model point of view? How did the 170 terrabyte number even come up? Aren't there different definitions for what constitutes 'information?' Also, who the hell spent their 20% on this problem when there was integral code for vital programs to write, such as Google Suggest and Google Suggest in Japanese?
PLEASE, SOMEONE EXPLAIN BEFORE I GO OFF INTO MORE OF A FLAMEBAIT RANT!
Martini Glasses
They take the rate of current indexing of data, then take the rate at which data is being added to the pile by looking at current trends and possible future trends of people hooking up to the net and adding to that pile, then take the rate at which their systems advance to do the indexing of that pile. They then pass those variables through a custom magic google app and wait a bit and then, tada, the answer 300 is spat out.
You need remember that they could be way off, if some major breakthrough in storage technolgy happens tomorrow all those figure would need be recalculated. At best it is a very very rough idea of how long it is going to take them to catch up to the worlds information and keep it in a current index.
Stuff like this (or years ago for LHC) is most likely following approach:
They astimated an amount of information that is "all information", like 480 000 Exabyte or so.
Then they look at their current capactity (storage and database cpupower) and just interpolate moore's law into the future and look when the demand will be met.
Of course, for stuff like the LHC that only interpolates 10-20 years into the future such a thing is possible, but 300 years? He should read up about the singularity...
HI O WISE PRINCE. WHT TOOK U SO DAM LONG?
Hey, check it out! I'm in the lead!!!! ^_^
I wouldn't consider the mad hatter mad. Just reality impaired. He sure can make a mean cup of tea.
"Of the approximately 5 million terabytes of information out in the world, only about 170 terabytes have been indexed, he (Schmidt) said earlier during his speech."
So ... how many terrabytes of info will be produced in the next 300 years, and does anyone really think that Google (and anyone, or everyone) could keep up?
Especially, once all 20 billion people who live in the Solar System are video-documenting every moment of their existence ...
OK, so I project and exaggerate ...
The article gives us no facts that we can use to verify the claim. Without a definition of information and a definition of indexing one cannot take this for accurate. There are many definitions of information and except that used in "Information Theory", which is a message received and decoded to its original form, I don't know of any definition that has sientific or mathematical rigour. In fact, in my opinion, Information Theory is a misnomer and is more properly called Communication Theory since it is about getting a message properly communicated, NOT about whether its contents are useful. Additionally, information as understood by most comes in many forms and types and each may require different ways of indexing. Finally, aren't the indexes information that needs to be indexed? How do you keep from recursing?
New here?
Using HTML in email is like putting sound effects on your phone calls. Just say <strong>no</strong>.
Uh oh...someone needs to visit Applied Cryogenics and knock 700 years off Fry's timer then.
They're almost there
i read the article, and this is what I got from it. i could be wrong.
-5 million TB of data.
-170 TB have already been indexed.
-it would take 300 years to index that data and make it searchable.
I don't think it's an exercise to index all knowledge. As you point out, that would be alogical. I think it's more of an understanding of what it would take to effectively and completely serve the world's information needs given current indexing capabilities.
I guess establishing a benchmark currently, both of how efficiently they index information, as well as a general number for the amount of data is out there, they can gauge how efficient they get relative to the rate at which the amount of potentially indexable data increases.
un burrito me trampeó.
Ask a stupid question...
Proverbs 21:19
Practice has shown that government ownership and operation of airports is inferior to private ownership.
Contribute to civilization: ari.aynrand.org/donate
He didn't clarify that 299 years of that was indexing all the Internet porn sites.
Don't worry, I just ordered a pizza for I. C. Wiener.
My other car is first.
Look, the problem is not how much data there is in the world, the problem is finding a general automatable algorithm for organizing it in such a way that J. Random User can rapidly find what he's looking for.
Stroll on down to the nearest university library. It's got a lot less information in it that Google is considering, and aboutt a hundred thousand man-years over a few centuries have gone into finding clever ways to organize it all: card catalogs, shelving systems (e.g. Dewey and his decimals), nowadays searchable electronic catalogs, reference books, specialized indices for law and science and medicine, citation indices, reviews, reviews of reviews...and so on and so forth forever. And yet, it can still be immensely difficult to track down a particular piece of information you want. Even if it can be done, often it takes a fair amount of expertise in a field just to know where to look. Where do you find public information on patents for desalination processes? How do you find out if anyone has synthesized a polymer resin that melts between 130 C and 150 C and is resistant to acid, with a tensile strength about X? What was the common law meaning of "ownership in fee simple" in 1680s England? Even to start looking for the answers, you often need great experience in the relevant field, so you know where to start looking -- the "search terms" we might say.
Google may be feeling its oats because they can now very rapidly provide the most obvious things people want -- directions to San Diego from Ukiah, the times and places Serenity is playing on Sunday, the lead story of the New York Times "Style" section last Sunday, or the names and addresses of the six pizzerias closest to me still open at 11:25 PM.
But this is utterly small potatoes compared to the problem of organizing information generally, so that it is useful to professionals during the weekday as well as for amusement on the weekend. It is first, generally speaking, an unsolved problem -- no library or information index I've ever used fails to have at least one frustrating "feature" that leaves me scratching my head, wondering what the heck the designers were thinking. Secondly, I very much doubt Google has the depth of professional expertise in-house to even begin to figure out how to organize all the giant repositories of information in law, science, engineering, literature et cetera in such a way that professionals can use them, let alone amateurs.
And finally, they don't have the money to do it, and it will be very hard for them to raise it. Indices have suffered from this problem for a long time: any given user will only pay a very small price per search, but it costs a huge amount to make the index. Heretofore, makers of indices and dictionaries and references have relied on selling them at very high prices to libraries, which in turn raise the money in small bits from their patrons, or taxes. But Google would cut out the library middleman -- you search directly. So how are they going to cover their costs? They've no easy way to charge you $0.005 every time you do a Google search, for example.
In short, this sounds like the 21st century equivalent of that 1950s nuclear energy braggadocio, "energy too cheap to meter." Call it "information too cheap to meter." Color me skeptical.