Slashdot Mirror
Google Sorts 1 Petabyte In 6 Hours
krewemaynard writes "Google has announced that they were able to sort one petabyte of data in 6 hours and 2 minutes across 4,000 computers. According to the Google Blog, '... to put this amount in perspective, it is 12 times the amount of archived web data in the US Library of Congress as of May 2008. In comparison, consider that the aggregate size of data processed by all instances of MapReduce at Google was on average 20PB per day in January 2008.' The technology making this possible is MapReduce 'a programming model and an associated implementation for processing and generating large data sets.' We discussed it a few months ago. Google has also posted a video from their Technology RoundTable discussing MapReduce."
Consider a data set of two numbers, each .5 petabyte big. It should only take a few minutes to sort them and there's even a 50% chance the data is already sorted.
It looks like Google saw Yahoo crowing about winning the 1 TB sort contest using Hadoop and decided to one up them!
Let's see if Yahoo responds!
It's not enough to sort by blond, black, gay, scat, etc. Some categories are a combination that don't belong in a hierarchy.
That is where tagging comes in. Sorting can be done on-the-fly, with no one category intrinsically more important.
I suggest you read Slashdot
Assuming it was written in binary in a font that allows for 1 digit per 2mm, the length of the data would be 183251938 m, or 1145324 times the perimeter of an olympic-sized swimming pool.
You got baited into a flame in a very elaborate scheme to mock your intelligence (or lack thereof).
There is no category meta-flamebait, so you're proving the mods right I'd say.
I hope this helps.
Show a man some news, distract him for an hour. Show a man some mod points, distract him for the rest of his life.
With a little bit of excel, if takes 4,000 servers 362 minutes to calculate a 1PB job It takes 1440 (24 hours) on 20,111.11 server to sort 20pb (if it was just plain sorting they were doing). And just on a side note, from their number one of their servers can compute 741.5 MB per minute!
i make this about 48GB/s, my hard drive manages about 20MB/s, even my mid-range ram manages only ~6.4GB/s, and top end ram will reach only ~13GB/s (according to wiki) so even ignoring the ability to process that much data in that time, the ability to simply move that much data is quite impressive (at time of print, may not hold one year down the line)
Yes, but how much is that in football fields?
You silly sod, you can't measure something in football fields! There's internationalization to take into account!
Canadian football fields are 100x59m, American football fields are 109x49m, and the rest of the world doesn't even play the same game on a football field. And THEIR sport has a standard range, anywhere from 90-120m by 45-90m (Thank you wikipedia).
You've now introduced variable-variables! We can't get an absolute number!
If you feel the urge to play with MapReduce (or reade the paper), you don't need a fancy Linux distro to do it. MapReduce is simply the map() and reduce() functions, exactly as implemented in Python. Granted, Google implementation can work with absurdly large data sets, but for small data sets, Python is all you need.
Not that this wasn't entirely predictable.
Parallel/distributed sorting doesn't eliminate the need for map/reduce, it just helps spread the problem set across machines.
Here's the thing though...its the distributing of the problem set and the combining of the results that is the hard part - not map/reduce.
Map and reduce are simple functional programming paradigms. With map, you apply a function to a list - which could be either atomic values or other functions. With reduce, you take a single function(like add or multiply, for instance) and use that to condense the list into a single value or object.
That's my understanding of map/reduce from my functional language classes in school and that's exactly how Google describes it. I don't really see what the big deal is with map/reduce in itself.
Like I said, its the distributing the problem among thousands of machines that is the hard part.
No, Thursday's out. How about never - is never good for you?
I developed a search engine in c++ in grad school and I remember applying the concepts of vector space model, okapi model in it. Looking at the relatively snail pace speed I got from the c++ stl datastructures (with limited hardware resources), I still can't help but wonder how google pulls off the "magic" of split second results.
MapReduce is a powerful concept and a good starting point is hadoop. Google uses its own proprietary system called GFS.
We once had a session with Mr.Raghavan who was the yahoo search chief.He advised that students in general should learn to work in projects that involve a large corpus of data and that phds are especially valued in the field.