Google Sorts 1 Petabyte In 6 Hours

Posted by Soulskill on Sunday November 23, 2008 @04:53AM from the sort-of-fast dept.

krewemaynard writes "Google has announced that they were able to sort one petabyte of data in 6 hours and 2 minutes across 4,000 computers. According to the Google Blog, '... to put this amount in perspective, it is 12 times the amount of archived web data in the US Library of Congress as of May 2008. In comparison, consider that the aggregate size of data processed by all instances of MapReduce at Google was on average 20PB per day in January 2008.' The technology making this possible is MapReduce 'a programming model and an associated implementation for processing and generating large data sets.' We discussed it a few months ago. Google has also posted a video from their Technology RoundTable discussing MapReduce."

3 of 166 comments (clear)

Min score:

Reason:

Sort:

Re:That's Easy by farker+haiku · 2008-11-23 05:16 · Score: 5, Informative

I think this is the data set. I could be wrong though. The article (yeah yeah) says that
In our sorting experiments we have followed the rules of a standard terabyte (TB) sort benchmark.
Which lead me to this page that describes the data (and it's available for download).

--
Your sig(k) has been stolen. There is a puff of smoke!
Re:That's Easy by Anonymous Coward · 2008-11-23 05:16 · Score: 5, Informative

From TFA: they sorted "10 trillion 100-byte records"
Re:Sort? Sort what? by nedlohs · 2008-11-23 05:34 · Score: 5, Informative

I realize, slashdot..., but maybe you could glance at the article which states:
10 trillion 100-byte records