Google Sorts 1 Petabyte In 6 Hours

krewemaynard writes "Google has announced that they were able to sort one petabyte of data in 6 hours and 2 minutes across 4,000 computers. According to the Google Blog, '... to put this amount in perspective, it is 12 times the amount of archived web data in the US Library of Congress as of May 2008. In comparison, consider that the aggregate size of data processed by all instances of MapReduce at Google was on average 20PB per day in January 2008.' The technology making this possible is MapReduce 'a programming model and an associated implementation for processing and generating large data sets.' We discussed it a few months ago. Google has also posted a video from their Technology RoundTable discussing MapReduce."

Comments

Kudos to Google

[Anonymous Coward]

for knowing how important the Library of Congress metric is to us nerds!

Re:Kudos to Google

[canuck57]

for knowing how important the Library of Congress metric is to us nerds!

But at least now we know Google can sort out petafiles.

Re:Kudos to Google

[shutdown -p now]

Bah! To pay true homage, they need to add it to the list of units in Google Calc!

Re:Kudos to Google

[LingNoi]

So Google can sort through 12 LoCs in 6 hours.

Wow, that's 2 LoC/pH

Re:Kudos to Google

[Anonymous Coward]

Woooosh, dipshit.

Unit conversion

[Zarhan]

Yay! We finally have unit conversion from 1 LoC to bytes! So...20 PB = 6LoC, means that 1 LoC = 3,333... PB :)

Re:Unit conversion

[xZgf6xHx2uhoAj9D]

This is an excellent point. No American football player has used his feet since the NFL adopted hoverchairs into the rules in 1974.

Finally...

[aztektum]

I will be able to catalog my pr0n in my lifetime:

Blondes, Brunettes, Red heads, Beastial^H^H^H^H^H "Other"

Re:Finally...

[Pugwash69]

How do you catalogue the topics? I mean "Clown" and "Monkey" are so different, but something with both elements could be difficult to sort.

Re:That's Easy

[sakdoctor]

And yet google don't even convert petabytes to libraries of congress in the google calculator.
Or perhaps I got the syntax wrong.

Re:That's Easy

[sakdoctor]

Huh? This isn't the parent post I was trying to reply to.

Its About Time....

[Anonymous Coward]

Finaly... A system with enough power to run vista efficiently.

Re:Its About Time....

[peragrin]

Not only that the extra processors aren't covered under the EULA and require special extra licenses.

Not impressive...

[g0dsp33d]

Not a big deal, that's just the data they have on you.

Re:tagging

[gardyloo]

pr0n for Geeks, volume 18: Sorting On-the-Fly

0s and 1s

[johno.ie]

That's a lot of computing power to use just to get 4,000,000,000,000 0s and 4,000,000,000,000 1s.

nice one, Google...

[Tastecicles]

...fancy doing my mp3 collection?

Re:Sort? Sort what?

[Dpaladin]

Sorting a petabyte sounds pretty impressive, but I don't think it was a whole yotta work.

Amazing feat...

[Duncan3]

Today from Google, the god of all things and doer of all things good in the universe, many millions of dollars in computer equipment were able to sort lots of things, in about the amount of time you would think it would take for millions of dollars of equipment to sort things.

In other news, a woodchuck was found chucking wood as fast as a woodchuck could chuck wood.

Congrats Google, you have a HUGE data set, and an even bigger wallet.

Re:BCA's?

[SEWilco]

Can we convert that to number of bad car analogies?

Sure, it's -4.15 Edsels.

USSR

[Anonymous Coward]

In Soviet Russia, 6 petabytes sort YOU in ONE hour.