Slashdot Mirror

← Back to Stories (view on slashdot.org)

Yahoo Releases Open Source Hadoop Distribution

Posted by timothy on from the spread-it-out-in-little-chunks dept.

ruphus13 writes "Yahoo has been a vociferous Apache Hadoop user and supporter for several years now, and uses it extensively within its Search technologies. Hadoop has been gaining popularity in the Cloud Computing space, with companies like the NYTimes converting 4TB and 11 million articles to PDFs in under 24 hours using Hadoop and EC2 in late 2007. Hadoop has been made available in Amazon's cloud and Yahoo has now released its own Hadoop version. From the article: 'At today's Hadoop Summit in Silicon Valley, Yahoo! announced the availability of the Yahoo! Distribution of Hadoop, a source-only version of Apache Hadoop that Yahoo! uses within its own search engine. [Hadoop] is an open source software framework that helps process very large data sets, and is widely used in large-scale data mining applications as well as in search tools at sites like Facebook and many others. For developers and users interested in Hadoop, it's worth noting that the Yahoo! Distribution of Hadoop has been widely tested and developed at Yahoo! for years now.'"

19 of 49 comments (clear)

  1. Timely article by C18H27NO3+ · · Score: 3, Informative

    Perhaps the Ask Slashdot inquirer in this thread will find this news usefull.

  2. Hadoop? by ickleberry · · Score: 5, Insightful

    Can we bring back the ordinary, sensible pre-Web 2.0 names please?

    1. Re:Hadoop? by fancellu · · Score: 2, Informative

      Its the name of the main developer's kid's toy elephant.

    2. Re:Hadoop? by ickleberry · · Score: 3, Funny

      Wow. I'd never have thunk it. Thought it was short for "I had a poop"

    3. Re:Hadoop? by Just+Some+Guy · · Score: 4, Insightful

      Like Yahoo!?

      --
      Dewey, what part of this looks like authorities should be involved?
    4. Re:Hadoop? by phantomfive · · Score: 2, Funny

      Only if you are vociferous about it.

      --
      Qxe4
    5. Re:Hadoop? by whereiswaldo · · Score: 2, Funny

      ... and the Slashdot mob's credibility spirals downward...

  3. Hadoop is awesome by fancellu · · Score: 5, Informative

    Not only is it used by Yahooo, but also by Facebook, who get 15TB of new data a day to handle. Checkout the very useful free vids from Cloudera. http://www.cloudera.com/hadoop-training-thinking-at-scale You can download a canned VM preloaded with Hadoop/Pig/Hive goodness, even a copy of Eclipse preconfigured. http://www.cloudera.com/hadoop-training-virtual-machine

    1. Re:Hadoop is awesome by Anonymous Coward · · Score: 2, Insightful

      It's actually 25TB of photos per week. He was only out by a factor of 4. Of course, this data isn't indexed, so he had no point.

    2. Re:Hadoop is awesome by DamnStupidElf · · Score: 2, Informative

      "Hive/Hadoop cluster at Facebook stores more than 2PB of uncompressed data and routinely loads 15 TB of data daily."

    3. Re:Hadoop is awesome by zerocool^ · · Score: 2, Interesting

      We also use it extensively at Rackspace Email division. We generate about 200GB/day of logs from postfix and dovecot installs, and hadoop with mapreduce allows us to pull all sorts of metrics and diagnostic information in very short timeframes. It helps our customer facing support reps, as well as allows us to give more demanding customers the statistics and metrics that they want, plus it helps us with capacity planning and a bunch of other stuff.

      And it's designed to run on commodity hardware.

      http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data

      ~Wx

      --
      sig?
  4. Yahoo! and OSS by Alethes · · Score: 5, Insightful

    Yahoo! really does get a lot of flack around here, but I have to say, they have contributed quite a bit of free and open-source software for developers to use. The list of of APIs and web services that are available is quite impressive and many of them are better than Google's similar offerings (BOSS vs Google's AJAX search, for example). For anybody who's interested, I really recommend checking out the Yahoo! Developer Network site.

    1. Re:Yahoo! and OSS by linguizic · · Score: 2, Interesting

      THANK YOU!!!! I have found YDN enormously useful.

      It's also worth noting that Yahoo has made major contributions to PHP as Rasmus is a Yahoo himself.

      --
      Does this sig remind you of Agatha Christie?
    2. Re:Yahoo! and OSS by hairyfeet · · Score: 5, Interesting

      And folks like to make fun of Yahoo search, but after switching from Google I just can't ever even think about going back. The more/concept tab(that is the blue button below the search box) is just too nice to give up.

      Example- i just picked up "Blacksite:Area 51" for $5. I type in "blacksi" and there it is. From "Blacksite:Area 51" in the search box under more/related I have cheats,patch.system reqs, PS3.Xbox360, Midway games west,multiplayer modes, squad based shooters, release date by region, etc. Just from typing "blacksi" and picking area 51 from the drop down I have all those different avenues related to my search right there at the top where they are easy to get at. It really lets me hone in on an area, and in some cases like movies it finds me interviews with the director which i often don't even know who directed a particular flick.

      So those that haven't tried their search in a few years really ought to give it a whirl. The more/related concept tab at the top makes search so easy to drill down. Plus Yahoo has an opt out for ad matching if you are concerned about privacy. I looked and I don't think Google even has an "opt out" short of using ABP. So give it a go, its free and you might find the more/concepts button as useful as I do. And competition is always a good thing, right?

      --
      ACs don't waste your time replying, your posts are never seen by me.
  5. Why is this a big deal? by Eric+Smith · · Score: 2, Interesting
    Does the world need another Hadoop distribution? In a case like this, isn't a "distribution" just a fork going by a different name that has a more positive connotation? there some good reason they did it this way rather than just pushing their changes upstream to Apache? Did Apache not want them?

    I'll admit to knowing basically nothing about Hadoop, but if I saw the same article with "Hadoop" replaced by "GCC", "Postfix", or "OpenOffice", I wouldn't see it as being a good thing.

    1. Re:Why is this a big deal? by shadow42 · · Score: 3, Informative

      As far as I can tell, the distribution Yahoo is offering is just the vanilla Hadoop, but with Yahoo's patches on top of it. Yahoo is very involved in Hadoop's development (the project's founder is now employed by them), so a lot of their patches get incorporated back into Hadoop's source tree. Most of the changes Yahoo made are just performance/stability patches that haven't been incorporated into an official release yet. You could probably get the same distribution just by grabbing SVN trunk.

    2. Re:Why is this a big deal? by linguizic · · Score: 2, Interesting

      Does the world need another Linux distribution? The folks at Ubuntu thought so, and they've made an indelible mark on Linux. Just like Yahoo! is doing with Hadoop.

      --
      Does this sig remind you of Agatha Christie?
  6. Re:Obligatory Java is Slow Comment by dintech · · Score: 2, Informative

    Java has it's faults but being slow is no longer one of them. You should do some googling.

  7. Re:open source software by blueskies · · Score: 2, Informative

    Fail!

    ISO 32000-1:2008: PDF

    Because stitching together numerous TIFF files on your own is so much better!