Slashdot Mirror

Its not really that expensive. Its just complex. Cloudera Hadoop is free for download. The support is what costs money. If you wanted, you could hire a bunch of green employees straight out of college with CIS degrees and pay the $1500 each to get them all certified with Cloudera. Then just buy two racks worth of commodity servers or buy virtualized space in a cloud environment. All in, you are looking at under a Million Dollars to get an enterprise class big data platform that can store more than a Petabyte of data. The tricky part is the months or years required to start streaming or scooping all the data into the data lake.

An issue recently came up on my Engineering team where a pig mapreduce job that stores in hbase slowed over the course of completing tasks until all the tasks failed due to timeouts. What appeared to be happening was a gc failure and pause due to tenure region exaustion and the built in cluster function to kill off the garbage collecting regionserver. The link below describes the issue and possible workarounds by implementing a custom memory allocation strategy. It's also a must read for anyone who isn't a java garbage collection expert. http://www.cloudera.com/blog/2011/02/avoiding-full-gcs-in-hbase-with-memstore-local-allocation-buffers-part-1/

You can get a master/slave combo VMWare VM at http://www.cloudera.com./ They also have packages for Ubuntu, I made an at-home cluster of VMs with one master and a slave that I can replicate.

First, I'm not really sure what you mean by "copy it into HDFS", since that's usually where data is stored in the first place. Copying in lots of data without giving the cluster time to stabilize can cause it to go into safe mode, where it won't make changes until everything's properly distributed. It's version 0.20. Don't expect perfection just yet.

There are also experts out there who will be quite happy to help get things running better. My company has been using Hadoop quite successfully with their help.

Not only is it used by Yahooo, but also by Facebook, who get 15TB of new data a day to handle. Checkout the very useful free vids from Cloudera. http://www.cloudera.com/hadoop-training-thinking-at-scale You can download a canned VM preloaded with Hadoop/Pig/Hive goodness, even a copy of Eclipse preconfigured. http://www.cloudera.com/hadoop-training-virtual-machine

Not only is it used by Yahooo, but also by Facebook, who get 15TB of new data a day to handle. Checkout the very useful free vids from Cloudera. http://www.cloudera.com/hadoop-training-thinking-at-scale You can download a canned VM preloaded with Hadoop/Pig/Hive goodness, even a copy of Eclipse preconfigured. http://www.cloudera.com/hadoop-training-virtual-machine