Slashdot Mirror
Book Review: Hadoop Beginner's Guide
First time accepted submitter sagecreek writes "Hadoop is an open-source, Java-based framework for large-scale data processing.
Typically, it runs on big clusters of computers working together to crunch large
chunks of data. You also can run Hadoop in "single-cluster mode" on a
Linux machine, Windows PC or Mac, to learn the technology or do testing and
debugging. The Hadoop framework, however, is not quickly mastered. Apache's
Hadoop wiki cautions: "If you do not know about classpaths,
how to compile and debug Java code, step back from Hadoop and learn a bit more about Java before
proceeding." But if you are reasonably comfortable with Java, the well-written
Hadoop Beginner's Guide by Garry Turkington can help you start mastering this
rising star in the Big Data constellation." Read below for the rest of Si's review.
Hadoop Beginner's Guide
author
Garry Turkington
pages
374
publisher
Packt Publishing
rating
9/10
reviewer
Si Dunn
ISBN
9781849517300
summary
Explains and shows how to use Hadoop software in Big Data settings.
Dr. Turkington is vice president of data engineering and lead architect for London-based Improve Digital. He holds a doctorate
in computer science from Queens University of Belfast in Northern Ireland. His
Hadoop Beginner's Guide provides an effective overview of Hadoop and hands-on guidance in how to use
it locally, in distributed hardware clusters, and out in the cloud.
Packt Publishing provided a review copy of the book. I have reviewed one other Packt book previously.
Much of the first chapter is devoted to "exploring the trends that led to Hadoop's creation and its enormous success." This includes brief discussions of Big Data, cloud computing, Amazon Web Services, and the differences between "scale-up" (using increasingly larger computers as data needs grow) and "scale-out" (spreading the data processing onto more and more machines as demand expands).
Dr. Turkington writes, "One of the most confusing aspects of Hadoop to a newcomer is its various components, projects, sub-projects, and their interrelationships."
His 374-page book emphasizes three major aspects of Hadoop: (1) its common projects; (2) the Hadoop Distributed File System (HDFS); and (3) MapReduce.
He explains, "Common projects comprise a set of libraries and tools that help the Hadoop product work in the real world."
The HDFS, meanwhile, "is a filesystem unlike most you may have encountered before." As a distributed filesystem, it can spread data storage across many nodes. "[I]t stores files in blocks typically at least 64 MB in size, much larger than the 4-32 KB seen in most filesystems." The book briefly describes several features, strengths, weaknesses, and other aspects of HDFS.
Finally, MapReduce is a well-known programming model for processing large data sets. Typically, MapReduce is used with clusters of computers that perform distributed computing. In the "Map" portion of the process, a single problem is split into many subtasks that are then assigned by a master computer to individual computers known as nodes (and there can be sub-nodes). During the "Reduce" part of the task, the master computer gathers up the processed data from the nodes, combines it and outputs a response to the problem that was posed to be solved. (MapReduce libraries are now available for many different computer languages, including Hadoop.)
"The developer focuses on expressing the transformation between source and result data sets, and the Hadoop framework manages all aspects of job execution, parallelization, and coordination," Dr. Turkington notes. He calls this "possibly the most important aspect of Hadoop. The platform takes responsibility for every aspect of executing the processing across the data. After the user defines the key criteria for the job, everything else becomes the responsibility of the system."
In this 11-chapter book, the first two chapters introduce Hadoop and explain how to install and run the software.
Three chapters are devoted to learning to work with MapReduce, from beginner to advanced levels. And the author stresses: "In the book, we will be learning how to write MapReduce programs to do some serious data crunching and how to run them on both locally managed and AWS-hosted Hadoop clusters." ["AWS" is "Amazon Web Services."]
Chapter 6, titled "When Things Break" zeroes in on Hadoop's "resilience to failure and an ability to survive failures when they do happen.much of the architecture and design of Hadoop is predicated on executing in an environment where failures are both frequent and expected." But node failures and numerous other problems still can arise, so the reader is given an overview of potential difficulties and how to handle them.
The next chapter, "Keeping Things Running," lays out what must be done to properly maintain a Hadoop cluster and keep it tuned and ready to crunch data.
Three of the remaining chapters show how Hadoop can be used elsewhere within an organization's systems and infrastructure, by personnel who are not trained to write MapReduce programs.
Chapter 8, for example, provides "A Relational View on Data with Hive." What Hive provides is "a data warehouse that uses MapReduce to analyze data stored on HDFS," Dr. Turkington notes. "In particular, it provides a query language called HiveQL that closely resembles the common Structured Query Language (SQL) standard."
Using Hive as an interface to Hadoop "not only accelerates the time required to produce results from data analysis, it significantly broadens who can use Hadoop and MapReduce. Instead of requiring software development skills, anyone with a familiarity with SQL can use Hive," the author states.
But, as Chapter 9 makes clear, Hive is not a relational database, and it doesn't fully implement SQL. So the text and code examples in Chapter 9 illustrate (1) how to set up MySQL to work with Hadoop and (2) how to use Sqoop to transfer bulk data between Hadoop and MySQL.
Chapter 10 shows how to set up and run Flume NG. This is a distributed service that collects, aggregates, and moves large amounts of log data from applications to Hadoop's HDFS.
The book's final chapter, "Where to Go Next," helps the newcomer see what else is available beyond the Hadoop core product. "There are," Dr. Turkington emphasizes, "a plethora of related projects and tools that build upon Hadoop and provide specific functionality or alternative approaches to existing ideas." He provides a quick tour of several of the projects and tools.
A key strength of this beginner's guide is in how its contents are structured and delivered. Four important headings appear repeatedly in most chapters. The "Time for action" heading singles out step-by-step instructions for performing a particular action. The "What just happened?" heading highlights explanations of "the working of tasks or instructions that you have just completed." The "Pop quiz" heading, meanwhile, is followed by short, multiple-choice questions that help you gauge your understanding. And the "Have a go hero" heading introduces paragraphs that "set practical challenges and give you ideas for experimenting with what you have learned."
Hadoop can be downloaded free from the Apache Software Foundation's Hadoop website.
Dr. Turkington's book does a good job of describing how to get Hadoop running on Ubuntu and other Linux distributions. But while he assures that "Hadoop does run well on other systems," he notes in his text: "Windows is supported only as a development platform, and Mac OS X is not formally supported at all." He refers users to Apache's Hadoop FAQ wiki for more information. Unfortunately, few details are offered there. So web searches become the best option for finding how-to instructions for Windows and Macs.
Running Hadoop on a Windows PC typically involves installing Cygwin and openSSH, so you can simulate using a Linux PC. But other choices can be found via sites such as Hadoop Wizard and Hadoop on Windows with Eclipse".
To install Hadoop on a Mac running OS X Mountain Lion, you will need to search for websites that offer how-to tips. Here is one example.
There are other ways get access to Hadoop on a single computer, using other operating systems or virtual machines. Again, web searches are necessary. The Cloudera Enterprise Free product is one virtual-machine option to consider.
Once you get past the hurdle of installing and running Hadoop, Garry Turkington's well-written, well-structured Hadoop Beginner's Guide can start you moving down the lengthy path to becoming an expert user.
You will have the opportunity, the book's tagline states, to "[l]earn how to crunch big data to extract meaning from the data avalanche."
Si Dunn is an author, screenwriter, and technology book reviewer.
You can purchase Hadoop Beginner's Guide from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Packt Publishing provided a review copy of the book. I have reviewed one other Packt book previously.
Much of the first chapter is devoted to "exploring the trends that led to Hadoop's creation and its enormous success." This includes brief discussions of Big Data, cloud computing, Amazon Web Services, and the differences between "scale-up" (using increasingly larger computers as data needs grow) and "scale-out" (spreading the data processing onto more and more machines as demand expands).
Dr. Turkington writes, "One of the most confusing aspects of Hadoop to a newcomer is its various components, projects, sub-projects, and their interrelationships."
His 374-page book emphasizes three major aspects of Hadoop: (1) its common projects; (2) the Hadoop Distributed File System (HDFS); and (3) MapReduce.
He explains, "Common projects comprise a set of libraries and tools that help the Hadoop product work in the real world."
The HDFS, meanwhile, "is a filesystem unlike most you may have encountered before." As a distributed filesystem, it can spread data storage across many nodes. "[I]t stores files in blocks typically at least 64 MB in size, much larger than the 4-32 KB seen in most filesystems." The book briefly describes several features, strengths, weaknesses, and other aspects of HDFS.
Finally, MapReduce is a well-known programming model for processing large data sets. Typically, MapReduce is used with clusters of computers that perform distributed computing. In the "Map" portion of the process, a single problem is split into many subtasks that are then assigned by a master computer to individual computers known as nodes (and there can be sub-nodes). During the "Reduce" part of the task, the master computer gathers up the processed data from the nodes, combines it and outputs a response to the problem that was posed to be solved. (MapReduce libraries are now available for many different computer languages, including Hadoop.)
"The developer focuses on expressing the transformation between source and result data sets, and the Hadoop framework manages all aspects of job execution, parallelization, and coordination," Dr. Turkington notes. He calls this "possibly the most important aspect of Hadoop. The platform takes responsibility for every aspect of executing the processing across the data. After the user defines the key criteria for the job, everything else becomes the responsibility of the system."
In this 11-chapter book, the first two chapters introduce Hadoop and explain how to install and run the software.
Three chapters are devoted to learning to work with MapReduce, from beginner to advanced levels. And the author stresses: "In the book, we will be learning how to write MapReduce programs to do some serious data crunching and how to run them on both locally managed and AWS-hosted Hadoop clusters." ["AWS" is "Amazon Web Services."]
Chapter 6, titled "When Things Break" zeroes in on Hadoop's "resilience to failure and an ability to survive failures when they do happen.much of the architecture and design of Hadoop is predicated on executing in an environment where failures are both frequent and expected." But node failures and numerous other problems still can arise, so the reader is given an overview of potential difficulties and how to handle them.
The next chapter, "Keeping Things Running," lays out what must be done to properly maintain a Hadoop cluster and keep it tuned and ready to crunch data.
Three of the remaining chapters show how Hadoop can be used elsewhere within an organization's systems and infrastructure, by personnel who are not trained to write MapReduce programs.
Chapter 8, for example, provides "A Relational View on Data with Hive." What Hive provides is "a data warehouse that uses MapReduce to analyze data stored on HDFS," Dr. Turkington notes. "In particular, it provides a query language called HiveQL that closely resembles the common Structured Query Language (SQL) standard."
Using Hive as an interface to Hadoop "not only accelerates the time required to produce results from data analysis, it significantly broadens who can use Hadoop and MapReduce. Instead of requiring software development skills, anyone with a familiarity with SQL can use Hive," the author states.
But, as Chapter 9 makes clear, Hive is not a relational database, and it doesn't fully implement SQL. So the text and code examples in Chapter 9 illustrate (1) how to set up MySQL to work with Hadoop and (2) how to use Sqoop to transfer bulk data between Hadoop and MySQL.
Chapter 10 shows how to set up and run Flume NG. This is a distributed service that collects, aggregates, and moves large amounts of log data from applications to Hadoop's HDFS.
The book's final chapter, "Where to Go Next," helps the newcomer see what else is available beyond the Hadoop core product. "There are," Dr. Turkington emphasizes, "a plethora of related projects and tools that build upon Hadoop and provide specific functionality or alternative approaches to existing ideas." He provides a quick tour of several of the projects and tools.
A key strength of this beginner's guide is in how its contents are structured and delivered. Four important headings appear repeatedly in most chapters. The "Time for action" heading singles out step-by-step instructions for performing a particular action. The "What just happened?" heading highlights explanations of "the working of tasks or instructions that you have just completed." The "Pop quiz" heading, meanwhile, is followed by short, multiple-choice questions that help you gauge your understanding. And the "Have a go hero" heading introduces paragraphs that "set practical challenges and give you ideas for experimenting with what you have learned."
Hadoop can be downloaded free from the Apache Software Foundation's Hadoop website.
Dr. Turkington's book does a good job of describing how to get Hadoop running on Ubuntu and other Linux distributions. But while he assures that "Hadoop does run well on other systems," he notes in his text: "Windows is supported only as a development platform, and Mac OS X is not formally supported at all." He refers users to Apache's Hadoop FAQ wiki for more information. Unfortunately, few details are offered there. So web searches become the best option for finding how-to instructions for Windows and Macs.
Running Hadoop on a Windows PC typically involves installing Cygwin and openSSH, so you can simulate using a Linux PC. But other choices can be found via sites such as Hadoop Wizard and Hadoop on Windows with Eclipse".
To install Hadoop on a Mac running OS X Mountain Lion, you will need to search for websites that offer how-to tips. Here is one example.
There are other ways get access to Hadoop on a single computer, using other operating systems or virtual machines. Again, web searches are necessary. The Cloudera Enterprise Free product is one virtual-machine option to consider.
Once you get past the hurdle of installing and running Hadoop, Garry Turkington's well-written, well-structured Hadoop Beginner's Guide can start you moving down the lengthy path to becoming an expert user.
You will have the opportunity, the book's tagline states, to "[l]earn how to crunch big data to extract meaning from the data avalanche."
Si Dunn is an author, screenwriter, and technology book reviewer.
You can purchase Hadoop Beginner's Guide from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
How many people on the planet actually manage "Big Data"? Isn't that the kind of thing that happens as a happy accident when your humble web site becomes the next FaceBubbleSpace? You can't plan for that.
Sure, there are other places where it happens--large corporations, governments, maybe some academic studies.
Really though, I have a hard time imagining that there are really a lot of people who deal with BD. Does anybody have numbers on it? What's the definition, anyway? Is Slashdot's archive BD?
For all intensive purposes, "whom" is no longer a word. That begs the question, "who cares"?
Hadoop is not a browser plugin.
I read the internet for the articles.
Could you elaborate, or provide sources?
Hadoop provides an answer to very specific questions involving large amounts of data, and isn't intended to be a database or other storage mechanism.
LegendMUD
Remember: when you need a book reviewed on Slashdot, make sure you publish it on Packt.
It's HDFS ... apparently the book didn't teach the reviewer much. "Hadoop Distributed File System."
If it's just a bleeping tool for data processing, what should the user need to be a Java developer to use it?
Hadoop. For when you feel like something fun to do on a friday night.
You can't handle the truth.
There are alternatives to Java for writing your MR programs if you have better things to do that worry about class paths etc :-) http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/
And back in the 80's BT used PL/1 (plus a bit of Fortran 77) to do Map reduce as the core of a billing system for the dialcom systems -even manged ro sell it to other telcos as a replacement for the Dialcom product.
Well, I think that great amount of confusion results from the way Java is marketed. For a website that has a larger than normal number of technical people, /. as an aggregate still displays fundamental misunderstanding of what Java is and what it is not.
For example Java is not Javascript. A browser sandbox that runs Java plugin has nothing to do with server side Java applications, that for example can run in servlet containers and servers like Tomcat Apache, Jetty, iPlanet, Resin (by Caucho), Enhydra, and such.
Also Java is a platform, but often people complain about how many different paradigms it supports, how many various libraries and servers and different ideas and packages exist around it, as if anybody forces any developer or a company to work with all or any of that stuff. Again, complaints like: J2EE is too heavy, so Java sucks, well, you don't need J2EE for your dinky application and actually you don't need it if you develop a non-dinky application but still support various tech that a serious application requires (transactions, multi-threading and multi-user support, etc.)
For people who apparently pride themselves being 'above marketing', developers somehow fell victims to various marketing around Java and didn't bother to check for themselves what the hell all this is about.
You can't handle the truth.
For people who apparently pride themselves being 'above marketing', developers somehow fell victims to various marketing around Java and didn't bother to check for themselves what the hell all this is about.
As a developer who never had many reasons to do very much with (actual) Java, I have to say I lost of respect for the platform when Oracle started bundling the runtime with the Ask toolbar on Windows. Because nothing says "serious computing platform" like having to make sure it doesn't install a toolbar on every update. /rant
Momentarily, the need for the construction of new light will no longer exist.
As a developer who never had many reasons to do very much with (actual) Java, I have to say I lost of respect for the platform when Oracle started bundling the runtime with the Ask toolbar on Windows.
- I don't know, as a thinking human I have to say that what Oracle does with Java browser plugin reflects poorly on Oracle and has nothing to do with Java. As I said, the browser plugin is irrelevant for majority of what Java is actually used for.
Personally I don't even have Java browser plugin on any of my machines because I don't have a use for it. At the same time I have developed plenty Java applications (as in Java language running on a JVM that resides on a server and has some form of application container on top of the JVM where the application is actually running).
So you are exactly what I am talking about - a victim of a marketing campaign, whichever form it takes.
You can't handle the truth.
I can guarantee you that developers who use Java for serious development work never worry about the Ask toolbar.
I never worked with Java, so honestly I didn't know. All I hear left and right is "Java sux because vulnerabilities". So I was wondering.
But judging from the smug answers below, I made the impardonable mistake of not knowing EVERYTHING. Oh well.
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
Why is this insightful? Shouldn't this comment have a low moderation value. A high moderation leads people into wasting their time look at this post and into a parent post who needed some help. Why should anyone else care.
Of course this is not the fault of the parent or the poster. Is moderation really this crappy? I guess moderators want to moderate something in this article and there just aren't any good posts.
Chris Mesterharm
...which is a good reason for asking questions. It it NOT a good reason for others to be dicks about it.
...gis sdrawkcab (usually not responding to ACs; don't bother posting as AC)
As an FYI, it's not the browser plugin that has the opt-out toolbar installation, it's the JVM itself. Do I think it's truly a sign of Java's strength/weakness as a platform? No, but as you say, it's horrible marketing.
Momentarily, the need for the construction of new light will no longer exist.
Agreed, but the OP's point was about Java suffering from bad marketing, and IMHO, the opt-out toolbar is a prime example.
Momentarily, the need for the construction of new light will no longer exist.
You mean the JRE that is part of the plugin installation? It cannot be the JRE itself, it must the the installer of the plugin. I don't install JVM this way or Java plugin (at all), I just download the necessary installation package for the JVM/SDK as needed. If I want Oracle SDK I get it from here and when it installs, it does not install any 'ask bar' or anything like that, so I am not even clear as to how people get these things. I download the SDK or JVM for a Linux distro, which is either a binary that will decompress or just a tarball.
What you are saying is that when you are prompted for a browser plugin download, as part of the plugin you get the JRE (which is the Runtime Environment) and you also get some Ask bar or whatever.
But this scheme by Oracle then results in people getting angry with Java of all things and AFAIC this is similar to being angry at (for example) the C / C++ language and the x86 architecture because Microsoft ActiveX platform is sometimes used as a virus vector.
That's why I say there is so much confusion around Java. Java IS a language AND a platform, it's like C and x86 hardware architecture that can run a binary compiled from C. If somebody sells a computer full of Ask bars etc., the average user will not be inclined to believe that the C language and the CPU architecture are horrible (they may or may not be, but that would not be on the mind of a computer user).
But the marketing of Java is so poor (for such a useful platform, AFAIC), that it creates this type of ridiculous notion around it.
You can't handle the truth.
No, I mean the JRE itself. (Remember, I'm talking about Windows here.) Granted, this is not the SDK I'm talking about, just the standard JRE installed in Windows, not the browser, but I just updated it on this machine, and I had to uncheck the option to install the Ask toolbar. (link is not my blog, btw.)
I'm fully aware of the distinctions between Java, the browser plugins, Javascript, etc., and I realize that the toolbar is not part of the install on 'real' OSs. But the Windows installer is what most users see of Java.
Momentarily, the need for the construction of new light will no longer exist.
Well, I went looking at this and it's not the JRE that does this, it's the Oracle installer for Java. JRE is not the installer, it's the run time environment.
Here is one of these stories about this issue.
But yes, on Linux or Unix I don't get any of this nonsense, it's just a tarball that I untar into a directory, set the path and run the JVM for example. The binary installers for Linux or Unix are simple shell scripts with the same tarball basically as part of the script, it doesn't do any of this stuff.
As I said, this is a terrible marketing move, does huge amount of disservice to the entire concept of Java by bundling things with installers that clearly shouldn't be there. It's not enough that people are confused about what Java is or is not (because of such a huge number of things that people just call 'Java'), but adding this nonsense to installers is just evil. Personally I have various issues with Oracle, would have really preferred if IBM bought Sun's assets rather than Oracle.
You can't handle the truth.
All due respect...but even as a developer, to say it's not the JRE but the JRE's installer that bundles the toolbar, is pretty pedantic. I really just wanted to point out this comes with standard install of Java itself, not the browser plugin.
Momentarily, the need for the construction of new light will no longer exist.
Isn't the devil in the details, while everything else is just marketing?
You can't handle the truth.