Slashdot Mirror

First time accepted submitter sagecreek writes "Hadoop is an open-source, Java-based framework for large-scale data processing. Typically, it runs on big clusters of computers working together to crunch large chunks of data. You also can run Hadoop in "single-cluster mode" on a Linux machine, Windows PC or Mac, to learn the technology or do testing and debugging. The Hadoop framework, however, is not quickly mastered. Apache's Hadoop wiki cautions: "If you do not know about classpaths, how to compile and debug Java code, step back from Hadoop and learn a bit more about Java before proceeding." But if you are reasonably comfortable with Java, the well-written Hadoop Beginner's Guide by Garry Turkington can help you start mastering this rising star in the Big Data constellation." Read below for the rest of Si's review. Hadoop Beginner's Guide author Garry Turkington pages 374 publisher Packt Publishing rating 9/10 reviewer Si Dunn ISBN 9781849517300 summary Explains and shows how to use Hadoop software in Big Data settings. Dr. Turkington is vice president of data engineering and lead architect for London-based Improve Digital. He holds a doctorate in computer science from Queens University of Belfast in Northern Ireland. His Hadoop Beginner's Guide provides an effective overview of Hadoop and hands-on guidance in how to use it locally, in distributed hardware clusters, and out in the cloud.

Packt Publishing provided a review copy of the book. I have reviewed one other Packt book previously.

Much of the first chapter is devoted to "exploring the trends that led to Hadoop's creation and its enormous success." This includes brief discussions of Big Data, cloud computing, Amazon Web Services, and the differences between "scale-up" (using increasingly larger computers as data needs grow) and "scale-out" (spreading the data processing onto more and more machines as demand expands).

Dr. Turkington writes, "One of the most confusing aspects of Hadoop to a newcomer is its various components, projects, sub-projects, and their interrelationships."

His 374-page book emphasizes three major aspects of Hadoop: (1) its common projects; (2) the Hadoop Distributed File System (HDFS); and (3) MapReduce.

He explains, "Common projects comprise a set of libraries and tools that help the Hadoop product work in the real world."

The HDFS, meanwhile, "is a filesystem unlike most you may have encountered before." As a distributed filesystem, it can spread data storage across many nodes. "[I]t stores files in blocks typically at least 64 MB in size, much larger than the 4-32 KB seen in most filesystems." The book briefly describes several features, strengths, weaknesses, and other aspects of HDFS.

Finally, MapReduce is a well-known programming model for processing large data sets. Typically, MapReduce is used with clusters of computers that perform distributed computing. In the "Map" portion of the process, a single problem is split into many subtasks that are then assigned by a master computer to individual computers known as nodes (and there can be sub-nodes). During the "Reduce" part of the task, the master computer gathers up the processed data from the nodes, combines it and outputs a response to the problem that was posed to be solved. (MapReduce libraries are now available for many different computer languages, including Hadoop.)

"The developer focuses on expressing the transformation between source and result data sets, and the Hadoop framework manages all aspects of job execution, parallelization, and coordination," Dr. Turkington notes. He calls this "possibly the most important aspect of Hadoop. The platform takes responsibility for every aspect of executing the processing across the data. After the user defines the key criteria for the job, everything else becomes the responsibility of the system."

In this 11-chapter book, the first two chapters introduce Hadoop and explain how to install and run the software.

Three chapters are devoted to learning to work with MapReduce, from beginner to advanced levels. And the author stresses: "In the book, we will be learning how to write MapReduce programs to do some serious data crunching and how to run them on both locally managed and AWS-hosted Hadoop clusters." ["AWS" is "Amazon Web Services."]

Chapter 6, titled "When Things Break" zeroes in on Hadoop's "resilience to failure and an ability to survive failures when they do happen.much of the architecture and design of Hadoop is predicated on executing in an environment where failures are both frequent and expected." But node failures and numerous other problems still can arise, so the reader is given an overview of potential difficulties and how to handle them.

The next chapter, "Keeping Things Running," lays out what must be done to properly maintain a Hadoop cluster and keep it tuned and ready to crunch data.

Three of the remaining chapters show how Hadoop can be used elsewhere within an organization's systems and infrastructure, by personnel who are not trained to write MapReduce programs.

Chapter 8, for example, provides "A Relational View on Data with Hive." What Hive provides is "a data warehouse that uses MapReduce to analyze data stored on HDFS," Dr. Turkington notes. "In particular, it provides a query language called HiveQL that closely resembles the common Structured Query Language (SQL) standard."

Using Hive as an interface to Hadoop "not only accelerates the time required to produce results from data analysis, it significantly broadens who can use Hadoop and MapReduce. Instead of requiring software development skills, anyone with a familiarity with SQL can use Hive," the author states.

But, as Chapter 9 makes clear, Hive is not a relational database, and it doesn't fully implement SQL. So the text and code examples in Chapter 9 illustrate (1) how to set up MySQL to work with Hadoop and (2) how to use Sqoop to transfer bulk data between Hadoop and MySQL.

Chapter 10 shows how to set up and run Flume NG. This is a distributed service that collects, aggregates, and moves large amounts of log data from applications to Hadoop's HDFS.

The book's final chapter, "Where to Go Next," helps the newcomer see what else is available beyond the Hadoop core product. "There are," Dr. Turkington emphasizes, "a plethora of related projects and tools that build upon Hadoop and provide specific functionality or alternative approaches to existing ideas." He provides a quick tour of several of the projects and tools.

A key strength of this beginner's guide is in how its contents are structured and delivered. Four important headings appear repeatedly in most chapters. The "Time for action" heading singles out step-by-step instructions for performing a particular action. The "What just happened?" heading highlights explanations of "the working of tasks or instructions that you have just completed." The "Pop quiz" heading, meanwhile, is followed by short, multiple-choice questions that help you gauge your understanding. And the "Have a go hero" heading introduces paragraphs that "set practical challenges and give you ideas for experimenting with what you have learned."

Hadoop can be downloaded free from the Apache Software Foundation's Hadoop website.

Dr. Turkington's book does a good job of describing how to get Hadoop running on Ubuntu and other Linux distributions. But while he assures that "Hadoop does run well on other systems," he notes in his text: "Windows is supported only as a development platform, and Mac OS X is not formally supported at all." He refers users to Apache's Hadoop FAQ wiki for more information. Unfortunately, few details are offered there. So web searches become the best option for finding how-to instructions for Windows and Macs.

Running Hadoop on a Windows PC typically involves installing Cygwin and openSSH, so you can simulate using a Linux PC. But other choices can be found via sites such as Hadoop Wizard and Hadoop on Windows with Eclipse".

To install Hadoop on a Mac running OS X Mountain Lion, you will need to search for websites that offer how-to tips. Here is one example.

There are other ways get access to Hadoop on a single computer, using other operating systems or virtual machines. Again, web searches are necessary. The Cloudera Enterprise Free product is one virtual-machine option to consider.

Once you get past the hurdle of installing and running Hadoop, Garry Turkington's well-written, well-structured Hadoop Beginner's Guide can start you moving down the lengthy path to becoming an expert user.

You will have the opportunity, the book's tagline states, to "[l]earn how to crunch big data to extract meaning from the data avalanche."

Si Dunn is an author, screenwriter, and technology book reviewer.

You can purchase Hadoop Beginner's Guide from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.

Slashback tonight brings some corrections, clarifications, and updates to previous Slashdot stories including the Supreme Court declines Falwell's appeal, GP2X now shipping in the US, a new version of Systrace released, Lessig and Stallman look back at Sun's OpenDRM, NASA jumps on the anti-matter propulsion bandwagon, GoDaddy donates $10,000 to OpenSSH, Ellison explains why he would NOT acquire Novell or Red Hat, and pictures of the Ball State wireless 'sculpture' -- Read on for details.

The Supreme Court declines Falwell's Appeal. yEvb0 writes "The US Supreme Court has refused to hear the appeal of Jerry Falwell, who claims that "gripe site" http://www.fallwell.com/ infringes on his trademark by luring surfers away from his own site. Despite winning a case in federal court, the 4th U.S. Circuit Court of Appeals disagreed with Falwell last year and said that operator Christopher Lamparello was free to operate his site about Falwell's views on gays because he 'clearly created his Web site intending only to provide a forum to criticize ideas, not to steal customers.'"

GP2X now shipping in the US. An anonymous reader writes "The Gamepark GP2X, a Linux-based handheld gaming platform that runs native and emulated games, is now shipping in the US, according to LinuxDevices. The device can reportedly run more than a thousand classic arcade games, through open-source console game emulators such as MAME, SNES, Genesis, and PC Engine. It has a 3.5-inch QVGA (320x240) color TFT LCD screen, and includes a media player supporting MPEG, JPEG, and MP3 formats."

New version of Systrace released. Niels writes "I just recently released a new version of Systrace that runs on Linux without requiring any kernel patches. I termed it the Phoenix release because it has been almost three years since I did any work on Systrace. However, I finally had the need to do some sandboxing on Linux without being able to change the kernel. So, voila, after a few late nights, here we go: Systrace for Linux using a ptrace back end."

Lessig and Stallman look back at Sun's OpenDRM. H4x0r Jim Duggan writes "The Register has an excellent article featuring Lessig and Stallman on 'Open Source' DRM. The spark for the article came from comments made about Sun's 'OpenDRM' by Lessig which were not wholly negative and were interpreted by some as an endorsement. Lessig clarifies: 'There's no disagreement about where we should end up - No DRM.'"

NASA jumps on the anti-matter propulsion bandwagon. steveo777 writes "NASA has an interesting read about creating yet another form of rocket propulsion. They plan on using Anti-electrons (positrons) combine with normal electrons to release enough energy to fuel the way to Mars and back. Its byproduct will be lower energy gamma radiation. From the article, '"Our advanced designs, like the gas core and the ablative engine concepts, could take astronauts to Mars in half that time, and perhaps even in as little as 45 days," said Kirby Meyer, an engineer with Positronics Research on the study.'"

GoDaddy donates $10,000 to OpenSSH. wcbrown writes "Go Daddy has donated $10,000 to the OpenSSH project, which is apparently used extensively within the company." This is another great donation in what hopefully will continue to be a trend within the community. No word on when the blinking will stop.

Ellison explains why he would NOT acquire Novell or Red Hat. Robert writes to tell us CBROnline is reporting that a recent statement by Larry Ellison covered so extensively in the news regarding speculation about why Oracle might be "planning to buy Novell or Red Hat" may have been a little off base. The full transcript of his interview with the FT is illuminating precisely because it reveals why the company would NOT acquire either Novell or Red Hat, and - apparently - why the company did not buy JBoss.

Pictures of the Ball State Wireless 'sculpture'. popeguilty writes "Slashdot readers may recall the story about the Wireless 'Sculpture' at Ball State University. The artwork is up and running, and I've got a few pictures posted for general consumption."

BSDForums writes "OpenSSH 4.2 has been released. OpenSSH is a 100% complete SSH protocol version 1.3, 1.5 and 2.0 implementation and includes sftp client and server support. Changes since OpenSSH 4.1 include security bug fixes relating to GatewayPorts, and GSSAPI, which eliminates the risk of credentials being inadvertently exposed to an untrusted user/host. A new compression method, proactive changes for signed vs. unsigned integer bugs, and many additional bugfixes and improvements highlight this release."

heydrick writes "The OpenSSH project is five years old. Project member Damien Miller writes, 'Five years ago, in late September 1999, the OpenSSH project was started. It began with an audit, cleanup and update of the last free version of Tatu Ylonen's legacy ssh-1.2.12 code. The project quickly gathered pace, attracting a portability effort and, in early 2000, an independent implementation of version 2 of the SSH protocol. Since then, OpenSSH has led in the implementation of proactive security techniques such as privilege separation & auto-reexecution.' Yaa for OpenSSH."

AlanS2002 points out today's article from Iain Thomson on vnu.net, which says that "Linux systems are getting tougher for hackers to crack, security experts have reported today," summarizing "A study conducted by the Honeynet Project has found that it takes about 3 months before a unpatched Linux machine will be owned, compared with about 72 hours in the past. According to a report on the study default installations are now more secure with less services enabled by default, added to this is newer versions of software such as OpenSSH being more secure. Interestingly Solaris 8 and 9 did not fare so well."

thequbemaster writes "The OpenBSD project, responsible for OpenSSH, OpenBGPD, and OpenNTPD, has created OpenCVS, a BSD licensed implementation of CVS client and server. From the site: 'It aims to be as compatible as possible with other CVS implementations, except when particular features reduce the overall security of the system. The OpenCVS project was started after discussions regarding the latest GNU CVS vulnerabilities that came out. Although CVS is widely used, its development has been mostly stagnant in the last years and many security issues have popped up, both in the implementation and in the mechanisms.' No releases are available yet. The README in the OpenCVS CVS repository states that the server is not ready yet, but looks like the client is usable." Update: 12/15 20:18 GMT by T : This project was mentioned briefly the other day, too.

44BSD writes "The folks at OpenBSD have announced OpenNTPd, a BSD-licensed implementation of the NTP protocol. As with OpenSSH, there are two teams responsible for the code, which, like OpenSSH, is available in pure OpenBSD and portable versions." TLA FYI: This Wikipedia entry offers a quick overview of the Network Time Protocol. Read on below for some more on OpenNTPd.

"As explained at the project home page:

One team does strictly OpenBSD-based development, aiming to produce code that is as clean, simple, and secure as possible. We believe that simplicity without the portability "goop" allows for better code quality control and easier review. The other team then takes the clean version and makes it portable, by adding the portability "goop" so that it will run on many operating systems.

I have been using the OpenBSD variant of this daemon as it exists in the OpenBSD 3.5-current branch of the OS, and it has worked flawlessly with absolutely zero configurational effort. The supplied config file is sufficient to have the daemon synch against a randomly chosen stratum 2 server (pool.ntp.org, served up via round-robin A records). Aside from its simplicity, this daemon offers the ability to be selectively bound to a machine's network interfaces, rather than having to bind to all, as is the case with xntpd. The features provided by this implementation are probably sufficient for the majority of NTP users, and when the promised support for GPS and radio clocks is added, still more will be able to choose OpenNTPd. This additional diversity is welcome, indeed."

SiliconEntity writes "On the heels of multiple OpenSSH vulnerabilities, the OpenSSL project is now reporting a number of security vulnerabilities of its own. OpenSSL is a standard cryptographic library used in a wide variety of security applications. The new vulnerabilities range from denial-of-service attacks to stack corruption, which imply the possibility of running malicious code. New versions of the software are released today which address the vulnerabilities."

An anonymous reader writes "The OpenSSH team has uncovered multiple exploitable vulnerabilities in the days-old portable release of OpenSSH. That's right folks: time to patch *again*. 3.7.1p2 is now available. Instructions and mirror list here. Please note that this vulnerability only affects *portable* OpenSSH--so if you are running OpenBSD, you're safe. This vulnerability apparently has to do with PAM, so you can use the 'UsePam no' option in your config file. Info on the advisory here and here."

An anonymous reader writes "The OpenSSH team has uncovered multiple exploitable vulnerabilities in the days-old portable release of OpenSSH. That's right folks: time to patch *again*. 3.7.1p2 is now available. Instructions and mirror list here. Please note that this vulnerability only affects *portable* OpenSSH--so if you are running OpenBSD, you're safe. This vulnerability apparently has to do with PAM, so you can use the 'UsePam no' option in your config file. Info on the advisory here and here."

maiden_taiwan asks: "We're constructing a free groupware application by integrating well-known open source components: apache webserver, inn news server, ircd chat, scp for file transfer, etc. Unfortunately, each app has its own incompatible concept of a 'user identity.' Apache has the htpasswd module, IRC has nicknames, scp has public keys, NetNews has the poster's email address, and so forth. Has anyone managed to integrate a similar suite of apps using a single sign-on model, where a user has a single identity that is understood and carried through all these apps?"

cperciva writes "The original story is here. And more details are available from the guy's weblog here." Here's a mirror of that email message. Another reader writes, "Not really a trojan because all it does is make a connection to 203.62.158.32:6667." Still another writes "The tarball of the portable OpenSSH on ftp.openbsd.org is trojaned. The backdoor is only used during build - generated binaries are fine." There isn't much authoritative information available, but this appears legitimate - please be careful if you're updating any of your machines with code from ftp.openbsd.org, and we'll update this story with more links as information is available. Update: 08/01 19:13 GMT by M : OpenSSH now has an advisory.

RiverWolf asks: "As a Systems Administrator (a.k.a. 'paranoid security freak') I spend much of my time tightening down systems, loading patches, and just generally making sure no one does what they're not supposed too. While tools like ssh have become a staple for file transfer and terminal sessions, I recently began looking at all the little print servers we have throughout my offices and wondered "hmm, can those things be sniffed?". Until now, my focus for printing has always been 'just get it working', but if someone can sniff the print jobs (like payroll and other confidential information) as they go across the network, then it doesn't matter how locked down eveything else is. Is there a standard for secure (encrypted transmission) network printing, or does anyone know of a way to do this? I found this document that deals with it in a round about fashion, but with dozens of printers spread throughout multiple locations, I don't see it as an option."

Slashback tonight with another round of updates and errata regarding recent Slashdot stories. Read on for more on domain slamming, the process behind fixing and revealing the recent OpenSSH vulnerability, early photography, and a special note for residents of Maricopa County, Arizona.

Quick work by smart people. ciaweb writes "The OpenSSH group has revised its security advisory about the recent OpenSSH vulnerabilities. In it, they describe their decision-making process for releasing the bug information. It is interesting to contrast their procedure, which appears designed to maximize user protection, against Microsoft's, which appears to maximize Microsoft's protection."

Pardon me, sir, would you mind if I SLAMMED THIS HAMMER ON YOUR FINGERS?! D0wnsp0ut writes "I thumbed through my mail today and found what appeared to be a renewal notice for my domain. This one came from "Domain Registry of America." Verisign attempted something similar back in March and Bulkregister.com fought back and won an injunction, against the mailings. So watch out if your domain is getting close to expiring. I talked to my registrar (Register.com) and they're aware of it.
I'll scan the letter but have no place to post the pictures. Can anyone lend some bandwidth?"

Half the world has never eaten a Krispy Kreme donut, either. cshirky writes "I've just written an essay on the phrase ' Half the world has never made a phone call'. It's more 'voice telephony-y' than the usual telecom stories here, but after seeing the interest in media and the market that surfaced during my /. interview, I thought it might be of some interest."

Please stop sending my money to Redmond, OK? TrumpetPower! writes "All that brouhaha over Maricopa County's policy prohibiting companies or persons convicted of antitrust violations has had an effect. I just received the following note announcing a public forum scheduled for this coming Monday.

You recently inquired about the County's use of Microsoft products and the manner in which we license their software. We appreciate your interest in the County's technology plans. To provide a forum in which to discuss our technology direction and address any questions you may have, we will have Information Technology staff members available to meet with citizens at 8:30 am on Monday July 8th. The meeting location will be the County Administration Building at 301 W. Jefferson in Suite 420. Please RSVP your attendance so we can ensure that adequate facilities are available for the meeting.

Thank you for your inquiry,
Paul Allsing
Deputy CIO
Maricopa County
301 W Jefferson, Suite 420
Phoenix, AZ 85003"

Ah, but what about the first annoying family photographer? 7h3_B055 writes: "Contrary to this article on Slashdot claiming the first photograph was created in 1826, much evidence is pointing to the fact that the Shroud of Turin may have been an earlier example (substantially earlier) of photography using ingredients as basic as egg-white for treating cloth (the photopaper) and urine for developing it. The camera itself could have been a simple box with a hole in it and the exposure time would have been lengthy."

Of course, there are a lot of theories about the Shroud of Turin, and a google search is likely to intrigue you for days.

If you are following the flap over the use of the letters Ess, Ess and Aitch in product names -- SSH Communications Security Corporation has asked the OpenSSH project to stop using those letters in the name of their software -- a story on NewsForge adds more details. If you didn't catch it then, here's yesterday's NewsForge article as well. Good thing nobody is enforcing a trademark on "telnet," eh?

My favorite tidbit from the article is this: "[OpenBSD and OpenSSH Developer Theo] de Raadt cites U.S. trademark law that requires owners of trademarks to notify violators immediately ... de Raadt argues that Ylönen would have to be living under a rock not to be aware of OpenSSH before now. OpenSSH, released in December 1999 and in use before that, was used by more than 17% of all SSH users earlier this month, according to a study published on the University of Alberta Web site." Besides that, the story does a great job of listing other people whose products including "SSH" in their names have been left blissfully alone, making it seem that OpenSSH is getting what can only be called special treatment.

Of interest: here is a link to a page at openssh.com showing the legal papers received and scanned by members of the OpenSSH project, including the trademark application in question, showing an entirely lowercased "ssh" as the applied-for mark.

The European Unix Platform e-zine is running an interview with Theo de Raadt (OpenBSD head honcho) in which he talks a little about his thoughts on the WC/BSDI merger, embedded OpenBSD, and the openssh.{com,org} dispute. Alex de Joode, also involved in the dispute, has written a reply to Theo's points.

Bowie J. Poag was one of the folks who wrote to us about the domain name controversy regarding OpenSSH. (I've included the full letter below). They're in the interesting situation of /having/ to be a .com, because a squatter has taken the openssh.org domain name. Read the letter below - it's a stickier situation than the other squatting issues we've talked about. Update: 03/07 04:58 by E : Alex de Joode has written his own response here. I hope this can be resolved amicably.

Please be advised that OpenSSH.ORG is NOT the official domain name for OpenSSH development. The name was taken by a someone not affiliated with the OpenSSH development team when news of OpenSSH was first leaked to the community. The correct Web and e-mail address for the OpenSSH development effort is OpenSSH.COM instead of .ORG.

The OpenSSH developers wanted to register under the .ORG top level domain, traditionally meant for non-profit organisations such as OpenSSH, but the name had already been taken. They settled for the .COM in the interim.

The .ORG name is currently held by Mr. Alex de Joode <adejoode@zedz.net>, a proponent of open source cryptography who runs his own free crypto portal hosted by xs4all.nl, a well-known and respected Dutch ISP. Mr. de Joode has repeatedly refused requests to sell or turn the .ORG name over to the OpenSSH developers. This leaves us no choice but to issue this advisory.

The OpenSSH.ORG Web site currently is a blank page with a link to the official site. Please do not visit the .ORG site, nor send e-mail to anybody at the .ORG address. This is more than just a request to boycott: there could be privacy issues, possibly data mining or building a mailing list of security conscious users. We simply don't know Mr. de Joode's motives, and we recommend caution.

Any help or suggestions in breaking the deadlock are appreciated.

Regards

For the OpenSSH developers, Louis Bertrand <louis@openbsd.org>

howardjp writes "The December Edition of Daemon News is out! There are some cool articles on OpenSSH as well as Applixware. "