Slashdot Mirror
Data Munging with Perl
The Scoop Larry Wall, so goes the story, needed to glue together two systems on opposite sides of the country. Calling on the virtues of Laziness (why throw together something for just one job) and Hubris (why not write a new language?), he created Perl. Though it's found new niches in the post-web world, Perl earns its bread and butter munging data.
Dave Cross has put together a friendly and handy compendium of techniques, tricks, and best practices. Suitable for raw novices to experienced intermediates, Data Munging with Perl is a gentle but firm romp from flat text, past structured and binary files, to the realm of custom parsers. Clean examples and lots of modules accompany the explanations.
What's to Like? The book plots a natural course through topics ordered by complexity. It opens with a theoretical overview of data processing. This introduces terminology and outlines the general types of data one might encounger. Additionally, the author writes with the authority of experience when exploring the basic approaches and best practices. While other books aimed at novice users shy away from programs-as-filters and data structures, Cross prefers to instill good habits from the start.Beyond munging data, the book provides a decent introduction to idiomatic and effective Perl programming. While the brief tutorial won't magically produce new JAPHs, the thoughtful and continual devotion to good technique and skill will inspire smarter programmers. More important than knowing many useful tricks is knowing when and how to use a handful of tools -- and where to go for more.
The overall level of quality is excellent. The binary data chapter stands out as the clearest explanation available, and the information on munging dates and times will save readers plenty of grief. Additionally, the entire parsing section introduces a handful of powerful but sorely-underused tools to handle HTML, XML, and even creating custom parsers. Rounding out the curriculum is an appendix that explores the larger modules, mentioned earlier, in more detail (XML::Parser, DBI, Date::Manip).
What's to Consider? Only two things might turn readers from this book. The first is its deceptive length. While the text is short, the examples are clear and the text packs a lot of wallop in what's there. Careful readers who follow the links to other resources will have little trouble supplementing their education. (On the other hand, another ten pages describing Parse::RecDescent would have been a nice addition. It's hard to fault the author for deferring to the module's voluminous documentation.)Second, longtime Perl programmers may find little new material, particularly if they are familiar with the wealth of modules on the CPAN. The intended audience is clearly new and underexperienced programmers. While there's plenty of good advice presented well, the book falls more toward the tutorial side of the aisle than the reference section. This does not detract from the book, but it does narrow the base of potential readers slightly.
The SummaryManning Publications continues its fine line of Perl books with the consistent and powerful Data Munging with Perl. Coders looking to transform data somehow and hackers who want to take advantage of Perl's unique features will improve their knowledge and understanding. If you find yourself working with files or records in Perl, this book will save you time and trouble. Table of Contents- Introduction
- Data, data munging, and Perl
- General practices to use when munging data
- Generally useful Perl idioms
- Pattern matching
- Data Munging
- Unstructured data
- Record-oriented data
- Fixed-width & binary data
- Simple Data Parsing
- More complex data formats
- HTML
- XML
- Building your own parsers
- Conclusion
- Looking back -- and ahead
- Modules reference
- Essential Perl
You can purchase this book at ThinkGeek.
Hey! I happen to like those fruity portraits! It's what made "Object Oriented Perl" by Damian Conway stand out on the bookshelf. That book is invaluable.
Although I think "Data Munging with Perl" is probably less usefull to me, and I probably wont get it, the cover is cool.
By your use of the word "fruity" I take it to mean you are afraid of looking queer. Not to worry, I'm sure you already do.
The current version of HTML::Parser is written in C, with an XS interface. Maybe you aught to take a visit to CPAN?
Matt. Want XML + Apache + Stylesheets? Get AxKit.
Author: Joseph Hall (with Randal Schwartz)
ISBN: 0201419750
Publisher: Addison Wesley (1998)
Fun collection of Perl idioms and some good stuff on h2xs.
I'm pretty sure this is a troll, but munging has nothing to do with data mining. This is what the Jargon File has to say on the word 'munge':
/muhnj/ vt. 1. [derogatory] To imperfectly transform
munge
information. 2. A comprehensive rewrite of a routine, data structure or
the whole program. 3. To modify data in some way the speaker doesn't
need to go into right now or cannot describe succinctly (compare
{mumble}). 4. To add {spamblock} to an email address.
In this case Dave means 'doing stuff with data' akin to the Jargon File's third definition of the word.
--
Niklas Nordebo | niklas at nordebo.com
Books don't require batteries that might run down, or suffer from any of a dozen other complaints against the "true portability" of electronic systems for getting documentation.
If you're happy with phosphors, more power to you, but if I want a reference, I want one that is as portable as I am; one without leashes to the power grid (even if they're only intermittent), and one with some editorial intelligence up front to filter down to the topics I care about, rather than a kitchen soup like the web where I have to sift through 10000 google hits to find the page that really answers my question.
7 November 2006: The day Americans realized corruption and incompetence weren't addressing 11 September 2001
Spreadsheets and database tables are structured data. Word processing files are unstructured.
Best Slashdot Co
Embarrassing? Nope.. C is used for speed.
.pm to be put in one of Perl's library folders. Just put the files in the right directories and run.
.pm for each of your platforms and distribute it if you like, or you can just not use the C modules. Of course, you pay a price in speed for using only perl, but you can't have it all.
The people who create and maintain perl are smart enough to realise that no tool is universally useful.
Mixes of C and perl simply require the appropriate compiled
You can precompile a binary
If ease of distribution is paramount, write the parser in C, embed a perl interpreter in it and code the perl portion appropriately.
I gots ta ding a ding dang my dang a long ling long
I figured the use of "tomb" was intentional. After all, it's where you put dead trees.
Yes. It's embarassing that Perl needs help from C to ... manipulate strings.
(I don't want to use that because the mix of C and Perl makes portability more difficult. All-Perl code you just put in the right directory and run. Mixes of C and Perl require compilers, package managers, makefiles, and installers. The target is shared-hosting services, where users may not have shell access. It's seriously annoying that Perl does this simple operation so slowly.)
Manning's Object Oriented Perl is a great book with a terribly cheap binding. My copy is already falling apart. Anyone know if they have improved the binding quality on this one?
~
~
The best part about a book -- a well written book, not a "How to Be and Unleashed Dummy in 21 Days" book -- is the time and care put into it by a host of professionals, whereas a Web resource tends to be cobbled together from a community of geniuses and idiots alike.
Look at Slashdot -- some of it is great, some of it would wither a pile of dog poo it's so bad. php.net is similar -- the function reference is good if you're looking for arguments to a rarely used function, but the user-contributed stuff is off-and-on useful.
That's partially why you pay $50 for a good tech book -- the team of people needed to put together a *good* book is quite expensive. You need a knowledgeable author, a clued-in editor, a savvy fact-checker... all these people cost money.
"Beware by whom you are called sane."
Potato chips are a by-yourself food.
take a look at the source of a perl program.
Along with the Camel, "Effective Perl Programming" (Addison/Wesley, don't remember author's name), and the "Perl Cookbook," this has been one of my favorite programming books. Mind you, I'm not a seasoned hacker, so YMMV. But for anyone who already understands the basics of Perl, this book is a great way to learn something practical.
Like Chromatic, though, I really wished that the section on Parse::RecDescent had been longer...
We have an engine which processes such data, but it's slow, because it's in Perl. Most of the time goes into modules recommended in this book, like HTML::Parser. The big problem is that simple tokenizing, like extracting HTML tags, is incredibly slow in Perl. The classic "get next character, get character class for character, switch on character class" operation is something Perl does very badly.
Yes, you can write low-level C functions and call them from Perl to deal with such problems, but that kills portability.
One thing that I haven't seen mentioned yet is that books are easier to read than monitors. Monitors just can't match a book's DPI, and the higher resolution of the printed page can actually improve reading speed and retention and reduce eye strain. That may or may not be a big issue for you, but it can be a big deal and a reasonable justification for the extra expense. Another advantage of a printed book is that the author has already gone to the trouble of cobbling together the data for you so that you don't have to spend your time scrounging the web for it; if you're a consultant getting paid $100 per hour it doesn't take much time scouring the web for information to add up to more than the cost of the book.
OT Note: the correct term is tome (from the Greek word meaning to cut, and the same root as in medical procedures ending in -otomy, as tomes were originally produced by cutting a long scroll into smaller sections) not tomb (which is where somebody is buried).
There's no point in questioning authority if you aren't going to listen to the answers.
Books like this give corporations the tools they need to destroy our privacy and strip us of our rights. How do you think Double Click puts the information about you it sells into useable form? With techniques it learns from this type of book. Same goes for the corporate websites you visit, your supermarket, etc.
Information wants to be free, but not the information in this book. Data mining and Data munging techniques should never have left the hallowed halls of academe. Once they enter the public domain, they are immediately exploited by greedy corporations. The author should have thought about that before writing a book like this.
If you buy or support books like this, you have lost any right to complain about your privacy being violated. If you are serious about privacy, boycott this book!
If you have a problem with my views, REPLY, don't moderate!
Get a second monitor to read documentation from. Not only would it pay for itself within 4 books, but it's more useful than a stack of spent books.
--
Log files are generally fairly structured data.
CSV files are structured data.
Free flowing ASCII text is unstructured data. :).
Shakespeare's sonnets, however well formed, are unstructured data
(unless you can come up with a parser that recognizes iambic pentameter...
Falling somewhere in the middle is binary data. It has a structured format but freeform contents. Consider the various sound, image, and video formats. Maybe Shakespeare's sonnets could fall into this category too... :)
There are situations where you could want to analyze each form. Parsing Apache log files is a slightly different task than analysing formal XML documents or sloppy HTML pages or messy ASCII email. This book helps give you a feel for which situation you may be dealing with, and thus what tools & techniques might be useful for that situation.
Though some will tell you otherwise, this book has nothing to do with "Buffy the Vampire Slayer." Sorry, grep.
DO NOT LEAVE IT IS NOT REAL
Perhaps I'm still stuck in the paper age (somewhere between bronze & silicon), but I find myself spending $50 a pop for progamming books I only skim through. If I need reference material, I hit PHP.net (for my PHP projects).
Am I missing something?