Slashdot Mirror
Regular Expression Pocket Reference
Michael J. Ross writes "When software developers need to manipulate text programmatically — such as finding all substrings within some text that match a particular pattern — the most concise and flexible solution is to use "regular expressions," which are strings of characters and symbols that can look anything but regular. Nonetheless, they can be invaluable for locating text that matches a pattern (the "expression"), and optionally replacing the matched text with new text. Regular expressions have proven so popular that they have been incorporated into most if not all major programming languages and editors, and even at least one Web server. But each one implements regular expressions in its own way — which is reason enough for programmers to appreciate the latest edition of Regular Expression Pocket Reference, by Tony Stubblebine." Read below for the rest of Michael's review.
Regular Expression Pocket Reference, Second Edition
author
Tony Stubblebine
pages
126
publisher
O'Reilly Media
rating
9/10
reviewer
Michael J. Ross
ISBN
0596514271
summary
A pithy guide to regular expressions in many languages.
The second edition of the book was published by O'Reilly Media on 18 July 2007, under the ISBNs 0596514271 and 978-0596514273. On the book's Web page, the publisher makes available the book's table of contents and index, as well as links for providing feedback and any errata. As of this writing, there are no unconfirmed errata (those submitted by readers but not yet checked by the author to see whether they are valid), and no confirmed ones, either. In fact, in my review of the first edition, published in 2004, it was noted that there were no unconfirmed errata, despite the book being out for some time prior to that review. The most likely explanation is that the author — in addition to any technical reviewers — did a thorough job of checking all of the regular expressions in the book, along with the sample code that make use of them. These efforts have paid off with the apparent absence of any errors in this new edition — something unseen in any other technical book with which I am familiar.
Before discussing this particular book, it may be of value to briefly discuss the essential concept of regular expressions, for the benefit of any readers who are not familiar with them. As noted earlier, a regular expression (frequently termed a "regex") is a string of characters intended for matching substrings in a block of text. A regex pattern can match literally, such as the pattern "book" matching both "book" and "bookshelf." A pattern can also use special characters and character combinations — often termed metasymbols and metasequences — such as \w to indicate a single word character (A-Z, a-z, 0-9, or '_'). Thus, the regex "b\w\wk" would match "book," but not "brook."
Here is a simple example to show the use of regexes in code, written in Perl: The statement "$text =~ m/book/;" would find the first instance of the string "book" inside the scalar variable $text, which presumably contains some text. To substitute all instances of the string with the word "publication," you could use the statement "$text =~ s/book/publication/g;" ('g' for globally search) or use "$text =~ s/bo{2}k/publication/g;". In this simplistic example, the second statement makes use of a quantifier, {2}, indicating two of the preceding letter.
These examples employ only one metacharacter (\w) and one quantifier ({2}). The total number of metacharacters, metasymbols, quantifiers, character classes, and assertions (to say nothing of capturing, clustering, and alternation) that are available, in most regex-enabled languages, is tremendous. However, the same cannot be said for the readability of all but the simplest regular expressions — especially lengthy ones not improved by whitespace and comments. As a consequence, when using regexes in their code, many programmers find themselves repeatedly consulting reference materials that do not focus on regular expressions. These resources comprise convoluted Perl books, incomplete tutorials on the Internet, and confusing discussions in technical newsgroups. For too many years, there was no published book providing the details of regexes for the various languages that utilize them, in addition to a clear explanation of how to use regexes wisely.
Fortunately, O'Reilly Media offers two titles in hopes of meeting that need: Mastering Regular Expressions, by Jeffrey Friedl, and Regular Expression Pocket Reference, by Tony Stubblebine. In several respects, the books are related — particularly in that Stubblebine bases his slender monograph upon Friedl's larger and more extensive title, justifiably characterized by Stubblebine as "the definitive work on the subject." In addition, Stubblebine's book follows the structure of Friedl's book, and contains page references to the same. Another major difference is that Regular Expression Pocket Reference is, just as the title indicates, for reference purposes only, and not intended as a tutorial.
At first glance, it is clear that Stubblebine's book packs a great deal of information into its modest 126 pages. That may partly be a result of the terseness of most, if not all, of the regular expression syntax; a metasymbol of more than two characters would be considered long-winded! Yet the high information density is likely also due to the manner in which Stubblebine has distilled the operators and rules, as well as the meaning and usage thereof, down to the bare bones. But this does not imply that the book is bereft of examples. Most of the sections contain at least one, and sometimes several, code fragments that illustrate the regex elements under discussion.
The book begins with a brief introduction to regexes and pattern matching, followed by an even briefer cookbook section, with Perl-style regexes for a dozen commonly-needed tasks, e.g., validating dates. The bulk of the book's material is divided into 11 sections, each one devoted to the usage of regexes within a particular language, application, or library: Perl 5.8, Java,.NET and C#, PHP, Python, Ruby, JavaScript, PCRE, the Apache Web server, the vi programmer's editor, and shell tools.
Each of these sections begins with a brief overview of how regexes fit into the overall language covered in that section. Following this is a subsection listing all of the supported metacharacters, with a summary of their meanings, in tabular format. In most cases, this is followed by a subsection showing the usage of those metacharacters — either in the form of operators or pattern-matching functions, depending upon how regular expressions are used within that language. Next is a subsection providing several examples, which is often the first material that most programmers turn to when trying to quickly figure out how to use one aspect of a language. Each section concludes with a short listing of other resources related to regexes for that particular language.
There are no glaring problems in this book, and I can only assume that all of the regular expressions themselves have been tested by the author and by previous readers. However, there is a minor weakness that should be pointed out, and could be corrected in the next edition. In most of the sections' examples, Stubblebine wisely formats the code so that every left brace ("{") is on the same line as the beginning of the statement that uses that brace, and each closing brace ("}") is lined up directly underneath the first character of the statement. This format saves space and makes it easier to match up the statement with its corresponding close brace. However, in the.NET / C# and PCRE library sections, the open braces consume their own lines, and also are indented inconsistently, as are the close braces, which makes the code less readable, as well as less consistent among the sections.
Some readers may fault the book's sparse index. Admittedly, an inadequate index in any sizable programming book can make it difficult if not impossible to find what one is looking for. As a result, one ends up flipping through the book's pages hoping to luckily spot the desired topic. This is the rather unpleasant method to which a reader must resort when a technical book has no index, or one that is inadequate — which is far too often the case. Stubblebine's index offers only several dozen entries for all the letters of the alphabet, and only two symbols. Some readers might demand that all of the metacharacters and metasequences be listed in the index, so they can be found even faster than otherwise. But given the large number of metacharacters and metasequences, as well as method names, module functions, and everything else relevant, creating an exhaustive index would almost double the size of the book, and be largely redundant with the language-specific sections. Within each language, there is typically a limited enough number of pages that scanning through them to find a particular topic, would not be onerous. On the other hand, some of the index's inclusions and omissions are odd. For instance, two symbols are listed, and yet no others; why bother with those two? Also, a few key concepts are missing, such as grouping and capturing.
Yet aside from these minor blemishes, Regular Expression Pocket Reference is a concise, well-written, and information-rich resource that should be kept on hand by any busy software developer.
Michael J. Ross is a Web developer, writer, and freelance editor.
You can purchase Regular Expression Pocket Reference, Second Edition from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Before discussing this particular book, it may be of value to briefly discuss the essential concept of regular expressions, for the benefit of any readers who are not familiar with them. As noted earlier, a regular expression (frequently termed a "regex") is a string of characters intended for matching substrings in a block of text. A regex pattern can match literally, such as the pattern "book" matching both "book" and "bookshelf." A pattern can also use special characters and character combinations — often termed metasymbols and metasequences — such as \w to indicate a single word character (A-Z, a-z, 0-9, or '_'). Thus, the regex "b\w\wk" would match "book," but not "brook."
Here is a simple example to show the use of regexes in code, written in Perl: The statement "$text =~ m/book/;" would find the first instance of the string "book" inside the scalar variable $text, which presumably contains some text. To substitute all instances of the string with the word "publication," you could use the statement "$text =~ s/book/publication/g;" ('g' for globally search) or use "$text =~ s/bo{2}k/publication/g;". In this simplistic example, the second statement makes use of a quantifier, {2}, indicating two of the preceding letter.
These examples employ only one metacharacter (\w) and one quantifier ({2}). The total number of metacharacters, metasymbols, quantifiers, character classes, and assertions (to say nothing of capturing, clustering, and alternation) that are available, in most regex-enabled languages, is tremendous. However, the same cannot be said for the readability of all but the simplest regular expressions — especially lengthy ones not improved by whitespace and comments. As a consequence, when using regexes in their code, many programmers find themselves repeatedly consulting reference materials that do not focus on regular expressions. These resources comprise convoluted Perl books, incomplete tutorials on the Internet, and confusing discussions in technical newsgroups. For too many years, there was no published book providing the details of regexes for the various languages that utilize them, in addition to a clear explanation of how to use regexes wisely.
Fortunately, O'Reilly Media offers two titles in hopes of meeting that need: Mastering Regular Expressions, by Jeffrey Friedl, and Regular Expression Pocket Reference, by Tony Stubblebine. In several respects, the books are related — particularly in that Stubblebine bases his slender monograph upon Friedl's larger and more extensive title, justifiably characterized by Stubblebine as "the definitive work on the subject." In addition, Stubblebine's book follows the structure of Friedl's book, and contains page references to the same. Another major difference is that Regular Expression Pocket Reference is, just as the title indicates, for reference purposes only, and not intended as a tutorial.
At first glance, it is clear that Stubblebine's book packs a great deal of information into its modest 126 pages. That may partly be a result of the terseness of most, if not all, of the regular expression syntax; a metasymbol of more than two characters would be considered long-winded! Yet the high information density is likely also due to the manner in which Stubblebine has distilled the operators and rules, as well as the meaning and usage thereof, down to the bare bones. But this does not imply that the book is bereft of examples. Most of the sections contain at least one, and sometimes several, code fragments that illustrate the regex elements under discussion.
The book begins with a brief introduction to regexes and pattern matching, followed by an even briefer cookbook section, with Perl-style regexes for a dozen commonly-needed tasks, e.g., validating dates. The bulk of the book's material is divided into 11 sections, each one devoted to the usage of regexes within a particular language, application, or library: Perl 5.8, Java,.NET and C#, PHP, Python, Ruby, JavaScript, PCRE, the Apache Web server, the vi programmer's editor, and shell tools.
Each of these sections begins with a brief overview of how regexes fit into the overall language covered in that section. Following this is a subsection listing all of the supported metacharacters, with a summary of their meanings, in tabular format. In most cases, this is followed by a subsection showing the usage of those metacharacters — either in the form of operators or pattern-matching functions, depending upon how regular expressions are used within that language. Next is a subsection providing several examples, which is often the first material that most programmers turn to when trying to quickly figure out how to use one aspect of a language. Each section concludes with a short listing of other resources related to regexes for that particular language.
There are no glaring problems in this book, and I can only assume that all of the regular expressions themselves have been tested by the author and by previous readers. However, there is a minor weakness that should be pointed out, and could be corrected in the next edition. In most of the sections' examples, Stubblebine wisely formats the code so that every left brace ("{") is on the same line as the beginning of the statement that uses that brace, and each closing brace ("}") is lined up directly underneath the first character of the statement. This format saves space and makes it easier to match up the statement with its corresponding close brace. However, in the.NET / C# and PCRE library sections, the open braces consume their own lines, and also are indented inconsistently, as are the close braces, which makes the code less readable, as well as less consistent among the sections.
Some readers may fault the book's sparse index. Admittedly, an inadequate index in any sizable programming book can make it difficult if not impossible to find what one is looking for. As a result, one ends up flipping through the book's pages hoping to luckily spot the desired topic. This is the rather unpleasant method to which a reader must resort when a technical book has no index, or one that is inadequate — which is far too often the case. Stubblebine's index offers only several dozen entries for all the letters of the alphabet, and only two symbols. Some readers might demand that all of the metacharacters and metasequences be listed in the index, so they can be found even faster than otherwise. But given the large number of metacharacters and metasequences, as well as method names, module functions, and everything else relevant, creating an exhaustive index would almost double the size of the book, and be largely redundant with the language-specific sections. Within each language, there is typically a limited enough number of pages that scanning through them to find a particular topic, would not be onerous. On the other hand, some of the index's inclusions and omissions are odd. For instance, two symbols are listed, and yet no others; why bother with those two? Also, a few key concepts are missing, such as grouping and capturing.
Yet aside from these minor blemishes, Regular Expression Pocket Reference is a concise, well-written, and information-rich resource that should be kept on hand by any busy software developer.
Michael J. Ross is a Web developer, writer, and freelance editor.
You can purchase Regular Expression Pocket Reference, Second Edition from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
this looks good, want to send me a copy? ;)
"Don't hate the media, become the media." -Jello Biafra
I find that O'Reilly's books on regular expressions (beside the pocket reference there's Mastering Regular Expressions ) seem to assume a great deal of prior knowledge. Is there any introduction to regular expressions for total beginners, perhaps teaching through examples and including exercises?
You need help from the book in order to find the best way to search for its ebook on the internet
Now that would be an interesting pair of authors ;)
Tsunami -- You can't bring a good wave down!
a google search for "regex [your fav language goes here]"?
Any guest worker system is indistinguishable from indentured servitude.
...I have a pocket reference to regular expressions.
That's not Picasso, that's Kandinsky!
www.regexlib.com
Here's the regular expression that I found most useful in childhood:
"Hello, I'm a smart geeky person, please to not beat me up and take my lunch money. I can help you with your math homework"
Sheldon
A minor correction:
However, there is a minor weakness that should be pointed out, and could be corrected in the next edition. Specifically, the book includes a section on
The universe is a figment of its own imagination.
Because you just can't discuss regular expressions without bringing up this quote:
Some people, when confronted with a problem, think "I know, I'll use regular expressions."
Now they have two problems.
-- Jamie Zawinski, 1997, in alt.religion.emacs
There's already a built in regular expression tutorial:
man perlretutIf someone is passing you on the right, you are an asshole for driving in the wrong lane.
I use grep regularly enough to know generally how to build an expression, but not often enough to know each (I use grep in 3-4 different editors) application's quirks/implementation details off the top of my head, so I end up having to look up something regularly. I always use the application's Help file rather than the grep manual I've got laying around somewhere.
Opening the Help file for the app and using its search function is a lot quicker than having to leaf through a book (worse when the book has a bad index). The only time this is annoying is when I've got a lack of screen real estate, but that's usually when I'm on the road and won't have access to any books anyway.
I'll start with an Obligatory quote.
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems. --Jamie Zawinski, in comp.lang.emacs
I'll close with a somewhat depressing fact: Regular expression and string processing can be done quickly and efficiently (and was done that way back decades ago, with grep and awk), but is actually done in a horribly inefficient way in all modern/popular programming language regexp engines.
Well to apply a common saying to something in need. You can't grep dead wood.
Excuse me while I gather the virgin sacrifice and assemble the pentagram required to solve your problem
Once I found the functions startswith and endswith, my need to use for regular expressions dropped away, fast. Occasionally, I'd have to pony up for a more complex pattern match, still a pain though even though I'd "cracked" regex. I wonder if the rest of regex could be done away with in a similar fashion?
Patriotism is a virtue of the vicious
Another post links to a site for a regex visualizer utility for Windows and Linux.
Here's one for the Mac:
http://homepage.mac.com/roger_jolly/software/index.html#regexhibit
I'd rather stick knitting needles in my eyes than debug a regular expression.
The only cure for that is getting a good reference and having a go at some tutorials until you get good enough to slay the beast. Then you'll be everyone's buddy at the office, because a lot of people feel the same way.
Or you could just stick knitting needles in your eyes and slash your face with a razor and then everyone will leave you alone.
That's our life, the big wheel of shit. - The Fat Man, Blue Tango Salvage
How do you pronounce "regex"? I see four possibilities:
1) "regh-ex" (hard 'g', like 'ghost')
2) "rej-ex" (soft 'g', like 'gerbil')
3) "re-gex" (hard 'g')
4) "re-jex" (soft 'g')
I use the first one, since those are the two initial syllables of 'regular' and 'expression', but I can see arguments for the others.
Nothing for 6-digit uids?
Regular expressions are easier than you think and once you get comfortable with them you will be wishing you hadn't done so sooner. In my opinion the difficult part of learning them is just getting used the strange mess of dots, pluses, brackets, backslashes, etc. and what they mean in different contexts. Unfortunately it is hard to walk away from an article or howto on regexes and actually remember the meaning of all the symbols. Regular expressions are deliberately terse and that makes them hard to read and understand by humans.
Therefore I think the best way to learn regular expressions is by example. I highly recommend this small interactive program which will walk you through building regular expressions for a few different languagues. When you think you need a regex for a program, just fire it up and answer the questions.
http://txt2regex.sourceforge.net/
After a while you won't need txt2regex for simple stuff because you will have hopefully just absorbed the syntax. Once you have mastered the basic regexes which txt2regex can generate you will be able to dive into more advanced topics like capturing groups.
Parsing Expression Grammars are the future.
They are generally faster and more powerful. It's easy to do regular expression-like stuff as well as really complicated things like a programming language parser.
Lua's LPeg extension is really excellent.
What about emacs? Grep? Sed? This book sounds like a good idea, but it's not so useful without a wider selection of applications. Frankly, though, I just want a short guide as to which things need to be escaped to get which meanings, and what character classes are available.
If you read the link I posted, you will see that they are indeed evil and slow - and not for any good reason. The implementation of good regular expression engines is not difficult and known in CS theory for many decades.
"Premature optimization" is a nice slogan - but the regexp performance problems are real, and I have encountered them before (I was extremely surprised to see that the regexp matching is scaling far worse than O(N) as it was clear to me that matching that regexp should be at worst O(N)).
The reason it is depressing is because they got it right in the 60's, and are getting it wrong now. Stalling progress is sad. Deteriorating is depressing.
As for elisp regexps being faster than other elisp methods - its not very indicative, as the regexp engine is implemented in C. If you compare, however, the pathological regexps (see my original link) in elisp, compared to a naive elisp char-by-char iteration of strings, you'll see that the elisp code performs better.
About your link, it doesn't seem that he is prejudiced against Perl, it seems that he hates Perl and that implies no prejudice. Many of us dislike or even hate Perl because we find it less suitable for all tasks than other tools that we use, and because we find that it an extremely ugly hack that strongly encourages write-once read-never code.
lining lining lining the TBghjo 7u00 6fFGWDT^tvvttvtvtv^F^F^F^ : Ooo ;s;oflew kk lKL i ufedk hijnj n LJ L JJd hsytCRTCRCRqwe WHA iIKJDJ yrfge WHAT? iuYHIOO&Ylh j;|}|}}||}|}}|]\]\]\] fyhukYHLRFHJE> HFe fkllk>KKKL KL K f fiuewuiwuriw boog lining
A regular expression can be thought of as a program which generates a set of strings - or recognizes a set of strings, which is the same thing. Regular expressions correspond to finite state automatons, so just as a FSA cannot recognize the set of all palindromes, neither can a regular expression. Also languages like perl have extended the capabilities of their regular expression string matchers to include things like backreferences, which cannot be done in a true regular expression, so we tend to use the word 'regexp' nowadays.
Or perhaps I'm just playing the grumpy computer scientist here.
-- Ed Avis ed@membled.com
There are a few online regexp test sites: http://www.regextester.com/ http://www.regexp.com.br/ http://www.regexpal.com/
http://www.visibone.com/javascript/jcht4_869.jpg
You don't like regexes? So what am I supposed to do? I mean, I *have* to write tons of crappy code like this that's a lot more scatter-brained than a simple regex... It solves ONE problem (and no others), I have to read through a lot of code to figure out why I'm doing this, and it's not likely to be any faster.
/^$pattern$/);
/* Pattern is composed of the characters C and V depending on which we want to match. */
Look at all the crap (in pseudo-Java, not using regexes at all) I'd have to write to allow people to specify something simple, like taking a pattern in the form of CCCVV and matching a word against it. And compare that with Perl.
# Perl
sub matchCV {
# Returns 1 if true, 0 if false.
my ($pattern, $s) = @_;
# Compare how this case is handled with the pseudo-Java. It's easier to do it better.
die "Invalid pattern ${pattern}!\n" if ($pattern !~ m/^[CV]+$/i);
# Turn pattern into a real regex.
$pattern =~ s/V/\[AEIOU\]/gi; # V -> all vowels
$pattern =~ s/C/\[^BCDFGHJKLMNPQRSTVWXYZ\]/gi; # C -> all consonants
return 1 if ($s =~
return 0;
}
/* Pseudo-Java */
bool isVowel(char c) {
String vowels = "AEIOU";
if (vowels.indexOf(c.toUppercase()) > 0) {
return true;
} else {
return false;
}
}
bool matchCV(String pattern, String s) {
bool retVal = true;
if (pattern.length() != s.length) { return false; }
for (int i = 0; (i < s.length()) && retVal; ++i) {
switch (s.charAt(i)) {
case 'C':
case 'c':
if (isVowel(s.charAt(i))) { retVal = false; }
break;
case 'V':
case 'v':
if (!isVowel(s.charAt(i))) { retVal = false; }
break;
default:
Bug("Invalid pattern character " + s.charAt(i).toString() + " at position " + i.toString + ".");
}
}
return retVal;
}
$text = "The bookkeeper was very careful to keep proper books as he did not wish to be booked for fraud."
$text =~ s/book/publication/g;
Yeah, that will work. Not.
-- Will program for bandwidth
http://www.stklos.org/Doc/html/stklos-ref-5.html#Regular-Expressions STKLOS.org regex reference. I don't even know what the site is originally about, but the regex ref is the best!
Quack damn you!
KRegExpEditor gives you a nice GUI.
Mac does not do Tkl/Tc? If it does, the GP post should work on OSX as well.
Neither of these programs look as nice as KDE's Editor, and that too should work on Mac and Windows soon enough.
Buy this program: http://www.regexbuddy.com/
It is the best $40 I ever spent when doing a project involving tons of Regular Expressions. It has detailed tutorials on how Regular Expressions work, a reference guide, debugging mode, real-time feedback on what your expression is doing, error checking, and a built-in forum where you can post your problems and people including the developer himself will chime in and help you figure it out!
I'm not associated with JGSoft in anyway, but RegEx Buddy really is an awesome product. Also, you can change which language you are targeting and it understands the limitations of each one (and verifies your code will work) and he explains the differences between different language's implementations on that website (for free).
"To strive, to seek, to find, and not to yield." - Tennyson
The first edition copy I have is pretty dog-eared from constantly being stashed in my laptop bag. I don't use regexes every day, but I understand them, and just need a handy reference. Definitely great for a "how do you specify X" kind of problem.
Why can't I mod "-1 Idiot"?
Oh, if so, what are those reasons really?