Spidering Hacks

DrCarbonite writes "Spidering Hacks is a well-written guide to scripting and automating your data-seeking forays onto the Internet. It offers an attractive combination of the solving the problems you have and exposing you to solutions that you weren't aware you needed." Read on for Martin's review of the book. Spidering Hacks author Kevin Hemenway and Tara Calishain pages 402 publisher O'Reilly rating 8 reviewer Jeff Martin ISBN 0596005776 summary A wide-ranging collection of hacks detailing how to be more productive in Internet research and data retrieval

Introduction Spidering Hacks (SH), by Kevin Hemenway and Tara Calishain, is a practical guide to performing Internet research that goes beyond a simple Google search. SH demonstrates how scripting and other techniques can increase the power and efficiency of your Internet searching, allowing the computer to obtain data, leaving the user free to spend more time on analysis.

SH's language of choice is Perl, and while there are a few guest appearances by Java and Python, some basic Perl fluency will serve the reader well in reading the Hack's source code. However, regardless of your language preference, SH is still a useful resource. The authors discuss ethics and guidelines for writing polite and properly behaved spiders as well as the concepts and reasoning behind the scripts they present. For this reason, non-Perl coders can still stand to learn a lot of useful tips that will help them with their own projects.

Overview

Chapter 1, Walking Softly, covers the basics of spiders and scrapers, and includes tips on proper etiquette for Web robots as well as some resources for identifying and registering the many Web robots/spiders that exist on the Internet. Hemenway and Calishain should be credited for taking the time to be civically responsible and giving their readers appreciation for the power they are utilizing.

Chapter 2, "Assembling a Toolbox," covers how to obtain the Perl modules used by the book, respecting robots.txt, and various topics (Perls LWP and WWW::Mechanize modules for example) that will provide the reader with a solid foundation throughout the rest of the book. SH does a great job introducing some topics that not all members in its target audience may be familiar with (i.e., regular expressions, the use of pipes, XPath).

Chapter 3, "Collecting Media Files," deals with obtaining files from POP3 email attachments, the Library of Congress, and Web cams, among other sources. While individual sites described here may not appeal to everyone, the idea is to provide a specific example demonstrating each of certain general concepts, which can be applied to sites of the reader's choosing.

Chapter 4, "Gleaning Data from Databases," approaches various online databases. There are some interesting hacks here, such as those that leverage Google and Yahoo together. This chapter is the longest, and provides the greatest variety of hacks. It also discusses locating, manipulating, and generating RSS feeds, as well as other miscellaneous tasks such as downloading horoscopes to an iPod.

Hack #48, Super Word Lookup, is a good example of why SH is so intriguing. While utilizing a dictionary or thesaurus via a browser is simple, having the ability to do so with a command-line program allows the user an automated approach, reducing distractions.

Chapter 5, "Maintaining Your Collections," discusses ways to automate retrieval using cron and practical alternatives for Windows users.

Chapter 6, "Giving Back to the World," ends SH by covering practical ways the reader can give back to the Internet and avoid the ignominious leech designation. This chapter provides information on creating public RSS feeds, making an organization's resources available for easy retrieval by spiders, and using instant messaging with a spider.

Conclusion

There are extensive links provided throughout the book, and this indirectly contributes to SH's worth. The usual O'Reilly site for source code is available and Hemenway also provides some additional code on his site. A detailed listing of the hacks covered in SH is also available online from SH's table of contents.

The Hacks series is a relatively new genre for O'Reilly, but it is rapidly maturing and this growth is reflected in Spidering Hacks. Hemenway and Calishain have done good work in assembling a wide variety of tips that cover a broad spectrum of interests and applications. This is a solid effort, and I can easily recommend it to those looking to perform more effective Internet research as well as those looking for new scripting projects to undertake.

You can purchase Spidering Hacks from bn.com. Slashdot welcomes readers' book reviews -- to submit a review for consideration, read the book review guidelines, then visit the submission page.

Comments

Why spider when you can deepweb?

[pheared]

deepweb

Table of content is packed with great stuff!

[millette]

Have a look at the Table of Content - it has 100 items, some of it you wouldn't obviously qualify as spidering, but more like data mining, but whatever, it's all good stuff! There's also some php, besides the java and python code. Perl is the most predominant language.

I wonder if Tracking Packages with FedEx is using the new google feature. That would be too simple :)

Does anyone know the name of a small utility to query search engines on the command line? It think it was a 2-letter program, but I couldn't find it anymore :(

Re:Table of content is packed with great stuff!

[rog]

You're probably looking for surfraw

Re:Table of content is packed with great stuff!

[millette]

I was way off with my 2-letter name :)

Looking at the page, and I'm pretty sure it's the little program I had lost. Thanks for finding it again!

Pssst! Mod parent up!

Ok I admit it

[JSkills]

I have written a porn gathering spider to seek out large movie files. It beats using a browser to find stuff.

Oh the shame ...

Re:Ok I admit it

[trentblase]

Maybe it's time for pr0n.google.com

Re:Ok I admit it

[interiot]

Is it open source? I wish there was more adult open-sores software. UBH running from cron is what I use currently to automate porn consumtion, but I'm sure there are tons of other opportunities....

Re:Ok I admit it

[BigGerman]

Booble?

Re:Ok I admit it

[QuasiCoLtd]

I wish there was more adult open-sores software.


What kind of pr0n have YOU been looking at....

Re:Ok I admit it

[Ignis Flatus]

Try altavista.com.
Click on the video tab.
Turn Family Filter off.
Fap away.

Not Likely

[trentblase]

The authors discuss ethics and guidelines for writing polite and properly behaved spiders

Maybe the spammers will read the ethics section and have a change of heart!

XML interop?

[prostoalex]

From the review it looks like an excellent books to read and maybe have around. I will check it out on Safari, since it looks like they made it available to subscribers.

However, looking at these hacks:

68. Checking Blogs for New Comments
69. Aggregating RSS and Posting Changes
70. Using the Link Cosmos of Technorati
71. Finding Related RSS Feeds

Do they offer any hacks on working with XML, perhaps XML::RSS or other parsing engines from CPAN? Or is most of the XML handled through regexp?

Re:XML interop?

[millette]

Here's hoping you aren't serious. I mean, rx are one thing, but to parse xml, and to some degree html, there are way better tools specifically for the job. I usually filter html thru tidy a few times until I can more easily parse it with xml tools - but that's just me.

Re:XML interop?

[justMichael]

Hack 24 Painless RSS with Template::Extract

It's actually a good read. They try to stay away from regex parsing as it tends to be fragile. They do cover it in one of the hacks though.

Most of the hacks have to do with using various methods to walk the doc tree to look for what you want like a certain cell in a table (think header with names) then jumping up one to get that row then grabbing the next row to get your data cells.

Re:XML interop?

[BenitoM]

Try Perl and XML by O'Reilly: http://www.oreilly.com/catalog/perlxml/index.html

There are basically two styles of XML parser, event-based (SAX) and document-based (DOM). I find DOM-types easier to use.

Use of "hacker"

[davidstrauss]

When are people going to realize that hackers just care about computers and the crackers are the bad guys? Oh wait...

Be sure to get LWP & Perl

[toupsie]

This is one of my favorite O'Reilly books. It is amazing what you can do with a few lines of Perl code and LWP.

Re:Techniques used by spammers?

[tds67]

Other than using google, why would anyone use these techniques other than to harvest email like a spammer?

There's a lot more information on the Web than just e-mail addresses. Besides, why be reliant on search engines when you can do it yourself?

Yeecchhh!

[AtariAmarok]

"....a porn gathering spider to....

That thing's going to build one nasty sticky web!

perldoc LWP

[Ars-Fartsica]

Save yourself $30.

MOD UP!

[Anonymous Coward]

it IS time

That's not what this is used for...

[justMichael]

Think of datamining, prime example:

You want to track your rank on www.alexa.com and the ranking of some of your key competitors. You build a spider that goes out each night and scrapes the info you want and stores it localy. now you have history on your and you competitiors ranking over time.

This way you can see that when your traffic is down so is your competitor or maybe when yours is down theirs is up...

This also happens to be one of the examples in the book.

Re:That's not what this is used for...

[beacher]

Here's what it's used for - violating H.R. 3261, the Database and Collections of Information Misappropriation Act Guess I'm going to buy it very soon ;) -B

Re:New book: Hacking your way into a Spider Hole

[tsmccaff]

The term "spider hole" has been part military parlance since WWII, but gained common usage outside the military during Vietnam. It may refer to the trapdoor spider, who doesn't use a web, but rather pops out of a hole in the ground, surprising its prey.

Tracking yahoo popularity.

[Flat Feet Pete]

My server's going to die under the load, but I did this using Perl+Curl.

This page is used to source the data.

Is LWP the correct/new way to do this kind of stuff? I started with curl and hacked regex's to get the data.

Re:Tracking yahoo popularity.

[interiot]

LWP runs as part of perl, so it gives you a little easier control over the variety of options (eg. user agent and such). And it's easier to get working cross-platform (it's a bitch that you have to do extra work to get around the shell parsing of arguments to subprocesses on Win32). Also, you can do fancy asynchronous stuff with LWP, so you can have interactive programs, or stuff going in parallel, etc...

In general, most people use LWP, and if you write very many programs that use the web, you're going to want to go to LWP eventually, so you might as well start learning now (and there are easier interfaces to facilitate that too).

Re:Tracking yahoo popularity.

[ls -lR]

LWP really just replaces the fetching part, it doesn't do anything to extract the data. It will definitely be easier than curl on the command line, no parameter passing to worry about.

To get the data from the page you can either use a bunch of regexps (as you've done, apparently) or a parser like HTML::TokeParser::Simple. The advantage of a parser is that it makes it more robust and immune to site changes. You also get higher quality data, for example if something subtle changes in the site's html source you sometimes get something like "this is the data <A href="whatever"..." In other words, you don't have to worry about quoting or tag boundaries or anything like that. Naturally, if your script allows user interaction this will tend to be more secure as there is less chance of a XSS and/or SQL injection vuln.

But, using a parser takes a little bit of investment up front in terms of time. With the '::Simple' variant it's really pretty easy, but it still requires that you be a little familiar with the tree structure of the page so that you can pull out the stuff you want.

All in all, if it works don't switch, but in the future you'll have a more robust and maintainable setup if you use LWP and a parser instead of commandline curl and regexps.

Spidering and exceeding ISP bandwidth limits

[G4from128k]

I suspect that more than a few people are going to hit their ISP's bandwidth limits if they start playing with spiders. A spider running on a simple 768 kbps DSL line can probably schlep down more than 4 GB per day or 129 GB/month (assuming the CPU can keep up analyzing with the flow).

Re:Spidering and exceeding ISP bandwidth limits

[interiot]

If it's a full spider where you're considering competing with google or reimplementing google with extra features, then yes, you'd obviously need an industrial-strength account.

More likely though, you leave the big jobs to the big boys, and you want to do very specific things, maybe even building on top of google... eg. find porn movies, copying edmunds' database so you can sort cars by their power/weight ratio (or list all RWD cars, or find the lightest RWD car, or...), or make your own third-party feed of slashdot from their homepage since they watch you like a hawk when you download their .rss too often, but not when you download their homepage too often.

Little custom jobs like that can take a minimal amount of code (especially if you're a regex wizard), take minimal bandwidth, and take enough skill that target sites aren't likely to track you down because there's only three of you doing it.

Alternative ways of searching and spidering

[Anonymous Coward]

can be found from www.searchlores.org

Agents, anyone?

[Wingchild]

A few years ago, the big idea was that by some as-yet undetermined point in the future (say, 2005) all human beings would be freed from having to collect their own data by way of intelligent, semi-autonomous Agents that could be given some loose english-query type tasks and go on their merry way, fetching and organizing and categorizing data by relevance. It's not too far different from the proposed use of scripting talked about above.

The problem comes more in the last assertation of the story; that pulling in all of this data will free up more time for people to spend on the work of analysis. I want to say this isn't accurate, but it probably boils down to what you call "analysis" work.

The problem with spiders, agents, and their like -- yes, even those that are going out and fetching porn -- is that they are able to provide content without context, much as a modern search engine does. I can take Google and get super specific with a query (say, `pirates carribean history -movie -"johnny depp"`). That will probably fetch me back some data that has my keywords in it, much as any script or agent could do.

Unfortunately, while the engine could rank based on keyword visibility and recurrance, as well as applying some algorithms to try and guess whether the data might be good or not (encylcopedias look this way, weblogs about Johnny Depp look that way), the engine itself still has on way to physically read the information and decide if it's at all useful. A high-school website's page with a tidbit of information and some cute animated .gifs could theoretically draw more of a response from the engine than an official historian's personal recollections of his research while he was working on his master's thesis about the Jolly Roger. Any script (or engine) is only what you make of it.

The most tedious part of data analysis these days is not providing content (as spiders, scripts, and search engines all do) ... it's in providing a frame of context for the choosing, and, ultimately, rejection of sources.

What comes after that sorting process - the assimilation of good data and the drawing of conclusions there-from - that's what I call data analysis. A shame that scripts, spiders, agents, and robots haven't found a way to do that for us. :)

Re:Agents, anyone?

[LetterJ]

I think that some of the things being done to filter *out* spam might also apply to filtering *in* good information from things like agents.

I know that my Popfile spam filter is getting pretty good (with 35,000 messages processed) at not only spam vs. ham type comparisons, but also work vs. personal and other categories.

Bayesian filters are just one type of learning algorithm, but they work fairly well for textual comparisons. I've personally been toying with seeing how well a toolbar/proxy combination would work for predicting the relative "value" of a site to me. Run all browsing through a Bayesian web proxy that analyses all sites visited. Then, with a browser toolbar, sites can be moderated into a series of categories.

That same database could be used by spiders to look for new content, and, if it fits into a "positive" category according to the analysis, add it to a personal content page of some sort that could be used as a browser's home page.

With sufficient data sources (and with a book like this, it shows that there ARE plenty of sources), it could really bring the content you want to read together.

Sample hacks

[Jadsky]

Don't know if anyone's pointed it out, but there are some sample links up on the web site. Some really great stuff, just from what I saw. Made me want to buy the book. (Guess that's the point.)

Perl script to query the library

[Saint Stephen]

I have 3 library cards, and get a lot of DVDs, CDs, and books from them. (Lotsa free time).

I got tired of having to go to all 3 websites to see what to take back each day, so I wrote a small bash/curl script so I could do it at the command line.

There are *lots* of things like this that could be done if the web were more semantic.

An alternative

[toddcw]

It's a commercial app, but it's saved us skads of time: screen-scraper. It's also a lot less of a "hack".

cousin of spam?

[GCP]

The easier and more widespread the techniques for spidering become, the more websites will get hammered with the unintended equivalent of DOS attacks, the way spam is the equivalent of a DOS attack on your email account.

I don't have any solutions in mind. I don't want anti-spidering legislation, for example, because *I* want to be able to spider. I just don't want *you* to do it. ;-)

Really, I'm just observing that as the Web evolves we could see another spam-like problem emerge, at least for the more interesting sites.

Re:cousin of spam?

[1iar_parad0x]

Well, if you space the time between HTTP requests, it wouldn't be spam.

This might be obvious or just a non-issue, but ignoring IMG tags in your bots (saves on bandwidth costs). You're probably not effecting their bandwidth by downloading text.

Incidently, most spammers are glorified script kiddies, not data miners or AI people. The kind of "hard-earned" money in data mining isn't the kind of money spammers are looking for.

The real problem with data mining is increased server load. Perhaps running your scripts late at night would help.

Of course, if server load was spam, then Slashdot would have a lot of explaining to do. :)

Re:cousin of spam?

[sethx9]

There is an industry built on teaching businesses and web designers how to increase ranking by making pages spider-friendly. The inverse of those same techniques could be used to protect a site.

If "bad" spiders became so common that businesses began needing to weigh the pros of page ranking against the cons of data theft then the indexing services (those that wanted to remain relevant) would develop other methods for accessing web content.

On a side note: I actually bought this book a couple of weeks ago as a tool to help me learn perl. Over the past few years I've built and used scraping tools and when I saw this book I was thrilled to have so many real-world examples that weren't about building front-end grids and tables to databases!

Re:Techniques used by spammers?

[Washizu]

"why would anyone use these techniques other than to harvest email like a spammer"

1. Archiving data on the web
2. Getting your files back when you forget your FTP password
3. Researching the link structure of the Internet and how it changes over time
4. Playing a joke on a friend by scraping his site and reposting the content, filtered in your favorite dialect
5. Reading your favorite site in an RSS reader, even if they don't provide an RSS feed
6. Counting how often certain words on used on the net
7. Checking to see if you have any broken links on your site
8. Testing to make sure every link is reachable on your site, and finding out how deep the deepest link is
9. Taking data from a public website and compiling useful statistics, such as GPA calculations, average completion times for cross country races, or the total number of points scored last night in the NHL.
10. Showing people that the Internet can be more than just a web browser

WWW::Mechanize is your friend

[andy@petdance.com]

You Perl folks who want something a bit easier than LWP for your spidering and scraping, take a look at WWW::Mechanize Besides the six hacks in the book that discuss Mech:

• #21: WWW::Mechanize 101
• #22: Scraping with WWW::Mechanize
• #36: Downloading Images from Webshots
• #44: Archiving Yahoo! Groups Messages with WWW::Yahoo::Groups (which uses Mech)
• #64: Super Author Searching
• #73: Scraping TV Listings

here are some other online resources to look at:

• WWW::Mechanize::Examples
  A random bunch of examples submitted by users, included with the Mechanize distribution.
• http://www.perl.com/pub/a/2003/01/22/mechanize.htm l
  Chris Ball's article about using WWW::Mechanize for scraping TV listings. (repurposed into hack #73 above)
• http://www.stonehenge.com/merlyn/LinuxMag/col47.ht ml
  Randal Schwartz's article on scraping Yahoo News for images.
• http://www.perladvent.org/2002/16th/
  WWW::Mechanize on the Perl Advent Calendar 2002, by Mark Fowler.

How much can you screen-scrape legally ?

[JPMH]

Question: how much screen-scraping can you do, before the legal questions start ?

In the USA, trading information that has cost somebody else time and money to build up can be caught under a doctrine of "misappropriation of trade values" or "unfair competition", dating from the INS case in 1918.

Meanwhile here in Europe, a collection of data has full authorial copyright (life + 70) under the EU Database Directive (1996), if the collecting involved personal intellectual creativity; or special database rights (last update + 15 years) if it did not.

I've done a little screen-scraping for a "one name" family history project. Presumably that is in the clear, as it was for personal non-commmercial research, or (at most) quite limited private circulation.

But where are the limits ?

How much screen-scraping can one do (or advertise), before legally it becomes a "significant taking" ?

Spidering Google Illegal?

[jetkust]

From Google Terms of Service:

No Automated Querying You may not send automated queries of any sort to Google's system without express permission in advance from Google. Note that "sending automated queries" includes, among other things:

using any software which sends queries to Google to determine how a website or webpage "ranks" on Google for various queries; "meta-searching" Google; and performing "offline" searches on Google.

Please do not write to Google to request permission to "meta-search" Google for a research project, as such requests will not be granted.

Re:Spidering Google Illegal?

[the pickle]

And what, exactly, constitutes "meta-searching" Google?

p

Re:Spidering Google Illegal?

[Jadsky]

This book requires that you submit to Google for a key to search with and use their API. In the hacks that require Google access, it'll just say something like

idkey = "insert your key here!"

AFAIK, this is standard practice for most sites with API access. (If you're interested, do it yourself at google.com/apis.) If you try to pull Google info down with an HTTP object programatically, Google will just return a 403 and tell you to read its terms of service. (Unless you spoof the header, but that requires doing it from scratch, and it will also get you in trouble if you try to use it commercially.)

RE: Booble

[Anm]

Been there.. done that...
http://www.booble.com/

Re:In other news...

[SparafucileMan]

There was a sign in arabic outside his shack that read "robots.txt...do not archive rug, rug/styrofoam, rug/styrofoam/hole, rug/styrofoam/hole/saddam", etc......let this be a lesson to all: security through obscurity does not work!!

Re: Booble

[Anonymous Coward]

Why'd you have to post that link? I have alot of important work to do and now it isn't going to get done...