Slashdot Mirror

← Back to Stories (view on slashdot.org)

How To Build a Web Spider On Linux

Posted by kdawson on from the five-eyes dept.

IdaAshley writes, "Web spiders are software agents that traverse the Internet gathering, filtering, and potentially aggregating information for a user. This article shows you how to build spiders and scrapers for Linux to crawl a Web site and gather information, stock data, in this case. Using common scripting languages and their collection of Web modules, you can easily develop Web spiders."

3 of 104 comments (clear)

  1. some points by cucucu · · Score: 5, Interesting
    • Don't forget to check and respect robots.txt. Python has a module that helps you parse that file
    • Samie and its Python port Pamie are your friends. You can automate IE so your script is treated as an human and not discriminated as a robot.
    • I use such beasts to do one-click time reporting at work and one-click cartoon collecting in my favorite newspaper.
    • And once I even repeatedly voted on an online poll and changed the course of history.
    • Ah, yes, TFA was about building a spider on Linux. I didn't check if my one-click IE scripts work on IE/Wine/Linux.
    • If I write an one-click script for online shopping, does it infringe the infamous Amazon patent?
    • When will Firefox's automation capabilities match those of IE?
  2. Actually... by SanityInAnarchy · · Score: 3, Interesting

    Some websites do not have good search functionality. Sometimes it's an area that Google doesn't crawl (robots.txt and such), and sometimes I'm looking for something very, very specific.

    Regardless, I do, in fact, build spiders. For instance, in an MMO I play, all users can have webpages, so it's very useful to have a spider as part of a clan/guild/whatever to crawl the webpages looking for users who have illegal items and such. In a more general way, there is a third-party site which collects vital statistics of everyone who puts those in their user page, so you can get lists of the most powerful people in the game, the richest people, etc.

    --
    Don't thank God, thank a doctor!
  3. Re:Hmm... by strstrep · · Score: 3, Interesting

    PHP lightweight? Ha!

    The PHP interpreter is over 5 megabytes in size. And it isn't thread-safe. That's a lot of memory overhead for a program that's going to be blocking on I/O most of the time, seeing how you'll have to fork() a new process for each new "thread" you want.

    Also, languages like Perl and Python have binaries that are about 1 megabyte in size. Now, while they'll probably need to load in extra files for most practical applications, these extra files are typically small. Most importantly, Perl and Python are thread-safe.

    Perl, for example, includes libraries such as Thread::Queue, which allows you to very easily create a threading model with worker threads, without having to worry too much about condition variables, mutexes, and the like.

    Disclaimer: All measurements done on x86 Debian Linux.