How To Build a Web Spider On Linux

IdaAshley writes, "Web spiders are software agents that traverse the Internet gathering, filtering, and potentially aggregating information for a user. This article shows you how to build spiders and scrapers for Linux to crawl a Web site and gather information, stock data, in this case. Using common scripting languages and their collection of Web modules, you can easily develop Web spiders."

Comments

some points

[cucucu]

• Don't forget to check and respect robots.txt. Python has a module that helps you parse that file
  
• Samie and its Python port Pamie are your friends. You can automate IE so your script is treated as an human and not discriminated as a robot.
  
• I use such beasts to do one-click time reporting at work and one-click cartoon collecting in my favorite newspaper.
  
• And once I even repeatedly voted on an online poll and changed the course of history.
  
• Ah, yes, TFA was about building a spider on Linux. I didn't check if my one-click IE scripts work on IE/Wine/Linux.
  
• If I write an one-click script for online shopping, does it infringe the infamous Amazon patent?
  
• When will Firefox's automation capabilities match those of IE?