Slashdot Mirror
Company Offers Customizable Web Spidering
TechReviewAl writes "A company called 80legs has come up with an interesting new web business model: customized, on-demand web spidering. The company sells access to its spidering system, charging $2 for every million pages crawled, plus a fee of three cents per hour of processing used. The idea is to offer Web startups a way to build their own web indexes without requiring huge server farms. 'Many startups struggle to find the funding needed to build large data centers, but that's not the approach 80legs took to construct its Web crawling infrastructure. The company instead runs its software on a distributed network of personal computers, much like the ones used for projects such as SETI@home. The distributed computing network is put together by Plura Processing, which rents it to 80legs. Plura gets computer users to supply unused processing power in exchange for access to games, donations to charities, and other rewards.'"
Lets assume that spidering a page costs 10 kB of data.
So thats $2 for 1M pages, or 10 GB of data download.
So thats at least $1 of data transfer that is being shifted onto the suckers, err "volunteers" who's home network is running this app.
Test your net with Netalyzr
Sounds like a legitimate front for identity thieves, spammer, or even worse... Marketers.
I suppose its easier to do than running your own bot net.
"I am the king of the Romans, and am superior to rules of grammar!"
-Sigismund, Holy Roman Emperor (1368-1437)
Seems like an awfully cheap way to spider millions of pages of porn. It would be worthwhile if Google didn't do it already for free.
This is apparently the service that caused a lot of controversy when people discovered it was somewhat hidden in Digsby.
I can see how they might get a fair number of people to donate their spare cycles for this, if the rewards are seen as sufficiently interesting. But are there really a whole bunch of startups (or other companies) that are really champing at the bit to create a new search engine? Other than marketers or malware purveyers, I mean. And do these searches honor robots.txt exclusions?
BTW I took a quick look at 80legs' website in an attempt to get these answers. I came up empty in that regard - so I will comment on how the CEO's hair makes him look like an in-disguise member of the Conehead family. Seriously, what's with the hair?
#DeleteChrome
"I mean, if you're going to be some kind of start-up search engine or "semantic company" (whatever that means), shouldn't Web spidering be your core competency? If you're going to differentiate yourself in the market, how can you buy spidering as a commodity?"
Raw spidering is pretty much a commodity already. You're issuing GET requests over HTTP (for the most part). The "semantic" stuff comes in to play analyzing the results and doing interesting things with raw information you get back. If people can spend more time focused on doing the 'interesting bits' and less time on having to scale up to pull in the raw data to analyze, they'll be better off for it and more likely to be able to focus on creating something new/interesting/distinguishing.
People (generally) don't write their own web servers, nor their own TCP/IP stacks, often don't write their own session handling logic, or security code. All of these things have been commoditized. Perhaps too many people are relying on 'cloud computing' these days, but hosting and storage 'in the cloud' is where all the cool kids are playing right now (I don't necessarily agree with it, and probably wouldn't put all my eggs in that basket myself, but others are doing so). Spidering may be the next frontier to get commoditized.
Perhaps not everyone is comfortable 'partnering' with Google for everything? If someone was going to work on developing the 'next big thing', would you rather invest in something where the people had spent an inordinate amount of time building network capacity up to do drone work, or used a service like 80legs, or built the prototypes on Google's servers? Depending on the project, any of those make sense, but I'd prefer to use a service like 80legs myself. They're small enough and hungry enough they should give top notch customer service at this stage, whereas Google's not going to give you a number to call for direct service (maybe they do if you're spending loads of money, but then you're back to wise use of money).
The P2P aspect of how they're doing the spidering may be clever, but I'd rather see a more direct use of data-center resources around the globe, rather than relying on a seti-like participation model.
creation science book