Slashdot Mirror

← Back to Stories (view on slashdot.org)

Company Offers Customizable Web Spidering

Posted by ScuttleMonkey on from the my-legs-are-longer-than-yours dept.

TechReviewAl writes "A company called 80legs has come up with an interesting new web business model: customized, on-demand web spidering. The company sells access to its spidering system, charging $2 for every million pages crawled, plus a fee of three cents per hour of processing used. The idea is to offer Web startups a way to build their own web indexes without requiring huge server farms. 'Many startups struggle to find the funding needed to build large data centers, but that's not the approach 80legs took to construct its Web crawling infrastructure. The company instead runs its software on a distributed network of personal computers, much like the ones used for projects such as SETI@home. The distributed computing network is put together by Plura Processing, which rents it to 80legs. Plura gets computer users to supply unused processing power in exchange for access to games, donations to charities, and other rewards.'"

6 of 46 comments (clear)

  1. Ah, abusing someone else's bandwidth... by nweaver · · Score: 3, Insightful

    Lets assume that spidering a page costs 10 kB of data.

    So thats $2 for 1M pages, or 10 GB of data download.

    So thats at least $1 of data transfer that is being shifted onto the suckers, err "volunteers" who's home network is running this app.

    --
    Test your net with Netalyzr
  2. Hrm... by vertinox · · Score: 4, Insightful

    Sounds like a legitimate front for identity thieves, spammer, or even worse... Marketers.

    I suppose its easier to do than running your own bot net.

    --
    "I am the king of the Romans, and am superior to rules of grammar!"
    -Sigismund, Holy Roman Emperor (1368-1437)
  3. Seems cheap! by Pedrito · · Score: 3, Insightful

    Seems like an awfully cheap way to spider millions of pages of porn. It would be worthwhile if Google didn't do it already for free.

  4. Occam's razor. by icepick72 · · Score: 2, Insightful

    The levels of indirection present to support this system -- distributed clients, incentives for being a distributed client, power supply vs demand, payment for custom spidering -- make the system many things at the same time and unnecessarily complex, because those things already exist for free and in less complex ways. Many needs are sufficed by the simpler mechanisms and always have been.

  5. Reality... by JuSTCHiLLiN · · Score: 2, Insightful

    Plura gets computer users to supply unused processing power in exchange for access to games, donations to charities, and spyware.

    --
    What's a Sig?
  6. Re:Buried in Digsby by Anonymous Coward · · Score: 1, Insightful

    Why the hell would the embed plura into an IM client anyway?

    Unfortunately, it's all about money.