Slashdot Mirror

← Back to Stories (view on slashdot.org)

Robotcop: It's the Law

Posted by michael on from the you-have-ten-seconds-to-comply dept.

Voivod writes: "Inspired by the recent Slashdot and Evolt discussions about Blocking Bad Spiders, we set out to write an Apache module that solves this problem. The result is Robotcop and it's ready for action. We believe that it's the best solution to protecting Apache webservers from spiders currently available. Install it and help us make life hell for e-mail harvesting software!"

1 of 54 comments (clear)

  1. What about spoofing spiders? by regen · · Score: 3, Interesting
    It's an interesting idea, but it looks like the spiders have to be well behaved to get caught. If the spider never reads the robots.txt file and it claims to be a friendly user agent (not a spider) it seems the only way it could get caught is if it falls into a trap directory. This doesn't seem likely.

    How do you prevent users from finding and falling into a trap directory? It seems like it wouldn't be that difficult to write a spider to get around the restrictions imposed by robotcop.

    Am I missing something?

    --
    The Economics of Website Security