Slashdot Mirror

Still, poisoning might not be too bad for spiders that misbehave.

Hold back the excitment, people, it's another episode of story recycling.

This site is pretty handy, now that I'm on the topic. Also make sure to check out RobotCop. Out for Apache now, coming soon for IIS and Zeus!

Apache 1.3 on Unix is multiprocess, and Robotcop uses libMM for sharing things like spider lists between these processes. Apache 1.3 on Win32 is multithreaded, and would require a totally different approach to this. At the time I thought Apache 2 was closer and let me avoid this work, but I'm thinking your estimate is more accurate now. I'm not too familiar with the Apache on Win32 community, but your comment is request #2 for Robotcop to support it, so I might go ahead and do it.

Regarding your src vs binary comments, you might be interested to know that about 95% of visitors to the Robotcop site choose to download the source instead of the much easier to install binary. It'll be interesting to see how Win32 is different once I have something up for that.

P.S. Don't call people "consumers". Even if they are Windows users, it's not nice. :-)

The Robotcop download page states that no binaries are available for versions of Apache HTTP Server designed for M$ Windows, and the binaries that do exist (for Red Hat Linux x86 and FreeBSD x86) aren't very compatible with mod_ssl.

"So compile it yourself!" For one thing, according to the compilation instructions, those who want to compile Robotcop for Windows will have to wait a year (estimated) until Apache 2.0 is no longer eta but Released. For another, not everybody can afford a license for M$ Visual Studio, which is required to build Apache HTTP Server; apparently, this popular Win32 version of GCC doesn't cut it.

In other words, Robotcop won't work for consumers who serve web pages from their home workstation that runs Windows.

The Robotcop download page states that no binaries are available for versions of Apache HTTP Server designed for M$ Windows, and the binaries that do exist (for Red Hat Linux x86 and FreeBSD x86) aren't very compatible with mod_ssl.

"So compile it yourself!" For one thing, according to the compilation instructions, those who want to compile Robotcop for Windows will have to wait a year (estimated) until Apache 2.0 is no longer eta but Released. For another, not everybody can afford a license for M$ Visual Studio, which is required to build Apache HTTP Server; apparently, this popular Win32 version of GCC doesn't cut it.

In other words, Robotcop won't work for consumers who serve web pages from their home workstation that runs Windows.

looking over the technical review and the readme, a few initial, random, and sporadic thoughts:

the blocking of valid users seems rather annoying (NAT users, some proxy users) and a bad spider could get around the short interval by increasing its sleep time.

IPv6 could screw your implementation. If i have access to a huge number of IP addresses then i could access your website through any one of those addresses. A spider could run an initial probe of a few million websites through one ip, change ips, then grab a second page from all those websites, change ips, grab webpage, etc etc.

if i know a website is running robotcop, can i screw over valid users by forging my ip address, accessing robots.txt, then accessing a honeypot dir? can i screw over all users by cycling through all ips and doing this (yeah that's time consuming, maybe i could just screw over users from one range?)?

The main problems i see from the robotcop approach is that it assumes everyone who accesses robots.txt is a robot and it assumes valid users will not follow certain paths through the website.
This is different for email poisoners b/c if i'm a user and i get to page with a bunch of (invalid) email addresses, it doesn't matter. i click back and continue on my way. but for something that actually *blocks* users, it's a bit different.

As it stands now, i could go to an internet cafe (often they use nat) and block every other user from seeing any site protected by robotcop.

How about tying both User-Agent and IP address to form valid/invalid users? that way a bad user behind NAT might get blocked while a good user could go on. The more information you can tie to one particular thread of access, the more likely you are to single out one particular user.

Instead of only blocking ips that seem to be bad spiders, why not feed themm specific information? that way if it is a user you can let them go on - "if you are a valid user, enter the word in the graphic below in this text field and click 'ok'!"

It really seems that whatever you do, it is possible to work around. Set cookies? i write a bot that keeps track of cookies. hidden webbugs/urls? my bot avoids these.
I can see robotcop as working in small cases, like for a limited number of servers on the internet, b/c then it is not worth the bot writer's time to implement work arounds. But once it becomes worth their time, you have a game of evolution.

Not that that's bad; keep a small enough base of users and you probably wont need to update methods all that often.

looking over the technical review and the readme, a few initial, random, and sporadic thoughts:

the blocking of valid users seems rather annoying (NAT users, some proxy users) and a bad spider could get around the short interval by increasing its sleep time.

IPv6 could screw your implementation. If i have access to a huge number of IP addresses then i could access your website through any one of those addresses. A spider could run an initial probe of a few million websites through one ip, change ips, then grab a second page from all those websites, change ips, grab webpage, etc etc.

if i know a website is running robotcop, can i screw over valid users by forging my ip address, accessing robots.txt, then accessing a honeypot dir? can i screw over all users by cycling through all ips and doing this (yeah that's time consuming, maybe i could just screw over users from one range?)?

The main problems i see from the robotcop approach is that it assumes everyone who accesses robots.txt is a robot and it assumes valid users will not follow certain paths through the website.
This is different for email poisoners b/c if i'm a user and i get to page with a bunch of (invalid) email addresses, it doesn't matter. i click back and continue on my way. but for something that actually *blocks* users, it's a bit different.

As it stands now, i could go to an internet cafe (often they use nat) and block every other user from seeing any site protected by robotcop.

How about tying both User-Agent and IP address to form valid/invalid users? that way a bad user behind NAT might get blocked while a good user could go on. The more information you can tie to one particular thread of access, the more likely you are to single out one particular user.

Instead of only blocking ips that seem to be bad spiders, why not feed themm specific information? that way if it is a user you can let them go on - "if you are a valid user, enter the word in the graphic below in this text field and click 'ok'!"

It really seems that whatever you do, it is possible to work around. Set cookies? i write a bot that keeps track of cookies. hidden webbugs/urls? my bot avoids these.
I can see robotcop as working in small cases, like for a limited number of servers on the internet, b/c then it is not worth the bot writer's time to implement work arounds. But once it becomes worth their time, you have a game of evolution.

Not that that's bad; keep a small enough base of users and you probably wont need to update methods all that often.

You just put hidden links in your HTML which only a spider's HTML parser would notice and follow. This technique is already widely used by wpoison which is a Perl CGI solution to the spider problem.

Check out the robotcop.org site. It has examples of how to set all this up.