Meet Cyveillancebot

gulker writes "A rant about making a new 'acquaintance'... Googlebot is like the UPS driver who comes to the door in a uniform, and will happily show you his ID and business card: Cyveillancebot is like a coarse, unshaven, itchy guy with his hat pulled down lurking near your half-open bedroom window. This after Cyveillance defeats a 'protection mechanism' - robots.txt - and grabs 155 copyrighted files from my Web server, which files it will presumably share with others, for a profit..."

Comments

Re:And this is why many ISPs don't give log access

[km790816]

I totally agree...but...

This is classic American business practices.

We are a good, upstanding corporation.
We want to protect our turf.
We employ a company to help us.
We don't ask about that companies means or, more likely, turn a blind eye.

Dell would never agree that applications on the Internet should, in general, act the way that Cyveillancebox does.

I believe that the author understands your point. He's not whining.

He is, however, pointing out the hypocrisy, which I think is valuable. I'll think twice about buying another Dell.

Re:Cyveillance in a nutshell

[PurpleFloyd]

To me, these actions (hammering databases, getting caught in recursive loops that could be easily avoided) are much worse than ignoring robots.txt. While the whole robots.txt issue could be justifiable from their position (so people couldn't hide copyrighted info via robots.txt), bringing down servers through what amounts to a DOS attack is simply inexcusable.

There are any number of spiders out there that are smart enough to index whole sites, including dynamically-generated pages, without taking a site down or even hitting it harder than a couple of simeltaneous users. This behavior is not only negligent, but malicious. Any site brought down by Cyveillance would probably have good grounds for legal action (I am not a lawyer, this is not legal advice, talk to a lawyer if you want legal advice, etc.).