Meet Cyveillancebot
gulker writes "A rant about making a new 'acquaintance'... Googlebot is like the UPS driver who comes to the door in a uniform, and will happily show you his ID and business card: Cyveillancebot is like a coarse, unshaven, itchy guy with his hat pulled down lurking near your half-open bedroom window. This after Cyveillance defeats a 'protection mechanism' - robots.txt - and grabs 155 copyrighted files from my Web server, which files it will presumably share with others, for a profit..."
And this is why many ISPs don't give log access to the people they host for
Too many user will run around screaming "Somebody's stealing my stuff! WAAAAAAH!"
Look, robots.txt is a gentleman's agreement. The internet is open for all, not just gentlemen.
WELCOME TO THE INTERNET!
HOW WOULD YOU LIKE TO BE ABUSED TODAY?
Please take some time to re-read that, and disabuse yourself of the idea that you can control other people - the only control you have over them is going to be control they will also have over you.
Secondly, if it really bothers you, block, ban, tarpit, data-spam, whatever the heck you want out of them. If they dare contact you, you can give them whatever you want, however you want. Send them data from random at a rate of 2 bytes per second. Drop half the packets they send you. Abuse the TCP/IP protocol and see if their software is robust enough to handle it. Make it just annoying enough to contact your website in certian ways that they will put you on their 'do not spider' list. Here's a simple example.
But, man get a grip. Seriously. Do you honestly think you are the first person to discover the dark underbelly of corporate money making schemes on the net?
-Adam