Re:Spidering Google Illegal?
on
Spidering Hacks
·
· Score: 1
I know for a fact it records the UserAgent of the program sending the queries. I once wrote a Perl module to harvest sentences using the groups.google.com collection of articles. Well I miscalculated a variable and the queries started to be sent too rapidly. Sure enough they started to time-out. However, switching the UA was a quick fix, even from the same IP. I didn't keep sending queries after this started though...;)
The protocol is nice and I use it quite a bit, but it suffers from the fact that you need to find the.torrent files before you can use it. This to me is a point of failure that should be avoided. Why isn't it possible for a client/tracker to share the torrent files themselves as part of the overall transaction?
While people may want to hide what torrents they have, this could go a long way to eliminate the problem.
Slashdot Mirror
I know for a fact it records the UserAgent of the program sending the queries. I once wrote a Perl module to harvest sentences using the groups.google.com collection of articles. Well I miscalculated a variable and the queries started to be sent too rapidly. Sure enough they started to time-out. However, switching the UA was a quick fix, even from the same IP. I didn't keep sending queries after this started though... ;)
The protocol is nice and I use it quite a bit, but it suffers from the fact that you need to find the .torrent files before you can use it. This to me is a point of failure that should be avoided. Why isn't it possible for a client/tracker to share the torrent files themselves as part of the overall transaction?
While people may want to hide what torrents they have, this could go a long way to eliminate the problem.