mayby they can also split the data into small chunks that aren't worth it alone. (like in a packet switched network)
one would somehow have to sniff their whole network and access to just one machine wouldn't be enough anymore.
this whole asking the saleguy thing remembers me of us going to our local computer dealer in win3.11-times for the fun of 'format c:'. They found ways to stop us pretty fast;)
The problem seems to be the amzn_assoc crawler that alexia uses on amazons behalf to find out about broken links.
the bot was scanning through some kind of cgi script thus generating thousands of requests.
At issue is not only the frequency of page retrievals, but also the duration of the crawl. For example, on Nov 26th, amzn_assoc visited one of my sites 13,406 times over a period of 17 hours, consuming approximately 200 Mbytes of bandwidth via calls to MrRat's CGI script.
Thats called outsourcing...
They pay an external company (ie. Sun) to employ open-source coders for them!
IMHO Sun is being quite innovative by behaving like this!
like.. history is a while loop that never ends.
mayby they can also split the data into small chunks that aren't worth it alone. (like in a packet switched network)
;)
one would somehow have to sniff their whole network and access to just one machine wouldn't be enough anymore.
this whole asking the saleguy thing remembers me of us going to our local computer dealer in win3.11-times for the fun of 'format c:'. They found ways to stop us pretty fast
like switching to vim after years of saving docs with alt-d-s in notepad?
we ignore /robots.txt and we'll circumvent every actions you take to not let us crawl your cgi-bins
kneel down and I'll spank you, associates...?
the bot was scanning through some kind of cgi script thus generating thousands of requests.
.. something about not accepting any cookies? cookie filtering is just great ;)