I must agree here.
Its isp that has same background guys as the pirate bay and they really respect your privacy and freedom to say what you want.
I know its mostly to route server traffic, but it makes great use in your personal stuff aswell.
Several major crawlers support a Crawl-delay parameter, set to the number of seconds to wait between successive requests to the same server: [1] [2]
User-agent: *
Crawl-delay: 10
I must agree here. Its isp that has same background guys as the pirate bay and they really respect your privacy and freedom to say what you want. I know its mostly to route server traffic, but it makes great use in your personal stuff aswell.
It actually does open on drag&drop. Just dont drop on the text area, but menu bar instead.
Crawl-delay directive
Several major crawlers support a Crawl-delay parameter, set to the number of seconds to wait between successive requests to the same server: [1] [2]
User-agent: *
Crawl-delay: 10