Slashdot Mirror

← Back to Stories (view on slashdot.org)

Who Isn't Paying Attention to ROBOTS.TXT?

Posted by Cliff on from the bad-spider-no-donut dept.

Kickstart asks: "After wading through the Apache logs, after being hit hard for three hours by a very unfriendly spider, I see that there appear to be real, legitimate, search engines that do not follow robots.txt rules. Looking around, I see that some specialized search engines make no mention of their policy on this or say what servers their spiders come from. Does anyone have information on who follow this standard and who doesn't?"

5 of 85 comments (clear)

  1. zerg by Lord+Omlette · · Score: 3, Interesting

    The next question should be, "How do we make them regret their non-compliance?"

    --
    [o]_O
    1. Re:zerg by Eric+Giguere · · Score: 4, Interesting

      Start returning 500 errors... Or 302s that redirect them back to themselves...

      Eric
      PS: Is there some kind of bot storm going on, I'm getting all kinds of weird accesses to my site today, they're all fetching just the home page and leaving, and the referrer tag is null for everyone... They may be committing click fraud through my site, which makes me mad...
  2. Making them Pay by Kelson · · Score: 3, Interesting

    How about Stopping Spambots?

  3. whitehouse.gov/robots.txt by CommandoB · · Score: 5, Interesting

    The whitehouse seems to take a "pre-emptive" approach. Just in case they ever put stuff on the internet that they might someday not want you to see (or that they might not want archived by google), they seem to cover all the bases in their 92KB robots.txt file.

    My personal favorites:
    Disallow: /911/iraq
    Disallow: /911/patriotism/iraq
    Disallow: /911/patriotism2/iraq
    Disallow: /911/sept112002/iraq [sic.]

    There's a theme here. Can you spot it? I'd like to think it's intentional, but at 2255 lines, it may just be that all permutations of Republican buzzwords have been covered.

    --
    Not that I post on slashdot or anything.
  4. Re:On a similar note... by afidel · · Score: 3, Interesting

    I asked rob and he said they check for DDoS's whenever someone try's to post anonymously from an address. I told him it was busted because no one posted anonymously from my IP, and furthermore it's bad netiquet to port scan someone just because they accessed your site. Don't think he cares.

    --
    There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.