Slashdot Mirror

It's not the web host, it's the .ly TLD DNS servers that are having issues. If I had a .ly domain, I would probably be increasing my TTL records to whatever I thought I could get away with. I remember it being something like 2 weeks, before people start ignoring your TTL and using a "sane" default.

http://www.kloth.net/services/dig.php
dig: couldn't get address for 'dns1.lttnet.net': failure

  ; > DiG 9.3.2 > @localhost bit.ly A +trace
  ; (2 servers found) ;; global options: printcmd
  . 3600000 IN NS C.ROOT-SERVERS.NET.
  . 3600000 IN NS D.ROOT-SERVERS.NET.
[snip]
  . 3600000 IN NS A.ROOT-SERVERS.NET.
  . 3600000 IN NS B.ROOT-SERVERS.NET. ;; Received 500 bytes from 127.0.0.1#53(127.0.0.1) in 1 ms

  ly. 172800 IN NS dns1.lttnet.net.
  ly. 172800 IN NS dns.lttnet.net.
  ly. 172800 IN NS ns-ly.ripe.net.
  ly. 172800 IN NS auth02.ns.uu.net.
  ly. 172800 IN NS phloem.uoregon.edu. ;; Received 263 bytes from 192.33.4.12#53(C.ROOT-SERVERS.NET) in 24 ms

and a second time:

  ; > DiG 9.3.2 > @localhost bit.ly A +trace
  ; (2 servers found) ;; global options: printcmd
[snip]

  ly. 172800 IN NS phloem.uoregon.edu.
  ly. 172800 IN NS ns-ly.ripe.net.
  ly. 172800 IN NS dns.lttnet.net.
  ly. 172800 IN NS dns1.lttnet.net.
  ly. 172800 IN NS auth02.ns.uu.net. ;; Received 263 bytes from 192.203.230.10#53(E.ROOT-SERVERS.NET) in 180 ms

  bit.ly. 172800 IN NS ns3.p26.dynect.net.
  bit.ly. 172800 IN NS ns2.p26.dynect.net.
  bit.ly. 172800 IN NS ns4.p26.dynect.net.
  bit.ly. 172800 IN NS ns1.p26.dynect.net. ;; Received 110 bytes from 128.223.32.35#53(phloem.uoregon.edu) in 193 ms

  bit.ly. 3600 IN A 168.143.172.53
  bit.ly. 86400 IN NS ns2.p26.dynect.net.
  bit.ly. 86400 IN NS ns3.p26.dynect.net.
  bit.ly. 86400 IN NS ns1.p26.dynect.net.
  bit.ly. 86400 IN NS ns4.p26.dynect.net. ;; Received 126 bytes from 208.78.71.26#53(ns3.p26.dynect.net) in 13 ms

It's pretty easy if you go to a site like this. Just copy the number it gives you to your address bar and away you go. As long as their links are all relative, and don't hard-code the hostname, the rest of the site will work. A more technical user might edit their hosts file or change their DNS configuration, but this is a solution so trivial most non-technical users can use it.

Maybe they're giving different answers to different geographical regions, or maybe you have a transparent DNS resolver in your network path. Try from a different network. I've already posted one webinterface to check from in a sibling post to yours. Here's another:

http://www.kloth.net/services/nslookup.php

I'm sure you can find more.

Yep. And this bot trap that I use does just that. Works like a champ.

Google, Yahoo and MSN have already done this. Simply insert 'rel="nofollow"' into all the tags that people post in the comments, and although they still show up it makes it pointless for those spammers trying to increase their PageRank.

I know this won't help with the unsightly comments on your website, but since this is the slashdot crowd just flag all the comments with URLs in them as 'hidden' and on a daily/whenever basis go through them deleting spam and unhiding legitimate comments. Stick this all in a central control panel and it's unlikely to take up more than 10 minutes of your time.

In addition to that, just stop any client with a useragent string that contains a URL or one of the known spambot names.

http://www.kloth.net/internet/bottrap.php - A quick implementation of a bot-trap, which bans bots which don't follow your robots.txt directions.

Oh, yeah, and to actually answer the OPs question, there are lists of known bad bots out there...

Why did you need to contact your friends/relatives to check whether or not your domain gets propagated?
Couldn't you just query DNS servers directly using nslookup and/or dig?
Querying them directly would eliminate you from wondering if the machine you are checking from has the DNS cached and you wouln't need to flush it (why would you need your friends/relatives to reboot their machines?). Not to mention the amount of time you would spend in having to coordinate this type of testing.
Even if you don't want to use nslookup and/or dig from your Windows/Linux/Mac/whatever, there are tools available via the web that can help as well.
This certainly is not a list of all the tools, or even the best ones... they're just ones that I have used in the past:

dig Web-based "dig" tool
nslookup Web-based "nslookup" tool
DNS Report Checks for DNS errors and provides nicely formatted information on a given domain
DNS Stuff Various web-based DNS tools

Why did you need to contact your friends/relatives to check whether or not your domain gets propagated?
Couldn't you just query DNS servers directly using nslookup and/or dig?
Querying them directly would eliminate you from wondering if the machine you are checking from has the DNS cached and you wouln't need to flush it (why would you need your friends/relatives to reboot their machines?). Not to mention the amount of time you would spend in having to coordinate this type of testing.
Even if you don't want to use nslookup and/or dig from your Windows/Linux/Mac/whatever, there are tools available via the web that can help as well.
This certainly is not a list of all the tools, or even the best ones... they're just ones that I have used in the past:

dig Web-based "dig" tool
nslookup Web-based "nslookup" tool
DNS Report Checks for DNS errors and provides nicely formatted information on a given domain
DNS Stuff Various web-based DNS tools

Catch Bad Bots in a Bot Trap

You put a line in your robots.txt saying that bots are not allowed to access a certain directory or file. Then you put an invisible link to said directory or file on your home page. Any host that makes a request for the forbidden file is an evil bot, and gets blacklisted and/or reported to some other authority.

.... is the profile of the average spammer. Most of my spam is poorly spelled and frequently points to sites that don't have anything to sell. My suspicion, and I have no way of verifying it, is that most of these messages are sent by people who get suckered into a "Make Money From Home!" offer, send a few messages to a giant list of addresses, and then give up when they're not living in MC Hammer's mansion by the end of the week.

Does anyone know who the average spammer is?

Another cool piece of spam research I've never seen mentioned on Slashdot is the Bot Trap, which I learned about from this Little Green Footballs entry. If you're the admin for any web server, I strongly recommend setting this up. You probably don't make a huge dent in spam, but you get the satisfaction of seeing the list of IP's you thwarted.