Slashdot Mirror
Providers Ignoring DNS TTL?
cluge asks: "It seems that several large providers give their users DNS servers that simply ignore DNS time to live (TTL). Over the past decade I've seen this from time to time. Recently it seems to be a pandemic, affecting very large cable/broadband and dial up networks. Performing a few tests against our broadband cable provider has shown that only one of the three provided DNS servers picked up a change in seven days or less. After turning in a trouble ticket with that provider - two of the three provided DNS servers were responding correct - while the third was still providing bad information more than two weeks after that specific change. What DNS caches ignore TTL by default? Is there a valid technical reason to ignore TTL?"
"This struck me as odd, and I decided to run a few tests using my own domain. Lowering the TTL to twenty four hours, and making changes and then checking to see when a change was picked up. I queried twelve outside DNS servers/caches that I had access to (Thanks to my friends and relatives with dial ups and DSL who put up with me and my requests to reboot their machine daily!). Checks performed against these outside DNS servers indicate that it may take as much as four to five weeks before a DNS change is picked up! Most DNS servers picked up the change within 48 hours. A small number did not (three out of twelve - that's a quarter of them!)
This merits more study, and prompts a few questions. So, before I begin with a more serious broad study, I'd like to get some feedback on the problem as I've seen it. I know the tin foil hat crowd will see the failure to propagate DNS correctly as censorship, and the OS/bind/djb/whatever zealots will simply see this as an argument for their particular religion.
Based on the responses I get, I will then setup and test a couple of domains with different DNS servers for 6 weeks and report back the findings. [volunteers welcome!]"
This merits more study, and prompts a few questions. So, before I begin with a more serious broad study, I'd like to get some feedback on the problem as I've seen it. I know the tin foil hat crowd will see the failure to propagate DNS correctly as censorship, and the OS/bind/djb/whatever zealots will simply see this as an argument for their particular religion.
Based on the responses I get, I will then setup and test a couple of domains with different DNS servers for 6 weeks and report back the findings. [volunteers welcome!]"
The whole thing about the web is it's based on trust... it's amazing it works as well as it does... I can see providors not obeying TTL simply to keep their DNS servers from being stuffed (or whatever the word du jour is)... It's all about trust, and the lack of it nowadays on the web... get used to this sorta thing happening more and more.
---
Programming is like sex... Make one mistake and support it the rest of your life.
How would I check my ISPs DNS servers for this?
RoadRunner
They can't run a DNS server properly to save their lives.
TTL is ignored, SpamAssassin is a "trojan DDOSing our network"
There have been DNS security concerns lately, specifically the 'cache poisoning' vulnerabilities reported by cert, sans et al.
Maybe some ISPs have altered their DNS servers to provide better protection, and in the process caused the 'ttl' problem ? (improbable imo, but who knows...)
It'd be interesting to know if this is recent or if it's already an old problem.
Knowing if it appeared suddenly or progressively would help as well.
Too bad there isn't such a thing as the wayback machine for dns and other services...
Perhaps this measure is meant to save some ressources on IPS's DNS servers if they have to query a lot of foreign DNS with low (and possibly overestimated) TTLs ?
I don't really know in-depth DNS mechanisms, but maybe ISPs are keeping a minimum TTL according to the average time between two updates of a given DNS entry ?
DNS queries are figgin tiny...
So, what's it really save you, even if you're a massive ISP to not obey the TTL's?
The only thing it's going to save you is from having to go out to the root servers and pull it again when the TTL expires. And, I speculate that this really is a very, *very* small amount of traffic compared to the other traffic to those servers.
I'd expect the highest bandwidth/resource users, by a very large margin, to be standard "in-TTL" answers to DNS clients.
So, these cranks, for lack of a better term, simply bork the systems they manage for no appreciable gain from doing so. Reducing spam by 0.0001% would have vastly more impact on the servers they maintain than ignoring TTL's.
Has anyone done any measurement stats on DNS queries. How much of the total traffic is DNS. I can't imagine it's even 0.5% of an average ISP.
Cheers,
Greg
Saving bandwidth is the only reason. It still a pain in the ass though. I found out the hard way when I though I was moving a website to a different IP address for the first time. I normally set the TTL to 7 days. So 7 day before the planned move I set the TTL to 1 hour. We switched the IP address and everthing was fine using ISP that followed DNS rules. AOL still had the address cached though. We didn't find out about teh problem with AOL right away. Our additude at the time was well screw the AOL users, there's no one importing using AOL anyways. I think it took a month before AOL finally updated to the right IP address. This definately makes it hard to do dns moves. At least with smtp you can add the mx record of the new server in advanced.
Some spammers setup DNS for web sites so that it's continually rotating through a number of different IPs, probably a number of them zombied PCs with web servers. The real stuff like transactions gets passed to other servers, but these disposable boxes act as lightning rods: A spam run won't be wasted if a few of them get complaints and taken down.
One line blog. I hear that they're called Twitters now.
Parent is the answer to why some providers ignore TTL, to prevent nutjobs like this from breaking the Internet.
I submitted a story similar to this one about a month ago regarding my experiences with direcway.com.
One of my customers was behind their network and we moved his email to our server. They couldn't access their domain name of course since it didn't exist on the server direcway's dns pointed to.
So I called them. Huge mistake. I spent hours on the phone escalating through foreign phone monkies until I made it to someone in management. Her attitude was that I was in the wrong regardless of what I had to say. Finally she lowered her defense just long enough to see that I was right but told me there was nothing they could do and that I wasn't allowed to talk to the people that run the DNS servers.
So I wrote a nasty little letter to corporate. 4 days later it was fixed. Not sure if the letter helped or not.
The man who trades freedom for security does not deserve nor will he ever receive either. - Benjamin Franklin
I've heard from a few people that many people are setting their TTL to like 5 minutes; due to this the ISPs are ignoring the TTL.
rooooar
I solved the problem by routing around the providers' poor DNS servers by pointing my home LAN to my own DNS server on my colo box. I run a DNS server in my house which then identifies my colo DNS server(s) with the forwarders option. Instead of relying on who-knows-what from the ISPs I use my own DNS server that I set up and am responsible for.
I know it's not a solution for everyone, but it does let me avoid stupidity. And having my own, reliable DNS server sure came in handy recently since Comcast has been having bad DNS problems the last couple of weeks.
Just to be on the record, a number of local ISPs here in Vermont don't update DNS records for 2-3 weeks. Sovernet, our big local provider, is among them. Very frustrating.
Waaa.. Waaa.. somebody ignored my TTL.
Listen -- We are SOA for around 11,000 domains. Both myself and the other uber-admins get tickets like this "escalated" when some clueless newbie wet behind the ears freaking junior admin DOESN'T RTFM and doesn't realize that if the serial #'s don't change then TTL is ignored.
We interface with nearly every major ISP -- I assure you, their name servers are configured just fine --- It's yours that is broke.
For those of you who just aren't DNS afficiandos -- so how retarded is this? Well here is another story ideas for slashdot that is along the same lines:
OMG! Two major RG vendors (NetGear and Dlink) don't follow RFC798 (TCP). See when I block port 80 on my firewall, the web stops working -- Imagine the whole web stops working by blocking just one little port. This is a huge problem! It needs to be addressed!
How about this little doosey:
I've just uncovered a SCO/Microsoft conspiracy. They've apparently teamed up because after reinstalling Windows XP onto another partition XP disabled my Linux partition -- the boot menu doesn't come up anymore. Clearly Microsoft is doing this to help SCO protect their intellectual property! Quick tell Groklaw!!!!
If you don't get either of the two above -- I can't help you. (seriously -- WTF are you reading slashdot for??)
I swear
I've recently "disconnected" my .COM version of my home domain (solely using the .US version now). As I get -0- spam to the .US address' (so far :) it is very apparent that a LOT of DNS servers [world wide] simply ignore TTL. A couple of weeks before the "switch" I set the TTL very low (60 seconds) -- I can easily handle the DNS traffic across my DNS servers (peppered here and there :).
.COM domain as I set _everything_ to IP address 127.0.0.1 (just to screw with the spammers high-jacked computers). Yet the spam [attempts] still come in. Every minute.
I would expect, now almost two months later, to be getting -0- traffic to the
Technically -- even running your own DNS servers there is nothing you can do if you move/add/delete something and others out there decide not to honor it. Everybody loses.
Usually on big providers overriding the TTL of the zone is a usual practice for sure, I do that myself in the ISP I'm working for (it's middle sized).
Why?
this is my sig
Yep -- we're seeing similar kinds of things. Recently, in preparation to migrating to some new servers we lowered our TTLs on the web server entries. Yesterday, we switched them over, and found that it wouldn't switch the traffic over within the TTL specs. Very frustrating to be sure, and we came to the same conclusion ... many (most?) Internet providers are ignoring DNS TTL.
Thanks,
Neil Ticktin
Publisher, MacTech Magazine
The point of TTLs is so people can cache domain name lookups for appropriate amounts of time. Only the source can say what is appropriate. The person caching the lookup has no idea what is appropriate outside of their own network.
So videotron can set TTLs for hosts in the videotron.ca domain but they should be reading TTLs for all other domains at the same time they make the query. They can't tell if the host they are looking up is a static server for years or someones dynamic host for a dialup modem that changes every 30 minutes.
i.e. they query ns1.google.com for "www.google.com", get back 216.239.37.99 and a TTL value of (for example) 6hours. So they cache that IP for that host for the next 6 hours. It is up to google to say what TTL is suitable, not videotron.
Here's a little tidbit that most people don't realize...
The Sun JVM implementation implements it's own DNS caching for any name resolution done by the networking APIs. By default the TTL for cached entries is... FOREVER. Not only that, but they will cache NEGATIVE LOOKUPS, so that if your resolution fails the first time it will fail forever.
The only solution is to restart your app (duh) or set the TTL as a system property on JVM startup.
I personally spent a few minutes staring at the monitor in shock when I first found this behavior by debugging a problem all the way down to the Java API source. Boggles the mind. Everyone else I've read who've 'discovered' this little known problem have had similar reactions.
This is unrelated to the TTL issue discussed in the article, but I try to take every opportunity possible to scream 'WTF Sun!?!?'
Well that sounds like a good explanation for why major ISPs might ignore the TTL. That's alot of wasted lookups and server load. For a large ISP we can expect that someone is going to most Akamai cites every minute so that is at least one lookup per minute per Akamai DNS record.
If other people do this as well ISPs might not have much of a choice but to ignore TTL and maybe there isn't any easy way to force the machine to use a minimum TTL but respect TTL above that limit.
If you liked this thought maybe you would find my blog nice too:
So rather than bitching about how stupid everyone is, why not take the time to explain why it is done this way? Take 5 minutes out of your busy escalated-ticket day, and tell us why it works the way it does - at least make an effort, rather than complaining about how much better you are than the rest of the world.
There are enough places (ISPs, like AOL), where TTL is completely ignored.
Case in point. A former high-traffic client (in the million hits per day range) did an 'inverse outsource' and reclaimed their site and corresponding domains. To prep for the move, the client migrated their DNS service to their new servers and changed the NS records with NetSol, several weeks in advance. They dropped their TTL down to two hours around that time. Early on the day of the cutover, changed the DNS records to point to the new location/IP range. That should've been enough, yes?
Well, they got a lot of complaints on the first few days that the site wasn't working. We ended up having to put create some URL redirection, ising the new IPs, on their old servers, and it took over two weeks for the traffic to die down. It took that long for some companies to flush their cache completely.
Companies are breaking the 'rules' and there's really not much you can do about it.
I'm MUCH more interested in how to get Mozilla to stop keeping its own DNS cache, or even remembering where the nameserver came from; at least on OS X.
Any time my IP changes (wired to wireless, say) while Mozilla is running, it goes off into never-never land and stays there.
Even on Windows it has its own resolver library, so if you're mucking about with changing a DSL gateway's identity, you have to quit completely and restart it.
Grrrr.
I hate to bear bad news, but there's nothing new here. Back in the 90s I observed similar situations--that no matter what the TTL, and even if you were careful to increment your version IDs, there were plenty of DNS servers that wouldn't notice DNS changes for weeks.
Hence whenever someone tells me that they're going to move their web hosting, I always advise them to allow for a couple of weeks of overlap, so that they don't lose a ton of traffic. Often they ignore my advice, because of course they have set their TTL low so their changes will take effect immediately, or so they think.
And then they wonder why they're getting hundreds of e-mails from people telling them that the site is down. And I forward them a copy of the e-mail I sent them beforehand warning them of the problem.
My gut feeling is that screaming about other people's DNS servers refusing to observe your TTLs is going to get you about as far as screaming about other people's SMTP servers refusing to deliver your spam. It's their server, they can do what they like. For instance, any TTL under 24 hours will be ignored by my caching DNS server. (RFCs say TTLs should be at least a day, more like 1-2 weeks.)
GCHQ Quantum Insert installed. If only our tongues were made of glass, how much more careful we would be when we speak
Lowering the TTL to twenty four hours, and making changes and then checking to see when a change was picked up.
To my knowledge, downstream caching nameservers will not check for changes in TTL before the latest cached TTL expires. Consequently, if the TTL is set to one year, then changed to 24h, the changed TTL will go on unnoticed for up to a year.
So I would like to know if the OP did allow the old TTL to expire after changing it, before he carried out the test. If not, the results of the test may be misleading.