Slashdot Mirror
98% of DNS Queries at the Root Level are Unnecessary
LEPP writes "Scientists at the San Diego Supercomputer Centerfound that 98% of the DNS queries at the root level are unnecessary. This doesn't even take into account the 99.9% of web pages suck or are unnecessary anyways. This means that the remaining 2% of necessary DNS queries are probably not necessary either."
99% of slashdot posts are unnecessary.
And they assumed the other 12 were exactly the same? Wouldn't looking at 2 at least be merited?
On a similar note, I noticed that AOL causes a lot of DNS lookups. From what I can see from my firewall logs, each TCP connection from an AOL user is handled by a separate proxy. Each proxy then does its own lookup on the host. So, for a normal sized webpage with some images or whatever, you get like 10 TCP connections for the content and 10 UDP connections for the DNS lookup. Seems kind of excessive to me.
Real man know IPs.
That's a thought! But we'd have to create servers for .vim, .pico, and .emacs as well...
98% of the DNS queries at the root level are unnecessary. [...] This means that the remaining 2% of necessary DNS queries are probably not necessary either."
Uhh... right, eliminate 100% of the root queries, they aren't needed..
sheeeesh..
Trolling is a art,
Just my 2p/2 worth.
Matt Thompson - Actuality - Insert product here.
is it that hard to configure a firewall to explicitly allow outgoing traffic rather than allow all? It seems that everyone thinks that the only bad traffic is the stuff coming in from the outside...
It's no wonder these servers have so many problems - there's thirteen of them! They need a lucky #14 - a Bilbo Baggins for their horde of dwarves. That'll stop those DoS attacks and unnecessary requests right away!
This doesn't even take into account the 99.9% of web pages suck or are unnecessary anyways.
What standard is this based on? My website wite sucks and is only necessary for my own amusement but it is similar to my favorite kind of sites on the web. I would use the web a lot less if it wasn't for those 99.9% of web sites. Most blogs, for instance, suck and are unnecessary but at the same time the total of all the blogs is having a big impact on news outlets and the media.
FoundNews.com - get paid to blog.,
From the article:
"Researchers believe that many bad requests occur because organizations have misconfigured packet filters and firewalls, security mechanisms intended to restrict certain types of network traffic. When packet filters and firewalls allow outgoing DNS queries, but block the resulting incoming responses..."
It's nice to see a story with info I can take and use. This is actually "stuff that matters".
Kudos to the researchers, and now I am off to check my firewall.
There are 01 kinds of cars in the world. The General Lee, and everything else.
Is it just me, or is this a description of a reverse lookup? How does that qualify as unnecessary? This is a pretty common step in troubleshooting, and some software does a reverse lookup following a forward lookup to verify that the hostname it gets back is the same one it started with.
Chuckles
I see this kind of thing all the time on /.--completely unedited, barely literate, rant-style submissions. Why don't the /. editors tone down or eliminate the rhetoric from submissions about otherwise worthy topics, or at least fix the grammar and typos?
I know, I'm going to get blasted for saying this, but I'm convinced it's one of those "little things" that makes /. look to the rest of the world more like a bunch of know-nothing kids typing at each other than a group of technically literate activists with something of value to contribute.
I now return you to your regularly scheduled rant...
Why don't DNS servers have a list of correct top-level domains, in order to answer directly, without going to a root server? The list is short, compared to the information the DNS server caches already, and the content of the list doesn't change so often. This list could be downloaded once in a day or so, from the DNS root servers.
When packet filters and firewalls allow outgoing DNS queries, but block the resulting incoming responses, software on the inside of the firewall can make the same DNS queries over and over, waiting for responses that can't get through
Why the hell does a firewall accept outgoing queries to black-listed domain names, if they are configured to block the response to these queries? This seems like a serious misconception to me.
JB.
74.4% of all statistics are made up on the spot.
"If anyone needs me, I'm in the angry dome."
"Scientists at the Vatican Praying Center found that 98% of the prayer queries at the God level are unnecessary."
If the authors actually thought how the DNS works they would realise the reason for this. A DNS server that gets a request for .com will consult the root the first time and then cache the result. So even though the server might then get a million hits in .com it won't ask the root again.
If the server tries to query for a non existent domain it will get back a 'non-existent' response. Now it will cache that response for some time but the chances of getting a cache hit is actually pretty low.
So if you have a properly configured DNS with a bunch of web surfers that view 1 million pages in 20 TLDs and 1,000 bogus ones they will generate 20 hits they would classify as genuine and 1,000 that were 'unnecessary'.
That is how the system is meant to work.
The 70% of repeated requests are likely to include outright attacks as well as misconfigured DNS systems.
The problem dealing with these issues is that a DNS query is pretty cheap to handle, cheaper in fact than most of the proposed defenses. It is probably more expensive for a DNS server to check IPs against a blacklist than to just return the damn data...
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
And that's a problem? My understanding was dealing with this sort of thing was exactly the purpose of the root DNS servers. If every ISP's DNS server was pre-configured to recognize valid and invalid top-level domains, you could just set them up to go straight to the specific DNS servers handling those domains (.com, .net, .org, etc.) There would be no need for a root-level system.
The argument for allowing this kind of cracked query through to the root server is that it makes it easy to add new domains (.elvis, .corp, what have you) without forcing everyone to reconfigure their DNS boxes for each new top-level domain.
Ummm... what does IPv6 have to do with DNS vanishing? With 128-bit IP addresses in an ugly hex-colon notation... DNS will be even more important when people move to IPv6.
The problem with DNS (and SMTP) is that they are protocols developed during a time where everyone on the internet was operating in a cooperative mode. Now that there is a proliferation of SPAM and DOS attacks, these old protocols break down because they were not developed with security in mind.
DNS will not go away. But the protocol will probably change at some point.
--
"What do you want me to do? Whack a guy? Off a guy? Whack off a guy? Cause I'm married."
1. Bad request received by a root server
2. Root server notices it's of the 'non-existent top level domain' variety.
3. Root server sends back information pointing to an ip that shows a web page with a nicer version of 'either you clicked a FrontPage created link, you are a monkey banging a banana on the keyboard, or your ISP administrators don't have a clue'.
Advantages: It'll embarrass ISP's. It'll cut down on the traffic to the Root Servers.
Disadvantages: It'll only be noticeable with web queries.
This is not a dream, not a dream...we are transmitting from the year 1-9-9-9.
DNS *does* suck! I mean, who wants to go through all of the trouble of laboriously remembering and typing "slashdot.org" in a browser when they can much more easily remember and type in "234.54.197.233.90.222"?? I pray for that day, also.
How about coming up with a DNS Moderation system.
The root servers give say 50 karma points to each IP address issuing a query.
If the query is unnecessary, it gets modded "-1 redundant".
When karma hits 0, it stops responding to further queries.
DNS eventually stops working at that site, admin pulls head out of ass and fixes the problem causing the redundant DNS queries.
Beauty is in the eye of the beerholder.
Huh?
Maybe I've been asleep at the wheel when it comes to all of the advantages of IPv6, but how on earth does it alleviate the need for a functioning DNS service?
Do you imagine that it will somehow be easier for people to remember IP addresses that are 128 bits in length than it is to remember them in their current 32 bit dotted decimal form?
I guess these will be what we have to look forward to in your DNS-free world of the future:
Riiiight.
For those that would die defending it, Freedom
has a sweet taste that the protected will never know.
One factor is that I suspect people are increasingly lowering their TTL's, expires, or whatever that parameter is. Most of the manage-it yourself DNS providers now allow an option toreduce that to a few minutes, which makes it much easier to move hosts around. And while a low setting increases DNS traffic, it rarely if ever incurs an extra cost to the domain holder.
It obviously seems to be a lot of junk traffic, but the only part we can say is bad requests are part 3 and 4 from the chart. Bad spellings must go to the root since there may be such domains!
It would be nice to analyze the 70% repeated or identical queries, probably lots of traffic can be explained for (or else there are a bunch of administrators out there who need a good manual on bind).
Scientists at the San Diego Supercomputer Center found that 98% of the remaining (necessary) DNS queries are related to porn websites!
2. The amount of time it takes to set up DNS correctly and effeciently with the existing products, especially BIND, is a lot more than it takes to just get them functioning.
3. The research would have been more interesting if they had gone and looked at say 1000 random requestors who where doing things screwed up and find out why and how they were screwed up.
4. It would be nice if the local DNS servers had a list of valid top level domains so that it would kill requests to non-existant ones.
THAT would be stuff that matters!
I'm surprised that they did not mention massive numbers of "broken" requests from Windows 2000/XP systems. I see this all the time due to misconfigurations. Administrators often set up the Windows 2000 DNS servers incorrectly and Windows 2000/XP systems(workstations and servers) configured such that they constantly try dynamic DNS updates to the wrong DNS servers, even the root servers.
Linux too, has some issues here. Obviously misconfigured DNS servers will always be a problem but, distros like Red Hat have IPv6 support compiled into the BIND RPM, this results in an IPv6 formatted query folllowed by an IPv4 query for every request.
Finally someone who makes a relevant comment. Though I wonder how the 'search from address bar'-feature has affected the number of non-existent queries.
This is somewhat of an invalid metaphor for both the way dns works, and the way computer caching works. Pretty much every local DNS server(unless my information is wrong), has some sort of caching system of varying degrees of efficiency. The problem is that unlike humans who are more likely to remember things if they are repeated, caching usually just consists of a series of entries which can quite easily be overwritten, older entries will be overwritten if they aren't updated or caching would never work for new frequently accessed sites. It's quite easy to get an access pattern which would remove even the most frequently accessed files from a list especially on a server with a great deal of users. By providing different servers for each chunk of users you can diminish this problem but then you'll get requests from each server. DNS is an ugly system because it does and ugly job.
A way to tell would be to see how many of the queries were looking for mx records.
I suspect that people using dummy email addresses like 'a@b.c' for subscriptions are another major cause of the misfires.
The browsers doing search from the address bar probably reduces the number of misfires. A modern browser will only go to DNS if it sees something like foo.bar. If it just sees foo it will typically try foo.com and then go bang a search engine.
Another reason I suspect spam is a major issue in the misfires is that lots of spam filters do lookup on sender addresses and those frequently point to non existent domains. Also the spam senders rarely do the most basic filtering on their lists - you can tell that since every now and again you get a spam with a full sender list at the top and you can see the broken addresses right there.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
That's a local problem, between the user and AOL's DNS servers. The article is descibing a different, higher-level problem between, for example, AOL's DNS servers and the root-level servers. If an AOL user's machine makes ten DNS requests for the same host, only one request should propagate past AOL's nameservers, but instead a misconfigured DNS will propagate all ten.
I can suddenly see lots of slashdot users thinking-- oh, I should fix my firewall, I have all these DNS requests; but that's normal operation for a client workstation. Your firewall would be broken only if all your DNS queries failed, and you'd know it pretty fast if that were the case.
So long, and thanks for all the Phish
It all comes out in the end anyway. Say AOL has 100 proxies. If 10,000 AOL users visit your site, then it'll look like only 100 unique visitors. Granted this is more than the 1 unique visitor that it would look like for most proxies, but it's still less than the actual number, not more. Presumably there are significantly less proxies at AOL than there are users. It only really matters to small sites like yours and mine, where we're getting excited about each and every visitor, and 10 all at once makes us need a new keyboard.
To do a reverse lookup, the resolver sends a different request type, asking for a PTR resource record. The form is to put the IP address (or network address) backwards, and append
If you have your own DNS server and watch your DNS traffic, you can see these two effects happening differently.
For a forward (A or MX record) lookup:
Local server queries root server for an A record
Root server responds with NS record for the registry of the domain
Local server contacts registry server for A
Registry server responds with NS records for the domain
Local server contacts the domain's server, which responds with an A record
Local server answers the resolver with the A record.
For a reverse (PTR) lookup, the resolver traverses the netblock providers:
Local server queries the root servers with a properly constructed PTR request (z.y.x.w.in-addr.arpa.)
Root server knows only where major net blocks are allocated, and returns the NS record of a Regional Internet Registry (RIPE, APNIC, etc)
Local server again queries an RIR NS with the PTR
RIR NS knows which ISPs hold which blocks, so responds with the ISP NS record
Local server again queries the ISP NS server, which either has the reverse hostname, or once again returns the NS record of the the local DNS server.
The two different types of queries follow different paths, either Name Registries or Netblock Providers. This article points out that many resolvers are broken because they allow obvious reverse lookups to pass as forward lookups, and then can't deal with the resulting error messages.
I have often seen broken resolvers repeatedly query DNS servers I manage, possibly because as the article points out, fucked firewalls allow the requests out, but block the requests from getting back to the resolver. It happens so much I just ignore it when I see it, its not worth notifying the admins because they are usually too clueless to know how to fix the problem.
the AC
Hemos is like...sci-fi fans;he thinks technology is cool, but he hasn't bothered to understand the science it's based on
If the First 98% are unnecessary and the last 2% are unnecessary as well...that's 100%...
That means that you just explained and wished for the Internet to go away...
or...you some how figured out how an end user can magically come up with the IP for a Host name from thin air. Go You. Your a Millionaire.
www.fotoforay.com
Really, we should have some sort of gnutella-like system for distributing zone files. The problem with DNS is that it was designed a LONG time ago before the more recent advances in P2P networks.
There shouldn't be much argument at this point that we need DNS2 - the current system is vulnerable to attack.
The problem is that, if you distribute zone files (or pieces of zone files) among a loosely-connected network, then you will need to establish trust. These zone files would have to be signed, and the certificate authority then becomes the bottleneck.
It hurts my head.
The fact that DNS, a 20 year old design, still works after being scaled several magnitudes beyond its original environment is proof that DNS doesn't need to be redesigned. The initial design is nothing short of genius. The extensions to the initial design (dynamic updates) build upon already solid technology.
I run a DNS server, I've looked at DNS packets, and every time I ask the Internet to tell me who the heck slashdot.org is and it comes back with an IP address I'm amazed. My network asks strangers for help and those strangers say: Hey, try here. Bam! Slashdot.org pops up in my browser.
You cannot "combine" DNS, DHCP, and Routing all into a single protocol. Hell, get three network engineers together sometime and try to get them to agree upon the best Internal Gateway Routing protocol sometime... EIGRP, OSPF, RIP.
Routing information is extremely different from domain name information. The two have nothing in common other than IP Addresses. You have to include not only information about who your neighbors are, but also what type of links are between you and your neighbors, and how congested those links are. Now, what about your neighbor's neighbors? Oh, we'll track that to, and also keep a set of tables that show us the next two best reconfigurations should any of the links stop working. Unless you're just talking about RIP for routing.
DHCP on the other hand is about getting clients configured for a network. They can then use DDNS to update their DNS record in a local DNS server. DHCP can do much more than just say: Here's your IP. It can also tell a client: here's where you should get your operating system from, and here's the voice over IP gateway, and here's the server where you should send your management info to, and here's the best local printer to use. Most people don't have clients that can handle that type of information, however.
It's not just "if it's not broke, don't fix it" this is a case of "it frelling works great, keep your hands off of it or I'll kick you in the jimmy."
The Internet was shown to be a scale-free network by U. Notre Dame physicist Barabasi. It means that the majority of the Web Page Requests is only for a fraction of the total Web Pages (the 'hubs').
Thus the 98% DNS Queries might be needed for only a minority of connections (I am assuming that Web Traffic is the bulk of Internet Traffic here).
Well, that's the theory. In practice, however, there are millions of servers out there that do not cache NXDOMAIN at all, and just keep querying, over and over and over again, for TLDs that they've already been told don't exist. Microsoft's name server has been known to do this.
At one point, f.gtld-servers.net was seeing millions of repeated queries per hour from the same two .mil servers asking the same question and refusing to accept the NXDOMAIN. For long periods, these two servers were asking the same question multiple times per click of F's timer. That's.. ummm.. Bad.
I suggest that you read the actual CAIDA paper, and the other papers on the subject that Evi Nemeth and others at CAIDA have produced. They *have* thought about how the DNS actually works in practice. You've only thought about how it would work if every implementation worked perfectly, according to your expectations.
Repetitive queries from the same nameserver in rapid succession, full-blown email addresses, search engine queries -- those are unnecessary, illegitimate queries that indicate not only bad nameserver configuration, but also bad application software. How many assorted DNS query permutation tricks have the various versions of Netscape Navigator tried over the years?
It should be a pre-requisite to remember each and every ip address before accessing the internet. :)
Problem solved.!!
yes, but now everyone's IP can look like:
: ::be:a:beef:ace
3ffe:10d7:::dead:beef:cafe:babe
and
2001:234d
what exciting words and numbers can you come up with 1234567890ABCDEF!
Reverse lookups go by sending a PTR request containing an IP address to a DNS server, versus a A request with a name as a snippet from this TCPdump shows a request from one my boxen to my DNS server:
Reverse:
12:59:31.814847 defender.licensedaemon > gimpy.domain: 20091+ PTR? 1.65.0.199.in-addr.arpa. (41)
12:59:31.816003 defender.1029 > arrowroot.arin.net.domain: 19500 [b2&3=0x10] [1au] PTR? 1.65.0.199.in-addr.arpa. (52)
Forward (complete request cycle from defender to gimpy):
13:11:54.760484 defender.globe > gimpy.domain: 47604+ A? www.gtei.net. (30)
13:11:54.761597 gimpy.1029 > dnsauth1.sys.gtei.net.domain: 51438 A? www.gtei.net. (30)
13:11:54.977584 dnsauth1.sys.gtei.net.domain > gimpy.1029: 51438*- 1/3/3 A 128.11.42.31 (167) (DF)
13:11:54.978626 gimpy.domain > defender.globe: 47604 1/3/0 A 128.11.42.31 (119)
DNS & BIND is the first book to use for more info, though.
// Agent Green (Ian / IU7 / KB1JQO)
// IEEE 802.3: All 10base Are Belong To Us
A DNS query for an IP address is a *BAD REQUEST* contrary to what some of these other posters have said. Asking a root server to resolve anything in the first place, is bad - they should only be asked for NS records - and in the second place, an IP address is not a valid domain name (unless ICANN has serripitiously added 256 new top level domains, namely, the numbers 0 thru 255).
Most networks that I've seen, are badly broken this way. The usual problem is that the network in question may use private address space (192.168.1.0/24 for example), but fail to install reverse dns for these addresses, causing delays and other problems when machines try to get the name associated with their ip address or that of a local machine connecting to them. Yes, you heard right - if you use any of the 192.168.x.x, 10.x.x.x, or 172.16-32.x.x addresses, you are broken unless you install dns to resolve for those addresses! This also goes for any ip netblock in general, although most isp's these days are setting up dummy records for their unused ip space that'll cover their customers allocations ok.
Let me get this straight. This is a study of DNS conducted by CAIDA at SDSC at UCSD? I need a host list for these acronyms!
Linux at home
That is hardly suprising since a lot of servers don't even cache the positive hits.
The report said 70% of the hits were repeated requests. Again this is not too suprising, the root zone caches really well. There are less than 200 domains after all and only 20 of those have a significant degree of activity. The TLD configurations change so infrequently that the TTL could be set at a month without inconveniencing anyone.
So the 'necessary' traffic for the root servers is negligible. Even with a million odd DNS servers out there each root need see no more than a few tens of thousands of hits an hour.
It makes no real difference since the roots have to be scaled to be able to survive a sustained DDoS attack for at least as long as it takes remediation measures to kick in. Get rid off all the bozo queries and you still need the same size box because of the script kiddies.
There are a bunch of changes that could be put in place that would reduce the DDoS problem. First we could follow the proposals of Mark Kosters and Paul Vixie to start using anycast (this looks like it is going ahead).
Another thing we could do is to change the DNS logic so that servers keep records in their cache beyond the TTL and use those as backup if the root or TLD is unavailable. Then even a DDoS that succeeded would have only marginal effect.
Looking for an Information Security student project suggestion?
Try http://dotcrimeManifesto.com/
I'll bet a large percent of the queries, especially for bogus top-level domains, are due to lookups by MTAs when receiving SPAM. Think of the numbers!
This doesn't mean that even these queries shouldn't be handled better -- just that SPAM lookups cause a bunch of 'em.
Well, apparently, you only have to fool the majority of people for a little while.
Actually, I've always had a theory that Microsoft coined ".msn" because they wanted to get their own top level domain.
It's not wasting time, I'm educating myself.
I've read through every comment on this page, hoping someone could explain this to me, but alas, I guess I'll just ask.
How does a query containing a valid IP address ever get to a root server?
It just seems to me that anything with an IP would bypass DNS altogether. Thanks!
You don't know what you are talking about, so you rant.
.elvis DNS request. ICANN might decide to open up .elvis registrations tomorrow and program the root servers to respond to them. If every DNS server had to be reprogrammed every time a new TLD was added, it would be a maintenance problem whenever the TLDs were expanded.
There is nothing malformed about a
The elegant part of the design was to define the protocol to look up unknown TLDs and unrecognized TLDs at the roots. It didn't anticipate a few million monkeys typing search terms into browser address lines.
The fault for the excess lookups lies in the applications programmers.
I have discovered a truly marvelous sig, unfortunately the sig limit is too small to contain i
I can think of some important things to consider about the interaction of your network and root servers.
.GPRS used for 3G phone networks (example) might be also query the wrong root level servers.
/etc/hosts file to pretend you have DNS on a nat'ed network. dnsmasq on freshmeat is very nice choice.
1. Some 3rd party DNS programs, like NetIQ and Preside. Require you to have root servers configured. Some will even break if you put false root information into it.
2. All unknown queries are sent to root servers, like your DMZ'ed networks. (Depends on your software has failsafe mode, but Bind can be disabled.)
3. Other TLD's are queried by the root domains.
4. With the security being a hot topic, networks are switching from recursive lookups to iterative mode. Which makes for more visible lookups on a network sniff by increased traffic.
Also, you can offload dns on your home networks by using a local dns server. Really handy, caches lookups, saves bandwidth, easier to setup than bind, and can use your