Slashdot Mirror
Dealing With ISPs That Use NXDomain Redirection?
Vrtigo1 writes "I work for a small company that has about 50 staff on the road relying on VPN back to our office at any given time. Many ISPs have implemented NXDomain redirection services that hijack DNS traffic to show you sponsored links and other related ads when you mistype a domain name. These services are incompatible with most VPN software, since they prevent the computer from resolving internal hostnames. Large ISPs typically provide an opt-out on their sponsored links page that immediately opts you out of the DNS redirection, but I've noticed that some smaller ISPs and CLECs have opt-out links that don't actually appear to do anything. I don't have a good solution for employees using these ISPs, and our employees are getting frustrated because the problem is becoming more prevalent and we can't fix it for them. I've tried calling a few of these smaller ISPs for help, but it's been like talking to a wall. Manually changing DNS servers works temporarily, but the user can't resolve internal hostnames when they connect to the office LAN again. Have you had to deal with ISPs using non-standard DNS servers? What is your solution?"
If the small local ISP is screwing up, and refuses to respond in any useful way despite your best repeated efforts, it sounds like its time to take your business elsewhere, maybe to one of those large ISPs you mentioned. And make sure you tell them WHY. Who know, maybe the threat alone will be enough to get them to make a sudden change in policy for you, with a month or two of free service to boot.
Last time I setup a VPN, was with a Cisco PIX firewall, (its been awhile) but there was a spot to specify which DNS servers to use when connected to the VPN. I had specified that when connected, they would use our DNS, since they otherwise couldn't resolve \\file-server\share or whatever..
What are we going to do tonight Brain?
to force use of internal DNS servers while connected.
Done.
If you're splitting your connection between a VPN tunnel and a non-VPN protected internet connection, you're a security risk to your infrastructure.
Have your administrator configure full tunnel support where ALL of your traffic goes through the encrypted tunnel. That solves a security problem AND it fixes your DNS problem because you don't use your local internet provider's DNS servers.
Check out my sysadmin blog!
http://en.wikipedia.org/wiki/Split-horizon_DNS
What's the benefit of blocking your internal DNS? You're firewalled off, or they wouldn't need the VPN. What's going on here is that you're doing something broken - you must have some kind of NXDOMAIN redirector running on the remote machine, and the ISP is doing something wrong, because its NXDOMAIN redirector is fooling your NXDOMAIN redirector. If you just follow the standards, the fact that they have a broken NXDOMAIN redirector wouldn't affect you.
Another option is to set up a DNS resolver that's reachable from outside your network, and also inside your network, but only answers for your internal names if the query comes from inside. Then configure all your VPN machines to always use that nameserver, and not use your ISP's nameserver.
Even if your ISP filters DNS and answers in place of your nameserver, you're okay, because as soon as the VPN is set up, all the queries will go across the VPN (since this server is on your local network). At that point you'll start getting answers for local domains because now the query is coming from a local (VPN) IP address.
This second solution is a bit more work, and of course being a DNS geek I'm biased toward just doing the right thing in the first place, so I recommend just opening up your DNS, but either way ought to work.
There are still small ISPs left where you live?
Mod parents up, please.
And then we can all go home. This is an easy problem to solve once you see it from the right angle, and that angle is described above.
Kid-proof tablet..
Level 3's resolvers were VERY slow earlier this week, to the point where our IDS system noticed it. I've generally been glad to use them when an ISP screws up their DNS but it IS a free service and you can't expect great performance from it for that reason.
There are 4 boxes to use in the defense of liberty: soap, ballot, jury, ammo. Use in that order. Starting now.
This guy sounds like a manager or IT worker who is having problems with his employees connecting to the work VPN.
it sounds more like he has not stated the problem correctly.
how is it possible that a VPN connection is doing DNS to an external name server? Should not every internet request flow over the vpn from the client to the server. once it reaches the internal vpn server the server should know how to route the internal addresses and for external addresses it could use an external domain name server. the problem described seems like it should not exist. what am I missing?
Some drink at the fountain of knowledge. Others just gargle.
... and it should be stopped. Forced to stop if no other approach works.
Redirecting my web request to somewhere else, as far as I am concerned, is equivalent to re-routing my snail mail to their own office if someone has moved. That is not acceptable. I want a "not at this address" notice, nothing else.
A logon script here loads a hosts file that null-routes a lot of known bad (spyware, etc) sites.
Could you do the same for your internal hosts so that when on the VPN it doesn't even need to do a DNS lookup?
Don't blame me, I voted for Kodos
Some ISPs already won't let you connect to port 25 on any server that isn't theirs (forcing you to relay outgoing mail through them), ostensibly to prevent zombies from sending spam. The ones that monetize NXDOMAIN could easily do the same for DNS. All they'd need is some flimsy pretext, and maybe not even that.
I wonder if the actual problem is this:
1. User goes to internal site, gets ISP not found page.
2. User goes "Whoops, need to turn on VPN". Turns on VPN
3. User hits refresh. Still goes to ISP not found page.
Is he sure this isn't an issue of just needing the user to close their browsers to clear the browser dns cache?
The first thing your secure VPN tunnel should be doing is altering the client's DNS profile to only use the DNS servers on the other side of the tunnel. Anything else is totally insecure.
Level3 is in the process of ACLing off 4.2.2.1 from the world so only downstream transit customers can use it. Google the Outages mailing list.
What I'd love is my own DNS Server but I can't find one free for XP anywhere...
I think it's called linux. (Also, see VirtualBox or VMware server).
This is in fact why NXDomain breaks things in the way the poster describes, however, unless you're the kind of employer who wants to see EVERYTHING your subordinates are doing it's not actually the best practice to filter everything through the VPN.
Filtering everything through their VPN increases overall costs in bandwidth and hardware as Intron indicated. These are very real, very costly expenses that many employers overlook when implementing broad policies... and it's a fantastic point you raised that all too many companies forget.
Why should my connection to slashdot.org, for example, be secure on the company VPN? My ssh and nfs connections have very real reasons to be secure however!! On the other hand you could fix this by filtering DNS traffic through the VPN, but not web traffic. The cost of DNS traffic is marginal comparatively to other services, but the benefit for companies facing these specific issues is obvious.
Your external VPN interface should have its own IP address. That is a security best practice. If you are load balancing VPN connections, you should be using a VIP. Changes to VPN server IP addresses won't matter to the client. Using names for IP resolution works great for VPN connections until your DNS get hijacked!
I do not think your ssh connection needs to tunneled through a VPN at all. Ssh is a secure way to transmit and recieve information without a VPN. I suppose you could use a VPN with ssh, but it seems redundant. NFS is another matter, though.
SSH tunnels get around that without difficulty. If you know the address, it's as simple as assigning local port 2222 to 10.1.0.100:22 and you can now SSH to that machine by connecting to localhost:2222. Get a SOCKS capable SSH client, and you don't need to set up the tunnel for each connection.
That's not limited to small ISPs. Verizon FiOS, for example:
"Oh, sure, we will let you opt out - just click on the link that shows your router"
BROKEN LINK
Hmmm, guess I will click on a similar router...
THEY ARE ALL BAD LINKS
Gee, I guess I will click on the "change OS settings" link then...
BAD LINK
Somebody's going to point out that you can Google and find where helpful geeks have posted the instructions to opt-out without Verizon's assistance. But that's not the point, really, is it? Verizon had working opt-out links exactly long enough to get a favorable review in Consumer Reports, and then it all mysteriously broke. I cannot explain this coincidence, personally, you will have to come to your own conclusions.
In the case of the router vulnerability, this is something that you can control on the corporate side of things by simply not accepting packets down the VPN tunnel that don't come from the IP address that's the far endpoint of that tunnel. I'm not a VPN expert, but I would be surprised if this isn't how your VPN is configured by default.
You can also filter packets on the receiving end of the VPN. That's how I configured our firewall at work. The VPN tunnel simply looks like another network interface to our firewall, so I apply a slightly less restrictive set of rules to that connection than I do to the default external interface. Giving someone keys to your network just because they are an authenticated VPN user is not a very good idea.
My main complaint with DNS tampering is the outright DNS hijacking that Sprint does with their AirCard (EVDO) service. You can't even query a different DNS server-- your packets are intercepted and redirected to Sprint's own DNS. Unfortunately, their records are often out-of-date as it appears that they also manipulate TTLs to keep the churn down on their servers. It's a real problem when you're relying on something like an AirCard for doing things like network penetration testing.
dnsmasq, avalable in most distrobutions, is a light weight dns server that you can tell the ips of bogus NXDomain sends and will turn them back to what they should be. You can also point your computers to level3's free dns service at 4.2.2.3 4.2.2.4 4.2.2.5 4.2.2.6
I don't know that I would leave that hole open in my VPN configuration, but have you tried using OpenDNS (http://www.opendns.com/)? I don't know if it'll work in your situation or not, but I hardcode it vs. picking up the automatically assigned ISP's DNS and it works great. It doesn't have the problems with the redirection for advertisement when an incorrect URL is entered. In fact, that's one of my primary reasons for using it. Give it a try. Their site will give you the two IP addresses you need to use them, and best of all... it's free.
I worked for an ISP that provided service to hotels. VPN configs were the major source of problems. We implemented a captive portal to try to smooth over issues like
SMTP rejection (SMTP-AUTH was not common, the portal provided silent redirect to local mail server)
Accountability/Abuse -- The rooms were hard-wired, and captive portal gave us some retroactive sense of what room was generating abusive traffic.
Splash-screen/terms-of-service
DNS redirection is one of the core techniques for establishing captive portals. I rather doubt that many smaller ISPs are doing the "sponsored link" DNS redirect. Maybe things have changed since I left, but I suspect there is no significant benefit and some real cost involved for sponsored redirects for all but the largest ISPs.
Most of the support calls were over VPN software. Since all traffic was redirected until the splash screen was agreed to, a small but significant segment of VPN client configs broke. I very much suspect that is the real source of the initial posters issues.