DNS based Website Failover Solutions?

Chase asks: "I run a couple of websites(including for my work). I'd like to have a backup web server that people would hit when my server goes down. My primary host is on my companies T1 line and even though I've had my server die once the most common reason for my sites to be offline is that our T1 goes down. I've looked at the High-Availability Linux Project but it seems that almost everything there is for failover using ip takeover which isn't an option if my network link dies and my backup server is on a different network. ZoneEdit seems to offer what I'm looking for but I'm wanting a do it myself solution. The only software I've found is Eddie and it seems to have stopped development around 2000. I know DNS based failover doesn't give 100% uptime but with a low cache time and decent monitoring it seems like it's the best solution for having my backup server at a differnt location and on a differnt network. Anyone know of a good solution? (Using Linux and/or Solaris hosts)"

Comments

Dyndns

[pbulteel73]

Dyndns.org offers free DNS services for dynamic ip addresses. They also offer a fee service that allows you to use your own domain name. Why not set it up with them? If your web server is unreachable by the other server, it will send a dyndns update query with the new address. Just a thought. -P

Re:Dyndns

[anicklin]

dyndns is pretty good in that with a custom domain, you can set an 'offline' redirect URI. However, this has to be done manually with an internet connection - kind of a problem if the dedicated public connection is unavailable, although you could always revert to some sort of dialup to get onto their web site and update it.

They will let you configure custom TTL values on A (host) records. I set mine to 5 minutes and it works just fine.

There are some automated engines out there which will update the dyndns service automatically, but I have not seen any which will automatically set the unavailable URI if the primary internet connection isn't available.

dyndns is more oriented at people who want to host but their address changes frequently, whether for black-hat, white-hat or ISP DHCP reasons. However, while reliability has never been a problem with their service, it may not suit the needs of a more commercial customer.

Just my two cents as a happy user.

Depends whether you want to pay for it . . .

[unixbob]

If I understand you correctly you you are looking for a F/OSS project to do what you are after.

However if you do actaully have a budget to spend have a look at the 3DNS product from F5 Networks. it does the failover you describe and although it works better if it is intereacting with F5's server load balancing product, it can still monitor and react to standard web servers becoming unavailable.

uhhhh

[nocomment]

If your T1 is down tht often I'd change providers. My T1 has been 'slow' once in the past year with 1 outage that lasted for about an hour when we first installed it.

Re:uhhhh

[nocomment]

If you need the QoS, but not necessarily a full T1 maybe you should look at SDSL. With ADSL the phone company owns the switching equipment and can turn it off/move/upgrade/whatever whenever they want. But with SDSL the provider (ie speakeasy, covad(if covad does sdsl)) owns the switching equipment and will skip over it when doing their moves/upgrades/whatever. Speakeasy has a QoS guarantee. I still feel safer with a T1 though :-)

backhoes are easy to fix, I remember when I worked at Mindspring (pre-Earthlink) there was major outage (a hurricane I think) in NY that not only broke the T1 (there was exposed fiber) but it was also under 30' of water. It took 7 days to drain the water before the cables could be repaired.

A few ways..

[ADRA]

1. Use colocation/Web hosting as the primary site. Their uptimes are usually very strong.

2. You will need a second line. Mandatory. If you really want insane uptime, you'll need dynamic routes ala BGP from both ISP's. If you don't need that much, you could maybe work with an automated probe-and-dnsupdate script which can run outside the network. It would switch the primary DNS to and from the backup IP address which is on the isolated network.

3. Have an equalized DNS entry for both IP addresses. It gives the client a 50% chance of connecting once its dead, but its better than nothing.

4. Tell the site visitors to connect to www1.mysite.com if they're having troubles reaching your site and have www1 pointing to your backup IP. Make sure your DNS servers are network redudant as well, or the whole excersize is pretty pointless.

You could always use IPv4 Anycasting.

[Mordant]

More information here.

Linux server hacks and the slashdot-effect...

[kwench]

Read all about IP take over and distributing server load as sample chapter of O'Reilly's Linux Server Hacks.
Don't know if it works for your setup.
My favorite quote:
If you serve a particularly popular site, you will eventually find the wall at which your server simply can't serve any more requests. In the web server world, this is called the Slashdot effect, and it isn't a pretty site (er, sight)

RFC 2136 + Net::DNS + your monitoring software

[embobo]

Ignoring the fact that DNS wasn't designed to handle this (setting your ttl to a low time (e.g., 5min) generates a good amount of useless traffic when your site is up), here is how you might do it:

First, you need to have a monitoring system on the Internet somewhere, not through your T1 because if that goes down it won't be able to update your DNS. You have that already, I'm sure, to test your web site accessibility from the Internet. Of course, at least one of your name servers must be accessible when the T1 goes down too, so that will have to be somewhere (other than on your T1) on the Internet as well.

On this name server enable dynamic updates. Modify your monitor system that checks availability of your site to use Net::DNS to update the IP address of your web server when the monitor fails.

Going all open source, I'd use Net::DNS and nagios for the monitoring software, bind for the name server (which supports dynamic updates), with Linux as the OS.

Re:RFC 2136 + Net::DNS + your monitoring software

[byolinux]

Nagios

with Linux as the OS

Kernel! And anyway, does the fact you're using GNU/Linux or *BSD actually make a difference to this?

Re:RFC 2136 + Net::DNS + your monitoring software

[FistFuck]

I do it now with two shell scripts.

The key is that I use tcpclient from DJBs ucspi-tcp package:

http://cr.yp.to/ucspi-tcp.html

Don't hurt yourself with BIND, either. Parsing that file is going to hurt your brain. I use grep -v to manage my data file for tinydns:

http://cr.yp.to/djbdns.html

Maybe I'll get around to publishing my work. A brief synopsis:

I do a tcp connection to port 80 on my webservers with a 5 second timout. If the connection fails it pulls all IPs assoicated with that server out of my DNS. Not only does this determine if the server is up but it also determines if the server needs less load because it can't get to my request
in time.

There's a state file for each webserver, ie webserver.up or webserver.down. That's easy to look for later to determine if I need to change the DNS tables.

I run the check every 60 seconds. I only have two servers so it's not too tough.

I also check www.yahoo.com and www.google.com availability over each ISP to determine if an ISP is available. I update DNS based on the ISP conditions as well.

I say again, try to avoid BIND if you can, I can't think of a sane way to process your zone files with shell scripting.

Re:RFC 2136 + Net::DNS + your monitoring software

[ptudor]

...I can't think of a sane way to process your zone files with shell scripting.

Luckily, when moving to tinydns there is a sane way to convert your zone files with shell scripting.

It depends....

[RedHat Rocky]

First thing you need to do if decide what kind of downtime is acceptable. 5 seconds, minutes, hours?

Then you need to look at your services you're offering from your website, is it all static, session-based or what?

Combine the two to figure out how much your downtime is going to actually cost you. For example, if my personal site, which is static, is down for 5 hours the only person who is going to really care is me. And I don't pay myself much. :)

Flipside, on an ecommerce site with shopping cart, that 5 minutes of downtime could cost a lot of lost sales.

In otherwords, your redundancy plan should match how much you think you'll lose if Bad Things Happen.

Now, you're on a T1 with some personal stuff, let's assume 5 minutes is fine, money lost is minimal, but any more time will be irritating. Your content is static. Here's a cheap DIY solution and yes it's DNS based.

Setup identical webservers on seperate networks. Have those servers also be the nameservers for the website in question. Configure each webserver to only answer an A query as itself. The ttl for the A record needs to be low (5-10 minutes). Now, if one of the servers/networks goes down, clients can only resolve DNS by reaching a server; server down, can't query it, they'll hit the other server.

This method has some downsides, as mentioned bandwidth usage will be higher as more DNS queries will be made. Session-based stuff also won't work, no guarantee which server any given request will hit.

see p2pweb.net

[p2pweb]

I'm working on a similar project : it's called p2pweb.net.

The site is distributed on 4 web servers : 3 on ADSL lines, one on SourceForge. I use 3 independant DNS to announce the web site. On each DNS I also run NAGIOS to monitor each web site. When one of the web site goes down (or up) a special handler (in perl) is called by NAGIOS and dynamicaly update the DNS entry

see global Load balancing for more details and code examples (in french only, but I am working on an English translation).

I set up the DNS TTL to 300 seconds, and NAGIOS can detect a state change in 2 or 3 minutes. So I can have global fail over in less than 10mn.

I have the system running for some month, and it works very well.

It's a king of "poor man's" akamai.

Multiple Master Name Servers

[fdragon]

Most registrars will provide you the ability to run at least 2, and usually more name servers (I think 6 is the limit). By using this fact, and the the fact that a client will request dns and use the first authoritive response it gets we can impliment something like the following.

Colocation facility 1 machine gets named "DNS1.domain.com" and is a reverse proxy to your real site. Colocation facility 2 machine gets named "DNS2.domain.com" and is also a reverse proxy to your real site. Add cache content sharing between these two servers for extra availability.

You will also be adding DNS servers to each one of those colocated servers. They run as masters (not slaves). The contents of the zones will make each server the single point of contact for your content.

With this setup the following happens when users request your content :

Browsers requests DNS lookup.

Client name server queries all the DNS servers for that domain for the request. First response wins.

Browser contacts your colocation server for content.

Colocation server checks its cache of your site.

if content does not exist, it will ask the cache partner for content, and then will query the real site.

Real site serves content to the proxy server at a much reduced rate.

I've done it

[crmartin]

... for a VoIP project. It's a really stupid way of getting very high availability, but it can be made to work, and it is cheap to implement.

Basics are:

(1) you need a heart beat to confirm the master machine is running.

(2) You write a simple script using dnsupdate(8) that removes your master and inserts the backup.

(3) You look up the special magic to tell DNS caching to flush on other machines.

Don't use DNS failover.

[Harik]

more then one large company enforces a minimum TTL to cut down on outbound lookups. Notably, AOL clients keep hitting the old address up to 24 hours after the switchover. Other ISPs/firewalled companies do the same.

Then again, if it dosn't matter to you, don't worry about it. Just do RR-DNS and manually cut out the failed IP. "most" people will get the still-working servers.