Quickly Switching Your Servers to Backups?

moogoogaipan writes "After a few days thinking about the quickest way to bring my website back to the internet users, I am still stuck at DNS. From experience, even if I set the TTL for my DNS zone file as low as 5 minutes, there are still DNS servers out there won't update until a few days later (Yeah. I'm looking at you, AOL). Here is my situation. Say that I have my web servers and database servers at a remote backup location, ready to serve. If we get hit by an earthquake at our main location, what can I do in a few hours to get everyone to go to our backup location?"

Comments

BGP

[ckdake]

Same Provider at both (N) locations, Same IPs for servers/services, Just don't advertise the prefixes via BGP from the backup location until the primary one goes down.

Re:BGP

[georgewilliamherbert]

Bingo.

This is exactly what BGP (or OSPF feeding in to your providers' BGP) is made for.

If you're big enough, you can get your own AS number and do this without having the same provider at each end (useful if the disaster that happens is that the software on all of provider X's core routers goes insane all at once, which happens from time to time).

DNS just can't be assumed to fail fast enough for very high reliability services. You can do DNS right... low TTLs and all... and some providers just cache the results and do the wrong thing, and some client systems will never look up the changed data if the old IP stops responding, until someone reboots the browser or workstation.

BGP.

Server Clusters

[Tuoqui]

NLB (Network Load Balancing) Cluster, link the two together and have them both serve the website. Not only will it not go down (barring freak accidents like both locations being hit at once) but it will also have the added benefit of presumably double the bandwidth and such.

Only problem is if you're locating them in two separate locations that they need to be able to communicate with each other and keep identical copies of the website and be able to connect to any databases you may need.

Basically any server clustering type setup if you can connect the two remotely would probably be a good starting point for your website assuming it is that important that it dont go down ever.

Scale back your expectations

[eln]

You could spend a bundle of money doing global load balancing and maintaining a full hot spare site, or you could figure out how critical it really is that your website be up within 5 minutes of some major disaster like an earthquake.

In the event of a major disaster, the need for "immediate" recovery is actually defined as being able to be back up and running within 24 hours of the event. This is true even for business critical functions. Unless your business would cease to exist within 24 hours if your website went down, I would consider a 72 hour return to service to be perfectly adequate, and it would cost a whole lot less time and money to set up. Keep in mind that we are talking about an eventuality that would only occur if your primary site was entirely disabled for an extended period of time, which is highly unlikely to happen if you're hosted in any kind of modern data center.

DNS failover

[linuxwrangler]

We have used DNS failover from dnsmadeeasy.com for a couple years and have put it to the test a couple times. They have had perfect reliability and a low cost (typically well under $100/year).

The method is not perfect, but it is plenty good enough for our needs to protect against something that takes a datacenter down for a prolonged time (several minutes/hours/days). And the price

And to those who recommend avoiding "disaster prone" places: they all have people. People like the backhoe guy who took out the OC192 down the street. Or the core drillers who managed to punch both the primary and secondary optical links to a building of ours at a point where they were too close to each other.

You can roll your own by having a DNS server at each site and DNS 1 always issues IP of server 1 while DNS 2 always issues IP of server 2. But there are a number of issues like traffic hitting both sites at the same time. And you will have to detect more than just a down link so you will be scripting web test and DNS update systems. By the time you are done, you will have spent decades' worth of dnsmadeeasy fees.

Note: dnsmadeeasy isn't the only game in town. Just the one we happen to use.

Re:DNS failover

[Joe5678]

dnsmadeeasy doesn't solve the problem the OP is asking about. They simply monitor your services and start serving a different DNS record if your primary is down.

The OP is concerned with all the DNS servers that aren't yours that would then have a cached version already, and continue to serve up the dead DNS record until their (incorrectly configured) TTL expired.

As another poster already mentioned, BGP is really the only technical solution to this problem. All other "solutions" are going to be convincing people that they don't really need instant failover in the event of a major disaster.

Re:DNS failover

[mother_reincarnated]

Disclosure time: I deal with this stuff every day and I have a vested interested in commercial solutions to this issue.
dnsmadeeasy doesn't solve the problem the OP is asking about. They simply monitor your services and start serving a different DNS record if your primary is down.

The OP is concerned with all the DNS servers that aren't yours that would then have a cached version already, and continue to serve up the dead DNS record until their (incorrectly configured) TTL expired.

As another poster already mentioned, BGP is really the only technical solution to this problem. All other "solutions" are going to be convincing people that they don't really need instant failover in the event of a major disaster.

The problem is that this is somewhat specious of a concern. It tends to be browser/os combos that don't honor TTLs, and there are very few of those around anymore. In that case all the client has to do is close and reopen their browser.

BGP is cumbersome but a valid solution (if you've got your own AS). You probably want to suppliment it with a DNS based solution though- far more of your outages will be less than whole site events. The more viable (and likely) scenario to keep in mind is a partial outage at a the primary DC where you can use DNS to send 99% of the users of affected applications to the DR site, and the 1% who are non RFC compliant can be handled via more draconian methods (either 302's, l3 backhaul, or even shared l2 if you have it between sites). A properly designed solution will be able to do all of this for you and also know what is really available without manual intervention.

I really want to address your comment about convincing people they "don't really need instant failover in the event of a major disaster." Well they probably don't [need 100% of users to have instant failover during a major disaster]. Metrics get wrapped around BC for a good reason. People often fail to grasp is that if your entire primary site goes 'poof' you, and likely most of the world, will not notice the time it takes either a DNS or BGP based solution to 'reconverge,' and you certainly won't care if a fractional percent of your users need to close their browsers (btw many users probably won't be able to get to your site because of route flapping mayhem et al. caused by said disaster...) Frankly it won't be anyone's biggest concern at that point.

Layered approaches are often needed, your particular requirements and bugdet determine how deep you go. A DNS based first layer will do 99%+ of the job.

[FYI for dealing with the bad boy superproxy types like AOL you could segment that traffic out and handle it differently than everyone who 'plays fair'... If you've got the money to play with the best of breed solutions then it's all been done before- but you probably wouldn't be asking /.]

Re:Location

[bahwi]

I've got my personal/small business server(does nothing but a crappy webpage) so it's not critical down in florida. It's gone down because of router issues but never a hurricane, and oh yes, it's been hit. I actually think those places may be better as they are built to weather those types of storms.

Yeah, I've heard lots of people sweating and panicking because a back ho was working somewhere near the datacenter. On site and beads of sweat on their forehead.

Buy "Scalable Internet Architectures"

[tedhiltonhead]

For a real answer, buy Theo Schlossnagle's book, "Scalable Internet Architectures". Theo presented a lengthy and highly-informative session at OSCON last year, and I subsequently bought and read his book. Worth every penny if you're professionally involved in providing reliable Internet services of any kind.

Re:geographical load balancing

[passthecrackpipe]

F5 mainly uses DNS for its Global Traffic Management solution. There are other bits and pieces, but that is the core, really.

Try asking at webhostingtalk.com

[Wabbit Wabbit]

The industry pros discuss this sort of thing there all the time. The colocation sub-forum would be the best place to ask. I know that sounds odd, but that's the area on WHT where the best network/transit/BGP people hang out.

Back in the day I used Exodus to do this

[ejoe_mac]

When $ is no issue, a tier 1 colocation provider with their own services would be the best option. They've got big pipes, and will work with you to have the additional services needed. I'd go as far to say that you're going to want to have a failover script that they would follow in the event of site A going offline. You'd need redundant equipment, or use a DR firm for getting back up.