Slashdot Mirror
Minimizing Downtime When Switching IP Addresses?
GeekTek asks: "As we all know, prices for co-location have plummeted since the height of the dot.com era. We've been shopping around and found a solution that works for us. We have a small setup of about a dozen Debian boxen, a few Windows servers and we run our own name servers (BIND 8.x). Most of our domain names are managed through our OpenSRS account. My concern is switching all of our server's IP addresses. I can not have any down time and I want to minimize the number of trips to the current co-lo (it's >2 hours away). What is the best way to do it? What experiences can you share in similar situations?"
if it's just http, you can use redirects to mitigate the dns delay. https complicates this since you'll have to get certs for your temporary dns names (i.e., https://old/ redirecting to https:/new/, which also responds to https://old/)
if it's just about anything else, you might consider setting up a vpn between your existing datacenter and your new datacenter, and setup both environments to answer to the same dns names and have your old environment tunnel to your new one.
in short, doing this with zero downtime is highly dependent on the services you are running. it's not a generic problem that has one solution.
There aint no pancake so thin it doesn't have two sides.
Then when you're done moving a machine, change the IP in DNS. When it seems solid, you put the TTL back to a reasonable (1 day) number.
During the transition, you can also keep a machine at the old IP address and forward the services to the new IP address using tools such as rinetd or xinetd. This assures that you have all traffic going to the correct machine (possibly through the old machine) but that the old IP address is available during the move for clients that have broken DNS resolvers that don't correctly honor DNS TTL values. The rinetd/xinetd purpose machine can easily be a temporary box, such as a laptop - it's not doing any real processing.
If you're also moving your DNS machines, move one a week before the big move, update whois, and make sure everything settles down. Then move the other a day or two after the big move.
Sticking feathers up your butt does not make you a chicken - Tyler Durden
Could you setup redundancy? Setup your DB to from colo to old place? Web pages/apps need to be duped.
Now mind you, if the latency between the two would have to be low enough that your DB doesn't choke doing redundancy/syncronization between your two sites.
If that can be setup, just change your namesevers to point to the new colo and watch the traffic transition over. Once your old site hits no traffic, you are done.
There are prolly better ways.. maybe just plain other ways.. but this is off the top of my head.
-
ping -f 255.255.255.255 # if only
Start reducing your DNS time-to-live. If it is now a week, set it to a day. A couple of days ahead set it to an hour. A few ours ahead set it to 5 minutes (do consider your traffic and make sure that you won't swamp your DNS server - if you are planning a move at night then load should be low and even setting ttl to a minute or two should be fine). Note: there are broken name servers out there and you will get requests to the old address for many hours. Also most browsers don't seem to ask based on DNS TTL so they may not get the new IP until they at least leave your site or perhaps close their browser.
(Note: I'm assuming you have duplicate equipment since that's the only way to physically move with no downtime unless your configuration allows you to remove half of your stuff and still keep running.)
Depending on your needs and current design you can also play NAT/Proxy games. Ie. set up a proxy server or use NAT to make your old IP contact your new servers to catch all the misdirected traffic until DNS propogates.
Last couple of times I did this it was fun to watch. I pulled the trigger on the DNS and could watch the load flow from the old to the new site (we were in the top 500 sites in traffic and did the move during the day so there was a statistically valid sample to work with).
~~~~~~~
"You are not remembered for doing what is expected of you." - Atul Chitnis
Second place is just the first loser.
I've always wondered why host (A) entries are not treated more like MX entries.
Why can't you set up a priority and if IP #1 (priority of 100) isn't available go to IP#2 (priority of 200).
This would be helpful for not only the poster's scenario but if you have a backup ISDN line or DSL for your primary T1. T1 goes down and the client knows to check your secondary DSL IP add'y to see if it can contact it.
Hrmmm.
Is there any way that your IPS can do some GRP routing to forward requests to the current IP to the new co-located address? I'm not all that familiar with GRP.
When I changed colo's I moved my nameservers first. A week later I tarred the home directory's, dumped the mysql databases, and changed the IP's in DNS. Finally, I changed the mailserver IP's. If you're using qmail you can make all of the mail that hits your old server forward to the new one by adding the new IP to the smtproutes file in /var/qmail/control/smtproutes.
Having done this, I can tell you that the simplest, most painless way is to mirror the machine, point DNS from the old location, and change your DNS IP's at opensrs. Once traffic stops flowing at the old colo center, down that machine, and move it to the new noc. Then use your mirror box to do the same to another machine. We did it with Linux (rh 6.2-7.3). Windows may be more difficult but I don't know anything about that.
We did all this when moving from 4 T1's to a DS3, last year.
Nobodies Prefect
Tidbits for Techs Technology Blog
Decrease the TTL of the DNS records during the switchover. If your current TTL is a day, then at least one day earlier, change it to, say, 300 (5 minutes). You'll experience a higher DNS query rate during that time, but probably nothing you can't handle.
Actually, you can reduce the DNS query rate by continuously setting the TTL to about half the time until the switchover. For instance, 24 hours before the switchover, set it to 12 hours. Then keep decreasing the TTL until it's down to about five minutes. This way, you won't get a continuous flood of DNS requests during the day before the switchover.
Will I retire or break 10K?
Use DHCP for server addresses instead of static IP.
Even though my home network is only a two, sometimes three machines, I administer IP addresses through DHCP. The server has a static IP, everything else gets its IP served from DHCP, with a static MACIP mapping. My DNS is on the same machine.
For your situation, switch the machines to DHCP at the old location, and have everything running. You would need a temporary machine to act as the DHCP/DNS machine at the new location. When you move your machines, they should simply come up. Watch out for hardcoded IPs in other configs.
I presume your servers are on a DMZ, and you could arrange one machine as a DHCP/DNS server. Heck, a WalMart $200 box could more than do the job.
The living have better things to do than to continue hating the dead.
- Configure a duplicate server (Or pull one of your redundant ones off a rack)
- Lower your TTL. 10 minutes is good.
- Drive it to the new location. Configure, test.
- Change your DNS server to point to the new location.
- Contact internic. Move your DNS servers to the new IPs. (Change the Host records rather then all the NS records of each individual domain.
- After a few days, shutdown everything on the old block and drive it to the new location.
Simple, easy and effictive. Probably skipped steps that are fairly obvious (Like configuring a box to answer on the new nameserver addresses, etc)--Dan
...I suggested he change his old website to forward to the new server's IP. It worked just fine, and held everything together while everyone's DNS updated. After a week he was able to take down the old site.
'Course if the old IP is completely dead, you've got problems. If you're physically moving the server, then I'm sure you can dig up an old 486 to run Apache on as a redirect.
Pay them. We techies need jobs, damnit. It's bad enough so many of us are giving our code away for free. Now we need to give free consulting too?
Every site is different, with different services, operator skill sets, requirements, demands, and cash.
Lay out your priorities. You say "everything has to stay up" - maybe that's true, but I moved a rather large commercial site stuck in one colo elsewhere, in pieces, when we had a *lot* of money, and when cost analysis started being done, it turned out we could afford downtime.
Look at your traffic records, worry about what has to be up and what doesn't. Think *hard* about dependencies.
Perhaps you can afford two trips (which is what we did), in which case, you move a skeleton crew to the new site (pre configured and tested, of course) , switched DNS (you did think about your TTL, yes?), waited for it to be picked up from a site I knew had not cached the DNS, and completed the move.
Perhaps you can buy/borrow from the office/use spares (but be careful about occupying your spares!) for the move.
Perhaps you can offload the bulk of your traffic elsewhere (Akamai or something to move the demand on machines off the machines while you're doing it.)
I can't speak to your situation, but there's always a way to make it work - like I said, it is pure operations. Analyse, plan, plan again, execute.
More hints -
- Before you're slouching in the colo breaking down the network, copy all data where it needs to be from the comfort of your office. Doublecheck you got it right.
- when disassembling equipment, label all interconnects, in order, unless every box is flat on a local net, with nothing hanging off of them. Don't forget routers, and don't assume it's stupid to label something obvious. Assume you're going to be brain dead when you put it back together - if something unexpected happens (someone flips the truck?), you will be brain dead. And even if you're not, it does help, esp. with messy SCSI configs, etc.
- Write out a timeline, and give yourself more time than you need. Make sure other people concerned know what it is.
- Oh, _back up your machines_. I know, it is obvious, but I know of one company that screwed this up royally.
- Bring one more person than you need. They might be helpful, and if not, they can at least fetch coffee and donuts when you need them.
- Bring snacks, lots of them.
- Convince your accountant insurance is worth it, if they don't belive it already. We were moving ~2M worth of gear, and I would have been even more freaked out than I was if we hadn't insured it while it was being transported.
- Have a wad of company cash/credit card on hand. You never know what comes up.
- Ditto for spares, whatever you can - is that disk that's been spinning for 4 years going to come back up? Cat 5?
- If you have heavy gear, think about whether or not you're going to move it yourself.
- Overplan it. You'll be glad. Think contigencies and fall back positions.
- Make sure your staff is well rested before you do it, and that they have whatever they need before you start.
Hope this helps.
-j
I forget what 8 was for.
1) Mirror both servers.
2) Change the DNS
3) When the DNS updates you're done.
4) Pat self on back.
5) Get back to work you damn code/admin monkey.
(This is how it worked at my job.)
No, I'm not talking about a DNS setting.
It's a proven fact your lifespan will actually decrease due to the stress involved in moving a network's IP address and the debugging that goes along with it.
"And like that
There is something else which I wanted to ask here. What about those who buy the web accounts from other webhosting company for your company?
How you deal with lot of different domains that is registered though various registars plus moving them to the new DNS Server? That would introduce lot more complexity than just simple change of the IP Addresses and DNS records. We talking about major movement of all Domains from one DNS Server to new DNS server plus new IP address allocation. What would you do to ensure the smooth movement of those?
-- Amazing how the Internet still humms along.... -- Dispite all the flaws of Micro$oft in their software!
Find out what slashdot's admins did the couple of times they moved their servers.
Then don't do that.
But then again, I could be wrong.
Remember this puzzle? Find the answer, and then start moving your servers!
The farmer (you) is taking his fox (servers), duck (DNS) and corn (customer web space) to market (colocation provider). He is currently stuck on the left bank of a major river (highway). The good news is that he has a boat available (pickup truck). The bad news is that the boat will only hold him and one other item for each crossing (changing over servers without service loss).
He dare not leave the duck alone with the corn as the corn would get eaten (no servers for DNS), or the duck alone with the fox as the duck would get eaten (no customer web space while swapping drives?). Also the farmer knows from prior experience that he cannot leave the corn alone on the right bank of the river (old location) since a large flock of crows (customers) is waiting to devour it (and you).
Can you help the farmer get everything across the river safely? (Ask Slashdot can!)
...
To get around this, there are two scenarios:
1) Use outside nameservers as your authoritative servers for your domain. You may even be able to get your registrar to do this. Some registrars offer it as a feature and others may charge. In any case, having a separate set of nameservers means you can move from colocation facility to colocation facility with relative ease as mentioned in earlier posts.
2) Set up two servers at the new colo facility as DNS servers and set all of your TTLs etc to the desired values. Registers those IPs as nameservers with Network Solutions (you may be able to do this through your registrar). Then change the IP numbers of your nameservers for the domain names. Wait 48 hours for total propagation and proceed as has been outlined in previous posts.
Please note that you really should contact your registrar and find out what the proceedure is for changing the IP address of a nameserver. I know that in the past when we had to do it, there was a template sent to Network Solutions specifically for this task. This is most likely easier now and probably different for each registrar.
two days ago, but only with my one server which is co-lo'd by a friend of mine who is an ISP. They simply set up routing so both the old and new IP adresses could reach my machine and then I only had to set up a second IP adress on my ethernet adapter like this:
;-)
/etc/sysconfig/SuSEfirewall2 and running "rcSuSEfirewall2 restart"
;-)
ifconfig eth0:1 123.45.67.89 netmask 255.255.255.224 broadcast 123.45.67.255
That's it. Now the people at my friend's company have set up the DNS to report the new IP adress and let it propagate through the 'net. One hour later or so all my domains targeted the new IP adress, everything went fine, with zero downtime.
The best is: everything was done through ssh, I didn't had to move my lazy ass
Only one pit to be aware of: don't forget to tell your firewall ! In my case it was simply adding eth0:1 to the list of firewalled interfaces in SuSEs
Now that everything works I could kick out the old IP adress and stuff... but I'm lazy
I've only dealt with this problem on personal sites, but I'd recommend outsourcing DNS for at least a few weeks before and after your move.
I've been real happy with zoneedit for DNS services. In fact, from looking at the identical sign-up forms on the Verisign and Zoneedit websites, I think Verisign is reselling their services (not that what those guys do should necessarily be an inspiration to anyone).
This might help to eliminate one very hairy variable from your already complex equation.