Transitioning Major Commercial Networks Between Providers?

Kalon asks: "I am the network manager for a large business park connecting to our countries largest telco/ISP. We've recently negotiated to move our Internet services to another major provider and I'm stuck with a problem: our range of IP addresses (8 Class C's) is half 'owned' and half 'leased' from of our current provider. With some 90 companies relying on our connection for continuous Internet uptime, how can I transition forward and reverse DNS services for those companies with websites/mailservers unlucky enough to be on leased addresses, and route mapping for those whole blocks I own? Considering I don't have APNIC membership, what tips do you have to speed or ease the transition, considering I have to deal with lumbering telcos who won't play nicely together, and 90 different companies all demanding Internet presence?"

Comments

Some more details...

[trims]

First off, all the suggestions in the post above by chrismcc (here) are excellent. I am going to expand on them a bit.

First, you really should have an AS number and get a dedicated IP allocation from APNIC - it will be immensely useful, and save you a whole bunch of money and headaches in the future. There are some upfront fees (I just checked, and a /19 (32 class C) runs about $8192, with an AS number cost $500, both of which are one-time fees, and you don't have to be an APNIC member), but they're well worth it, and not excessive.

After you get the AS and IP block, talk to your new ISP and your old ISP. Advise them that they will need to be advertising your AS now.

If you don't already have it, upgrade all your border routers to support BGP, so you can actually use the AS number, and also so you can potentially use multiple different ISP pipes for redunancy and load balancing. Depending on your setup, this might cost some money. However, given your setup, I suspect that you don't have anything more than a Cisco 3600-series border router, which is fine (and the BGP upgrade isn't very expensive).

Using the new APNIC address block and AS number, number all your network equipment with the new IP space, keeping the original IPs from the old ISP in place.

Have the new ISP begin advertising the AS number and your new IP block. Do some testing from outside to make sure you can reach all segments of your network.

For all important machines and equipment (primarily servers, but stuff that generally retains a static IP), give them a new IP. Almost everything supports virtual interfaces now, so it's trivial to have two different IPs assigned to the same machine these days.

Do some more testing, to check that you can reach these machines via the new IP (do both internal and external testing, as required).

Update DNS to include both the new and old IP for all assigned machines. Change the TTL to something VERY LOW, like an hour or so.

Test DNS starting the next day, and do repeatedly for the next 3-4 days.

Update any servers providing DHCP or BootP or similar dynamic service to provide IPs from the new block. Continue testing.

After about a week of running both the new and old DNS entries in parallel with all machines assigned a NEW IP address, consider removing several of the DNS entries for the old IP space. Do this, and check to see if anything breaks. Fix it.

Remove all old IPs from DNS. Wait a week or so to see if anything lingering breaks, and see if you get any reports from outside as to problems.

Remove all of the old IP numbers from all machines and network equipment, and notify the old ISP that you have completed renumbering, and that they should change routing so that IP space no longer points to you. Inevitibly, you probably will run into something you missed. Fix it.

Ditch the old ISP, or keep them for redundancy/load balancing, but make sure they're publishing your AS number and new BGP information.

Many of these steps above will require coordination with your client companies. However, if you do it right, there should be NO DOWN TIME, and the transition will be transparent to your clients.

-Erik