Transitioning Major Commercial Networks Between Providers?

Kalon asks: "I am the network manager for a large business park connecting to our countries largest telco/ISP. We've recently negotiated to move our Internet services to another major provider and I'm stuck with a problem: our range of IP addresses (8 Class C's) is half 'owned' and half 'leased' from of our current provider. With some 90 companies relying on our connection for continuous Internet uptime, how can I transition forward and reverse DNS services for those companies with websites/mailservers unlucky enough to be on leased addresses, and route mapping for those whole blocks I own? Considering I don't have APNIC membership, what tips do you have to speed or ease the transition, considering I have to deal with lumbering telcos who won't play nicely together, and 90 different companies all demanding Internet presence?"

Comments

Get into the fetal position and cry

[leviramsey]

...cause you're in for a hell of a difficult time.

Buy 4 more Class C

[lewiscr]

Buy 4 more Class C subnets, and transition the people on the leased subnets now.

Then when you move, its just a matter of updating the routing.

Of course, changing the subnet routes could be another problem. A lot of the big routers don't propigate routes for anything smaller than a /19 subnet. I could be wrong about the size, but a /24 is right out. If your 4 existing subnets are all together, you've got a /22, and that might just work. Getting the 8 subnets together for a /21 would be better.

Set the TTLs LOW

[netringer]

As you get close tot the cutover time, set the TTLs (Time to live) on every DNS record you can get your hands on to a very low value.

Any host that does a lookup can legally only cache the answer as long as the TTL time. After that it has to ask again. With a low value you get a lot more requests to the DNS servers, but the host requesting will know about the change quickly so it'll find your servers at the new address as it changes.

The value is in seconds. 3600 would be an hour.

DNS and NAT

[funky womble]

If you have use of the old addresses for a little while after the new addresses are available, maybe you could run a NAT box on the old addresses for a short while so that inbound traffic could be directed to the new addresses. Maybe something like this -

1. setup the NAT box
2. change IP addresses on a machine
3. add NAT rule to direct traffic to new address
4. if more_machines_to_move goto 2
5. beer/pizza/etc, you earned it (:

Or instead of using a NAT box like that, you could possibly put it in front of customer servers and/or routers, so they could keep the configuration with their old addresses for a while longer, giving you a little more breathing space (obviously you need to update DNS while you're doing that - you might find tinydns, part of the djbdns package, useful as it can control the TTLs, making it fairly easy to expire DNS records at a particular time and replace them with new ones).

I know an ISP that has been putting off renumbering for several years, I don't think their former transit providers are too happy about having part of their own netblocks announced at them :/

Think of it as a challenge and also proof that there are certain (fairy limited) circumstances where NAT doesn't completely suck!

been there...

[chrismcc@netus.com]

Hello...

I've done this. It isn't fun, but it's doable.

quick options
1: use two providers
you get to keep your address
you have redundancy

2: hire someone that has done this before

long solution:
get your own address from arin or suitable authority. with 90 companies connected, getting a /20, /19, or /18 should not be a problem

bring up second connection. establish BGP sessions with both providors announcing all (old and new) ip addresses.

plan on transitioning a few customers per day. I imagine most of your customers are 9-5ers. They are easy. Save the more critical ones to last as you'll have had a lot of practice.

Using DHCP on their side helps

Co-ordinate with the companies IT person. You update the routers/routes, and they update the DHCP server. Easiest done as they leave for the day.

do not do NAT in the routers. It will kill your performance.

The whole transition could take up to two or three months. Don't get in a hurry. Do it right the first time.

Setup DNS servers on both old and new ip address blocks.

Consider keeping the old connection (maybe at a lower bandwidth) for redundancy. or kill old provider connection.

You're joking right?

[anthony_dipierro]

You're the "network manager for a large business park" and you don't know how to do this? I suggest you go to your boss right now and tell him to fire you.

Some more details...

[trims]

First off, all the suggestions in the post above by chrismcc (here) are excellent. I am going to expand on them a bit.

First, you really should have an AS number and get a dedicated IP allocation from APNIC - it will be immensely useful, and save you a whole bunch of money and headaches in the future. There are some upfront fees (I just checked, and a /19 (32 class C) runs about $8192, with an AS number cost $500, both of which are one-time fees, and you don't have to be an APNIC member), but they're well worth it, and not excessive.

After you get the AS and IP block, talk to your new ISP and your old ISP. Advise them that they will need to be advertising your AS now.

If you don't already have it, upgrade all your border routers to support BGP, so you can actually use the AS number, and also so you can potentially use multiple different ISP pipes for redunancy and load balancing. Depending on your setup, this might cost some money. However, given your setup, I suspect that you don't have anything more than a Cisco 3600-series border router, which is fine (and the BGP upgrade isn't very expensive).

Using the new APNIC address block and AS number, number all your network equipment with the new IP space, keeping the original IPs from the old ISP in place.

Have the new ISP begin advertising the AS number and your new IP block. Do some testing from outside to make sure you can reach all segments of your network.

For all important machines and equipment (primarily servers, but stuff that generally retains a static IP), give them a new IP. Almost everything supports virtual interfaces now, so it's trivial to have two different IPs assigned to the same machine these days.

Do some more testing, to check that you can reach these machines via the new IP (do both internal and external testing, as required).

Update DNS to include both the new and old IP for all assigned machines. Change the TTL to something VERY LOW, like an hour or so.

Test DNS starting the next day, and do repeatedly for the next 3-4 days.

Update any servers providing DHCP or BootP or similar dynamic service to provide IPs from the new block. Continue testing.

After about a week of running both the new and old DNS entries in parallel with all machines assigned a NEW IP address, consider removing several of the DNS entries for the old IP space. Do this, and check to see if anything breaks. Fix it.

Remove all old IPs from DNS. Wait a week or so to see if anything lingering breaks, and see if you get any reports from outside as to problems.

Remove all of the old IP numbers from all machines and network equipment, and notify the old ISP that you have completed renumbering, and that they should change routing so that IP space no longer points to you. Inevitibly, you probably will run into something you missed. Fix it.

Ditch the old ISP, or keep them for redundancy/load balancing, but make sure they're publishing your AS number and new BGP information.

Many of these steps above will require coordination with your client companies. However, if you do it right, there should be NO DOWN TIME, and the transition will be transparent to your clients.

-Erik