Testing Network Changes When No Test Labs Exist?

vvaduva writes "The ugly truth is that many network guys secretly work on production equipment all the time, or test things on production networks when they face impossible deadlines. Management often expects us to get a job done but refuse to provide funds for expensive lab equipment, test circuits and for reasonable time to get testing done before moving equipment or configs into production. How do most of you handle such situations, and what recommendation do you have for creating a network test lab on the cheap, especially when core network devices are vendor-centric, like Cisco?"

Comments

Document and test at night

[jdigriz]

Step 1) Make a formal request for the test lab. Make it as detailed as possible. Explain the impact to business if various components fail. Make a plain-language executive summary calling out risks. step 2) Once the request is denied, make sure you have a paper trail of the rejection step 3) If possible test network changes on the production equipment at 2am so that impact on users will be less step 4) Once the inevitable failure occurs, haul out the paper trail and get the bean counter fired. Repeat until test lab is approved. Note, step 4 may get you fired instead. Business decisions are somewhat nondeterministic.

Re:Document and test at night

[Keruo]

step 3) If possible test network changes on the production equipment at 2am so that impact on users will be less

Been there, done that. Sadly the only way to see how your setup works is to try it in production.
Sure it helps if you can test it beforehand, but sometimes your lab might not reflect what happens in real network when you roll something out.
Just make sure you can clock those am hours as overtime/nighttime work.
And remember to backup the running config twice so you can restore the production network if something goes fubar.

Packet Life

[z4ns4stu]

Stretch, over at Packet Life has a great lab set up that anyone who needs to test Cisco configurations on can sign up for and use.

Re:Virtualization?

[value_added]

Specifically, GNS3 allows you to create test networks in a virtual environment, then import software images for your Cisco routers, switches, PIX firewalls, Juniper hardware, etc, all run on hypervisor technology.

For anyone unfamiliar with GNS3, a link to the website. There are versions available for Windows, Linux, and OS X. FreeBSD already has it in ports.

As a side note, I'd add that maintaining a home lab (to the extent practicable and useful) is one way to side-step limitations of what your employer provides. Consider it a combination of "Ongoing Professional Education" and "Proactive Job Security Measures" (i.e., "I better test this shit to save my ass tomorrow").

Tools

[Tancred]

Here are a few tools:

GNS3 - http://www.gns3.net/ - free network simulator, based on Dynamips Cisco emulator
Opnet - http://www.opnet.com/ - detailed planning of networks, from scratch
Traffic Explorer - http://packetdesign.com/ - plan changes to an existing network

Go virtual!

[leegaard]

If you are unable to recycle old equipment into your testlab you should go virtual.

For Cisco routers, GSN3/Dynamips (www.gns3.net) is your friend. Any recent PC or laptop will allow you to build a large and complex topology that will satisfy most experiments and even support you when doing certification preparation. It will only work for routers so switch-based platforms are out (like the 3570,6500 and 7600). The good news is that the features are more or less the same and they more or less behave the same way. If "more or less" is not close enough you need a replica of your production network or at least a few devices of each to test what can be labelled as critical.

For Juniper routers, google juniper Olive. It will run a juniper router the same way dynamips runs a Cisco router.

In both cases a proactive partnership deal with the vendor will be a good idea. Both Cisco and Juniper (and I am sure all other major network vendors) have programs where they will more or less advise, test and prepare the configurations for you. If you run a critical network this is money well spent.

In the end it comes down to the level of risk your management is willing to take. Ask them if they will allow the network to be less up since you are unable to properly test your changes before implementation.