Testing Network Changes When No Test Labs Exist?

vvaduva writes "The ugly truth is that many network guys secretly work on production equipment all the time, or test things on production networks when they face impossible deadlines. Management often expects us to get a job done but refuse to provide funds for expensive lab equipment, test circuits and for reasonable time to get testing done before moving equipment or configs into production. How do most of you handle such situations, and what recommendation do you have for creating a network test lab on the cheap, especially when core network devices are vendor-centric, like Cisco?"

Comments

Document and test at night

[jdigriz]

Step 1) Make a formal request for the test lab. Make it as detailed as possible. Explain the impact to business if various components fail. Make a plain-language executive summary calling out risks. step 2) Once the request is denied, make sure you have a paper trail of the rejection step 3) If possible test network changes on the production equipment at 2am so that impact on users will be less step 4) Once the inevitable failure occurs, haul out the paper trail and get the bean counter fired. Repeat until test lab is approved. Note, step 4 may get you fired instead. Business decisions are somewhat nondeterministic.

My last resort

[tchdab1]

I call my buddies at RIM and test my mods on their system.

Re:The tag says it all

[DigiShaman]

Not all changes are a one-way trip. Having a rollback plan is also important. Should something very unexpected happen, be prepared to roll back any and all changes to undo what has just been done.

Tools

[Tancred]

Here are a few tools:

GNS3 - http://www.gns3.net/ - free network simulator, based on Dynamips Cisco emulator
Opnet - http://www.opnet.com/ - detailed planning of networks, from scratch
Traffic Explorer - http://packetdesign.com/ - plan changes to an existing network

Re:Pretty simple, really

[symbolset]

Oh, no. We do this all the time. Around the holidays we rewire the production server racks so their ethernet cables droop over the aisles, so we can hang up Christmas cards. Jimmy has a script that blinks the blue UID lights for a festive holiday display.