Slashdot Mirror
Ask Slashdot: How To Evacuate a Network
First time accepted submitter gpowers writes "I am the IT Manager for Shambhala Mountain Center, near Red Feather Lakes, Colorado. We are in the pre-evacuation area for the High Park Fire. What is the best way to load 50+ workstations, 6 servers, IP phones, networking gear, printers and wireless equipment into a 17-foot U-Haul? We have limited packing supplies. We also need to spend as much time as possible working with the fire crew on fire risk mitigation."
Quickly?
Less posting to Slashdot would be step 1...
Pack what's critical first. Servers. Critical networking gear. Workstations. Ignore the phones, printers and wireless gear unless you've got extra time. And good luck.
"But we decide which is right, and which is an illusion"
Nuke it from orbit. It's the only way to be sure.
First thought is put half of them in your own car. Then put the other half in the truck and abandon it in the fire's path. Then eBay.
Take lots of pictures before you unplug your cables. It will save you time when you have to reconnect everything.
The "best" way to evacuate a data center is to already have off-site back-up for your data in place, drop a fresh copy to portable media, and walk out. The hardware should be insured. The life of your and your people (at least some of whom should probably be helping their families evacuate) are far more valuable than a few months of making your insurer pay for rented hardware until your new machines show up.
1. label all hard drives. Drive position, and server.
2. place hard drives in anti-static bags
3. pack drives in foam.
4. get drives far, far away.
Hard drives are both the most valuable, and the most fragile part. Do not load them in a stiff suspension vehicle like a truck, as this bounces the drives. Choose a soft-suspension normal car.
Next take servers and network gear. Desktops are a maybe, as are phones. Ignore printers.
Tape a piece of cardboard over the face of an LCD monitor to protect it from casual bumps.
Above all, no data is worth a human life. No heroics. You're not paid for heroics.
Log off Slashdot before reading this :-)
If you've got labeling stuff around, use it (not fancy label makers, just the basic "Hello My Name Is" and a Sharpie.)
Grab the servers, grab the workstation bodies, grab the phones and anything else that's easily portable, and any backup media you've got. Unfortunately, rack-mounted equipment is usually harder to grab, but that's probably your most expensive and critical stuff. And it'll be your critical path, so start unbolting it first. All of that will fit, put it in first, braced as well as you can.
Monitors and keyboards are nice, but they're just money, not data. Grab a few of them, but leave the rest for last. If you have packing material left, great, but if not you'll just have some breakage. If you've got any CRTs, leave them, they're heavy.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Two years ago we go a call telling us the levies might not hold and if they burst (1 block away) we'd have 8 ft of water. We didn't really have a battle plan and we had a lot less to deal with than it sounds like you did, but we learned some lessons.
1) praise the lord we had good network documentation. Now is not the time to be writing down how the firewall and public and private LANs are plugged in together. Shut stuff down, and start placing network hardware in big plastic tubs. Have tubs handy for this, they nest nicely when not in use. Toss cables in a different tub. just wind them up best you can into loops and toss them in. there's probably not time for neatness, you can deal with that later. TAKE THE DOCUMENTATION WITH YOU. You'll feel mighty silly if that's left pinned on the wall. or I assume you have an electronic copy you can print when you get offsite. Make sure any servers with complex cable attachments (like to phone banks or security systems) have labels on the connectors.
1b) got your phone system documented too? this is a whole 'nother can of worms that often is forgotten about. Does anyone have a diagram of where all those punched down wires go on each block? If you have phone switching hardware to pack, make sure the cables are labeled, they will all probably look the same with the giant connectors that attach to the blocks. "We'll just call Al, he does our phone stuff." Oh, you don't think Al is going to be BUSY helping everyone else that is returning? Nothing's as fun as a 2-3 day wait to get your phones back up and running huh?
2) Label ac adapters. You need to know which unit wants 12vdc and which has 24vac, you don't want to fry stuff when you are trying to reassemble. every pack should have the model of the unit it goes to written on it. Gear WILL get separated from its pack during the evac.
3) label staff's hardware. It's very annoying trying to figure out whose beige box is whose later. and they will probably fight over monitors and keyboards later. save yourself the headache. If you are already under the gun, run to the store and get a dozen rolls of masking tape and sharpies and have the staff label their equipment while you're packing things up, full initials or names, I bet you have duplicate first names you don't want to deal with later. Make sure you label the phones.
4) have a plan for things you can't easily move. the corp office was also forecast to get 8ft of water and they were on the WRONG side of the dike so it was more of a "when" than "if". they had a very expensive multifunction printer that the service people told them they could have a tech out to take it apart (so it fit out the door) in three days, which obviously was silly. They rushed in a bunch of cinder blocks and lifted it up and set it on them 8.5' up. (I have no idea how they lifted it) In retrospect, the building got 14" of water and totaled it, they SHOULD have killed power to the building and took a saws all to a wall. OR at least watertight wrapped it before lifting. I've seen this done with entire cars when faced with an incoming flood or hurricane. Even if it doesn't keep out the water 100%, at least it will keep out the mud, which you may be very grateful later. Got a plan for your big server room ups's? those can be quite large and heavy, and are often hardwired into the AC, are you able and qualified to unhook it? Maybe you should call in an electrician now and change that armored cable to a dryer type plug? Have a place you can move big stuff that can't be evac'd to where it will be at least more likely to survive. Think of flood, fire, and tornado/hurricane, there's probably not one single place that will work best in all three cases. Smoke damage can be very destructive, simply having something wrapped in mover's visqueen may prevent unnecessary loss that the fire missed but the smoke got. Do you have a plan for that rack that's bolted down or won't even fit through the door?
5) Document what's been left behind. A simple way to
I work for the Department of Redundancy Department.
Save your data and your people
but not in that order.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.