Slashdot Mirror
Ask Slashdot: How To Evacuate a Network
First time accepted submitter gpowers writes "I am the IT Manager for Shambhala Mountain Center, near Red Feather Lakes, Colorado. We are in the pre-evacuation area for the High Park Fire. What is the best way to load 50+ workstations, 6 servers, IP phones, networking gear, printers and wireless equipment into a 17-foot U-Haul? We have limited packing supplies. We also need to spend as much time as possible working with the fire crew on fire risk mitigation."
Quickly?
Less posting to Slashdot would be step 1...
Pack what's critical first. Servers. Critical networking gear. Workstations. Ignore the phones, printers and wireless gear unless you've got extra time. And good luck.
"But we decide which is right, and which is an illusion"
Nuke it from orbit. It's the only way to be sure.
First thought is put half of them in your own car. Then put the other half in the truck and abandon it in the fire's path. Then eBay.
Basically you're fighting, not avoiding. Relocate. Avoid. Cheers!
I come to Slashdot only to read sigs. One you are reading is mine.
Damn if I'd go in to work to remove hardware when a fire is threatening.
I'm not paid enough to risk my life. Period.
Take lots of pictures before you unplug your cables. It will save you time when you have to reconnect everything.
The "best" way to evacuate a data center is to already have off-site back-up for your data in place, drop a fresh copy to portable media, and walk out. The hardware should be insured. The life of your and your people (at least some of whom should probably be helping their families evacuate) are far more valuable than a few months of making your insurer pay for rented hardware until your new machines show up.
First, triage the equipment.
You likely do not have time to pull disks from systems, so pack computers and
external drives first. Get blankets to protect things. Blankets start at the bottom
to act like a shock absorber.
Things like networking gear and wireless stuff is irrelevant compared to the
computers, and probably lighter. If you CAN, sure, save all that stuff too.
But the data comes first. Don't forget backups.
If there are computers with really really important or sensitive stuff, put
those in someones car in the backseat, again with blankets. If I seem
blanket obsessed, it's because I've found them to be available quickly
either from individuals or stores. Yes, bubble wrap or sorbathane would
be better but you aren't likely to have that stuff lying around.
Not trying to be mean, but you should have already had a plan in place...this late in the game without a plan means you just have to go with asses and elbows (just get what you can while you can and forget trying to install a plan to do it!)
The good news is you can become an example for other IT people! Everyone should look at their disaster plans and make sure you have accounted/planned for all emergencies that may happen in your area.
Whatever stores data first -- if it's a SAN, then your RAID chassis and metadata controllers, and if you have time, the SAN fabric switches and cabling, but you can replace the latter if you have to, and if it's ordinary SAS, the servers if they're all internal storage, or the RAID chassis or whatever's external. Definitely grab any non-offsite backup media with that. Rest of it in descending order of priority after you grab the most valuable stuff, mostly to avoid having to replace it.
Best strategy overall is to think "what if we had to abandon this evacuation mid-process and run?" Try to have what you most want already in the truck at any given moment, and concentrate on data before hardware -- the data is far more valuable in most cases.
If you haven't done an offsite backup, for god/dess' sake do one *now* and get the backup media to a safe location .. :/
1. label all hard drives. Drive position, and server.
2. place hard drives in anti-static bags
3. pack drives in foam.
4. get drives far, far away.
Hard drives are both the most valuable, and the most fragile part. Do not load them in a stiff suspension vehicle like a truck, as this bounces the drives. Choose a soft-suspension normal car.
Next take servers and network gear. Desktops are a maybe, as are phones. Ignore printers.
Tape a piece of cardboard over the face of an LCD monitor to protect it from casual bumps.
Above all, no data is worth a human life. No heroics. You're not paid for heroics.
Many servers and disk arrays specify that drives have to be shipped separately simply because their mass is so high that the rack doesn't pass shock and vibe tests. So either pull out the drives (mark which slot they go into) and pack them separately in bubble wrap or, if that's not an option, put as much cushioning as you can around the servers and strap them down so they don't bounce. In any case, be prepared for some disk drive damage or degradation.
thegodmovie.com - watch it
Log off Slashdot before reading this :-)
If you've got labeling stuff around, use it (not fancy label makers, just the basic "Hello My Name Is" and a Sharpie.)
Grab the servers, grab the workstation bodies, grab the phones and anything else that's easily portable, and any backup media you've got. Unfortunately, rack-mounted equipment is usually harder to grab, but that's probably your most expensive and critical stuff. And it'll be your critical path, so start unbolting it first. All of that will fit, put it in first, braced as well as you can.
Monitors and keyboards are nice, but they're just money, not data. Grab a few of them, but leave the rest for last. If you have packing material left, great, but if not you'll just have some breakage. If you've got any CRTs, leave them, they're heavy.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
thinking this through, your priority is to make it as quick as possible to get essential gear out the door. so, consider:
* having the servers and desktops *already* in easy-to-carry crates, with large handles on the outside and packing materials surrounding the machines.
* have the machines stacked off the ground so that people don't have to waste time bending down and possibly injuring themselves by jolting weight that's too much for them
* have all essential equipment nearest to the doors plural, prioritised by criticality
* yes doors plural: add an extra door next to the existing one (or replace the one door with easy-to-open double-doors with those pushable handles) so that at least two people side-by-side can get through at once, carrying the crates, and can just "barge through them" rather than having to twist the handles.
* make sure that the crates are stackable and sturdy but also light enough to carry!
* even consider having the machines already loaded onto 4-wheeled trollies and left on them, permanently.
* if time is _seriously_ critical, consider putting guillotines next to all cables (and test them) so that people don't have to waste time unplugging cables: just cut them and go - but only consider this if the guillotines are sharp enough and easy enough to operate, and only if it's seriously seriously critical to save seconds. don't put power cables through the guillotine though!
* consider getting convenient light-weight but sturdy cabinets made for all LCD monitors, with double doors that fold back 180 degrees out of sight, and a top (with a handle) that locks automatically when it's flipped over. have the LCD monitors mounted onto the cabinets with rubber bushes so that they don't need to be placed or positioned into the cabinets - just pull out the cables, shut the doors, slam the top over and pick it up by the handle: done.
* consider getting 12v powered LCD monitors instead of 240v/120v AC mains, so that the power cables can be guillotined rather than pulled.
* instead of guillotining, consider breaking all the tabs on the network and telephone cables (the ones that "click and lock") and affixing them *loosely* with gaffa tape to all devices (network hubs, machines etc.) - this way it will be possible to just pull (hard) and out pop the cables. or, if someone forgets, and gets to the end of the wire, they won't trip or be yanked backwards: the cable will just come out, clean.
* get 4-port hubs instead of 8, 16 or 24-port. 4 gaffa-taped cables are easier to pull out than 8, 16 or 24, and if one of the 4-port hubs is lost to a fire, so what, big deal. a 24-port hub however starts to get expensive.
* stop people from putting the bloody screws in the bloody cables - you know the ones: parallel ports, VGA cables, serial cables etc. the ones that are always bloody irritating when it comes to fixing or moving a machine and you find that the bloody VGA cable needs a bloody screwdriver to remove the damn thing. take the screws *OUT* of the cables; that way people can't go "oh look: screws - let's tighten them".
so - yeah. make it easy to just shift everything. have practice drills. set a deadline (say 1 minute) and see how much kit people can get out in that time, without damaging it.
oh - and you know how i suggested making it easy to shift everything? uh... make sure the insurance is up to date, and get good security. no point making it easy for *other people* to shift all that expensive gear, eh? oh. and sort out some off-site backups, eh? :) i use rsync; my friend uses backuppc (because he has a lot of machines). /peace
Walk out with your backups, and save the people first.
the problem. Like the other people say here, load up the most important machines first, pack them with bubblewrap, stack them and tie them down. Same with the monitors if you can do everything in one go. I suppose you want to save the machines first, data is money, monitors are cheap. 3 guys load 200 machines in under an hour. (been there done that). Be careful the most dangerous thing to the machines are bumpy roads. Take it easy. Hard bumps can kill a disk, and generally, any vibrations will loosen cables. Especially SATA cables. Don't panic if something doesn't work after moving, open machine, fasten all cables. :)
Can I light a sig ?
Doesn't cover pre-existing conditions (fire).
Had you adopted "The Cloud" sooner, this would not have been a problem!
Make sure you have off site backups of everything needed to reconstruct your network.
At this point that would a "shoulda done" thing. I'm guessing since he's asking, they don't have a clue. Hopefully you practiced proper source control/workstation backups. Grab the servers and place them in a car(s). After that, the most important workstations, any truly expensive pieces of networking gear, and then whatever else you can. Realize anything in the U-Haul may not survive the trip, even with packing material.
The cesspool just got a check and balance.
Two years ago we go a call telling us the levies might not hold and if they burst (1 block away) we'd have 8 ft of water. We didn't really have a battle plan and we had a lot less to deal with than it sounds like you did, but we learned some lessons.
1) praise the lord we had good network documentation. Now is not the time to be writing down how the firewall and public and private LANs are plugged in together. Shut stuff down, and start placing network hardware in big plastic tubs. Have tubs handy for this, they nest nicely when not in use. Toss cables in a different tub. just wind them up best you can into loops and toss them in. there's probably not time for neatness, you can deal with that later. TAKE THE DOCUMENTATION WITH YOU. You'll feel mighty silly if that's left pinned on the wall. or I assume you have an electronic copy you can print when you get offsite. Make sure any servers with complex cable attachments (like to phone banks or security systems) have labels on the connectors.
1b) got your phone system documented too? this is a whole 'nother can of worms that often is forgotten about. Does anyone have a diagram of where all those punched down wires go on each block? If you have phone switching hardware to pack, make sure the cables are labeled, they will all probably look the same with the giant connectors that attach to the blocks. "We'll just call Al, he does our phone stuff." Oh, you don't think Al is going to be BUSY helping everyone else that is returning? Nothing's as fun as a 2-3 day wait to get your phones back up and running huh?
2) Label ac adapters. You need to know which unit wants 12vdc and which has 24vac, you don't want to fry stuff when you are trying to reassemble. every pack should have the model of the unit it goes to written on it. Gear WILL get separated from its pack during the evac.
3) label staff's hardware. It's very annoying trying to figure out whose beige box is whose later. and they will probably fight over monitors and keyboards later. save yourself the headache. If you are already under the gun, run to the store and get a dozen rolls of masking tape and sharpies and have the staff label their equipment while you're packing things up, full initials or names, I bet you have duplicate first names you don't want to deal with later. Make sure you label the phones.
4) have a plan for things you can't easily move. the corp office was also forecast to get 8ft of water and they were on the WRONG side of the dike so it was more of a "when" than "if". they had a very expensive multifunction printer that the service people told them they could have a tech out to take it apart (so it fit out the door) in three days, which obviously was silly. They rushed in a bunch of cinder blocks and lifted it up and set it on them 8.5' up. (I have no idea how they lifted it) In retrospect, the building got 14" of water and totaled it, they SHOULD have killed power to the building and took a saws all to a wall. OR at least watertight wrapped it before lifting. I've seen this done with entire cars when faced with an incoming flood or hurricane. Even if it doesn't keep out the water 100%, at least it will keep out the mud, which you may be very grateful later. Got a plan for your big server room ups's? those can be quite large and heavy, and are often hardwired into the AC, are you able and qualified to unhook it? Maybe you should call in an electrician now and change that armored cable to a dryer type plug? Have a place you can move big stuff that can't be evac'd to where it will be at least more likely to survive. Think of flood, fire, and tornado/hurricane, there's probably not one single place that will work best in all three cases. Smoke damage can be very destructive, simply having something wrapped in mover's visqueen may prevent unnecessary loss that the fire missed but the smoke got. Do you have a plan for that rack that's bolted down or won't even fit through the door?
5) Document what's been left behind. A simple way to
I work for the Department of Redundancy Department.
in order of importance, fragility, price and density.
put some spare mattresses on the floor/sides of the uhaul and put your servers down there. next to each other. (you do have mattresses, right? you're a retreat center and a big fire is coming...)
next desktop boxes, lined up next to each other.
on top networking and ip phones, combined into a few bags/pillowcases etc. these, particularly the phones are light and wont damage each other.
next screens, wrapped in blankets and stabilized. you'll find the screens most fragile, and requiring the most careful packing, but they are also not so expensive to replace so don't worry too much.
come to think of it, you probably can throw some meditation pillows in there between the screens and anywhere else you need them.
that should give you a fast pack of everything critical. you also hopefully have made offsite backups, though.
Personally I wouldn't take anything unless it is 100% un-replaceable (discontinued systems and since-last-offisite-transfer backups). Remember, your insurance will (if the person that negotiated it wasn't a complete moron) cover ALL hardware that is caught in the fire, they might NOT cover hardware that you broke in the U-Haul truck while trying to save it. You should already have offsite backups, so at the most you should save the "didn't make it to offsite yet" recent backups (1 day to 1 week's worth depending on your setup). For everything else: let it burn, that's what you pay those high insurance premiums for! If your insurance company doesn't like that plan, THEY can move it out of the f*$ing building.
If you find out you work for a business with large quantities of server equipment that doesn't have FIRE INSURANCE, the only things you should take with you are
- 1 UPS
- 1 computer (desktop/laptop/whatever)
- 1 printer
- 1 reem of paper
Now you have all you need to print resume's while driving the hell away from that building as fast as you can!
As everyone has stated so far, WTF are you doing on Slashdot, when your in an emergency situation. Now, onto my part, WTF are you doing as a IT Manager, and you don't have a critical battle plan, How the hell did you get this job in the first place. Your previous background was not in IT somehow. /done.
First thing first, Grab the essentials. Since you stated 50+ workstations 6 Servers and random IT 'junk' then your most important priority is obviously the data. Whatever your company's business is, I presume you have six servers because the important stuff is done on these servers, and the workstations are really just end user stuff. Park the U-Haul close to the building and get ready to launch.
--1) Mark your machines. IP addresses, Hostnames, etc each machine should be labeled before disconnection. Your first hour can go quick, if you use programs like Nessus, or LanGuard to map your existing network infrastructure. Getting a logical map saved of everything connected is a wise thing to do. Once a logical map is made, and physically your equipment is labeled... Step 2.
--2) Unplug the important. So you have some networking equipment along with your processing equipment(computers/servers) You probably want everything that could be brought back, plugged backed in, and literally your back up and running. Well. Time to grab the essentials. Grab your routers, switches, the Modem will be helpful. Grab the servers, and THEN grab the workstations. Load them up. You have laser printers? Are these printers networked? If so, you may want to grab the most productive Printer you have, and take it with you.
--3) Segment the piles. Obviously people want to keep their 'junk' together, however in a emergency situation, the most important stuff needs to be cared for first. So the IP phones, are probably expendable. Maybe not thePBX device because those can cost upwards in the thousands, so if its located on the wall where it should be, you can probably disconnect it and take it with you. The phones themselves can be expensive as well, but not even close to the device that serves them.
--4) When your out of the way of harm, you probably want to setup temporary shop. So wherever you end up, you are only going to setup the marginal needs to ensure you can pull records/data. So a Hotel, U-Store-It, or even a different office building, your going to need to obtain some replacement temp cabling, and desks.
As you can tell, if you left behind anything during your grab, your going to be spending money to have replacements. Hope the insurance coverage wasn't botched, because this is where you learn your company can get back on its feet. Otherwise, if you lost your data, your dead anyways. At least you have your data.
I could care less about 'Cloud Computing", but it really is a blessing to have some sort of off-site backup procedure in place. While your prepping to move, you could have had your data backing up online/offsite. During the long process, you could have recorded serial numbers, taken photographs of each room (before) along with itemized lists of what was in each room (or associated with each workstation). If you did a good job, you probably now have sufficient records of all your assets ahead of time. But if you suck at your job, you are probably frantic right now, and making bad decisions like asking Slashdot for advice, when you have few hours left.
Now my example of an emergency situation that came up on me at a datacenter (10k workstation/server environment).
Flashflood warning popped up on me, and corporate was out of town. Our datacenter was setup with a T1, Satellite, and a backup (consumer based) Cable Modem services. With our primary switches and routers located in a locked room. When employees started panicking that water was entering the building from the east end(where the ditches and drainage areas were), I kne
Laptops/notebooks, tablets where possible. Anything that can be used in a zipper case is a plus.
Wireless of course. Several sets of access points, two in the trailer. The production ones can be left behind to burn.
Many of today's notebooks can do a good job as a server, remember to use power settings that make sense for server duty.
Backups of course, but probably external drives.
If you have the flexibility to choose your server OS, one that offers a resilient filesystem is good, since you may have to power down in a vehicle of some sort. Pulling the external drive off when running will exercise the resilience.
Bugging out would consist mostly of closing lids, zipping up cases (maybe) and running. Servers go the same way. IF you can grab the APs, fine. For 50 users, this will not take a half hour. Crates to take the zip cases should let you essentially drop stations in there. You can build crates that cushion your servers better. External drives get better cushioning also, but using notebook drives enhances their durability, some. Everything else can survive.
Keep a set of UPSes in the trailer, or at least by the back door, charging. These will get handy when you arrive at the new location, and if you save the old ones, you may use two sets to give you instant power while you get everything running, and find the outlets for permanent power. A generator would be handy, and it need not be big. Propane rigs are easier to handle than gasoline. If your evac point is within 2 hours' drive, you may even be able to safe the servers, park the drives, and take them on the trip running. You ARE writing scripts to do emergency shutdowns, safe modes, parks, and closing critical apps/saving data, riiiight?
I'm assuming you may not always have 24 hours' notice. If you will have a guaranteed 2 hour notice, then use short racks that can be wheeled around, and you can have fairly conventional servers and wired network, just plan on abandoning the cabling, which is entirely expendable IMHO. Leaving the servers and switches cabled together is helpful, and sme simplified interconnect to mutiple cabinets will help. Plenty of cables in the trailer, and a spool of cable with a bag of plugs and at least 2 crimpers also. And a simple tester. Trust me on this, no point in guessing if you made it right. Making those 200' cables to solve a problem would be handy.
Lots of diagrams laminated to the cabinets is handy, even a grease penciled fill in the blanks chart to show what was built is a blessing when you reconnect.
Somehow, I suspect the military has some advice for you on this. Someone in Interior or the Forest Service must have a contact.
I would love to be in that business. Nothing like having to make DR plans that have to accomodate the loss of the facility to sharpen your focus and get the juices flowing. The last project like that I was in, a financial institution needed a similar plan, and we even has a BOM at a distributor ready to be ordered and shipped on notice, updated quarterly. Almost got to do it for real, but they fixed the gas leak without blowing up the building. Darn. :)
deleting the extra space after periods so i can stay relevant, yeah.
You have 10 minutes to load a 14 foot trailer with all your gear before the place burns down! I bet you're wishing you hadn't cracked down on employee "slacking off" with Tetris now! Oooh maybe we'd know how to load that trailer if our boss hadn't told us to get back to work!
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Fully agree with this AC. If this is a geek/nerd posting this question you should be ashamed of yourself. I was expecting some sort of photographic evidence with layout and resulting HOWTO. HOWTO: Evacuate your data-centre quickly.
1. Highlight highest value equipment first
2. Arrange the physical design of the data-centre to quickly remove priority rack-mounted equipment first.
3. Prioritize: Stuff not currently backed up; expensive equipment.
4. Flood the place with flame retardant before leaving.
5. etc
Too late for that now I suppose.
In lieu of that grab the drives and run.
Second this. If you're thinking about this now then it's too late. Forget all of these idiotic elaborate answers on how to spend time in the path of a wildfire "evacuate" the network as they're likely to get you killed. Wildfires are extremely dangerous and fickle things. You may think you have 30 minutes, but then the wind shifts slightly and you're dead. Grab some drives and run is pretty much what you're left with.
Aside from that, your insurance may cover equipment loss by fire but it may not cover damage caused when trying to save it. Next time colo the important stuff for redundancy. Colocated servers have an advantage over "cloud" of being removable from the colo to become instant infrastructure if the building does burn. At least automated offsite backups of important data so you can trigger it remotely and watch from a safe distance.
this is my sig
http://eddie.niese.net/images/Wiring_BAD.jpg
(A) have employees take their desk stuff... the laptops/desktops. four wraps of bubble wrap, and go. screw the rest. there are usually a bunch of apps and work files on individual PCs that are irreplaceable. some are even business-critical. but it's THEIR machine, they're up now because it's set up and ready to go. add DSL/cable/whatever and they can work from home.
(B) send somebody west with backup set A. send somebody east with backup set B. then take out the servers and go. if you have very freaky setups in firewall appliances and routers, bring them, too. I like the idea of cheap-ass mattresses. fill the floor, and set the racks flat on their backs on the mattresses. yeah, it's overkill, but you have your inter-rack wiring, etc there.
(C) insure you have critical business papers... server and software licenses, articles of incorporation, insurance policies... with you.
screw the rest. employees can go anyplace for a monitor and keyboard/mouse kit. you can get a X-pack of twice-recycled cell phones at any of the corporate stores within 12-24 hours to operate on temporarily. if you can't get into building space at once and need to operate the business, haul the server room stuff to an ISP with colocation facilities and have your MX record transferred to their pipe.
if this is supposed to be a new economy, how come they still want my old fashioned money?
Save your data and your people
but not in that order.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Your data is worth zero to looters and will be overwritten by pr0n anyway.
To The Cloud!
If there's a wildfire coming your way, everybody who owns a Civic is going to be at the truck rental places ahead of you in line. Your facility will burn before you even get to the counter.
Buy a truck and park it on the facility property. Or buy two small vans (which would probably have more computer-friendly suspensions) and keep them there. They'll be useful every time the company needs to evacuate ahead of a fire...or move from one office to another...or help the CEO move when (s)he buys a new home...or help relo new employees from out of town.
Anytime there's hardware that can't be replicated in advance in an offsite disaster recovery facility, you really need to be able to move it 24x7. And without the truck, your entire disaster recovery plan goes down the toilet. The truck is the critical path, and needs to be treated as such.
What I'm seeing here is a disagreement on the time frames involved. I liked one of the posters who mentioned 'acute emergency'. A wildfire miles away, with the area being declared a 'pre-evacuation' zone indicates that there is not only no exceptional danger yet, but that there's an emergency alert system in place to tell you BEFORE it becomes acute.
In which case acting to protect property is still on the cards. Choices include 'shelter in place' and 'evacuate'. For a fire, the answer is generally evacuate for people, but the equipment can still shelter.
The ideal time to be planning for an emergency is always NOW. Plan for fire even if there's no wildfires in the state, no fire in the building, but if it's too late for that, you can still improve your situation by doing the planning. A bit more expensive, results aren't going to be quite as good, but it can still be done.
Long term(fire not expected other than general threat), if you're in an area where wildfires are a possibility, involves turning your building into a shelter. Don't put vinyl siding up, put up aluminum or even steel, backed by the appropriate fire-resistant insulation. Install fireproof shutters. I've seen vinyl siding melt with just radiant heat; once it's gone the wood behind it can catch. Clear an area around the building, short grass or better yet fire resistant plants such as Sage or Yarrow for within 30 feet of the building. Remember, keep the plants trimmed, and water when fire is expected!
Medium term(fire is expected within days) - too late to change what's planted, your roof/siding, etc... You still have time, depending on your available resources, to remove dead vegetation, trees too close to the building, mow the lawn short(and the only time I'd bag and remove the cuttings). After that, wet everything down to protect against cinders.
Short term(hours) - remove what dead vegetation and trees that you can, set up what water you can, leave early so that you're not fighting with the other last minute evacuees.
Acute - leave already! Forget everything except maybe the backup tapes. Which should have been in an appropriate firesafe/cabinet already. Remember: A firesafe rated for paper is NOT rated for magnetic tape for the same temperatures/times. The average 'fireproof' safe or filing cabinet is only rated for paper. You can get removable type drive bays that are rated for fire/water. But at 15 minutes of protection, they're unlikely to work against a wildfire sparked building fire.
I don't read AC A human right