Slashdot Mirror
Ask Slashdot: How To Evacuate a Network
First time accepted submitter gpowers writes "I am the IT Manager for Shambhala Mountain Center, near Red Feather Lakes, Colorado. We are in the pre-evacuation area for the High Park Fire. What is the best way to load 50+ workstations, 6 servers, IP phones, networking gear, printers and wireless equipment into a 17-foot U-Haul? We have limited packing supplies. We also need to spend as much time as possible working with the fire crew on fire risk mitigation."
Get one?
Quickly?
Seriously... you're a geek... use your own imagination. This question is just plain idiotic. Not that it's any surprise that it would be on Slashdot these days....
Less posting to Slashdot would be step 1...
Pack what's critical first. Servers. Critical networking gear. Workstations. Ignore the phones, printers and wireless gear unless you've got extra time. And good luck.
"But we decide which is right, and which is an illusion"
Nuke it from orbit. It's the only way to be sure.
you probably have lots of those available
careful over the bumps, and godspeed
First thought is put half of them in your own car. Then put the other half in the truck and abandon it in the fire's path. Then eBay.
Basically you're fighting, not avoiding. Relocate. Avoid. Cheers!
I come to Slashdot only to read sigs. One you are reading is mine.
Damn if I'd go in to work to remove hardware when a fire is threatening.
I'm not paid enough to risk my life. Period.
If something breaks, it can be easily replaced. If you leave everything behind, that'd be much more expensive.
Take lots of pictures before you unplug your cables. It will save you time when you have to reconnect everything.
The "best" way to evacuate a data center is to already have off-site back-up for your data in place, drop a fresh copy to portable media, and walk out. The hardware should be insured. The life of your and your people (at least some of whom should probably be helping their families evacuate) are far more valuable than a few months of making your insurer pay for rented hardware until your new machines show up.
The workstations and servers presumably contain data that is either irreplaceable, or labor-intensive to recreate. the printers, IP Phones, and non-configurable networking equipment can be re-ordered when the dust settles. Pack the former first, and leave the latter if you are pressed for time.
Focus on "Properly packing" the Servers and workstations. Properly packing in this context is retaliative, but I bet servers and workstations are more sensitive to getting banged around in the back of a u-haul than ip phones, printers, and wireless gear. Your network gear is probably in group 2 - more important than the "Phones, printers, and Wireless gear"
"Give a woman two glasses of wine and some pad thai, and they'll agree to just about anything." the Sports Guy
First, triage the equipment.
You likely do not have time to pull disks from systems, so pack computers and
external drives first. Get blankets to protect things. Blankets start at the bottom
to act like a shock absorber.
Things like networking gear and wireless stuff is irrelevant compared to the
computers, and probably lighter. If you CAN, sure, save all that stuff too.
But the data comes first. Don't forget backups.
If there are computers with really really important or sensitive stuff, put
those in someones car in the backseat, again with blankets. If I seem
blanket obsessed, it's because I've found them to be available quickly
either from individuals or stores. Yes, bubble wrap or sorbathane would
be better but you aren't likely to have that stuff lying around.
Best suggestion I can think of ...
1) servers (take the entire racks), documentation, backups, certificates (to prove you own what's on the servers), network gear, etc first. Make sure you have EVEYRTHING needed, core-wise, to operate.
2) THEN start loading workstations. If you have to leave some behind, and the place burns down, that's what insurance is for.
You can operate a business on leased workstations.
You cannot operate a business without all your core servers, and you wouldn't want to wait for the downtime required to rebuild them.
Not trying to be mean, but you should have already had a plan in place...this late in the game without a plan means you just have to go with asses and elbows (just get what you can while you can and forget trying to install a plan to do it!)
The good news is you can become an example for other IT people! Everyone should look at their disaster plans and make sure you have accounted/planned for all emergencies that may happen in your area.
Put the most valuable stuff in first and best packed. You can probably take the computer boxes themselves and just line them up in a big rectangular area and rope it off so they don't move. Monitors (LCD) will be the hardest because they are awkwardly shaped and easily damaged. You can wrap the screens with cardboard and lay them on their sides, interlocking, if shape permits. CRT, if you have those, can just go in like the computers, they're pretty resilient. Can't say anything about the rest, you are probably best off sticking it in boxes.
In general, rope is your friend. You can keep stuff from moving pretty effectively with rope.
Great Intellect...
I'd say worry about your family first and get them the hell out of there first. And than from there start at whats essential and most important
http://theworkaround.com/
Irreplaceable: Load it and get the hell out.
Next time: Planning and preparation.
NetInfo connection failed for server 127.0.0.1/local
You definitely need to apply this: http://en.wikipedia.org/wiki/Knapsack_problem ;)
I would take backup tapes first in a separate vehicle. that way you can recover. Servers. Work station computers next, not the monitors keyboards and mice. Then leave the rest. Insurance should cover it. The most important thing is the data, and they sit on the backup tapes and don't take much space.
Whatever stores data first -- if it's a SAN, then your RAID chassis and metadata controllers, and if you have time, the SAN fabric switches and cabling, but you can replace the latter if you have to, and if it's ordinary SAS, the servers if they're all internal storage, or the RAID chassis or whatever's external. Definitely grab any non-offsite backup media with that. Rest of it in descending order of priority after you grab the most valuable stuff, mostly to avoid having to replace it.
Best strategy overall is to think "what if we had to abandon this evacuation mid-process and run?" Try to have what you most want already in the truck at any given moment, and concentrate on data before hardware -- the data is far more valuable in most cases.
If you haven't done an offsite backup, for god/dess' sake do one *now* and get the backup media to a safe location .. :/
Label the server disks, then pull them from the servers. Use your limited packing materials to protect them. If you have time to load the servers into the truck, then great - but as long as you know what servers those disks came from, you can buy more of them (or put in service calls on the ones you have if they were damaged in the truck).
Dump your switch and router configs to text file and copy them to USB key AND PAPER.
If you're looking to do a straight toss-and-drive: workstations on the bottom, then servers, then switches. Printers off to the side. Everything else (phones, etc) on top of the switches.
Labelled hard drives in a separate, padded box that doesn't leave your sight, and switch configs in your pocket.
Hope that your users don't save everything on their desktops...
Heavy/sturdiest stuff on bottom, fill left & right sides of the truck then fill in the middle w/ crap that can easily be tossed out for easy access. Seriously though, just play tetris next time you take a shit & the tetris effect will show the way. Oh, & label the workstations. http://en.wikipedia.org/wiki/Tetris_effect
1. label all hard drives. Drive position, and server.
2. place hard drives in anti-static bags
3. pack drives in foam.
4. get drives far, far away.
Hard drives are both the most valuable, and the most fragile part. Do not load them in a stiff suspension vehicle like a truck, as this bounces the drives. Choose a soft-suspension normal car.
Next take servers and network gear. Desktops are a maybe, as are phones. Ignore printers.
Tape a piece of cardboard over the face of an LCD monitor to protect it from casual bumps.
Above all, no data is worth a human life. No heroics. You're not paid for heroics.
Many servers and disk arrays specify that drives have to be shipped separately simply because their mass is so high that the rack doesn't pass shock and vibe tests. So either pull out the drives (mark which slot they go into) and pack them separately in bubble wrap or, if that's not an option, put as much cushioning as you can around the servers and strap them down so they don't bounce. In any case, be prepared for some disk drive damage or degradation.
thegodmovie.com - watch it
Log off Slashdot before reading this :-)
If you've got labeling stuff around, use it (not fancy label makers, just the basic "Hello My Name Is" and a Sharpie.)
Grab the servers, grab the workstation bodies, grab the phones and anything else that's easily portable, and any backup media you've got. Unfortunately, rack-mounted equipment is usually harder to grab, but that's probably your most expensive and critical stuff. And it'll be your critical path, so start unbolting it first. All of that will fit, put it in first, braced as well as you can.
Monitors and keyboards are nice, but they're just money, not data. Grab a few of them, but leave the rest for last. If you have packing material left, great, but if not you'll just have some breakage. If you've got any CRTs, leave them, they're heavy.
Bill Stewart
New Fast-Compression-only CPR http://preview.tinyurl.com/dy575ks
Catapult. Quick as hell...
Drive up to a Home Depot and honk your horn.
P.S.
Buy them lunch.
Make sure you have off site backups of everything needed to reconstruct your network.
After that it really doesn't matter. Either you can move everything out in time or you cannot. If you cannot then you move the people and forget the gear.
Just like in the fire drills for almost every other company out there.
It's called insurance.
thinking this through, your priority is to make it as quick as possible to get essential gear out the door. so, consider:
* having the servers and desktops *already* in easy-to-carry crates, with large handles on the outside and packing materials surrounding the machines.
* have the machines stacked off the ground so that people don't have to waste time bending down and possibly injuring themselves by jolting weight that's too much for them
* have all essential equipment nearest to the doors plural, prioritised by criticality
* yes doors plural: add an extra door next to the existing one (or replace the one door with easy-to-open double-doors with those pushable handles) so that at least two people side-by-side can get through at once, carrying the crates, and can just "barge through them" rather than having to twist the handles.
* make sure that the crates are stackable and sturdy but also light enough to carry!
* even consider having the machines already loaded onto 4-wheeled trollies and left on them, permanently.
* if time is _seriously_ critical, consider putting guillotines next to all cables (and test them) so that people don't have to waste time unplugging cables: just cut them and go - but only consider this if the guillotines are sharp enough and easy enough to operate, and only if it's seriously seriously critical to save seconds. don't put power cables through the guillotine though!
* consider getting convenient light-weight but sturdy cabinets made for all LCD monitors, with double doors that fold back 180 degrees out of sight, and a top (with a handle) that locks automatically when it's flipped over. have the LCD monitors mounted onto the cabinets with rubber bushes so that they don't need to be placed or positioned into the cabinets - just pull out the cables, shut the doors, slam the top over and pick it up by the handle: done.
* consider getting 12v powered LCD monitors instead of 240v/120v AC mains, so that the power cables can be guillotined rather than pulled.
* instead of guillotining, consider breaking all the tabs on the network and telephone cables (the ones that "click and lock") and affixing them *loosely* with gaffa tape to all devices (network hubs, machines etc.) - this way it will be possible to just pull (hard) and out pop the cables. or, if someone forgets, and gets to the end of the wire, they won't trip or be yanked backwards: the cable will just come out, clean.
* get 4-port hubs instead of 8, 16 or 24-port. 4 gaffa-taped cables are easier to pull out than 8, 16 or 24, and if one of the 4-port hubs is lost to a fire, so what, big deal. a 24-port hub however starts to get expensive.
* stop people from putting the bloody screws in the bloody cables - you know the ones: parallel ports, VGA cables, serial cables etc. the ones that are always bloody irritating when it comes to fixing or moving a machine and you find that the bloody VGA cable needs a bloody screwdriver to remove the damn thing. take the screws *OUT* of the cables; that way people can't go "oh look: screws - let's tighten them".
so - yeah. make it easy to just shift everything. have practice drills. set a deadline (say 1 minute) and see how much kit people can get out in that time, without damaging it.
oh - and you know how i suggested making it easy to shift everything? uh... make sure the insurance is up to date, and get good security. no point making it easy for *other people* to shift all that expensive gear, eh? oh. and sort out some off-site backups, eh? :) i use rsync; my friend uses backuppc (because he has a lot of machines). /peace
Why does an IT manager not have a disaster recovery plan?
How did you get to be the IT manager and not do this basic planning? I mean you are the _manager_ after all. You don't do any real (i.e. in the trenches) work any more, so what are you doing if you are not at least doing regular IT manager stuff like planning for disaster?
Walk out with your backups, and save the people first.
go with Ryder (much better then U-Haul)
the problem. Like the other people say here, load up the most important machines first, pack them with bubblewrap, stack them and tie them down. Same with the monitors if you can do everything in one go. I suppose you want to save the machines first, data is money, monitors are cheap. 3 guys load 200 machines in under an hour. (been there done that). Be careful the most dangerous thing to the machines are bumpy roads. Take it easy. Hard bumps can kill a disk, and generally, any vibrations will loosen cables. Especially SATA cables. Don't panic if something doesn't work after moving, open machine, fasten all cables. :)
Can I light a sig ?
Having disaster plans costs money, most companies aren't willing to spend.
And when all is said and done, what do you think this little fiasco will cost them? "Penny wise, pound foolish" applies to this situation.
Thank me later!
Ask the Monks to all prey very fast and in ernest, and get all your disk / tape storage out now...... the heat in the area alone may be enough to kill tape if you have any. but disks, san, backup data is 100% a must...... insurance hopefully would cover what got left behind.
Well, the equipment should be fully insured. Have a plan in place to replace it quickly. The data should be backed up offsite. If you are not doing offsite backup then TPTB obviously place the value of that data at $0. The only thing you should be walking out the door with in the event of a wildfire coming your way is yourself, your laptop and possibly some drives with the most recent local backup.
But if the fire is literally on your doorstep now, I'd tell the boss to fuck off and get my ass out of there, the hell with the job and the data and the company.
Pretty much at this point you should be opening book 1 and looking for plan b , the one you have actually practiced a time or two. but it sounds too late for that. so take data first, if you can't move it send the data it's self to the cloud, PCs pretty much get work done so they go next, VOIP phone server equipment should be the last to go, can you auto forward to cell phones before the bug out? the employees may be able to lighten the load by taking their own equipment when they bug out.
The rest of the readers out there Plan, Practice, review plan, practice, and repeat again.
Had you adopted "The Cloud" sooner, this would not have been a problem!
For rapid packing it would be best to have moving pallets prefabbed. Each employee grabs there gear, puts it into the prefab units, then you wheel the lot out. That way you're not pissing about trying to arrange everything so that it fits. As long as the truck holds the prefab units, everything will fit.
Two years ago we go a call telling us the levies might not hold and if they burst (1 block away) we'd have 8 ft of water. We didn't really have a battle plan and we had a lot less to deal with than it sounds like you did, but we learned some lessons.
1) praise the lord we had good network documentation. Now is not the time to be writing down how the firewall and public and private LANs are plugged in together. Shut stuff down, and start placing network hardware in big plastic tubs. Have tubs handy for this, they nest nicely when not in use. Toss cables in a different tub. just wind them up best you can into loops and toss them in. there's probably not time for neatness, you can deal with that later. TAKE THE DOCUMENTATION WITH YOU. You'll feel mighty silly if that's left pinned on the wall. or I assume you have an electronic copy you can print when you get offsite. Make sure any servers with complex cable attachments (like to phone banks or security systems) have labels on the connectors.
1b) got your phone system documented too? this is a whole 'nother can of worms that often is forgotten about. Does anyone have a diagram of where all those punched down wires go on each block? If you have phone switching hardware to pack, make sure the cables are labeled, they will all probably look the same with the giant connectors that attach to the blocks. "We'll just call Al, he does our phone stuff." Oh, you don't think Al is going to be BUSY helping everyone else that is returning? Nothing's as fun as a 2-3 day wait to get your phones back up and running huh?
2) Label ac adapters. You need to know which unit wants 12vdc and which has 24vac, you don't want to fry stuff when you are trying to reassemble. every pack should have the model of the unit it goes to written on it. Gear WILL get separated from its pack during the evac.
3) label staff's hardware. It's very annoying trying to figure out whose beige box is whose later. and they will probably fight over monitors and keyboards later. save yourself the headache. If you are already under the gun, run to the store and get a dozen rolls of masking tape and sharpies and have the staff label their equipment while you're packing things up, full initials or names, I bet you have duplicate first names you don't want to deal with later. Make sure you label the phones.
4) have a plan for things you can't easily move. the corp office was also forecast to get 8ft of water and they were on the WRONG side of the dike so it was more of a "when" than "if". they had a very expensive multifunction printer that the service people told them they could have a tech out to take it apart (so it fit out the door) in three days, which obviously was silly. They rushed in a bunch of cinder blocks and lifted it up and set it on them 8.5' up. (I have no idea how they lifted it) In retrospect, the building got 14" of water and totaled it, they SHOULD have killed power to the building and took a saws all to a wall. OR at least watertight wrapped it before lifting. I've seen this done with entire cars when faced with an incoming flood or hurricane. Even if it doesn't keep out the water 100%, at least it will keep out the mud, which you may be very grateful later. Got a plan for your big server room ups's? those can be quite large and heavy, and are often hardwired into the AC, are you able and qualified to unhook it? Maybe you should call in an electrician now and change that armored cable to a dryer type plug? Have a place you can move big stuff that can't be evac'd to where it will be at least more likely to survive. Think of flood, fire, and tornado/hurricane, there's probably not one single place that will work best in all three cases. Smoke damage can be very destructive, simply having something wrapped in mover's visqueen may prevent unnecessary loss that the fire missed but the smoke got. Do you have a plan for that rack that's bolted down or won't even fit through the door?
5) Document what's been left behind. A simple way to
I work for the Department of Redundancy Department.
Bring in an expert.
you'd know that fire kills trolls
Most posters don't seem to have understood the question. The location is *not* in immediate danger; there is no current risk to life or property. However, if the fire behaves badly, they may have a problem a few days from now. The idea is to be ready for that circumstance. So there's plenty of time to worry about valuable property as well as people.
These would be perfect for emergencies.
Shai Schticks:"You don't make peace with friends, you make peace with enemies"
in order of importance, fragility, price and density.
put some spare mattresses on the floor/sides of the uhaul and put your servers down there. next to each other. (you do have mattresses, right? you're a retreat center and a big fire is coming...)
next desktop boxes, lined up next to each other.
on top networking and ip phones, combined into a few bags/pillowcases etc. these, particularly the phones are light and wont damage each other.
next screens, wrapped in blankets and stabilized. you'll find the screens most fragile, and requiring the most careful packing, but they are also not so expensive to replace so don't worry too much.
come to think of it, you probably can throw some meditation pillows in there between the screens and anywhere else you need them.
that should give you a fast pack of everything critical. you also hopefully have made offsite backups, though.
you have no disaster plan? 1.) offsite backup everything, 2.) get everyone out. 3.) file an insurance claim, 4.) rebuild.
1. Inventory *everything* in terms of hardware, disks, monitors, etc. Insurance will want this (and you do have insurance, don't you?)
2. Save all configs that you can to someplace secure/remote. This would make setting things up again more easy if it is the case that your entire workplace gets destroyed. A switch / wireless AP / IP phone is very easy to replace and really not worth saving if there's a fire coming down the mountain.
Consider this: do you really need the entire workstation or just the hard drive? Even then, do you even need the data on it or can you rebuild it with an image and a restore from a network share?
Insurance and off site backup.
You likely don't have the knowledge and skills to quickly pack and load a truck like this - stuff will get broken, and you'll be slow.
Better to hire professional moves who can come in, grab the critical stuff, and pack the truck so that nothing gets damaged. Probably stuff like equipment racks can be dollied out in one piece and tied down in the truck - forget pulling individual drives.
Three Squirrels
If you have only 30 minutes or so chances are one or two people will only be able to grab a few servers at most, and with a large organization thats going to boil down to probably credit card processing server, customer account database server, imaging server(s), and maybe a web server and application server. Everything else be sure to take pictures of so its easy to prove it existed for insurance purposes.
Bonus. If the stuff doesnt' get damaged, its easier to put 4 or 5 servers back in the racks than it is a bunch of racks.
Don't rush or panic while moving critical gear. Take careful methodical action and accidents will be less likely to happen. Don't drop the critical server while trying to save it.
Your backup tapes/drives/cloud thinger should already be well outside of the evacuation zone. Turn off the lights, and go.
If you don't have that ability, then your organization has failed at responsible IT 101.
Too late for that now I suppose.
In lieu of that grab the drives and run.
I am very small, utmostly microscopic.
As everyone has stated so far, WTF are you doing on Slashdot, when your in an emergency situation. Now, onto my part, WTF are you doing as a IT Manager, and you don't have a critical battle plan, How the hell did you get this job in the first place. Your previous background was not in IT somehow. /done.
First thing first, Grab the essentials. Since you stated 50+ workstations 6 Servers and random IT 'junk' then your most important priority is obviously the data. Whatever your company's business is, I presume you have six servers because the important stuff is done on these servers, and the workstations are really just end user stuff. Park the U-Haul close to the building and get ready to launch.
--1) Mark your machines. IP addresses, Hostnames, etc each machine should be labeled before disconnection. Your first hour can go quick, if you use programs like Nessus, or LanGuard to map your existing network infrastructure. Getting a logical map saved of everything connected is a wise thing to do. Once a logical map is made, and physically your equipment is labeled... Step 2.
--2) Unplug the important. So you have some networking equipment along with your processing equipment(computers/servers) You probably want everything that could be brought back, plugged backed in, and literally your back up and running. Well. Time to grab the essentials. Grab your routers, switches, the Modem will be helpful. Grab the servers, and THEN grab the workstations. Load them up. You have laser printers? Are these printers networked? If so, you may want to grab the most productive Printer you have, and take it with you.
--3) Segment the piles. Obviously people want to keep their 'junk' together, however in a emergency situation, the most important stuff needs to be cared for first. So the IP phones, are probably expendable. Maybe not thePBX device because those can cost upwards in the thousands, so if its located on the wall where it should be, you can probably disconnect it and take it with you. The phones themselves can be expensive as well, but not even close to the device that serves them.
--4) When your out of the way of harm, you probably want to setup temporary shop. So wherever you end up, you are only going to setup the marginal needs to ensure you can pull records/data. So a Hotel, U-Store-It, or even a different office building, your going to need to obtain some replacement temp cabling, and desks.
As you can tell, if you left behind anything during your grab, your going to be spending money to have replacements. Hope the insurance coverage wasn't botched, because this is where you learn your company can get back on its feet. Otherwise, if you lost your data, your dead anyways. At least you have your data.
I could care less about 'Cloud Computing", but it really is a blessing to have some sort of off-site backup procedure in place. While your prepping to move, you could have had your data backing up online/offsite. During the long process, you could have recorded serial numbers, taken photographs of each room (before) along with itemized lists of what was in each room (or associated with each workstation). If you did a good job, you probably now have sufficient records of all your assets ahead of time. But if you suck at your job, you are probably frantic right now, and making bad decisions like asking Slashdot for advice, when you have few hours left.
Now my example of an emergency situation that came up on me at a datacenter (10k workstation/server environment).
Flashflood warning popped up on me, and corporate was out of town. Our datacenter was setup with a T1, Satellite, and a backup (consumer based) Cable Modem services. With our primary switches and routers located in a locked room. When employees started panicking that water was entering the building from the east end(where the ditches and drainage areas were), I kne
For now, I'm with those who say to get data offsite, then prioritize the rest. But for my clients in high risk zones, I generally recommend that they build their DCs using half height racks that can be lifted out with the equipment still in them, forklift sized DC doors, don't forget to actually have a forklift, put related equipment (power, air) into each rack, and label all the patch panels in each rack (the ones that connect to equipment outside the rack) consistently and thoroughly. That's enough to get you a semi-mobile DC in a war zone. It is expensive though. For most people, it's better to have hot-hot DCs geographically separated, and let the insurance company buy you new gear while you run on your remaining site(s).
-- Two men say they're Jesus. One of them must be wrong. - Dire Straits
Laptops/notebooks, tablets where possible. Anything that can be used in a zipper case is a plus.
Wireless of course. Several sets of access points, two in the trailer. The production ones can be left behind to burn.
Many of today's notebooks can do a good job as a server, remember to use power settings that make sense for server duty.
Backups of course, but probably external drives.
If you have the flexibility to choose your server OS, one that offers a resilient filesystem is good, since you may have to power down in a vehicle of some sort. Pulling the external drive off when running will exercise the resilience.
Bugging out would consist mostly of closing lids, zipping up cases (maybe) and running. Servers go the same way. IF you can grab the APs, fine. For 50 users, this will not take a half hour. Crates to take the zip cases should let you essentially drop stations in there. You can build crates that cushion your servers better. External drives get better cushioning also, but using notebook drives enhances their durability, some. Everything else can survive.
Keep a set of UPSes in the trailer, or at least by the back door, charging. These will get handy when you arrive at the new location, and if you save the old ones, you may use two sets to give you instant power while you get everything running, and find the outlets for permanent power. A generator would be handy, and it need not be big. Propane rigs are easier to handle than gasoline. If your evac point is within 2 hours' drive, you may even be able to safe the servers, park the drives, and take them on the trip running. You ARE writing scripts to do emergency shutdowns, safe modes, parks, and closing critical apps/saving data, riiiight?
I'm assuming you may not always have 24 hours' notice. If you will have a guaranteed 2 hour notice, then use short racks that can be wheeled around, and you can have fairly conventional servers and wired network, just plan on abandoning the cabling, which is entirely expendable IMHO. Leaving the servers and switches cabled together is helpful, and sme simplified interconnect to mutiple cabinets will help. Plenty of cables in the trailer, and a spool of cable with a bag of plugs and at least 2 crimpers also. And a simple tester. Trust me on this, no point in guessing if you made it right. Making those 200' cables to solve a problem would be handy.
Lots of diagrams laminated to the cabinets is handy, even a grease penciled fill in the blanks chart to show what was built is a blessing when you reconnect.
Somehow, I suspect the military has some advice for you on this. Someone in Interior or the Forest Service must have a contact.
I would love to be in that business. Nothing like having to make DR plans that have to accomodate the loss of the facility to sharpen your focus and get the juices flowing. The last project like that I was in, a financial institution needed a similar plan, and we even has a BOM at a distributor ready to be ordered and shipped on notice, updated quarterly. Almost got to do it for real, but they fixed the gas leak without blowing up the building. Darn. :)
deleting the extra space after periods so i can stay relevant, yeah.
Have an off-site backup. 1) Now, Visio diagram everything in racks, cabling included. 2) Ensure off-site backup works, connect to it as main source of data and direct back backup flow 3) If absolutely necessary, grab drives from critical machines only. 4) Call insurance company immediately upon entering a safe zone, let them know what's happened. Ensure building is COMPLETELY empty, shut down EVERYTHING in the server room, Yell to verify the rooms are completely empty and secure the rooms as best as you can, Priority number 2 here is securing the data however it is necessary. Priority 1 is, naturally, life. Unless you are trained in fire prevention and use of a fire extinguisher and licensed as such, I would not toy at all with a fire. If you panic and trip and fall and a piece of hardware gets broke, too bad. Upon returning, depending on nature of security. investigate the room thoroughly, even if no fire touched the building. Validate the power down of old equipment didn't leave any odd smells. Shut down everything's power switch, EVERYTHING. Turn power back on for the room. Give Management an ETA of 2.5-4x real time, with the off-record note that it will probably be quicker but you are being dutiful of potential problems (This should all already be written down and signed by your boss/manager and upper management should be aware) Power on equipment in clusters of 2-5 machines at a time, most relied upon first, last relied upon last. (Networking, SAN,backend servers, frontend-ish servers) Have a co-worker help you with this. If a piece of hardware has failed, determine why as quickly as possible, if not possible, follow replacement procedures and, if necessary, buy new until you are able to facilitate insurance claim/RMA/etc. Have admins/yourself test certain functions of the network/SAN/Whatever to verify proper operation. Be ready for calls. There are always calls. This is summarized, of course. Insert what you need, take out what you don't. If you have off-site full operations, ensure your managers know. Keep your manager and possibly one step up from him in the loop and facilitate questions as they arise but ensure they know the importance of you not being sidetracked. I've forgotten a few things, etc, but I THINK this is a good start. If you are in a fire situation, and a fire breaks out in your server room, Alarm and evac. Do NOT put out he fire unless you are certified. Insurance companies have a statement, commonly, that relieves them of heroic attempts at hardware rescue by non-heroes. Another key thing, your manager has to play middle-man. He HAS to take all the heat from your clients and his management. Questions are alright. Oh, and don't overwork yourself. You need to be there (Mentally) for remedial efforts. Burning yourself out to bring a system up in half the time is a no go. This may sound very government-esque and lazy, but put yourself in the shoes of somebody in the situation and just think of the stress you'd be under to bring everything back up. Oh god, I did not realize it would all bunch together when I hit enter... grr. Or, so the preview says.
penske is even better with newer trucks
unless you have offsite replication you may be better off telling people to stop working than have them work till the last momrent and then shutdown.
If you do nightly backups and some people do crital stuff this morning you can loose that work or restore to an inconsstant state. for some information it may be better to be down then inconsustant.
you dont want customers to have reciepts for transactions you do not have backups on your end.
Most of the posts on here are absolutely rubbish; just ignore them, and listen to what I have to say. I've helped businesses in this situation before - here you go: Your shop is not that big if you're only 6 servers, and you're only in the PRE-evacuation. You have time to do this right. Everything out of the datacenter but the wiring. Leave the racks too. The only thing you need is the physical equipment, your data, and a way for continuation after re-establishment somewhere else (Backups). Ensure you've grabbed all backup tapes that may be on-site. 6 servers will not take you that long to evacuate. Your datacenter is most likely a wiring closet or something equivalent, so take the extra time to label things. Make a diagram or map before you start removing things so you know how to put it back together. If you have some networking equipment scattered throughout the premises, ensure to grab that stuff too. Remember, you're not on fire yet. You have time to think this over and get all of the REALLY important stuff. You have two options for how you want to handle the client workstations and IP phones. The first option is for you and your IT team to go around to each one with a cardboard box and pack it all up. The second option is to give each employee a cardboard box and have them pack their own workspace - why should you guys have to do everything? Unfortunately, NON-IT workstation users are stupid and do not always store company data on their assigned network storage, so your workstations are actually assets, that could be holding data that is not replaceable. Because you mentioned all of the items (servers, workstations, networking equip. and phones) in your post, I'm assuming skipping them is not an option. Go with the fastest - each employee packs their own workspace. Get support from management to make it happen, it would save you and your team time as well as expedite the evacuation process. Each employee packing their own workspace also ensures they grab the things that are necessary that may be hiding in their desks/around their workspace. The rest is up to you. Really, and I know you've probably heard this 100x and read about it several times from the comments above - but the organization is not taking itself seriously if it does not have any type of protection from this type of disaster/condition. I don't have to spell out the obvious, but if the organization exists in an area that is prone for this type of disaster, then the business should have been designed to "hit the road" in the event this type of condition threatened the business... as well as protection and correction.
You have 10 minutes to load a 14 foot trailer with all your gear before the place burns down! I bet you're wishing you hadn't cracked down on employee "slacking off" with Tetris now! Oooh maybe we'd know how to load that trailer if our boss hadn't told us to get back to work!
I'm trying to teach myself to set people on fire with my mind... Is it hot in here?
Unplug the whole shit
Use a showel to load the truck.
Evacuate
Ask your kids to untangle the wires as an after-school game.
Packing rules should be made as part of your disaster recovery plan. But as a rule, I save the packing for large server and switching equipment. That way there is no issues with packing.
Workstations I don't save the packing after 30 days. In that case I would plastic wrap them and ask employees to assist if due to circumstances all available shipping options are exhausted.
Good luck.
Hack
Got Geometrodynamics? Awe, too hard to figure out? Too bad.
Shipping Containers. Put your nodes on grid-by-grid basis inside the shipping container(s). Ship it offsite. Of course, having DR plan with out-of-region, target site would have been prefferable
We are not talking about a single computer. The guy is talking about hundreds.
Sure take a picture .... of hundreds of same colored cables. Yeah ... that will give any clue.
Any competent IT person would LABEL each cable before connection.
Yeah, wrap ESD sensitive hardware with carpet. Even ESD ready can carpet produce electrostatic discharges if not grounded properly.
Great idea .... NOT!!.
I know this won't help the OP now, but If you live in a disaster-prone area and you could ever be hours away from a sudden evac, consider setting up your hardware as if you were doing a mobile installation. 19" rack cabinets on casters, which can be quickly pushed to a truck. Note: just because you have wheels on your rack doesn't mean they're meant to be moved when loaded down with equipment. But the appropriate equipment. If you want to see something like this in action, go to a major televised sporting event a couple of days before the event starts and ask nicely to talk to the tech manager.
I would leave all the cable behind really, no time to pack those up, just gets annoying. It might actually be easier to go through them with some hedge cutters (after you turn off the power off course).
If you have small racks to move, just make sure they're free to move and put them on a cart with casters, just be careful they don't tip over but a decent forklift will help with that. That only works for racks that are not fully used (like half racks or smaller). 42U racks that are filled can't be moved, get a battery powered drill to unscrew the devices and pull them out. Don't waste time unplugging the hard drives or anything as some others mention, your data should be backed up and hard drives that aren't spinning can withstand quite some abuse.
Really, why are you even moving stuff out? You should already have a backup in place for your most important stuff, with automatic fail overs.
Custom electronics and digital signage for your business: www.evcircuits.com
Short answer: you don't do what you're asking about, and you're probably fucked.
Do your disaster recovery planning in advance: make sure the servers are sufficiently backed up they can be brought up elsewhere at short notice (VMs you can bring up on Amazon are ideal for the purpose) and insurance to cover your workstations.
The time to start thinking about this is about a year ago. I realise there are many circumstances in which doing this is not feasible, but that doesn't make you any less fucked.
http://rocknerd.co.uk
You probably won't get it very well done.
Don't forget to document you failures after the fact, that will serve a lot of other people in the future.
Step 1: Encrypt all drives that contain sensitive data. Yes, all.
Step 2: Make sure there is a single switch that can turn off the power to all computers that contain those drives. Yes, all.
Step 3: Make sure you have working offsite daily backup (encrypted too, of course).
When the emergency occurs, flip the switch and walk away. Your backups are already safe, your hardware is insured, and with the power off and the disks encrypted no one can break in and steal your data while you're evacuated.
Dude, I'm in FC and down w/ the D/L. If you need a place to bring all that stuff - email me.
Use a virtual desktop infrastructure with dumb terminals (aka zero client). VMView comes to mind but Xen may have a solution that works for you too. You don't have the move servers that reside in a different location.
Walk out. Don't spend a second of time worrying about hardware, and restore from your already existing remote backup if the fire levels the place.
If you don't have remote backup, you're a moron.
You don't really need much packing. Just make sure everything touches and "nests" so items don't slap each other much. Flexy shit like power cords can cut down rattles.
"This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
You have offsite backups and insurance to cover any damaged equipment, so whats the problem? Its not a disaster, its an excuse to upgrade. If you don't have offsite backups and insurance, well son, cut the power, break out the chainsaws, cut the wires, quit wasting you time posting to slashdot, and hope your customers/investors don't get wind of your incompetence.
Good luck
"17-foot U-Haul? We have limited packing supplies."
Do a quick round up of peoples bedding supplies: box springs, mattresses, blankets and pillows.
Build all your workstations and servers onto racks made from hand trucks, you know the ones with 2 rubber wheels that you use to move fridges and the like. You can build little shelves etc onto them with places to hook all the various cables then when you need to move you simply unplug the cables and wheel them out the door. It's cheap and effective you don't need people tripping over themselves trying to carry heavy bit of a gear onto a truck. If the gear already has wheels on it it's much easier to move safely.
You do have backups of all the data, including from workstations, at a backup site well away from the fire zone, right? If not, grab the disks first and do nothing else until those are out of danger. Then deal with what will get you back on online and in operation somewhere, first. Don't bother what what insurance can replace.
now we need to go OSS in diesel cars
I'd first start unracking the servers, the SANs/NAS devices, etc. load those into your Uhaul first. Grab some extra blankets from the house to use to help pack em in. Secondly, id go for the network gear, the switches, the routers, etc. Third, id grab the CPUs and get all that crap loaded. Monitors and printers can be replaced. Pack anything else you can easily grab after that, and head for the hills! or well the hills away from the fire!
1. Put blankets (heavy ones for padding) on the floor of the truck, all over. Get at least 2" of padding.
2. Label everything, which hard drive goes in which position in each server.
3. Label all the cables, and definitely photograph everything. Get a copy of that SD card, make maybe 3 copies.
4. Pull the server hard drives and put them in a regular car, the truck's ride will be too rough.
5. Once you have all the servers and workstations in the truck, it hopefully will load down the truck and soften the ride a bit.
6. Wrap workstations in blankets as well, it will help cushion them.
7. The phone system is probably connected via 25-pair cables, which will disconnect in one pull. Leave the other cables in place, if the building survives that will allow it to be put back in service quickly.
8. Insurance would be great, but it won't cover everything. Make sure you have coverage on the truck, and coverage for the materials being carried. Get the full damage waiver, the likelihood of someone bumping the truck is pretty high in this situation.
9. Offsite backups etc. are also wonderful, but they need hardware to put it on. What you're doing is decreasing the time needed to get back online by at least a week.
10. Make sure not to put yourself in danger, get out promptly and don't take any chances. Make sure you have a full tank of fuel in the truck as often as possible. Make sure if it's a diesel to pay attention and use the right fuel. Gas in a Diesel tank will ruin the whole day.
11. Make sure to bring water and food for the drivers, and expect to be driving in areas where all the gas stations are closed, and no new resources are available.
12. I'd leave the rails from the servers behind, leave UPS's in the racks, leave monitors and keyboards. All of that stuff is easy to replace.
General rule, heaviest stuff on the bottom, try not to stack stuff up high. Get a nice heavy padlock so you can keep the truck locked, no point in letting looters grab your hard work out from under you. Combination locks are nice but make sure at least three people, especially ones not in the evac crew know the combo.
Get a Data Centre Container from Rittal (there maybe other's that do a similar concept). Use it as your server room. Store on a back of a truck (or have a forklift/crane/lift setup to get it onto a truck). In an emergency, cut all the hard lines running into it (mains power and network), drive it to the new loc and plug it in. Make all your workstations laptop's and everyone who uses one has the responsiblity to pick it up and carry it with them.
All you need at the new site is basic structured cabling and extra fuel. The DCC is basically a server room in a shipping container, so has UPS, cooling, racks, lighting, generator all built in.
(A) have employees take their desk stuff... the laptops/desktops. four wraps of bubble wrap, and go. screw the rest. there are usually a bunch of apps and work files on individual PCs that are irreplaceable. some are even business-critical. but it's THEIR machine, they're up now because it's set up and ready to go. add DSL/cable/whatever and they can work from home.
(B) send somebody west with backup set A. send somebody east with backup set B. then take out the servers and go. if you have very freaky setups in firewall appliances and routers, bring them, too. I like the idea of cheap-ass mattresses. fill the floor, and set the racks flat on their backs on the mattresses. yeah, it's overkill, but you have your inter-rack wiring, etc there.
(C) insure you have critical business papers... server and software licenses, articles of incorporation, insurance policies... with you.
screw the rest. employees can go anyplace for a monitor and keyboard/mouse kit. you can get a X-pack of twice-recycled cell phones at any of the corporate stores within 12-24 hours to operate on temporarily. if you can't get into building space at once and need to operate the business, haul the server room stuff to an ISP with colocation facilities and have your MX record transferred to their pipe.
if this is supposed to be a new economy, how come they still want my old fashioned money?
If you can afford it, and if you have time to bring in outside people, think about doing this:
Label the equipment that you want to save. Make a video of the equipment, with the labels visible, and with an employee verbally describing the equipment, make it easier to set up the equipment again. Then send your employees home.
Bring in IT people that you trust, who live outside the fire danger area. Hire them to pack up your equipment, papers, etc. Have them bring plenty of packing and padding material.
Because these temporary IT employees live outside the danger area, they can concentrate on packing your equipment. They won't be glued to their phones, talking to insurance people, or arranging to evacuate their families.
Assumes you are doing this in a few hours.
1 Get out early, don't die in the fire.
2 Get somebody else to do these steps while you think of other useful things to do.
3 Take pictures of everything - lots and lots and LOTS of pictures. Zoom in on labels. If needed to identify the system (back of rack may not be informative enough in a picture) put a postit note on that gets photographed saying "server rack #3" or whatever.
4 Try to bundle and leave the cables where they are after disconnecting them, use tape to quickly bundle them into place. You can scrape the tape off later.
5 Load the data into your u-haul first.
6 Leave a sprinkler running off a garden hose up on the roof. Close all the doors and windows.
If you haven't done a `copy run flash` or whatever to save your cisco configs now would probably be a good time.
If Not. Any former active duty military on staff? As them to help orchestrate.
People first!
ORDERLY shutdown everything, then physically disco the UPSs. If staff is still on site and bugging out in their own vehicles, consider having them evac their own workstations. Your shit is already in the wind, this might save _some_ things you might otherwise miss.
The C-level officers and their secretaries PCs are important. They are not the same priority as joe shitbag in marketing. Prioritize. Printers, monitors, etc do NOT matter.
If you have one of those fancy document center (printer scanner fax wtf-ever) gizzies, and YOU HAVE TIME, rape the HD out of it. It has more juicy data than you would believe....
Servers. If you can grab em all, do so. Label EVERYTHING.
If not, grab the drives, as others have advised, and LABEL EVERYTHING. Package as well as you can. Ziplocking each drive is not a bad idea, and gives you the op to label the bag. Raid your shipping department for packing material, and when you run out, rape the padding in the office furniture.
Remember, people first!
Network infrastructure is less important than your corp data. All that being said, if you have time, now is a good time to dump the configs on the routers, firewalls, etc. to HARDCOPY to take with.
Same applies to the PBX.
Have fun....... and quit reading /. when you need to be saving your bacon!
Plan ahead next time, OK?
Red
assuming you did your racks right, you shoudl be able to unplug power one or two other cables, and roll them onto the Uhaul. Simply strap them down and go. If you didn't, I'd just unplug the connecting bits (leaving the business ends in the machines) and plastic wrap them (heavily) Anything non-rack-mounted is, in my opinion, not serious computing, and therefore a casualty of the evac.
as for workstations and the like, I typically like plastic wrap... lots and lots of it. You can use it to wrap cpu, monitor, keyboard and phone together. stack 'em deep and high between your racks, and throw ample blankets between layers.
Save your data and your people
but not in that order.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
1) If you aren't prepared for a tornado, fire, or other disaster to take out all of your equipment suddenly, you are't prepared. This means having offsite backups and a plan to buy replacement hardware and restore data within a reasonable period of time, or make the business decision that you can live with losing "everything" from an IT perspective.
2) If you have the luxury of a few extra hours to evacuate, shut down the servers and take the drive or the boxes that the drives are in to the car. But this is just a bonus, so you can save time rebuilding later.
3) If you have to ask this question when you are in the "pre-evacuation" zone, it's far, far, too late to be asking the question. DO come back next month and do some tabletop disaster-scenario exercises.
Knowledge is how to play a game, intelligence is how to win, wisdom is knowing what game to play.
Your data is worth zero to looters and will be overwritten by pr0n anyway.
"What is the best way to load 50+ workstations, 6 servers, IP phones, networking gear, printers and wireless equipment into a 17-foot U-Haul?"
Elbow grease.
Alternate answer: talent.
Alternate alternate answer: you should have asked Slashdot (or a consultant) when you were putting together your DR plan. That's not something you want to throw together at the last possible moment before your shit gets disastered.
If that were my network, I would have arrange proper insurance to start with, that should include finance to have equipment purchased to replace those destroyed by fire. I would also have regular (And corrrectly tested) backups stored off site. With a bit of immagination, you would have set up regular dumps of your data to a remote location, so the whold u-haul would have not been necesary (If you use some sort of *nix, you can put a couple of rsync lines in the crone and you have nothing to worry about.....)
Insure and ignore the workstations.
Get a hold of a transportable container and mount the servers/critical machines IN IT. Like sun was/is doing with server farms in containers. When the warning comes down, you just safely tow it with you. It might take some (basic) engineering to ensure proper power supply and ventilation, but shouldn't be hard to do.
Don't wait for the disaster to arrive. Buy a trailer and move everything into it now. If you don't have a garage door on your data center, build one now and back the trailer into it. When the data center is inside a trailer, running away becomes simpler.
1. Disconnect the network cable (one high-capacity fiber link)
2, Disconnect the power cable.
3. Disconnect the external air conditioner that blows cold air through the trailer.
4. Open the garage door
5. back in the pickup truck and hitch up the trailer.
6 DRIVE!
Make sure you have the right insurance. Network switches and air conditioning and anything that stays put, needs normal fire and contents insurance. But the stuff in the trailer needs insurance that covers moving it, the potential for a car crash, etc.
Make all your PCs do network boot, and mount iSCSI volumes for their storage. Enforce it by removing all PC hard drives. That way all your data is in the trailer and is safe. Make sure that all the documentation about PC configuration and network wiring is also in the trailer.
For next time...
http://en.wikipedia.org/wiki/Firebreak
http://en.wikipedia.org/wiki/Firewall_(construction)
Taking off the tech hat...others have given good advice for triage on the equipment, etc.
Safety of you and your staff is the priority. Anybody who does not need to be there should get to safety now, before the evacuation order comes. The fewer people to be accounted for, the less chaos there will be if you need to leave on short notice. Make sure to exchange cell phone numbers (if you haven't already), so you can rendezvous with the others later on.
Assign a person to be the safety monitor--to stay at the phone and radio, and get the word out to all others if the evacuation order comes. Vigilance is their only assignment. The safety monitor should have a list of everybody still on site, to be sure nobody is left behind. If you get the call to evacuate, you'll have only minutes to get everybody out and down the road to safety.
Have N+1 vehicles ready to go, where N is the number you need to get everybody to safety. If one doesn't start, don't mess with it--leave it behind and call the insurance company later.
You should have two ways out. If one becomes impassible because of the fire, head down the other way immediately.
I hope you and your center escape the fire. But if it comes to a choice between you and the data center, let the servers melt!
Forest fires requires trees. Cut everything down for a couple of hundred yards around valuable structures, and keep any brush and growth cut in the future.
"This post is an artistic work of fiction and falsehood. Only a fool would take anything posted here as fact."
First, I hope you have offsite backups. A just in case is better than no fall back.
If you're in a hurry and limited on resources, have your people unplug their own equipment and bundle the cords with it, and tag their phone, tower, monitor and cord bag. If you're moving workstations and servers in a uhaul, I'd pull your disks and pack them in a couple coolers with padding. It's easier to bad the insides of a couple of those than to cushion a large part of a truck. Towers and servers without disks are a lot more able to handle the bumps and would only need minimal protection around them to stop them from sliding/falling. Do splurge on a few of the big padded things Uhaul offers, and don't be afraid to get a couple $20 air mattresses to use as inflated padding too. Protect them from sharp corners and they work very well. Bags of tagged cords can be put in garbage bags (or if you're taking garbage cans, fill them with them). Phones can be put in paper boxes and taped shut.
Big printers are more difficult than other hardware. They don't stack well, and no one keeps their boxes. If they're floor model printers, use webbing to secure them to the hooks/catches on the uhaul walls.
you'll get locked out of the building as part of phase 1 response.
go and buy some doubletake or setup rsync .. if you are worried about it do it NOW...
even after a hurricane I got locked out of an office in florida for over a month... zero damage to the building too..
Don't load ANY of your gear into a U-Haul truck. Move it if you must, but step one is to get a truck from another provider, because U-Haul vehicles are almost always rolling piles of crap with bald tires, crumby brakes, sketchy steering, and a million other problems that arise from the complete lack of maintenance they receive. Get a decent truck (from Budget, Penske, or ANYONE other than U-Haul, and consider a professional mover) unless you want to take the very real chance of having a breakdown and having to unload/load everything again. Seriously, it would be like using duct tape to hold that loose wing onto your airplane because you are only flying 50 miles - it might just work out fine, but that is a chance you need not take.
Actual vehicle choice aside, you may find a professional moving company that is well insured and capable of handling your delicate equipment. They work quickly and efficiently and are reliable. You do the unhooking and setup and let them handle the transport. Tell them exactly what you are moving and why, and ask them to explain how they will secure everything in transit. I am a major proponent of DIY everything, but I am also a big fan of professional movers.
If it is too late for all that, get everyone to back their minivans, SUVs, and "sport" wagons up to the door and start chuckin' the good stuff in the back... and hope your backups and insurance are in order.
This is a hacked account, for which the owner can not be held responsible.
...leave it. Phones. Printers. Networking gear. If it doesn't have a hard drive, leave it behind.
Take the servers. Take the dozen workstations and laptops most likely to have stored mission-critical work on local drives. You already know who the knuckleheads are. And then go.
Let insurance take care of the rest. If you've time, take some pictures so you can prove the state of the office. And if you're self-insured, well, now you know better for next time.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Backup everything one last time, and send the backup away to a safe place. If you have time to pack the computers, put padding between them and the walls, and strap them in (hitting the walls is what is most likely to do damage -- the sudden stop). If no time to pack the computers, pull the disk drives and label them. Plan to take no more than half the time that it is estimated that you have. Later tell the county to encourage annual brush burnoff.
You should have offsite backups and a disaster plan already in place.
Well no fucking shit, that wasn't the question and above all completely fucking useless this time.
Stupid fucker really needs to me a mod option.
Why not setup your server room in the 17ft u-haul?
Spent a few weeks up at RMSC myself years ago. If the hard drives aren't SSDs, bumping them around in transit and while packing/unpacking might risk head crashes. Since you have a lot of them, cushion the drives on a layer or three of the zabutons you have lying around.
I really hope the stupa survives the fire. Please follow up after you're able to return, and good luck.
Offsite back-up is right, like in the cloud. That way, once they get new equipment, they can always restore what's been lost.
Like in the cloud, you mean MegaUpload?
I was promised a flying car. Where is my flying car?
If you want your drives to fail, put the servers in a u-haul without any protective packaging. Just pull the drives pack those carefully (label them first) and then start putting in expensive small equipment if you have time and no insurance. You do have your backups offsite already, right?
I was promised a flying car. Where is my flying car?
A container costs $300 or so. Cover it with red lead paint. Dry line it so it is pleasant. Cost $2000 maybe so far.
Put air conditioning in it (that's the bit I don't know).
Keep your valuable stuff (the rack mounted servers) in the container.
If you have loads of space left, have a place to put the workstation carcasses in.
When the fire arrives, everyone picks up their workstation carcass, puts it in the container, you unplug it from the mains, pick it up and drive away
(For the snarky commentators I say the submitter has to do the best he can in a difficult situation. Asking /. may give insights, DOs or DONTs so is reasonable thing to do.)
When you have your plan then its a matter of implementing it. IMHO the best way to get many hands working together is to give a clear objective, basic instructions and definite parameters. For example "Normal business will stop at 3pm. Get the home phone numbers of your work colleagues. etc. etc. "
As far as getting the tech out I would aim to steal enough with the objective of having a working system set up somewhere else within hours and then doing it in say your living room. This objective, ie. a working server/router etc. and this method means the next day when you go back you're a lot wiser about the details that matter.
(Obvious, but people don't think of these things.) Park your transport, whether private cars or trucks right close to the place where you're taking stuff from. Get a tail-lift truck for heavy gear.
The old houses used to have storm shutters. These were wooden panels that could be closed and locked in place over the glass windows. Unlike boarding up a house with plywood, a hammer and nails, the storm shutters could be deployed in minutes. You'd think they'd catch on again in hurricane zones. You do see some fake shutters from time to time, but these are useless decorations nailed fast to the side of the house.
Beta is broken and the link to classic doesn't work. Stop wasting our time or there won't be anybody left here.
Since you apparently don't have one, and assuming this really is an emergency situation, try the following:
- Prioritise kit that holds irreplaceable data: anything else you can work around
- That means don't even consider shifting the phones, printers, displays... unless you find you've got a lot more time than you first thought.
- Consider the impact on the whole business. If the fire hits the Centre, how long for the rest of the business to recover? IT may not be a priority in the Big Picture.
- Keep calm, do what you can, stay safe.
I've done more IT Security Risk assessments (Australia) which which involve these sorts of fire threats than I can remember and frankly, you don't, don't evacuate a network that is. Get insurance. Equipment can be replaced, data can't. Focus on data. I've not seen one cost-benefit analysis which shows that having portable equipment and infrastructure to support any IRP/DRP is more cost-effective than proper insurance with proper data protection in place but you never know, you may be that unique scenario. Of course none of this helps you now but after this is over this should be your priority.
Build your entire environment into standard shipping 'containers' that can be lifted onto a container moving truck and if the order comes to evacuate, the container(s) go on the truck(s) and off you go.
That being said I am completely with those who say leave the hardware behind and get out.
blindly antisocialist = antisocial
...do you mean a fuckton of weed? Because if common sense ruled, you would know that PROPER off-site backup and insurance is the answer.
Don't. Buy fire and theft insurance and leave the kit there. Buy new kit if the worst happens.
You could consider removable disk cadies so you can remove all the data in a real hurry if you have to.
To The Cloud!
All of your workstations should already be image-backed to one of your servers daily, so there should be no need to take any of the workstations with you. If one of your servers has the space, I'd be scrambling to at least get a windows image backup done right now. At least then you'd just be throwing the server boxen in the back of your station wagon since you probably don't have an off-site backup plan.
If it's simply a matter of funds and you can't afford this stuff, at least get some decent rackmount cases for your servers and put them in a 24U rack that you can just wheel out on a hand truck.
Insurance will replace the stuff left behind. If you don't have insurance, you deserve to be bitch-slapped.
I think what you have to ask yourself is:
If there was a fire on your back door, would you risk your life (or actually expect anyone else to) to protect the companies equipment?
Or would you rather drive home and make sure your family is safe?
If your asking this question then you don't have a contingency plan (a violation in some states) for disasters.
Let it all go, and spend you time either looking for your next employer (because they probably won't be very happy with you)
and learn about and develop a contingency plan for the risks that are likely in your area, including fire, flood, and possibly earthquake and weather.
Since many disasters can leave you without services for 72 hours (even if you didn't evacuate) there should be plans in place to set up operation either at another location or alternate mirror site within 24 hours at a place that is not likely to experience the same problem at the same time. Partnering with other companies, even competitors (yes, this does happen) can help reduce down time.
But all of this needs to be set in place BEFORE a disaster happens.
If there's a wildfire coming your way, everybody who owns a Civic is going to be at the truck rental places ahead of you in line. Your facility will burn before you even get to the counter.
Buy a truck and park it on the facility property. Or buy two small vans (which would probably have more computer-friendly suspensions) and keep them there. They'll be useful every time the company needs to evacuate ahead of a fire...or move from one office to another...or help the CEO move when (s)he buys a new home...or help relo new employees from out of town.
Anytime there's hardware that can't be replicated in advance in an offsite disaster recovery facility, you really need to be able to move it 24x7. And without the truck, your entire disaster recovery plan goes down the toilet. The truck is the critical path, and needs to be treated as such.
First, you should have off-site backups of your critical data. Put it in the cloud or some off-site company server.
Second, your life is worth more then the value of any of the equipment or data you are trying to save. You can always walk away from a company that lost all its critical data, but risking your life to protect that data is stupid.
Lastly, your employees (and you and IT?) will most likely love it if all their equipment burned to the ground, chances are they are running ancient crap (like 99% of all companies that want to save a buck by NOT investing in the software and equipment their employees use every day), so if the company burns down then the insurance will pay for everyone to get all nice new shiny upgrades.
Life is the most valuable so if you see smoke, leave the equipment behind, help others and run away.
I haven't thought of anything clever to put here, but then again most of you haven't either.
not sure what the best answer is, but sponsoring a contest and allowing multiple people to attempt to actually solve the problem ( maybe with sets of mock equipment) and giving them a nice prize for the solution would be a good way to find out.
Make it easy to move.
Blade systems are one way. Two people to pull it out of the rack, and a dolly ride to the truck, and you've just moved 20 systems. Repeat twice, you've got all 50 out of there. Not to mention, since the blade chassis usually have internal networking, there are minimal cables to unplug and reconnect.
For desktop, virtualize them, and use $300 PCoIP clients. *IF* the employees want to take them, it takes two minutes to unhook, and it might even fit in some pants pockets. The admins can carry out the blade chassis that has all of the disks and CPUs, again on a single trip with a dolly.
With such a setup, having EVERYTHING out inside of 15 minutes is feasible. If you have pre-assigned teams to parallelize moving the servers, 10 minutes is quite doable.
But, as others have said, consider just letting them burn and replacing with insurance money.
If you have a solid backup system, you should already have a disaster recovery plan wrapped around your off site data, and a backup Tape Drive/NAS if you can grab it it. So when the bell rings, you walk out...and grab a most current backup device on your way out if you have the time. You should be able to start recreating core services at a new site. The way to avoid trust issues is to never need to trust very much.
Don't give me an excuse to make fun of Sham-bhala. It's not fair.
I though you guys were going to meditate and magically put the fire out.
Noblesse oblige. I pray for your continuing service as a savior.
Hardware is replaceable, data is not. So:
All the workstations would be thin clients - no data loss if someone forgot a workstation.
I'd mount the servers onto racks on wheels that I know will fit into the truck. They are also chained to the floor to prevent theft. In the event of an emergency the servers are unchained and turned off, then rolled out the door and into the vehicle. I would test this evacuation once a year. Possible bonus: You can now rearrange the server room with greater ease.
Writing as someone who knows nothing about networks or fire or anything even closely related, but I am a bridge player, this is taken from bridge, and I know a bit of game theory...
I would act on the scenario that minimizes your loss.
If the fire doesn't reach you, you don't want to have a big loss from having to build everything back, debugging everything from scratch.
Don't take apart too much that is hard to put back together.
But, be ready for the worst case. Take the data, everything easily movable and easily put back in its place.
What you should really be doing is minimize loss incurred over all actions, integrating for every action over the probability of every possible outcome. The strategy I outlined above is supposed to be an approximation to this exact formula....
What I'm seeing here is a disagreement on the time frames involved. I liked one of the posters who mentioned 'acute emergency'. A wildfire miles away, with the area being declared a 'pre-evacuation' zone indicates that there is not only no exceptional danger yet, but that there's an emergency alert system in place to tell you BEFORE it becomes acute.
In which case acting to protect property is still on the cards. Choices include 'shelter in place' and 'evacuate'. For a fire, the answer is generally evacuate for people, but the equipment can still shelter.
The ideal time to be planning for an emergency is always NOW. Plan for fire even if there's no wildfires in the state, no fire in the building, but if it's too late for that, you can still improve your situation by doing the planning. A bit more expensive, results aren't going to be quite as good, but it can still be done.
Long term(fire not expected other than general threat), if you're in an area where wildfires are a possibility, involves turning your building into a shelter. Don't put vinyl siding up, put up aluminum or even steel, backed by the appropriate fire-resistant insulation. Install fireproof shutters. I've seen vinyl siding melt with just radiant heat; once it's gone the wood behind it can catch. Clear an area around the building, short grass or better yet fire resistant plants such as Sage or Yarrow for within 30 feet of the building. Remember, keep the plants trimmed, and water when fire is expected!
Medium term(fire is expected within days) - too late to change what's planted, your roof/siding, etc... You still have time, depending on your available resources, to remove dead vegetation, trees too close to the building, mow the lawn short(and the only time I'd bag and remove the cuttings). After that, wet everything down to protect against cinders.
Short term(hours) - remove what dead vegetation and trees that you can, set up what water you can, leave early so that you're not fighting with the other last minute evacuees.
Acute - leave already! Forget everything except maybe the backup tapes. Which should have been in an appropriate firesafe/cabinet already. Remember: A firesafe rated for paper is NOT rated for magnetic tape for the same temperatures/times. The average 'fireproof' safe or filing cabinet is only rated for paper. You can get removable type drive bays that are rated for fire/water. But at 15 minutes of protection, they're unlikely to work against a wildfire sparked building fire.
I don't read AC A human right
Have a "fire" sale on the equipment, once you removed the hard drives!~
Tweet that you need help moving the stuff and pass it on. When we had the big fire in Boulder pretty much everything was handled long before official responses but people on the #boulderfire twitter feed. Ask everyone in your social network to retweet the request. You'd be surprised how effective it is.
Every rule has more than one consequence.
Hi there!
I am retired IT - a former US Forest Service IT director living in the mountains near Golden. Do you need some physical help? I can be at the fire scene in about one hour?
I'm sure you don't have enough boxes. Consider this: rip the cushions off all the chairs and couches, and cover the floors. Actually, you should be able to get heavy blanketing for the floors. Rolls of bubble wrap are sold at UHaul, too. Cover the floor. Flatten your carboard boxes, then pack the mess on them. Save a box or three for the absolutly critical machines. Then more heavy blankets on the sides and top, and rope it down, tight - you don't want it all bouncing, I think.
PULL OUT THE DRIVES of anything that has hot-swap, and package them more carefully, and put them in your car.
That's what I'd do.
mark
Rent a wood chipper; should be able to fit all that in a 17 foot Uhaul no problem.
Seriously, stick them into a big cement hole with some sort of water-cooling or other method such as heat-transferance into the cold, hard earth.
Then again, I suppose you don't really need computing power during a bushfire... so shut them down, close the cement door, and drive away like the Dukes of Hazzard!
You can come back after the flames. With a shovel. And some water, cos you'll need it when shovelling the ash away from the front door.
Go update your resume and start job hunting.
Put everything in cloud. Pray for rain. How.
Defining Statistics and Social Research
Seriously, how do these questions even get to Slashdot? Call a moving company or PODS and tell them to pack it up. Problem solved.