Slashdot Mirror
Outfitting a Brand New Datacenter?
An anonymous reader writes "We completed our new 4,000 sq. ft. data center (Tier II/III, according to The Uptime Institute) and just recently moved our core systems from our old data center to the new. We've been up and running for several months now and I'm preparing to close out the project. The last piece is to purchase some accessories and tools for the new location. The short list so far consists of a Server Lift, a few extra floor tile pullers, flashlights and a crash cart. We'll also add to the tools in the toolbox located in one of the auxiliary rooms — these things seem to have legs! What are we missing? Where can we find crash carts set up more for a data center environment (beyond the utility cart with and LCD, keyboard, and mouse strapped to it)?"
You will probably need a series of tubes.
Ear protection
O2 masks for when the Halon drops
arrows on the floor directing people to the nearest exit
a 'Battleship' style row/column marker for every row/column of racks
near-Draconian access control policies
I want to delete my account but Slashdot doesn't allow it.
"We've been up and running for several months now" ... "What are we missing?"
You've migrated several months ago - if you don't know what you are missing yet you either haven't been paying attention to what you need locally or haven't been paying attention to the recent news. Any small items you've probably already needed and know about.
Large things - like accomodating power outages (see 365 Main St) need to be prepared for. I'd guess after a successful migration you've likely covered most things.
At the DC I work at we have a crap load of extra gear. Make sure you have one emergcy kit in your core room, ensure that no one is to use it unless it is an emergmcy. The kit should have but not limited to the following: screw drivers mounting screws/cage nuts knife (a Leatherman multi-tool) spare patch/cross-over cables (Copper) (various length) spare fibre patch cables (Various length) Cable tester (Copper/fibre) couplers for fibre fibre cleaning kits Patch panel punch tool spare hard ware for core gear We have more gear however i'm drawing a bit of a blank as I haven't needed to look at the kit for a while.
And a good sized crescent wrench. Absolutely indispensable.
Drop it across the terminals of one of your backup batteries -- when it's disconnected from the grid. When the wrench cools off, store it in a safe place. Makes a great scapegoat when things go wrong. Could save your career...
I've used crash carts from a company called Ergotron: http://www.ergotron.com/tabid/158/language/en-US/d efault.aspx
:) (@ 90 watts/sq foot of cooling). But they did not(at the time, they wised up July of last year and now strictly enforce their cooling capacity at this particular data center).
At my current and my past company, they work real well. I looked high and low for a good crash cart and nothing seemed to come close to these. Maybe I was just searching the wrong terms(and apparently my vendors were too). They are a bit pricey though, ~$1500 or so to start. I have a Styleview LCD cart at my current job, and had a LCD cart and a laptop cart at my last place (servers were co-located in a ~900 sq foot cage, 8 feet between rows, so plenty of space for the carts).
I also bought a KVM over IP/CAT5 solution from raritan(http://www.raritan.com/), which worked out real well for those situations where a serial console wasn't enough(unless you have fancy out of band management, some do, some don't). I setup tables in the front of the cage, hooked up a couple of the raritan hardware clients. Typically ran one CAT5 cable w/KVM hookup to each rack, so it could be plugged into any system fairly easily. Range of 1000 feet. This was pretty pricey too, with the adapters and all it was about $25k. Though in the grand scheme of things it was cheap at the time. I had cyclades terminal servers in every rack, with serial consoles on all the servers and network gear.
Also I hooked up a temperature sensor board, from Sensatronics(http://www.sensatronics.com/) I think. I think it was a 16 port board, and I bought all 300 foot cables for all of the sensors, and cut them to length. This ended up being about $5k I think(I went way overkill on the cable lengths).
At my current company we use servertech(http://www.servertech.com/) PDUs, their higher end models come with optional temperature/humidity sensors so we use those instead of the senatronics.
Despite it being a co-location, we had 500kW of power going into that cage(standard setup was ~12kW/rack), if the data center had followed their own procedures(AT&T enterprise network services), we would of had to have about a 5,500 sq foot cage, comparable to your data center
posting as AC, since I don't have an account. I read slashdot daily but I post maybe once every 2-3 years, so I haven't bothered to make an account.
This is a little late a troll for slashdot opinions after you've built the place, isn't it?
http://www.ariamedical.com/crash-cart-inventory-ch ecklist.html
Under the influence of Post-Cyberpunk Gonzo Journalism
A zero point module to only be used for defence of the datacenter , powering the shield, or incase of power outages.
My UID is prime is yours?
We are using a Presto Manual Stacker as a server lift at my Data Centre. Ours is hand cranked, but Presto also makes an electric variant that would be suitable.
...a time machine, preferably in a Faraday cage (to shield your data center from unwanted interference), so you can implement the necessary changes a couple of months ago.
You need a monkey. Why? If a monkey can manage to bring down even a single server, you've not secured the place enough.
Colin Dean Go a year without DRM
For those times when the internal security system is working, but not according to spec...
Just junk food for thought...
Get a nice comfy Plantronics headset for the POTS line nearby. In a noisy datacenter, while on a mission critical tech support call, the last thing you need is your hand pressing the phone to your ear and/or crappy cell phone audio.
Would be a middle-aged Scottish man to sit in the middle of it with an intercom to say "She canna' take it any more!" when usage gets high.
Great Intellect...
Unless your data center is full of doctors, you should really use an AED.
-Peter
We recently had a catastrophic failure of our data center. We had a planned generator test that went horribly wrong. Unfortunately we'd added so many computers that our battery backups only had 10 or 15 minutes of power. Unfortunately the computer operator missed something and everything went down hard. Except for one computer system.
The Tandem that houses our main clinical application had this big array of D batteries. We'd always made fun of the administrators because of it, but miraculously it stayed up when everyone else went down. I now bow down before their primitive greatness.
$50 is not too much to spend on a screwdriver. Especially when you want it to work 100% of the time and work smoothly. I can't count the number of bogus $25 "ratchet" screwdrivers my bosses used to buy from Home Depot. I've had my Snap-On for 10 years and I'll probably have it many, many more.
Plus, if you have decent tools, there's a chance people won't destroy them first chance they get and not replace them.
I like music
Every Data Center should have a Hybrid at its center. After all, your data center will be run by human-looking machines, right? :)
Talk to Michael Vick about that.
I've seen several good suggestions already with specific suggestions on tools or parts. Start with those. My suggestion is quite simple, actually: Why GUESS what you need, when you can find out for sure?
Tear down one ENTIRE rack. (Or several, if they have any variations.)
Now, look at this big pile of parts in front of you and imagine what you would do WHEN *ANY* one of them breaks.
Get several spares for each of those parts and put into the cart.
Whatever tools you needed for disassembly, put into a crash cart.
Then make another, identical cart. When the brown stuff hits the spinnie thingie, and multiple systems are down, the last thing you want to be doing is fighting over tools. Get spares of EVERYTHING so at least TWO people can work on things at the same time! You'll thank me when there's two of you trying to work on both sides of a rack.
NOTE: Be sure to inventory what you put into each cart! Tools have a way of growing legs and you want to be able to check and make sure that you STILL have ALL the tools.
And please consider getting a big-ass UPS for your cart (At least 1KVA). If your power is wonky, you want to be sure your cart's equipment (laptop, hub, switch, router, etc.) won't be flaking out as the power comes and goes. Even with the power out, you can plug one server into the UPS and restore/repair it while the power is still out. While you're at it, also get some LONG extension cords (100-foot) made of AT LEAST 12-gauge wire. Plug the UPS into the extension cord.
Think you're all set? Now, using ONLY the tools on ONE crash cart, put the rack back together. With the power out. (i.e. no mains)
When you have done this, not only will you be CERTAIN that you have all the tools you needed to [re]assemble everything, you'll actually have done so and will have run into (hopefully) most of the problems that you could encounter.
That's it off the top of my head. Best of luck to you! P.S. One last thing: MANY rolls of Duct Tape! <grin>
I'd suggest extreme emergency supplies for situations where extra cables and backup supplies will prove fruitless.
This includes, but not limited to:
A bottle of whisky
A bottle of scotch
A glass
A Shotgun, pref with ammo
Sleeping pills
Pep pills....
In all seriousness, a good first aid kit should be in the center. Nothing sucks more than a dull headache and not having any asprin for it.
Plus, when someone cuts their hand on a server rack, it'll patch their hands up to keep them from bleeding all over them.
import system.cool.Sig;
tm
Support TBI Research: http://www.raisinhope.org
I've used a bunch of different crash carts and the best by far is the old standby AV cart with a monitor keyboard and mouse strapped to the top. Its cheap, it fits down the aisles and it has lots of convenient flat space on which to set tools, screws, cds and whatever else you need to solve the current problem.
- 23676--RUB9T28
http://www.eofficedirect.com/ProductDetails--895-
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion.
Shotguns fall under the category of LARTs, and we all know how useful LARTs are.
I agree 100% about the meds, make sure to have a stack of headache tablets (my personal preference is neurofen) in your first aid cabinet. Theres nothing worse than a headache when you are trying to solve a problem/disaster when your brain feels like it's oozing out of your ears due to the noisy machines.
Computer hardware isn't so much an issue - although, if you don't have some kind of maintenance contract, you want at least 2 of everything, up to and including 'entire servers'.
Depending on how much you're doing 'in house' things like cagenuts, spare cable management thingies, and tools to deploy said items will save a lot of grief.
Serial cables, and consoles, if you're running unix hardware. Get a set that you _know_ works. All too often you only ever need these when things have gone a bit wrong, which is entirely the wrong time to be wondering whether that's the right cable.
Spare UPS battery modules - if your whole DC isn't on a clean UPS supply, then you'll need standalone units for all your servers. And they will have batteries going bad, and it will always be a nuisance when they do.
Little labling utility thing, like a Dymo. The key to a happy datacentre, is to label and label and label. Even put labels on top of other labels saying you think this label is wrong, but haven't had a chance to check it. Label everything you can think of, with what it's for, where it goes, and who's in charge of it. Servers need hostnames, IP addresses, and anything that I might need to know about it right there and then. Cables need where they're going, and what they're plugged into. Go nuts with your labels, if I can't tell something just by looking at it, and I might need to know it 'here and now' then it should have a label with that information on it.
if your AC fails, you have a surpringly short time before heat will become a major problem. A portable aircon unit or 5 and some door wedges, combined with your largest sysadmins on guard duty, could save your bacon...
we use them here for medical electronic data capture carts. they'll sell you all the mouting brackets, arms, trolleys etc that you could possibly need. they aren't cheap, but they are good. i particularly like their "command post" system for putting pc workstations in hostile industrial environments...
Mod parent up.
If theres one thing I have learned from Slashdot it's this: YOU NEED A BEOWULF CLUSTER AND IT HAS TO RUN LINUX!!11
Armed Guards. With dogs possibly.
Lots of good ideas out there, here's a few more I don't think I saw.
Label maker. Preferably a decent one that can take different types of labels for different types of cables. Then go crazy with it. This will be a few hours of tedious work (but what are PFY, erm, Jr. Admins for?), and well worth it when you need it.
We have a large locking toolbox for our tools since they kept on walking. Funny, place is quite secure for servers, but screwdrivers ended up being left in pockets.
Talking about secure, if you have electronic access is it still locked during a power loss situation? If not, that's a problem. If it is, make sure all your admins have the key/punch combo/whatever to get in anyway.
If your lighting isn't on generator, a few wall mounted rechargeable flashlights to supplement the head-mounted battery one.
Good luck,
=Blue(23)
LITTLE GIRL: But which cookie will you eat FIRST? C. MONSTER: Me think you have misconception of cookie-eating process.
One of those little toy alligator mouths on a stick. The ones that you pull a lever in the handle and the mouth opens and closes. I can't tell you how many times I have wished I had one of those in my server rack - for when I drop things from the ladder, or want to grab the end of a cable, or hold a few cables approximately in place while I move something, or want to tap a coworker on the shoulder because he can't hear me over the constant noise.
The masses are the crack whores of religion.
A redundant datacenter at a different location.
Heck, my office is under the cafeteria. Forget plain old water, how about a soup spill (minor quantities, but messy) or a dishwasher malfunction? (major quantities, almost as messy, per litre.)
The living have better things to do than to continue hating the dead.
Some greedy bugger'll run off with it and use it at home in their Perpetual Motion Machine.
I would think you have a service contract for your power and mechanical systems but I would at least get:
- a drain snake and a drain cleaner. If you go to grainger, they have one that runs on co2 carts(like an air rifle uses). Condensation lines from air handlers will clog from time-to-time and the under-floor water detection system will alert you when you have a drain back up (you did get an under-floor water detection system?)
- a shop vac. to pull up any water should a drain back up.
- a multimeter with a big clamp. you can check if a whip is energized by just wrapping the clamp around the whip.
- POTS phones next to the cancel buttons on your fire system with a current contact list next to it. If you're alone in a data center using your finger to keep the sprinklers/air suppression from firing, you'll appreciate the phone being there.
- extra floor tiles & extra vents.
- install whips in your PDUs that match the longest length required that match up to your most frequently used connections (L5-20, L6-20, L6-30). Should you have a breaker fail on you in the PDU, you can run the spare whip anywhere on the floor and bring the rack back up.
- spare breakers in every size used in your PDUs
- a couple of big industrial fans. Helpful in any cooling system failure.
- an ir thermometer gun. To spot check temps around the data center.
STFU & GBTW
Don't forget the breathalyser! *hic*
1, 2, 3, 4, 5... That's the combination on my luggage!
I'd say you'll thank yourself if you have some of these items:
A Spot Cooler- If you have a CRAC unit crap out and need some coverage while replacing part(s).
Replacement compressor(s)
A variety of above floor fans, and below floor fans (in case of water under the floor).
As many spare breakers as they'll let you buy. (that UPS is no good to you with a bad breaker downstream of it).
Don't just get tarps, get these tarps.
Extra long load bank cables. Have your electrician make them up for you. If you make them extra long and store them onsite, you can use them to jumper out inside switchgear if you suffer a catastrophic failure (it might be ugly, but if done right, it can save your ass).
Flashlights that will work.
Hand operated pumps. If you have a pump fail and you need to get diesel fuel from your storage tanks to the "day tanks" of you generators, you'll be glad this is on the shelf.
A megger.
A phase rotation meter.
A good circuit tracer.
That's a pretty good start.
Troll? I think this should be Modded UP.
AskShashdot should be renamed ThinkForMe.
At worst, if this was a truly professional setup, these should be things people already know.
Dude, chill for a moment.
:D
Tell me something - if you're going to ask for casual advice on outfitting a data center, WHERE BETTER to ask than Slashdot?
I've built 2 from the ground up now. I don't mind sharing little nuggets.
When I have a question about how to put the finishing touches on a house, I don't see the harm in someone going to a home builders forum and asking for casual advice. Of course you want professionals doing the bulk of the work, but it's the little things that always get you, and we can certainly share those.
Karma: Chameleon (mostly due to the fact that you come and go).
Whoa there. Don't lose your cool - what the commenting system is for, ultimately, is an exchange of ideas, yo.
First off, it's not like the guy is asking for advise on critical systems like cooling or something. It's more like a last minute gizmo checkup(or a way to rid themselves of budget leftovers?) Some people can come up with things the "asker" or the other readers hadn't thought of - for example, it would never occur to me about tarps in a DC(credit: a few posts below) because mine is sandwiched between 10-12 floors of office space and an underground parking lot. But maybe to the next person it will be of some use. Besides, the lower one is on the n00bness-to-pro scale(and I am!:D), the more useful this kind of old pro information is.
P.S. Funny how appropriate my sig is today, eh?
P.P.S. Pedant alert - I took the liberty of correcting your title:) o-c-y instead of o-c-i-t-y.
Microsoft put the "sucks" in "success".
I actually had some batteries that were swollen into a UPS last summer. I had to tear it down to get the batteries out. I ought to take pictures of the needle nose pliers that aren't so need-nosed anymore from arcing. ANYWAY...
:D
After I got them out, the frame was rather disfigured, but salvageable, but I needed a hammer to fix it. Off to Home Depot I go...
What do you know...they have all kinds of small sledge hammers, but one stood out to me. It had a label on it:
"Engineer's Hammer"
Oh. Hell. Yes.
(and I don't swear!)
Fixed up the case, put new batteries in, life is good. The Engineer's Hammer stays at my desk at all times. All of the developers and salespeople are now aware of the Engineer's Hammer. I have made them aware that the Engineer's Hammer can fix just about anything, and FIX IT REAL GOOD. INCLUDING STUPIDITY.
Karma: Chameleon (mostly due to the fact that you come and go).
I have one of those. Unfortunately, it doesn't actually say "Engineer's Hammer" on it. I bought it because it reminded me of a certain mythologically excellent hammer.
.. They will go to my tool box and retrieve the correct hammer. Either that, or they will be chided for possessing insufficient leetness.
When I tell a tech, "Hey, pass me Mjolnir!"
Do daemons dream of electric sleep()?
The first time we had an A/C drain problem flooding under the raised floor, we had only a small shop vac to remove the water. Took forever, and I was amazed we lost no power. The next time we had a bigger shop vac-- more powerful, more volume, bigger hose, and we cleaned that sucker out in no time.
Make sure you have water alarms under the floor as well. Ideally they notify someone directly when there's a problem.
Flashlights by the doors. Preferably rechargeable, and plugged in. But at least available, and check them every so often.
I really want a transparisteel skylight in there, but no luck so far.
We had a water pipe spring a leak at one of our dehumidifiers (oh the irony), on a Saturday afternoon, on the wrong side of the shut off valve. It was a slow leak but, had been going for about 12 hours when I found it. The mops were locked in the janitors closet, and a shop vac really would have slowed the course of water across the server room floor, and made cleanup a lot easier.
09 F9 11 02 9D 74 E3 5B D8 41 56 C5 63 56 88 C0
Keep it current, keep it organized keep it available, and dammit Mike, put the pubs @$#%#@ back where they belong when you aren't using them!
Common hand tools in this box, commonly used special tools in that one. Rarely used tools in this other one. And dammit Mike, put the @$#%#@ tools back in the @$#%#@ box and put the @$#%#@ box where it @$#%#@ belongs when you aren't using them!
The main key is less in having lots of stuff than in keeping what you do have organized and available.
Once you try it, you will never go back
Yeah, we had a pipe from a bathroom above break once. Of course it was after hours when no one from that office was around to let someone in to shut it off right away. That accounted for the water coming in from the ceiling, through the lights and right into the top of the racks.
We also had storm water funnelled into the raised floor are where the power was all running. The outside flooded and apparently where some of the data conduits went out of the building, water could come in and then pour down onto the floor. I seem to recall a third incident as well, but it's been long enough that I'm a little fuzzy on the details, which I thus won't try to relate.
I do like someone else's idea to keep a shop vac handy. That'd help for a smaller water incident. Maybe some foam "dams" as well to channel the water away from critical stuff while you try to get it stopped. I imagine some of the packaging servers get shipped in would work well as a dam.
The party of stupid and the party of evil get together and do something both stupid and evil, then call it bipartisan.
Meh, he's just pissed he couldn't pickup a consulting job and "askslashdot".
Under the influence of Post-Cyberpunk Gonzo Journalism
... a guy could have a pretty good weekend in Vegas with all that stuff.
- Maj. T.J. 'King' Kong (dec.)
ceci n'est pas un sig.
We had a major problem with tools "walking" away when building our DC. Too many contractors, outsiders, and poorly-disciplined insiders. It was a MAJOR headache, with many hours lost hunting, deadlines missed, tempers and bad feelings, etc.
I went to the local big-box store and bought Plasti-Dip in five different colors. (Plasti-Dip is a liquid coating you dip the handles of tools into. It cures to a nice, thick, insulating layer, and makes for a good grip, too.) I assembled five complete sets of tools, gave each set a uniform color with the Plasti-Dip. I colored everything: hand tools, voltmeters, probes, cables, clipboards, everything. I assigned one color to each of our four techs. The fifth set turned out this horrible pink color and I set them aside as spares/loaners.
I had each tech sign for his complete set of tools, making sure he now understood that he was responsible for them, knowing how to use them, care for them, and knowing where they were at all times. I said the company would replace a tool if he turned in a broken one, otherwise it came out of his paycheck. I gave each tech permission in writing to refuse to loan out "his" tools -- I had HR put it in the Employee manual.
This worked pretty well. When someone lost a tool, they had to use one from the pink set -- which, of course, earned them a lot of snide comments until they replaced it themselves. Tool-care became a matter of pride. The tool walk-away problem declined, but never really went to zero.
Then, one day, one of our contractors tried to exit the building with a WAN diag instrument stuffed down his pants! Building security spotted the pink Plasti-Dip color. They didn't know what it was, but they knew tech took a horrible ribbing for having to use a pink tool. I got called to the scene, told the guy he was fired, and escorted him to his car. I get to his car and, looking through the window, see a couple other of our tools on his back seat! I very quickly did a search of his car (maybe not kosher, but I outweigh the guy by 80 lbs, so what's he going to do?)
Anyway, I come up with 9 or 10 of our items in all different colors. They were on the floor, under the seats, under the carpet, everywhere! And not just hand tools, but instruments of various kinds, specially cables, etc. totaling in the thousands of dollars. By this time, the contractor is yelling and screaming and stomping about. My boss comes out to check on this ruckus and immediately calls the cops. Soon, there's four cop cars in the parking lot, handcuffs, sirens, the whole bit.
As I'm walking back to the building, I can see people watching out the windows, crowding around the door and stairs.
"What's going on?" someone asked.
"He had a pink tool," I said.
I stayed with that company for another two years -- we never lost another tool.
Tarps will also come in handy when you decide to do anything drastic with your ceiling tiles. Granted, this may take several years in your new datacenter, but in the old one, when the boss decides that a cheap and easy way to impress the bigwigs on Monday is to replace all the ceiling tiles with nice shiny ones. On Saturday.
Cover all the racks and equipment before you begin, or you'll be sorry. Fiberglass and/or gypsum dust circulating forever, grime and dust and rat droppings hurdling toward every fan, vent hole, and orifice..
It no is pretty, my friend.
+1 Awesome
Check out my sysadmin blog!