Outfitting a Brand New Datacenter?

An anonymous reader writes "We completed our new 4,000 sq. ft. data center (Tier II/III, according to The Uptime Institute) and just recently moved our core systems from our old data center to the new. We've been up and running for several months now and I'm preparing to close out the project. The last piece is to purchase some accessories and tools for the new location. The short list so far consists of a Server Lift, a few extra floor tile pullers, flashlights and a crash cart. We'll also add to the tools in the toolbox located in one of the auxiliary rooms — these things seem to have legs! What are we missing? Where can we find crash carts set up more for a data center environment (beyond the utility cart with and LCD, keyboard, and mouse strapped to it)?"

Comments

Safety equipment

[Gothmolly]

Ear protection
O2 masks for when the Halon drops
arrows on the floor directing people to the nearest exit
a 'Battleship' style row/column marker for every row/column of racks
near-Draconian access control policies

Re:Safety equipment

[_Sharp'r_]

Tarps.

Lots of cheap plastic tarps that are stored out of the way but that you can deploy quickly in the event of a water event.

I know, you think you'll never use them, but if you do (storm leaks, broken pipe above, etc...), they'll be the most valuable tools you could have spent $100 acquiring a whole bunch of.

You just haven't lived if you haven't empirically tested (even accidently) how long it takes for power circuits under six inches of water to blow, or how those drop ceiling flourescent tube lights look when they're full of water and still going, or how long servers and switches stay up with water pouring down the racks into them.... :)

Re:Safety equipment

[numbski]

Having built a "ghetto" data center from used gear in an old loft, I may not have all state of the art, but here's some things I found that weren't immediately obvious at the time:

1. A list of what UPS' take what batteries, and contact with a vendor that can get them to you ground in one day without having to pay overnight shipping. This is priceless. Trust me. If you have a full-center battery backup this actually makes it even more important, not less.

2. A *REPUTABLE* generator service person, and a service agreement. I never realized how hard it was to find someone that was reliable, and that when they say "24-hour service" it means that someone actually answers the phone, and responds withing an hour, or heck, even a day.

3. Lots and lots of screws, cage nuts, and cage nut removal tools. All of the prior have a habit of up and walking off. Have plenty on hand.

4. Hideaway bed. I'm not joking. Have a place where a tech can crash after a 2 am service call. Better than driving exhausted.

5. AdderLinkIP, phone line installed to core rack, and Modem. Hook the AdderLinkIP to your KVMs, and code in all hosts. Now you can get console remotely on any box, PLUS if your internet connection should go down, someone can dial into the AdderLink, and be able to troubleshoot the net connection from the inside without having to drive. Priceless.

6. If there is not a break area nearby with a fridge, get a mini fridge. Stock the fridge with caffeine. Make this a habit that the company provides. Caffeine without having to leave to get it is priceless during a crash too.

7. Anything else you can think of to make your network engineers LESS repulsed at the idea of having to be there. After a while being "the guy(s)" to go to when things go down has it's novelty wear off. If it's within an office, an arcade cabinet, darts, an XBox, ANYTHING....ask them what they enjoy doing in their off time. Heck, alcohol in small quantity is even done in our office (we have a micro-brewery across the street - I have 2 growlers in the office for those long weekend projects).

By saying you're Tier II/III, it leads me to believe you don't have to present as much of an image (hyper-clean room environment with lots and lots of racks, and nothing else), so do what makes you feel comfortable there. You'll be spending more time than you'd like there soon.

What are we missing?

[Mistlefoot]

"We've been up and running for several months now" ... "What are we missing?"

You've migrated several months ago - if you don't know what you are missing yet you either haven't been paying attention to what you need locally or haven't been paying attention to the recent news. Any small items you've probably already needed and know about.

Large things - like accomodating power outages (see 365 Main St) need to be prepared for. I'd guess after a successful migration you've likely covered most things.

Re:What are we missing?

[vilain]

Some things I saw in the last datacenter I worked at that I found indispensible:

- one of those headlamp lights for hands-off work on servers (put this in the tool box)
- a way to track who has the tools in the toolbox (check it at start and end of each shift and record such)
- at least 2 cordless headset phones (ever try to move around a server room tied to a cord)
- a supply of batteries for everything that needs them
- a couple 7-day temperature gage chart recorders at various locations in the center + supply of graph paper (useful for A/C issues)
- status check at start and end of each shift (temperature, server status lights, A/C, UPS, equipment in toolbox, etc.)
- a way to log all operations status (we used an in-house Access database which had to be updated at end of each shift)
- install 2 large UPS systems and connect the dual power supplies one to each USP
- instigate a policy "If you change any system stuff on a server, reboot it to ensure it comes back in a known state" Schedule downtime if needed
- don't offer or expect 7x24x365 availability unless you've built fault-tolerant servers that can do this--every system needs downtime for one reason or another and have a slot allocated for regular downtimes on a monthly basis. Emergency hardware outages don't count against this. But when are you going to roll out patches you've tested in the test environment (you _have_ a test environment that somewhat duplicates production, right?)
- NO DEVELOPERS ALLOWED ON PRODUCTION SERVERS. THIS IS A TERMINATION OFFENSE (WITH EXTREME PREJUDICE).
- Limit who has root to groups of servers. Only the datacenter manager should have root to everything.

Have a server shutdown procedure (order that servers go down in the event of a power or A/C event)

If you have a motor generator for backup power, test it quarterly so it will kick in when there's a city power outage. This will avoid the problems seen in the 365 Main Street outage in S.F.

I had to chuckle when I heard about 365 Main. The old datacenter manager would have covered that with the periodically tested motor generator.

Wireless Headset(s)

[GreggBz]

Get a nice comfy Plantronics headset for the POTS line nearby. In a noisy datacenter, while on a mission critical tech support call, the last thing you need is your hand pressing the phone to your ear and/or crappy cell phone audio.

Re: Tarps

[Tmack]

Its modded funny... but should be Informative. One of our datacenters had a bathroom located on the floor above. For fear of something overflowing and dripping into the racks, plastic was kept on standby. Notice I said "had a bathroom"... we finally worked out a deal with the building mgmt, now its more office space for us. Plastic sheets are a must. If for nothing else, the roof might leak, fire suppression might go off (now days replacing servers can be cheaper than refilling the "halon" tanks), some random pipe in the ceiling/floors above might break, or someone might decide to drive their car into the wall and make a new door. Having the plastic on standby is a good idea.

tm

Re:Don't guess!

[dpilot]

Now that you mention it, did you include some sort of work lights on that crash cart? I don't think I've seen anyone mention work lights yet, or maybe that's such a given that nobody felt it was worth mentioning.

Re:100% IDIOCITY. Hire a ***ing consultant!

[numbski]

Dude, chill for a moment.

Tell me something - if you're going to ask for casual advice on outfitting a data center, WHERE BETTER to ask than Slashdot?

I've built 2 from the ground up now. I don't mind sharing little nuggets.

When I have a question about how to put the finishing touches on a house, I don't see the harm in someone going to a home builders forum and asking for casual advice. Of course you want professionals doing the bulk of the work, but it's the little things that always get you, and we can certainly share those. :D

Might be helpful

[DerekLyons]

When I was in the Navy, I operated a largish weapons control system... After we left the shipyards our big 'stuff' problem came down to two: ('Mike' ia a) a pseduonym and b) not me, I swear. I was the responsible for the tools and spent a lot of time getting after 'Mike'.)
 

1. Documentation
   Keep it current, keep it organized keep it available, and dammit Mike, put the pubs @$#%#@ back where they belong when you aren't using them!
    
    
2. Tools
   Common hand tools in this box, commonly used special tools in that one. Rarely used tools in this other one. And dammit Mike, put the @$#%#@ tools back in the @$#%#@ box and put the @$#%#@ box where it @$#%#@ belongs when you aren't using them!
    
    

The main key is less in having lots of stuff than in keeping what you do have organized and available.