Slashdot Mirror
Ask Slashdot: Designing a Telecom Configuration Center?
First time accepted submitter Big Jim Taters (1490261) writes "I have been tasked with helping move our config center from one location to our Headquarters. I have a small budget and no choice in location. I do, however, have an opportunity to design the space fresh (well, kinda.) What we will be configuring is routers, switches, firewalls, and other telecom related devices. What I cannot find is any "Best Practices" or "Lessons Learned" out there. So I ask you fine folks: What are some of the best and worst designs, practices, procedures, and work flows that you have encountered in sitting down to stage anywhere from 2 to 200 devices at once to get configured?"
and don't forget the raised floor
Either you padded your resume or your boss is dumb. Either way, you're taking a job away from one of the many qualified engineers who could succeed in this task.
Many places use uniform cable colors to indicate the purpose of each wire (red = DMZ, yellow = uplink, blue = intranet, green = servers).
But I find that makes it very hard to trace individual cables. Tracing one red cable among a bunch of of other red cables is madness.
Rather, I'd suggest explicitly using as many random different colors as you can.
Label the end of each cable where it plugs in with its purpose if its not abundantly clear, but otherwise, tracing a single red-wire through a bunch of multi-colored wire becomes Much, much easier.
List out all your common changes, and produce a checklist template for implementation. This checklist should NOT be page after page of screenshots that nobody but the greenest admin will ever read. They should be concise, and contain just enough information to have all the implementation data, and jog the memory of the admin as to precisely which steps need to be done.
On the template, you should record all the data you can possibly need to implement the change. If you could not fill out the checklist, and then hand it to another admin for implementation, the checklist isn't good enough.
So, that covers the change request part of the checklist.
In the actual implementation part, record ALL the steps where there's a decision point. (As in, you don't need steps for "Remote in to admin console, Login to Switch Config App, Login to Switch, Enter Config mode, enter VLAN subsystem, etc.) "Add VLANs to switch, using information listed above" is fine. Make sure the checklist includes updating whatever documentation you have.
Each line on the checklist should contain the date/time the step was completed. (If the admin just has to put an "X" there, guaranteed they'll ignore the checklist and just put in the "X"'s at the end.
Make the filled-out checklist itself part of the change record. Your change records should be complete enough that you should, in theory, be able to take the pre-change-system config, execute the tickets one after another, and end up with the same final config.
Lastly, do NOT require mgmt. approval for routine changes. Your checklist should already cover giving the appropriate people warning of the change. If you require mgmt. approval (or a change control board) for the most trivial changes, it quickly becomes rubber-stamping, which is even worse than wasting everybody's time. Save the change review process for changes not covered by the checklist.
You're seriously starting from scratch? Oh man... wish I could do that.
Lets see... #1 thing... when I started way-back-when, we had this giant display board that would show alarms on equipment. It looked neat, and made us look fancy. Don't do that... It was useless to those of us fixing stuff. The only people that looked at it were executives. Every hour or two some VP would come running over "Why is NewYork blinking red?!?!?!" "Because the gateways is down." "Is that bad?!?!" yada yada
Setup a wiki... make sure everyone has permissions to edit it. Make sure you have procedures in place for how and when to edit it.
I don't know what equipment you're using, but if you have a choice, try and go to a monoculture of one manufacturer if you can. We had a huge mess of every type of device you could imagine from every company on earth. Over in California we had 3 Juniper routers. Every time something "strange" happened with those we had to call the "Juniper guy" You should all be experts in everything you have and the easiest way to do that is to go with one vendor. It makes it easier to hire for. "We're a Cisco shop" or "We're a Juniper shop" whatever... That's up to you and how talented the people your managements willing to pay for. If they only will hire people green out of techschool like my old boss was, then this is likely a good idea.
Setup a centralized on-call list that charts whos responsible for what. One of the worst things that can happen during an outage is that you're screwing around calling Bob to get Toms number, because he has to change the firewall to let Tim into the device he needs to fix. This goes all the way down to your facilities people. I had an outage once that we couldn't address because the basement of the building was flooded and no-one knew who to call to get the pipes fixed.
Is everything you do software? Or will you be handing the equipment as well? Everything I did was via-remote. I rarely actually touched equipment, we had field techs for that. But my buddy that does the same job I did, now has a job where he actually unboxes the equipment and installs it personally. If you're going to do that I'd recomend a good printer, cable labeling templates... practice using both. TOOLS! Specifically "Easyouts" for stripped screws. A Small hammer. Hand-screw-drivers. Wire snips, etc...
Along those same lines, a benchtop with every type of OS that will pass through your stuff. We had Every version of Windows, MACs, Linux, etc... usually these were just to prove the vendor wrong... We'd submit a ticket for a bug and they'd say "Oh, that only happens with a MAC!!!" so we'd test it out "No it doesn't" etc...
I'm not sure if I got overly basic with you there, but that's how we did it.
For actually configuring devices, we'd have one of the leads write a script and then foist it off on "the new guy" to put it on every device. It's been about 5yrs since I've done a config though so things might not be as simple with all the modern GUI stuff.
and I like to get paid for my work. I expect most of my peers feel similar. So as unhelpful as you may find this, hire someone who's done it before, and ask them nicely to let you tag along and learn. Then you can become one of the professionals.
The above statement may sound condescending, but it's not meant that way. It takes years to learn the stuff you're asking, and differentiates the juniors from the seniors. Asking the seniors to train you for free isn't likely to be that well received by most of us.
You are asking quite a broad question so I'll just throw in some random ideas since the topic is so broad you'll need to specify exactly what you have problems with and ask about it... and then iterate and ask again since it would be a long process.
1) Mount infrastructure systems like UPS and AC in advance and let them dry run so you can test if they behave properly f.e. ACs dont produce fog (I've dealt with such problem and it required changes from vendor) UPSs produce weird vibrations and so on. Just put the basic stuff to work as soon as possible so it you can overcome any gotchas it may cause.
2) Plan for expansion in advance - make proper room for additional AC and UPS units. Wire it accordingly. Put exhaust tunnels and everything needed just don't put the unit itself - this will make it much easier for you to expand if needed as you'll have all the work done already. Also add all racks you'll probably need later and wire them. Just don't use them - leave as spare. Racks and wires are cheap - labour is not cheap and it consumes time when needed.
3) Put additional canals (just empty space) below AC units as they can leak fluid.
4) Invest into cable management system so you can label each end of each cable with barcode and have a database of this configuration so you can scan every end of every cable and lookup what is its function where and to what it is connected etc. Don't bother with manually labeling each cable since it is tedious and probably will change in time. Just use generic barcodes attached to ends of wires. MAINTAIN this database as it is the most important stuff. Every single change must be noted in database. Peroid. No exeptions. Also don't bother with color coding the cables - usualy you will end with a mess unless you have a simple network.
5) Invest in security system like webcams on entry doors, RFID IDs and so on. It is not as expensive and will give you audited information on who, when had physical access to the facility. Be strict and anal implementing this - in case of non compilance f.e. somebody is trying to get unauthorized access just raise the alarm to local security staff. It is better to have false positives here.