Slashdot Mirror
Ask Slashdot: Designing a Telecom Configuration Center?
First time accepted submitter Big Jim Taters (1490261) writes "I have been tasked with helping move our config center from one location to our Headquarters. I have a small budget and no choice in location. I do, however, have an opportunity to design the space fresh (well, kinda.) What we will be configuring is routers, switches, firewalls, and other telecom related devices. What I cannot find is any "Best Practices" or "Lessons Learned" out there. So I ask you fine folks: What are some of the best and worst designs, practices, procedures, and work flows that you have encountered in sitting down to stage anywhere from 2 to 200 devices at once to get configured?"
and don't forget the raised floor
Either you padded your resume or your boss is dumb. Either way, you're taking a job away from one of the many qualified engineers who could succeed in this task.
Don't ask Slashdot; ask the people who have to work there.
Really, It belongs there.
Moderating "-1, Disagree" is simple censorship. Have the guts to post your opinion. -- Spazmania (174582)
Many places use uniform cable colors to indicate the purpose of each wire (red = DMZ, yellow = uplink, blue = intranet, green = servers).
But I find that makes it very hard to trace individual cables. Tracing one red cable among a bunch of of other red cables is madness.
Rather, I'd suggest explicitly using as many random different colors as you can.
Label the end of each cable where it plugs in with its purpose if its not abundantly clear, but otherwise, tracing a single red-wire through a bunch of multi-colored wire becomes Much, much easier.
List out all your common changes, and produce a checklist template for implementation. This checklist should NOT be page after page of screenshots that nobody but the greenest admin will ever read. They should be concise, and contain just enough information to have all the implementation data, and jog the memory of the admin as to precisely which steps need to be done.
On the template, you should record all the data you can possibly need to implement the change. If you could not fill out the checklist, and then hand it to another admin for implementation, the checklist isn't good enough.
So, that covers the change request part of the checklist.
In the actual implementation part, record ALL the steps where there's a decision point. (As in, you don't need steps for "Remote in to admin console, Login to Switch Config App, Login to Switch, Enter Config mode, enter VLAN subsystem, etc.) "Add VLANs to switch, using information listed above" is fine. Make sure the checklist includes updating whatever documentation you have.
Each line on the checklist should contain the date/time the step was completed. (If the admin just has to put an "X" there, guaranteed they'll ignore the checklist and just put in the "X"'s at the end.
Make the filled-out checklist itself part of the change record. Your change records should be complete enough that you should, in theory, be able to take the pre-change-system config, execute the tickets one after another, and end up with the same final config.
Lastly, do NOT require mgmt. approval for routine changes. Your checklist should already cover giving the appropriate people warning of the change. If you require mgmt. approval (or a change control board) for the most trivial changes, it quickly becomes rubber-stamping, which is even worse than wasting everybody's time. Save the change review process for changes not covered by the checklist.
200?
I like to use electrical tape bands on the ENDS to denote which is which. There is already a colour code table to translate numbers into basic colours, see resistors.
It may sound obvious, but staging that many devices requires some organization.
... double check the version. It could save you hours of frustration.
Simple things, such as one pile (table/area) for unstaged devices; one table for staged devices and one for problem devices.
If you are required to document the process (most likely you are) - serial numbers, user assignments, etc - then the help of a USB bar code scanner and a spreadsheet can really help. If you don't have a bar code scanner then a good magnifying glass may prove itself quite useful.
A mobile internet connection, a good power bar or two (or more), extension cord and a set of PC tools is recommended.
And always be sure to ask for the latest version of documentation! I can't count how many times the documentation was outdated on projects... even when sent the day before
Make sure that you have a support number or resource in case things go awry.
I don't know the details of your project; i'm just going from personal experience setting up large amounts of equipment.
Hope this helps!
Political correctness is really just herd psychology pushed by insecure people who desperately seek social conformity.
save config as XML then run it through diff - a great way to see what you actually changed...
Do an immaculate job of running lines. Make sure all lines are not just functional but that everything looks nice and is well labeled. The moment one person does a sloppy job of installing/repairing some lines or equipment, everyone after them, will likewise do a sloppier job until the place is a complete disaster.
If you do this all yourself, its less of an issue, but expect to grow. Expect other people to not have your own common sense.
You're seriously starting from scratch? Oh man... wish I could do that.
Lets see... #1 thing... when I started way-back-when, we had this giant display board that would show alarms on equipment. It looked neat, and made us look fancy. Don't do that... It was useless to those of us fixing stuff. The only people that looked at it were executives. Every hour or two some VP would come running over "Why is NewYork blinking red?!?!?!" "Because the gateways is down." "Is that bad?!?!" yada yada
Setup a wiki... make sure everyone has permissions to edit it. Make sure you have procedures in place for how and when to edit it.
I don't know what equipment you're using, but if you have a choice, try and go to a monoculture of one manufacturer if you can. We had a huge mess of every type of device you could imagine from every company on earth. Over in California we had 3 Juniper routers. Every time something "strange" happened with those we had to call the "Juniper guy" You should all be experts in everything you have and the easiest way to do that is to go with one vendor. It makes it easier to hire for. "We're a Cisco shop" or "We're a Juniper shop" whatever... That's up to you and how talented the people your managements willing to pay for. If they only will hire people green out of techschool like my old boss was, then this is likely a good idea.
Setup a centralized on-call list that charts whos responsible for what. One of the worst things that can happen during an outage is that you're screwing around calling Bob to get Toms number, because he has to change the firewall to let Tim into the device he needs to fix. This goes all the way down to your facilities people. I had an outage once that we couldn't address because the basement of the building was flooded and no-one knew who to call to get the pipes fixed.
Is everything you do software? Or will you be handing the equipment as well? Everything I did was via-remote. I rarely actually touched equipment, we had field techs for that. But my buddy that does the same job I did, now has a job where he actually unboxes the equipment and installs it personally. If you're going to do that I'd recomend a good printer, cable labeling templates... practice using both. TOOLS! Specifically "Easyouts" for stripped screws. A Small hammer. Hand-screw-drivers. Wire snips, etc...
Along those same lines, a benchtop with every type of OS that will pass through your stuff. We had Every version of Windows, MACs, Linux, etc... usually these were just to prove the vendor wrong... We'd submit a ticket for a bug and they'd say "Oh, that only happens with a MAC!!!" so we'd test it out "No it doesn't" etc...
I'm not sure if I got overly basic with you there, but that's how we did it.
For actually configuring devices, we'd have one of the leads write a script and then foist it off on "the new guy" to put it on every device. It's been about 5yrs since I've done a config though so things might not be as simple with all the modern GUI stuff.
Use fiber for everything, setup a pfsense box, set the switches to unmanaged, and use one collision domain. I suggest 10.0.0.0/8.
and I like to get paid for my work. I expect most of my peers feel similar. So as unhelpful as you may find this, hire someone who's done it before, and ask them nicely to let you tag along and learn. Then you can become one of the professionals.
The above statement may sound condescending, but it's not meant that way. It takes years to learn the stuff you're asking, and differentiates the juniors from the seniors. Asking the seniors to train you for free isn't likely to be that well received by most of us.
Get yourself plenty of serial console servers. 48 port ones are normal, so maybe you need 5. Then you can have specific places where each one goes. People go in once place, equipment goes in another. There's nothing worse than working next to hardware.
ps, it seems like people are quite mislead on what the OP requires here. He is staging telco equipment, not building a NOC.
or a data centre, or a network! sheesh ;-)
... pay someone who knows.
For telecom, the best things we did was to put plywood up on the walls so we could mount any 110 or 66 blocks there as well as any surface mount equipment. I've also highly enjoyed the b-line flextray cable management system to move cables from point A to B within a room. Likewise, put a good cable management flange down the sides of your 2 post racks to reduce the temptation to just waterfall all your cables over the rest of you equipment. If you're dealing with servers, I recommend a mobile crash cart like you find in a hospital that you can just keep an UPS, computer, & 2-port KVM with all your tools ready to go and just wheel it between racks & servers.
Put plywood up on your walls for surface mount things like 110 or 66 blocks and any surface mount equipment. From a product standpoint, I've found the b-line flextray cable management system to be very nice and easy to install. I'd also invest in any kind of vertical flange cable management for your 2 post racks just to help keep from waterfalling your cables over the equipment.
because they are way different.
remote config, you just need secure telnet and good management pipes, on top of the normal basic gizmo office (overpowered PCs, hopefully dual screens, redundant power, etc.)
local config, you will also need an open half-rack on casters at each workstation area, good AC and "battery" power, I like the idea of a cube terminal server 8 or more ports for you to get into the craft ports and push scripts, then localize afterwards. or a localization generator which saves all the configs on a common ultra-backed-up server, and you just tftp them into the remote equipment.
there is almost always a TL1 interface and port number on telco equipment, so you could automate all the localization generator stuff into TL1 as well. since you're telco, also get the AC1 MIB documents for your stuff so you can catch and decode the SNMP traps and autogenerate repair tickets.
none of this is high-buck stuff. it's the coding in the end that determines how much of the scutwork is manual and time-waste, and how much nonsense actually is handled when the outage clock starts ticking and your techs get after the repair.
if this is supposed to be a new economy, how come they still want my old fashioned money?
I've been asked to complete a project that is waaaaaay over my head. I'm in deep trouble here and my employer may figure out that I don't know shit about my job. Please help.
You are asking quite a broad question so I'll just throw in some random ideas since the topic is so broad you'll need to specify exactly what you have problems with and ask about it... and then iterate and ask again since it would be a long process.
1) Mount infrastructure systems like UPS and AC in advance and let them dry run so you can test if they behave properly f.e. ACs dont produce fog (I've dealt with such problem and it required changes from vendor) UPSs produce weird vibrations and so on. Just put the basic stuff to work as soon as possible so it you can overcome any gotchas it may cause.
2) Plan for expansion in advance - make proper room for additional AC and UPS units. Wire it accordingly. Put exhaust tunnels and everything needed just don't put the unit itself - this will make it much easier for you to expand if needed as you'll have all the work done already. Also add all racks you'll probably need later and wire them. Just don't use them - leave as spare. Racks and wires are cheap - labour is not cheap and it consumes time when needed.
3) Put additional canals (just empty space) below AC units as they can leak fluid.
4) Invest into cable management system so you can label each end of each cable with barcode and have a database of this configuration so you can scan every end of every cable and lookup what is its function where and to what it is connected etc. Don't bother with manually labeling each cable since it is tedious and probably will change in time. Just use generic barcodes attached to ends of wires. MAINTAIN this database as it is the most important stuff. Every single change must be noted in database. Peroid. No exeptions. Also don't bother with color coding the cables - usualy you will end with a mess unless you have a simple network.
5) Invest in security system like webcams on entry doors, RFID IDs and so on. It is not as expensive and will give you audited information on who, when had physical access to the facility. Be strict and anal implementing this - in case of non compilance f.e. somebody is trying to get unauthorized access just raise the alarm to local security staff. It is better to have false positives here.
Long work benches with a single long shelf high above, long power strips mounted between the vertical struts that hold the shelf, and label every end of every cable. 1/1, 2/2, 3/3. That's all there really is to it. There are some really nice tech benches with integrated cable management and monitor arm mounts etc... if you're budget allows for it. If the room is not well ventilated think about where to place a fan, and pick up a cheap speaker set to connect your smartphone to for music.
Probably the most important thing is to get support from management for the door to be limited access so that the unwashed masses can't get in to disturb you.
So NSA is realy asking on slashdot how to outfit that "Cisco Modding Lab"? ....
Your goals here are make it quick and easy to get stuff out of the box, configured, and back out the door as quickly and efficiently as possible.
Things I'd do to start:
If doing racks, consider shelves so you can slide equipment in and out quickly. Some racks will let you do shelves that mount to the sides rather than taking up 1U for a shelf, these may let you get more density in the rack and need fewer racks.
If doing shelves, don't stack equipment, try to put it like books on end, makes it a lot easier to get one piece out without moving a bunch of others.
Plenty of power cords/outlets where you need it, make sure if everything isn't a C13 that you account for this. Newer switches are starting to use C15s or C19s for larger equipment. Make sure you have a large enough UPS to handle startup current for all these devices. Constantly turning up/down equipment is hell on your power feeds, good clean UPS power is important.
Patch cables wired in and velcro'ed off to the rack where you need them, and run extras. That way if you have a suspected bad cable or a broken end you aren't worrying about replacing it right away to get the equipment out the door.
Terminal servers are a godsend in an environment like that. Configure them so you know that TS1, port 1 is the top (or bottom) device in the rack. Keep them in order or you'll be tearing your hair out why the wrong device just rebooted.
As someone else mentioned, USB barcode scanners if you have to do any kind of inventory tracking is a GREAT tool to have.
Separate but adjacent boxing/unboxing room with a sturdy table. And a sturdy cart to move equipment back and forth between them. You want to keep all the cardboard and styrofoam out of the equipment config area.
I wish to answer several points here and clarify a few others: First, let me start by saying that I'm not asking anyone to give up their job as a designer and do my work for me. I am a RCDD in training and can do my own design work (as an understudy.) I'm simply asking for any specific tips of things that did or, probably more importantly so, what did NOT work. E.g. Don't go build a huge alert board (as Charliemopps pointed out.) TheBrez also has the right idea. We will be configuring the equipment as a refresh to our other remote sites (I work for a very large company that has ~175-200 sites scattered throughout the US alone.) that will be configured and shipped back out to their respective sites. Also, please consider that this is a configuration center of not-yet-in-production equipment and not an in-production data center or computer room. One example of a design conflict I'm having is racks with shelves vs. tabletop space for configuring equipment. Other good suggestions so far (and thanks to all who have contributed): Raised floors are good for permanent cabling, but poor for heavy turnaround cabling. Barcode systems for organization are HUGE. And just for those who feel I'm reaching out because I'm in over my head....well, there's probably nothing to say that would convince you otherwise, but that is not the case. I've worked a few configuration areas before and have some experience, though on the order of a few months of it rather than years. My design, though sounding vague in text-only form, is to have essentially 3 dedicated areas for the whole space: a storage area for both received equipment not ready for configuration and empty boxes of equipment that is a work in progress, an area for unboxing/boxing the equipment with a typical shipping/receiving feel to it, and the actual configuration space. I lobbied for, and apparently won, the racks w/ shelves design (ideally each shelf representing its own TR.) Cable management is a must, of course, and thank you everyone for making that abundantly clear. In closing, I'm just looking for those tidbits of wisdom that only come from living inside the config area and saying "Man, I really hate that we have ________."
Be sure to properly document all of your network connections in a Data Center Infrastructure Management tool. I like OpenDCIM http://www.opendcim.org/downlo... because it's free and open source
Small Budget.....eh...
Things to ponder on:
1) Primary and Secondary UPS/Generator to ensure good clean supply at all times. Depending on your power source and and stability, you may or may not need this. If you have a budget, go for it.
2) Primary and Secondary Temperature/Humidity Control to ensure a stable environment. Can get pretty hot in a Telecom room when AC is not working. Depending on your geo-location, you may or may not need this.
3) Raised Floor/Ceiling space to run cables. If budget allows, Cable Trays to run cables from one side of the room to another....could be done neatly and on a budget.
5) Position of Racks to ensure Accessibility....especially cable racks. You never know when you need to punchdown new cables and getting behind the little space behind a rack can be a pain. Make sure you can easily work around your server/cable racks.
6) Proper electrical grounding for all equipment. Ground the racks properly...equipment can thus be grounded via racks.
7) Physical separation of Electrical cabling and Data cabling as much as possible.
8) Labeling
9) Physical Security to the room and building room is in. - Fire/Water Alarms. Temperature sensor if budget allows. Break-in protection...etc.
10) Position of Telephone - During remote support this can be a pain unless you have a cordless phone.
11) Consider how many people will be in room at same time.
Best way to start is plan a layout using some software. Position everything. AC/ UPS/ RACKS/ TELEPHONE/ PEOPLE IN ROOM.
Just some basic considerations....among others...
All the best!
First, I do this for a living, building and populating and managing data centers. That's why I'm an A/C. The words are true but I don't want them to reflect on my competitors in the market or on the vendors and clients all of us rely on.
Question 1:
- When you ask for things that fit 2-200 servers you're talking about entirely different things. You also don't tell us what KIND of servers (network? disk? ASP? iaas? sap? I could go on). Are they 1RU power-hungry bitcoin hashers, 2RU JBODs, 4RU monster Oracle/SAS boxes? What?
Racks/Cooling/Power:
- For two servers you don't require a rack. You sure should get one because it prevents people from kicking the power cable out, but there you go. For 200 you need active hot/cold airflow design as well as an enclosed rack, power distribution unit, management of same, and multiple power sources for A/B power. Am I going overboard? See question 1
Physical
- Raised floor? Not for two servers. Not for 5 racks (42 1RU servers per rack = up to 210 servers). You may consider a 4-rack "enclosed tower" with cooling and power distribution in it, but since you don't REALLY have 200 servers, this is academic. Or as they say in the academic world, moot.
- Hot/Cold aisles? Not for 5 racks. But that enclosed tower thing would work well for growth
Cabling
- Invest heavy in overdoing cabling infrastructure. Patch panels. Many more runs than you need. FDPs and Cat6s.
Schemas
- Come up with a naming schema for everything. Datacenter (one day you might turn another office into another one, so might as well label this one "1"). Racks (one day you may have a second rack and maybe even more than 9 so label this one "01"). RUs, label either from the top or the bottom but choose a schema and stick to it. You'd be surprised how much easier it is to allign something in the center of the cabinet (for common-length cabling to devices in the rack) when you can quickly see the RU21/RU22 stickers on both sides. SERVER NAMES. No, not "dopey, sneezy, etc" some schema where the server name helps YOU locate it when it's down, not something that the rest of the world thinks is cute. I've seen DC1R02RU05 work wonders for telling a tech INSTANTLY what to go look at.
- Come up with a control schema. KVMs? Remote Spyders? Serial control? Power port control? Whatever. If you design it in initially you'll find it makes troubleshooting a breeze and you don't have to drive down to do it. IPMI is a must. that and modern controllers and you'll never have to look at the machines physically once you install them.
- Come up with a sechema for naming things and make sure all your schemas above comply with it. I like and then define the NOUN and the N for each type of object.
- Come up with a documentation schema that tells everyone WHERE this stuff is documented. Note that the thirst part of any good documentation is to refer to itself and explain what it's for, where it's stored, and who should be allowed to use it.
Oh I could go on.
But you've been tasked with this, and you have no money, and you want to make it all cool, so I wish you the best.
M
"I have been tasked with helping move our config center from one location to our Headquarters"
How many people has been allocated to the task and what is your budget and timeframe?