Slashdot Mirror
Required Practices for a Network Operations Center?
hayduke.com asks: "I've recently been assigned to a program that is designing a 'Network Operation Center (NOC)'. I started to look for books, online material and other sources to help define a baseline for the Services Level Agreement for our intended customers. Not having any customers yet we are trying to incorporate the design elements that will provide the best possible level of service to the largest number of customers. A search on my favorite search engine brings up a lot of articles that have companies boasting that they have been recognized for being 'Best Practice' leaders in their respective fields but there are no references as to what those practices are. As this will be a NOC (pro-active) as opposed to a Call Center (reactive), I would like to know what other people think that NOC should be at bare minimum or if there are 'standards' that all NOCs should be held to."
2) Lots and lots and lots of blinking lights.
3) Biometric-based access, such as finger, palm and retinal scanners.
4) Big, ugly dude guarding the front door.
5) Hire Linux weeniers from slashdot to run it all for you.
I have been asked to do a job I know jack-all about. I don't know why they asked me, perhaps because I lied and said I knew something about it. Anyway, all that doesn't matter now, because if I don't find someone pretty damn quick to do my job for me, and for free, I'm going to be found out. So please if a few of you nice slashdotters could submit a draft for the design of a Network Operation Center, I'd be ever so grateful. I'd still have a job, and I'll still be able to convince the dummies that I'm not one of them.Of course the only thing in it for you is the satisfaction of saving some dumb freeloader's butt and getting him out of a tight situation.
Yours,
L. Ardass
If you are providing and charging for backup services, make logs available to individual clients. At the very least, have SOME way to let me know how many bytes you backed up each night if there is an arbitrary monthly limit.
Warnings on the first day of the month (if day 1's use * 30 will exceed limit) would be appreciated.
Warnings on days with more than 50% more than normal days use deserve a warning of some sort. Each client should be able to configure their own warning levels.
If network badwidth limits are known in advance, warnings at 90% and 95% would be very useful.
"God is dead." - Frederik Nietzsche
I started to look for books, online material and other sources to help define a baseline for the Services Level Agreement for our intended customers.
Host a box at Exodus or Level3 and have a read of the SLA they give you. Beter yet, just call them up and ask for a quote and a salesman to call, no need to spend any of your own money. It's probably copyrighted so you can't just use it for your own customers, but it'll give you an idea of where to start.
I strongly recommend you read this, definitive guide to establish your procedures and develop your SLAs.
Not a flame or anything, but seriously. Get someone who knows what the hell they are doing to do some consulting for you. There are lots of them on the market, and you can get them cheap. Hey, you plan to make money with this, right? Don't want to lose your ass? Then you need someone with experience. You woudn't start a business without consulting a lawyer and CPA would you? While you don't mention what experience YOU have, I'm assuming that you have SOME, but not much based on your questions.
If you are gonna provide an SLA, you want TECHNICAL advice and LEGAL advice. Most SLA's are actually toothless in real life. The lawyers give you enough outs that you will never have to pay up with most customers (a few have the talent to see through the crap and make changes to your contract to put teeth back in.) Even though lawyers are expensive, it pays for itself in the long run.
The advise on slashdot is going to be spotty at best, especially in the light that so many NOC's are run poorly. Without experience on hand, you will run into the SAME traps / problems that most NOC's with inexperienced leaders run into.
Well, here are a few things that you may need.
A TESTED disaster recovery plan for servers, network, power, and cooling.
A trouble ticket system customized for your needs usable by inside and out (internally generated tickets and customer generated tickets.)
A network monitoring / management system that tracks not only subsystem availability but performance and keeps a history.
A customer management system that can bring up EVERYTHING you need to know about a customer, their syetems, their people, notification proceedures, etc. (this is VERY non-trivial)
A change control system - what happened, who approved it, who did it, how long did it take, what did they do, how did they do it, when was it active, how do you revert, etc.
Sigh. Setting up a NOC correctly is one of the most difficult tasks in IT.
If you're in the Silicon Valley you'll want some industrial or retro funiture, but if your company is located anywhere else, you'll want the "me too" look that only Aeron chairs can provide.
I know we're just joking about these requirements, but they're scarily familar to three NOCs I've been involved with. I think there must be some unwritten ruleset that goes something along these lines:
1) 50% of NOC budget must be spent on funriture and flat-panel displays.
2) Trendy lighting in NOC must seriously interfere with trendy displays. (example: if room is equiped with halogen spot lighting, at least one non-movable light should be aimed at a projection screen).
3) NOC must be located in the most inconvenient area of the most inconvenient building.
4) Actual NOC computers must be running the latest, untested wiz-bang buggy software on the latest, untested wiz-band buggy hardware.
5) Half of the NOC staff must be completely unskilled, impersonal, and unwashed.
6) The other half othe NOC staff must be anal, uptight, and permanently pissed off.
Server Room layout is another story... but does match rule #2 quite well... the perfect server room is often located as far from the building's loading dock as possible. With a proper pallet jack, it should take at least 30 minutes to haul a crated SGI Origin or Sun Enterprise server from the loading dock to the server room. Smaller items should take no less than 15 minutes. Shaky ramps, cramped elevators, and narrow hallways are a plus.
The only book you need is the US Code, Title 11. Pay special attention to Chapter 13.
I've never worked in a NOC, but I've been a customer with a couple of big names, and the most important thing to a customer is not the SLA - if we have to even read it, things have gone seriously wrong, and rather than litigate, we'll just leave.
The blinkenlights, CRM processes, trouble ticketing systems etc are all lovely, but the thing that makes a difference is the people. In one case, there were 2 network guys we spoke to - one was great, used his initiative to sort stuff out, never lied to us or tried to fob us off, and kept us in the loop with what was going on. The other guy was technically prob. better, but used all the company's processes to hide from us. He was reluctant to acknowledge problems, rarely responded to voice or email, and gave the impression we were not really important to him. They both worked to the same SLA, processes, standards, etc. One was good at his job, the other merely good at technology.
So, I would suggest that instead of spending a lot of time on SLAs, you spend time finding good people. Monitor your performance not by "uptime" (one of our suppliers was monitoring our site using the internal network. They got 99.999% uptime, while anyone outside the NOC got "server not found"), but by customer satisfaction - contact your customers once every 3-6 months and ask them to fill out a satisfaction survey. Deal with issues they raise. Treat a customer who leaves you like a company crisis. Encourage your people to think about outcomes, not processes.
Sure, you need to know how your network is performing, what your customer's uptime statistics are, and have the facilities we have come to expect (including blinkenlights). Just make sure you've also got some cat5, screwdrivers, and free drinks for your customers, and that you don't get carried away with all the fancy stuff.
It's all very well in practice, but it will never work in theory.
1. Good ticket system. I've used several and I can tell you that this can be a major issue. RequestTracker appears to be sufficient, although I've never used it for a larger datacenter. The tracking system and the PROCESS can make or break your datacenter.
2. Phone calls sound like a good idea to clients but they can be a pain in the butt.. especially since many clients have difficult accents. They also require a lot of a technician's time. Consider having no telephone support, or only for large clients.
3. You need a good customer management and billing systems.
4. Good inventory system, tie to billing system.
5. A web-based interface to your router(s) and switches is advised.. it is also advised to tie this into your other software, don't physically unplug suspended servers, just suspend them from the billing interface and have it automatically use SNMP to disable their port.
6. Make sure the person handling your routing needs is sharp, I've seen datacenters where the 'router god' was learning on the job (not a bad way to learn, just a bad way to run a NOC)
7. Make sure that you do proper cable management, keep the facilities in good order, working restrooms. I've seen datacenters which would make the homeless cry.
8. Server monitoring system. For the convience of your technicians, a projection or large lcd would be preferred.. one of those cheaper dot-matrix LED displays would be ok.
9. I've been places where I've had to answer telephone calls, answer tickets, lookup passwords, and monitor servers.. all from different webpages behind the LAN. INTEGRATE. The closer and easier something is to access, the more useful it becomes.
Btw, I'm currently writing management software for a NOC; although it is proprietary software belonging to the NOC and I don't think they have any plans to sell it.