Slashdot Mirror
Ideas For a Great Control Room?
lewko writes "Our company is about to build a central monitoring facility and I'm looking for ideas/suggestions about the best hardware and the best way to make it comfortable for those manning a screen. It will be manned 24x7 and operators will be monitoring a variety of systems including security, network, fire, video and more. These will be observed via local multi-monitor workstations and a common videowall. This is going to be a massively expensive exercise and we only get one chance to get it right. The facility is in a secure windowless bunker and staff will generally be in there for many hours at a time. So we have to implement design elements which make it a 'happy' place. At the same time, it has to be ergonomically sound. Lastly, we will be showing it to our clients, so without undoing the above objectives, it would be nice if it was 'cool' (yet functional). Whilst Television doesn't transfer to real life always, think 'CTU' from 24."
You need one of those old cabinets with tape reels ticking around. Adds to the atmosphere and will remind clients of Thunderbirds etc,
Some fake windows, even just glass blocks with lights behind them, will do wonders. Also, make it so that people have to get up from chairs once in a while.
Whatever you do, don't make it dark with blue-ish lighting, like on TV. That strains your eyes. Provide good lighting, and make sure the persons can sit or stand comfortably while watching the screens.
Plenty of ideas here:
http://www.villainsource.com/lairs.html
A big red button with a sign above it that says 'DO NOT PRESS'.
'We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress.' RPF
Well, it sort of goes against 'being in a bunker', but if I was going to work somewhere for many hours, I'd like some natural light.
Of course it's still possible to achieve that using reflective tubing or such like, though it might still undo whatever it is you seek to achieve by being underground.
If it's not possible, I'd suggest paying lots of attention to lighting. And add some real plants too - they'll generate oxygen as well as making the environment seem less bunker like.
Here is a blog post with some thoughts about the control room design at the LHC. Here is a picture of the CMS detector control room for comparison. You might also take a look at pictures of the CERN control room for some ideas.
I sincerely hope your "client" is not the US Government. The number of contractors I have seen build "CSI" control rooms to try to impress their government counterparts is incredible. Typically these control rooms control very little, or at least, very little worthwhile. At any rate, I would give the advice: form follows function.
Numerous commentors here point out the error of using cool colored dim lighting for a facility where actual work gets done. But you can have your cake (or Cake, if you are using cool indie rock background music) and eat it too - just have a button that switches the lighting to "cool mode" whenever a visitor comes in. Meaningless but cool looking graphic "screen savers" could also pop up on the screens.
Starships were meant to fly, Hands up and touch the sky - Nicky Minaj
Do ensure that it has nearby and adequate washroom facilities. Nothing worse than having to travel up a couple flights of stairs every time personnel has to take care of nature, know what I mean?
Adequate lighting, and ventilation / heating / air conditioning is also something to consider. Nothing worse then working in winter with cold fingers, let me tell you.
I've been NOC for just shy of 3 years now, and I can tell you the environment you work in plays a huge role in how comfortably you handle the workflow. Its nice to focus on the more technical bits such as equipment and infrastructure, monitors, etc, but do not forget that people have to comfortably be there for hours at a time. We do 12 hour shifts here, and the most important consideration would be the temperature and air quality, imho.
user@host$ diff
I would contract with a design/build firm that specializes in these facilities. You don't want to do this by the seat of your pants relying on Slashdot advice.
My company invested millions of dollars into a central monitoring facility, with a large video wall driven by Crestron equipment. The idea was the video wall could display news/weather alongside alarms/outages in real time, with geographic mapping capabilities. Workstations were quad displays on adjustable motorized desks which sat atop a raised platform for simple network runs. A large executive "war room" style conference room was built with a glass wall overlooking the platform and video wall believed to be useful in the event of some catastrophic failure. All other staff sat in cubicles surrounding the platform with glass cube walls anywhere that would otherwise obstruct the view of the platform/video wall. A secure mantrap was put in place to restrict access to the facility. Dedicated bathrooms were installed with showers in the monitoring area in case critical staff were quarantined for extended periods of time.
It was impressive when it was built, but within a couple years, the video wall has been dismantled and parts sold off due to its impracticality. The right software was never found to perform the type of "geographic" monitoring conceived, partly due to bureaucracy. Network redundancy was overlooked, which made the monitoring facility itself non-functional during an outage. The facility lacked appropriate backup generators and UPS to keep the facility running during a thunderstorm. The platform desks required too much real estate and allowed no room for growth, so they have been replaced by cubicles. The secure mantrap was an inconvenience for upper management, so the inner door was disabled, defeating the mantrap. The quad displays ironically obstructed the view of the video wall when it was still in place, and did not fit in the cubicles when they were installed, so these were reduced to 2. All critical staff were sent home to telecommute because they took up too much real-estate required for day-to-day operations, and it made more sense to not have critical staff in a single central location anyway.
The point is, don't get too caught up in building 'CTU' from 24. The right monitoring software platform makes all the difference, as does intelligent network redundancy, telephony and backup power.
I'm not an expert, but I play one on slashdot.
Lastly, we will be showing it to our clients
In other words its going to all be for show.
Big projected screens showing something management finds important to brag about but the employees will never glance at.
All individuality quashed, no pics of the family etc.
My advice, in all honesty, is to build two. One that actually works, and one that is a star trek mock up. Whenever they did marketing picture shows they hired college age models to "staff" our network management center anyway, so non-operational equipment is not exactly a problem for the models to pose with.
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
I used to know someone who had worked for an alarm monitoring company. She said the chairs and other furniture were used 24 hours a day, yet the accountants were depreciating them like ordinary office furniture. As a result, the furniture was not replaced often enough and was falling apart and uncomfortable. Make sure to plan on fast depreciation for your furniture.
steveha
lf(1): it's like ls(1) but sorts filenames by extension, tersely
So, a couple things to think about:
1. How much money do you really have? Lots of people think they have money, but run out when it comes to all the details.
2. Do you want flash, or functionality? The two are sometimes complementary, but often one trumps the other.
3. How many people will staff? What's the schedule? This helps you figure out workstation configurations.
4. Are you putting multiple tiers in the same room? This is "best practice" if you do it right.
5. Are you handling customer calls directly? Do you deal with customers?
Basically, you need to figure out a lot of goals. A true "global" NOC can cost $50M easily for a telecom or comparable organization.
I've been a big fan of Barco for large projectors, and their IP-based solution is quite powerful. Recently, I rolled out a "public safety" SOC (security operations center) with 8 SXGA+ rear-projection displays. The largest I've worked on was 40+ of that style of display. Your garden-variety projector isn't cut out to handle this kind of duty-cycle. They're not cheap, but they're designed to operate 24x7x365, and many models have multiple lamps, etc. so that you can service them while they're online. So here's a few more things to think about:
* What goes on the "big screen" has to be useful. It must be grokable in a very short period of time. If you can't look at it for 2 seconds and get a good idea of what's going on, it's too complicated.
* Multiple displays per operations person
* Operational "graphs" that show overall statistics that matter to the people working, not to management.
* Good task lighting. Good lighting period is everything. Pay a real designer to do this.
* Good seating. We have let operations people pick chairs that fit their needs. Expect to spend $800-1k/person on seating.
* Sound deadening/management. NOCs get loud, and managing the acoustics is important to make sure that people can "think" and they can interact with one another.
* Ticketing is everything. Look at systems that are available commercially and for free. Consider writing your own if needed. If the system is streamlined to your own business, it will always be an impediment to getting the job done, which means people won't use it. If they don't use it, lots of knowledge is lost and post-mortems are more difficult.
Also, a few things that seem superfluous, but ended up being critical in some places I've worked (not all these were at the same place):
* Virtualized desktops (think RDP, X11, etc.) so that people can move and maintain their setup :)
* Color-shifting lighting to compensate for normal rhythms of people on weird shifts. Turns out green is effective after lunch at helping people maintain focus. This isn't cheap, but it sure does have a big impact.
* Keep your customers OUT OF THE NOC. A glass wall into the NOC is fine, but actually letting them in is distracting, and depending, can come with legal issues around privacy, HIPAA, etc. Best to keep them at a distance.
* Before you let customers see the NOC, you warn people. We had a blinking lighting strip under the displays that was linked into the Crestron system so that you couldn't flip the LCD-glass for 10 seconds to give NOC operators a warning. You don't want customers seeing people picking their nose.
Finally, as nice as good facilities are, if you don't have the process and people, it's useless. People people people people. Good people create good processes. Promote from within, and develop a strategy to give people a career path. Otherwise, you'll burn people out, and get huge turnover. That sucks for everyone.
You can purchase some really high end equipment to manage multiple monitors on a videowall, but you shouldn't. Use standard PC level hardware (or lower end rackmount depending on space requirements) with no more than two display cards each. Drive all your monitors separately then tie them together with Synnergy. You can still administer them all from a single workstation, fairly seamlessly, but you don't have a single point of failure, and you've probably saved hundreds of dollars. The videowall systems can also run some light duty servers especially system monitoring. (I like Xymon over Nagios, but it depends on what you want to do with it.)
So far as the monitors themselves, purchase flat-panel HDTV's. They are likely to be cheaper than similarly sized monitors, and you won't want greater resolution than an HDTV can handle for a video wall anyway. This gives you the added benefit of being able to tie in training videos, or third shift entertainment on to one or more screens if needed. Also, if one of your videowall servers goes down right before clients come to view the installation, you can quickly switch those monitors over to CNN, CNBC or another relevant channel.
The workstation tables should be glass or some other surface that can support either dry erase or grease-pen writing. Being able do simple notes on your desk will reduce scratch paper usage and make maximum use of available areas. Glass cubicle walls will cut down on noise like a cubicle would, but does not give as much of the feeling of being in a box as standard cubicles. They allow unobstructed view of the video-wall and you can write on them with grease pens.
Have more workstations than you need, and do not tie people their workstations. If someone wants to claim one that is fine, but some people will really like being able to log off, walk across the room, and log back on. This will also allow you to bring in off-shift workers when shit hits the fan.
As a security measure, get a dot-matrix printer on your firewall. Feed tail -f /var/log/authlog directly to it. If anyone gets in that shouldn't they will NOT be able to erase their tracks.
Put in a breakroom or break area that still has a view of the common videowall. When your people are taking a break during downtime, they should still be able to see if it is suddenly no longer downtime.
For the love of God (and your staff) put in a drink fridge or soda fountain and a coffee pot.
Little Brother, watching the watchers
If this doesn't end up at least vaguely resembling the bridge from either the TOS or TNG Enterprise, you're FIRED.
24x7 ops in a confined space that you want to actually work inside means caring about the two things overlooked by both IT and Management.
1) Human waste
Unless able/willing to get time outside the tank, people _are_ going to: snack,drink coffee & water, etc at their stations.
The bathroom (mens and womens) is going where? Outside the double tier biometric locked doors past the guard?
In a crisis or crunch the time use for bathrooms actually goes up as more coffee and crap food (fatty, sugary, glutten, etc) are ingested & a trip to the sewage system is required.
Also bathroom facilities need to be built with those things necessary for those with disabilities; with obestity / diabetes and those with really bad digestion (IE get a really good odour neutralizaing vent system) Also some sound damping is necessary. After a shift change would you want to hear 14 flushes in a row at your station?
If you want a real good idea of how human smells take over a confined space take Greyhound to anywhere on a trip longer than 8 hours.
After a few hours you'll be begging for fresh air & a decent restroom as well. You can figure out the max time people can hold it as would need to if they pass outside the security areas of your ops center.
2) Housekeeping & Janitorial
How are and just who will clean the bathrooms as well as the control room area?
The guys & gals making so little as opposed to the IT guys, but do the really important grunt work during the night to porter the bathroom with their cleaning carts and supplies.
Really, although the janitors have a key to everywhere they typically will block doors open. The cleaning team is so "trusted" it isn't questioned about bypass of security doors.
If Solid Snake could hide in a janitors cart not a cardboard box - he'd go anywhere.
Additionally these are also the people that are going to empty the waste bins, recycle bins, spritz down empty cubes/stations with disinfectant / de oderizers.
My former company had a policy: if techs are on a station that station will not be cleaned.
You have to have some sort of desk rotation to move out your personnel (that monitor now covered by a different station) so the area can be cleaned adequately.
If this is not done you deserve the thick stank that will descend upon your control room.
If your running really critical Ops: A HR policy on proper hygiene and showers should also a clause in the personnel contract. Everyone laughs until you have to term a tech for stinking to much.
As a bonus I'll add that my experience is that the AC design will never be adequate nor will the heating. It will be visited often by AC techs in the first 5 years until everyone gives up hope.
The failure is that AC is typically the retail mall design of a large scale dumping of cold dry air into a large volume of space and somebody's desk (hopefully not yours) is just under where this happens. Hot and cold spots are intractable in a large open floor plan arrangement.
I had a short stint with a fire department in a fairly large metro area. They handle emergency response for a few million people and have a pretty efficient set up for their war room.
It is fairly roomy so that you can get up and walk around.
desks are set up in pods with groups a people sitting facing towards the center of the hexagons they are on the perimeter of.
They have phone communications with headsets that allow them to move around
there are multiple monitors on each desk and large monitors on every wall that can be switched to show any desktop
The lighting and noise were somewhat subdued, but in no way dimly lit or overly comfortable (no high back chairs)
I do not remember the snack food situation, but there is a lot of security and it would be a pain to have to walk in and out all the time)
fwiw, I would save your company a ton of money and just use IP kvms and a software kvm management solution to tie together a bunch of desktops in a relatively open area together where the operators have some room to walk around, but are not overly distracted or lulled to sleep
ya know, massive operations centers are just soooo glass house IS anyway, totally 80's thinkin
Wherever You Go, There You Are
What hardware you have doesn't matter much. Any run of the mill dual headed desktop will do what I need - a few browser windows, a dozen xterms, and some email up in the corner. Anything else is just for show.
What's really valuable when I'm on point is documentation. I need to know:
What does this server do?
What's the procedure to rebuild it?
Who built it last time, in case I hit trouble?
Who's the business owner?
How do I reach them? A cell number would help.
What's the escalation path when they're AWOL?
Where are the contact numbers for our transit providers?
What are their SLAs?
Where's a map of our network, in case I have to creatively reroute traffic in ways that OSPF won't?
Is it up to date?
What services are exposed in the DMZ?
Where's the ticket requesting this port be opened?
Are there supposed to be 100,000 different IPs connecting here, or just three?
Where is the password vault?
What's the procedure to update the password vault if I have to change one?
Being able to find these things quickly will make me a much happier sysadmin than any creature comforts, excluding caffeine.
If you want to get into creature comforts in a windowless bunker, make it lighting. I don't want it bright, but it should be well designed to cover the space well. Good warm triphosphor fluorescents with high frequency electronic ballasts are much much nicer than the old cold ones with 120Hz flicker from magnetic ballasts. Color rendering index matters. That makes a good base lighting for the workspace. Then get a few of those industrial HID grow lamps, and have them light a big picture of a forest scene covering an entire wall. Or actually grow plants under them, and don't pay too close attention to what else people plant when you're not looking. Careful not to make it too bright, but the sun-like spectrum will break up the monotony of the fluorescents. Add some bright halogen task lights for when you need to see something well, without having to flood the whole room.
Raise the ceiling as high as you can. Rip out the ceiling tiles. Suspend the lights on cables. Let the ductwork show. Paint it all black. I hate living in a box. Exposing all the HVAC and cabling breaks it up, and I actually like the minimalist industrial aesthetic. If you want a softer look, hang some tapestries up amongst the machinery.
I used to work for a large insurance company in Chicago. The director charged with building our NOC in 2000 basically traveled throughout the country visiting other large IT organization's NOC's and took the best ideas and made them work for us - and it did resemble 24.
Take a large crescent shaped room with a 30' or more ceiling. The video wall was three different sections (this is important for separation of displays and multiple tools at the same time). The display units were high end rear projection systems that were each hooked up to computers that drove the display and were roughly 3'x5' each. Of course there's no seam or separation between the screens. Any group of screens can be used to display anything you want (1 screen, 2, 4, 6, all, etc). Pretty basic stuff nowadays, but it was great ten years ago. The left and right banks had three screens stacked on top of eachother, by either 4 or 5 wide. The center bank was 3 high by either 8 or 10 wide.
Three rows of crescent tables with low walls in front separating them, and minimal separation between workspaces - you want people in a NOC to work very closely with eachother, especially in case of an outage. Each station had two or three LCD screens mounted on articulating arms, but not to be stacked on top of eachother like those trading desks you see with 6 or 8 LCD screens at them. That would be too tall, and you couldn't look over the top to see the main video wall without standing. The room sat close to 50 people. Around the edges of the room are various cabinets, printers, personal storage for the three shifts of employees that work in the NOC, etc. Of course high end chairs are important as others have noted. Lighting is also equally important. You have to be very careful with making sure it is as close to natural lighting as possible. The lighting we used was recessed and inset so that no lightbulb shone directly out or down on the people - it made it less harsh, but still very bright in the room based on a good design. Wireless headsets are important, and also minimizing speakerphones and any other distracting noise.
Behind the rows of tables at the back of the crescent in the donut hole section if you will is an enclosed room large enough to sit 30 people comfortably with power, phones, and network connections to cover it. The walls facing the NOC are floor to ceiling glass, and it has connectivity to the videowall of the NOC so that displays from their can be sent to the meeting room as well. It has every high end normal conference room tool you could need - multiple video conferences, smartboard, integrated microphones and speakers, etc. Everything was hidden inside builtin cabinets made of high end wood. This main room is the situation room. During a large outage, 2nd and 3rd level staff will work out of the room in conjunction with the NOC teams. Directly upstairs from the situation room is another identical room, also with floor to ceiling glass walls looking out to the video wall of the NOC. This upper room was reserved for senior and executive management use during a large outage. Engineers and Executive management have different needs during an outage and require separate spaces and separate functions, although constant information does need to feed between the two. The upper room was more of the showpiece room. It had a motorized curtain that you could press a button on the wireless control panel to open and close. The entrance from the building going up to the second floor board room does not give anything away for what the NOC itself looked like, so once everybody was assembled in the room and the button was hit, it never failed to impress first time visitors. They would always leave their chairs at the conf table and walk right up to the glass wall to look down at the people working in the NOC and see what was displayed on the board.
It was an extremely impressive setup. I am now in sales and visit customer sites on a daily basis and I have yet to see something that even approaches what this
I am the OP. Thanks for your advice. You've saved us millions. No, billions.
Now that you've established I am clueless, go away and read some of the genuinely insightful comments on this thread and you'll see precisely why I asked Slashdot. It's called canvassing a broad range of opinions. Even the world's best consultants/designers would have only designed a handful of these personally. I don't want to make any expensive mistakes and I am quite happy to consider the opinion of a low-ranking staff member earning a few bucks an hour sitting in these places before I subject our low-ranking staff to the same mistakes.
And FYI, my consultants are reading this thread and loving it.
Do you or your partner snore? - Visit www.snoring.com.au