Slashdot Mirror
Ideas For a Great Control Room?
lewko writes "Our company is about to build a central monitoring facility and I'm looking for ideas/suggestions about the best hardware and the best way to make it comfortable for those manning a screen. It will be manned 24x7 and operators will be monitoring a variety of systems including security, network, fire, video and more. These will be observed via local multi-monitor workstations and a common videowall. This is going to be a massively expensive exercise and we only get one chance to get it right. The facility is in a secure windowless bunker and staff will generally be in there for many hours at a time. So we have to implement design elements which make it a 'happy' place. At the same time, it has to be ergonomically sound. Lastly, we will be showing it to our clients, so without undoing the above objectives, it would be nice if it was 'cool' (yet functional). Whilst Television doesn't transfer to real life always, think 'CTU' from 24."
Whatever you do, don't make it dark with blue-ish lighting, like on TV. That strains your eyes. Provide good lighting, and make sure the persons can sit or stand comfortably while watching the screens.
Here is a blog post with some thoughts about the control room design at the LHC. Here is a picture of the CMS detector control room for comparison. You might also take a look at pictures of the CERN control room for some ideas.
Numerous commentors here point out the error of using cool colored dim lighting for a facility where actual work gets done. But you can have your cake (or Cake, if you are using cool indie rock background music) and eat it too - just have a button that switches the lighting to "cool mode" whenever a visitor comes in. Meaningless but cool looking graphic "screen savers" could also pop up on the screens.
Starships were meant to fly, Hands up and touch the sky - Nicky Minaj
I would contract with a design/build firm that specializes in these facilities. You don't want to do this by the seat of your pants relying on Slashdot advice.
My company invested millions of dollars into a central monitoring facility, with a large video wall driven by Crestron equipment. The idea was the video wall could display news/weather alongside alarms/outages in real time, with geographic mapping capabilities. Workstations were quad displays on adjustable motorized desks which sat atop a raised platform for simple network runs. A large executive "war room" style conference room was built with a glass wall overlooking the platform and video wall believed to be useful in the event of some catastrophic failure. All other staff sat in cubicles surrounding the platform with glass cube walls anywhere that would otherwise obstruct the view of the platform/video wall. A secure mantrap was put in place to restrict access to the facility. Dedicated bathrooms were installed with showers in the monitoring area in case critical staff were quarantined for extended periods of time.
It was impressive when it was built, but within a couple years, the video wall has been dismantled and parts sold off due to its impracticality. The right software was never found to perform the type of "geographic" monitoring conceived, partly due to bureaucracy. Network redundancy was overlooked, which made the monitoring facility itself non-functional during an outage. The facility lacked appropriate backup generators and UPS to keep the facility running during a thunderstorm. The platform desks required too much real estate and allowed no room for growth, so they have been replaced by cubicles. The secure mantrap was an inconvenience for upper management, so the inner door was disabled, defeating the mantrap. The quad displays ironically obstructed the view of the video wall when it was still in place, and did not fit in the cubicles when they were installed, so these were reduced to 2. All critical staff were sent home to telecommute because they took up too much real-estate required for day-to-day operations, and it made more sense to not have critical staff in a single central location anyway.
The point is, don't get too caught up in building 'CTU' from 24. The right monitoring software platform makes all the difference, as does intelligent network redundancy, telephony and backup power.
I'm not an expert, but I play one on slashdot.
So, a couple things to think about:
1. How much money do you really have? Lots of people think they have money, but run out when it comes to all the details.
2. Do you want flash, or functionality? The two are sometimes complementary, but often one trumps the other.
3. How many people will staff? What's the schedule? This helps you figure out workstation configurations.
4. Are you putting multiple tiers in the same room? This is "best practice" if you do it right.
5. Are you handling customer calls directly? Do you deal with customers?
Basically, you need to figure out a lot of goals. A true "global" NOC can cost $50M easily for a telecom or comparable organization.
I've been a big fan of Barco for large projectors, and their IP-based solution is quite powerful. Recently, I rolled out a "public safety" SOC (security operations center) with 8 SXGA+ rear-projection displays. The largest I've worked on was 40+ of that style of display. Your garden-variety projector isn't cut out to handle this kind of duty-cycle. They're not cheap, but they're designed to operate 24x7x365, and many models have multiple lamps, etc. so that you can service them while they're online. So here's a few more things to think about:
* What goes on the "big screen" has to be useful. It must be grokable in a very short period of time. If you can't look at it for 2 seconds and get a good idea of what's going on, it's too complicated.
* Multiple displays per operations person
* Operational "graphs" that show overall statistics that matter to the people working, not to management.
* Good task lighting. Good lighting period is everything. Pay a real designer to do this.
* Good seating. We have let operations people pick chairs that fit their needs. Expect to spend $800-1k/person on seating.
* Sound deadening/management. NOCs get loud, and managing the acoustics is important to make sure that people can "think" and they can interact with one another.
* Ticketing is everything. Look at systems that are available commercially and for free. Consider writing your own if needed. If the system is streamlined to your own business, it will always be an impediment to getting the job done, which means people won't use it. If they don't use it, lots of knowledge is lost and post-mortems are more difficult.
Also, a few things that seem superfluous, but ended up being critical in some places I've worked (not all these were at the same place):
* Virtualized desktops (think RDP, X11, etc.) so that people can move and maintain their setup :)
* Color-shifting lighting to compensate for normal rhythms of people on weird shifts. Turns out green is effective after lunch at helping people maintain focus. This isn't cheap, but it sure does have a big impact.
* Keep your customers OUT OF THE NOC. A glass wall into the NOC is fine, but actually letting them in is distracting, and depending, can come with legal issues around privacy, HIPAA, etc. Best to keep them at a distance.
* Before you let customers see the NOC, you warn people. We had a blinking lighting strip under the displays that was linked into the Crestron system so that you couldn't flip the LCD-glass for 10 seconds to give NOC operators a warning. You don't want customers seeing people picking their nose.
Finally, as nice as good facilities are, if you don't have the process and people, it's useless. People people people people. Good people create good processes. Promote from within, and develop a strategy to give people a career path. Otherwise, you'll burn people out, and get huge turnover. That sucks for everyone.
24x7 ops in a confined space that you want to actually work inside means caring about the two things overlooked by both IT and Management.
1) Human waste
Unless able/willing to get time outside the tank, people _are_ going to: snack,drink coffee & water, etc at their stations.
The bathroom (mens and womens) is going where? Outside the double tier biometric locked doors past the guard?
In a crisis or crunch the time use for bathrooms actually goes up as more coffee and crap food (fatty, sugary, glutten, etc) are ingested & a trip to the sewage system is required.
Also bathroom facilities need to be built with those things necessary for those with disabilities; with obestity / diabetes and those with really bad digestion (IE get a really good odour neutralizaing vent system) Also some sound damping is necessary. After a shift change would you want to hear 14 flushes in a row at your station?
If you want a real good idea of how human smells take over a confined space take Greyhound to anywhere on a trip longer than 8 hours.
After a few hours you'll be begging for fresh air & a decent restroom as well. You can figure out the max time people can hold it as would need to if they pass outside the security areas of your ops center.
2) Housekeeping & Janitorial
How are and just who will clean the bathrooms as well as the control room area?
The guys & gals making so little as opposed to the IT guys, but do the really important grunt work during the night to porter the bathroom with their cleaning carts and supplies.
Really, although the janitors have a key to everywhere they typically will block doors open. The cleaning team is so "trusted" it isn't questioned about bypass of security doors.
If Solid Snake could hide in a janitors cart not a cardboard box - he'd go anywhere.
Additionally these are also the people that are going to empty the waste bins, recycle bins, spritz down empty cubes/stations with disinfectant / de oderizers.
My former company had a policy: if techs are on a station that station will not be cleaned.
You have to have some sort of desk rotation to move out your personnel (that monitor now covered by a different station) so the area can be cleaned adequately.
If this is not done you deserve the thick stank that will descend upon your control room.
If your running really critical Ops: A HR policy on proper hygiene and showers should also a clause in the personnel contract. Everyone laughs until you have to term a tech for stinking to much.
As a bonus I'll add that my experience is that the AC design will never be adequate nor will the heating. It will be visited often by AC techs in the first 5 years until everyone gives up hope.
The failure is that AC is typically the retail mall design of a large scale dumping of cold dry air into a large volume of space and somebody's desk (hopefully not yours) is just under where this happens. Hot and cold spots are intractable in a large open floor plan arrangement.
I worked for a couple of months in doing NOC work, you pick up some basic must-haves really quickly in regards to comfort, I'd like to add them. Temperature and air quality is among the very top priorities for sure, here are a few other points.
Get the best and most comfortable chairs you can find and then get the ones that are even better. People will be sitting in these 24/7, you need the best you can get or they'll get worn down in mere months, both chairs and people. Full range of adjustability is essential, as is full back and lumbar support. Look at the seats in taxis, cop cars, any sort of car where people do long shifts, look at long-haul trucks if you can. Taxis where I live are almost uniformly Mercedes-Benz or Volvo, partly because of their excellent seat ergonomics.
Get people out of the chairs. I know this sounds silly considering you've just bought amazing chairs, but give people options. Exercise balls, kneeling chairs etc., let people mix it up so they aren't stuck in an ordinary sitting position for hours on end. Some basic exercise equipment is good too, doesn't have to be anything more fancy than some wall-mounted bars etc., just to let people stretch a bit.
Quick access to toilets and kitchen(ette). This type of work demands as little downtime as possible. Make sure the kitchen has at least two microwave ovens, depending on the number of people who'll be working there, a good meal is essential to getting through a night shift and even the slight inconvenience of having to wait a bit longer can be amplified by being sleepy and in a job where you deal with very stressful situations on a daily basis. Night shifts have been shown to have a correlation with heart disease and possibly cancer. Don't make people wait too for their food as well :-) The place where I worked had lots of people doing 9-5 jobs as well and the cafeteria always kept well-stocked vending machines with sandwiches and the hot dish of the day for the night shifters. It was highly appreciated.
Speaking of the kitchen, get really good coffee. No scratch that, get great coffee and reliable high-capacity coffee makers. These people will suck down black coffee like you wouldn't believe. Keep the fridge(s) well-stocked with ice-cold Coke (or whatever caffeinated soft drink they like) for the non-coffee drinkers.
I cannot stress this enough: Have fresh fruit available at all times. This was an absolutely life saver for me. You haven't had a tasty apple until you've bitten into a fresh, ripe Golden Delicious at 4:30 in the morning after a long nights stressful work. Keeps people from gorging on chips and other unhealthy snack foods as well.
Lastly, let people have their distractions. We used fancy multi-monitor setups where we could put just about any system we monitored on whichever monitor we wanted, very slick. I can't remember what they called it, but it seemed to work pretty well. My point is that the universal setup seemed to be one monitor for desktop stuff, one for TV (most people had 24/7 news on) and the rest for monitoring etc.. The TV feed was brilliant, during the downtime you could catch up on the news and so on, it helped you get through the boring parts of a night shift.
I'm sure I've forgotten a lot of stuff, but these things stood out in my mind as the most important.
Eat the rich.