Slashdot Mirror
Design and Evaluation of Central Control Room Operations
brothke writes "In aviation today, technically advanced airplanes present a unique paradox. Technically advanced airplanes, in theory, have more available safety, and the outcome should be that there are fewer accidents. But without proper training for their pilots, they could be less safe than airplanes with less available safety. The FAA found that without proper training for the pilots who fly them, technically advanced airplanes don't advance safety at all. The reason is that technically advanced airplanes present challenges that under-prepared pilots might not be equipped to handle." Read on for the rest of the review.
Human Factors in the Design and Evaluation of Central Control Room Operations
author
Neville Stanton, Paul Salmon, Daniel Jenkins, Guy Walker
pages
446
publisher
CRC Press
rating
10/10
reviewer
Ben Rothke
ISBN
978-1439809914
summary
Invaluable reference that can be used for the design, assessment, evaluation an operations of NOCs and SOCs
In the IT world, staff members are often expected to install, configure, maintain and support technically advanced software. Companies often buy huge infrastructure software, such as CRM, ERP, PKI, identify management, intrusion detection and more, without first understanding how to make them work in their complex environment. Management often is oblivious to the fact that just because they can buy and install the software that it will not work on its own. The reason why so many large software deployments fail miserably is that the IT staff often doesn't have the proper training, support and assistance that they need.
Human Factors in the Design and Evaluation of Central Control Room Operations is a fantastic book that shows what it takes to ensure support staff work and operate together, in a formal and efficient manner. The book integrates the topics of human factors and ergonomics to create an incredibly valuable tome. The book details the interactions between people and their working environment, and shows in depth how the work environment can and must be designed to reduce errors, improve performance, improve the quality of work, and increase the work satisfaction of the workers themselves.
While the book was written primarily for control room settings, it is relevant for those in IT if they have any involvement in remote support, security operation centers (SOC) and network operation centers (NOC).
While the book is of value to anyone involved in operation, those who will find the most value are those charged with the management and operations or large groups or operations. If they have management support to deploy the formal methods detailed in the book, they will find that they can create significantly higher levels of customer and end-user satisfaction.
The authors note that all SOC and NOC's have a common feature in that the people operating them are often remote from the processes that they are monitoring and controlling, and the operations function on a 24/7 basis. The many demands of remote and continuous operation place special considerations on the design of the SOC and NOC. The output of the book is that it can be used to effectively to design these operating centers.
The books presents a comprehensive and all-inclusive on the topic of human factors on the following 14 topics: competencies, training, procedures, communications, workload, automation, supervision, shift patterns, control room layout, SCADA interfaces, alarms, control room environment, human error, and safety culture. Each chapter includes extensive diagrams and flowcharts to show how the processes develop.
The book also provides a highly analytical approach to each topic. It details the required processes and procedures necessary to make each subject area work. The book is not only based on the four author's expertise; they quote heavily from other experts and their research.
Chapter 2 opens with the observation that the safe and efficient operation of operating centers and control rooms is dependent upon the competence of the operators working within them. It details how to create competence assessments to ensure that staff is capable of carrying out their tasks safely and efficiently by assessing their skills and knowledge. The authors stress that it is not acceptable for organizations to assume that their staff are competent based on only their exposure to training and experience. They suggest that organizations create a program to determine those competence levels.
Chapter 3 goes into detail about how to create effective training programs to ensure worker competence. The benefit of a trained worked is that they can yield higher productivity and provide better service. Well-trained workers often have better morale and produce less errors. The chapter details the importance of a training needs analysis to properly determine what needs to be in the curriculum.
Chapter 4 is on procedures and is particularly important to those working in a SOC or NOC. If consistent and repeatable procedures are created, staff can provide much a more effective and dependable levels of service. Even with the benefits of well crafted procedures, its development process is a complex one involving the identification of all of the tasks that require procedures, a judgment on the level of assistance required, identification of the type or format of procedure required, writing and reviewing the procedures, and obtaining approval for them.
The importance of procedures is underscored when the book notes research that 70% of accidents and incidents within the nuclear power companies occurred when workers failed to properly follow procedures. In the petrochemical industry, 27% of incidents were caused by situations for which there were inadequate or no procedures available.
The percentage of failed IT projects and large software rollout catastrophes is both staggering and appalling. No other sector but IT would tolerate such failures. A book like as Human Factors in the Design and Evaluation of Central Control Room Operations goes a long way to stop that. The book is a rare one in that it both provides all of the factors involved in the problem at hand, and then provides all of the details needed to obviate those problems.
Ben Rothke is the author of Computer Security: 20 Things Every Employee Should Know.
You can purchase Human Factors in the Design and Evaluation of Central Control Room Operations from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
Human Factors in the Design and Evaluation of Central Control Room Operations is a fantastic book that shows what it takes to ensure support staff work and operate together, in a formal and efficient manner. The book integrates the topics of human factors and ergonomics to create an incredibly valuable tome. The book details the interactions between people and their working environment, and shows in depth how the work environment can and must be designed to reduce errors, improve performance, improve the quality of work, and increase the work satisfaction of the workers themselves.
While the book was written primarily for control room settings, it is relevant for those in IT if they have any involvement in remote support, security operation centers (SOC) and network operation centers (NOC).
While the book is of value to anyone involved in operation, those who will find the most value are those charged with the management and operations or large groups or operations. If they have management support to deploy the formal methods detailed in the book, they will find that they can create significantly higher levels of customer and end-user satisfaction.
The authors note that all SOC and NOC's have a common feature in that the people operating them are often remote from the processes that they are monitoring and controlling, and the operations function on a 24/7 basis. The many demands of remote and continuous operation place special considerations on the design of the SOC and NOC. The output of the book is that it can be used to effectively to design these operating centers.
The books presents a comprehensive and all-inclusive on the topic of human factors on the following 14 topics: competencies, training, procedures, communications, workload, automation, supervision, shift patterns, control room layout, SCADA interfaces, alarms, control room environment, human error, and safety culture. Each chapter includes extensive diagrams and flowcharts to show how the processes develop.
The book also provides a highly analytical approach to each topic. It details the required processes and procedures necessary to make each subject area work. The book is not only based on the four author's expertise; they quote heavily from other experts and their research.
Chapter 2 opens with the observation that the safe and efficient operation of operating centers and control rooms is dependent upon the competence of the operators working within them. It details how to create competence assessments to ensure that staff is capable of carrying out their tasks safely and efficiently by assessing their skills and knowledge. The authors stress that it is not acceptable for organizations to assume that their staff are competent based on only their exposure to training and experience. They suggest that organizations create a program to determine those competence levels.
Chapter 3 goes into detail about how to create effective training programs to ensure worker competence. The benefit of a trained worked is that they can yield higher productivity and provide better service. Well-trained workers often have better morale and produce less errors. The chapter details the importance of a training needs analysis to properly determine what needs to be in the curriculum.
Chapter 4 is on procedures and is particularly important to those working in a SOC or NOC. If consistent and repeatable procedures are created, staff can provide much a more effective and dependable levels of service. Even with the benefits of well crafted procedures, its development process is a complex one involving the identification of all of the tasks that require procedures, a judgment on the level of assistance required, identification of the type or format of procedure required, writing and reviewing the procedures, and obtaining approval for them.
The importance of procedures is underscored when the book notes research that 70% of accidents and incidents within the nuclear power companies occurred when workers failed to properly follow procedures. In the petrochemical industry, 27% of incidents were caused by situations for which there were inadequate or no procedures available.
The percentage of failed IT projects and large software rollout catastrophes is both staggering and appalling. No other sector but IT would tolerate such failures. A book like as Human Factors in the Design and Evaluation of Central Control Room Operations goes a long way to stop that. The book is a rare one in that it both provides all of the factors involved in the problem at hand, and then provides all of the details needed to obviate those problems.
Ben Rothke is the author of Computer Security: 20 Things Every Employee Should Know.
You can purchase Human Factors in the Design and Evaluation of Central Control Room Operations from amazon.com. Slashdot welcomes readers' book reviews -- to see your own review here, read the book review guidelines, then visit the submission page.
I'm not reading the review. If it's half as painful to read as the summary was, I think I'll save myself the agony.
Seriously, redundant much?
They are running Linux, thus making it unnecessarily complicated :P
Fellow slashdotters - this is a book with tips for improving our own command centres!
One that hath name thou can not otter
"In today's world, technically advanced cars present a unique paradox. Technically advanced vehicles, in theory, have more available safety, and the outcome should be that there are less accidents. But without proper training for their drivers, they could be less safe than cars with less available safety. NHTSA found that without proper training for the drivers who drive them, technically advanced cars don't advance safety at all. The reason is that technically advanced vehicles present challenges that under-prepared drivers might not be equipped to handle."
This could explain some of the Toyota crashes. The drivers don't understand what they need to do to slow down and stop the car when the accelerator acts like it's stuck.
If I used a sig over again, would anyone notice?
Wrong summary much?
-- I really need to bleed off some of this
Technically advanced airplanes are technically advanced. And airplanes that are technically advanced are advanced, technically.
My one and only car crash, due to my fault, is due to ABS. It was my first ABS car and I tried to pump the brake, and failed to stop before hitting the other car.
It was a minor fender bender, never mind any injury, but it burned it into my skull: slam the brake.
Fuck systemd. Fuck Redhat. Fuck Soylent, too. Wait, scratch the last one.
and it was all like 'beepbeepbeep', and I was like 'huh?' It devoured my airplane. And it was a really good airplane."
It's not really a paradox. The simple solution is this: upgrade the technologies as the budget allows and train the damned pilots. It's not the kind of system where you have to have everything happen at once - this can definitely be done in phases.
Hard stuff not obvious, people need training.
I am Slashdot. Are you Slashdot as well?
I designed a Nagios interface especially for Control Rooms. My program can be run on a large screen hanging on the wall and then display a list of problems. Of course has a nice web-interface for remote configuration ;-)
It is called CoffeeSaint.
www.vanheusden.com - home of Multitail, HTTPing, CoffeeSaint, EntropyBroker, rsstail, bsod, listener, nagcon, nagi
The authors note that all SOC and NOC's have a common feature in that the people operating them are often remote from the processes that they are monitoring and controlling, and the operations function on a 24/7 basis. The many demands of remote and continuous operation place special considerations on the design of the SOC and NOC. The output of the book is that it can be used to effectively to design these operating centers.
You need a big fridge, and a microwave.
Another common feature is at least of all NOCs I've seen is marketing wants the most stylish looking facility they can get, which is often/always completely at odds with the goals of an effective facility.
Common noc mistakes:
1) Everyone crammed in like sardines so "we can work together". Except that no one noticed that we don't work together. All it makes is a lot of noise and interference. No space to open a stack of manuals, closely related to no space for usable computer monitors (as opposed to the ones used only for show). Even worse, designers seem addicted to adding "static noise" masking generator, crappy elevator music, and/or a PA system for other departments blaring away. Thru careful work, its possible to include features to make it look like its ideal for cooperation, yet make actual cooperation impossible due to noise level etc.
2) Extraordinarily expensive big screen TVs / monitors / projectors on all the forward facing walls, that no one actually uses. Too small, too low res, no actual business purpose. This is a killer two ways, first of all its a huge capital expense that could have paid the salary for extra techs for years, which would have a measurable positive effect. The other way its a killer is you'll actually take people off productive work to "fix" the big screens so marketing is happy. Would anyone in the NOC have a problem doing their job if all the projector bulbs burned out? No, but marketing would freak out.
3) Second class citizen status. "Real" employees can have family pictures in their work area. The dogs of the "noc", not so much. This attitude flows thru the organization in many other ways, producing discontent. Promotion out of the noc becomes a goal, not to "advance" but just to get the hell out.
4) Constant over the shoulder monitoring. No matter if its marketing, or management, there seems an utterly desperate desire to perch over the NOC workers shoulders, either physically or virtually. A great employer-employee attitude if you are 17 years old and working at taco bell. Not a great attitude in a professional noc environment. There seems something inherent in all NOC management that makes them distrust their employees, that you generally don't see in most other departments. Kind of like having the ability to treat them as serfs inevitably makes it a requirement to treat them like serfs.
5) You know those 1970's "sunken livingrooms"? alive and well in the nocs of the world. How about the 1960s original star trek theatre in the round concept with a bridge in the middle? alive and well in the nocs of the world. Remember the set of "wargames" from 1983? Why can't a noc be designed that doesn't look like a throwback or parody? At least try something different, like a medieval dungeon or something?
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
A distant relative by marriage bought a top of line new Mercedes. He has to attend a Saturday long training class for four Saturdays.
After laughing in disbelief. I asked why does he need a training class. In a nutshell, apparently there's so much electronic bullshit (I guess to justify the price of the car somehow) that they need classes. GPSes, internet and whatnot
I started thinking about all that distraction for the driver. I can just see liability insurance eventually going through the roof - while the old rich guy is fucking around with his car, he goes and rear ends someone or runs over a pedestrian.
There's a point where all this complexity becomes counter productive and makes the car more unsafe.
The notion that technology saves the day without introducing new tasks, quirks, and procedures is naive thinking at its worst. The Office 2007 ribbon might be a fair example. People who memorized countless shortcut keys and menus for Office '97 through 2003 suddenly have to reacquaint themselves with the shortcut bar for n00bs, with no simply way to revert to the old setup. Fun times are had by all. Also, as noted above:
Management often is oblivious to the fact that just because they can buy and install the software; that it will work on its own
This tends to be responsible for about 90% of all tech woes for deployments I've ever encountered.
or does the summary have nothing to do with the review?
The author of this article must be a tautologist!
XKCD address this a while ago.
http://xkcd.com/703/
.....Bill-yuns and bill-yuns of itty-bitty buttons, all alike.
Regards;
My late Grandfather used to tell me stories all the time of stuff that happened in the air traffic control tower at Charlotte-Douglas airport (he was the chief of the tower for 20 years). 99% of the time, human error is to blame, for stuff like military pilots trying to land at the wrong airport and not listening to comms to stuff like poor pronounciation over comms to pilots causing close calls. By the end of his career, he found that work ethic was more important than credentials, because laziness was the cause of a large majority of mishaps. It is difficult to train work ethic, and easy to impart knowledge.
Rewarding diligence and establishing a culture of consistency was their solution. I've seen it work in other industries over the years as well. But, when you think about it, isn't it just common sense to do it? Why do you need a book to explain that?
... is quite involved and requires careful thought. Training and procedures are important, but the best UIs should make the next step(s) in a task obvious. A symptom of an overly complex, poorly thought out UI is the high level of training and checklists needed to identify the next step(s) or locate required data. Using the flight deck model, older airplanes needed a large amount of training (and a third crew member) because all of the instrumentation and controls were just mounted on a few panels, with no guidance as to which dials and knobs would require special attention under various different flight conditions. The modern flight deck makes use of flexible displays that remain quiet (dark) until they demand attention. Then, they are presented in a manner which suits the particular task at hand. Like an automated checklist.
If thought out carefully, a good UI can be quite intuitive to use, with little or no training. Apple is particularly good at creating such intuitive interfaces. I've created a few apps that were deployed to the shop floor with no training required whatsoever (other than the URL for the top page). One in particular was developed as a proof of concept demo and leaked to the shop. It was intended to demonstrate one possible approach to replacing a crappy command line interface. But when the factory people saw it (it was implemented using production data) and just started using it, management just made it the standard tool. I was never approached (until much later, for process documentation reasons) to write a users manual, or any instructions whatsoever, for its use. Subsequent additions of functions were also accomplished with no training requirements as well. If a new feature was added, it appeared as a selectable option (sometimes with an animated "New" flag) in the appropriate location for the process being performed.
Have gnu, will travel.
Here's an example of just that - the new Moesk control center for Moscow's electric network.
Take a look at the pictures. This looks like a movie set for a Bond movie. The architects got completely out of control here.
Notice the suspended transparent bubble for top management. It looks like it retracts into the ceiling. The lower operator's platform has steeply slanted sides, no railings, and chairs with wheels. The huge room only has eight operator positions.
I'll bet that, within a year or two, the people who actually have to run the grid set up a "field control center" with about twenty people with PCs, cork boards on the walls, 2-way radios for talking to field crews, a conference/map table, and some printers. The real work will be done there. A few people will sit in the big room and answer questions for management.
There is a broader lesson to be drawn from this -- the ability of the staff to provide the correct results depends not only on their having the necessary tools and skills but being able to apply them effectively. What the FAA found in the latest Buffalo crash -- that the crew were so fatigued that their judgement was impaired applies elsewhere. A number of family members work in medicine -- where long hours and screwy work schedules are the norm. One would think that in areas where judgement affects lives that there would be serious attention to work schedules that would maximize their ability to perform. But the policy appears to be the reverse -- scheduling their work in ways that seem designed to minimize their effectiveness. So IT is not the only place where there is a curious acceptance of diminished outcomes for reasons under managements' control.
If it isn't obscenely expensive, incredibly wasteful and even more incredibly useless, it wasn't designed by a Russian.
Face it, you think the US is bad about being showy, but Russia takes the cake in large, showy, unnecessary things.
I'll bet that, within a year or two, the people who actually have to run the grid set up a "field control center" with about twenty people with PCs, cork boards on the walls, 2-way radios for talking to field crews, a conference/map table, and some printers. The real work will be done there. A few people will sit in the big room and answer questions for management.
At a previous employer, for a Very Important Photoshoot for marketing, they hired college age models to staff our center in the pictures, apparently because the real personnel were far too unphotogenic. I believe the age of all the models added together still didn't reach the age of some of our old timers.
Kind of like how anytime you see a call center in marketing material, its always staffed by stereotypical beauty pageant white women, where in reality most (not all) call centers have been moved to prisons and 3rd world countries. I've often wondered what the prisoners and 3rd worlders think when they see those advertisements (other than the obvious, americans are idiots, etc)
"Science flies us to the moon. Religion flies us into buildings." - Victor Stenger
I say this as someone who doesn't work in the industry - what a cool design. Completely back asswards in some respects but it looks cool.
Man blir trött av att gå och göra ingenting.
Wow, someone hit the CAD program right after watching Minority Report, didn't they?
I toured a company (big web analytics company in Utah) awhile back and they showed us a NOC that was so impractically designed, I honestly thought it was fictional-- I actually told my co-workers, "I bet that's just for showing during tours, and the real NOC is in the basement somewhere." I wonder if the two or three guys who were staffing it were just shoved in there and told, "look busy! There's a tour coming through!"
Comment of the year
(see under Sperry Attitude Gyro and under-trained pilot).
Hey! The Cone of Silence was a required deliverable insisted on by management, okay?
The appropriate response would be tl;dr
Enjoy the rest of your stay!
Set your phasers on "funky"!
Hey! The Cone of Silence was a required deliverable insisted on by management, okay?
Some of those pictures are renders, and some are real. It turns out that the suspended transparent bubble was deleted before actual construction, along with the oval lighting rings and the curved-glass viewing galleries. Those are renders. The ones with conventional lighting on the ceiling are real, including the fancy podium. In this picture, the PCs, which look like conventional mini-towers, and their cables are in place.
So it's really a big, square mostly-empty room, with a bunch of Barco rear-projection monitors in front, and a fancy podium that cost US$90,000. The architect writes "this is main room in large building of company, which controls all electricity in Moscow. In this room 6 people working 24/7. Working group will not expand, but space is needed for groups of high ranked visitors."
That might actually be a reasonable design, given that problem. I can see the chief operator saying to the architect "Put enough empty space behind us so that when the oligarchs come to visit, they don't get in the way. Don't give them any chairs, or they'll hang around and muck things up. Put a VIP lounge somewhere else in the building, with the booze and the girls. Stick a few screens in there to repeat the big board, so they think they're running things."
Looks like the designer had just finished watching Men In Black...
From what I hear the setups of such rooms often play out like penis-length comparisons.
If someone has 100 modules, the next one wants 150 and so on.
And this does not only go for the Russians.