Ah, yes & no - those protections you speak of are in the PLC and controller code, which may well be able to be changed via these apps or vulnerabilities exposed to or by these apps.
Of course, we try to ensure no console/operator can blow things up, but they can do many bad things, like mix explosive chemicals, run at unsafe speeds/temps with various material mixtures, over-tension, etc. The control system can't know everything in complex systems.
Plus lots of systems have manual modes and sequencing that depend on operator brains and skills, and perhaps maintenance modes.
Having any Internet or phone connection to control systems boggles this old control engineer's mind; foolish in the extreme.
YES we are, in every area, but jobs are in Shanghai. We are in fact looking for NOC engineers and process people. Senior engineers in all areas: Linux, DBA, Security, Performance, Troubleshooting, tools, managers and much more. We are building the world's top MSP and running numerous multi-hundred mullion user systems, doing the most difficult things on the Internet today.
Hmm, as the OP I value Slashdot's input and ideas on these things.
Our life and what we do is a tad more complicated than most others, in fact, quite a bit more complex than anyone I talk to, and despite my and our decades of experience in these areas, and sustained global searches for solutions, we often have to invent our own systems and technology - you'll see more of this from us over the next 24 months as we open source our best Ops and Management tools.
By the way, my thread on password management resulted in nothing useful as seems what we need does not exist. Good SaaS opportunity, I think. And that's our small password issue, we have much larger and more challenging security challenges that need world-class solutions we may have to yet again invent.
In this case, we have limited experience on modestly large NOCs and what people are doing for the PC selection, mounting, wiring, etc. as this is not our area, hence asking all of you for your input - and lots of good ideas and thoughts here - we'll post pictures and diagrams of what we end up with.
OP here and agreed - we are using professional display room vendor and thus the screens are commercial duty 7x24, mounted on modular walls. Though realistically screens are cheap enough that replacing them over time is not a huge burden, as long as we can manage the sizes which is my biggest worry (i.e. new 42" screen same size as old 42" or 41", etc.)
OP here and the goal is both though if we have to trade-off, good for tours is more important, but I can have both, frankly as we've been doing this many years and this is just a new upgrade. We run our primary and secondary monitoring systems up on the screens, active ticket lists, rule-based alerting on both business and tech stuff, action plan status, notice-of-the-day info and changes, change controls, active engineer work and ssh, email and IM session/ticket tracking, and a lot more across all these systems.
This is where we do real-time analysis, task routing, ticket management, communications and escalation, and more - 10-15 people will work in this room across 5+ areas 7x24 (Support, Coordination, Alerts, Requests, Security, Performance, DBA, Escalation, Scheduling, Leads and Managers).
The NOC handles about 1,000 events/week (in addition to the automated systems, tasks, responses) so it's a very busy place. Our core SLAs are 5-15 minutes for hundreds of customers and thousand of servers/systems.
Plus for emergencies we can/want to route laptops to big screens for shared team work so we have several meeting tables in the room to test this process (the offices have other dedicated rooms for this, too).
As the original poster, I want both - yes, we are in China and flashy tours of shiny command centers are helpful to customers who see this as professional - we have one now and it is very useful in the tour and sales pitch. Amazingly effective, actually, hence why I'll invest in it.
You are correct that the actual usefulness of lots of monitors are limited, though we do have lots of systems and info to display, dashboards, alert and rule systems, and much more so there is actual info architecture here, too - and we'll run some emergencies from the NOC also where more displays routed from laptops for team troubleshooting is helpful.
Beyond that, actually this is hardly a dingy area but in fact a Class A full-buildout with state-of-the-art offices (though open plan for large teams), glass walls, soft lighting, large cafe with game areas, rest areas, private phone/1:1 rooms, and more - think Google and AT&T, rather than dingy cubes.
Original Poster here - yes, these are all good suggestions and we should add more LDAP (we have large multi-thousand host LDAP systems now), but a lot, if not most of these systems we need, especially various SaaS tools, don't support this well, if at all. So a full SSO system is a real challenge - we are looking at AD integration next year to handle the ones that can.
But I don't really need this today - what I need is to TRACK all the system access, in part just to know what systems Johnny in Ops Engineering, etc. needs access to at what level, to notify the system owners to add/remove that, to track who added access and when, etc. as this happens over several days/weeks for new employees.
And to manage changes, which are of course frequent as this fall we add at least one new system per week - the cloud and SaaS is great, but managing users is not (assuming the system owner even reads the docs, manuals, sets roles correctly, etc.).
Today we have a huge XLS for this with common all-employee systems like HR, ERP, Email, etc. then per department blocks, then per role, then special stuff. It's pages long, and each item ties to an SOP, system access owner, etc.
And this is all just business systems, totally separate from our customers' operational systems, AWS/Alibaba/Rackspace/etc. IAM integrations, and our real work, which is totally separated and managed differently (hence the big LDAP systems, ticket integration, password managers, etc.)
So thinking we need to build a basic auth-like system but just that tracks users, roles, systems, roles in those systems, requests, approvals, changes, etc. But would have hoped this already existed.
I don't think I agree with his analysis of writing tests first; or rather, I believe he misses the larger point. Yes, programmers will tend to write tests that mirror the programs they are testing, but I find they also tend to force developers to at least think about testing, what's correct, and how things fail. It's not perfect and if everything is well-designed it may not matter, but in real applications, writing tests first can cut out at least a few code/run/fix cycles.
Slashdot Mirror
Ah, yes & no - those protections you speak of are in the PLC and controller code, which may well be able to be changed via these apps or vulnerabilities exposed to or by these apps.
Of course, we try to ensure no console/operator can blow things up, but they can do many bad things, like mix explosive chemicals, run at unsafe speeds/temps with various material mixtures, over-tension, etc. The control system can't know everything in complex systems.
Plus lots of systems have manual modes and sequencing that depend on operator brains and skills, and perhaps maintenance modes.
Having any Internet or phone connection to control systems boggles this old control engineer's mind; foolish in the extreme.
YES we are, in every area, but jobs are in Shanghai. We are in fact looking for NOC engineers and process people. Senior engineers in all areas: Linux, DBA, Security, Performance, Troubleshooting, tools, managers and much more. We are building the world's top MSP and running numerous multi-hundred mullion user systems, doing the most difficult things on the Internet today.
I know you are probably being a bit facetious, but our career site:
http://careers.chinanetcloud.c...
Hmm, as the OP I value Slashdot's input and ideas on these things.
Our life and what we do is a tad more complicated than most others, in fact, quite a bit more complex than anyone I talk to, and despite my and our decades of experience in these areas, and sustained global searches for solutions, we often have to invent our own systems and technology - you'll see more of this from us over the next 24 months as we open source our best Ops and Management tools.
By the way, my thread on password management resulted in nothing useful as seems what we need does not exist. Good SaaS opportunity, I think. And that's our small password issue, we have much larger and more challenging security challenges that need world-class solutions we may have to yet again invent.
In this case, we have limited experience on modestly large NOCs and what people are doing for the PC selection, mounting, wiring, etc. as this is not our area, hence asking all of you for your input - and lots of good ideas and thoughts here - we'll post pictures and diagrams of what we end up with.
OP here and agreed - we are using professional display room vendor and thus the screens are commercial duty 7x24, mounted on modular walls. Though realistically screens are cheap enough that replacing them over time is not a huge burden, as long as we can manage the sizes which is my biggest worry (i.e. new 42" screen same size as old 42" or 41", etc.)
OP here and the goal is both though if we have to trade-off, good for tours is more important, but I can have both, frankly as we've been doing this many years and this is just a new upgrade. We run our primary and secondary monitoring systems up on the screens, active ticket lists, rule-based alerting on both business and tech stuff, action plan status, notice-of-the-day info and changes, change controls, active engineer work and ssh, email and IM session/ticket tracking, and a lot more across all these systems.
This is where we do real-time analysis, task routing, ticket management, communications and escalation, and more - 10-15 people will work in this room across 5+ areas 7x24 (Support, Coordination, Alerts, Requests, Security, Performance, DBA, Escalation, Scheduling, Leads and Managers).
The NOC handles about 1,000 events/week (in addition to the automated systems, tasks, responses) so it's a very busy place. Our core SLAs are 5-15 minutes for hundreds of customers and thousand of servers/systems.
Plus for emergencies we can/want to route laptops to big screens for shared team work so we have several meeting tables in the room to test this process (the offices have other dedicated rooms for this, too).
As the original poster, I want both - yes, we are in China and flashy tours of shiny command centers are helpful to customers who see this as professional - we have one now and it is very useful in the tour and sales pitch. Amazingly effective, actually, hence why I'll invest in it.
You are correct that the actual usefulness of lots of monitors are limited, though we do have lots of systems and info to display, dashboards, alert and rule systems, and much more so there is actual info architecture here, too - and we'll run some emergencies from the NOC also where more displays routed from laptops for team troubleshooting is helpful.
Beyond that, actually this is hardly a dingy area but in fact a Class A full-buildout with state-of-the-art offices (though open plan for large teams), glass walls, soft lighting, large cafe with game areas, rest areas, private phone/1:1 rooms, and more - think Google and AT&T, rather than dingy cubes.
Original Poster here - yes, these are all good suggestions and we should add more LDAP (we have large multi-thousand host LDAP systems now), but a lot, if not most of these systems we need, especially various SaaS tools, don't support this well, if at all. So a full SSO system is a real challenge - we are looking at AD integration next year to handle the ones that can.
But I don't really need this today - what I need is to TRACK all the system access, in part just to know what systems Johnny in Ops Engineering, etc. needs access to at what level, to notify the system owners to add/remove that, to track who added access and when, etc. as this happens over several days/weeks for new employees.
And to manage changes, which are of course frequent as this fall we add at least one new system per week - the cloud and SaaS is great, but managing users is not (assuming the system owner even reads the docs, manuals, sets roles correctly, etc.).
Today we have a huge XLS for this with common all-employee systems like HR, ERP, Email, etc. then per department blocks, then per role, then special stuff. It's pages long, and each item ties to an SOP, system access owner, etc.
And this is all just business systems, totally separate from our customers' operational systems, AWS/Alibaba/Rackspace/etc. IAM integrations, and our real work, which is totally separated and managed differently (hence the big LDAP systems, ticket integration, password managers, etc.)
So thinking we need to build a basic auth-like system but just that tracks users, roles, systems, roles in those systems, requests, approvals, changes, etc. But would have hoped this already existed.
I don't think I agree with his analysis of writing tests first; or rather, I believe he misses the larger point. Yes, programmers will tend to write tests that mirror the programs they are testing, but I find they also tend to force developers to at least think about testing, what's correct, and how things fail. It's not perfect and if everything is well-designed it may not matter, but in real applications, writing tests first can cut out at least a few code/run/fix cycles.